REVIEW 21 cited by
COMET: A Neural Framework for MT Evaluation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We present COMET, a neural framework for training multilingual machine translation evaluation models which obtains new state-of-the-art levels of correlation with human judgements. Our framework leverages recent breakthroughs in cross-lingual pretrained language modeling resulting in highly multilingual and adaptable MT evaluation models that exploit information from both the source input and a target-language reference translation in order to more accurately predict MT quality. To showcase our framework, we train three models with different types of human judgements: Direct Assessments, Human-mediated Translation Edit Rate and Multidimensional Quality Metrics. Our models achieve new state-of-the-art performance on the WMT 2019 Metrics shared task and demonstrate robustness to high-performing systems.
Forward citations
Cited by 21 Pith papers
-
GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark
GigaSpeechBench is a new 680-hour in-the-wild multilingual ASR/AST benchmark with five modules for low-resource languages, Chinese dialects, English accents, domain terminology, and age-varied speech, showing model pe...
-
Translate With Care: Addressing Gender Bias, Neutrality, and Reasoning in Large Language Model Translations
A new genderless-to-English benchmark shows that fine-tuning mBART-50 on carefully curated examples cuts gender stereotyping and pronoun-reasoning errors, beating larger proprietary systems on that benchmark.
-
From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set
Using per-example in-context demonstrations built from historical same-source human MQM ratings makes an LLM judge dramatically better at fine-grained MT evaluation on WMT'23 and WMT'24.
-
Self-evolving Agentic Customer Support System at LinkedIn
A production customer-support agent that continuously evolves its prompts and retrieval in a closed evaluation loop improved live self-serve resolution and routing accuracy at LinkedIn.
-
Building Large-Scale English-Romanian Literary Translation Resources with Open Models
LoRA-fine-tuning a 12B open model on 15,000 GPT-o3 fable translations yields rubric scores close to GPT-o3 (4.83 vs 4.92) for English-to-Romanian literary translation at roughly one percent of the API cost.
-
Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voice
An end-to-end simultaneous speech-to-speech translation model with voice cloning, trained with a two-stage reinforcement learning reward scheme, reports high accuracy and low latency on the authors' RealSI benchmark.
-
Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive Summarization
Across eight languages, n-gram metrics such as ROUGE correlate less with human ratings in fusional languages than in isolating and agglutinative ones, while the neural metric COMET correlates better, especially in low...
-
SeqPO-SiMT: Sequential Policy Optimization for Simultaneous Machine Translation
SeqPO-SiMT uses sequential policy optimization with a combined quality-and-latency reward to improve simultaneous machine translation, beating supervised fine-tuning on six En-Zh and Zh-En datasets.
-
Word Level Timestamp Generation for Automatic Speech Recognition and Translation
The paper teaches the Canary ASR and speech-translation model to output word-level start and end timestamps directly using forced-alignment teacher labels.
-
A 2-step Framework for Automated Literary Translation Evaluation: Its Promises and Pitfalls
A rubric-plus-question-answering LLM framework for literary translation evaluation beats traditional MT metrics but still trails human agreement, especially on Korean honorifics.
-
Reasoning Before Translation: Enhancing Legal Machine Translation with Structured Reasoning
On Swiss legal translation, reinforcement learning with a ChrF reward improves small open models more than supervised fine-tuning, but frontier reasoning models still score higher.
-
Introducing the Swiss Food Knowledge Graph: AI for Context-Aware Nutrition Recommendation
The paper introduces SwissFKG, a knowledge graph integrating Swiss recipes, nutrients, allergens, and dietary guidelines, populated via an LLM pipeline and used for a Graph-RAG question answering demo.
-
Psychology-Driven Enhancement of Humour Translation
A decomposition-and-recomposition prompt method for humor translation reports large gains on LLM-based metrics, but the evaluation lacks human validation and statistical checks.
-
An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability
The reliability of LLM-as-a-Judge depends strongly on scoring rubrics and reference answers; sampling with averaging outperforms greedy decoding, and chain-of-thought reasoning adds little when rubrics are clear.
-
Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMs
Selective pre-translation, translating only some prompt components into English, generally outperforms both full prompt translation and direct inference across tasks and languages, with the largest gains for low-resou...
-
Style-agnostic evaluation of ASR using multiple reference transcripts
Multireference scoring with verbatim and nonverbatim human transcripts cuts measured WER for state-of-the-art ASR models by about half, but the reduction is partly guaranteed by the math and not validated against cont...
-
CHAI for LLMs: Improving Code-Mixed Translation in Large Language Models through Reinforcement Learning with AI Feedback
CHAI trains a reward model on GPT-4o preference labels and uses PPO to align Llama-3.1-8B for English-to-Hinglish translation, claiming a 25.66% human win-rate improvement over baselines.
-
From Sentences to Sequences: Rethinking Languages in Biological System
A new RNA inverse folding model (RiFold) using stochastic-order decoding and structure-aware metrics outperforms prior methods, and the paper shows sequence recovery and structural recovery are correlated but not equivalent.
-
Evaluating Machine Translation Models for English-Hindi Language Pairs: A Comparative Analysis
Google Translate outperforms NLLB-200, OPUS-MT, and IndicTrans2 on English-Hindi translation on 18,000+ parallel sentences and a 400-question FAQ corpus, with all models degrading as sentence length grows.
-
Comparative analysis of subword tokenization approaches for Indian languages
Across SMT and NMT models for 11 Indian languages, SentencePiece gives the highest BLEU for most language pairs, while BPE wins in the multilingual model.
-
TransBench: Benchmarking Machine Translation for Industrial-Scale Applications
TransBench is a proposed e-commerce MT benchmark with a three-level evaluation framework and a fine-tuned quality-scoring model, but the paper contains no results and no released data or code.
Discussion (0). Continue with ORCID to comment.