REVIEW 18 cited by
Large Language Models Are State-of-the-Art Evaluators of Translation Quality
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We describe GEMBA, a GPT-based metric for assessment of translation quality, which works both with a reference translation and without. In our evaluation, we focus on zero-shot prompting, comparing four prompt variants in two modes, based on the availability of the reference. We investigate nine versions of GPT models, including ChatGPT and GPT-4. We show that our method for translation quality assessment only works with GPT~3.5 and larger models. Comparing to results from WMT22's Metrics shared task, our method achieves state-of-the-art accuracy in both modes when compared to MQM-based human labels. Our results are valid on the system level for all three WMT22 Metrics shared task language pairs, namely English into German, English into Russian, and Chinese into English. This provides a first glimpse into the usefulness of pre-trained, generative large language models for quality assessment of translations. We publicly release all our code and prompt templates used for the experiments described in this work, as well as all corresponding scoring results, to allow for external validation and reproducibility.
Forward citations
Cited by 18 Pith papers
-
From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set
Using per-example in-context demonstrations built from historical same-source human MQM ratings makes an LLM judge dramatically better at fine-grained MT evaluation on WMT'23 and WMT'24.
-
TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation
Distilling agreement-filtered multi-LRM jury annotations into Gemma-3-12B improves MQM translation quality evaluation from 52.63% to 55.03% average segment-level accuracy, approaching closed LRMs.
-
Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems
AgentLocate localizes multi-agent LLM failures to a responsible agent and earliest decisive step via judge hypotheses, confidence-weighted multi-evaluator verification, and LoRA refinement.
-
How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAG
A new GraphRAG evaluation framework using graph-grounded questions and bias-correction yields much smaller win rates than earlier reports, casting doubt on reported GraphRAG gains.
-
LLM as Effective Streaming Processor: Bridging Streaming-Batch Mismatches with Group Position Encoding
Streaming LLMs need only fix the input-attention mismatch; group position encoding (source and target positions numbered separately) removes the need for re-encoding and outperforms specialized streaming baselines.
-
Generalised Probabilistic Modelling and Improved Uncertainty Estimation in Comparative LLM-as-a-judge
A generalised Product-of-Experts framework with a new 'probability of reordering' selection metric that reduces the number of LLM comparisons needed for ranking by about 50%.
-
LLM Sensitivity Evaluation Framework for Clinical Diagnosis
An evaluation framework and dataset show that GPT-4 and other LLMs frequently fail to adjust diagnoses when key patient information is perturbed, achieving only 5.28% accuracy on such changed cases.
-
Hierarchical Divide-and-Conquer for Fine-Grained Alignment in LLM-Based Medical Evaluation
HDCEval splits medical answer grading into relevance, correctness, and expression checks, uses reward-token-trained expert models, and reports improved agreement with human doctors.
-
RecSys Arena: Pair-wise Recommender System Evaluation with Large Language Models
LLM judges role-playing as users rank two recommender systems pairwise, producing orderings that track AUC and, the authors argue, sharper separation between closely matched systems.
-
LLMPirate: LLMs for Black-box Hardware IP Piracy
LLMPirate uses eight LLMs to rewrite Verilog netlists into functionally equivalent circuits that evade GNN4IP, MOSS, Jplag, and SIM on most tested designs.
-
Psychology-Driven Enhancement of Humour Translation
A decomposition-and-recomposition prompt method for humor translation reports large gains on LLM-based metrics, but the evaluation lacks human validation and statistical checks.
-
Can LLMs Play \^O \u{A}n Quan Game? A Study of Multi-Step Planning and Decision Making
In 50-game matches, an 8-billion-parameter Llama beat a 70-billion-parameter Llama more often than it lost, while larger models generated longer reasoning traces but not reliably better scores.
-
Thinking Short and Right Over Thinking Long: Serving LLM Reasoning Efficiently and Accurately
SART reduces LLM serving latency by combining redundant branch sampling with early stopping and PRM-based dynamic branch pruning, while keeping answer accuracy comparable to baseline methods.
-
LLM-Based Evaluation of Low-Resource Machine Translation: A Reference-less Dialect Guided Approach with a Refined Sylheti-English Benchmark
A dialect-guided prompt with tokenizer extension and a regression head improves reference-less machine translation evaluation for Sylheti-English, with best Spearman gain +0.1083 over an annotation-guided baseline.
-
Can Large Language Models Serve as Evaluators for Code Summarization?
An LLM prompt that makes the model role-play a code reviewer scores code summaries with 81.59% Spearman correlation with human judgment, outperforming BLEU and BERTScore on a 300-sample benchmark.
-
Large Language Models for Security Operations Centers: A Comprehensive Survey
A systematic review of 138 papers classifying LLM applications in SOC workflows by phase, model family, datasets, and maturity.
-
Comparing Large Language Models and Traditional Machine Translation Tools for Translating Medical Consultation Summaries: A Pilot Study
In a 34-translation pilot, Google Translate, Bing, and DeepL generally beat GPT-4o, LLAMA-3.1, and GEMMA-2 on BLEU, CHR-F, and METEOR for medical consultation summaries, with LLMs strongest for Vietnamese and Chinese ...
-
Personalizing Education through an Adaptive LMS with Integrated LLMs
The paper presents a hybrid expert-system and LLM adaptive LMS prototype and a benchmark of ten LLMs on standardized tests, showing self-hosted models are competitive with proprietary ones in reading, writing, and cod...
Discussion (0). Continue with ORCID to comment.