Pith. sign in

REVIEW 2 cited by

To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2107.10821 v2 pith:U6BNUWMH submitted 2021-07-22 cs.CL

Tom Kocmi , Christian Federmann , Roman Grundkiewicz , Marcin Junczys-Dowmunt , Hitokazu Matsushita , Arul Menezes This is my paper · ORCID
classification cs.CL
keywords metricshumanjudgementsautomatictranslationcollectiondeploymentevaluation
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Automatic metrics are commonly used as the exclusive tool for declaring the superiority of one machine translation system's quality over another. The community choice of automatic metric guides research directions and industrial developments by deciding which models are deemed better. Evaluating metrics correlations with sets of human judgements has been limited by the size of these sets. In this paper, we corroborate how reliable metrics are in contrast to human judgements on -- to the best of our knowledge -- the largest collection of judgements reported in the literature. Arguably, pairwise rankings of two systems are the most common evaluation tasks in research or deployment scenarios. Taking human judgement as a gold standard, we investigate which metrics have the highest accuracy in predicting translation quality rankings for such system pairs. Furthermore, we evaluate the performance of various metrics across different language pairs and domains. Lastly, we show that the sole use of BLEU impeded the development of improved models leading to bad deployment decisions. We release the collection of 2.3M sentence-level human judgements for 4380 systems for further analysis and replication of our work.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Distilling agreement-filtered multi-LRM jury annotations into Gemma-3-12B improves MQM translation quality evaluation from 52.63% to 55.03% average segment-level accuracy, approaching closed LRMs.

  2. Towards a Science of Causal Interpretability in Deep Learning for Software Engineering

    cs.SE 2025-05 conditional novelty 5.0 of 10

    The dissertation presents docode, a causal interpretability method for neural code models, and uses a case study to show that some correlations between code properties and model performance are confounded rather than causal.

Pith tools