REVIEW 3 major objections 4 minor 40 references
On The Evaluation of Machine Translation Systems Trained With Back-Translation
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper shows back-translation improves translation quality for natural and translationese inputs alike, while BLEU stays blind to the gain.
desk verdict Back-translation's quality gains are real but BLEU-blind; the cross-direction comparisons need a model-config clarification before fully trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central instrument is a paired test design built from double translations: each source-original sentence $X$ is paired with a translationese version $X^{**}$ that maps to the same reference $Y^*$, so the effect of translationese input can be isolated while content is held fixed, and the same construction is used on the reverse side. Around this, the paper wraps a pair of Transformer language models trained on the outputs of the back-translation and bitext-only systems, and uses their perplexity scores (lower means closer to the text type being scored) as a probe for closeness to translationese versus natural text. The probe produces the load-bearing numbers: on translationese references the two systems score nearly identically (perplexity 37.2 for the bitext-only system, 36.8 for the back-translation system), while on naturally occurring target text the back-translation output is far closer (82.2 versus 57.4), identifying fluency as the hidden dimension that human judges perceive and BLEU cannot measure.
What would settle it
Take a held-out test set where every source is naturally occurring and the human reference is also naturally occurring target text instead of translationese; the paper's account predicts back-translation still wins with professional judges and still receives lower language-model perplexity. If either prediction reverses, the central claim about human preference or the fluency explanation would be refuted.
Extended reading notes
Core claim
On its own terms, the paper establishes that back-translation improves translation quality on both parts of a test set—source-original and target-original sentences—and that the appearance of a reverse-only benefit is an artifact of how quality is measured. Using a paired design in which the same human reference is shared by a natural source and its double-translated version, the authors show that translationese inputs are easier for both systems but give no special advantage to back-translation. Professional translators, rating outputs without seeing the reference, prefer back-translation output in every language direction tested. The reason BLEU cannot reflect this is that the outputs of the back-translation and bitext-only systems are equally close to translationese, so n-gram overlap with a translationese reference cannot separate them; the back-translation output is, however, much closer to natural target text, which is what the human judges reward. The paper consequently recommends distinguishing direct and reverse test portions and complementing BLEU with a language-model perplexity score that measures fluency.
Load-bearing premise
The explanation for BLEU's blind spot assumes the two language models used to measure 'closeness to translationese' are equally well calibrated, so their nearly equal scores on translationese references genuinely mean both systems' outputs are equally translationese-like; if one language model is simply better, the measured tie could be an artifact.
Editorial extensions
If this is right
- Back-translation improves quality for naturally occurring inputs as well as translationese, so the common restriction of test sets to source-original sentences will not erase its advantage.
- Automatic metrics built on n-gram overlap with translationese references will systematically underestimate the quality of a more fluent system, so BLEU ties should not be read as quality ties.
- Adding a language-model fluency score, computed on held-out naturally occurring monolingual data, separates the fluency gain human judges see from the adequacy signal BLEU already captures.
- Because both systems benefit equally from translationese inputs, the large reverse-direction BLEU gain of back-translation is not explained by translationese simplicity; it is better attributed to the training-data match.
Reading between the lines
- The same reference-dialect blind spot should affect other augmentation techniques that push outputs toward natural target text, such as tagged back-translation or denoising-based generation, so their reported BLEU gains may be understated for the same reason.
- A concrete extension is to rank systems on two axes—BLEU for adequacy and held-out language-model perplexity for fluency—turning the recommendation into a two-dimensional evaluation standard that could replace single-number leaderboards.
- If the fluency explanation is right, the gap between BLEU and human judgment should widen as back-translated data scales up, a prediction that could be checked against published systems of different sizes.
- The conceptual point extends beyond machine translation: any evaluation that compares generated text against reference text that is itself translated or otherwise non-representative may inherit this kind of metric blindness.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies why back-translation (BT) improves machine translation quality according to human judges even though automatic metrics such as BLEU show gains only on target-original (reverse) test sentences. Using four language directions (En-De, De-En, En-Ru, Ru-En), the authors compare models trained on parallel data only (OP) with models trained on parallel plus back-translated data (BT). They collect sentence-level human references and double translations, and they run professional human evaluations with both source-based and target-based direct assessment. The main findings are: (1) professional translators prefer BT over OP both for source-original and target-original test sentences, while BLEU does not distinguish the systems on source-original sentences; (2) translationese inputs are easier to translate for both OP and BT, so this does not explain the BLEU discrepancy; (3) language-model perplexities suggest that BT and OP outputs are equally close to translationese, which the authors argue explains why BLEU fails in direct mode; and (4) BT outputs are judged more fluent and receive lower perplexity under a held-out Common Crawl language model. The paper concludes by recommending that BLEU be complemented with a language-model fluency score when human evaluation is unavailable.
Significance. If the results hold, the paper makes a valuable empirical contribution to MT evaluation. It provides new human-annotated reference sets, demonstrates with professional translators that BLEU under-reports the quality of back-translation systems on direct sentences, and offers a concrete, practical recommendation to add a language-model score. The study is broader than most prior work: it covers four language pairs, uses source- and target-based human assessment, applies bootstrap significance testing, and uses a held-out Common Crawl language model that is disjoint from the back-translation data. The public release of the additional references and the use of standard fairseq models are strengths that support reproducibility. The central limitation is that the causal claim about back-translation is potentially confounded by model-capacity and ensembling differences for three of the four language directions.
major comments (3)
- [3.2] The OP versus BT comparison is not demonstrably matched for De-En, En-Ru, and Ru-En. Section 3.2 states that for De-En and En-Ru the authors "increased model capacity by using larger FFN size (8192) and we also used an ensemble of models trained with three different seeds," but it does not state that the OP baselines for these directions use the same capacity and ensembling, and Ru-En is not mentioned at all. Since the BT systems are the top-ranked WMT'19 FAIR systems while the OP baselines are only described as trained on parallel data, the human preferences in Table 3 and the fluency judgments in Table 5 for three of four directions may reflect model capacity, checkpoint averaging, or ensembling rather than the presence of back-translated data. The authors should either document that OP and BT are identical in architecture and training recipe except for the data, or add a matched-control ablation that isolates the effect of back-translation.
- [4.5, Table 4] The claim that BT and OP outputs are equally close to translationese rests on comparing perplexities from two different language models, one trained on BT outputs and one on OP outputs (37.2 versus 36.8 on Y*). This comparison is only valid if the two language models generalize equally well on the evaluation sets, and the paper provides no evidence for that assumption. A single held-out language model applied to both system outputs, or a cross-perplexity check in which each LM is scored on both output sets, is needed to support the abstract's claim that BLEU fails because the outputs of both systems are equally close to the translationese references.
- [5, Table 6] The proposed language-model fluency metric is reported without significance tests or confidence intervals, and the statement that BT "attains the same BLEU score" as OP is not strictly supported by Table 1: for En-De, De-En, and Ru-En the BT direct-mode BLEU is numerically lower than OP (32.3 versus 33.7, 38.6 versus 40.3, and 41.2 versus 43.8, respectively). The claim that the LM score breaks BLEU ties should be supported by statistical testing or softened to reflect the observed BLEU differences.
minor comments (4)
- [1] There are several typos: "signifcanlty" in the introduction, "disproof" for "disprove," and "translationes inputs" in the Table 2 caption.
- [3.4] The human evaluation section does not report the number of raters per language direction, the number of sentences actually retained after applying the 30-point disagreement criterion, or confidence intervals for the reported z-scores; adding this information would strengthen the significance claims.
- [4.4] The correlation of 0.90 between source-based and target-based assessment has a very wide 95% confidence interval (0.55 to 0.98), probably because it is computed over a small number of systems; the paper should acknowledge this limited precision.
- [References] The reference list contains duplicate entries for Poncelas et al. 2018a and 2018b, which appear to be the same paper.
Circularity Check
No significant circularity: the paper's human-preference and fluency findings are independent empirical measurements, not derivations from their own inputs.
full rationale
The paper's central claims are empirical measurements: professional human translators prefer BT over OP in blinded, randomized evaluations (Section 3.4, Table 3), and native speakers judge BT outputs more fluent in pairwise comparisons (Table 5). These judgments are independent of both BLEU and the language-model analyses. The explanation for BLEU's failure is tested, not assumed: Section 4.5 trains separate LMs on BT and OP outputs and evaluates them on held-out translationese (Y*) and natural text (Y) portions of newstest; the perplexities in Table 4 are measurements on held-out data, not fitted parameters repackaged as predictions. The proposed evaluation metric in Section 5 is explicitly guarded against circularity: the LM is trained on Common Crawl data 'disjoint from the monolingual dataset used for generating back-translated data for BT training,' with the paper noting the restriction is 'critical, otherwise the language model is likely to assign higher probably to BT generations simply because training and evaluation sets overlap.' Self-citations to the authors' earlier WMT systems (Edunov et al., 2018; Ng et al., 2019) supply the systems under test and the back-translation recipe; they do not by themselves justify the paper's conclusions, which rest on evaluations performed in this work. No uniqueness theorem is imported, no ansatz is smuggled in by citation, and no known result is merely renamed. The possible ambiguity about whether capacity increases and ensembling in Section 3.2 apply to both OP and BT is an experimental confound, not a circularity, because the paper does not define the systems' equivalence into existence; it measures their outputs. Overall, the derivation chain is self-contained against external benchmarks and the central claim does not reduce to its inputs by construction.
Assumptions & free parameters
assumptions (4)
- domain assumption Translationese is simpler and easier to translate than naturally occurring text.
- domain assumption The double translations X** and Y** are faithful translationese versions of X and Y that preserve meaning.
- domain assumption Language model perplexity on a held-out corpus is a valid proxy for closeness to naturally occurring text.
- domain assumption Professional human translators are the gold standard for translation quality.
Cite this review
Pith. "Pith review of On The Evaluation of Machine Translation Systems Trained With Back-Translation." pith.science (2026). https://pith.science/paper/IDB3LIOT
@misc{pith2026190805204,
author = {Pith},
title = {Pith review of: On The Evaluation of Machine Translation Systems Trained With Back-Translation},
year = {2026},
howpublished = {\url{https://pith.science/paper/IDB3LIOT}},
note = {Machine review of arXiv:1908.05204}
}
read the original abstract
Back-translation is a widely used data augmentation technique which leverages target monolingual data. However, its effectiveness has been challenged since automatic metrics such as BLEU only show significant improvements for test examples where the source itself is a translation, or translationese. This is believed to be due to translationese inputs better matching the back-translated training data. In this work, we show that this conjecture is not empirically supported and that back-translation improves translation quality of both naturally occurring text as well as translationese according to professional human translators. We provide empirical evidence to support the view that back-translation is preferred by humans because it produces more fluent outputs. BLEU cannot capture human preferences because references are translationese when source sentences are natural text. We recommend complementing BLEU with a language model score to measure fluency.
Figures
Reference graph
Works this paper leans on
-
[1]
Alexei Baevski and Michael Auli. 2018. Adaptive input representations for neural language modeling. arXiv, abs/1809.10853
arXiv 2018
-
[2]
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR
2015
-
[3]
Mona Baker. 1993. Corpus linguistics and translation studies: Implications and applications. Text and technology: In honour of John Sinclair, 233:250
work page 1993
-
[4]
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In In Proc. of ACL Workshop on Intrinsic and Extrinsic Evaluation Measure for Machine Translation
work page 2005
-
[5]
Ondrej Bojar and Ales Tamchyna. 2011. Improving translation model by monolingual data. In Proc. of WMT
work page 2011
-
[6]
Ond r ej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation ( WMT18 ). In Proc. of WMT
work page 2018
-
[7]
Ond r ej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2019. Findings of the 2019 conference on machine translation (wmt19). In Proc. of WMT
work page 2019
-
[8]
Isaac Caswell, Ciprian Chelba, and David Grangier. 2019. Tagged back-translation. In Proc. of WMT
work page 2019
Show all 40 references
-
[9]
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proc. of EMNLP
2018
-
[10]
Markus Freitag, Isaac Caswell, and Scott Roy. 2019. Text repair model for neural machine translation. arXiv, abs/1904.04790
2019 arXiv
-
[11]
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional Sequence to Sequence Learning . In Proc. of ICML
2017
-
[12]
Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2017. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, 23(1):3–30
2017
-
[13]
Yvette Graham, Barry Haddow, and Philipp Koehn. 2019. Translationese in machine translation evaluation. arXiv, abs/1906.09833
2019 arXiv
-
[14]
Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. 2018. Iterative back-translation for neural machine translation. In Proc. of 2nd Workshop on Neural Machine Translation and Generation
2018
-
[15]
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proc. of EMNLP
2004
-
[16]
Moshe Koppel and Noam Ordan. 2011. Translationese and its dialects. In Proc. of ACL
2011
-
[17]
David Kurokawa, Cyril Goutte, and Pierre Isabelle. 2009. Automatic detection of translated text and its impact on machine translation. In Proc. of MT Summit
2009
-
[18]
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018. Phrase-based and neural unsupervised machine translation. In EMNLP
2018
-
[19]
Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2011. Language models for machine translation: Original vs. translated texts. In Proc. of EMNLP
2011
-
[20]
Marco Lui and Timothy Baldwin. 2012. langid. py: An off-the-shelf language identification tool. In Proc. of ACL: Demonstrations
2012
-
[21]
Qingsong Ma, Johnny Wei, Ondrej Bojar, and Yvette Graham. 2019. Results of the wmt19 metrics shared task: Segment-level and strong mt systems pose big challenges. In Proc. of WMT
2019
-
[22]
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook fair's wmt19 news translation task submission. In Proc. of WMT
2019
-
[23]
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. https://doi.org/10.18653/v1/N19-4009 fairseq: A fast, extensible toolkit for sequence modeling . In Proc. of NAACL: Demonstrations
2019 doi
-
[24]
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proc. of WMT
2018
-
[25]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proc. of ACL
2002
-
[27]
Shterionov, Andy Way, Gideon Maillette de Buy Wenniger, and Peyman Passban
Alberto Poncelas, Dimitar Sht. Shterionov, Andy Way, Gideon Maillette de Buy Wenniger, and Peyman Passban. 2018 b . Investigating backtranslation in neural machine translation. arXiv, 1804.06189
2018 arXiv
-
[28]
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proc. of WMT
2018
-
[29]
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 a . Improving neural machine translation models with monolingual data. In Proc. of ACL
2016
-
[30]
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 b . Neural machine translation of rare words with subword units. In Proc. of ACL
2016
-
[31]
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proc. of AMTA
2006
-
[32]
Milos Stanojevic and Khalil Sima'an. 2014. Beer: Better evaluation as ranking. In Proc. of WMT
2014
-
[33]
Sara Stymne. 2017. The effect of translationese on tuning for statistical machine translation. In Proc. of NoDaLiDa
2017
-
[34]
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proc. of NIPS
2014
-
[35]
Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the unattainable? reassessing claims of human parity in neural machine translation. In Proc. of WMT
2018
-
[36]
Gideon Toury. 2012. Descriptive translation studies and beyond: Revised edition, volume 100. John Benjamins Publishing
2012
-
[37]
Gomez, Lukasz Kaiser, and Illia Polosukhin
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need . In Proc. of NIPS
2017
-
[38]
Mike Zhang and Antonio Toral. 2019. The effect of translationese in machine translation test sets. arXiv, abs/1906.08069
2019 arXiv
-
[39]
Weinberger, and Yoav Artzi
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv, abs/1904.09675
2019 arXiv
-
[40]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...
-
[41]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.