{"id":"a0b51c68-55a5-481f-b7c2-83590cd50626","arxiv_id":"1908.05204","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Back-translation produces more fluent, human-preferred output even when BLEU is flat, so evaluation should combine BLEU with a language model score.","lead":"A large-scale human evaluation shows that back-translation improves machine translation quality for both natural text and translationese, overturning a widely cited conjecture. The same study explains why BLEU scores hide this gain and recommends adding a language model fluency score to automatic evaluation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BT/OP comparison may be confounded by model capacity and ensembling for three of four language pairs; the human-preference claim needs a matched-control check.","rationale":"The reader's weakest assumption concerns the language-model perplexity comparison in Section 4.5, which affects only the proposed explanation of why BLEU fails. My concern instead targets the primary human-preference result itself: if the OP and BT systems are not matched in capacity, ensemble size, or decoding recipe, the observed human preference may not be attributable to back-translation. The manuscript does not provide enough detail in Sections 3.1-3.2 to rule this out, and the Related Work claim of comparing 'the same NMT system' is not backed by a table or explicit statement of matching configurations for all four directions. Since the paper's headline claim is causal and the human evaluation is its main evidence, this potential confound is load-bearing. The appropriate response is not rejection; the human evaluation and controlled En-De setup are valuable. Rather, the paper should be accepted only conditional on demonstrating that the OP baselines match the BT systems in model capacity, ensembling, and tuning, or on rerunning the comparison with a capacity-matched baseline.","tokens_in":1461,"tokens_out":1252,"duration_ms":84043,"concrete_test":"For at least one affected direction (e.g., Ru-En), train an OP baseline using the exact WMT19 BT recipe: same Transformer Big FFN size, same number of seeds, same ensemble size, and same checkpoint averaging, with identical data filtering and preprocessing. Then rerun the Section 3.4 source-based direct assessment on the direct portion. If BT remains significantly preferred over this capacity-matched OP, the causal claim survives; if the preference shrinks or disappears, the reported BT gain is confounded.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central causal claim that back-translation improves human-rated quality requires that BT and OP systems differ only in the use of back-translated data. Sections 3.1-3.2 do not establish this for De-En, En-Ru, and Ru-En. BT models are the top-ranked WMT'19 FAIR systems, while OP baselines are described only as trained on parallel data. Section 3.2 states that for De-En and En-Ru 'we increased model capacity by using larger FFN size (8192) and we also used an ensemble of models trained with three different seeds,' without specifying whether this recipe applies to both OP and BT or only to one. If the WMT'19 BT systems use a larger FFN, checkpoint averaging, and/or a multi-seed ensemble while the OP baselines do not, Table 3's human preferences (and Table 5's fluency judgments) could reflect capacity or ensembling rather than back-translation. Near-equal direct BLEU scores do not control for this, since larger or ensembled systems can match BLEU while producing more fluent output. The Related Work assertion that the comparison is 'the same NMT system trained with and without back-translated data' is not operationalized in the experimental section, so the core human-preference finding is at risk for three of four language directions.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies why back-translation (BT) improves machine translation quality according to human judges even though automatic metrics such as BLEU show gains only on target-original (reverse) test sentences. Using four language directions (En-De, De-En, En-Ru, Ru-En), the authors compare models trained on parallel data only (OP) with models trained on parallel plus back-translated data (BT). They collect sentence-level human references and double translations, and they run professional human evaluations with both source-based and target-based direct assessment. The main findings are: (1) professional translators prefer BT over OP both for source-original and target-original test sentences, while BLEU does not distinguish the systems on source-original sentences; (2) translationese inputs are easier to translate for both OP and BT, so this does not explain the BLEU discrepancy; (3) language-model perplexities suggest that BT and OP outputs are equally close to translationese, which the authors argue explains why BLEU fails in direct mode; and (4) BT outputs are judged more fluent and receive lower perplexity under a held-out Common Crawl language model. The paper concludes by recommending that BLEU be complemented with a language-model fluency score when human evaluation is unavailable.","tokens_in":11304,"tokens_out":4086,"duration_ms":41612,"significance":"If the results hold, the paper makes a valuable empirical contribution to MT evaluation. It provides new human-annotated reference sets, demonstrates with professional translators that BLEU under-reports the quality of back-translation systems on direct sentences, and offers a concrete, practical recommendation to add a language-model score. The study is broader than most prior work: it covers four language pairs, uses source- and target-based human assessment, applies bootstrap significance testing, and uses a held-out Common Crawl language model that is disjoint from the back-translation data. The public release of the additional references and the use of standard fairseq models are strengths that support reproducibility. The central limitation is that the causal claim about back-translation is potentially confounded by model-capacity and ensembling differences for three of the four language directions.","major_comments":[{"comment":"The OP versus BT comparison is not demonstrably matched for De-En, En-Ru, and Ru-En. Section 3.2 states that for De-En and En-Ru the authors \"increased model capacity by using larger FFN size (8192) and we also used an ensemble of models trained with three different seeds,\" but it does not state that the OP baselines for these directions use the same capacity and ensembling, and Ru-En is not mentioned at all. Since the BT systems are the top-ranked WMT'19 FAIR systems while the OP baselines are only described as trained on parallel data, the human preferences in Table 3 and the fluency judgments in Table 5 for three of four directions may reflect model capacity, checkpoint averaging, or ensembling rather than the presence of back-translated data. The authors should either document that OP and BT are identical in architecture and training recipe except for the data, or add a matched-control ablation that isolates the effect of back-translation.","section":"3.2"},{"comment":"The claim that BT and OP outputs are equally close to translationese rests on comparing perplexities from two different language models, one trained on BT outputs and one on OP outputs (37.2 versus 36.8 on Y*). This comparison is only valid if the two language models generalize equally well on the evaluation sets, and the paper provides no evidence for that assumption. A single held-out language model applied to both system outputs, or a cross-perplexity check in which each LM is scored on both output sets, is needed to support the abstract's claim that BLEU fails because the outputs of both systems are equally close to the translationese references.","section":"4.5, Table 4"},{"comment":"The proposed language-model fluency metric is reported without significance tests or confidence intervals, and the statement that BT \"attains the same BLEU score\" as OP is not strictly supported by Table 1: for En-De, De-En, and Ru-En the BT direct-mode BLEU is numerically lower than OP (32.3 versus 33.7, 38.6 versus 40.3, and 41.2 versus 43.8, respectively). The claim that the LM score breaks BLEU ties should be supported by statistical testing or softened to reflect the observed BLEU differences.","section":"5, Table 6"}],"minor_comments":[{"comment":"There are several typos: \"signifcanlty\" in the introduction, \"disproof\" for \"disprove,\" and \"translationes inputs\" in the Table 2 caption.","section":"1"},{"comment":"The human evaluation section does not report the number of raters per language direction, the number of sentences actually retained after applying the 30-point disagreement criterion, or confidence intervals for the reported z-scores; adding this information would strengthen the significance claims.","section":"3.4"},{"comment":"The correlation of 0.90 between source-based and target-based assessment has a very wide 95% confidence interval (0.55 to 0.98), probably because it is computed over a small number of systems; the paper should acknowledge this limited precision.","section":"4.4"},{"comment":"The reference list contains duplicate entries for Poncelas et al. 2018a and 2018b, which appear to be the same paper.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong empirical study, but the central causal claim about back-translation is currently supported by a clean matched comparison only for En-De. If the authors can confirm that the OP and BT systems are identical in architecture, capacity, and ensembling for all four directions, or add a matched control for the other three directions, the paper would be suitable for publication. Otherwise, the claims should be reframed as an observational comparison of deployed top-ranked systems rather than a controlled demonstration of the effect of back-translation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The double-translation test set is a genuinely clean design, and the human evaluation shows BT beats OP even where BLEU is flat. The main soft spot is a reporting gap: for three of four language pairs we are not explicitly told that the bigger FFN and three-seed ensemble apply to both OP and BT. That needs fixing before the cross-direction claims are fully trusted.\n\nWhat is new and good: they compare source-original X against translationese X** with the same reference, which directly tests the 'translationese is easier' story and shows both OP and BT benefit equally. The fluency explanation is well supported: LMs trained on outputs of BT and OP give similar perplexity on translationese references, but BT is much closer to natural text, and native-speaker fluency judgments agree. The proposed LM-as-fluency-companion is practical, and they were careful to train it on Common Crawl data disjoint from the back-translation monolingual set, so the metric is not circular. The human eval is credible: professional translators, source- and target-based assessments correlated at 0.90, three judgments per sentence, bootstrap significance.\n\nSoft spots: the stress-test concern about model capacity is not crazy. Section 3.2 says they increased FFN size and used an ensemble for De-En and En-Ru, but never explicitly says 'for both OP and BT.' If only the WMT'19 BT systems got the bigger config, part of the human preference could be capacity rather than back-translation. The text around it suggests the intended comparison is matched, and En-De, where the config is explicit and single-model, shows the same human preference. So it is a reporting gap, not a demonstrated flaw, but it deserves a one-paragraph fix. Minor: perplexity comparisons in Table 4 lack confidence intervals, so 37.2 vs 36.8 is treated as equal without uncertainty; and the BLEU+LM recommendation is directional, not validated against human judgments beyond its observed sign.\n\nWho is it for? Anyone working on MT evaluation or using back-translation in production. It is a serious empirical study worth refereeing. With the config clarification it becomes a solid paper.","headline":"Back-translation's quality gains are real but BLEU-blind; the cross-direction comparisons need a model-config clarification before fully trusted.","tokens_in":726,"tokens_out":1450,"would_cite":true,"duration_ms":57676,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows back-translation improves translation quality for natural and translationese inputs alike, while BLEU stays blind to the gain.","keywords":["back-translation","machine translation evaluation","BLEU","human evaluation","translationese","fluency","language model scoring","neural machine translation"],"falsifier":"Take a held-out test set where every source is naturally occurring and the human reference is also naturally occurring target text instead of translationese; the paper's account predicts back-translation still wins with professional judges and still receives lower language-model perplexity. If either prediction reverses, the central claim about human preference or the fluency explanation would be refuted.","tokens_in":10882,"feed_emoji":"📊","tokens_out":9993,"duration_ms":92812,"temperature":0.7,"pith_summary":"Back-translation—training a reverse model to synthesize extra parallel sentences from monolingual target text—improves translation quality judged by professional human translators for both naturally occurring source text and translationese, the paper shows. This overturns the widespread claim that back-translation's gains are confined to test sentences whose source is itself a translation. The paper traces BLEU's blindness to the fact that the reference translations used for natural-source test sentences are translationese, and that both a back-translation system and a bitext-only system produce outputs equally close to that translationese dialect. Professional translators nevertheless prefer the back-translation outputs because they are more fluent, closer to naturally written target text. The practical consequence is that automatic leaderboards built on translationese references under-report exactly the quality gain back-translation delivers, and a language-model fluency score should be added alongside BLEU.","feed_headline":"Human judges prefer back-translation; BLEU stays blind","feed_subtitle":"Translationese references hide the fluency boost, so a language-model score should complement BLEU.","key_machinery":"The central instrument is a paired test design built from double translations: each source-original sentence $X$ is paired with a translationese version $X^{**}$ that maps to the same reference $Y^*$, so the effect of translationese input can be isolated while content is held fixed, and the same construction is used on the reverse side. Around this, the paper wraps a pair of Transformer language models trained on the outputs of the back-translation and bitext-only systems, and uses their perplexity scores (lower means closer to the text type being scored) as a probe for closeness to translationese versus natural text. The probe produces the load-bearing numbers: on translationese references the two systems score nearly identically (perplexity 37.2 for the bitext-only system, 36.8 for the back-translation system), while on naturally occurring target text the back-translation output is far closer (82.2 versus 57.4), identifying fluency as the hidden dimension that human judges perceive and BLEU cannot measure.","core_discovery":"On its own terms, the paper establishes that back-translation improves translation quality on both parts of a test set—source-original and target-original sentences—and that the appearance of a reverse-only benefit is an artifact of how quality is measured. Using a paired design in which the same human reference is shared by a natural source and its double-translated version, the authors show that translationese inputs are easier for both systems but give no special advantage to back-translation. Professional translators, rating outputs without seeing the reference, prefer back-translation output in every language direction tested. The reason BLEU cannot reflect this is that the outputs of the back-translation and bitext-only systems are equally close to translationese, so n-gram overlap with a translationese reference cannot separate them; the back-translation output is, however, much closer to natural target text, which is what the human judges reward. The paper consequently recommends distinguishing direct and reverse test portions and complementing BLEU with a language-model perplexity score that measures fluency.","pith_inferences":["The same reference-dialect blind spot should affect other augmentation techniques that push outputs toward natural target text, such as tagged back-translation or denoising-based generation, so their reported BLEU gains may be understated for the same reason.","A concrete extension is to rank systems on two axes—BLEU for adequacy and held-out language-model perplexity for fluency—turning the recommendation into a two-dimensional evaluation standard that could replace single-number leaderboards.","If the fluency explanation is right, the gap between BLEU and human judgment should widen as back-translated data scales up, a prediction that could be checked against published systems of different sizes.","The conceptual point extends beyond machine translation: any evaluation that compares generated text against reference text that is itself translated or otherwise non-representative may inherit this kind of metric blindness."],"forward_implications":["Back-translation improves quality for naturally occurring inputs as well as translationese, so the common restriction of test sets to source-original sentences will not erase its advantage.","Automatic metrics built on n-gram overlap with translationese references will systematically underestimate the quality of a more fluent system, so BLEU ties should not be read as quality ties.","Adding a language-model fluency score, computed on held-out naturally occurring monolingual data, separates the fluency gain human judges see from the adequacy signal BLEU already captures.","Because both systems benefit equally from translationese inputs, the large reverse-direction BLEU gain of back-translation is not explained by translationese simplicity; it is better attributed to the training-data match."],"supporting_citations":[{"why":"Introduces back-translation as a data-augmentation technique, the method under evaluation.","marker":"Sennrich et al. 2016a"},{"why":"Provides the large-scale sampled back-translation recipe and specific systems used as the BT models.","marker":"Edunov et al. 2018"},{"why":"Origin of the claim that translationese is easier to translate and that BT gains concentrate on reverse sentences, which this paper tests.","marker":"Toral et al. 2018"},{"why":"Concurrent finding that BLEU correlates poorly with human judgments, which this paper contrasts with its controlled design.","marker":"Graham et al. 2019"},{"why":"Defines the BLEU metric whose blind spot in direct mode is the paper's central diagnostic.","marker":"Papineni et al. 2002"},{"why":"Supplies the Transformer language-model architecture used to measure closeness to translationese and natural text.","marker":"Baevski and Auli 2018"},{"why":"Provides an implementation of BLEU with standardized tokenization, used for all reported BLEU scores.","marker":"Post 2018"}],"fun_headline_variants":["Back-translation wins with humans, not BLEU","BLEU hides back-translation's fluency gains","Human judges see what BLEU can't in back-translation","Back-translation: humans prefer it, BLEU can't tell","BLEU misses fluency boost from back-translation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The explanation for BLEU's blind spot assumes the two language models used to measure 'closeness to translationese' are equally well calibrated, so their nearly equal scores on translationese references genuinely mean both systems' outputs are equally translationese-like; if one language model is simply better, the measured tie could be an artifact.","fun_headline_variants_meta":{"raw":{"variants":["Back-translation wins with humans, not BLEU","BLEU hides back-translation's fluency gains","Human judges see what BLEU can't in back-translation","Back-translation: humans prefer it, BLEU can't tell","BLEU misses fluency boost from back-translation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000134,"raw_usage":{"total_tokens":1110,"prompt_tokens":884,"completion_tokens":226,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":144}},"tokens_in":500,"tokens_out":226,"duration_ms":3132,"temperature":1.0,"reasoning_tokens":144,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:20:08.465937+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out test set where every source is naturally occurring and the human reference is also naturally occurring target text instead of translationese; the paper's account predicts back-translation still wins with professional judges and still receives lower language-model perplexity. If either prediction reverses, the central claim about human preference or the fluency explanation would be refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Origin of the claim that translationese is easier to translate and that BT gains concentrate on reverse sentences, which this paper tests."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides an implementation of BLEU with standardized tokenization, used for all reported BLEU scores."}],"review_version":1}