{"id":"7b47bcd4-3e8e-42cb-8dfe-3792d218e508","arxiv_id":"1909.01383","paper_version":2,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A monolingual sequence-to-sequence model, trained on round-trip translations, repairs cross-sentence inconsistencies in neural machine translation output.","lead":"This paper introduces DocRepair, a post-processing system that fixes cross-sentence inconsistencies in machine translation using only monolingual text. It improves translation quality and consistency in English-Russian translation, making document-level MT easier to deploy without document-level parallel data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Round-trip synthetic corruptions are an imperfect proxy for real MT context errors; the paper demonstrates this for VP ellipsis, but does not quantify coverage of the 30m round-trip training distribution over other inconsistency types.","rationale":"The reader's weakest_assumption identifies the round-trip synthetic data assumption, and that is exactly the place where the argument is least secure. The paper's Section 6.2 is the right test of this assumption, and it partially validates it: one-way training is only slightly better than round-trip on deixis, lexical cohesion, and ellipsis inflection, and substantially better only for VP ellipsis, which the authors explicitly acknowledge. The 30m monolingual round-trip model also outperforms the smaller parallel-filtered model on most contrastive sets, so the monolingual route is not clearly inferior. The BLEU and human evaluations are conducted on actual pipeline outputs with access to the source text, providing independent evidence that the repair helps real translations, not just synthetic corruptions. The concern is therefore a genuine limitation but not a refutation of the central claim. A targeted coverage check on the 30m round-trip distribution would make the boundary of the claim precise, but the available evidence already supports acceptance without changing the verdict.","tokens_in":10608,"tokens_out":11773,"duration_ms":127048,"concrete_test":"On a held-out parallel fragment set, compute two error inventories: (a) actual En→Ru baseline consistency errors, categorized as deixis, lexical cohesion, ellipsis inflection, VP ellipsis, or other, and (b) round-trip corruptions of the same target Russian sentences. Measure the recall of each real error category in the round-trip corruption set. If a category beyond VP ellipsis has near-zero recall, the training distribution is missing a real failure mode and the monolingual claim needs qualification; if only VP ellipsis is undercovered, the paper's stated limitation is bounded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central assumption is that inconsistencies produced by translating isolated sentences in round-trip (Ru→En→Ru) match the inconsistencies of the actual En→Ru baseline at test time. Section 6.2 and Figure 3 show this fails for VP ellipsis: back-translated English rarely contains the auxiliary 'do', so DocRepair is not exposed to the corresponding Russian verb-filling errors, and its contrastive score drops 4.8 points below CADec. The one-way versus round-trip comparison in Table 7 is limited to a 2.5m-instance subset (1.5m parallel plus 1m monolingual) rather than the full 30m setup used for the headline numbers, and it reports only contrastive scores, not BLEU or human preference. Because the training corruption is produced by the same En→Ru model used at test time but on back-translated source text, the mismatch is structural; the paper does not measure how well the round-trip corruption covers the broader space of context inconsistencies. The central quality claim survives for the tested language pair and data because BLEU and human evaluation are measured on real pipeline outputs, but the generality of the monolingual-only repair mechanism beyond the four tested phenomena remains incompletely established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DocRepair, a monolingual sequence-to-sequence model that post-edits groups of sentence-level MT outputs to improve cross-sentence consistency. Training data is generated by taking target-language monolingual document fragments, replacing each sentence with an isolated round-trip translation (Russian→English→Russian), and training a Transformer to map such inconsistent fragments back to the original consistent ones. At test time the model is applied to the output of a context-agnostic English→Russian system. Experiments on OpenSubtitles report a BLEU gain from 33.91 to 34.60, large improvements on contrastive test sets for deixis, lexical cohesion, and ellipsis inflection, a drop on VP ellipsis, and a human evaluation showing that annotators prefer DocRepair corrections in 73% of the cases where they express a preference. The authors additionally analyze data-size effects, one-way versus round-trip corruption, and learning dynamics, and they release code and data.","tokens_in":10851,"tokens_out":8924,"duration_ms":93039,"significance":"If the results hold, this is a practically important contribution to document-level MT: it decouples context repair from the MT system itself, requires no document-level parallel data for the repair model, and could in principle be applied to any black-box MT engine. The paper is strong on evaluation: BLEU, external contrastive test sets from Voita et al. (2019), human evaluation, and several ablations. The machine-checked release of code and data and the honest, detailed analysis of the VP-ellipsis failure mode are particular strengths. The main open risk is the central training-data assumption—that round-trip corruptions of isolated sentences faithfully imitate the inconsistencies of the actual context-agnostic system—which is demonstrated for some phenomena but not systematically quantified.","major_comments":[{"comment":"The claim that DocRepair 'requires only monolingual document-level data' is overstated as written. Section 4.3 shows that generating the round-trip training signal requires both a Russian→English and an English→Russian sentence-level MT model trained on 6m parallel sentence pairs. The method therefore assumes access to sentence-level parallel data (or pre-existing bidirectional MT systems), even though it does not require document-level parallel data. Please rephrase the contribution as 'no document-level parallel data is required' and state this assumption explicitly in the abstract and introduction.","section":"Abstract and §2"},{"comment":"The comparison of one-way and round-trip corruptions is run on 2.5m training instances and reports only contrastive accuracies, whereas the headline results use 30m round-trip fragments and include BLEU and human evaluation. Because round-trip corruptions are produced by back-translating target monolingual text with the same En→Ru system used at test time, the mismatch is structural; the paper demonstrates this for VP ellipsis but does not quantify how representative the round-trip training distribution is of the broader space of context inconsistencies, nor whether the one-way advantage persists at larger scale or carries over to BLEU and human preference. Please either add BLEU (and, if feasible, human evaluation) for the one-way setup, or explicitly restrict the generality claim to the four tested phenomena and to the 30m round-trip distribution.","section":"§6.2, Table 7"}],"minor_comments":[{"comment":"The human evaluation samples only fragments where DocRepair changed at least one sentence; this selection is stated in a footnote but should be discussed in the main text so that the 52% 'equal quality' figure is not misread as applying to unchanged fragments. Reporting inter-annotator agreement would also strengthen the evaluation.","section":"§5.3"},{"comment":"The contrastive-score differences between one-way and round-trip training are not accompanied by significance tests or run-to-run variance; the 6-point VP-ellipsis gap is large, but the smaller differences on other phenomena may be within training noise.","section":"§6.2, Table 7"},{"comment":"The round-trip sampling temperature (0.5), random token replacement probability (10%), and group size (4) are not ablated; a brief discussion of their sensitivity would help establish that the headline results do not hinge on these specific choices.","section":"§4.3"},{"comment":"The term 'BLEU with the baseline' is used in the text before it is defined; please define it explicitly when first introduced so that the learning-dynamics discussion is self-contained.","section":"§7, Figure 4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a genuinely useful contribution to document-level MT. It shows that a monolingual repair model, trained on round-trip translations of isolated sentences, can fix a large share of cross-sentence consistency errors without any parallel document-level data. The gains on deixis and lexical cohesion are substantial, the BLEU improvement is small but consistent, and the human evaluation backs it up. The VP ellipsis failure is analyzed honestly and explained well.\n\nThe new thing is simple and clean: prior APE work repaired sentences in isolation; DocRepair operates on groups of sentences, so it can learn to make translations agree. Training on round-trip translations is a neat way to generate the right kind of inconsistencies. The experimental work is thorough: contrastive test sets, BLEU, human preference, ablations on data size, and a direct one-way vs. round-trip comparison. Code and data are released.\n\nThe soft spot is the one the stress-test note flags: the round-trip corruption is an imperfect proxy for real MT inconsistencies. The paper demonstrates this for VP ellipsis, but it doesn't quantify how well the 30m-example training distribution covers other error types. The one-way vs. round-trip comparison in Section 6.2 is limited to a 2.5m subset and only contrastive scores, so we don't know how much the mismatch costs in BLEU or human preference in the full setting. That's a real gap, but it's a gap in generality, not a fatal flaw. The main result—that monolingual repair works for several phenomena—is solid.\n\nI'd send this to peer review without hesitation. It is a good paper for the subfield: useful to anyone working on context-aware NMT, APE, or low-resource settings where document-level parallel data is scarce. The authors are clear about what works and what doesn't. My only request to the authors would be to widen the coverage analysis, but that's a future-work item, not a blocker.","headline":"Solid, honest paper on monolingual context repair for NMT; the round-trip assumption has a real but well-documented limitation.","tokens_in":11390,"tokens_out":2165,"would_cite":true,"duration_ms":20461,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Monolingual repair fixes MT context errors without parallel data","keywords":["context-aware machine translation","monolingual repair","automatic post-editing","round-trip translation","discourse consistency","deixis","lexical cohesion","ellipsis"],"falsifier":"A decisive test would compare DocRepair's corrections against real errors where round-trip generation cannot reproduce them: for example, if a language pair had a systematic discourse error that is invisible to round-trip translation (as VP ellipsis is here) and the model failed to correct or worsened that error on a held-out contrastive set, the generality of the monolingual-repair claim would be refuted. The paper's own VP-ellipsis result is a partial instance; a broader demonstration across several phenomena and languages would settle the question.","tokens_in":10399,"feed_emoji":"🔧","tokens_out":9540,"duration_ms":71406,"temperature":0.7,"pith_summary":"Modern machine translation systems translate sentences one at a time, and the resulting translations of a document often disagree—a pronoun may change gender, a name may be rendered two different ways, or an ellipsis may be filled with the wrong verb. This paper claims that these inconsistencies can be corrected by a separate, monolingual repair model, called DocRepair, which is trained without any parallel document-level data. The key move is to generate synthetic inconsistencies by round-trip translating each sentence of a monolingual text in isolation, then train a sequence-to-sequence model to map those inconsistent groups back into the original consistent text. At test time, the model post-edits the output of any sentence-level MT system. If this works, it would make context-aware translation possible in the common situation where parallel document-level data is scarce, and it would make the repair independent of the MT system's internal representations.","feed_headline":"Monolingual repair fixes MT context errors without parallel data","feed_subtitle":"DocRepair lifts BLEU from 33.9 to 34.6 and improves deixis and lexical cohesion.","key_machinery":"The central object is the DocRepair model: a standard Transformer-based sequence-to-sequence model trained to map inconsistent groups of sentences into consistent ones. Inconsistent groups are formed by taking four consecutive sentences from monolingual Russian text, round-trip translating each sentence in isolation (Russian to English, then sampling an English-to-Russian translation), giving a pseudo-document that contains the same type of gender, lexical, and ellipsis agreement errors a real context-agnostic MT system makes. Consistent groups are the original monolingual sentences. Sentences are concatenated with a reserved separator token, and the model is trained to output the original consistent text. At test time the model receives the output of a sentence-level MT system for a four-sentence fragment and rewrites it; the paper shows most of the BLEU gain comes from cross-sentence repair rather than sentence-level post-editing alone.","core_discovery":"The central claim is that the inconsistencies a context-agnostic MT system produces between sentences can be imitated by round-trip translations of isolated sentences, and that a monolingual sequence-to-sequence Transformer trained on this synthetic data—mapping inconsistent groups of sentences back into the original consistent ones from monolingual target-language data—can automatically correct those inconsistencies in real translations. On English-to-Russian, this DocRepair model improves BLEU from 33.91 to 34.60, outperforming both the sentence-level baseline and a prior two-pass document-level model (CADec) that requires parallel document-level data. On targeted contrastive tests, accuracy rises from 50.0 to 91.8 for deixis, 45.9 to 80.6 for lexical cohesion, and 53.0 to 86.4 for ellipsis inflection, and human annotators prefer the corrected translations to the baseline in 73% of the cases where they expressed a preference. However, VP ellipsis accuracy (75.2) trails the parallel-data CADec (80.0), indicating that some phenomena are harder to imitate round-trip.","pith_inferences":["Editorial inference: the round-trip synthetic-inconsistency procedure could be adapted to other sequence-generation tasks beyond MT, such as dialogue response generation or summarization, wherever cross-sentence consistency is a known weakness.","Editorial inference: the modest BLEU improvement (0.7) understates the consistency gains shown by contrastive tests; BLEU may be insensitive to the discourse-level errors this model targets, so deployment decisions should weigh both metrics.","Editorial inference: since the model only needs monolingual data, it could be applied to low-resource language pairs where parallel document-level corpora are rare, provided a sentence-level MT system exists."],"forward_implications":["Context-aware MT can be improved without parallel document-level data; monolingual target-language documents suffice.","The repair model is decoupled from the MT system, so it can in principle correct translations from any black-box sentence-level MT system.","The largest consistency gains are in deixis and lexical cohesion; these phenomena are well imitated by round-trip translations.","VP ellipsis is the hardest phenomenon to capture with synthetic data; closing this gap likely requires parallel document-level data.","The repair model changes a minority of sentences (often none), acting as a cautious post-editor rather than a full rewrite."],"supporting_citations":[{"why":"supplies the contrastive test sets for deixis, lexical cohesion, and ellipsis, the CADec baseline, and the parallel-data training setup the paper compares against","marker":"(Voita et al., 2019)"},{"why":"provides the Transformer architecture used for the sentence-level MT baseline, the back-translation models, and DocRepair itself","marker":"(Vaswani et al., 2017)"},{"why":"introduces round-trip translations as synthetic data for automatic post-editing, the generation technique DocRepair adapts to the document level","marker":"(Junczys-Dowmunt and Grundkiewicz, 2016)"},{"why":"concurrent work on monolingual repair for automatic post-editing that the paper extends from sentence level to multi-sentence groups","marker":"(Freitag et al., 2019)"},{"why":"established back-translation with monolingual data to create synthetic parallel training data, the basis for the round-trip approach","marker":"(Sennrich et al., 2016a)"},{"why":"is the OpenSubtitles2018 corpus from which both the monolingual fragments and the parallel training data are drawn","marker":"(Lison et al., 2018)"},{"why":"provides the alignment filtering criterion (relative time overlap of subtitle frames) used to build the parallel training data","marker":"(Voita et al., 2018)"}],"fun_headline_variants":["DocRepair corrects MT context errors using only target-language data","Monolingual DocRepair boosts MT consistency, no parallel data needed","Fix NMT context errors with only monolingual data","DocRepair: post-edit sentence-level MT for context consistency","Context-aware repair: fixing MT inconsistencies without parallel data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the errors created by round-trip translating each sentence in isolation—gender slips, name variants, wrong ellipsis fillings—are representative enough of the errors the real sentence-level MT system makes that repairing the synthetic inconsistencies will transfer to repairing the real ones.","fun_headline_variants_meta":{"raw":{"variants":["DocRepair corrects MT context errors using only target-language data","Monolingual DocRepair boosts MT consistency, no parallel data needed","Fix NMT context errors with only monolingual data","DocRepair: post-edit sentence-level MT for context consistency","Context-aware repair: fixing MT inconsistencies without parallel data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000625,"raw_usage":{"total_tokens":2911,"prompt_tokens":983,"completion_tokens":1928,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":1841}},"tokens_in":599,"tokens_out":1928,"duration_ms":12711,"temperature":1.0,"reasoning_tokens":1841,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:18:26.338844+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test would compare DocRepair's corrections against real errors where round-trip generation cannot reproduce them: for example, if a language pair had a systematic discourse error that is invisible to round-trip translation (as VP ellipsis is here) and the model failed to correct or worsened that error on a held-out contrastive set, the generality of the monolingual-repair claim would be refuted. The paper's own VP-ellipsis result is a partial instance; a broader demonstration across several phenomena and languages would settle the question.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the contrastive test sets for deixis, lexical cohesion, and ellipsis, the CADec baseline, and the parallel-data training setup the paper compares against"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the Transformer architecture used for the sentence-level MT baseline, the back-translation models, and DocRepair itself"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"introduces round-trip translations as synthetic data for automatic post-editing, the generation technique DocRepair adapts to the document level"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"concurrent work on monolingual repair for automatic post-editing that the paper extends from sentence level to multi-sentence groups"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"is the OpenSubtitles2018 corpus from which both the monolingual fragments and the parallel training data are drawn"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the alignment filtering criterion (relative time overlap of subtitle frames) used to build the parallel training data"}],"review_version":1}