{"id":"af900092-261a-4469-97e1-eddd8d9d4d6a","arxiv_id":"1908.03402","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-source transformer with adaptive embedding-noise de-noising achieves small BLEU improvements over the do-nothing baseline on WMT19 English-German automatic post-editing.","lead":"This paper describes a system for the WMT 2019 English-German automatic post-editing shared task, adapting a multi-source transformer to correct machine translation errors. The system earns modest BLEU gains over the do-nothing baseline (75.03 vs 74.73 on the test set), but the improvement is small and within typical model variance.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported test-set gain cannot be attributed to joint de-noising: the APE-only Base model was not evaluated on the test set, and dev results (Table 2) show Base BLEU overlapping or exceeding joint-trained models.","rationale":"The reader's weakest assumption--that embedding-noise de-noising transfers to real APE errors--is also the most load-bearing condition for the paper's central contribution. The test-set numbers do not by themselves establish it, because no non-joint Base model was scored on the test set. The dev-side evidence is actually negative for the transfer assumption: Base's maximum BLEU (77.13) is higher than either joint-trained variant's maximum (77.08 for Gaussian, 77.10 for Uniform), and the paper explicitly speculates that noise training was biased by the synthetic/real gap. The absence of significance testing compounds this: a +0.30 BLEU difference on a single test set is within the range one would expect from seed or checkpoint variation, especially given the roughly 0.17-0.22 BLEU ranges across the four averaged models on dev. I therefore frame the concern as an omitted critical ablation rather than as an internal inconsistency. This is not an objection to the paper's honesty: the authors report the dev table, the synthetic-data BLEU gap, and their own interpretation that APE gains over NMT output are small. Those disclosures are exactly what make the missing Base test result identifiable as the decisive missing datum. The appropriate disposition remains conditional acceptance: the paper provides a useful description of a shared-task submission, and the multi-source architecture plus eSCAPE training may well be a working recipe, but the novel de-noising component should not be credited before the Base-versus-joint comparison on the test set is reported. Releasing code would also help, but the ablation is the scientifically necessary check. This matches, and slightly sharpens, the reader's conditional verdict, so no change to the verdict is needed.","tokens_in":6496,"tokens_out":6638,"duration_ms":68192,"concrete_test":"Run the missing ablation: train and evaluate the APE-only Base model (same multi-source transformer, same eSCAPE data, same checkpoint-averaging and beam=4 settings) on the WMT19 test set, and report BLEU/TER alongside Table 3. Add a paired bootstrap significance test over test sentences for Base vs. Gaussian and Base vs. Uniform. If Base's BLEU is at or above 75.03, or its confidence interval overlaps the joint models' intervals, the de-noising component is not demonstrated to help on real APE data, and the paper's methodological claim should be weakened accordingly; if Base is clearly below 75.03 with non-overlapping intervals, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is a multi-source transformer jointly trained with a de-noising encoder (Eq. 1, Section 2.2). All submitted systems are jointly trained, and the reported test-set result (Table 3) is 75.03 BLEU vs. 74.73 for 'MT as PE'. The load-bearing question is whether the de-noising component is actually doing useful work on real post-editing data. That question is left open: Table 2 reports only dev BLEU for Base (76.91-77.13), Gaussian (76.94-77.08) and Uniform (77.01-77.10), showing Base's maximum exceeds both joint-trained variants. No Base model was scored on the WMT19 test set, so we do not know whether the +0.30 BLEU over the do-nothing baseline comes from joint de-noising, from the multi-source architecture, or from checkpoint-selection noise. The authors themselves note in Section 3.2 that 'there is a gap between the generated machine translation outputs with noise and the real world machine translation outputs, which biased the training,' and that even the ensemble did not 'significantly improve' BLEU (+0.46). Without a test-set Base score and without significance testing, the central claim that the de-noising encoder transfers to real APE errors is not supported. The stated reason for submitting joint models--smaller variance among averaged checkpoints--does not establish transfer; it is a model-selection criterion on the dev set. The manuscript's own dev data are the strongest evidence that the de-noising assumption may fail.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports the UdS English-German submission to the WMT19 shared task on automatic post-editing. The authors adapt a multi-source transformer architecture originally proposed for document-level NMT (Zhang et al., 2018b), with two encoders -- one for the source sentence and one for the MT output -- and a decoder that generates the post-edited sentence. The main novelty is a joint-training scheme (Section 2.2) in which the APE task is trained together with a de-noising encoder task: during part of training, the MT input is replaced by the gold post-edit with noise added to its embedding (Eq. 1), and the overall loss is a weighted sum (Eq. 2) with lambda = 0.5. Using the WMT19 training and development data plus the synthetic eSCAPE corpus, the authors train multi-source transformers either for APE alone (Base) or jointly with Gaussian or uniform noise. On the development set, all models achieve BLEU around 77, with Base's best averaged checkpoint at 77.13 and joint models at 77.08/77.10; the ensemble reaches 77.22. All submitted models are the jointly trained ones, and on the test set they report BLEU 75.03 compared to the do-nothing MT-as-PE baseline of 74.73. The paper is candid about the small gains and about the gap between synthetic eSCAPE data (37.68 BLEU vs gold PE) and real post-editing data (above 77 BLEU).","tokens_in":6765,"tokens_out":7549,"duration_ms":67909,"significance":"The paper is a clearly written and fully specified system description with reproducible training details (architecture, BPE, optimizer, checkpoint averaging). Its strength is honesty about limitations: the authors state that joint training did not improve over the APE-only Base on dev (Base max BLEU 77.13 vs 77.10 for joint models), that the synthetic-noise-to-real-error gap may bias training, and that the test-set gains are not significant. If the de-noising transfer claim were established, the paper would offer a useful regularization technique for APE in high-quality NMT settings. However, as it stands the paper does not demonstrate the central contribution: the APE-only Base model was not evaluated on the test set, so the +0.30 BLEU over MT-as-PE cannot be attributed to the de-noising component. The paper is adequate as a shared-task system description, but not yet as a journal-length demonstration of the method's benefit.","major_comments":[{"comment":"The central claim that joint training with the de-noising encoder is beneficial is not supported because the APE-only Base model was not scored on the WMT19 test set. All submitted models are jointly trained, so the reported test-set BLEU of 75.03 versus 74.73 for MT-as-PE could be due to the multi-source architecture, preprocessing, or checkpoint averaging rather than to the de-noising task. On the development set, Table 2 shows that the Base model's maximum BLEU (77.13) exceeds both jointly trained variants (77.08 and 77.10), and Section 3.2 itself says the gap between synthetic noise and real MT outputs 'biased the training.' Please provide a test-set score for the Base model (the WMT19 test set is publicly available with references), or explicitly reframe the paper as a descriptive system report rather than a demonstration of the de-noising method's benefit.","section":"3.2 (Tables 2 and 3)"},{"comment":"The de-noising task's transfer to real APE errors is an unvalidated assumption. The paper adds noise to the combined word and positional embedding of the gold post-edit (Eq. 1), arguing that this simulates synonyms and word-order errors, but no evidence is given that these corruptions resemble actual differences between MT output and post-edits. The paper's own data (Table 1) show that the synthetic eSCAPE corpus, which is the only direct source of realistic MT-style noise, is far from real data (37.68 vs 77.15/77.42 BLEU), and Section 3.1 says the noise strength (0.2) and joint-training weight (0.5) were not tuned due to lack of time. Without a validation set analysis comparing the induced errors to actual MT errors, Equation (1)'s choice of noise distribution is arbitrary and the claim that joint training makes the model 'more robust' is not testable from the reported results.","section":"2.2 (Eq. 1) and Section 3.1"},{"comment":"The paper uses words like 'significantly' without statistical support, and the reported differences are small relative to the observed variance. On the test set, all submitted models give identical BLEU (75.03) and TER within 0.07; on dev, the min-max ranges for Base (76.91-77.13), Gaussian (76.94-77.08), and Uniform (77.01-77.10) overlap, with only four averaged checkpoints per condition. The statement that jointly trained models were submitted 'because the performance gap between the best and the worst model of jointly trained models is smaller' is a model-selection heuristic, not a statistically meaningful measure of variance or transfer. Please provide bootstrap confidence intervals or a paired significance test (e.g., over test segments) for the test-set comparison, and either support or remove the word 'significant' in the discussion.","section":"3.2 (Tables 2 and 3)"}],"minor_comments":[{"comment":"In the sentence 'We explore both standard Gaussian distribution and uniform distribution of [−1.0,−1.0] as N,' the interval should be [−1.0, 1.0] (or another two-sided interval), since the printed degenerate interval [−1.0,−1.0] would add a constant noise value rather than uniform random noise.","section":"2.2"},{"comment":"The dev BLEU between MT and PE is 77.15 in Table 1 but 76.76 for 'MT as PE' in Table 2; the paper should state which preprocessing (truecasing, tokenizer flags) applies to each table so the reader can reconcile the numbers.","section":"Tables 1 and 2"},{"comment":"The description of the training data would be clearer if the sizes of the real WMT training set and the synthetic eSCAPE corpus were given separately, before reporting the merged 6.5M triples.","section":"3.1"},{"comment":"The symbol 'emb' is described as the embedding matrix, but in the forward pass the noise must be applied to the embedding lookup for the current batch; please clarify the notation so that Eq. (1) is not read as a global rescaling of the full embedding matrix.","section":"2.2, Eq. (1)"},{"comment":"The entry 'Biao Zhang, Deyi Xiong, and jinsong su jinsong. 2018a' should be corrected to 'Jinsong Su' with proper capitalization.","section":"References"},{"comment":"The abstract uses 'de-noising encoder' while Section 2.2 uses both 'denoising' and 'de-noising'; please unify the spelling.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is a WMT shared task system description, and the community's typical threshold for such papers is descriptive transparency rather than hypothesis testing. For a journal, however, the missing Base-on-test-set result is a load-bearing gap because the paper's only novel component (joint training with de-noising) is not independently evaluated. The fix is within scope: the WMT19 test set and references are publicly available for scoring, and the authors kept the model checkpoints. If the Base test-set score turns out to match the joint models, the manuscript can be accepted as a system description with an explicit negative or neutral result; if Base is better, the paper must be reframed. I agree with the reader's conditional verdict and recommend major revision rather than rejection because the missing experiment is feasible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a tidy WMT 2019 shared-task system description. It re-uses an existing multi-source transformer, adds one cross-attention module and an adaptive-noise de-noising objective, and reports a +0.30 BLEU gain over the do-nothing baseline on the test set. The paper is honest about the gain being small and about the synthetic/real gap. The weak spot, as you'd guess from the dev table, is that the de-noising joint training is not actually shown to help. Base, the APE-only model, has the highest single-model dev BLEU (77.13) and overlaps with the joint-trained variants (76.94–77.10). Only joint-trained models were submitted to the test set, so the +0.30 BLEU over 'MT as PE' could just as easily come from the multi-source architecture or checkpoint averaging, not from the noise. The authors' stated reason for submitting joint models is smaller variance among checkpoints: that is supported by the dev ranges (Base range 0.22, Gaussian 0.14, Uniform 0.09), but it is a model-selection story, not evidence that de-noising transfers to real APE errors. They do flag the gap themselves, and they never claim the test-set gain is due to de-noising, so the paper is not misleading. It is just a system paper with a limited contribution.\n\nWhat's genuinely new: the specific architecture—context-gate-free dual-source transformer with an extra cross-attention in the encoder and adaptive embedding noise scaled by |emb|—and the joint training with lambda weighting. These are incremental but clearly described. The paper is well written, gives full hyperparameters, and reports min/max over four averaged checkpoints, which is more than many shared-task papers do.\n\nSoft spots in proportion: no significance tests or confidence intervals (minor for a system paper, but would help); no test-set result for Base (this is the main hole, and it prevents attributing the gain to anything); no release of code (they cite their own Neutron toolkit, but the actual adaptation is not available). The adaptive noise is a reasonable tweak to Cheng et al.'s approach, but with only one noise strength and one lambda tried, and no ablation, we can't tell if it matters.\n\nBottom line: this paper is for people working on APE or document-context transformer variants. It deserves a serious referee: it is a competent, honest system description with reproducible-enough settings. I would accept it at a workshop or as a short paper, but I would push the authors to either add a test-set Base score or temper the robustness language in the abstract. It is not a paper I'd cite for the de-noising idea; the existing citations cover that better.","headline":"Honest, incremental shared-task system paper; the joint de-noising story is not proven on real data, but the paper never over-sells it.","tokens_in":7365,"tokens_out":2739,"would_cite":false,"duration_ms":28207,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a multi-source transformer adapted for automatic post-editing and jointly trained with a de-noising encoder can correct English-German MT output, reaching 75.03 BLEU and 16.77 TER on the WMT19 test set, a small gain…","keywords":["automatic post-editing","multi-source transformer","joint training","de-noising encoder","WMT 2019 shared task","English-German","neural machine translation","embedding noise"],"falsifier":"Train the same multi-source transformer on the same real and synthetic data without the de-noising task and evaluate it on the WMT19 test set; if its BLEU and TER match or beat 75.03 and 16.77, the de-noising joint training is not needed. Alternatively, run the jointly trained model with noise strength 0.2 replaced by a grid including 0, 0.1, 0.5, and 1.0 on the same dev set; a flat response would show the 0.2 value is not load-bearing.","tokens_in":6243,"feed_emoji":"✍️","tokens_out":5896,"duration_ms":54645,"temperature":0.7,"pith_summary":"The paper tries to show that a multi-source transformer, originally built to use document context in neural machine translation, can be turned into an automatic post-editing (APE) system for English-German and that jointly training it with a de-noising encoder makes the model more robust. APE aims to correct errors in machine translation output, and the paper enters the WMT 2019 shared task with this architecture. On the official test set the system reaches 75.03 BLEU and 16.77 TER, a small improvement over the do-nothing baseline of 74.73 BLEU and 16.84 TER. A sympathetic reader would care because neural machine translation output is already good enough that post-editing gains are tiny, so the paper tests whether synthetic noise and a de-noising objective can find any remaining headroom.","feed_headline":"Post-editing NMT gains just 0.3 BLEU","feed_subtitle":"A multi-source transformer with a de-noising task scores 75.03 BLEU on WMT19 En-De, barely above the do-nothing baseline.","key_machinery":"The load-bearing mechanism is the multi-source transformer with a de-noising auxiliary task. In the architecture, one encoder reads the source sentence while a second encoder reads the machine translation output and also attends to the source through an extra cross-attention module; the decoder then attends to both encoded inputs to generate the post-edit. The de-noising objective corrupts the combined word-plus-positional embedding of the gold post-edit as $\\mathrm{emb}_{\\mathrm{out}} = \\mathrm{emb} + \\mathrm{strength} \\cdot |\\mathrm{emb}| \\cdot N$, with strength $0.2$ and $N$ drawn from either a standard Gaussian or a uniform distribution, and the total loss is $\\lambda$ times the APE loss plus $(1-\\lambda)$ times the de-noising loss with $\\lambda = 0.5$. This simulates new machine-translation-like errors by perturbing words and their order, and the adaptive $|\\mathrm{emb}|$ scaling is meant to stop the transformer from trivially shrinking weights to ignore the noise.","core_discovery":"On the paper's own terms, the central discovery is that the adapted multi-source transformer, trained jointly with a de-noising encoder, is a workable APE system whose main benefit on top of a dual-source transformer is reduced variance between checkpoints rather than higher peak BLEU. The model encodes the source sentence and the machine translation output with separate six-layer encoders, adds a cross-attention module so the MT encoder can attend to the source, and decodes the post-edited result while attending to both. Joint training randomly alternates between the APE task and a de-noising task in which the gold post-edit embedding is corrupted with noise $N$ scaled by $0.2 \\cdot |\\mathrm{emb}|$ and the model must reconstruct the clean output; the two losses are balanced by $\\lambda = 0.5$. On the development set the averaged base model reaches up to 77.13 BLEU, slightly above the jointly trained models' maxima of 77.08 and 77.10, while the ensemble of five jointly trained models reaches 77.22. The paper nonetheless submits jointly trained models because their spread across averaged checkpoints is smaller.","pith_inferences":["A natural extension the paper leaves implicit is to measure post-editing quality by human effort or TER rather than BLEU, since BLEU differences below 0.5 are near the level of tokenization noise.","The adaptive noise strength 0.2 could be annealed over training or tuned per token type; a grid search over strength values would show whether the particular value is important or whether any mild corruption works.","Because the base model's dev BLEU peak exceeds the jointly trained peaks, the paper's own numbers suggest the de-noising task may help variance more than accuracy; a direct test-set comparison of the base model alone would settle that.","Combining the synthetic corpus with domain adaptation or fine-tuning on real post-edits, which the paper mentions but does not run, is the most direct route to closing the synthetic-real gap."],"forward_implications":["If the central claim holds, automatic post-editing of modern neural MT output should expect BLEU gains well under one point, so system comparisons should treat the do-nothing baseline as the primary reference.","The joint de-noising objective buys stability across checkpoints and ensembling, not peak single-model accuracy, so it is a variance-reduction technique for APE.","The 37.68 BLEU score of the eSCAPE synthetic corpus versus over 77 BLEU on real post-editing data means synthetic data must be treated as regularization, not as a proxy for real errors.","The extra cross-attention from the MT encoder to the source, which prevents parameter sharing between the two encoders, is the architectural price paid for enabling the de-noising task.","Ensembling five jointly trained models gives the submission's final test numbers, indicating that checkpoint averaging and ensembling remain important for APE."],"supporting_citations":[{"why":"Supplies the dual-source transformer APE baseline, the shared-embedding advice, and the 20-times upsampling used in training.","marker":"Junczys-Dowmunt and Grundkiewicz, 2018"},{"why":"Source of the multi-source context transformer architecture the paper adapts by removing the context gate.","marker":"Zhang et al., 2018b"},{"why":"Motivates adding noise to embeddings rather than sequences for robust training, adapted here to APE.","marker":"Cheng et al., 2018"},{"why":"Provides the eSCAPE synthetic corpus whose 37.68 BLEU gap motivates the de-noising task.","marker":"Negri et al., 2018"},{"why":"Defines the transformer base model, learning-rate schedule, and positional-embedding mechanism used throughout.","marker":"Vaswani et al., 2017"},{"why":"Documents the finding that APE over NMT outputs yields only small gains, framing the paper's modest results.","marker":"Chatterjee et al., 2018"}],"fun_headline_variants":["APE transformer trims variance, not peak BLEU","Denoising joint training stabilizes APE scores","Multi-source transformer: even BLEU, steadier runs","APE gains 0.3 BLEU, but stability improves","Context-aware NMT for APE: consistent not higher"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that adding scaled Gaussian or uniform noise to gold post-edit embeddings produces simulated machine-translation errors similar enough to real ones that the de-noising objective helps the APE task, even though the paper's own dev set shows the base model reaching higher BLEU than either jointly trained model.","fun_headline_variants_meta":{"raw":{"variants":["APE transformer trims variance, not peak BLEU","Denoising joint training stabilizes APE scores","Multi-source transformer: even BLEU, steadier runs","APE gains 0.3 BLEU, but stability improves","Context-aware NMT for APE: consistent not higher"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000174,"raw_usage":{"total_tokens":1224,"prompt_tokens":832,"completion_tokens":392,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":309}},"tokens_in":448,"tokens_out":392,"duration_ms":4544,"temperature":1.0,"reasoning_tokens":309,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:13:21.449009+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same multi-source transformer on the same real and synthetic data without the de-noising task and evaluate it on the WMT19 test set; if its BLEU and TER match or beat 75.03 and 16.77, the de-noising joint training is not needed. Alternatively, run the jointly trained model with noise strength 0.2 replaced by a grid including 0, 0.1, 0.5, and 1.0 on the same dev set; a flat response would show the 0.2 value is not load-bearing.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the dual-source transformer APE baseline, the shared-embedding advice, and the 20-times upsampling used in training."}],"review_version":1}