Pith. sign in

REVIEW 4 major objections 5 minor 28 references

The Transference Architecture for Automatic Post-Editing

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A transformer decoder turned into an unmasked encoder lifts automatic post-editing by about one BLEU point.

desk verdict Simple, plausible APE architecture with honest reporting, but the 1-BLEU headline is unsupported on NMT and untested on PBSMT. read the letter →

arxiv 1908.06151 v2 pith:T5JOTO2T submitted 2019-08-16 cs.CL

Santanu Pal , Hongfei Xu , Nico Herbig , Sudip Kumar Naskar , Antonio Krueger , Josef van Genabith This is my paper · ORCID
classification cs.CL
keywords automaticpost-editingtransferencearchitecturemulti-sourcetransformerunmaskeddecoderBLEUWMTsharedtaskEnglish-Germansecondencoder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Automatic post-editing (APE) corrects machine-translated text, and this paper claims that a simple recombination of standard transformer blocks can do it better. The proposed architecture, called transference, encodes the source sentence with a normal transformer encoder, passes the machine-translated text through a decoder-style block with self-attention but no masking, and then lets a final decoder produce the corrected text. On WMT English–German shared tasks, the paper reports roughly a 1 BLEU gain over the previous best ensemble, and shows that the depth of the inserted second encoder matters more than the depth of the decoder. A sympathetic reader would take the central claim to be that this unmasked decoder block is a simple and effective way to fuse source and translation context before making post-editing decisions.

What carries the argument

The key mechanism is the unmasked decoder block placed between the source encoder and the output decoder. In the transformer, decoder blocks use masked self-attention so each position can only look at earlier positions; the paper removes that mask when processing the machine-translated text, so the block acts as a bidirectional encoder that combines the translation with the source via cross-attention. The argument is that this fused representation gives the final decoder access to both source and translation context simultaneously, which is meant to mirror how a human post-editor first compares the source against the machine output and then applies corrections.

What would settle it

Train both the transference ensemble and the wmt18smt best x4 ensemble multiple times with different random seeds on the same data, compute bootstrap confidence intervals for BLEU and TER on test2016 and test2017, and check whether the 1-point gap falls inside the intervals; if it does, the claimed superiority collapses.

Watch

Extended reading notes

Core claim

The paper's central discovery is the transference architecture ({src, mt}tr → pe), which recombines the standard transformer blocks by placing an unmasked decoder block between the source encoder and the output decoder. That block performs self-attention on the machine-translated text and cross-attention over the encoded source, producing a fused representation that the final decoder attends to while generating the post-edited sentence. On the WMT 2016 and 2017 PBSMT test sets and the WMT 2018 NMT development and test sets, an ensemble built around transference reaches roughly +1 BLEU and -0.5 TER over the best WMT 2018 baseline ensemble, and the authors claim a new state of the art in APE. They also show that reducing the depth of the second encoder hurts all four scores, whereas reducing the depth of the final decoder leaves performance largely intact.

Load-bearing premise

The claimed ~1 BLEU advantage over the strongest baseline ensemble is not shown to be statistically significant, because the baseline's sentence-level scores were unavailable, so the paper assumes the gap is larger than run-to-run variation.

Editorial extensions

If this is right

  • The transference ensemble outperforms the WMT 2018 SMT winner ensemble by about +1 BLEU and -0.5 TER on test2016 and test2017.
  • The transference model trained on eScape and NMT data, fine-tuned, and checkpoint-averaged improves over the raw NMT baseline by +1.02 BLEU and -0.69 TER on test2018.
  • Both the single-encoder mt→pe and concatenated {src+mt}→pe models improve over raw MT, and adding src through a separate encoder yields further gains.
  • Reducing second-encoder layers hurts performance, while reducing decoder layers does not, indicating that encoders carry more weight in transformer-based APE.
  • The architecture is generic and can be applied to other multi-source tasks such as multi-source translation or summarization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the architecture's advantage is robust, it suggests the bottleneck in APE is the fusion of source and translation contexts before decoding, not the decoding capacity itself; similar unmasked encoder-style blocks could help other correction or editing tasks.
  • The reported inability to establish statistical significance against the x4 ensemble means the ~1 BLEU gap could be within noise; a direct replication with multiple seeds and the baseline's sentence-level scores would settle the claim.
  • The finding that decoder depth is comparatively unimportant might extend to transformer-based generation beyond APE, as the paper itself hints, but that generalization is not tested here.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a multi-source transformer architecture for automatic post-editing (APE) called transference. The model encodes the source with a standard transformer encoder, processes the machine-translated text with an unmasked decoder block that acts as a second encoder, and generates the post-edited text with a final decoder. The authors train the model on WMT APE English-German data plus synthetic corpora and report BLEU and TER results on the 2016/2017 PBSMT tasks and the 2018 NMT task. They claim that the transference model outperforms the state of the art by about 1 BLEU point across all tasks, and they present an ablation showing that reducing the depth of the second encoder hurts performance more than reducing the decoder depth. The central state-of-the-art claim is not fully supported by the reported numbers: the PBSMT gain is not tested for significance, and the NMT BLEU comparison on dev2018 is actually negative.

Significance. If the reported gains were robust, the transference architecture would be a useful contribution to APE. Its main idea, reusing an unmasked transformer decoder as a second encoder to combine source and machine-translation contexts, is simple, well motivated, and potentially transferable to other multi-source tasks. The evaluation uses external WMT shared-task test sets and compares against strong published baselines, which is methodologically appropriate. However, the headline claim of a 1 BLEU improvement over the state of the art is not established: the PBSMT comparison lacks a statistical significance test and is confounded with ensembling, while the NMT results show essentially no BLEU advantage over the official winner on dev2018. The architecture is still of interest, but the paper's claims must be substantially revised and, where possible, supported by significance testing.

major comments (4)
  1. [Abstract, §5.3, Table 2] The abstract and conclusions claim that transference outperforms the state of the art by 1 BLEU on the WMT 2018 NMT task. Table 2 does not support this: Exp. 10 reaches 77.67 BLEU on dev2018 versus 77.74 for wmt18nmt_best, which is a deficit of 0.07 BLEU, and the test2018 gain is only +0.22 BLEU (75.75 vs. 75.53). The 1 BLEU figure appears only relative to the raw NMT baseline, not relative to the state of the art. The TER improvements are more consistent, but they are not what the abstract claims. Section 5.3 itself acknowledges the lower BLEU on dev2018, so the abstract and conclusions should be revised to state task-specific BLEU and TER differences accurately.
  2. [§5.5, Table 1] The central PBSMT claim is that the transference ensemble beats wmt18smt_best(x4) by roughly +1 BLEU. Section 5.5 explicitly states that no significance test was possible because sentence-level scores for wmt18smt_best(x4) are unavailable. On 2,000-sentence test sets, a 1 BLEU gap is within run-to-run variance for transformer ensembles. Moreover, the comparison is not matched: Exp. 4.2 (ensemblesmt(x3)) combines two random-init models with a four-checkpoint ensemble, whereas wmt18smt_best(x4) is an ensemble of four single models. The architecture effect is therefore confounded with ensemble size and ensembling strategy. Please add a matched ensemble comparison and paired significance testing, or state explicitly that the superiority claim is not statistically established.
  3. [§4.2, Table 2 note] Table 2 notes that the test2018 results for Exp. 10 were obtained by the WMT 2019 task organizers, not by the authors' own evaluation pipeline. Without details on tokenization, casing, and reference preprocessing used by the organizers, these numbers cannot be directly compared with the authors' dev2018 results, and the +0.22 BLEU gain on test2018 should be treated with caution. Please provide the authors' own evaluation on test2018 or describe the exact evaluation conditions under which the organizer-provided score was computed.
  4. [§5.1, Table 1] The PBSMT comparison against the state of the art uses the originally published wmt18smt_best numbers (Exp. 1.3 and 1.4), whose training pipeline differs from the present paper's setup. The authors do retrain the baseline under their own configuration (Exp. 1.5 and 1.6), but the headline 1-BLEU claim is made against the original numbers, not against the matched retrained versions. Since the retrained baselines are 1-2 BLEU lower than the original ones, the transference advantage over a matched baseline should be reported separately, and the claim of a 1 BLEU gain over 'the state of the art' should be qualified by the training-condition mismatch.
minor comments (5)
  1. [Table 1] The table footnote says the ±X value is the improvement over wmt18smt_best(x4), but Exp. 1.2's ± values are relative to raw SMT. Please clarify in the table or caption.
  2. [§4.2] The text says 'we explore two experimental setups' but then lists more than two variants (generic,smt, generic,nmt, fine-tuned, and averaged). Please revise the enumeration.
  3. [Table 2] The ± values in parentheses are relative to Raw NMT, not to wmt18nmt_best; the caption should state this explicitly.
  4. [Abstract, §4.1] The phrase 'WMT 2018 ... (PBSMT and NMT)' is misleading because the paper reports no WMT 2018 PBSMT experiments; the PBSMT results are from the 2016 and 2017 tasks.
  5. [§5.3, Exp. 5.1] The claim that reducing decoder layers to 6-6-4 'does not hurt the performance' is too strong: test2017 BLEU drops from 70.05 to 69.82 in Exp. 5.1. A phrase like 'has only a small mixed effect' would be more accurate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the transference architecture is evaluated against external WMT test sets, and no load-bearing claim reduces to a fitted parameter or self-citation.

full rationale

The paper's central claim is an empirical result: the transference transformer variant ({src, mt}tr -> pe) is compared on the WMT 2016/2017 PBSMT and WMT 2018 NMT APE test sets against published baselines (wmt18smt_best, wmt18nmt_best) and against retrained versions of those baselines under the paper's own setup (Exp. 1.5/1.6). The architecture is defined independently of the benchmark data; no equation defines transference's output in terms of the target BLEU scores, and no parameter is fitted to the test data and then reported as a prediction. The motivational translator-workflow analogy in Section 1 is not used as evidence. The self-citations (Pal et al. 2016a, 2016b, 2018) are related-work references and are not load-bearing: the WMT baselines come from Junczys-Dowmunt and Grundkiewicz (2018) and Tebbifakhr et al. (2018), not from the authors' own prior results. The explicit limitation in Section 5.5 that statistical significance could not be tested because sentence-level scores for wmt18smt_best (x4) are unavailable concerns uncertainty in the comparison, not a reduction of the result to its inputs; likewise the weaker NMT BLEU result against wmt18nmt_best is an empirical discrepancy, not circularity. All load-bearing evidence is external to the model's own construction, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on public benchmark data and standard hyperparameter choices; no fitted scalar parameters or new entities are introduced. The main assumptions are the reliability of the datasets and the transferability of transformer base settings.

assumptions (3)
  • domain assumption The publicly available WMT APE datasets and the synthetic corpora (artificial data, eScape) are reliable and were preprocessed consistently.
    The experiments compare models trained on these data; different cleaning choices for eScape (Section 4.1) could shift absolute scores, but the comparison between systems uses the same data.
  • domain assumption The transformer base hyperparameters from Vaswani et al. (2017) transfer to the APE task and to the retrained baselines.
    The setup fixes layers to 6, warmup to 8000, batch to 25K tokens, and beam to 4 without per-task tuning. The retrained wmt18smt best baseline uses the same setup, making the comparison internally consistent, but absolute performance could differ with tuned hyperparameters.
  • ad hoc to paper The two-step translator workflow, compare src and mt, then edit, is a useful inductive bias for APE.
    This motivation in Section 1 guides the architecture design; it is not independently established, but it is not used as evidence for the empirical claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Transference Architecture for Automatic Post-Editing." pith.science (2026). https://pith.science/paper/T5JOTO2T

@misc{pith2026190806151,
  author       = {Pith},
  title        = {Pith review of: The Transference Architecture for Automatic Post-Editing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T5JOTO2T}},
  note         = {Machine review of arXiv:1908.06151}
}
read the original abstract

In automatic post-editing (APE) it makes sense to condition post-editing (pe) decisions on both the source (src) and the machine translated text (mt) as input. This has led to multi-source encoder based APE approaches. A research challenge now is the search for architectures that best support the capture, preparation and provision of src and mt information and its integration with pe decisions. In this paper we present a new multi-source APE model, called transference. Unlike previous approaches, it (i) uses a transformer encoder block for src, (ii) followed by a decoder block, but without masking for self-attention on mt, which effectively acts as second encoder combining src -> mt, and (iii) feeds this representation into a final decoder block generating pe. Our model outperforms the state-of-the-art by 1 BLEU point on the WMT 2016, 2017, and 2018 English--German APE shared tasks (PBSMT and NMT). We further investigate the importance of our newly introduced second encoder and find that a too small amount of layers does hurt the performance, while reducing the number of layers of the decoder does not matter much.

Figures

Figures reproduced from arXiv: 1908.06151 by the authors.

Figure 1
Figure 1. The transference model architecture for APE ({src, mt}tr → pe). 4 Experiments We explore our approach on both APE sub-tasks of WMT 2018, where the 1st-stage MT system to which APE is applied is either a phrase-based sta￾tistical machine translation (PBSMT) or a neural machine translation (NMT) model. For the PBSMT task, we compare against four baselines: the raw SMT output provided by the 1st-stage PBSMT system, the… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 25 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate . In International Conference on Learning Representations (ICLR), San Diego, CA, USA

    work page 2015

  4. [4]

    Ond r ej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 Conference on Machine Translation (WMT17) . In Proceedings of the Second Conference o...

    work page 2017

  5. [5]

    Ond r ej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of ...

    work page 2016

  6. [6]

    Ond r ej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 Workshop on Statistical Machine Translation . In Proceedings of the Tenth Workshop on Statistical Machine Tr...

    work page 2015

  7. [7]

    Amin Farajian, Matteo Negri, Marco Turchi, Ankit Srivastava, and Santanu Pal

    Rajen Chatterjee, M. Amin Farajian, Matteo Negri, Marco Turchi, Ankit Srivastava, and Santanu Pal. 2017. Multi-Source Neural Automatic Post-Editing: FBK's participation in the WMT 2017 APE shared task . In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers , pages 630--638, Copenhagen, Denmark. Association for Comput...

    work page 2017

  8. [8]

    Rajen Chatterjee, Matteo Negri, Raphael Rubino, and Marco Turchi. 2018. Findings of the WMT 2018 Shared Task on Automatic Post-Editing . In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, Brussels, Belgium. Association for Computational Linguistics

    work page 2018

Show all 28 references
  1. [9]

    Tobias Domhan. 2018. https://www.aclweb.org/anthology/P18-1167 How much attention do you need? a granular analysis of neural machine translation architectures . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ...

    2018

  2. [10]

    Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2016. http://www.aclweb.org/anthology/W/W16/W16-2378 Log-linear Combinations of Monolingual and Bilingual Neural Machine Translation Models for Automatic Post-Editing . In Proceedings of the First Conference on Machine Translation...

    2016

  3. [11]

    Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2017. http://www.aclweb.org/anthology/W17-4774 The AMU-UEdin Submission to the WMT 2017 Shared Task on Automatic Post-Editing . In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 63...

    2017

  4. [12]

    Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2018. http://www.aclweb.org/anthology/W18-6468 MS-UEdin Submission to the WMT2018 APE Shared Task: Dual-Source Transformer for Automatic Post-Editing . In Proceedings of the Third Conference on Machine Translation, Volume 2: Share...

    2018

  5. [13]

    Kevin Knight and Ishwar Chander. 1994. Automated Postediting of Documents . In Proceedings of the Twelfth National Conference on Artificial Intelligence (Vol. 1), AAAI '94, pages 779--784, Seattle, Washington, USA

    1994

  6. [14]

    Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ond r ej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Mac...

    2007

  7. [15]

    Jind r ich Libovick\' y , Jind r ich Helcl, Marek Tlust\' y , Ond r ej Bojar, and Pavel Pecina. 2016. CUNI System for WMT16 Automatic Post-Editing and Multimodal Translation Tasks . In Proceedings of the First Conference on Machine Translation, pages 646--654, Berlin, Germany....

    2016

  8. [16]

    Matteo Negri, Marco Turchi, Rajen Chatterjee, and Nicola Bertoldi. 2018. ESCAPE: a Large-scale Synthetic Corpus for Automatic Post-Editing . In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Lang...

    2018

  9. [17]

    Jan Niehues, Eunah Cho, Thanh-Le Ha, and Alex Waibel. 2016. Pre-Translation for Neural Machine Translation . In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1828--1836, Osaka, Japan. The COLING 2016 Organiz...

    2016

  10. [18]

    Santanu Pal, Nico Herbig, Antonio Krüger, and Josef van Genabith. 2018. http://www.aclweb.org/anthology/W18-6469 A Transformer-Based Multi-Source Automatic Post-Editing System . In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages ...

    2018

  11. [19]

    Santanu Pal, Sudip Kumar Naskar, and Josef van Genabith. 2016 a . Multi-Engine and Multi-Alignment Based Automatic Post-Editing and Its Impact on Translation Productivity . In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical...

    2016

  12. [20]

    Santanu Pal, Sudip Kumar Naskar, Mihaela Vela, and Josef van Genabith. 2016 b . http://anthology.aclweb.org/P16-2046 A Neural Network Based Approach to Automatic Post-Editing . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2...

    2016

  13. [21]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation . In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, pages 311--318, Philadelphia, Pennsylvania

    2002

  14. [22]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers

    2016

  15. [23]

    Jaehun Shin and Jong-Hyeok Lee. 2018. http://www.aclweb.org/anthology/W18-6471 Multi-encoder Transformer Network for Automatic Post-Editing . In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 853--858, Belgium, Brussels. Associa...

    2018

  16. [24]

    Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of Translation Edit Rate with Targeted Human Annotation . In Proceedings of Association for Machine Translation in the Americas, pages 223--231, Cambridge, Massachusetts, USA

    2006

  17. [25]

    TAUS/CNGL Report . 2010. http://www.cngl.ie/tauscngl-machine-translation-post-editing-guidelines-published Machine Translation Post-Editing Guidelines Published . Technical report, TAUS

    2010

  18. [26]

    Amirhossein Tebbifakhr, Ruchit Agrawal, Rajen Chatterjee, Matteo Negri, and Marco Turchi. 2018. http://www.aclweb.org/anthology/W18-6472 Multi-Source Transformer with Combined Losses for Automatic Post Editing . In Proceedings of the Third Conference on Machine Translation, Vo...

    2018

  19. [27]

    Dusan Varis and Ond r ej Bojar. 2017. CUNI System for WMT17 Automatic Post-Editing Task . In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 661--666, Copenhagen, Denmark. Association for Computational Linguistics

    2017

  20. [28]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf Attention Is All You Need . In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fe...

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.