REVIEW 1 major objections 6 minor 21 references
UDS--DFKI Submission to the WMT2019 Similar Language Translation Shared Task
T0 review · 1 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A two-encoder transformer with word and subword input placed second among ten Czech-to-Polish systems in the 2019 similar-language shared task.
desk verdict Solid, externally verifiable shared-task result with a muddy architecture description and an over-claimed dev improvement; worth a serious referee but not a breakthrough. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The transference model is the central mechanism: a multi-encoder transformer in which encoder $enc_1$ reads source word forms $f_w$, a sub-encoder $enc_2$ reads byte-pair-encoding tokens $f_s$, and a second encoder (called both $enc_{src \to mt}$ and $enc_{1 \to 2}$) performs cross-attention over these representations to prepare what the decoder attends to. Byte-pair encoding is a subword segmentation that splits rare words into frequent units; here Czech and Polish share one BPE vocabulary, and the decoder consumes only BPE text. Training rests on a data-selection step that scores each parallel sentence by the sum of absolute cross-entropy differences between a domain-specific and an out-of-domain language model (Equation 1); the 500K most in-domain-like sentences form the fine-tuning set, and the final model averages the eight best checkpoints.
What would settle it
Train the described word-plus-subword two-encoder transformer on the cleaned Czech–Polish corpus, select and fine-tune on the 500K most in-domain-like sentences as specified, and decode the development set; if the BLEU score is not close to the reported 25.1 (or the official test BLEU is not close to 7.6), the submitted system was not the model as described. A reader could also inspect a handful of submitted translation pairs for artifacts inconsistent with joint word and subword encoding.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that a two-encoder transformer—encoding the Czech source once at word level and once at subword level, then fusing these representations through a second cross-attention encoder—produces Czech-to-Polish translations that placed second of ten entries, 0.3 BLEU behind the first-place system. The same pipeline without fine-tuning scored 12.9 BLEU lower on the development set, and the official test set, drawn from a different domain, returned low BLEU and high TER (translation edit rate) for every participating system, which the paper reads as evidence that the out-of-domain test made the task uniformly hard. The paper presents this as a competitive result for direct translation between similar languages without English as a pivot.
Load-bearing premise
The load-bearing premise is that the model described in the paper is the system that actually produced the submitted translations; the description's notation is internally inconsistent and no code or outputs are released, so that link cannot be verified from the paper.
Editorial extensions
If this is right
- If the official evaluation and the reported system description are accurate, the dual word-plus-subword encoder architecture is a competitive design for similar-language pairs without English as a pivot.
- The 12.9 BLEU improvement from fine-tuning on selected in-domain-like data implies that data selection and fine-tuning can substantially adapt a noisy general-domain corpus to a small in-domain set.
- The uniformly low BLEU and high TER scores on the official test set imply that out-of-domain test data can dominate and obscure differences in model quality.
- Joint BPE vocabularies and shared embeddings between the two languages are direct enablers of the approach, so the method is best suited to closely related languages.
Reading between the lines
- Because the reported fine-tuning gain is larger than the gap between first and second place, a natural next experiment is an ablation that separates the dual-encoder architecture from the data-selection and fine-tuning step; the paper does not provide that comparison.
- The transference idea—two source views, surface and subword, fused by cross-attention—should carry over to other similar-language pairs and to dialect-to-standard translation, where morphological differences are concentrated in subword units.
- A testable extension would be to swap off the second encoder while keeping word-level input, or to fine-tune a standard single-encoder transformer on the same 500K subset; either comparison would isolate the architecture's contribution.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes the UDS-DFKI system submitted to the WMT2019 Similar Language Translation shared task for Czech-to-Polish translation. The system is a two-encoder transformer ('transference model') that encodes word-level and BPE-level source representations, combined with cross-entropy difference data selection (transference500K) and fine-tuning. The paper reports that the system ranked second among ten entries in the official evaluation, with BLEU 7.6 versus the winner's 7.9, and it analyzes the effect of out-of-domain test data.
Significance. The central empirical claim, that UDS-DFKI ranked second in the official Czech-to-Polish evaluation, is supported by the transcription of the organizers' rank table and is externally verifiable in the WMT19 findings, so the result is credible. The paper's method, if clearly described, would be a useful data point for similar-language translation with multi-encoder transformers. However, the paper's contribution is limited by an inconsistent architecture description and lack of released code, which prevents verification of the exact system that produced the submitted outputs. The paper also makes a strong claim of 'significant' improvement that is not supported by significance testing and is much smaller on the official test set.
major comments (1)
- [Section 4] The notation in Section 4 is internally inconsistent: the output of the word-form encoder (fw) and the output of the BPE sub-encoder (fs) are both denoted enc2, and the 'second encoder' is referred to as both encsrc→mt and enc1→2. Because these names denote distinct components in the text, a reader cannot determine the actual connectivity of the model, and the claim that this architecture is the one that produced the ranked system output cannot be assessed. Please unify the notation and provide a precise diagram or layer-by-layer specification.
minor comments (6)
- [Abstract / Section 1] The abstract and introduction state that the system ranked 'second among ten teams,' but Table 2 lists ten entries from five teams. Please correct to 'ten entries' or 'five teams.'
- [Section 3] The data description is confusing: the text says the first 1,000 sentences of the sorted General corpus are used for validation, and then says 1,000 top-scored sentences are excluded. Please clarify whether these are the same set or whether two separate 1,000-sentence sets are involved.
- [Section 6] The claim that the fine-tuned system provides 'significant performance improvement' is based only on the development set (+12.9 BLEU); the official test set shows a 0.5 BLEU difference. Please avoid the word 'significant' unless a statistical test is provided, and discuss the discrepancy between dev and test improvements.
- [Table 2] The table would benefit from a caption noting that the ten entries come from five teams, and 'Constrastive' should be corrected to 'Contrastive.'
- [Section 5.1] No baseline Transformer model is included; adding a standard Transformer trained on the same data would help separate the effect of the transference architecture from the effect of data selection and fine-tuning.
- [General] Releasing the code or providing a detailed configuration file would greatly improve reproducibility, especially given the architecture description ambiguities.
Circularity Check
No significant circularity: the second-place claim rests on the official WMT19 organizer ranking, not on the paper's own derivation.
full rationale
The paper's central claim—'UDS-DFKI achieved competitive performance ranking second among ten teams in Czech to Polish translation'—is an empirical result measured by the WMT19 shared-task organizers. The evidence is Table 2, which presents the organizer-provided rank list, and this ranking is externally verifiable in the WMT19 Findings report; it is not generated by any equation or assumption in the paper. The system choices, including the 500K data-selection threshold and fine-tuning, were made using the development set and then evaluated on the official test set, so no fitted parameter is relabeled as a prediction. Citations to the authors' own prior work, such as Pal et al. (2015) for corpus cleaning, support only a preprocessing step and are not load-bearing for the ranking claim. The notational inconsistencies in Section 4 and the lack of released code are reproducibility concerns, not circularity. No self-definitional reduction, fitted-input prediction, or uniqueness-imported-from-authors pattern is present. Although the abstract's 'ten teams' is imprecise because Table 2 lists ten entries from five teams, that wording does not alter the official second-place rank.
Assumptions & free parameters
free parameters (4)
- transference500K training set size =
500,000 sentence pairs
- validation set size =
1,000 sentences
- in-domain LM corpus size =
2,050 sentences
- BPE vocabulary size =
28,000 tokens
assumptions (4)
- domain assumption Official WMT 2019 evaluation of BLEU/TER and the ranking in Table 2 are correct and correspond to the submitted system.
- domain assumption The eScape corpus is a valid out-of-domain corpus for the out-of-domain language model in Equation 1.
- domain assumption The development set is a good in-domain reference for selecting training data that will help on the official test set.
- domain assumption A shared Czech-Polish BPE vocabulary benefits translation because the languages are similar.
Cite this review
Pith. "Pith review of UDS--DFKI Submission to the WMT2019 Similar Language Translation Shared Task." pith.science (2026). https://pith.science/paper/RSVMEV7R
@misc{pith2026190806138,
author = {Pith},
title = {Pith review of: UDS--DFKI Submission to the WMT2019 Similar Language Translation Shared Task},
year = {2026},
howpublished = {\url{https://pith.science/paper/RSVMEV7R}},
note = {Machine review of arXiv:1908.06138}
}
read the original abstract
In this paper we present the UDS-DFKI system submitted to the Similar Language Translation shared task at WMT 2019. The first edition of this shared task featured data from three pairs of similar languages: Czech and Polish, Hindi and Nepali, and Portuguese and Spanish. Participants could choose to participate in any of these three tracks and submit system outputs in any translation direction. We report the results obtained by our system in translating from Czech to Polish and comment on the impact of out-of-domain test data in the performance of our system. UDS-DFKI achieved competitive performance ranking second among ten teams in Czech to Polish translation.
Reference graph
Works this paper leans on
-
[1]
Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain Adaptation via Pseudo In-domain Data Selection . In Proceedings of EMNLP
work page 2011
-
[2]
Lo \"i c Barrault, Ond r ej Bojar, Marta R. Costa-juss \`a , Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias M \"u ller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 Conference on Machine Translation (WMT19) . In Proceedings of WMT
work page 2019
-
[3]
Ond r ej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, et al. 2017. Findings of the 2017 Conference on Machine Translation (WMT17) . In Proceedings of WMT
work page 2017
-
[4]
Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al. 2016. Findings of the 2016 Conference on Machine Translation . In Proceedings of WMT
work page 2016
-
[5]
Marta R. Costa-juss \`a . 2017. Why Catalan-Spanish Neural Machine Translation? Analysis, Comparison and Combination with Standard Rule and Phrase-based Technologies . In Proceedings of VarDial
work page 2017
-
[6]
Costa-juss \`a , Marcos Zampieri, and Santanu Pal
Marta R. Costa-juss \`a , Marcos Zampieri, and Santanu Pal. 2018. A Neural Approach to Language Variety Translation . In Proceedings of VarDial
work page 2018
-
[7]
Federico Fancellu, Andy Way, and Morgan O’Brien. 2014. Standard Language Variety Conversion for Content Localisation via SMT . In Proceedings of EAMT
work page 2014
-
[8]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition . Proceedings of CVPR
work page 2016
Show all 21 references
-
[9]
Diederik P Kingma and Jimmy Lei Ba. 2015. Adam: A Method for Stochastic Optimization . Proceedings of ICLR
2015
-
[10]
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ond r ej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Mac...
2007
-
[11]
Surafel M Lakew, Aliia Erofeeva, and Marcello Federico. 2018. Neural Machine Translation into Language Varieties . arXiv preprint arXiv:1811.01064
2018 arXiv
-
[12]
Santanu Pal, Sudip Naskar, and Josef van Genabith. 2015. U d S -sant: E nglish -- G erman hybrid machine translation system. In Proceedings of WMT
2015
-
[13]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of ACL
2002
-
[14]
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units . In Proceedings of ACL
2016
-
[15]
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation . In Proceedings of AMTA
2006
-
[16]
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting . J. Mach. Learn. Res., 15(1):1929--1958
2014
-
[17]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need . In Proceedings of NIPS
2017
-
[18]
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Ahmed Ali, Suwon Shon, James Glass, Yves Scherrer, Tanja Samard z i \' c , Nikola Ljube s i \' c , J \"o rg Tiedemann, Chris van der Lee , Stefan Grondelaers, Nelleke Oostdijk, Dirk Speelman, Antal van den Bosch , Ritesh Kumar, ...
2018
-
[19]
Marcos Zampieri, Shervin Malmasi, Yves Scherrer, Tanja Samard z i \' c , Francis Tyers, Miikka Silfverberg, Natalia Klyueva, Tung-Le Pan, Chu-Ren Huang, Radu Tudor Ionescu, Andrei Butnaru, and Tommi Jauhiainen. 2019. A Report on the Third VarDial Evaluation Campaign . In Proce...
2019
-
[20]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...
-
[21]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.