REVIEW 3 major objections 5 minor 27 references
Incorporating Word and Subword Units in Unsupervised Machine Translation Using Language Model Rescoring
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read An unsupervised German-to-Czech translation system reaches 10.6 BLEU by combining word- and subword-level NMT with PBSMT pseudo-parallel data, unknown-word replacement, and Czech language-model rescoring.
desk verdict A transparent WMT19 system description for German-Czech; the UWR trick is genuinely new but unvalidated, and the 0.3 BLEU rescoring gains could be noise. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a candidate-generation-and-selection pipeline. Two unsupervised NMT models—one word-level with a 50k vocabulary, one subword-level with 40k byte-pair-encoding (BPE) units—are initialized with separately trained FastText embeddings aligned into a shared space by MUSE, an unsupervised cross-lingual embedding alignment procedure. A PBSMT model built with a unigram phrase table and back-translation supplies pseudo-parallel sentences for fine-tuning and supplies substitution words for the unknown-word replacement step. Finally, a Czech QRNN language model (test-set perplexity 93.54) rescores beam-search candidates, keeping the lowest-perplexity translation.
What would settle it
Run the full pipeline on a German-to-Czech test set, then repeat the unknown-word replacement step after shuffling the word order of the PBSMT output before matching context words. If the roughly 2 point BLEU gain persists under shuffled order, the mechanism is not doing the positional matching the paper claims; if the gain collapses, the order-correspondence assumption is confirmed.
Extended reading notes
Core claim
The central claim is that combining word-level and subword-level (BPE) input representations inside an unsupervised NMT framework, then refining outputs with PBSMT-generated pseudo-parallel fine-tuning, unknown-word replacement (UWR), and language-model rescoring, produces the best German-to-Czech translation in this submission. In the ablations, subword-level NMT beats word-level NMT by about 1.5 BLEU; fine-tuning on PBSMT pseudo-parallel data lifts the subword model from 9.4 to 9.8 BLEU; UWR raises the word-level model from 7.9 to 10.1 BLEU; and Czech LM rescoring adds about 0.3 BLEU on top of each candidate set. The final ensemble of the best word-level and subword-level models, both rescored, achieves 10.6 BLEU (cased).
Load-bearing premise
The load-bearing assumption is that the word-level NMT output and the PBSMT output stay aligned closely enough in word order that matching context words around an unknown token finds the correct replacement; if that ordering breaks, the unknown-word replacement procedure injects wrong words and the reported ~2 BLEU gain disappears.
Editorial extensions
If this is right
- The method does not require German and Czech to share a vocabulary: separate BPE embeddings are trained and then aligned, so the recipe transfers to another language pair without changing the alignment machinery.
- PBSMT-generated pseudo-parallel data can improve a subword-level neural model even when the PBSMT model itself scores lower than the NMT model, because the extra data increases the variety of back-translation training signal.
- Language-model rescoring over beam candidates adds a consistent gain of about 0.3 BLEU per application and provides a principled way to ensemble outputs from heterogeneous models.
- Unknown-word replacement based on a secondary PBSMT translation is a viable alternative to a larger vocabulary for word-level NMT, recovering more BLEU than the gap to subword-level NMT.
Reading between the lines
- The roughly 2 BLEU gain from UWR suggests the word-level NMT and PBSMT outputs are often locally in sync in translation order; a testable extension is to replace the heuristic context matching with learned alignments so the method works for pairs with larger word-order divergence.
- Because the BPE embeddings are aligned rather than shared, the approach points toward applying the same two-step alignment to unrelated languages with little lexical overlap, where a shared subword vocabulary would be empty.
- The 0.3 BLEU rescoring gain may understate the value of the language model: on longer or more ambiguous sentences, a high-coverage target-side LM could be doing more work than a corpus-level BLEU gain reveals.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper describes the CAiRE submission to the WMT 2019 unsupervised machine translation track for German-to-Czech. The system combines a word-level unsupervised NMT model with MUSE-aligned FastText embeddings, a subword-level (BPE) unsupervised NMT model, an unsupervised PBSMT model, fine-tuning on PBSMT-generated pseudo-parallel data, an unknown word replacement (UWR) procedure for the word-level model, language-model-based rescoring of beam-search candidates, and a final ensemble with post-processing. On the WMT19 test set, the system achieves 10.6 BLEU (cased), with the main ablations showing that UWR adds about 2 BLEU to the word-level model, LM rescoring adds 0.1–0.3 BLEU per application, and the final ensemble outperforms each single model by a small margin.
Significance. If the reported results are reliable, the paper offers a useful practical recipe for unsupervised translation of morphologically rich languages, and the component-wise ablations in Table 2 are informative. A notable strength is the transparent, step-by-step evaluation with multiple metrics (BLEU, TER, BEER, CharacterTER), which makes the contribution of each component concrete and facilitates reproduction. The paper also makes explicit that the method uses no parallel data and relies only on monolingual corpora plus pretrained models, which is an advantage for low-resource scenarios. However, the reported gains are small and are not accompanied by statistical validation, and the largest gain (UWR) rests on an alignment assumption that is stated but never verified. The significance of the contribution would be substantially strengthened by additional analysis of the UWR mechanism and by uncertainty estimates for the BLEU differences.
major comments (3)
- [§2.2, Table 2] The 2 BLEU improvement attributed to unknown word replacement (word-level fine-tuned 7.7 -> fine-tuned + UWR 9.6 cased) depends entirely on the assumption stated in §2.2: “the translations from the word-level NMT model and PBSMT model are roughly aligned in order.” The paper provides no evidence for this assumption: no alignment accuracy measurement, no qualitative examples, and no analysis of how often the replacement word is correct. Because the UWR gain is the largest single component improvement and UWR is one of the two claimed contributions, this is a load-bearing point. I recommend the authors validate the assumption, for example by sampling sentences and manually annotating whether the UWR candidates are correct, or by performing a permutation test in which the word order of the PBSMT translation is shuffled before replacement. If the BLEU gain is insensitive to the order, the current justification is misleading; if it is sensitive, the authors should report the alignment accuracy and discuss failure cases.
- [§3.5, Table 2] The paper claims that LM rescoring “helps the model improve by around a 0.3 BLEU score each time,” citing differences such as 9.5 vs. 10.0 for subword-level fine-tuned and 9.6 vs. 9.9 for word-level + UWR. These differences are very small and are reported without error bars, confidence intervals, or significance tests, and it appears that each condition was evaluated with a single run. For NMT systems, run-to-run variance of 0.3–0.5 BLEU is common, so the rescoring improvement may be indistinguishable from noise. Since LM rescoring is the second of the two stated contributions, the authors should provide at least three independent runs with mean and standard deviation, or bootstrap confidence intervals over the test set, and state whether the observed differences are statistically reliable.
- [§2.1.2] The description of subword-level NMT initialization is ambiguous. The abstract and introduction say “training byte-pair encoding (BPE) embeddings for German and Czech separately, and they are aligned using MUSE,” and §2.1.2 says “we train FastText word vectors separately for German and Czech, and apply MUSE to align these embeddings.” MUSE is designed for word-level embeddings; if the aligned embeddings are actually subword units, the alignment procedure and its effectiveness on BPE units should be described explicitly (e.g., whether MUSE is applied to subword embeddings trained on BPE-segmented corpora, and how the alignment quality is measured). If the aligned embeddings are word-level, the paper should clarify how they are used to initialize a model whose vocabulary consists of BPE units. This ambiguity affects reproducibility.
minor comments (5)
- [§2.3] The rescoring section does not state whether the language model perplexity is normalized by sentence length. If it is not, the rescoring may systematically favor shorter sentences, which would be a confound. Please specify the length normalization used when comparing candidates from beam search.
- [§4.1] There is a typo: “senario” should be “scenario.”
- [§3.4 and §3.5] The paper reports “BEER 2.0” in Table 2 without defining or citing the metric. Please add a citation or a brief explanation of what BEER 2.0 indicates.
- [§2.2, Figure 2] The UWR procedure has several free parameters (context window size, the allowed edit distance of the last two characters, and the candidate selection rule). The paper does not analyze their sensitivity. Reporting how the BLEU score varies with the context window size or the edit-distance tolerance would help assess robustness.
- [§3.3] The sentence “two out of 12 million sentences are randomly selected from the the monolingual dataset” contains a duplicated article and does not specify whether the selection is per iteration. Please clarify.
Circularity Check
No circularity: the paper is an empirical WMT system report whose components are externally sourced or independently trained, and no fitted quantity is presented as a prediction.
full rationale
The paper reports an unsupervised German-to-Czech MT pipeline and evaluates it with BLEU/TER against the WMT19 test set. There is no derivation chain in which an output is defined in terms of its own input. The main components (MUSE-aligned embeddings, FastText vectors, Moses PBSMT with KenLM, subword NMT, QRNN Czech language model) are either standard external tools or independently trained on monolingual data. The language model rescoring step selects among beam-search candidates by lowest perplexity; it is not fitted to the BLEU score it is used to improve, so it is not a fitted input called a prediction. The unknown word replacement procedure assumes 'the translations from the word-level NMT model and PBSMT model are roughly aligned in order' (Section 2.2), but this is an untested empirical alignment assumption rather than a circular reduction: the replacement candidates come from a separate PBSMT system, not from the word-level NMT output or from the evaluation metric itself. The paper contains one self-citation (Lee et al., 2019, a SemEval hyperpartisan news detection paper), but it is not load-bearing anywhere in the method or results. No uniqueness theorem, ansatz smuggled via citation, or renaming of a known result is present. The central claim is therefore self-contained as an experimental finding, though its robustness is limited by the unvalidated ordering assumption in the UWR step.
Assumptions & free parameters
free parameters (3)
- Context window size for unknown word replacement =
2
- Beam size for candidate generation =
5
- Vocabulary frequency threshold for Czech LM =
15 occurrences
assumptions (4)
- domain assumption MUSE aligns German and Czech word/BPE embeddings into a shared space with sufficient accuracy for NMT initialization.
- domain assumption Word-level NMT and PBSMT translations are roughly aligned in order, so unknown words can be replaced from corresponding PBSMT positions.
- domain assumption A Czech QRNN language model's perplexity is a reliable selector among beam search candidates.
- domain assumption Back-translation and denoising autoencoding from Lample et al. (2018b) transfer to German-Czech with the described hyperparameters.
Cite this review
Pith. "Pith review of Incorporating Word and Subword Units in Unsupervised Machine Translation Using Language Model Rescoring." pith.science (2026). https://pith.science/paper/7RHUK3ZW
@misc{pith2026190805925,
author = {Pith},
title = {Pith review of: Incorporating Word and Subword Units in Unsupervised Machine Translation Using Language Model Rescoring},
year = {2026},
howpublished = {\url{https://pith.science/paper/7RHUK3ZW}},
note = {Machine review of arXiv:1908.05925}
}
read the original abstract
This paper describes CAiRE's submission to the unsupervised machine translation track of the WMT'19 news shared task from German to Czech. We leverage a phrase-based statistical machine translation (PBSMT) model and a pre-trained language model to combine word-level neural machine translation (NMT) and subword-level NMT models without using any parallel data. We propose to solve the morphological richness problem of languages by training byte-pair encoding (BPE) embeddings for German and Czech separately, and they are aligned using MUSE (Conneau et al., 2018). To ensure the fluency and consistency of translations, a rescoring mechanism is proposed that reuses the pre-trained language model to select the translation candidates generated through beam search. Moreover, a series of pre-processing and post-processing approaches are applied to improve the quality of final translations.
Figures
Reference graph
Works this paper leans on
-
[1]
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451--462
work page 2017
-
[2]
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018 a . Unsupervised statistical machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3632--3642
work page 2018
-
[3]
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018 b . https://openreview.net/forum?id=Sy2ogebAW Unsupervised neural machine translation . In International Conference on Learning Representations
work page 2018
-
[4]
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings
work page 2015
-
[5]
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135--146
2017
-
[6]
Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv\'e J\'egou. 2018. Word translation without parallel data. In International Conference on Learning Representations (ICLR)
work page 2018
-
[7]
Ryan Cotterell, Sebastian J Mielke, Jason Eisner, and Brian Roark. 2018. Are all languages equally hard to language-model? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 536--541
work page 2018
-
[8]
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1243--1252. JMLR. org
2017
Show all 27 references
-
[9]
Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor OK Li. 2018. Universal neural machine translation for extremely low resource languages. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol...
2018
-
[10]
Kenneth Heafield. 2011. Kenlm: Faster and smaller language model queries. In Proceedings of the sixth workshop on statistical machine translation, pages 187--197. Association for Computational Linguistics
2011
-
[11]
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting ...
2007
-
[12]
Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291
2019 arXiv
-
[13]
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018 a . https://openreview.net/forum?id=rkYTTf-AZ Unsupervised machine translation using monolingual corpora only . In International Conference on Learning Representations
2018
-
[14]
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018 b . Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP)
2018
-
[15]
Nayeon Lee, Zihan Liu, and Pascale Fung. 2019. Team yeon-zi at semeval-2019 task 4: Hyperpartisan news detection by de-noising weakly-labeled data. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 1052--1056
2019
-
[16]
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018 a . An Analysis of Neural Language Modeling at Multiple Scales . arXiv preprint arXiv:1803.08240
2018 arXiv
-
[17]
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018 b . https://openreview.net/forum?id=SyyGPP0TZ Regularizing and optimizing LSTM language models . In International Conference on Learning Representations
2018
-
[18]
Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1, pages 160--167. Association for Computational Linguistics
2003
-
[19]
Alec Radford, Luke Metz, and Soumith Chintala. 2016. Unsupervised representation learning with deep convolutional generative adversarial networks. 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings
2016
-
[20]
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 a . Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 86--96
2016
-
[21]
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 b . Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1715--1725
2016
-
[22]
Reut Tsarfaty, Djam \'e Seddah, Yoav Goldberg, Sandra K \"u bler, Marie Candito, Jennifer Foster, Yannick Versley, Ines Rehbein, and Lamia Tounsi. 2010. Statistical parsing of morphologically rich languages (spmrl): what, how and whither. In Proceedings of the NAACL HLT 2010 F...
2010
-
[23]
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan Gomez, Stephan Gouws, Llion Jones, ukasz Kaiser, Nal Kalchbrenner, Niki Parmar, et al. 2018. Tensor2tensor for neural machine translation. In Proceedings of the 13th Conference of the Association for Machine T...
2018
-
[24]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6000--6010. Curran A...
2017
-
[25]
Jiawei Wu, Xin Wang, and William Yang Wang. 2019. Extract and edit: An alternative to back-translation for unsupervised neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu...
2019
-
[26]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...
-
[27]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.