Pith. sign in

REVIEW 3 major objections 4 minor 23 references

On NMT Search Errors and Model Errors: Cat Got Your Tongue?

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Beam search fails to find the model's best-scoring translation on most sentences, and the model often scores the empty translation highest.

desk verdict First exact search reveals NMT models often prefer empty translations while beam search hides it; read this for the result, brace for the unproven termination. read the letter →

arxiv 1908.10090 v1 pith:6K6BOLKM submitted 2019-08-27 cs.CL

Felix Stahlberg , Bill Byrne This is my paper · ORCID
classification cs.CL
keywords neuralmachinetranslationexactinferencebeamsearchdepth-firsterrorsmodelemptylengthbias
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to separate two sources of error in neural machine translation: search errors, where the decoder fails to find the translation the model scores highest, and model errors, where the model itself scores a bad translation highest. The authors construct an exact inference procedure that combines beam search with depth-first search and returns the global best model score for the monotone scoring function of vanilla NMT. Applied to the full WMT15 English-German test set with a Transformer base model, exact search reveals that beam search misses the global best score on most sentences, and that for 51.8% of sentences the model's global best translation is empty, a single end-of-sentence token. The authors conclude that beam search errors are partly masking a severe model bias toward shorter translations, and that simply improving search would make translation quality worse unless the model's adequacy failure is fixed.

What carries the argument

The load-bearing mechanism is the monotonicity of NMT log-scores: because every conditional log-probability in the chain-rule factorization is negative, the score of any partial hypothesis is strictly lower than the score of any of its prefixes. The exact search takes the beam search result to initialize a lower bound $\gamma$, then explores the translation space depth-first and prunes any branch whose accumulated score falls below $\gamma$; whenever a better complete hypothesis is found, the bound is raised. Placing the end-of-sentence token first in the vocabulary order raises $\gamma$ early and sharpens pruning. For length-constrained decoding, the scheme is generalized to length-dependent lower bounds $\gamma_k$, which is what allows exact search under length normalization.

What would settle it

Recompute the global best scores with an independent exact method, such as dynamic programming over length-bounded hypotheses, and compare on a random subset of the 2,169 WMT15 sentences; any disagreement, or any DFS run that fails to terminate within a fixed node budget, would invalidate the reported exact search-error and empty-translation rates.

Watch

Extended reading notes

Core claim

The paper's central discovery is a quantitative separation of search and model errors in NMT under exact inference. For the Transformer base model on WMT15 English-German, greedy decoding has a 73.6% search-error rate and beam-10 has 57.7%, yet beam-10 achieves BLEU 30.3; exact search achieves BLEU 2.1 with length ratio 0.06 because 51.8% of the time the global best model score belongs to the empty translation. The authors show the bias is not an artifact of one architecture: an LSTM baseline has 47.7% empty global bests and SliceNet 41.2%, and even a strong Transformer-Big system has 25.8%. They further show that constraining exact search to a minimum length does not remove the model's preference for very short translations, while length normalization restores realistic lengths but still does not match beam search BLEU. The paper's conclusion is that vanilla NMT relies on search errors to hide a model-level adequacy failure, which is unsatisfactory from a modeling perspective.

Load-bearing premise

The exact depth-first search terminates on every sentence and returns the true global best model score; the paper asserts no execution time limits are needed but concedes in its discussion of related work that the algorithm often converges in practice.

Editorial extensions

If this is right

  • Improving search quality alone, such as raising the beam size, lowers BLEU because fewer search errors means the model's short-translation bias is exposed more often.
  • Exact inference can serve as a diagnostic tool that quantifies, in absolute terms, how often a trained NMT model assigns its best score to degenerate translations.
  • Decoding fixes that only change search, such as heuristics that reward length, address symptoms: length normalization gives plausible length ratios but does not recover beam search's BLEU in the paper's experiments.
  • The empty-translation problem worsens with source length, so models that score well on short sentences may be substantially more degraded on long inputs.
  • The reported rates imply that model-level adequacy, not search, is the binding constraint for these systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the empty-global-best phenomenon generalizes across language pairs and domains, then reporting BLEU under beam search systematically overstates model adequacy; benchmark results should be complemented by exact-search or length-controlled diagnostics.
  • A natural testable extension is to apply the exact search to models trained with objectives designed to fix length bias, such as global conditioning or margin-based training, and check whether the empty-global-best rate drops; the paper does not run this experiment.
  • The monotonicity-based pruning could be replaced by an admissible A*-style search with a length-aware heuristic to independently verify the reported exact scores on a random subset, avoiding reliance on DFS termination in practice.
  • The length-constrained exact-search results suggest a practical inference-time recipe: decode with a minimum length constraint set by the source length, then compare exact scores across that constrained space; the paper's numbers hint this could restore adequacy without retuning, but they do not claim it.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents an exact decoding algorithm for neural machine translation that combines beam search with depth-first search, exploiting the monotonicity of locally normalized log-probability scores to prune partial hypotheses. Using this algorithm on the WMT15 English-German test set with a Transformer base model, the authors report that beam search with beam size 10 fails to find the global best model score on 57.7% of sentences, and that for 51.8% of sentences the model assigns the global best score to the empty translation (a single end-of-sentence token). They further show that larger beams reduce search errors but degrade BLEU because translations become too short, and they use length-constrained exact search and length normalization experiments to argue that the empty-translation phenomenon reflects an inherent model bias toward shorter translations.

Significance. If the empirical claims are correct, this is a valuable and striking quantification of the interplay between search errors and model errors in NMT, and it provides concrete evidence for the previously suspected length bias of locally normalized sequence models. The paper's strengths include an open-source implementation in the SGNMT decoder, replication of the main finding across four architectures, and a simple, parameter-free pruning criterion whose monotonicity argument is mathematically sound. The beam-derived lower bound is a legitimate lower bound rather than a circular constraint. However, the exactness of the headline result depends on a termination guarantee that is asserted but not proved, and the length-constrained experiments are run on tractability-based subsets, so the secondary conclusions are less secure.

major comments (3)
  1. [Section 3, Algorithm 2, footnote 6] The central exactness claim for Table 1 is not backed by a termination guarantee. Monotonicity (Eq. 3) only tells us that extending a prefix lowers the score; it does not rule out an infinite branch whose scores converge to a limit strictly above the current lower bound γ. On such a branch, the condition p′ ≥ γ in line 7 of Algorithm 2 remains true forever, so the DFS would not terminate and the global maximum over the finite-string search space may not exist. The paper's own Related Work says the DFS 'often converges in practice,' and footnote 6 asserts unconstrained exact search 'does not need maximum execution time limits' without proof. Please supply a termination theorem under explicit model assumptions or an empirical certificate (e.g., maximum visited depth and search time for every one of the 2,169 sentences), and state how the 51.8% empty-translation and 57.7% search-error percentages would be affected if any sentence required a stopping heuristic.
  2. [Section 4, footnote 6, Tables 3 and 4] The length-constrained experiments cover only subsets (73.0% and 48.3%) of the test set, yet the abstract and discussion draw conclusions about length bias and length normalization from them. Because footnote 6 says decoding was stopped after one day, the subsets may be selected by tractability, which would bias the comparison between Beam-10 and exact search. Please describe how the subsets were selected, report the source-length and difficulty statistics of included versus excluded sentences, and state which conclusions are claimed only for the feasible subset.
  3. [Section 4, exact inference under length constraints] The generalized exact search under length normalization is not specified in sufficient detail to verify the claim of finding global best translations for each length k. Equation (4) gives only the initial lower bounds; the length-dependent pruning rule, the range of k, and the termination condition are left to the cited SGNMT implementation. Please provide a formal algorithm or pseudocode for this variant, or restrict the claim to the unconstrained exact search.
minor comments (4)
  1. [Section 3, Table 1] Please state the beam size used to compute the initial lower bound γ for the exact search; the exactness of Algorithm 2 does not depend on it, but reproducibility does.
  2. [Section 4, Table 3] The row label 'Exact for Beam-10 length' should define whether the constraint fixes the exact target length to the length of the best Beam-10 hypothesis or imposes an inequality; the text suggests the former, but the table is ambiguous.
  3. [Footnote 3] The sentence 'Comparable with http://matrix.statmt.org/' is incomplete; please specify which BLEU score is comparable and under what tokenization.
  4. [Figures 3 and 5] The histograms would benefit from labeled bin edges and counts so that the peak in [0.0, 0.1] and the subset size in Figure 5 (73.0%) are directly readable.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the exact-search comparison is self-contained and its conclusions do not reduce to fitted inputs or self-citations.

full rationale

This paper reports an empirical comparison between approximate beam search and a proposed exact DFS search over a fixed pretrained Transformer. The 'global best' scores are obtained by Algorithm 2, whose only input besides the model is gamma, the beam-search score used as a lower bound. Using a lower bound from a weaker decoder is a valid algorithmic technique and does not shape the result: DFS either finds a complete hypothesis with score at least gamma or proves none exists, and gamma is updated upward when better hypotheses are found. The empty-translation phenomenon is a property of the trained model's score landscape discovered by exhaustive search; it is not defined in terms of beam search or any fitted parameter. The paper fits no parameters and retrains no model; Tables 1 and 2 compare fixed models. Self-citations are limited to pointing to the SGNMT decoder (a code release), the preprocessing pipeline from the same group's WMT18 system, and a prior constraints-based analysis; none of these citations carries a load-bearing assumption needed for the exact-search result. The only notable weakness is the unproved termination of the DFS: the monotonicity of Eq. 3 does not by itself bound the number of prefixes with score above gamma, since per-token log-probabilities can be arbitrarily close to zero. The paper itself says the algorithm 'often converges in practice' (Related Work) and asserts in footnote 6 that unconstrained exact search 'does not need maximum execution time limits.' This is a correctness or completeness risk about whether the reported 51.8% empty-translation and 57.7% search-error numbers cover all 2,169 sentences, not a circularity: the claim would fail on external mathematical grounds (non-termination), not by reducing to its own inputs. Therefore no circular step can be exhibited, and the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central empirical claims do not depend on fitted free parameters. The main load-bearing premises are the monotonicity of NMT scores, which is mathematically valid, the practical termination of exact search, which is assumed rather than proven, and the representativeness of the four architectures.

assumptions (3)
  • domain assumption The conditional log-probabilities in an NMT model are strictly negative, so extending a partial hypothesis always lowers its score (Eq. 3).
    This monotonicity property is the basis for admissible pruning in Algorithm 2. It holds for vanilla locally normalized models, but not for length normalization or word rewards, as the paper notes in Section 2.
  • ad hoc to paper The exact depth-first search terminates on all test sentences without external time limits.
    The paper states in footnote 6 that unconstrained exact search is much faster and needs no time limits, but provides no termination proof; Related Work says the algorithm 'often converges in practice', so this is an unproven practical assumption.
  • domain assumption The four tested model architectures (Transformer-Base, LSTM, SliceNet, Transformer-Big) are representative enough to support conclusions about 'vanilla NMT in its current form'.
    The abstract and conclusion generalize from these four trained models to NMT broadly; this assumes the failure is not specific to these particular systems.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On NMT Search Errors and Model Errors: Cat Got Your Tongue?." pith.science (2026). https://pith.science/paper/6K6BOLKM

@misc{pith2026190810090,
  author       = {Pith},
  title        = {Pith review of: On NMT Search Errors and Model Errors: Cat Got Your Tongue?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6K6BOLKM}},
  note         = {Machine review of arXiv:1908.10090}
}
read the original abstract

We report on search errors and model errors in neural machine translation (NMT). We present an exact inference procedure for neural sequence models based on a combination of beam search and depth-first search. We use our exact search to find the global best model scores under a Transformer base model for the entire WMT15 English-German test set. Surprisingly, beam search fails to find these global best model scores in most cases, even with a very large beam size of 100. For more than 50% of the sentences, the model in fact assigns its global best score to the empty translation, revealing a massive failure of neural models in properly accounting for adequacy. We show by constraining search with a minimum translation length that at the root of the problem of empty translations lies an inherent bias towards shorter translations. We conclude that vanilla NMT in its current form requires just the right amount of beam search errors, which, from a modelling perspective, is a highly unsatisfactory conclusion indeed, as the model often prefers an empty translation.

Figures

Figures reproduced from arXiv: 1908.10090 by the authors.

Figure 1
Figure 1. BLEU over the percentage of search er￾rors. Large beam sizes yield fewer search errors but the BLEU score suffers from a length ratio below 1. -12.2 -12.0 -11.8 -11.6 -11.4 -11.2 -11.0 -10.8 10 20 30 40 50 60 70 80 90 100 55% 60% 65% 70% Log-likelihood #Search errors Beam size Log-likelihood #Search errors [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Even large beam sizes produce a large num [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 5
Figure 5. Histogram over length ratios with minimum [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Number of search errors under Beam-10 and [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 12 canonical work pages

  1. [1]

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR

    2015

  2. [2]

    Nicolas Boulanger-Lewandowski, Yoshua Bengio, and Pascal Vincent. 2013. Audio chord recognition with recurrent neural networks. In ISMIR, pages 335--340. Citeseer

    work page 2013

  3. [3]

    Yining Chen, Sorcha Gilroy, Andreas Maletti, Jonathan May, and Kevin Knight. 2018. https://doi.org/10.18653/v1/N18-1205 Recurrent neural networks as weighted language recognizers . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pag...

    doi:10.18653/v1/n18-1205 2018

  4. [4]

    Wei He, Zhongjun He, Hua Wu, and Haifeng Wang. 2016. Improved neural machine translation with SMT features. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pages 151--157. AAAI Press

    work page 2016

  5. [5]

    Liang Huang, Kai Zhao, and Mingbo Ma. 2017. https://doi.org/10.18653/v1/D17-1227 When to finish? O ptimal beam search for neural text generation (modulo beam size) . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2134--2139, Copenhagen, Denmark. Association for Computational Linguistics

    doi:10.18653/v1/d17-1227 2017

  6. [6]

    S \'e bastien Jean, Orhan Firat, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. https://doi.org/10.18653/v1/W15-3014 Montreal neural machine translation systems for WMT ' 15 . In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 134--140, Lisbon, Portugal. Association for Computational Linguistics

    work page doi:10.18653/v1/w15-3014 2015

  7. [7]

    Lukasz Kaiser, Aidan N Gomez, and Francois Chollet. 2017. Depthwise separable convolutions for neural machine translation. arXiv preprint arXiv:1706.03059

    arXiv 2017

  8. [8]

    Nal Kalchbrenner and Phil Blunsom. 2013. http://aclweb.org/anthology/D13-1176 Recurrent continuous translation models . In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1700--1709. Association for Computational Linguistics

    work page 2013

Show all 23 references
  1. [9]

    Philipp Koehn and Rebecca Knowles. 2017. https://doi.org/10.18653/v1/W17-3204 Six challenges for neural machine translation . In Proceedings of the First Workshop on Neural Machine Translation, pages 28--39, Vancouver. Association for Computational Linguistics

    2017 doi

  2. [10]

    Aviral Kumar and Sunita Sarawagi. 2019. Calibration of encoder decoder models for neural machine translation. arXiv preprint arXiv:1903.00802

    2019 arXiv

  3. [11]

    Kenton Murray and David Chiang. 2018. https://www.aclweb.org/anthology/W18-6322 Correcting length bias in neural machine translation . In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 212--223, Belgium, Brussels. Association for Computation...

    2018

  4. [12]

    Jan Niehues, Eunah Cho, Thanh-Le Ha, and Alex Waibel. 2017. Analyzing neural MT search and model performance. In Proceedings of the First Workshop on Neural Machine Translation, pages 11--17

    2017

  5. [13]

    Myle Ott, Michael Auli, David Grangier, et al. 2018. Analyzing uncertainty in neural machine translation. In International Conference on Machine Learning, pages 3953--3962

    2018

  6. [14]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. https://doi.org/10.18653/v1/P16-1162 Neural machine translation of rare words with subword units . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    2016 doi

  7. [15]

    Pavel Sountsov and Sunita Sarawagi. 2016. https://doi.org/10.18653/v1/D16-1158 Length bias in encoder decoder models and a case for global conditioning . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1516--1525, Austin, Texas....

    2016 doi

  8. [16]

    Felix Stahlberg, Adri \`a de Gispert, and Bill Byrne. 2018 a . https://www.aclweb.org/anthology/W18-6427 The University of C ambridge ' s machine translation systems for WMT 18 . In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 504--512,...

    2018

  9. [17]

    Felix Stahlberg, Eva Hasler, Danielle Saunders, and Bill Byrne. 2017. https://doi.org/10.18653/v1/D17-2005 SGNMT -- A flexible NMT decoding platform for quick prototyping of new models and search strategies . In Proceedings of the 2017 Conference on Empirical Methods in Natura...

    2017 doi

  10. [18]

    Felix Stahlberg, Danielle Saunders, Gonzalo Iglesias, and Bill Byrne. 2018 b . http://aclweb.org/anthology/W18-1821 Why not be versatile? Applications of the SGNMT decoder for machine translation . In Proceedings of the 13th Conference of the Association for Machine Translatio...

    2018

  11. [19]

    Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104--3112

    2014

  12. [20]

    Gomez, Stephan Gouws, Llion Jones, ukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit

    Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, ukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. http://arxiv.org/abs/1803.07416 Tensor2tensor for neural machine transl...

    2018 arXiv

  13. [21]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998--6008

    2017

  14. [22]

    Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144

    2016 arXiv

  15. [23]

    Yilin Yang, Liang Huang, and Mingbo Ma. 2018. https://www.aclweb.org/anthology/D18-1342 Breaking the beam search curse: A study of (re-) scoring methods and stopping criteria for neural machine translation . In Proceedings of the 2018 Conference on Empirical Methods in Natural...

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.