Pith. sign in

REVIEW 5 minor 27 references

Simple and Effective Noisy Channel Modeling for Neural Machine Translation

T0 review · 0 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A backward translation model beats ensembles by up to 3.2 BLEU

desk verdict A simple, carefully tested seq2seq channel model that improves NMT by a few BLEU; the empirical claims hold up, with one missing online-decoding comparison. read the letter →

arxiv 1908.05731 v1 pith:TEEPA5YU submitted 2019-08-15 cs.CL

Kyra Yee , Nathan Ng , Yann N. Dauphin , Michael Auli This is my paper · ORCID
classification cs.CL
keywords noisychannelmodelneuralmachinetranslationsequence-to-sequenceTransformerrerankinglanguagebeamsearchexplainingaway
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the channel half of noisy channel machine translation can be an ordinary target-to-source sequence-to-sequence model—no latent variables, no prefix-specific architecture. A Transformer trained on full sentence pairs in the reverse direction is pressed into service as $p(x|y)$, scoring a complete source from whatever target prefix has been generated so far. Combined with a neural language model and the direct model, this channel model consistently beats ensembles of direct models and right-to-left reranking, improving WMT'17 German-English by up to 3.2 BLEU and showing gains on all four language pairs tested. The practical point is that a strong noisy channel system can be assembled from components that already exist.

What carries the argument

The load-bearing object is the noisy channel decomposition with a standard Transformer as the channel: $p(y|x) \propto p(x|y)p(y)$, with $p(x)$ dropped as constant. The channel $p(x|y)$ is computed by summing per-token log probabilities over the source in the reverse direction. During beam search the direct model proposes $k_1 \times k_2$ candidate extensions, the channel plus language model rescore them, and the final score is the length-normalized combination \[\frac{1}{t}\log p(y|x) + \$\lambda$\frac{1}{s}\big(\log p(x|y) + \log p(y)\big),\] where $t$ is the target prefix length and $s$ is the source length. The per-word scaling by $t$ and $s$ is what keeps the direct and channel scores comparable as decoding progresses.

What would settle it

On a held-out parallel set, take a beam of candidate translations from the direct model, truncate each hypothesis after 5, 10, and 20 target tokens, and check whether the standard reverse Transformer's channel scores rank the oracle (best-BLEU) hypothesis above the others. If the standard model fails to discriminate at short prefixes while a model trained on all target prefixes succeeds, the core robustness claim would be refuted.

Watch

Extended reading notes

Core claim

The central claim is empirical: a standard Transformer sequence-to-sequence model trained to translate from target to source is a well-calibrated enough channel model that decoding by $\arg\max_y \log p(x|y) + \log p(y)$ works, even though inference asks it to condition on incomplete target prefixes it never saw in training. The paper finds the train/test mismatch is harmless after roughly fifteen tokens of target context, and that scoring the entire source—rather than a prefix as in previous neural noisy channel models—is what gives the approach its edge. On full sentences the reverse seq2seq channel even beats a model trained on all possible target prefixes by 5.7 BLEU, which the authors attribute to the prefix model splitting its capacity across long and short inputs. The result is not just a new model; it is evidence that the expensive, specially designed channel model is unnecessary.

Load-bearing premise

The reverse Transformer is assumed to output meaningful probabilities when conditioned on target prefixes it was never trained on, so that $p(x|y_1,\dots,y_k)$ is a reliable score during beam search.

Editorial extensions

If this is right

  • Adding the channel and language model improves online decoding over a two-model ensemble by 0.9 BLEU on WMT'17 German-English news2017, and over the single direct model by 1.9 BLEU.
  • In n-best reranking the channel approach gains more from larger beams than alternatives, rising 2.4 BLEU from beam 5 to beam 100 on WMT De-En news2016.
  • The channel model makes better use of a language model than direct-plus-LM reranking, because conditioning on $p(x|y)$ counters explaining-away effects that let the direct model ignore the source.
  • For any target prefix fraction, scoring the entire source is at least as good as scoring any source prefix, so future channel architectures should condition on the full source.
  • The same channel model is beneficial across WMT'17 German-English, English-German, Chinese-English and IWSLT'14 German-English.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit is using this channel scoring outside translation: any conditional generation task with a full conditioning context could be reranked by a reverse seq2seq model plus a language model, e.g., summarization or data-to-text.
  • Because the channel model is just a reverse Transformer, it inherits every future architecture improvement; one could also ensemble channel models or train several random seeds to push the reranking gains further.
  • The authors' prefix-model comparison suggests an easy testable variant: train the reverse model only on long target prefixes (say, at least 15 tokens), which might close the gap on short prefixes while avoiding the full-training mismatch.
  • The single scalar $\lambda$ in the scoring combination was chosen for simplicity; a length-dependent or token-dependent weight, or a learned gating between direct and channel scores, could improve short-target decisions in online decoding.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 5 minor

Summary. The paper proposes modeling the noisy channel probability p(x|y) in neural machine translation with a standard Transformer sequence-to-sequence model trained on complete sentence pairs, and using this channel in combination with a direct model and an external language model. The setup is evaluated in two modes: online decoding with a two-step beam search, where the direct model pre-prunes extensions and the channel model rescores the resulting partial target prefixes, and n-best list reranking, where complete target hypotheses are scored. Experiments on WMT'17 De-En, En-De, Zh-En and IWSLT'14 De-En show consistent BLEU gains over the direct model, an ensemble of direct models, and right-to-left reranking, with the largest gains reported on large beams. The paper includes ablations on per-word score normalization, an analysis of target-prefix length, and results averaged over three random initializations with standard deviations.

Significance. The contribution is practically and empirically significant. The proposed channel parameterization is much simpler than the latent-variable noisy channel of Yu et al. (2017), and the paper demonstrates that it yields large and consistent improvements, including up to 3.2 BLEU on WMT'17 De-En and up to 4.0 BLEU over direct decoding in the reranking setting with beam 100. The experimental methodology is careful: three seeds with standard deviations, four language pairs, comparisons against strong alternatives, and released code and pretrained models. The observation that a vanilla sequence-to-sequence model works as a channel model under a train/test mismatch is interesting and practically useful. The main caveat is that the support for the claimed robustness to incomplete target prefixes is somewhat indirect, but this does not undermine the headline empirical results.

minor comments (5)
  1. [Section 3.1, Figure 1] The claim that standard sequence-to-sequence models are robust to incomplete target prefixes is supported by a proxy experiment: Figure 1 measures source-prediction BLEU rather than the actual combined score in Eq. (1). An online-decoding comparison against a prefix-trained channel model, or an explicit sentence stating that Figure 1 is a proxy, would make the robustness claim more precise.
  2. [Table 1 and Eq. (1)] The ablation without per-word scores drops from 36.2 to 35.1 BLEU on news2017, which shows that the length normalization in Eq. (1) is important for the reported online-decoding gains; the paper could state this more explicitly and explain why normalizing the language model term by the source length s rather than the target length t is appropriate.
  3. [Section 3.2-3.5] The paper references the latent-variable channel model of Yu et al. (2017) as motivation but does not include it as an experimental baseline; a sentence explaining why it is omitted would help the reader calibrate the contribution.
  4. [Experimental settings] The tuned values of the interpolation weight lambda and the length penalty are not reported for individual configurations; providing these values, or stating that they are available in the released code, would improve reproducibility.
  5. [Minor editorial issues] There are several typos and small inconsistencies, including 'Mach ine' in the title, 'langauge' in Section 3, and the caption of Table 4 referring to news2017 while the body text for Table 1 says news2017 and the appendix table header says news2017; these should be corrected.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the channel-model derivation is a legitimate factorization, evaluated on held-out test sets, with only non-load-bearing self-citations.

full rationale

The paper's central step is Bayes' rule applied to a separately trained reverse-direction Transformer channel model p(x|y), a direct model p(y|x), and a language model p(y). This is a genuine factorization rather than a tautology: the channel model is trained on the same parallel corpus as the direct model, but that is the intended noisy-channel decomposition, not a re-use of the predicted quantity as an input. The evaluation is on held-out news test sets against external baselines (direct model, ensemble, right-to-left reranking), with averages and standard deviations over three random initializations. The only tuned quantities are the interpolation weight λ and a length penalty, both selected on validation; no test-set fitting occurs. The acknowledged train/test mismatch for partial target prefixes is addressed empirically in Section 3.1 and Figure 1 via a proxy experiment, which supports the robustness claim rather than assuming it. Self-citations (e.g., Ng et al. 2019 for WMT usage, Baevski and Auli 2018 for the LM architecture) are contextual and not load-bearing: the headline BLEU improvements in Tables 1–3 are computed in this paper and do not reduce to any cited result. No uniqueness theorem, fitted-parameter-renamed-as-prediction, or ansatz-smuggled-via-citation pattern is present. The strongest claim, that channel+direct+LM improves BLEU by up to 3.2 on WMT'17 De-En, is a direct measurement, not a consequence of how the scoring function is defined.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method's success depends on Bayes' rule (standard), the empirical robustness of seq2seq models to partial-input conditioning, a strong language model prior, and the chosen length normalization scheme. No new entities are introduced. The only fitted quantity is the interpolation weight lambda (plus a length penalty), both tuned on validation data.

free parameters (3)
  • lambda (interpolation weight) = tuned on validation, specific values not reported in paper
    Equation (1) combines the direct score and the channel+LM score with a tunable weight lambda; the paper tunes lambda and a length penalty on the validation set per configuration.
  • length penalty = tuned on validation, value not reported
    Used in beam search scoring; tuned together with lambda on the validation set.
  • beam sizes k1=5, k2=10 = fixed at 5 and 10
    Chosen for the decoding algorithm; not tuned per dataset, but they affect the search space and thus the results.
assumptions (4)
  • standard math Bayes' rule: p(y|x) = p(x|y)p(y)/p(x), with p(x) constant and ignored.
    Used in Section 2 to derive the noisy channel scoring from the direct model; standard probability theory.
  • domain assumption A standard Transformer seq2seq model trained on full sentence pairs produces meaningful conditional probabilities when conditioned on partial target prefixes, despite never seeing such inputs during training.
    Central train/test mismatch in Section 2; the paper's entire method rests on this empirical robustness.
  • domain assumption The language model p(y) estimated on monolingual data is a good prior for target sentences.
    The method depends on a strong LM; the paper trains Transformer LMs on billions of words, and the gains come largely from the LM contribution.
  • domain assumption Length normalization (1/t and 1/s) and a single interpolation weight lambda are sufficient to balance direct, channel, and LM scores throughout decoding.
    Equation (1) and the ablation 'per word scores' in Table 1 show this choice matters; it is a modeling assumption rather than a proven necessity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Simple and Effective Noisy Channel Modeling for Neural Machine Translation." pith.science (2026). https://pith.science/paper/TEEPA5YU

@misc{pith2026190805731,
  author       = {Pith},
  title        = {Pith review of: Simple and Effective Noisy Channel Modeling for Neural Machine Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TEEPA5YU}},
  note         = {Machine review of arXiv:1908.05731}
}
read the original abstract

Previous work on neural noisy channel modeling relied on latent variable models that incrementally process the source and target sentence. This makes decoding decisions based on partial source prefixes even though the full source is available. We pursue an alternative approach based on standard sequence to sequence models which utilize the entire source. These models perform remarkably well as channel models, even though they have neither been trained on, nor designed to factor over incomplete target sentences. Experiments with neural language models trained on billions of words show that noisy channel models can outperform a direct model by up to 3.2 BLEU on WMT'17 German-English translation. We evaluate on four language-pairs and our channel models consistently outperform strong alternatives such right-to-left reranking models and ensembles of direct models.

Figures

Figures reproduced from arXiv: 1908.05731 by the authors.

Figure 1
Figure 1. Comparison of two channel models: a stan [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Impact of target prefix length for the channel [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 17 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Alexei Baevski and Michael Auli. 2018. Adaptive input representations for neural language modeling. arXiv, abs/1809.10853

    arXiv 2018

  4. [4]

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR

    2015

  5. [5]

    Peter F Brown, Vincent J Della Pietra, Stephen A Della Pietra, and Robert L Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational linguistics, 19(2):263--311

    1993

  6. [6]

    Yongchao Deng, Shanbo Cheng, Jun Lu, Kai Song, Jingang Wang, Shenglan Wu, Liang Yao, Guchun Zhang, Haibo Zhang, Pei Zhang, Changfeng Zhu, and Boxing Chen. 2018. Alibaba's neural machine translation systems for wmt18. In Proc. of WMT

    work page 2018

  7. [7]

    Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018 a . Understanding back-translation at scale. In Proc. of EMNLP

    work page 2018

  8. [8]

    Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc'Aurelio Ranzato. 2018 b . Classical structured prediction losses for sequence to sequence learning. In Proc. of NAACL

    work page 2018

Show all 27 references
  1. [9]

    Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional Sequence to Sequence Learning . In Proc. of ICML

    2017

  2. [10]

    Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine translation. arXiv, abs/1503.03535

    2015 arXiv

  3. [11]

    Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, Will Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia...

    2018 arXiv

  4. [12]

    Marcin Junczys-Dowmunt. 2018. Microsoft's submission to the wmt2018 news translation task: How i learned to stop worrying and love the data. In Proc. of WMT

    2018

  5. [13]

    Dan Klein and Christopher Manning. 2001. Conditional structure versus conditional estimation in nlp. In Proc. of EMNLP

    2001

  6. [14]

    Philipp Koehn, Kevin Duh, and Brian Thompson. 2018. The jhu machine translation systems for wmt 2018. In Proc. of WMT

    2018

  7. [15]

    Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proc. of NAACL

    2003

  8. [16]

    Gurvan L'Hostis, David Grangier, and Michael Auli. 2016. Vocabulary selection strategies for neural machine translation. arXiv, abs/1610.00072

    2016 arXiv

  9. [17]

    Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook fair's wmt19 news translation task submission. In Proc. of WMT

    2019

  10. [18]

    Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proc. of NAACL System Demonstrations

    2019

  11. [19]

    Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proc. of WMT

    2018

  12. [20]

    Matt Post. 2018. A call for clarity in reporting bleu scores. arXiv, abs/1804.08771

    2018 arXiv

  13. [21]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 a . Improving neural machine translation models with monolingual data. In Proc. of ACL

    2016

  14. [22]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 b . Neural machine translation of rare words with subword units. In Proc. of ACL

    2016

  15. [23]

    Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to Sequence Learning with Neural Networks . In Proc. of NIPS

    2014

  16. [24]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need . In Proc. of NIPS

    2017

  17. [25]

    Yuguang Wang, Shanbo Cheng, Liyang Jiang, Jiajun Yang, Wei Chen, Muze Li, Lin Shi, Yanfeng Wang, and Hongtao Yang. 2017. Sogou neural machine translation systems for wmt17. In Proceedings of the Second Conference on Machine Translation, pages 410--415

    2017

  18. [26]

    Weijia Xu and Marine Carpuat. 2018. The university of maryland's chinese-english neural machine translation systems at wmt18. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 535--540

    2018

  19. [27]

    Lei Yu, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Tom \' a s Kocisk \' y . 2017. The neural noisy channel. In Proc. of ICLR

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.