REVIEW 4 major objections 5 minor 19 references
Efficient Bidirectional Neural Machine Translation
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Training a single encoder–decoder model to generate translations in both left-to-right and right-to-left order, then reranking the two candidate sets, beats unidirectional translation baselines by 0.8–1.32 BLEU while halving the parameter…
desk verdict A genuinely simple single-model bidirectional decoding idea; the BLEU gains are likely real in part, but the paper's ablations miss the candidate-pool-size control, so the headline numbers should be read with caution. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the shared-parameter bidirectional objective and its approximate inference loop. Equation (1) trains one decoder to maximize, for every target token $y_t$, both the probability given the left context $y_{<t}$ and the probability given the right context $y_{>t}$, with two learned start-token embeddings telling the decoder which direction it is generating in; reversed target sequences are mixed into the same mini-batch. At inference, Algorithm 1 generates $K$ beam candidates in each direction, scores every candidate with both directional probabilities, and picks the one maximizing the sum in Eq. (2) as an approximation to the intractable argmax over all sequences. This machinery is what lets one parameter set supply two complementary candidate pools and a joint ranking over them.
What would settle it
Run Algorithm 1 on WMT14 En→De with every candidate scored by the left-to-right probability alone instead of the sum in Eq. (2). The paper's claim predicts BLEU falls from 29.30 toward the 28.73 level of the no-right-to-left-candidates ablation; if BLEU stays at 29.30, the right-to-left scores are not the source of the gain.
Extended reading notes
Core claim
The central claim is that maximizing the sum of left-to-right and right-to-left log-likelihoods under one parameter set—$\sum_t (\log P(y_t|y_{<t},x)+\log P(y_t|y_{>t},x))$—yields a single model that decodes well in both directions, and that reranking beam candidates by that same sum produces better translations than either direction alone. The paper argues the right-to-left pass is not just a regularizer: the ablation removing right-to-left candidates drops WMT14 En→De from 29.30 to 28.73 BLEU, close to the 28.40 unidirectional baseline, while 33.8% of the final selections come from the right-to-left beam. With only left-to-right decoding the trained model nearly matches the baseline (28.25 vs 28.40), so the gain is attributed to candidate combination rather than to a stronger per-token model. The method therefore claims bidirectional decoding at the cost of a single model, with only about 10% extra inference time for the reverse scoring pass.
Load-bearing premise
The load-bearing premise is that one parameter set can model the left-to-right and right-to-left target distributions accurately enough that their summed score ranks better translations above either direction's best candidate.
Editorial extensions
If this is right
- A single trained checkpoint can be decoded in either direction with no significant quality loss, so applications that need both directions no longer need two separate models.
- Bidirectional reranking becomes affordable for deployment: about half the parameters and 40% less training time than a two-model ensemble, with only roughly 10% extra inference cost.
- Any autoregressive sequence-generation task that suffers from error propagation could adopt the same training and reranking recipe; the paper names text summarization and image captioning as the next applications.
- When the decoder is made deeper (up to 6 layers), the gap over the unidirectional baseline grows, suggesting the benefit scales with decoder capacity.
Reading between the lines
- Editorial inference: because the paper's motivation is that right-side tokens degrade with length, a length-stratified BLEU breakdown would likely show the gains concentrate on longer sentences; the paper does not report that split.
- Editorial inference: the joint objective is a form of consistency training between two orderings of the same target, so the same trick could be applied to any autoregressive generator with two orderings, not only translation.
- Editorial inference: the near-parity of left-to-right-only decoding with the baseline suggests the objective may also act as a regularizer; comparing against a self-ensemble of the same model would separate regularization from directional complementarity.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a single encoder-decoder NMT model that is trained with both left-to-right and right-to-left objectives using direction-specific start tokens, so that one parameter set can decode in both directions. At inference, Algorithm 1 generates K candidate sequences from each direction, scores all 2K candidates with the sum of L2R and R2L log-probabilities (Eq. 2), and selects the best. Experiments on IWSLT14 De→En, WMT14 En→De, and WMT17 Zh→En report improvements of 0.8–1.32 BLEU over a unidirectional Transformer baseline, plus efficiency gains compared with training two separate directional models. The paper also claims a new state-of-the-art score of 29.30 BLEU on WMT14 En→De.
Significance. If the reported gains are real and attributable to the bidirectional mechanism, the method is a simple and useful contribution: it achieves bidirectional decoding in a single model with a modest training-time overhead relative to a unidirectional baseline and a large parameter savings relative to two-model ensembles. The method is clearly specified, the training and inference procedures are reproducible, and the three-task evaluation provides useful cross-lingual evidence. However, the empirical claims are weakened by two confounds that are not addressed in the current manuscript: the baseline uses a smaller candidate pool than the proposed method, and the proposed model is trained for more iterations. These issues are fixable, but they must be resolved before the central claim is secure.
major comments (4)
- [§3.2 and Algorithm 1] The reported improvements are not isolated from the effect of generating twice as many candidates. The bidirectional method generates 2K sequences (K per direction) and reranks them, while the baseline is a single beam search with K candidates. The ablation that restricts candidates to L2R only reduces the pool to K, so the drop from 29.30 to 28.73 in §3.2 could be due to pool size rather than to the value of the R2L candidates. The paper should include a unidirectional baseline with beam size 2K and a L2R-only variant with 2K candidates reranked using Eq. (2). Without these controls, the central claim that bidirectional decoding is responsible for the gains is not established.
- [§3.2, Training and Inference] The authors state that 'our method needs 1.2x iterations to converge.' Because the bidirectional model is trained for 20% more updates than the baseline, the improvement on WMT14 En→De (and possibly the other tasks) may reflect additional training rather than the bidirectional objective. The evaluation should match the number of training updates, for example by training the baseline for 1.2x iterations or by evaluating the bidirectional model at the same iteration count as the baseline, to separate the effect of the method from the effect of longer training.
- [Tables 2–4] The paper reports BLEU differences of 0.8–1.32 points but provides no confidence intervals, bootstrap resampling, or significance tests. Given the known variance of BLEU on these test sets, especially IWSLT14 with its small test set, these differences could be within noise. The authors should provide significance estimates (e.g., paired bootstrap resampling) or at least error bars for the main comparisons, so that the reader can assess whether the claimed improvements are statistically meaningful.
- [§3.2, 'Analyzing of the Contribution of Right-To-Left Decoding'] The sentence 'If we only use sentences generated in left-to-right as candidates and keep other factors unchanged' is inaccurate because the number of candidates is reduced from 2K to K. This changes a factor that the authors explicitly say is unchanged. The ablation therefore does not support the conclusion that the improvement is due to bidirectionally generated candidates rather than to self-ensemble or pool-size effects. A proper ablation should compare L2R-only with K candidates, L2R-only with 2K candidates, and the full bidirectional approach with 2K candidates, all reranked with the same scoring function.
minor comments (5)
- [Section 3, first sentence] The opening sentence says 'to very the effectiveness' but should be 'to verify the effectiveness.'
- [Table 2 footnote] The footnote states that the reimplemented baseline gives 28.36 BLEU but the table lists 28.40 from Vaswani et al. (2017b). It would be clearer and more consistent to use the reimplemented score for the comparison, or to explain why the original score is preferred.
- [Abstract and Section 3.2] The claim that the method 'improve[s] inference speed' is vague; the comparison appears to be against a two-model ensemble, but the abstract does not specify this. Clarify the reference point for the inference-time claim.
- [Section 2.1] The notation P(←y|x;θ) is not explicitly defined; the paper should clarify that it denotes the probability of the reversed target sequence under the right-to-left decoding direction.
- [Appendix C, Figure 1(b)] The text says 'We vary the beam size on IWSLT14 En→De validation set as well as the test set,' but the task in the paper is IWSLT14 De→En. Correct the direction.
Circularity Check
No significant circularity: the bidirectional objective, inference reranking, and external BLEU benchmarks are self-contained; self-citations are background only.
full rationale
The paper's central claim — that a single encoder-decoder trained with the joint left-to-right and right-to-left objective of Eq. (1) and decoded by the Eq. (2) reranking in Algorithm 1 improves BLEU over a unidirectional Transformer — is not circular. Eq. (2) is exactly the sum of the two directional log-likelihoods for a candidate sequence, so using it for reranking is the inference analogue of the training objective rather than a fitted quantity renamed as a prediction. The BLEU comparisons are made against external test sets (WMT14 En-De, WMT17 Zh-En, IWSLT14 De-En) and previously published scores; no parameter is fit to the test set and then reported as a prediction. The paper cites several works with overlapping authorship (e.g., Wu et al. 2018 and Ren et al. 2019 for the observation that right-side tokens are harder), but this observation is also supported by the paper's own WMT17 validation statistics (39.0% vs 28.1%) and is motivational rather than load-bearing. The ablation in Section 3.2 partially addresses the candidate-set concern by reporting 28.73 when only left-to-right candidates are reranked, and it explicitly acknowledges that the ensemble comparison is parameter-unfair; whether the residual gain is fully attributable to right-to-left candidates is an experimental-control question, not a circularity. No self-definitional, fitted-input, or self-citation-chain reduction is present.
Assumptions & free parameters
free parameters (2)
- Beam size K =
per-task validation choice; exact values not stated in main text
- Length penalty alpha =
not stated
assumptions (4)
- standard math The chain-rule factorization of sequence probability holds for both decoding directions (Eq. 1).
- domain assumption Reversing the target sequence and using a direction-specific start token teaches the shared decoder to generate right-to-left (Section 2.1).
- ad hoc to paper The sum of left-to-right and right-to-left log-probabilities in Eq. (2) is a valid scoring function for ranking candidates.
- domain assumption Beam search over 2K candidates plus bidirectional reranking approximates the argmax in Eq. (2).
Cite this review
Pith. "Pith review of Efficient Bidirectional Neural Machine Translation." pith.science (2026). https://pith.science/paper/S6QJXWCJ
@misc{pith2026190809329,
author = {Pith},
title = {Pith review of: Efficient Bidirectional Neural Machine Translation},
year = {2026},
howpublished = {\url{https://pith.science/paper/S6QJXWCJ}},
note = {Machine review of arXiv:1908.09329}
}
read the original abstract
The encoder-decoder based neural machine translation usually generates a target sequence token by token from left to right. Due to error propagation, the tokens in the right side of the generated sequence are usually of poorer quality than those in the left side. In this paper, we propose an efficient method to generate a sequence in both left-to-right and right-to-left manners using a single encoder and decoder, combining the advantages of both generation directions. Experiments on three translation tasks show that our method achieves significant improvements over conventional unidirectional approach. Compared with ensemble methods that train and combine two models with different generation directions, our method saves 50% model parameters and about 40% training time, and also improve inference speed.
Figures
Reference graph
Works this paper leans on
-
[5]
arXiv preprint arXiv:1610.02242
Temporal ensem- bling for semi-supervised learning. arXiv preprint arXiv:1610.02242. Lemao Liu, Andrew M. Finch, Masao Utiyama, and Eiichiro Sumita. 2016a. Agreement on target- bidirectional lstms for sequence-to-sequence learn- ing. In AAAI, February 12-17, 2016, Phoenix, Ari- zona, USA., pages 2630–2637. Lemao Liu, Masao Utiyama, Andrew M. Finch, and Ei...
-
[6]
Towards Robust Neural Networks via Random Self-ensemble
Towards robust neural net- works via random self-ensemble. arXiv preprint arXiv:1712.00673. Thang Luong, Hieu Pham, and Christopher D. Man- ning
-
[8]
Sequence level training with recurrent neural networks. CoRR, abs/1511.06732. Yi Ren, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu
-
[9]
In WMT 2017, Copenhagen, Denmark, September 7-8, 2017, pages 389–399
The university of edinburgh’s neural MT sys- tems for WMT17. In WMT 2017, Copenhagen, Denmark, September 7-8, 2017, pages 389–399. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh neural machine translation sys- tems for WMT
work page 2017
-
[10]
Outrageously large neural net- works: The sparsely-gated mixture-of-experts layer. CoRR, abs/1701.06538. Yanyao Shen, Xu Tan, Di He, Tao Qin, and Tie-Yan Liu
-
[11]
Dense Information Flow for Neural Machine Translation
Dense information flow for neural ma- chine translation. arXiv preprint arXiv:1806.00722. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu
-
[14]
In WMT 2017, Copenhagen, Denmark, September 7-8, 2017, pages 410–415
Sogou neural machine translation systems for WMT17. In WMT 2017, Copenhagen, Denmark, September 7-8, 2017, pages 410–415. Lijun Wu, Xu Tan, Di He, Fei Tian, Tao Qin, Jianhuang Lai, and Tie-Yan Liu
work page 2017
-
[15]
Beyond error propaga- tion in neural machine translation: Characteristics of language also matter. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 3602–3611. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V . Le, and et al
work page 2018
Show all 19 references
-
[16]
CoRR, abs/1609.08144
Google’s neural machine trans- lation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144. A Datasets Description for Neural Machine Translation IWSLT14 German-English We preprocess the IWSLT14 German-English (briefly, De →En) dataset (Cettolo et al.,
-
[17]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin
In WMT, pages 400–404. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017a. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, pages 6000–6010. Ashish Vaswani, Noam S...
-
[18]
After preprocess- ing, the dataset contains 160K bilingual training sentences and 7K validation sentences
using sub-word types based on byte-pair-encoding (BPE) (Sen- nrich et al., 2016b) 2, resulting in a shared vo- cabulary of about 31K tokens. After preprocess- ing, the dataset contains 160K bilingual training sentences and 7K validation sentences. Follow- ing the common practi...
2015
-
[19]
with β1 = 0 .9, β2 = 0.98,ε = 10−9 and follow the same learning rate schedule in Vaswani et al. (2017b). We train our models for WMT14 En →De and WMT17 Zh→En with 8 NVIDIA Tesla M40 GPUs on one machine, and for IWSLT14 De→En with one M40 GPU as it is of both small model size a...
2017
-
[2002]
with multi-bleu.pl6 on IWSLT14 De→En and WMT14 En→De tasks. We use case insensitive BLEU for IWSLT14 De→En and case 3https://nlp.stanford.edu/projects/nmt/ 4http://www.statmt.org/wmt17/translation-task.html 5https://github.com/rsennrich/subword-nmt 6https://github.com/moses-sm...
2017
-
[2014]
arXiv preprint arXiv:1412.6980
Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. Samuli Laine and Timo Aila
-
[2015]
In EMNLP 2015, Lis- bon, Portugal, September 17-21, 2015, pages 1412–
Effective approaches to attention-based neural machine translation. In EMNLP 2015, Lis- bon, Portugal, September 17-21, 2015, pages 1412–
2015
-
[2016]
CoRR, abs/1610.10099
Neural machine translation in linear time. CoRR, abs/1610.10099. Diederik Kingma and Jimmy Ba
-
[2017]
arXiv preprint arXiv:1706.05565
Neural phrase-based machine trans- lation. arXiv preprint arXiv:1706.05565. Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, A¨aron van den Oord, Alex Graves, and Koray Kavukcuoglu
-
[2018]
arXiv preprint arXiv:1803.05567
Achieving hu- man parity on automatic chinese to english news translation. arXiv preprint arXiv:1803.05567. Tianyu He, Xu Tan, Yingce Xia, Di He, Tao Qin, Zhibo Chen, and Tie-Yan Liu
-
[2019]
arXiv preprint arXiv:1905.02450
Mass: Masked sequence to sequence pre-training for language generation. arXiv preprint arXiv:1905.02450. Zhixing Tan, Boli Wang, Jinming Hu, Yidong Chen, and Xiaodong Shi
1905 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.