{"id":"5e7806ec-e0f1-42e4-b769-afd946515625","arxiv_id":"1908.09329","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A single encoder-decoder trained with both decoding directions beats a unidirectional Transformer by 0.8 to 1.3 BLEU and saves about half the parameters of a two-model ensemble.","lead":"This paper trains a single translation model to decode in both left-to-right and right-to-left order, then picks the best candidate with both scores. It reports BLEU gains of about one point on three benchmarks while roughly halving the parameter cost of the usual two-model ensemble.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed BLEU gains may be partly a candidate-pool-size artifact: Algorithm 1 reranks 2K candidates against a K-candidate baseline, and the §3.2 ablation does not control for this.","rationale":"The reader's verdict is already CONDITIONAL, mainly because the empirical evidence lacks error bars, candidate-pool-size control, and released code. My concern is more specific: the reported ablation controls for generation direction but not for candidate pool size. If the 2K-candidate control were run and the unidirectional baseline were to jump from 28.4 to roughly 29.0, then the paper's central novelty—that decoding in both directions with one model is what helps—would be substantially weakened. The method itself is plausible and the paper is honest about limitations, so this is not a rejection; it is a request for a specific control that would turn a conditional acceptance into a confident one. I therefore keep the reader's CONDITIONAL verdict unchanged, and note that the reader already flagged the pool-size issue in the rationale, though not as the primary weakest assumption.","tokens_in":9461,"tokens_out":5292,"duration_ms":51816,"concrete_test":"Re-run the WMT14 En→De unidirectional Transformer (same architecture, data, and number of training iterations as the §3.1 baseline) with beam size 2K, selecting the highest L2R-probability candidate with the same length penalty used in §3.1, and report BLEU. Also run the bidirectional model with 2K L2R-only candidates (no R2L beam search) and the same Eq. (2) reranking. If either condition reaches about 29.0 BLEU, then the R2L-specific contribution in Table 2 is largely a pool-size or scoring artifact rather than evidence for bidirectional decoding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that bidirectional decoding in a single model improves BLEU over a unidirectional baseline. Algorithm 1 (§2.2) forms 2K candidates (K from each direction) and reranks them by the Eq. (2) score. The unidirectional baseline is a single beam search, presumably with K candidates. Since BLEU selection over a larger candidate pool can improve even when the added candidates are drawn from the L2R model itself, the headline improvements are not yet securely attributable to the bidirectional mechanism. The paper's ablation in §3.2 compares 29.30 (2K candidates, both directions) with 28.73 (K L2R candidates, reranked by both directions), attributing the residual 0.57 to R2L candidates. But it never reports the unidirectional baseline with beam size 2K, nor a L2R-only variant generating 2K candidates. If a unidirectional Transformer with beam size 2K reaches roughly 29.0 BLEU, most of the gain over the 28.40 baseline comes from pool size, not from right-to-left decoding. A second, smaller confound: the bidirectional model trains for 1.2x more iterations than the baseline (§3.2), so part of the En→De gain could reflect additional training rather than the method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a single encoder-decoder NMT model that is trained with both left-to-right and right-to-left objectives using direction-specific start tokens, so that one parameter set can decode in both directions. At inference, Algorithm 1 generates K candidate sequences from each direction, scores all 2K candidates with the sum of L2R and R2L log-probabilities (Eq. 2), and selects the best. Experiments on IWSLT14 De→En, WMT14 En→De, and WMT17 Zh→En report improvements of 0.8–1.32 BLEU over a unidirectional Transformer baseline, plus efficiency gains compared with training two separate directional models. The paper also claims a new state-of-the-art score of 29.30 BLEU on WMT14 En→De.","tokens_in":9720,"tokens_out":4992,"duration_ms":45504,"significance":"If the reported gains are real and attributable to the bidirectional mechanism, the method is a simple and useful contribution: it achieves bidirectional decoding in a single model with a modest training-time overhead relative to a unidirectional baseline and a large parameter savings relative to two-model ensembles. The method is clearly specified, the training and inference procedures are reproducible, and the three-task evaluation provides useful cross-lingual evidence. However, the empirical claims are weakened by two confounds that are not addressed in the current manuscript: the baseline uses a smaller candidate pool than the proposed method, and the proposed model is trained for more iterations. These issues are fixable, but they must be resolved before the central claim is secure.","major_comments":[{"comment":"The reported improvements are not isolated from the effect of generating twice as many candidates. The bidirectional method generates 2K sequences (K per direction) and reranks them, while the baseline is a single beam search with K candidates. The ablation that restricts candidates to L2R only reduces the pool to K, so the drop from 29.30 to 28.73 in §3.2 could be due to pool size rather than to the value of the R2L candidates. The paper should include a unidirectional baseline with beam size 2K and a L2R-only variant with 2K candidates reranked using Eq. (2). Without these controls, the central claim that bidirectional decoding is responsible for the gains is not established.","section":"§3.2 and Algorithm 1"},{"comment":"The authors state that 'our method needs 1.2x iterations to converge.' Because the bidirectional model is trained for 20% more updates than the baseline, the improvement on WMT14 En→De (and possibly the other tasks) may reflect additional training rather than the bidirectional objective. The evaluation should match the number of training updates, for example by training the baseline for 1.2x iterations or by evaluating the bidirectional model at the same iteration count as the baseline, to separate the effect of the method from the effect of longer training.","section":"§3.2, Training and Inference"},{"comment":"The paper reports BLEU differences of 0.8–1.32 points but provides no confidence intervals, bootstrap resampling, or significance tests. Given the known variance of BLEU on these test sets, especially IWSLT14 with its small test set, these differences could be within noise. The authors should provide significance estimates (e.g., paired bootstrap resampling) or at least error bars for the main comparisons, so that the reader can assess whether the claimed improvements are statistically meaningful.","section":"Tables 2–4"},{"comment":"The sentence 'If we only use sentences generated in left-to-right as candidates and keep other factors unchanged' is inaccurate because the number of candidates is reduced from 2K to K. This changes a factor that the authors explicitly say is unchanged. The ablation therefore does not support the conclusion that the improvement is due to bidirectionally generated candidates rather than to self-ensemble or pool-size effects. A proper ablation should compare L2R-only with K candidates, L2R-only with 2K candidates, and the full bidirectional approach with 2K candidates, all reranked with the same scoring function.","section":"§3.2, 'Analyzing of the Contribution of Right-To-Left Decoding'"}],"minor_comments":[{"comment":"The opening sentence says 'to very the effectiveness' but should be 'to verify the effectiveness.'","section":"Section 3, first sentence"},{"comment":"The footnote states that the reimplemented baseline gives 28.36 BLEU but the table lists 28.40 from Vaswani et al. (2017b). It would be clearer and more consistent to use the reimplemented score for the comparison, or to explain why the original score is preferred.","section":"Table 2 footnote"},{"comment":"The claim that the method 'improve[s] inference speed' is vague; the comparison appears to be against a two-model ensemble, but the abstract does not specify this. Clarify the reference point for the inference-time claim.","section":"Abstract and Section 3.2"},{"comment":"The notation P(←y|x;θ) is not explicitly defined; the paper should clarify that it denotes the probability of the reversed target sequence under the right-to-left decoding direction.","section":"Section 2.1"},{"comment":"The text says 'We vary the beam size on IWSLT14 En→De validation set as well as the test set,' but the task in the paper is IWSLT14 De→En. Correct the direction.","section":"Appendix C, Figure 1(b)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript presents a simple and potentially attractive method, and the authors are well positioned to add the necessary controls. The main concern for the editor is that the headline claim—that bidirectional decoding in a single model improves BLEU—rests on comparisons that do not control for candidate pool size or training duration. I would ask the authors to run the additional experiments (2K-beam unidirectional baseline, L2R-only with 2K candidates, and matched-iteration training) and to report significance estimates before this is acceptable for publication. The paper's 'new record' claim on WMT14 En→De should also be contextualized against contemporaneous published systems, since it may be outdated by the time of publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know: this paper does something I haven't seen before—one shared decoder that can generate both left-to-right and right-to-left using learned start tokens. It's a clean trick, and it works reasonably well.\n\nThe strongest evidence is Table 4: decoding their shared model in L2R only roughly matches the unidirectional baseline, so parameter sharing doesn't hurt. The efficiency comparison vs two-model ensemble is honest: 50% fewer parameters, about 0.6x training compute, and only ~10% extra inference cost for the reranking.\n\nThe main weakness is the attribution of the BLEU gain to R2L candidates. Algorithm 1 reranks 2K candidates against a K-candidate baseline, and the §3.2 ablation does not include a L2R-only 2K candidate pool. Without that control, the 0.57 point difference between 29.30 and 28.73 is ambiguous. The 1.2x extra training iterations for the bidirectional model is a second confound, though a smaller one. No confidence intervals either, which is typical for this type of paper but worth noting.\n\nThat said, these are fixable with a few runs, not a fundamental flaw. The method itself is easy to reproduce, and the central claim—that a single model can serve both directions—appears sound.\n\nIf this lands on your desk, send it to review with a request for the missing controls. The idea is worth having in the literature, and a careful referee can prompt the right experiments.","headline":"A genuinely simple single-model bidirectional decoding idea; the BLEU gains are likely real in part, but the paper's ablations miss the candidate-pool-size control, so the headline numbers should be read with caution.","tokens_in":10216,"tokens_out":3937,"would_cite":true,"duration_ms":37439,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a single encoder–decoder model to generate translations in both left-to-right and right-to-left order, then reranking the two candidate sets, beats unidirectional translation baselines by 0.8–1.32 BLEU while halving the parameter…","keywords":["neural machine translation","bidirectional decoding","left-to-right decoding","right-to-left decoding","single encoder-decoder","beam search reranking","Transformer","BLEU"],"falsifier":"Run Algorithm 1 on WMT14 En→De with every candidate scored by the left-to-right probability alone instead of the sum in Eq. (2). The paper's claim predicts BLEU falls from 29.30 toward the 28.73 level of the no-right-to-left-candidates ablation; if BLEU stays at 29.30, the right-to-left scores are not the source of the gain.","tokens_in":9247,"feed_emoji":"🔄","tokens_out":12401,"duration_ms":104970,"temperature":0.7,"pith_summary":"The paper tries to show that a translation model does not have to choose between left-to-right and right-to-left decoding: one encoder–decoder can be trained to do both at once, and the two decoding directions can then vote on the final output. The motivation is that left-to-right decoding makes more errors on the right end of a sentence, so a right-to-left pass provides complementary candidates where the left-to-right model is weak. On three translation tasks the method beats a same-architecture unidirectional Transformer by 0.8 to 1.32 BLEU, reaching 29.30 BLEU on WMT14 English→German, and it does so with roughly half the parameters and 40% less training time than a two-model bidirectional ensemble. If the claim holds, bidirectional reranking no longer requires maintaining two full models.","feed_headline":"Decoding both ways with one model beats unidirectional baselines","feed_subtitle":"A right-to-left pass repairs the error-prone tail of left-to-right translations, adding up to 1.32 BLEU.","key_machinery":"The load-bearing object is the shared-parameter bidirectional objective and its approximate inference loop. Equation (1) trains one decoder to maximize, for every target token $y_t$, both the probability given the left context $y_{<t}$ and the probability given the right context $y_{>t}$, with two learned start-token embeddings telling the decoder which direction it is generating in; reversed target sequences are mixed into the same mini-batch. At inference, Algorithm 1 generates $K$ beam candidates in each direction, scores every candidate with both directional probabilities, and picks the one maximizing the sum in Eq. (2) as an approximation to the intractable argmax over all sequences. This machinery is what lets one parameter set supply two complementary candidate pools and a joint ranking over them.","core_discovery":"The central claim is that maximizing the sum of left-to-right and right-to-left log-likelihoods under one parameter set—$\\sum_t (\\log P(y_t|y_{<t},x)+\\log P(y_t|y_{>t},x))$—yields a single model that decodes well in both directions, and that reranking beam candidates by that same sum produces better translations than either direction alone. The paper argues the right-to-left pass is not just a regularizer: the ablation removing right-to-left candidates drops WMT14 En→De from 29.30 to 28.73 BLEU, close to the 28.40 unidirectional baseline, while 33.8% of the final selections come from the right-to-left beam. With only left-to-right decoding the trained model nearly matches the baseline (28.25 vs 28.40), so the gain is attributed to candidate combination rather than to a stronger per-token model. The method therefore claims bidirectional decoding at the cost of a single model, with only about 10% extra inference time for the reverse scoring pass.","pith_inferences":["Editorial inference: because the paper's motivation is that right-side tokens degrade with length, a length-stratified BLEU breakdown would likely show the gains concentrate on longer sentences; the paper does not report that split.","Editorial inference: the joint objective is a form of consistency training between two orderings of the same target, so the same trick could be applied to any autoregressive generator with two orderings, not only translation.","Editorial inference: the near-parity of left-to-right-only decoding with the baseline suggests the objective may also act as a regularizer; comparing against a self-ensemble of the same model would separate regularization from directional complementarity."],"forward_implications":["A single trained checkpoint can be decoded in either direction with no significant quality loss, so applications that need both directions no longer need two separate models.","Bidirectional reranking becomes affordable for deployment: about half the parameters and 40% less training time than a two-model ensemble, with only roughly 10% extra inference cost.","Any autoregressive sequence-generation task that suffers from error propagation could adopt the same training and reranking recipe; the paper names text summarization and image captioning as the next applications.","When the decoder is made deeper (up to 6 layers), the gap over the unidirectional baseline grows, suggesting the benefit scales with decoder capacity."],"supporting_citations":[{"why":"Supplies the Transformer architecture used for both the unidirectional baseline and the proposed model, and the 28.40 BLEU score on WMT14 En→De that the method claims to beat with 29.30.","marker":"Vaswani et al. (2017b)"},{"why":"Cited as prior observation that right-side tokens are less accurate and as an earlier target-bidirectional sequence-to-sequence approach.","marker":"Liu et al. (2016a)"},{"why":"Describes the two-model left-to-right/right-to-left reranking approach whose doubled parameter and training cost the single-model method is designed to avoid.","marker":"Sennrich et al. (2016a)"},{"why":"A WMT17 system used as a comparison baseline and an example of the two-model bidirectional reranking line of work.","marker":"Wang et al. (2017)"},{"why":"Cited with the error-propagation observation that motivates the claim that the right side of left-to-right output is weaker.","marker":"Wu et al. (2018)"},{"why":"Defines BLEU, the evaluation metric on which all reported comparisons and the 29.30 score rest.","marker":"Papineni et al. (2002)"}],"fun_headline_variants":["Single model bi-directional decoding improves BLEU","Right-to-left reranking fixes weak translation tails","One decoder, two directions, half the parameters","Efficient bidirectional NMT saves 50% parameters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that one parameter set can model the left-to-right and right-to-left target distributions accurately enough that their summed score ranks better translations above either direction's best candidate.","fun_headline_variants_meta":{"raw":{"variants":["Single model bi-directional decoding improves BLEU","Right-to-left reranking fixes weak translation tails","One decoder, two directions, half the parameters","Efficient bidirectional NMT saves 50% parameters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000558,"raw_usage":{"total_tokens":2620,"prompt_tokens":880,"completion_tokens":1740,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":1679}},"tokens_in":496,"tokens_out":1740,"duration_ms":12659,"temperature":1.0,"reasoning_tokens":1679,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:14:37.084103+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Algorithm 1 on WMT14 En→De with every candidate scored by the left-to-right probability alone instead of the sum in Eq. (2). The paper's claim predicts BLEU falls from 29.30 toward the 28.73 level of the no-right-to-left-candidates ablation; if BLEU stays at 29.30, the right-to-left scores are not the source of the gain.","supporting_citations":[],"review_version":1}