Pith. sign in

REVIEW 3 major objections 5 minor 14 references

Multi-Layer Softmaxing during Training Neural Machine Translation for Flexible Decoding with Fewer Layers

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Multi-layer softmaxing trains one translation model that decodes at any depth.

desk verdict Useful flexible-depth training trick, but the abstract's 'insignificant losses' claim does not hold for shallow decoder settings and no significance tests back it up. read the letter →

arxiv 1908.10118 v2 pith:4T3A6K4O submitted 2019-08-27 cs.CL cs.LG

Raj Dabre , Atsushi Fujita This is my paper · ORCID
classification cs.CLcs.LG
keywords multi-layersoftmaxingneuralmachinetranslationflexibledecodinglayer-wisetrainingmodelcompressionTransformerencoder-decoderspeed
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a training procedure that turns one encoder-decoder network into many models of different depths. Instead of computing a single cross-entropy loss from the top layers, the method computes a softmax loss for the output of every decoder layer derived from every encoder layer, aggregates the $N \times M$ losses, and back-propagates the sum. A single Transformer trained this way on WMT18 English-to-German can then be decoded with any number of encoder and decoder layers from one to six, with BLEU scores within about one point of separately trained vanilla models in most configurations. The authors report that this compresses 36 individual checkpoints into one 183M-parameter model, and that choosing fewer decoder layers roughly halves decoding time for a small BLEU cost.

What carries the argument

The key machinery is the multi-layer softmaxing objective of Algorithm 1. For each $i \in \{1,\dots,N\}$ and $j \in \{1,\dots,M\}$, the source is passed through encoder layers up to $i$, the decoder runs up to layer $j$ using that encoder representation, and the softmax over the target vocabulary is taken directly from the $j$-th decoder layer's output; a cross-entropy loss against the same target sentence is computed for every such $(i,j)$ pair. The $N \times M$ losses are averaged and back-propagated through the whole network. At decode time, one simply runs the first $n$ encoder layers and the first $m$ decoder layers and takes the softmax from layer $m$, with no extra adapters, projections, or fine-tuning. Conceptually this ties the parameters of all $N \times M$ submodels into one shared network; because gradients from every loss path that includes a layer flow into that layer, the shallow layers receive dense training signal and can stand alone.

What would settle it

On a held-out set other than newstest2018, decode the same $6 \times 6$ checkpoint at all 36 depth pairs and compare each score with a vanilla model of the same depth. If more than a few configurations exceed the paper's 1.0-BLEU gap, or if the rows with one or two decoder layers fall more than 3 BLEU below the full model, the claim of 'insignificant losses' for arbitrary fewer layers would be refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that a model trained by multi-layer softmaxing—computing the cross-entropy loss of the output of each of the $M$ decoder layers when it attends to each of the $N$ encoder layers, then aggregating all $N \times M$ losses—can be decoded with any $n \le N$ encoder layers and $m \le M$ decoder layers and still translate almost as well as a model trained specifically for that depth. In the WMT18 En→De experiments, one $6 \times 6$ Transformer achieved 24.24 to 34.95 BLEU across the 36 depth configurations, landing within less than 1.0 BLEU of the corresponding vanilla model in most configurations with more than one decoder layer. The same checkpoint stores 183M parameters instead of the 4,600M total across 36 individually trained vanilla models, and training it costs about 9.5 times the compute of one full model rather than the 25.5 times needed to train all 36 submodels. The authors summarize the result as 'essentially compressed 36 models into one' and show that using fewer decoder layers can halve decoding time while losing about 1.7 BLEU points.

Load-bearing premise

The load-bearing assumption is that every intermediate decoder layer, when attached to any intermediate encoder layer, can feed the shared output softmax and produce a meaningful word distribution without any extra projection or adaptation. If mid-depth representations are not already aligned with the output embedding space, the shallow layers would not train into usable decoders and the flexibility would be lost.

Editorial extensions

If this is right

  • A $6 \times 6$ checkpoint can be served at any depth: with 4 encoder and 3 decoder layers it gives BLEU 34.48 and decodes 1.3 times faster than the best vanilla model (151.79s vs 195.85s) at a cost of 0.87 BLEU.
  • Using 6 encoder and 2 decoder layers keeps BLEU at 33.61 and nearly halves decoding time (117.05s vs 195.85s), which is useful for low-latency translation without retraining.
  • Storage shrinks from 4,600M parameters across 36 separate checkpoints to 183M in one model, a 25-fold saving.
  • Training the flexible model costs about 9.5 times one vanilla training run, versus 25.5 times for all 36 submodels, so the flexibility is cheaper to obtain than an ensemble of depth-specific models.
  • The method is not tied to Transformer or to translation, so the same recipe can be applied to any deep encoder-decoder network.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the multi-layer objective gives every decoder layer its own loss, the paper's model is an anytime model: one deployment can trade quality for speed at request time. The authors do not frame it this way, but this makes it a natural fit for serving systems that route requests to different latency budgets.
  • The largest quality gap to vanilla models sits in the single-decoder-layer row (24.24 vs 27.07 BLEU for one encoder layer), suggesting that weighted loss aggregation or a small per-layer projection for shallow layers could close much of the gap; the paper only averages the losses.
  • The oracle-layer analysis implies that different sentences are best handled by different depths, so an input-dependent layer router—which the paper leaves as future work—could recover quality while decoding even faster than any fixed-depth choice.
  • The same training objective should transfer to other autoregressive generation tasks such as speech recognition or summarization, where decoder depth dominates latency; this is untested so should be read as a hypothesis, not a result.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes multi-layer softmaxing for training an encoder-decoder Transformer so that a single model can be decoded with any number of encoder and decoder layers up to the trained maximum. Instead of computing the cross-entropy loss only from the top decoder layer, the method computes N×M losses, one for each combination of an encoder layer output and a decoder layer output, averages them, and back-propagates the average. Experiments on WMT18 English-to-German compare one 6×6 N×M model against 36 individually trained vanilla models (1 to 6 encoder and decoder layers). For most configurations with two or more decoder layers, the N×M model is within about 1 BLEU of the corresponding vanilla model, and decoding-time measurements show large speedups when fewer decoder layers are used.

Significance. The idea is simple, easy to implement, and potentially useful for latency-quality trade-offs in deployed NMT. The paper includes a broad baseline covering all 36 vanilla models, a decoding-time table, and an oracle analysis. The main empirical finding—that a single multi-loss model can approximate separate models at mid-to-deep layer counts—is a useful practical result. However, the headline claim of arbitrary layer choice with insignificant losses is only supported for middle and deep configurations; shallow-decoder configurations lose between 1.3 and 2.8 BLEU. The absence of any significance testing also weakens the statement that differences are not statistically significant. These issues are fixable within the scope of the paper.

major comments (3)
  1. [§4.3 / Table 1] The abstract and conclusion claim that a single N×M model can be used for decoding with an arbitrary fewer number of layers with insignificant quality loss, but the shallow-column results do not support this: at decoder m=1 the BLEU gaps are -2.83 (n=1), -2.01 (n=2), -1.33 (n=3), -1.33 (n=4), -1.81 (n=5), and -2.13 (n=6); even at m=2, n=1 the gap is -1.40. The sentence in §4.3 admitting that vanilla models are 'significantly better' with a single decoder layer directly contradicts the unqualified 'arbitrary' claim. Please either restrict the claim to configurations with at least two decoder layers or provide explicit evidence that these shallow losses are acceptable for the intended low-latency use cases.
  2. [§4.3] The paper states that for more than one decoder layer 'there is no statistically significant difference' between vanilla and N×M models, but no statistical test or confidence interval is reported anywhere. All BLEU numbers come from a single training run per configuration and a single 2,998-sentence test set. To support the 'insignificant' language, please add paired bootstrap or approximate randomization tests over test sentences (or multiple training seeds) for at least the configurations used in the cost-benefit arguments (e.g., 4×3 and 6×2).
  3. [§5.3 / Figure 2] The claim that the N×M model can 'essentially do what 36 individually trained models can do' is not supported by the oracle sentence counts alone. Figure 2 shows that for 28.8% of test sentences a 1×1 N×M decoding is the best among the 36 N×M configurations, but the corpus-level BLEU of that configuration is 2.8 points below the vanilla 1×1 model. The oracle analysis does not compare the oracle-selected N×M outputs with the corresponding vanilla models. Please report the BLEU of the oracle combination and, if possible, the overlap with the vanilla oracle choices.
minor comments (5)
  1. [Algorithm 1] Line 5 uses dec_{j-1} but dec_0 is never defined; please make explicit that dec_0 is the decoder input embedding (e.g., dec_0 = Y).
  2. [§4.2] The text says the N×M model was trained on 2 GPUs with the halved batch size to ensure that both models see the same amount of training data; please specify the actual effective batch size and the number of parameter updates per epoch so that readers can verify the comparison.
  3. [§5.2] The parameter counts are reported only as approximate ratios (25.16 times more parameters); please state the exact total parameter counts for the N×M model and the 36 vanilla models, or give the formula used to compute them.
  4. [Figure 1] The caption states that 'the lowest layer in the stack receives the most updates' due to gradient flow, but this is not immediately obvious from the figure; a short derivation or a reference to the relevant gradient path would improve clarity.
  5. [General] Please correct the typo 'sof tmax' in Algorithm 1, and rephrase 'equals to tying' in Section 1 for readability.

Circularity Check

0 steps flagged · score 0.0 of 10

Empirical training-method paper; no circular derivation, though the shallow-layer quality claim is statistically under-supported.

full rationale

The paper's central claim is an empirical property of a training procedure, not a derived identity. Algorithm 1 defines the training objective as the aggregate (average) of N×M cross-entropy losses computed from each decoder layer using each encoder layer; this is the method being evaluated, not a fit to the test set. The supporting results in Table 1 compare this single model's BLEU on newstest2018 against 36 separately trained vanilla models, an external benchmark with no parameter fitted to the test data. No equation in the paper reduces a predicted quantity to an input by construction: the ability to decode with fewer layers follows from the training objective, but whether it yields usable BLEU is an empirical finding. The only self-citation (Nakazawa et al., 2018, cited in footnote 10 for the general point that BLEU is imperfectly correlated with human judgment) is peripheral and not load-bearing. The abstract's 'insignificant losses' wording is statistically under-supported, especially for 1x1 configurations (24.24 vs 27.07), but that is a correctness and evidence concern, not circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities. The main implicit assumptions are about the representational alignment of intermediate layers, the adequacy of simple loss averaging, and the fairness of the training comparison. These are not derived or ablated.

assumptions (4)
  • domain assumption Intermediate Transformer layer outputs are directly usable for softmax prediction without an additional projection layer.
    Algorithm 1 (lines 4 to 8) computes a softmax loss from each decoder layer output attending to each encoder layer output; the architecture has no adapter or projection for low layers.
  • ad hoc to paper Simple averaging of the NxM losses is a sufficient aggregation strategy.
    Footnote 3 states they averaged multiple losses in the experiments, but no ablation of other aggregation options (e.g., weighted or max) is provided.
  • domain assumption Training all models for the same 300k iterations is a fair comparison setup.
    Section 4.2 states this is fair to ensure each model sees roughly the same number of examples, but no validation of this choice is given.
  • ad hoc to paper Shallower layers are better trained because they receive more gradient updates.
    Stated in the Introduction and Figure 1 caption as motivation for why decoding with fewer layers might work; no direct measurement of layer-wise training quality supports this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Layer Softmaxing during Training Neural Machine Translation for Flexible Decoding with Fewer Layers." pith.science (2026). https://pith.science/paper/4T3A6K4O

@misc{pith2026190810118,
  author       = {Pith},
  title        = {Pith review of: Multi-Layer Softmaxing during Training Neural Machine Translation for Flexible Decoding with Fewer Layers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4T3A6K4O}},
  note         = {Machine review of arXiv:1908.10118}
}
read the original abstract

This paper proposes a novel procedure for training an encoder-decoder based deep neural network which compresses NxM models into a single model enabling us to dynamically choose the number of encoder and decoder layers for decoding. Usually, the output of the last layer of the N-layer encoder is fed to the M-layer decoder, and the output of the last decoder layer is used to compute softmax loss. Instead, our method computes a single loss consisting of NxM losses: the softmax loss for the output of each of the M decoder layers derived using the output of each of the N encoder layers. A single model trained by our method can be used for decoding with an arbitrary fewer number of encoder and decoder layers. In practical scenarios, this (a) enables faster decoding with insignificant losses in translation quality and (b) alleviates the need to train NxM models, thereby saving space. We take a case study of neural machine translation and show the advantage and give a cost-benefit analysis of our approach.

Figures

Figures reproduced from arXiv: 1908.10118 by the authors.

Figure 1
Figure 1. The general concept of multi-layer softmaxing for training multi-layer neural models with an example of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Distribution of oracle translations among 36 combinations of encoder and decoder layers for WMT [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 11 canonical work pages

  1. [1]

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. https://arxiv.org/pdf/1409.0473.pdf Neural machine translation by jointly learning to align and translate . In Proceedings of the 3rd International Conference on Learning Representations, San Diego, USA

    arXiv 2015

  2. [2]

    Yonatan Belinkov, Llu \'i s M \`a rquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2017. http://aclweb.org/anthology/I17-1001 Evaluating layers of representation in neural machine translation on part-of-speech and semantic tagging tasks . In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1...

    work page 2017

  3. [3]

    e nboer, C aglar G \

    Kyunghyun Cho, Bart van Merri \" e nboer, C aglar G \" u l c ehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. http://aclweb.org/anthology/D14-1179 Learning phrase representations using RNN encoder--decoder for statistical machine translation . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Pr...

    work page 2014

  4. [4]

    Zi-Yi Dou, Zhaopeng Tu, Xing Wang, Shuming Shi, and Tong Zhang. 2018. http://aclweb.org/anthology/D18-1457 Exploiting deep representations for neural machine translation . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4253--4262, Brussels, Belgium

    work page 2018

  5. [5]

    Markus Freitag, Yaser Al - Onaizan, and Baskaran Sankaran. 2017. http://arxiv.org/abs/1702.01802 Ensemble distillation for neural machine translation . CoRR, abs/1702.01802

    arXiv 2017

  6. [6]

    Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. http://arxiv.org/abs/1503.02531 Distilling the knowledge in a neural network . CoRR, abs/1503.02531

    arXiv 2015

  7. [7]

    Toshiaki Nakazawa, Shohei Higashiyama, Chenchen Ding, Raj Dabre, Anoop Kunchukuttan, Win Pa Pa, Isao Goto, Hideya Mino, Katsuhito Sudoh, and Sadao Kurohashi. 2018. http://aclweb.org/anthology/Y18-3001 Overview of the 5th workshop on Asian translation . In Proceedings of the 5th Workshop on Asian Translation, Hong Kong, China

    work page 2018

  8. [8]

    Yusuke Oda, Philip Arthur, Graham Neubig, Koichiro Yoshino, and Satoshi Nakamura. 2017. https://doi.org/10.18653/v1/P17-1079 Neural machine translation via binary code prediction . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 850--860, Vancouver, Canada

    work page doi:10.18653/v1/p17-1079 2017

Show all 14 references
  1. [9]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. http://aclweb.org/anthology/P02-1040 BLEU : A method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 311--318, Phi...

    2002

  2. [10]

    Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf Sequence to sequence learning with neural networks . In Proceedings of the 27th Neural Information Processing Systems Conference, pages ...

    2014

  3. [11]

    Liling Tan, Jon Dehdari, and Josef van Genabith . 2015. http://aclweb.org/anthology/W15-5009 An awkward disparity between BLEU / RIBES scores and human judgements in machine translation . In Proceedings of the 2nd Workshop on Asian Translation, pages 74--81, Kyoto, Japan

    2015

  4. [12]

    Gomez, ukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf Attention is all you need . In Proceedings of the 30th Neural Information Processin...

    2017

  5. [13]

    Qiang Wang, Fuxue Li, Tong Xiao, Yanyang Li, Yinqiao Li, and Jingbo Zhu. 2018. http://aclweb.org/anthology/C18-1255 Multi-layer representation fusion for neural machine translation . In Proceedings of the 27th International Conference on Computational Linguistics, pages 3015--...

    2018

  6. [14]

    Deyi Xiong, Biao Zhang, and Jinsong Su. 2018. http://aclweb.org/anthology/P18-1166 Accelerating neural transformer via an average attention network . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Long Papers, pages 1789--1798, Melb...

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.