Pith. sign in

REVIEW 3 major objections 6 minor 31 references

Towards Making the Most of BERT in Neural Machine Translation

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CTNMT keeps pre-trained BERT knowledge from being forgotten during NMT fine-tuning, gaining up to 2.9 BLEU on WMT14 English-German.

desk verdict A solid empirical recipe for BERT in NMT, marred by a sign error in the core distillation equation that must be fixed before the results are reproducible. read the letter →

arxiv 1908.05672 v5 pith:774XCGOO submitted 2019-08-15 cs.CL cs.LG

Jiacheng Yang , Mingxuan Wang , Hao Zhou , Chengqi Zhao , Yong Yu , Weinan Zhang , Lei Li This is my paper · ORCID
classification cs.CLcs.LG
keywords neuralmachinetranslationBERTpre-trainedlanguagemodelscatastrophicforgettingknowledgedistillationdynamicswitchrate-scheduledlearningTransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the usual way of fine-tuning BERT into a neural machine translation system fails in resource-rich settings because the NMT model forgets BERT's general linguistic knowledge during long training. It proposes CTNMT, a concerted training framework with three coordinated pieces: asymptotic distillation, which keeps the NMT encoder's hidden states close to BERT's; a dynamic switching gate, which adaptively fuses BERT features with the NMT encoder's own representations; and rate-scheduled learning, which updates BERT parameters on a different, slanted triangular schedule from the rest of the model. Together these prevent catastrophic forgetting and let a base Transformer beat much larger baselines. On WMT14 English-German the combined system reaches 30.1 BLEU against 27.2 for the Transformer base, tops the previous pre-training-aided NMT result by 1.4 BLEU, and on WMT14 English-French a base model surpasses Transformer big by 1.3 BLEU, with a 1.6 BLEU gain on English-Chinese.

What carries the argument

The load-bearing machinery is the joint training objective $L = \alpha L_{\mathrm{nmt}} + (1-\alpha) L_{\mathrm{kd}}$ with $L_{\mathrm{kd}} = -\|\hat{h}^{\mathrm{lm}} - h^{l}\|_2^2$, where the pre-trained language model's hidden state $\hat{h}^{\mathrm{lm}}$ is frozen as a teacher and $h^{l}$ is the $l$-th NMT encoder layer (the paper supervises the third layer). The dynamic switch computes an element-wise gate $g = \sigma(W h^{\mathrm{lm}} + U h^{\mathrm{nmt}} + b)$ and produces $h = g \odot h^{\mathrm{lm}} + (1-g) \odot h^{\mathrm{nmt}}$, fusing BERT's representation with the NMT encoder's. Rate-scheduled learning updates BERT parameters with $\eta_{\mathrm{lm}} = \rho \, \eta_{\mathrm{nmt}}$, where $\rho$ follows a slanted triangular schedule that increases until $T' = 10{,}000$ steps, decays until $T = 20{,}000$ steps, and is zero afterward. Distillation, gating, and scheduling attack forgetting from three complementary angles: they keep the student near BERT, adaptively choose which representation each element should trust, and gradually stop updating BERT so the final refinement is NMT-only.

What would settle it

Run CTNMT on WMT14 English-German with the teacher hidden states replaced by random vectors of the same dimension, keeping every other setting identical; if BLEU stays near 30.1, the distillation term is not carrying BERT's knowledge. A complementary check is to compare the learned gate values against the baseline and confirm whether the gains persist when the gate is fixed.

Watch

Extended reading notes

Core claim

The paper's central claim is that the disappearance of pre-training gains in large-scale NMT is not inevitable and can be countered by converting BERT into a fixed teacher and an adaptive feature source rather than a fragile initialization. The authors show that minimizing an L2 loss between the NMT encoder's third layer and BERT's second-to-last layer, gating that teacher's output against the NMT encoder's own embeddings with an element-wise sigmoid switch, and giving BERT parameters a slanted triangular learning rate that ramps up, decays, and then stops yield a model that keeps BERT knowledge while still fitting the parallel corpus. In their experiments each of the three techniques alone gives roughly 2 BLEU over the base Transformer on English-German, and combining all three is worth another half point, reaching 30.1 BLEU.

Load-bearing premise

The whole framework leans on the assumption that a fixed layer of BERT and a fixed layer of the NMT encoder represent the same information in compatible coordinates, so that minimizing the squared distance between them actually transfers linguistic knowledge; if the two spaces are not aligned, the distillation term is just an arbitrary regularizer and the reported gains could come from the gating or the schedule.

Editorial extensions

If this is right

  • Each of the three techniques is independently useful: on WMT14 English-German, rate-scheduling alone reaches 29.7 BLEU, dynamic switch 29.4, and asymptotic distillation 29.2, versus the 27.2 base Transformer.
  • The combined system reaches 30.1 BLEU on English-German, 42.3 on English-French, and 38.9 on English-Chinese, outperforming much larger models on the two WMT14 benchmarks.
  • Because the distillation signal is applied to the encoder, the recipe is compatible with different decoders and different pre-trained encoders; the paper finds BERT works best for the encoder and GPT-2 is preferable for the decoder.
  • The framework modifies only the training loss, the gating, and the learning-rate schedule, not the inference architecture, so it can be carried over to other NLP tasks that fine-tune large pre-trained models.
  • Asymptotic distillation alone beats the MultiCol baseline without adding parameters, which points to a practical recipe for resource-constrained translation systems that cannot afford to run BERT at inference time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension would be to learn or search the layer correspondence between the NMT encoder and BERT instead of fixing the third layer to the second-to-last BERT layer, since the paper does not justify that pairing.
  • If the mechanism generalizes, the same distillation-plus-gating-plus-schedule recipe should transfer to other bidirectional pre-trained encoders and to sequence-to-sequence tasks such as summarization or parsing; that is not tested here.
  • The per-element gate values could be inspected on source tokens: if rare or domain-specific tokens lean on BERT while frequent tokens lean on the NMT encoder, the switch is doing interpretable feature selection rather than averaging; the paper does not report such an analysis.
  • The asymptotic-distillation-only result suggests a cheap deployment recipe: keep BERT only during training and discard it afterward, which matters for mobile or low-latency translation; the paper notes the resource advantage but does not measure inference speed or memory.
  • The weakest link is the untested geometric alignment between BERT's hidden states and the NMT encoder's hidden states; probing that alignment directly would sharpen the claim that distillation, rather than the gate or the schedule, carries BERT's knowledge.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents CTNMT, a concerted training framework for integrating pre-trained language models (BERT, GPT-2) into neural machine translation. Three complementary techniques are proposed: (i) asymptotic distillation, which regularizes the NMT encoder's hidden states toward those of a fixed pre-trained LM (Eq. 4); (ii) a dynamic switch gate that adaptively fuses BERT and NMT encoder representations (Eqs. 6-7); and (iii) rate-scheduled learning, which applies a slanted triangular learning rate schedule to the pre-trained parameters (Eq. 10). The method is evaluated on WMT14 English-German, English-French, and English-Chinese, with reported BLEU gains of up to 2.9, 1.3, and 1.6, respectively, over the authors' base Transformer; the combined system also surpasses prior pre-training-aided NMT (Lample and Conneau, 2019) by 1.4 BLEU on English-German. Ablations in Section 4 attribute gains to each component, and a data-size study (Table 3) shows the benefit of asymptotic distillation as training data grows.

Significance. If the reported results hold, this is a practically valuable contribution: it addresses the catastrophic-forgetting problem when fine-tuning large pre-trained LMs for resource-rich NMT, and the three techniques are simple, general, and do not add inference-time parameters. The paper includes public code and model links, a data-size ablation, and component-wise ablations that go beyond a single aggregate result. The claimed improvements over strong Transformer baselines (up to about 3 BLEU on En-De, and more than 1 BLEU over Transformer big on En-Fr) are substantial for these benchmarks, and the finding that bidirectional BERT features help more on the encoder than the decoder is consistent with existing evidence. However, as detailed below, a sign error in the formal definition of the distillation loss, missing statistical uncertainty quantification, and an unexamined geometric assumption about the distillation target must be resolved before the claims can be fully accepted.

major comments (3)
  1. [Sec. 2.2, Eq. (4) and Eq. (5)] The distillation loss is defined as L_kd = -||h_lm - h_l||_2^2, but the text states that the objective is to 'penalize the mean-squared-error loss' and to 'mimic the original teacher network by minimizing the loss.' Combined with the total loss L = alpha * L_nmt + (1-alpha) * L_kd, minimizing L with the stated sign actually maximizes the squared distance between the NMT encoder layer and the BERT layer, which contradicts the stated intent of retaining pre-trained knowledge. This is a load-bearing inconsistency for the first contribution; all experiments labeled 'Asymptotic Distillation' in Table 1 and Table 3 rely on this objective. Please correct the sign (presumably to L_kd = +||h_lm - h_l||_2^2) and confirm via the released code or a re-run that the reported gains are produced by the intended distillation objective.
  2. [Sec. 4, Table 1 and all experimental tables] The paper claims 'significant' improvements (Abstract) and 'substantial improvement' (Sec. 4.4) but reports a single run per configuration without variance, confidence intervals, or significance tests. Several differences that support the additive contribution of the three components are small: e.g., Dynamic Switch 29.4 vs. Asymptotic Distillation 29.2 vs. Rate-scheduling 29.7, and the combined system 30.1 vs. the best single technique 29.7. These ~0.3-0.5 BLEU gaps are near typical run-to-run noise for these benchmarks. Please provide multiple seeds with means and standard deviations, or an equivalent significance analysis, for the main results and for the component ablations, so the reader can assess whether the differences are reliable.
  3. [Sec. 2.2 and Sec. 4.6] The asymptotic distillation assumes that the hidden-state space of the 3rd NMT encoder layer is geometrically comparable to the second-to-last BERT layer, so that minimizing mean-squared error between them transfers linguistic knowledge to translation. This assumption is not justified empirically or theoretically. The paper's own data (Table 6) show identical BLEU scores (29.2) for the second-to-last, third-to-last, and fourth-to-last BERT layers, which is suspicious and suggests either rounding or an insensitivity that deserves explanation. Since Asymptotic Distillation alone contributes 2.0 BLEU over the base model (29.2 vs. 27.2), it is important to verify that the distillation term transfers knowledge rather than merely acting as an arbitrary regularizer; a representational-alignment analysis (e.g., CCA or a learned projection) would address this concern.
minor comments (6)
  1. [Sec. 2.2] The phrase 'asymptotic distillation as the minic regularization' appears to contain a typo; presumably 'mimic regularization' was intended.
  2. [Sec. 2.2 and Sec. 4.6] The paper states in Sec. 2.2 that 'we use the last layer and find it is better to add the supervision signal to the top encoder layers,' but Sec. 4.6 says 'we add the supervision signal to the 3th encoder layer.' These statements should be reconciled to specify which NMT encoder layer and which BERT layer are used for distillation.
  3. [Table 4] The row 'Encoder w/o BERT init' reports a dash ('-') for the BLEU value. Either provide the numerical result or state explicitly why it is omitted (e.g., training did not converge), as the current presentation is uninformative.
  4. [Table 6] The identical reported BLEU scores (29.2) for Second-to-Last, Third-to-Last, and Fourth-to-Last Hidden layers should be clarified; if these are rounded values, report more decimals or state that the method is insensitive to the choice within this range.
  5. [Throughout] There are several typographical errors: 'ssymptotic distillation' in Related Work, 'drate-scheduled learning' in Sec. 2.4, and '3th encoder layer' in Sec. 4.6. These should be corrected.
  6. [Table 1] The row label 'CT NMT Transformer (base)' is slightly confusing because it denotes the authors' baseline without the three strategies, not a variant of CTNMT. Rename this row (e.g., 'Transformer (base, our implementation)') to avoid ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CTNMT reports measured BLEU scores from held-out test sets; no prediction reduces to a fitted input or self-citation chain.

full rationale

The paper's central claim is an empirical training framework evaluated on WMT benchmarks. The three components—asymptotic distillation (Sec. 2.2, Eqs. 4–5), dynamic switch (Sec. 2.3, Eqs. 6–7), and rate-scheduled learning (Sec. 2.4, Eqs. 8–10)—are defined as explicit objectives and update rules. The reported BLEU gains in Table 1 are measured on held-out test sets, not quantities derived from those formulas by construction. Hyperparameters such as alpha=0.9, T'=10000, T=20000, and the choice of the second-to-last BERT layer are tuned on validation data, not fitted to the test set, so no fitted input is renamed as a prediction. The slanted triangular learning-rate schedule is explicitly credited to Howard and Ruder (2018) and Smith (2017), and the distillation and gating designs are not justified by self-citations. The only overlapping-author reference (Tan et al. 2018 in the related-work survey) is not load-bearing. One non-circularity issue is flagged for completeness: Eq. 4 writes L_kd = -||h_lm - h_l||_2^2 while the text says the objective is to 'penalize the mean-squared-error loss'; taken literally, minimizing L = alpha*L_nmt + (1-alpha)*L_kd would maximize the MSE. This is a sign-consistency or reproducibility concern, not an equivalence of the result to its inputs. Overall, no derivation chain in the paper reduces by construction to its own assumptions, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The framework introduces no new physical or mathematical entities. The free parameters are standard training hyperparameters, and the core assumptions are about the usefulness of BERT hidden states and the reality of catastrophic forgetting.

free parameters (4)
  • alpha (α) = 0.9
    Balance coefficient between NMT loss and distillation loss in Eq. 5. Tuned on the WMT14 En-De validation set; it is a free hyperparameter that affects the final BLEU.
  • T_prime = 10000
    Switch point for the learning-rate schedule in Eq. 10. Set by preliminary experiments; a free schedule hyperparameter.
  • T = 20000
    Total fine-tuning steps for the LM learning-rate schedule in Eq. 10. Set by preliminary experiments; a free schedule hyperparameter.
  • BERT layer choice = second-to-last
    The paper selects the second-to-last BERT hidden layer (Sec. 3.2) and the 3rd NMT encoder layer for distillation (Sec. 4.6) based on validation performance; a categorical free choice.
assumptions (4)
  • domain assumption The Transformer architecture and its hyperparameters are a strong baseline; gains are measured relative to it.
    The paper uses Vaswani et al.'s Transformer settings without re-optimizing them; if the baseline is suboptimal, the relative gains may be inflated.
  • domain assumption The L2 distance between a fixed BERT layer and an NMT encoder layer is a meaningful knowledge-transfer objective.
    Invoked in Eq. 4 and Sec. 4.6; no evidence that the representation spaces of BERT and NMT are aligned in a way that makes MSE between a chosen pair of layers informative.
  • domain assumption Catastrophic forgetting is the correct explanation for the degradation of fine-tuned BERT in NMT, and the proposed regularizers mitigate it.
    The paper cites Goodfellow et al. and Edunov et al., but does not directly measure BERT's language modeling performance before/after fine-tuning; the forgetting is inferred from BLEU changes.
  • standard math BLEU, computed with multi-bleu.pl, is a valid metric for the comparisons.
    Used consistently with prior work; the evaluation protocol is stated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Making the Most of BERT in Neural Machine Translation." pith.science (2026). https://pith.science/paper/774XCGOO

@misc{pith2026190805672,
  author       = {Pith},
  title        = {Pith review of: Towards Making the Most of BERT in Neural Machine Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/774XCGOO}},
  note         = {Machine review of arXiv:1908.05672}
}
read the original abstract

GPT-2 and BERT demonstrate the effectiveness of using pre-trained language models (LMs) on various natural language processing tasks. However, LM fine-tuning often suffers from catastrophic forgetting when applied to resource-rich tasks. In this work, we introduce a concerted training framework (CTNMT) that is the key to integrate the pre-trained LMs to neural machine translation (NMT). Our proposed CTNMT consists of three techniques: a) asymptotic distillation to ensure that the NMT model can retain the previous pre-trained knowledge; b) a dynamic switching gate to avoid catastrophic forgetting of pre-trained knowledge; and c) a strategy to adjust the learning paces according to a scheduled policy. Our experiments in machine translation show CTNMT gains of up to 3 BLEU score on the WMT14 English-German language pair which even surpasses the previous state-of-the-art pre-training aided NMT by 1.4 BLEU score. While for the large WMT14 English-French task with 40 millions of sentence-pairs, our base model still significantly improves upon the state-of-the-art Transformer big model by more than 1 BLEU score. The code and model can be downloaded from https://github.com/bytedance/neurst/ tree/master/examples/ctnmt.

Figures

Figures reproduced from arXiv: 1908.05672 by the authors.

Figure 1
Figure 1. The overall CTNMT with asymptotic distillation and dynamic switch. representations. GPT-2 can be viewed as a causal language modeling (CLM) task consisting of a Transformer LM trained to fit the probability of a word given previous words in a sentence, while BERT is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers. Specifically, from an inp… view at source ↗
Figure 3
Figure 3. The slanted triangular learning rate schedule [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 12 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135--146

    2017

  4. [4]

    Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Lawrence Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2018. The best of both worlds: Combining recent advances in neural machine translation. In ACL

    work page 2018

  5. [5]

    Junyoung Chung, C aglar G \" u l c ehre, KyungHyun Cho, and Yoshua Bengio. 2014. http://arxiv.org/abs/1412.3555 Empirical evaluation of gated recurrent neural networks on sequence modeling . CoRR, abs/1412.3555

    arXiv 2014

  6. [6]

    Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493--2537

    work page 2011

  7. [7]

    George E Dahl, Dong Yu, Li Deng, and Alex Acero. 2012. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 20(1):30--42

    work page 2012

  8. [8]

    Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    arXiv 2018

Show all 31 references
  1. [9]

    Sergey Edunov, Alexei Baevski, and Michael Auli. 2019. Pre-trained language model representations for language generation. CoRR, abs/1903.09722

    2019 arXiv

  2. [10]

    Goodfellow , Mehdi Mirza , Da Xiao , Aaron Courville , and Yoshua Bengio

    Ian J. Goodfellow , Mehdi Mirza , Da Xiao , Aaron Courville , and Yoshua Bengio . 2013. http://arxiv.org/abs/1312.6211 An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks . arXiv e-prints, page arXiv:1312.6211

    2013 arXiv

  3. [11]

    Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor O.K. Li. 2018. https://doi.org/10.18653/v1/N18-1032 Universal neural machine translation for extremely low resource languages . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computat...

    2018 doi

  4. [12]

    Jeremy Howard and Sebastian Ruder. 2018. https://www.aclweb.org/anthology/P18-1031 Universal language model fine-tuning for text classification . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328--339,...

    2018

  5. [13]

    Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. neural information processing systems, pages 3294--3302

    2015

  6. [14]

    Guillaume Lample and Alexis Conneau. 2019. http://arxiv.org/abs/1901.07291 Cross-lingual language model pretraining . CoRR, abs/1901.07291

    2019 arXiv

  7. [15]

    Quoc Le and Tomas Mikolov. 2014. http://proceedings.mlr.press/v32/le14.html Distributed representations of sentences and documents . In Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 1188--1196...

    2014

  8. [16]

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. neural information processing systems, pages 3111--3119

    2013

  9. [17]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311--318. Association for Computational Linguistics

    2002

  10. [18]

    Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. https://doi.org/10.3115/v1/D14-1162 G love: Global vectors for word representation . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , pages 1532--1543, Doha,...

    2014 doi

  11. [19]

    Gabriel Pereyra, George Tucker, Jan Chorowski, ukasz Kaiser, and Geoffrey Hinton. 2017. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548

    2017 arXiv

  12. [20]

    Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. https://doi.org/10.18653/v1/N18-1202 Deep contextualized word representations . In Proceedings of the 2018 Conference of the North A merican Chapter of the Assoc...

    2018 doi

  13. [21]

    Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. https://doi.org/10.18653/v1/N18-2084 When and why are pre-trained word embeddings useful for neural machine translation? In Proceedings of the 2018 Conference of the North A merican Chapter o...

    2018 doi

  14. [22]

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training

    2018

  15. [23]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners

    2019

  16. [24]

    Prajit Ramachandran, Peter Liu, and Quoc Le. 2017. https://doi.org/10.18653/v1/D17-1039 Unsupervised pretraining for sequence to sequence learning . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 383--391, Copenhagen, Denmark. ...

    2017 doi

  17. [25]

    Leslie N. Smith. 2017. Cyclical learning rates for training neural networks. In WACV , pages 464--472. IEEE Computer Society

    2017

  18. [26]

    Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. pages 1631--1642

    2013

  19. [27]

    Zhixing Tan, Mingxuan Wang, Jun Xie, Yidong Chen, and Xiaodong Shi. 2018. Deep semantic role labeling with self-attention. national conference on artificial intelligence, pages 4929--4936

    2018

  20. [28]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf Attention is all you need . In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fe...

    2017

  21. [29]

    Gomez , Lukasz Kaiser , and Illia Polosukhin

    Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N. Gomez , Lukasz Kaiser , and Illia Polosukhin . 2017. Attention is all you need. neural information processing systems, pages 5998--6008

    2017

  22. [30]

    Dong Wang and Thomas Fang Zheng. 2015. Transfer learning for speech and language processing. asia pacific signal and information processing association annual summit and conference, pages 1225--1237

    2015

  23. [31]

    Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. https://doi.org/10.18653/v1/D16-1163 Transfer learning for low-resource neural machine translation . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1568--1575, Aus...

    2016 doi

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.