Pith. sign in

REVIEW 4 major objections 5 minor 39 references

Encoder-Agnostic Adaptation for Conditional Language Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Pseudo self attention adapts a pretrained language model to arbitrary conditional inputs by injecting the source directly into self-attention.

desk verdict Pseudo self attention is a simple, sensible way to inject arbitrary conditioning into a pretrained decoder, and the paper makes a decent empirical case for it, but the significance reporting and the unfair Repr-Transformer baseline keep me from fully buying the 'most consistent' claim. read the letter →

arxiv 1908.06938 v2 pith:G7OILYCD submitted 2019-08-19 cs.CL

Zachary M. Ziegler , Luke Melas-Kyriazi , Sebastian Gehrmann , Alexander M. Rush This is my paper · ORCID
classification cs.CL
keywords pseudoselfattentionencoder-agnosticadaptationpretrainedlanguagemodelsconditionaltextgenerationGPT-2fine-tuningtransferlearningforNLGabstractivesummarizationimageparagraphcaptioning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to answer how a pretrained transformer language model should be adapted to arbitrary conditional inputs, and claims that the best encoder-agnostic recipe is to inject the encoded source directly into the model's self-attention as extra key/value vectors, an approach it calls pseudo self attention. Across four diverse tasks — class-conditional movie reviews, CNN/DailyMail summarization, WritingPrompts stories, and Visual Genome paragraph captioning — this method outperforms two alternative adaptation schemes and is the most consistent. The authors trace the advantage to parameter stability: alternative schemes that add cross-attention layers or frozen representations drift further from the pretrained weights, eroding the language model's generation ability. If the claim holds, it gives practitioners a simple, modality-agnostic way to turn any pretrained autoregressive model into a conditional generator without redesigning the architecture.

What carries the argument

The load-bearing object is pseudo self attention, a modification of the transformer self-attention block in which the encoded source $X$ is concatenated into the keys and values via learned projection matrices $U^k, U^v$ of size $D$-by-$D'$, while the query stream remains the target tokens $Y$: $PSA(X,Y) = softmax((YW_q)[XU^k; YW_k]^T)[XU^v; YW_v]$. This injects conditioning additively into attention outputs without changing the pretrained query and output projections, so the pretrained weights need to move less during fine-tuning. The paper uses this as a direct test of its hypothesis that successful encoder-agnostic adaptation should minimize interference with pretrained parameters, and it measures interference as root median squared deviation of feed-forward parameters from their initial values.

What would settle it

Retrain the Repr-Transformer baseline with the same 12-layer, 12-head, 768-dim decoder and regularization as pseudo self attention on IMDb and the other three tasks; if it matches or beats pseudo self attention in perplexity, sentiment accuracy, ROUGE, and captioning scores, the paper's central claim of consistent superiority fails. A cheaper check is to vary Repr-Transformer width and see whether the Pseudo-Self margin shrinks as capacity grows.

Watch

Extended reading notes

Core claim

The central claim is that a pretrained autoregressive transformer can be conditioned on arbitrary source information by learning small projections that map the encoder output into the key/value space of the existing self-attention, rather than by adding a separate cross-attention mechanism or relying on frozen representations. Concretely, pseudo self attention computes $PSA(X,Y) = softmax((YW_q)[XU_k; YW_k]^T)[XU_v; YW_v]$, where $U_k$ and $U_v$ are new learned parameters and $X$ is any encoder output. Because self-attention is set-based, the model can use or ignore this pseudo-history, and only the new projections and the task encoder need to be trained. The paper's evidence is that this method wins on all four tasks, keeps source adherence nearly intact, is markedly more data-efficient than training from scratch, and produces qualitatively more coherent text.

Load-bearing premise

The fair-comparison assumption that the Repr-Transformer baseline was configured appropriately is load-bearing: the paper gives it a smaller 6-layer, 512-dim decoder 'to avoid overfitting' while pseudo self attention uses the full 12-layer GPT-2 decoder, so if capacities were matched the claimed advantage could shrink.

Editorial extensions

If this is right

  • Pseudo self attention gives a single encoder-agnostic recipe that improves over both a representation-frozen decoder and a cross-attention decoder across all four tasks tested.
  • Generation stays faithful to the source: on IMDb, sentiment classification of generated reviews drops only 0.4% versus a no-pretraining transformer, while the representation baseline drops 20.0%.
  • The method is data-efficient: with only 200 supervised examples it reaches 44.4 perplexity on IMDb, while a transformer trained from scratch exceeds 1000.
  • Larger pretrained models continue to help: switching from the 117M to the 345M GPT-2 lowers IMDb perplexity from 34.80 to 30.26 with essentially no loss in sentiment accuracy.
  • Human ratings on story generation show significant gains over a no-pretraining baseline on grammaticality, non-redundancy, consistency, and typicality, with grammaticality within 6.1% of human-written text.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Read as a learned key/value prefix, pseudo self attention anticipates parameter-efficient conditioning: the same additive injection could be used with a largely frozen language model and only a small trained encoder, which the paper's data-efficiency results already hint at.
  • Because the injection is modality-agnostic, the recipe should extend to other source types such as structured data, audio, or database rows, provided an encoder maps them to the decoder's hidden dimension; the paper only tests text, class bits, and CNN image features.
  • The paper's explanation — that keeping parameters near initialization preserves generation ability — is supported only by a feed-forward-layer drift plot on one task; a direct test would measure per-layer drift on all methods across tasks and correlate it with generation quality.
  • A fair comparison controlling total parameter count across adaptation methods is a natural extension and would determine whether the advantage is the injection mechanism itself or simply the larger decoder used by pseudo self attention.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies how to adapt a pretrained transformer language model (GPT-2) to conditional text generation in an encoder-agnostic setting, where the source can be a bit, text, or image features. It compares three adaptation methods: Repr-Transformer, which uses the pretrained model only to produce contextual target representations; Context-Attn, which inserts a randomly initialized cross-attention layer into a pretrained decoder; and the proposed Pseudo-Self attention, which injects learned encoder representations directly into the self-attention computation of the pretrained decoder. Experiments cover four tasks: class-conditional IMDb review generation, CNN/Daily Mail summarization, WritingPrompts story generation, and Visual Genome image paragraph captioning. The paper reports that Pseudo-Self attention outperforms the alternatives, is the most consistent across tasks, is data efficient, and produces more coherent outputs, with supporting human evaluation and a scaling experiment using a larger GPT-2 model.

Significance. If the empirical claims hold, the paper makes a useful and simple contribution: pseudo self attention is an elegant way to condition a pretrained decoder on arbitrary source modalities without adding a separate cross-attention block or retraining a full decoder. The study is broad, covering four tasks with different input types and output lengths, and it includes a human evaluation, a data-efficiency analysis, and a larger-model scaling point. The authors also release code, which supports reproducibility. The main weakness is that the central claim of consistent, statistically significant superiority is not backed by the reporting: the main tables claim significance but give no variance, number of seeds, or test procedure, and the key baseline is run with a smaller decoder than the proposed method. The reported numbers even contradict the literal "across all tasks" claim on one metric. The contribution is potentially significant, but the evidence as presented is not yet sufficient to establish the headline claim.

major comments (4)
  1. [Section 4, Tables 1-4] The captions of Tables 1, 2, 3, and 4 state that bold indicates statistically significant best results at p≤0.05, but the paper never reports the number of seeds, the standard deviation or confidence interval for any main-table score, or the statistical test used. This is a load-bearing issue because the central claim is that Pseudo-Self is the most consistent method, yet the margins are small precisely where the comparison is most direct: in Table 2, Pseudo-Self beats Context-Attn by only 0.13 ROUGE-1 (40.72 vs. 40.59), 0.21 ROUGE-2 (18.38 vs. 18.17), 0.22 ROUGE-L (37.46 vs. 37.24), and 0.25 PPL (6.43 vs. 6.68). Without variance or a defined test, the significance claim is not verifiable, and the "consistent" superiority claim is not established.
  2. [Section 4, first paragraph] The Repr-Transformer baseline is not configured fairly. The paper states that "for the Repr-Transformer model to avoid overfitting we use 6/8/512 layers/heads/dim for the decoder," while Context-Attn and Pseudo-Self use the full 12-layer/12-head/768-dim GPT-2 decoder. Because the headline claim is that Pseudo-Self outperforms strong baselines, this capacity mismatch is a confound. The overfitting justification is not empirically demonstrated, and a smaller decoder could explain part of Repr-Transformer's worse performance. The authors should either match the decoder capacity across methods or provide a controlled experiment measuring the effect of this architectural choice.
  3. [Section 4.3, Table 3] The literal claim in the Introduction that "pseudo self attention outperforms the other pretraining methods" across all tasks is contradicted by the paper's own reported numbers: in Table 3, Repr-Trans has PPL 21.16 while Pseudo-Self has PPL 21.21. This may be a small difference, but it directly weakens the "most consistent" formulation. Additionally, the Context-Attn collapse on this task (PPL >5000, rank accuracy 9.3) is reported without any diagnostic or explanation; since this is the only task where Context-Attn catastrophically fails, the paper should either explain the optimization failure or temper the consistency claim accordingly.
  4. [Section 5.2, Figure 3] The data-efficiency analysis reports error bars described as an "approximate 95% confidence interval," but the manuscript does not specify how the interval was computed, how the eight repeats were structured, or whether any variance information exists for the main results in Tables 1-4. This is the only place in the paper where uncertainty is visualized, so its methodology should be explicit; otherwise readers cannot calibrate the significance claims made elsewhere.
minor comments (5)
  1. [Section 2, Related Work] The sentence "This study compliments ours with positive results on dialogue generation" should use "complements" rather than "compliments."
  2. [Section 3, Figure 2] The metric "root median squared deviation" is not defined; please specify the formula and whether it is computed over all feed-forward parameters or a subset.
  3. [Section 5.4, Qualitative Examples] The text refers to the "Pseudo-Attention model," but the paper elsewhere uses "Pseudo-Self attention"; the terminology should be consistent.
  4. [Section 5.1, Table 5] The caption of Table 5 says "the larger 345M parameter GPT2 model to the 117M parameter GPT model," but both are GPT-2 models; please correct the wording.
  5. [Section 4.4, Visual Genome] The paragraph captioning experiment trains the encoder and decoder separately rather than end-to-end, unlike the other tasks; please state explicitly why this departure is made and how it may affect the comparison.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: empirical architecture comparison with external benchmarks; mild self-referential evaluation via an IMDb-trained sentiment classifier, but no derivation reduces to its inputs.

full rationale

The paper's central claim is an empirical comparison of encoder-agnostic adaptation methods across four generation tasks, measured with external benchmarks (ROUGE, CIDEr/BLEU, an external fastText sentiment classifier, and human evaluation). There is no derivation chain in which a predicted quantity is defined in terms of the same fitted quantity. The proposed Pseudo-Self attention is defined by an equation, PSA(X,Y) = softmax((YW_q)[XU_k; YW_k]^T)[XU_v; YW_v], but the paper does not derive its empirical superiority from this definition; it tests it against baselines. The closest mild self-referential element is the class-conditional generation evaluation: the sentiment classifier used to measure class adherence is trained on the same IMDb training set that also trains the generation models, so high class accuracy partially reflects the classifier recognizing the same distribution rather than an independent property. This is an evaluation-metric concern, not a circular derivation, and it is disclosed as an external classifier with reported 90.1% test accuracy. Self-citations to Gehrmann et al. (2018), Melas-Kyriazi et al. (2018), and Zhao et al. (2018) are used for baselines or standard components (bottom-up attention, paragraph captioning baselines), not as load-bearing justification for the main result. The paper is self-contained as an empirical study; therefore circularity is minimal.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the assumed transferability of GPT-2's pretrained weights, the harmlessness of adding extra attention keys/values, and the validity of the evaluation metrics. The main hand-chosen parameters are evaluation settings and an architecture choice for a baseline; no new physical entities are introduced.

free parameters (4)
  • Repr-Transformer decoder size = 6 layers, 8 heads, 512 dim
    Selected by hand to avoid overfitting; this architectural choice reduces the capacity of a key baseline and could bias the comparison in favor of Pseudo-Self (Section 4).
  • Generation temperature = 0.7, 0.75, 0.9
    Evaluation-time sampling temperatures tuned by hand; the claimed coherence and class adherence results depend on these values.
  • Beam size and trigram blocking = beam=5, trigram blocking
    Used for summarization decoding; consistent with literature, but a hand-chosen hyperparameter affecting ROUGE scores.
  • Top-k sampling = 100
    Used in story generation human evaluation; selected for generation quality.
assumptions (5)
  • domain assumption GPT-2 (117M) is a high-quality unconditional language model whose representations transfer to conditional generation tasks.
    The method assumes the pretrained LM provides a strong prior for target text; supported by prior work but not proven for all tasks.
  • domain assumption Injecting extra key/value vectors into self-attention does not disrupt the pretrained attention distribution in a harmful way.
    The PSA formulation relies on the model being able to ignore or use extra context; the paper provides indirect evidence (RMSD) but no theoretical guarantee.
  • domain assumption Perplexity, ROUGE, CIDEr, and a fastText sentiment classifier are valid measures of generation quality and source adherence.
    The central comparison uses these metrics as proxies; each has known limitations.
  • domain assumption Fine-tuning all pretrained weights end-to-end with cross-entropy loss is an appropriate optimization procedure for the adaptation.
    This is standard practice, but the paper does not compare with partial freezing or other optimization strategies.
  • domain assumption The fastText classifier trained on IMDb provides an unbiased measure of class adherence on generated reviews.
    The classifier and the generator are trained on the same dataset, so evaluation of class adherence is partly circular.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Encoder-Agnostic Adaptation for Conditional Language Generation." pith.science (2026). https://pith.science/paper/G7OILYCD

@misc{pith2026190806938,
  author       = {Pith},
  title        = {Pith review of: Encoder-Agnostic Adaptation for Conditional Language Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G7OILYCD}},
  note         = {Machine review of arXiv:1908.06938}
}
read the original abstract

Large pretrained language models have changed the way researchers approach discriminative natural language understanding tasks, leading to the dominance of approaches that adapt a pretrained model for arbitrary downstream tasks. However it is an open-question how to use similar techniques for language generation. Early results in the encoder-agnostic setting have been mostly negative. In this work we explore methods for adapting a pretrained language model to arbitrary conditional input. We observe that pretrained transformer models are sensitive to large parameter changes during tuning. We therefore propose an adaptation that directly injects arbitrary conditioning into self attention, an approach we call pseudo self attention. Through experiments on four diverse conditional text generation tasks we show that this encoder-agnostic technique outperforms strong baselines, produces coherent generations, and is data efficient.

Figures

Figures reproduced from arXiv: 1908.06938 by the authors.

Figure 1
Figure 1. Encoder-agnostic variants considered. All methods utilize a problem-specific source en [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Comparison of parameter changes in feed forward layers with different condition￾ing. Root median squared deviation between feed forward parameters, for the Pseudo-Self and Context-Attn models. The Context-Attn ap￾proach requires a larger deviation from the ini￾tialization to fit the data. Model PPL Cls Acc Test set - 90.1 GPT-2 41.21 - Simple Fusion 38.31 65.1 Transformer 105.43 92.7 Repr-Trans 39.69 72.7 Context-At… view at source ↗
Figure 3
Figure 3. Data efficiency analysis with IMDb. PPL shown in blue (left), classification accuracy shown in orange (right). Error bars show an ap￾proximate 95% confidence interval. To compare models we compute two metrics: perplexity (PPL) and prompt ranking. Perplexity is used as a proxy for generation quality, whereas prompt ranking is used to measure the relevance of the story to the prompt. To calculate prompt ranking, we us… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 18 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Bahl, Frederick Jelinek, and Robert L

    Lalit R. Bahl, Frederick Jelinek, and Robert L. Mercer. A Maximum Likelihood Approach to Continuous Speech Recognition . IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-5 0 (2): 0 179--190, 1983. ISSN 01628828. doi:10.1109/TPAMI.1983.4767370

    work page arXiv 1983

  3. [3]

    Diverse and coherent paragraph generation from images

    Moitreya Chatterjee and Alexander G Schwing. Diverse and coherent paragraph generation from images. In Proceedings of the European Conference on Computer Vision (ECCV), pp.\ 729--744, 2018

    work page 2018

  4. [4]

    An embarrassingly simple approach for transfer learning from pretrained language models

    Alexandra Chronopoulou, Christos Baziotis, and Alexandros Potamianos. An embarrassingly simple approach for transfer learning from pretrained language models. CoRR, abs/1902.10547, 2019

    arXiv 1902

  5. [5]

    Overview of DUC 2006

    Hoa Trang Dang. Overview of DUC 2006 . Proceedings of HLT-NAACL, 2006

    work page 2006

  6. [6]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding . arXiv preprint arXiv:1810.04805, 2018. ISSN 0140-525X. doi:arXiv:1811.03600v2. URL http://arxiv.org/abs/1810.04805

    arXiv 2018

  7. [7]

    Unified language model pre-training for natural language understanding and generation

    Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. ArXiv, abs/1905.03197, 2019

    arXiv 1905

  8. [8]

    Pre-trained Language Model Representations for Language Generation

    Sergey Edunov, Alexei Baevski, and Michael Auli. Pre-trained Language Model Representations for Language Generation . NAACL-HLT, 2019

    work page 2019

Show all 39 references
  1. [9]

    Hierarchical Neural Story Generation

    Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical Neural Story Generation . Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 889--898, 2018. ISSN 1473-3250. doi:10.1016/j.carres.2011.08.014. URL http://a...

    2018 arXiv

  2. [10]

    Sebastian Gehrmann, Yuntian Deng, and Alexander M. Rush. Bottom-Up Abstractive Summarization . Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018. URL http://arxiv.org/abs/1808.10792

    2018 arXiv

  3. [11]

    Large-scale transfer learning for natural language generation

    Sergey Golovanov, Rauf Kurbanov, Sergey Nikolenko, Kyryl Truskovskyi, Alexander Tselousov, and Thomas Wolf. Large-scale transfer learning for natural language generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 6053--605...

    2019

  4. [12]

    On Using Monolingual Corpora in Neural Machine Translation

    Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. On Using Monolingual Corpora in Neural Machine Translation . arXiv preprint arXiv:1503.03535, 2015. ISSN 1560-7917. doi:10.2807/1560-7917.ES2...

    2015 arXiv

  5. [13]

    Teaching Machines to Read and Comprehend

    Karl Moritz Hermann, Tom \' a s Ko c isk \' y , Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching Machines to Read and Comprehend . Proceedings of NIPS, pp.\ 1--14, 2015. URL http://arxiv.org/abs/1506.03340

    2015 arXiv

  6. [14]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. CoRR, abs/1902.00751, 2019

    1902 arXiv

  7. [15]

    Universal Language Model Fine-tuning for Text Classification

    Jeremy Howard and Sebastian Ruder. Universal Language Model Fine-tuning for Text Classification . Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018. ISSN 23330384. doi:arXiv:1801.06146v3. URL http://arxiv.org/...

    2018 arXiv

  8. [16]

    Bag of Tricks for Efficient Text Classification

    Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of Tricks for Efficient Text Classification . arXiv preprint arXiv:1607.01759, 2016. URL http://arxiv.org/abs/1607.01759

    2016 arXiv

  9. [17]

    Statistical Phrase-Based Translation

    Philipp Koehn, Franz Josef Och, and Daniel Marcu. Statistical Phrase-Based Translation . Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, 0 (June): 0 48--54, 2003

    2003

  10. [18]

    A hierarchical approach for generating descriptive image paragraphs

    Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. A hierarchical approach for generating descriptive image paragraphs. In Computer Vision and Patterm Recognition (CVPR), 2017

    2017

  11. [19]

    Cross-lingual language model pretraining

    Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291, 2019

    1901 arXiv

  12. [20]

    Learning Word Vectors for Sentiment Analysis

    Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning Word Vectors for Sentiment Analysis

  13. [21]

    Learned in Translation: Contextualized Word Vectors

    Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in Translation: Contextualized Word Vectors . 31st Conference on Neural Information Processing Systems, 2017. URL http://arxiv.org/abs/1708.00107

    2017 arXiv

  14. [22]

    Training for diversity in image paragraph captioning

    Luke Melas-Kyriazi, Alexander Rush, and George Han. Training for diversity in image paragraph captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.\ 757--761, 2018

    2018

  15. [23]

    Distributed Representations of Words and Phrases and their Compositionality

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality . Advances in Neural Information Processing Systems, 2013

    2013

  16. [24]

    Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer

    Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations . Proceedings of NAACL-HLT, 2018. doi:10.18653/v1/N18-1202. URL http://arxiv.org/abs/1802.05365

    2018 arXiv

  17. [25]

    GPT: Improving Language Understanding by Generative Pre-Training

    Alec Radford and Tim Salimans. GPT: Improving Language Understanding by Generative Pre-Training . arXiv, pp.\ 1--12, 2018. ISSN 1095-8290. doi:10.1093/aob/mcp031. URL https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language \_ understandi...

    2018 doi

  18. [26]

    Language Models are Unsupervised Multitask Learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners . 2019

    2019

  19. [27]

    Liu, and Quoc V

    Prajit Ramachandran, Peter J. Liu, and Quoc V. Le. Unsupervised Pretraining for Sequence to Sequence Learning . Proceedings of EMNLP, 2017

    2017

  20. [28]

    Style transfer from non-parallel text by cross-alignment

    Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from non-parallel text by cross-alignment. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pp.\ 6833--6844, USA, 2017. Curran Associates Inc. ISBN...

    2017

  21. [29]

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and T. M. Liu. Mass: Masked sequence to sequence pre-training for language generation. In ICML, 2019

    2019

  22. [30]

    Cold fusion: Training Seq2seq models together with language models

    Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates. Cold fusion: Training Seq2seq models together with language models . Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, 2018-Septe: 0 387--391, 2018. ISSN 19909...

    2018 doi

  23. [31]

    Simple Fusion: Return of the Language Model

    Felix Stahlberg, James Cross, and Veselin Stoyanov. Simple Fusion: Return of the Language Model . Proceedings of the Third Conference on Machine Translation: Research Papers, 1: 0 204--211, 2018. URL http://arxiv.org/abs/1809.00125

    2018 arXiv

  24. [32]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need . 31st Conference on Neural Information Processing Systems, pp.\ 5998--6008, 2017

    2017

  25. [33]

    Chenguang Wang, Mu Li, and Alexander J. Smola. Language models with transformers. CoRR, abs/1904.09408, 2019

    1904 arXiv

  26. [34]

    The Neural Noisy Channel

    Lei Yu, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Tomas Kocisky. The Neural Noisy Channel . arXiv preprint arXiv:1611.02554, pp.\ 1--13, 2017

    2017 arXiv

  27. [35]

    Pretraining-Based Natural Language Generation for Text Summarization

    Haoyu Zhang, Yeyun Gong, Yu Yan, Nan Duan, Jianjun Xu, Ji Wang, Ming Gong, and Ming Zhou. Pretraining-Based Natural Language Generation for Text Summarization . arXiv preprint arXiv:1902.09243, 2019. URL http://arxiv.org/abs/1902.09243

    1902 arXiv

  28. [36]

    Adversarially regularized autoencoders

    Junbo Zhao, Yoon Kim, Kelly Zhang, Alexander Rush, and Yann LeCun. Adversarially regularized autoencoders. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp....

    2018

  29. [37]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  30. [38]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  31. [39]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.