Pith. sign in

REVIEW 4 major objections 5 minor 48 references

Efficient Language Modeling for Low-Resource Settings with Hybrid RNN-Transformer Architectures

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Putting two cheap recurrent blocks ahead of a transformer beats a full attention stack at equal size and rivals far bigger models.

desk verdict Sensible hybrid RNN-Transformer idea with a promising internal comparison, but the single-seed significance claims are invalid and the code isn't released; worth a referee but needs a statistical fix. read the letter →

arxiv 2502.00617 v1 pith:PMRPBMEJ submitted 2025-02-02 cs.CL

Gabriel Lindenmaier , Sean Papay , Sebastian Padó This is my paper · ORCID
classification cs.CL
keywords hybridRNN-transformerlanguagemodelingquasi-recurrentneuralnetworksparameterefficiencylow-resourceNLPenwik8Wikitext-103
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether a language model can get transformer-level quality for a fraction of the compute by putting cheap recurrent layers where attention is least needed. The authors take a parameter-efficient transformer design, the PAR Transformer, and replace its first two attention layers with two AWD-QRNN blocks to form a Hybrid Transformer at the same parameter count. On enwik8 the hybrid reaches 1.013 bits-per-character, beating its own PAR baseline (1.047), the SRU++ (1.022), and the Adaptive Transformer (1.02); on Wikitext-103 it reaches 20.91 perplexity with about 60M parameters, beating Transformer-XL's 24.0 at 151M parameters. The claim is that a deliberate mix of recurrence and attention, rather than more attention alone, is the more efficient route to language modeling when data and compute are scarce.

What carries the argument

The central object is the AWD-QRNN block: a quasi-recurrent neural network layer, a partially parallelizable recurrent variant that convolves over the input and updates its state with gated pooling, with DropConnect-style weight dropout applied to its three gates. The Hybrid Transformer composes two such blocks ahead of a PAR Transformer while removing the first two attention layers and a few feed-forward layers so the parameter count stays level with the baseline. The composition does the work: the recurrent early layers supply a temporal inductive bias and local structure at low cost, the remaining multi-head attention with relative position information handles long-range dependencies in the deeper layers, and RNN-dropout with residual connections and layer normalization keeps training stable.

What would settle it

Re-train the SRU++, the Adaptive Transformer, the Longformer, and the Hybrid Transformer on enwik8 and Wikitext-103 under identical context lengths, GPU hardware, training budgets, and several random seeds. The central claim fails if the hybrid no longer beats comparable-parameter models on enwik8, or no longer matches Transformer-XL's 24.0 perplexity on Wikitext-103 at less than half its parameters.

Watch

Extended reading notes

Core claim

The central claim is that selectively replacing the first two attention layers of a PAR Transformer with two AWD-QRNN blocks yields a hybrid architecture that outperforms existing models with a comparable parameter count and matches larger models with substantially fewer parameters. In the paper's experiments the Hybrid Transformer achieves 1.013 bits-per-character on enwik8 test, improving on the PAR Transformer baseline (1.047), the SRU++ (1.022), and the Adaptive Transformer (1.02), with only the Longformer (1.00) scoring better at roughly 32 times the training compute. On Wikitext-103 the hybrid reaches 20.91 perplexity with about 60M parameters, passing Transformer-XL's 24.0 at 151M parameters. The paper interprets these results as evidence for its hypothesis that lower network layers learn comparatively simple local structure that recurrent layers handle well, while the deeper layers are where attention pays off, marking a 'sweet spot' for parameter- and compute-efficient language modeling.

Load-bearing premise

The load-bearing premise is that published results for other models, trained on different hardware and with different context lengths and training budgets, can be compared directly against the paper's single fixed-seed runs; if every baseline were re-run under identical conditions, the reported margins could shrink or reverse.

Editorial extensions

If this is right

  • A language model can spend most of its layer budget on cheap recurrent computation and still reap the benefits of attention, so near-transformer quality does not require a full stack of attention layers.
  • On Wikitext-103 the hybrid beats a transformer more than twice its size (59.9M versus 151M parameters), so the savings come from architecture, not from a smaller training budget.
  • The efficiency gain costs almost nothing in training time: the hybrid is only about 1.7% slower per batch than the PAR Transformer baseline.
  • The hybrid's edge carries over to PG-19, a book-corpus benchmark, across several tokenization and vocabulary sizes, with the gap over the PAR baseline growing at shorter token lengths.
  • The strongest gains appear exactly in the low-resource regime the paper targets: roughly 100M-token corpora and models of about 40 to 60M parameters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's layer-role explanation, recurrent layers for simple early structure and attention for deep structure, is supported only indirectly by end-to-end scores; a direct test would probe what the QRNN blocks encode compared with the attention layers they replaced.
  • If the pattern scales, a meaningful fraction of attention layers in production-sized models could be exchanged for recurrent blocks, cutting training energy and inference latency; the paper explicitly leaves scaling untested.
  • Because the early QRNN layers use a small fixed convolution window, the architecture builds in a locality prior that may fit English's relatively rigid word order better than freer-word-order or morphologically richer languages, a transfer question the paper lists as open.
  • The cleanest check of the headline claim would be a controlled re-run of the published baselines under the paper's exact context lengths and training budgets, separating architecture effects from setup differences.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes three language-modeling architectures for low-resource settings: the Attn-QRNN (an adapted SHA-LSTM), the PAR Transformer (a baseline adapted from Mandava et al., 2020), and the novel Hybrid Transformer, which replaces the first two attention layers of a PAR-style transformer with AWD-QRNN blocks. The models are evaluated on enwik8, WikiText-103, and, in an appendix, PG-19, with roughly 40–60M parameters and a fixed two-day training budget on an Nvidia Titan RTX. The central claim is that the Hybrid Transformer outperforms comparable-parameter models on enwik8 (1.013 BPC vs. 1.047 for the PAR baseline) and matches or beats larger models on WikiText-103 (20.91 PPL vs. 24.0 for a 151M-parameter Transformer-XL), while using fewer parameters and similar or less compute. The paper reports a single fixed-seed run for each architecture and supports the headline internal comparison with a Student's t-test (p < 0.001) and a Kolmogorov-Smirnov test, both computed from that single run.

Significance. If the central claim holds, the paper provides useful, concrete evidence that hybrid recurrent-transformer architectures can be more parameter- and compute-efficient than pure transformers in low-resource settings. The underlying hypothesis—that shallow recurrent layers can handle early, simpler linguistic structure while attention is reserved for deeper layers—is intuitive and worth testing empirically. The paper's strengths include a clearly specified architecture search space, detailed hyperparameter reporting, a stated fixed training budget, and a promising result direction. However, the significance is currently limited by the absence of valid uncertainty estimates for the internal comparisons and by the lack of controlled external baselines, so the headline claims should be treated as provisional until these issues are addressed.

major comments (4)
  1. [Section 4.3 and Section 5, Tables 2 and 3] The paper states in Section 4.3, 'We report results of a single fixed-seed run,' yet Section 5 claims that the 0.034 BPC improvement of the Hybrid Transformer over the PAR Transformer on enwik8 is 'a significant improvement according to Student’s t-test (p < 0.001).' With one run per architecture there is no between-run variance, so a valid t-test cannot be computed; if the test was instead applied to per-token or per-batch losses, those samples are strongly autocorrelated in language modeling, violating the independence assumption and making any nonzero difference spuriously significant. The same problem affects the Kolmogorov-Smirnov test used to claim that Hybrid and PAR results on WikiText-103 'are not significantly different.' The central internal claim that replacing attention with QRNN blocks improves performance requires multiple independent seeds (e.g., 5 or more) with reported means and standard deviations or confidence intervals, or another statistically valid comparison procedure.
  2. [Section 5, Tables 2 and 3] The external comparisons to published numbers are not controlled. For example, the SRU++ result of 1.022 BPC in Table 2 was trained for 37 GPU-hours on Nvidia 2080Ti GPUs, while the Hybrid Transformer was trained for 48 hours on an Nvidia Titan RTX; the Longformer comparison uses roughly 32x more compute, and the Transformer-XL comparison on WikiText-103 uses a 384-token training attention length versus 1024 tokens here. These differences in compute, hardware, and context length make it difficult to attribute the reported gains to the architecture alone. The abstract's claim that the reduced architectures 'outperform existing models with a comparable number of parameters' would be much stronger if at least one baseline (e.g., the PAR Transformer or an SRU++-like model) were retrained under the same training budget, context length, and seed protocol.
  3. [Section 4.1 and Section 5] Hyperparameters were tuned on enwik8 validation performance, and then the same enwik8 test set is used as a headline result. While the test set is distinct from the validation set, reporting a single run after tuning on the same dataset risks selection bias, especially with the statistically invalid significance claims described above. The paper should report multiple seeds and, ideally, state how hyperparameter selection was performed (e.g., early stopping on validation) so that the test numbers can be interpreted as unbiased estimates of generalization.
  4. [Appendix B] The PG-19 appendix reports Levene tests and Student's t-tests with extremely small p-values (e.g., p < 1e-60) to support the claim that the Hybrid Transformer is better than the PAR Transformer across tokenizations. These tests again appear to be computed from a single fixed-seed run per configuration, for the same reason as in Section 5, and are therefore invalid as supporting evidence. Additionally, the comparison to Compressive Transformer and Routing Transformer uses models with substantially more parameters (523M vs. ~40-58M), so the stated conclusion that 'the model parameter sizes are far too small for this large dataset' is not supported by a controlled comparison.
minor comments (5)
  1. [Section 5] The sentence 'All three architectures outperform their respective counterparts from other works' is too strong; for example, the Longformer achieves a better BPC (1.00 vs. 1.013) despite using more compute, so 'outperform' should be qualified with respect to the compute budget.
  2. [Section 3, Table 1] The architecture notation in Table 1 (e.g., '| + 3×(q|f) + (qafff)') is not formally defined; the paper should state the semantics of concatenation and repetition, and define what 'qafff' means in terms of the layer ordering.
  3. [Throughout] There are several typos and grammatical errors, including 'wich' for 'which', 'allevieate' for 'alleviate', 'V olume' in the bibliography, and 'far to small' in Appendix B. The manuscript would benefit from a careful proofreading pass.
  4. [Section 4.1] The claim that the Hybrid Transformer is only '1.7% slower' in training speed per data batch should be accompanied by a note on whether this includes the two-day wall-clock budget or only per-batch throughput; the current wording is ambiguous.
  5. [Abstract] The abstract promises that 'the implementation is going to be available at this URL,' but no working repository is provided at the time of review. If the code is not available, the reproducibility of the experimental claims is substantially reduced; the authors should either provide the code or clearly state its availability status.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical architecture comparison whose central claim is not derived from its inputs by construction.

full rationale

This paper makes no analytic derivation: its results are empirical benchmark numbers obtained by training three explicitly defined architectures (Attn-QRNN, PAR Transformer, Hybrid Transformer) and evaluating them on enwik8, WikiText-103, and PG-19. The central claim—that the Hybrid Transformer outperforms the PAR Transformer—is not an equation-level consequence of the model definitions; it is a comparison of test-set metrics obtained from training runs. The only input that could be called fitted is the hyperparameter schedule, which was tuned on enwik8 validation ('We tune hyperparameters based on performance on the enwik8 dataset', Section 4.1), but the reported enwik8 numbers are on the disjoint test split, and the same protocol is applied to all three architectures, so the comparison is not forced by construction. All external baselines cited (Merity 2019; Dai et al. 2019; Lei 2021; Mandava et al. 2020) are prior independent work, not self-citations of the present authors, and none is used to define the Hybrid architecture's output. There is no uniqueness theorem, no ansatz smuggled in via self-citation, and no renamed known result; the architecture definitions in Table 1 are given directly in terms of standard building blocks. Any statistical critique of the single fixed-seed significance tests (Sections 4.3 and 5) is a correctness or comparability concern, not circularity. Accordingly, no circular step can be exhibited with the required quote-and-reduction standard.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

This is an empirical architecture paper, so the central claim rests on benchmark comparability and standard training assumptions rather than on a mathematical derivation. The key free choices are the architecture's QRNN placement and count, the context lengths, and the hyperparameters tuned on enwik8. No new theoretical entities are proposed.

free parameters (3)
  • Test-time attention length = 2048 for enwik8; 1600/2048 for Wikitext-103
    Chosen per model and dataset, and longer than some baselines (e.g., Transformer-XL's 384 training attention), so the parameter-efficiency comparison is not matched on context.
  • Number of QRNN blocks in the Hybrid Transformer = 2
    Chosen by hand; no ablation varies the count or position, so the contribution of the specific "QRNN early, attention late" placement is not isolated.
  • Training hyperparameters (learning rates, dropout, weight decay) = See Table 4
    Tuned on enwik8 validation; the enwik8 test result is therefore not fully independent of hyperparameter selection, and all results come from a single fixed seed.
assumptions (3)
  • domain assumption Published comparison numbers from prior work (Merity 2019, Dai 2019, Lei 2021, etc.) are accurate and were obtained under conditions that make them comparable to the authors' runs.
    The central "outperforms existing models" claim is based on tables comparing to literature numbers without re-running baselines under identical conditions; this is load-bearing for the abstract's headline.
  • domain assumption A single fixed-seed run is representative enough to support the reported differences.
    Section 4.3 reports "a single fixed-seed run"; no seed variance is provided, yet the conclusions claim significant improvements.
  • domain assumption The significance tests over batches are valid indicators of architectural superiority.
    The t-test and Kolmogorov-Smirnov test are applied over batches within one run; language-model batches are autocorrelated, so the p-values may overstate significance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Language Modeling for Low-Resource Settings with Hybrid RNN-Transformer Architectures." pith.science (2026). https://pith.science/paper/PMRPBMEJ

@misc{pith2026250200617,
  author       = {Pith},
  title        = {Pith review of: Efficient Language Modeling for Low-Resource Settings with Hybrid RNN-Transformer Architectures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PMRPBMEJ}},
  note         = {Machine review of arXiv:2502.00617}
}
read the original abstract

Transformer-based language models have recently been at the forefront of active research in text generation. However, these models' advances come at the price of prohibitive training costs, with parameter counts in the billions and compute requirements measured in petaflop/s-decades. In this paper, we investigate transformer-based architectures for improving model performance in a low-data regime by selectively replacing attention layers with feed-forward and quasi-recurrent neural network layers. We test these architectures on the standard Enwik8 and Wikitext-103 corpora. Our results show that our reduced architectures outperform existing models with a comparable number of parameters, and obtain comparable performance to larger models while significantly reducing the number of parameters.

Figures

Figures reproduced from arXiv: 2502.00617 by the authors.

Figure 1
Figure 1. Building blocks of our models Sequential Network Linear Softmax Token Embedding RNN Dropout (a) Base model layout 3x Multi-Head Attention Add & Norm Dropout Feed Forward Block (b) PAR transformer block Feed Forward Block RNN Dropout QRNN Add & Norm (c) QRNN- plus feed￾forward layer (QRNN FF) QRNN Add & Norm QRNN FF Block RNN Dropout (d) QRNN block for the Hybrid Transformer [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Language model buildup & single PAR transformer and QRNN blocks. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The "Sequential Network" part of our three language modeling architecture layouts [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 17 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Alexei Baevski and Michael Auli. 2019. https://openreview.net/forum?id=ByxZX20qFQ Adaptive input representations for neural language modeling . In International Conference on Learning Representations

    work page 2019

  4. [4]

    Dzmitry Bahdanau, Kyung-Hyun Cho, and Yoshua Bengio. 2015. https://www.iclr.cc/archive/www/lib/exe/fetch.php Neural machine translation by jointly learning to align and translate . In Proceedings of ICLR

    work page 2015

  5. [5]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. http://arxiv.org/abs/2004.05150 Longformer: The long-document transformer . CoRR, abs/2004.05150

    arXiv 2020

  6. [6]

    Kaj Bostrom and Greg Durrett. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.414 Byte pair encoding is suboptimal for language model pretraining . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4617--4624, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.findings-emnlp.414 2020

  7. [7]

    James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. 2017. https://openreview.net/pdf?id=H1zJ-v5xl Quasi-recurrent neural networks . In International Conference on Learning Representations

    work page 2017

  8. [8]

    Sanzhu Cao and Yan Yue. 2019. https://doi.org/10.1109/IICSPI48186.2019.9095962 Financial news generation based on artificial intelligence technology . In 2019 2nd International Conference on Safety Produce Informatization (IICSPI), pages 398--401

    work page arXiv 2019

Show all 48 references
  1. [9]

    Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2018. https://doi.org/10.18653/v1/P18-1...

    2018 doi

  2. [10]

    Aaron Daniel Cohen, Adam Roberts, Alejandra Molina, Alena Butryna, Alicia Jin, Apoorv Kulshreshtha, Ben Hutchinson, Ben Zevenbergen, Blaise Hilary Aguera-Arcas, Chung ching Chang, Claire Cui, Cosmo Du, Daniel De Freitas Adiwardana, Dehao Chen, Dmitry (Dima) Lepikhin, Ed H. Chi...

    2022 arXiv

  3. [11]

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. https://doi.org/10.18653/v1/P19-1285 Transformer- XL : Attentive language models beyond a fixed-length context . In Proceedings of the 57th Annual Meeting of the Association for Com...

    2019 doi

  4. [12]

    Kunihiko Fukushima. 1975. https://doi.org/10.1007/BF00342633 Cognitron: A self-organizing multilayered neural network . Biol. Cybern., 20(3–4):121–136

    1975 doi

  5. [13]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  6. [14]

    \' E douard Grave, Armand Joulin, Moustapha Ciss\' e , David Grangier Facebook AI Research, and Herv\' e J\' e gou. 2017. Efficient softmax approximation for gpus. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 1302–1310. JMLR.org

    2017

  7. [15]

    Alex Graves. 2013. http://arxiv.org/abs/1308.0850 Generating sequences with recurrent neural networks . CoRR, abs/1308.0850

    2013 arXiv

  8. [16]

    Jie Hao, Xing Wang, Baosong Yang, Longyue Wang, Jinfeng Zhang, and Zhaopeng Tu. 2019. https://doi.org/10.18653/v1/N19-1122 Modeling recurrence for transformer . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistic...

    2019 doi

  9. [17]

    DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. 2022. https://doi.org/10.48550/ARXIV.2203.07852 Block-recurrent transformers . CoRR, abs/2203.07852

    2022 doi arXiv

  10. [18]

    Marcus Hutter. 2006. http://prize.hutter1.net/ The human knowledge compression contest . http://prize.hutter1.net/

    2006

  11. [19]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. http://arxiv.org/abs/2001.08361 Scaling laws for neural language models . CoRR, abs/2001.08361

    2020 arXiv

  12. [20]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2015. http://arxiv.org/abs/1412.6980 Adam: A method for stochastic optimization . In ICLR (Poster)

    2015 arXiv

  13. [21]

    Taku Kudo. 2018. https://doi.org/10.18653/v1/P18-1007 Subword regularization: Improving neural network translation models with multiple subword candidates . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page...

    2018 doi

  14. [22]

    Tao Lei. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.602 When attention meets fast recurrence: Training language models with reduced compute . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7633--7648, Online and Punta Ca...

    2021 doi

  15. [23]

    Wang, Hui Dai, and Yoav Artzi

    Tao Lei, Yu Zhang, Sida I. Wang, Hui Dai, and Yoav Artzi. 2018. https://doi.org/10.18653/v1/D18-1477 Simple recurrent units for highly parallelizable recurrence . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4470--4481, Bruss...

    2018 doi

  16. [24]

    Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021. https://proceedings.mlr.press/v139/liang21a.html Towards understanding and mitigating social biases in language models . In Proceedings of the 38th International Conference on Machine Learning, vo...

    2021

  17. [25]

    Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.463 Understanding the difficulty of training transformers . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),...

    2020 doi

  18. [26]

    Ilya Loshchilov and Frank Hutter. 2019. https://openreview.net/forum?id=Bkg6RiCqY7 Decoupled weight decay regularization . In International Conference on Learning Representations

    2019

  19. [27]

    Swetha Mandava, Szymon Migacz, and Alex Fit - Florea. 2020. http://arxiv.org/abs/2009.04534 Pay attention when required . CoRR, abs/2009.04534

    2020 arXiv

  20. [28]

    Stephen Merity. 2019. http://arxiv.org/abs/1911.11423 Single headed attention RNN: stop thinking with your head . CoRR, abs/1911.11423

    2019 arXiv

  21. [29]

    Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018 a . http://arxiv.org/abs/1803.08240 An analysis of neural language modeling at multiple scales . CoRR, abs/1803.08240

    2018 arXiv

  22. [30]

    Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018 b . https://openreview.net/forum?id=SyyGPP0TZ Regularizing and optimizing LSTM language models . In International Conference on Learning Representations

    2018

  23. [31]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. https://openreview.net/forum?id=Byj72udxe Pointer sentinel mixture models . In International Conference on Learning Representations

    2017

  24. [32]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  25. [33]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  26. [34]

    Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H

    Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendric...

    2021 arXiv

  27. [35]

    Rae, Anna Potapenko, Siddhant M

    Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. https://openreview.net/forum?id=SylKikSYDH Compressive transformers for long-range sequence modelling . In International Conference on Learning Representations

    2020

  28. [36]

    Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2021. https://doi.org/10.1162/tacl_a_00353 Efficient content-based sparse attention with routing transformers . Transactions of the Association for Computational Linguistics, 9:53--68

    2021 doi

  29. [37]

    Rogers, Inna Goncearenco, Giuseppe Sarli, Igor Galynker, Denis Peskoff, Marine Carpuat, Jules White, Shyamal Anadkat, Alexander Hoyle, and Philip Resnik

    Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, Pranav Sandeep Dulepet, Saurav Vidyadhara, Dayeon Ki, Sweta Agrawal, Chau Pham, Gerson Kroiz, Feileen Li, Hudson Tao, Ashay S...

    2024 arXiv

  30. [38]

    Smith, and Oren Etzioni

    Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2020. https://doi.org/10.1145/3381831 Green AI . Commun. ACM, 63(12):54–63

    2020 doi

  31. [39]

    Huajie Shao, Jun Wang, Haohong Lin, Xuezhou Zhang, Aston Zhang, Heng Ji, and Tarek Abdelzaher. 2021. https://doi.org/10.1145/3442381.3449838 Controllable and diverse text generation in e-commerce . In Proceedings of the Web Conference 2021, WWW '21, page 2392–2401, New York, N...

    2021

  32. [40]

    Leslie N. Smith. 2018. http://arxiv.org/abs/1803.09820 A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decay . CoRR, abs/1803.09820

    2018 arXiv

  33. [41]

    David So, Wojciech Ma\' n ke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V Le. 2021. https://proceedings.neurips.cc/paper/2021/file/2f3c6a4cd8af177f6456e7e51a916ff3-Paper.pdf Searching for efficient transformers for language modeling

    2021

  34. [42]

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. http://jmlr.org/papers/v15/srivastava14a.html Dropout: A simple way to prevent neural networks from overfitting . Journal of Machine Learning Research, 15(56):1929--1958

    2014

  35. [43]

    Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019. https://doi.org/10.18653/v1/P19-1032 Adaptive attention span in transformers . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 331--335, Florence, ...

    2019 doi

  36. [44]

    Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. https://doi.org/10.18653/v1/P19-1452 BERT rediscovers the classical NLP pipeline . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593--4601, Florence, Italy. Association for ...

    2019 doi

  37. [45]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Proceedings of the 31st Internat...

    2017

  38. [46]

    Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. https://proceedings.neurips.cc/paper/2020/file/92650b2e92217715fe312e6fa7b90d82-Paper.pdf Investigating gender bias in language models using causal mediation anal...

    2020

  39. [47]

    Li Wan, Matthew Zeiler, Sixin Zhang, Yann LeCun, and Rob Fergus. 2013. https://dl.acm.org/doi/10.5555/3042817.3043055 Regularization of neural networks using dropconnect . In Proceedings of the 30th International Conference on International Conference on Machine Learning - Vol...

    2013

  40. [48]

    Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. 2019. https://proceedings.neurips.cc/paper/2019/file/2f4fe03d77724a7217006e5d16728874-Paper.pdf Understanding and improving layer normalization . In Advances in Neural Information Processing Systems, volume ...

    2019

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.