Pith. sign in

REVIEW 3 major objections 5 minor 37 references

Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Dense local dependencies in text force self-attention to inflate its pre-softmax logits as sequence length grows, and this logit explosion is what destabilizes long-sequence transformer training under low-precision arithmetic; explicitly…

desk verdict A plausible but unproven mechanism for attention-logit growth; solid synthetic evidence and a practical fix, but the density premise is never tested on real attention. read the letter →

arxiv 2505.15548 v2 pith:3NWEWSSI submitted 2025-05-21 cs.LG

Suvadeep Hajra This is my paper · ORCID
classification cs.LG
keywords attentionlogitexplosiontraininginstabilitydenselocaldependencieslow-ranklong-sequencetransformersLS-attentionmixed-precision
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that attention-logit explosion during long-sequence transformer training is not primarily an optimizer or precision artifact but a representational failure: self-attention's parameterization $QK^T$ carries at most rank $d$, so when the ideal attention matrix is a dense banded matrix with effective rank $n$, the model can approximate it only by inflating the pre-softmax logits. The paper supports this with a synthetic task where a $2500\times 2500$ banded attention matrix must be recovered, showing that global attention's logits rise sharply while local attention stays stable, and with language-model experiments on PG19 in BF16 where longer sequences destabilize Flash-attention unless local heads are added. The proposed remedy, Long Short-attention (LS-attention), keeps one global head for long-range dependencies and makes the remaining heads local, suppressing the maximum logit to below one-twentieth of vanilla attention and training stably at sequence length 8192. If correct, the result makes explicitly modeling dense local dependencies a design principle for stable, efficient long-context transformers.

What carries the argument

The central object is the rank and degree-of-freedom mismatch between the target attention matrix $P$ and its implementation as $\operatorname{softmax}(QK^T/\sqrt{d_k})$: dense local dependencies demand an effectively rank-$n$ banded $P$, while $QK^T$ is at most rank $d$ (or $d_k$). The proposed mechanism is LS-attention, which assumes the full attention matrix approximately decomposes as a sum of local banded matrices $P_{S_i}$ with span $p\ll n$ plus global low-rank matrices $P_{L_j}$; local heads evaluate only a sliding window of $O(np)$ entries, and a small number of global heads retain long-range capacity, so the representation effort matches the structure of the task.

What would settle it

Train an autoregressive transformer in BF16 on a corpus where dependencies are deliberately sparse—for example, a synthetic language in which each token depends only on a single rare anchor token—so the ideal attention matrix is low-rank, and record the maximum pre-softmax logit. If logit explosion and loss divergence still occur at $n\gg d$, the dense-local-dependency mechanism is not the necessary cause; if they disappear, the paper's central claim is supported.

Watch

Extended reading notes

Core claim

The central claim is that dense local dependency patterns in text induce an effectively high-rank attention matrix $P$—in a causal next-token task where each token attends to the previous $l$ neighbors, the ideal pattern satisfies $P[i,j]>0$ for $0\le i-j\le l$, a banded matrix whose rank grows linearly with $n$. Since $P=\operatorname{softmax}(QK^T)$, the attention matrix is generated by $Q,K\in\mathbb{R}^{n\times d}$, giving the parameterization at most $nd$ degrees of freedom, far fewer than the $O(n^2)$ entries of a dense band when $n\gg d$. The paper argues that the model compensates for this mismatch by increasing the magnitude of the logits $QK^T$, which eventually destabilizes low-precision (BF16) training. It demonstrates the mechanism on a synthetic dense-local-dependency task and on PG19 language modeling, then shows that decomposing attention into local heads, which need only $O(nl')$ degrees of freedom, plus a low-rank global head removes the logit explosion and stabilizes training at long sequence lengths.

Load-bearing premise

The load-bearing premise is that real language-model attention is dominated by dense local dependencies, so the ideal attention matrix is an effectively full-rank banded matrix; if actual attention is instead dominated by a few keyword tokens or by another low-rank structure, the rank-mismatch mechanism would not trigger and the observed instability would need a different explanation.

Editorial extensions

If this is right

  • Attention-logit growth should increase with sequence length $n$ and weaken when the per-head key/query dimension $d_k$ increases, because both move the rank capacity of $QK^T$.
  • Adding dedicated local heads removes the primary driver of logit explosion: in the paper's runs, LS-attention keeps the maximum logit below one-twentieth of vanilla MHSA and trains stably at $n=8192$, where Flash-attention diverges.
  • The global heads can be few and low-rank: one global head plus five local heads reaches about 36 perplexity on PG19, whereas all-local attention reaches only about 42, showing that long-range capacity and stability do not conflict.
  • LS-attention reduces inference time by up to 36% at sequence length 8192, and the saving should approach the head-count ratio $H/l$ for longer sequences because local heads add near-linear cost.
  • Stabilization methods that merely normalize logits can stop divergence without fixing the mismatch: QK-norm in the paper lands at 112.17 perplexity, over 2.5 times worse than LS-attention after the same steps, while full-precision training reaches comparable quality only after more than 20 times the GPU hours.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test a scaling law, but its degree-of-freedom argument predicts that the sequence length at which logit growth becomes steep should scale roughly with the attention dimension $d_k$; a model with $d_k=128$ should tolerate about twice the sequence length of one with $d_k=64$ before explosion.
  • Because the mechanism is about the structure of the attention matrix rather than language specifically, the same instability should appear in other domains with dense banded dependencies, such as raw audio or genomic sequences; the LS-attention decomposition would be a natural transfer but the paper leaves this untested.
  • The synthetic task fixes the band density at Bernoulli probability 0.5; varying the density continuously would directly test the paper's claim that density, not locality alone, drives logit growth, and would predict that sparse bands cause no explosion even for large window sizes.
  • The local span in the paper is fixed (50 or 100) regardless of sequence length; if real text contains multi-scale local structure, a hierarchical or multi-span local design could preserve stability while capturing longer local context, an extension the author does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that dense local dependencies in autoregressive language modeling produce an effectively high-rank attention target matrix that the low-rank parameterization of self-attention logits cannot faithfully represent, and that the optimizer compensates by inflating pre-softmax logits as sequence length grows, causing training instability under low-precision arithmetic. It proposes LS-attention, which splits attention into local and global heads, and reports synthetic experiments plus PG19 training runs showing that LS-attention mitigates logit explosion and training instability while also reducing inference time.

Significance. If the proposed mechanism were established, it would provide a concrete, design-relevant explanation for long-sequence training instability and a simple architectural remedy. The paper is empirically grounded: it contains a synthetic controlled task, real-language-model experiments on PG19, comparisons with QK-normalization and full-FP32 training, and an inference-time comparison. The main limitation is that the theoretical core—rank mismatch forcing logit inflation—is not actually proven, and the central premise that real language-model attention is dominated by dense local dependencies is never directly tested. Those gaps make the current version a promising but not yet fully supported contribution.

major comments (3)
  1. [Section 3, first two paragraphs] The rank argument is not sound as written. The paper states that because P is derived from QK^T, its degrees of freedom are bounded by nd and hence a rank-n target P′ cannot be represented by a low-rank QK^T. However, P = softmax(QK^T) is obtained by a nonlinear row-wise map, and the rank of softmax(S) is not bounded by the rank of S. For example, a rank-2 S can already produce a full-rank 2×2 softmax output. Therefore the observation that P′ has rank n does not by itself imply that a low-rank logit matrix cannot approximate it; the manuscript needs a direct approximation-theoretic bound on softmax(QK^T) or a different capacity argument.
  2. [Abstract and Section 3.1] The abstract claims that logit growth is driven by the density of local dependencies rather than by locality alone, but no experiment varies density. The synthetic task fixes the Bernoulli probability at p = 0.5 and never sweeps it, and the PG19 experiments compare LS-attention (local plus one global head) against vanilla MHSA, changing both attention span and the number of global heads simultaneously. The paper also never measures the actual density or effective rank of learned attention matrices on real language data. The premise that real LM attention is dominated by dense local dependencies therefore remains unsupported, and the central causal claim is not isolated from other confounds.
  3. [Section 3 and Figure 2] The step from 'a low-rank QK^T cannot exactly represent P′' to 'the optimizer inflates the logits' is asserted rather than derived. In a softmax approximation problem, the optimizer could in principle settle for a bounded approximation error without growing logits; the manuscript does not supply a theorem or a controlled experiment showing that logit inflation is the forced response. The synthetic experiment only reports training curves and max logits for one density setting, and it does not state the loss function, learning rate, initialization, or whether the reported logit growth is robust to these choices.
minor comments (5)
  1. [Metadata vs. full text] The arXiv title 'Dense Local Dependencies Induce Attention-Logit Explosion...' differs from the title in the full text, 'Short-Range Dependency Effects on Transformer Instability and a Decomposed Attention Solution'; the final version should use one consistent title.
  2. [Section 5.2] The text says the logit curves for n = 128, 512, 2048 are plotted in Figure 4c, but the data appear in Figure 3c; Figure 4c instead compares Flash-attention and LS-attention.
  3. [Section 3.1] The synthetic task does not specify the loss being minimized (e.g., mean squared error between O and PV, or cross-entropy over attention rows) or the optimizer hyperparameters; adding these details is important for reproducibility.
  4. [Throughout] There are several typos and formatting issues, including 'dependensy' (Section 3), 'ingore' (footnote 1), 'explotion' (Figure 4 captions), 'upto' (Section 1), 'larger than2.5' (Section 5.3), and the table entries '0 .52' and '2 .11' in Table 1.
  5. [Figure 2d] The local-attention logit curve also increases over training, though less sharply than the global curve; the paper should briefly explain why this residual growth is consistent with the claim that local attention avoids logit explosion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central mechanism is an analytical DOF argument plus a controlled synthetic probe; the dense-local premise is an untested assumption but not a circular step.

full rationale

No circularity found. The core claim in Section 3 is an analytical argument from the degree-of-freedom count of softmax(QK^T) to the difficulty of representing a banded, effectively full-rank target attention matrix P, followed by a synthetic probe in Section 3.1 that constructs a banded P and trains global and local attention to match it. The synthetic task is a controlled probe: the target matrix is fixed ex ante and is not derived from the trained logits, so observing logit growth in global attention is evidence rather than a consequence of construction. The PG-19 experiments compare LS-attention with vanilla MHSA and two external stabilization baselines; the results are external empirical comparisons, not fitted values renamed as predictions. There are no self-citations, no uniqueness theorems imported from the authors' prior work, and no ansatz smuggled in via citation. The paper's own Limitations section concedes that the mechanism applies mainly when n >> d and when dense local dependencies are present; whether real language-model attention is indeed dominated by dense local dependencies is an untested causal premise, which is a correctness or validity risk rather than circularity. No equation is shown to reduce to itself, and no fitted parameter is reused as a prediction. Score 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

LS-attention is an architectural modification, not a newly postulated physical or mathematical entity, so the invented-entities ledger is empty. The free parameters are hand-set experimental choices, and the axioms are the mathematical facts and domain assumptions on which the causal explanation rests.

free parameters (2)
  • local_attention_span = 50 tokens for n <= 2048; 100 tokens for longer sequences
    Hand-set hyperparameter of LS-attention; the paper does not sweep it, and the stability and speed benefits depend on this choice.
  • synthetic_band_density = 0.5 (Bernoulli probability)
    The synthetic target matrix fills each allowed band entry with probability 0.5, defining the density of local dependencies; density is never varied even though the abstract claims density is the driver.
assumptions (4)
  • standard math The ideal banded attention matrix P' for dense local dependencies has rank n.
    Section 3: P' is an n by n matrix with nonzero entries only in a band of width l, and the paper asserts it has rank n, so O(nd) parameters of QK^T are insufficient.
  • domain assumption Real language modeling attention is dominated by dense local dependencies.
    Section 3 and 5.2: the explanation assumes nearly every token relies on neighboring tokens, making the ideal attention matrix effectively full-rank and banded.
  • domain assumption Long-range attention patterns are low-rank and the overall attention matrix decomposes into a sum of local and low-rank global components.
    Section 4: LS-attention assumes P is approximately a sum of local bands plus low-rank global matrices, motivated by keyword tokens; no direct evidence for this decomposition is provided.
  • ad hoc to paper When QK^T cannot represent the target pattern, the optimizer responds by inflating the logits.
    Section 3: 'forces the model to compensate by inflating the logits S' is asserted as an optimization fact; no theorem or controlled experiment isolates this response from other possible failure modes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training." pith.science (2026). https://pith.science/paper/3NWEWSSI

@misc{pith2026250515548,
  author       = {Pith},
  title        = {Pith review of: Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3NWEWSSI}},
  note         = {Machine review of arXiv:2505.15548}
}
read the original abstract

Autoregressive transformer language models frequently exhibit training instability when trained on long sequences, particularly under low-precision arithmetic. Although this instability is often accompanied by attention-logit explosion, its underlying cause remains poorly understood. In this work, we present analytical insights and empirical evidence that dense local dependencies are a major contributor to attention-logit explosion. We demonstrate that dense local dependency patterns yield an effectively high-rank attention structure, which the low-rank parameterization of self-attention can only approximate with increasingly large logits as the sequence length grows. This logit inflation ultimately leads to training instability under low-precision arithmetic. We support this explanation through extensive experiments on synthetic and language modeling tasks. Our results consistently show that attention-logit growth increases with sequence length, is mitigated by increasing the attention dimension, and is substantially reduced by explicitly modeling dense local dependencies. Furthermore, we show that this growth is driven by the density of local dependencies rather than by locality alone. More broadly, our findings suggest that explicitly modeling dense local dependencies constitutes an important design principle for developing stable, efficient, and scalable long-context transformer architectures for autoregressive language modeling.

Figures

Figures reproduced from arXiv: 2505.15548 by the authors.

Figure 1
Figure 1. Mitigation of training instability and logit ex [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of representing dense local dependencies by local and global attention. (a) Global attention [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Training instability and logit explosion in Flash-attention at longer sequence lengths. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Mitigation of logit explotion and training instability using LS-attention. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Performance comparison of LS-attention (in mixed BF16) with two alternatives: (1) Flash-attention [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Impact of global attention head in LS￾attention. performance. LS-attention with a global head achieves a validation perplexity of around 36, while the variant with no global head reaches only around 42 perplexity after the first 50K training steps. The results suggest …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 25 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Joshua Ainslie, James Lee - Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr \' o n, and Sumit Sanghai. 2023. GQA: training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023 , pages 4895--4901. ...

    work page 2023

  4. [4]

    Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, and Furu Wei. 2022. Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL...

    2022

  5. [5]

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual

    work page 2020

  6. [6]

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. 2022. Maskgit: Masked generative image transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 11305--11315. IEEE

    work page 2022

  7. [7]

    Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. 2020. Generative pretraining from pixels. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research, pages 1691--1703. PMLR

    work page 2020

  8. [8]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, and 48 others. 2023. Palm: Scaling language modeling with pa...

    work page 2023

Show all 37 references
  1. [9]

    Tri Dao. 2024. Flash A ttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR)

    2024

  2. [10]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R \' e

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R \' e . 2022. Flash A ttention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2...

    2022

  3. [11]

    Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme Ruiz, Matth...

    2023

  4. [12]

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hum...

    2019

  5. [13]

    Alicia Golden, Samuel Hsia, Fei Sun, Bilge Acun, Basil Hosmer, Yejin Lee, Zachary DeVito, Jeff Johnson, Gu - Yeon Wei, David Brooks, and Carole - Jean Wu. 2024. Is flash attention stable? CoRR, abs/2405.02803

    2024 arXiv

  6. [14]

    Anmol Gulati, James Qin, Chung - Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. 2020. Conformer: Convolution-augmented transformer for speech recognition. In 21st Annual Conference of the International Speech C...

    2020

  7. [15]

    Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. 2020. Query-key normalization for transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020 , volume EMNLP 2020 of Findings of ACL , pa...

    2020

  8. [16]

    Wei - Ning Hsu, Benjamin Bolte, Yao - Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE ACM Trans. Audio Speech Lang. Process. , 29:3451--3460

    2021

  9. [17]

    Akhil Kedia, Mohd Abbas Zaidi, Sushil Khyalia, Jungho Jung, Harshith Goka, and Haejun Lee. 2024. Transformers get stable: An end-to-end signal propagation theory for language models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July ...

    2024

  10. [18]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings

    2015

  11. [19]

    Conglong Li, Minjia Zhang, and Yuxiong He. 2021. Curriculum learning: A regularization method for efficient and stable billion-scale GPT model pre-training. CoRR, abs/2108.06084

    2021 arXiv

  12. [20]

    Conglong Li, Minjia Zhang, and Yuxiong He. 2022. The stability-efficiency dilemma: Investigating sequence length warmup for training GPT models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 20...

    2022

  13. [21]

    Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. 2020. Understanding the difficulty of training transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020 , pages 5747--5...

    2020

  14. [22]

    Igor Molybog, Peter Albert, Moya Chen, Zachary DeVito, David Esiobu, Naman Goyal, Punit Singh Koura, Sharan Narang, Andrew Poulton, Ruan Silva, Binh Tang, Diana Liskovich, Puxin Xu, Yuchen Zhang, Melanie Kambadur, Stephen Roller, and Susan Zhang. 2023. A theory on adam instabi...

    2023 arXiv

  15. [23]

    Kosuke Nishida, Kyosuke Nishida, and Kuniko Saito. 2024. Initialization of large language models via reparameterization to mitigate loss spikes. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-1...

    2024

  16. [24]

    Vittorio Pippi, Fabio Quattrini, Silvia Cascianelli, Alessio Tonioni, and Rita Cucchiara. 2025. Zero-shot styled text image generation, but make it autoregressive. CoRR, abs/2503.17074

    2025 arXiv

  17. [25]

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently scaling transformer inference. In Proceedings of the Sixth Conference on Machine Learning and Systems, MLSys 2023, Miami...

    2023

  18. [26]

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. https://openai.com/research/language-unsupervised Improving language understanding by generative pre-training . OpenAI Blog

    2018

  19. [27]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. https://openai.com/research/language-unsupervised Language models are unsupervised multitask learners . OpenAI Blog

    2019

  20. [28]

    Rae, Anna Potapenko, Siddhant M

    Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . Ope...

    2020

  21. [29]

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. 2024. Flash A ttention-3: Fast and accurate attention with asynchrony and low-precision. CoRR, abs/2407.08608

    2024 arXiv

  22. [30]

    Noam Shazeer. 2019. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150

    2019 arXiv

  23. [31]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Annual Conference on Neural Information Processing Systems 2017, USA , pages 5998--6008

    2017

  24. [32]

    Guoxia Wang, Shuai Li, Congliang Chen, Jinle Zeng, Jiabin Yang, Tao Sun, Yanjun Ma, Dianhai Yu, and Li Shen. 2025. Adagc: Improving training stability for large language model pretraining. CoRR, abs/2502.11034

    2025 arXiv

  25. [33]

    Liu, Lechao Xiao, Katie E

    Mitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie E. Everett, Alexander A. Alemi, Ben Adlam, John D. Co - Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl - Dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. 2024. Small-sc...

    2024

  26. [34]

    Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao

    Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. 2022. Tensor programs V: tuning large neural networks via zero-shot hyperparameter transfer. CoRR, abs/2203.03466

    2022 arXiv

  27. [35]

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. 2022. Scaling autoregressive models for conte...

    2022

  28. [36]

    Susskind

    Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M. Susskind. 2023. Stabilizing transformer training by preventing attention entropy collapse. In International Conference on Machine Learning, ICML 2023, 23-2...

    2023

  29. [37]

    Barrett, Zhangyang Wang, and Beidi Chen

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R \' e , Clark W. Barrett, Zhangyang Wang, and Beidi Chen. 2023. H2O: heavy-hitter oracle for efficient generative inference of large language models. In Advan...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.