Pith. sign in

REVIEW 4 major objections 7 minor 57 references

Taming Transformer Without Using Learning Rate Warmup

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Spectral energy concentration in the query-key weight product is what crashes Transformer training without warmup, and a learning-rate cap derived from Weyl's inequality prevents it.

desk verdict A useful warmup-free optimizer with an unproven mechanism—worth refereeing for the method, not for the theory. read the letter →

arxiv 2505.21910 v1 pith:I53ZPSLA submitted 2025-05-28 cs.LG cs.CV

Xianbiao Qi , Yelin He , Jiaquan Ye , Chun-Guang Li , Bojia Zi , Xili Dai , Qin Zou , Rong Xiao This is my paper
classification cs.LGcs.CV
keywords spectralenergyconcentrationmalignantentropycollapselearningratewarmupAdamW2Weyl'sinequalitytransformertrainingstabilityself-attentionsingularvalues
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims to explain why Transformer training diverges when learning-rate warmup is omitted, and to remove the need for warmup altogether. The culprit, it argues, is spectral energy concentration (SEC): the top few singular values of the query-key product $W_q^\top W_k$ come to dominate its entire spectrum, so the attention map becomes sparse and low-rank at the same time, a state the authors call malignant entropy collapse. The proposed remedy, AdamW2, uses Weyl's inequality to cap the learning rate for each weight matrix at a small multiple of the ratio between the current spectral norm of the weights and the spectral norm of the update. With that cap, singular values grow smoothly instead of concentrating, and ViT, Swin-Transformer, and GPT models train stably with no warmup and no architectural change.

What carries the argument

The load-bearing objects are the SEC index, the Jacobian identity for self-attention, and Weyl's inequality. The SEC index $\mathrm{SEC}(d_q,s)=\sum_{i=1}^s \sigma_i^2(W_q^\top W_k)/\sum_{i=1}^{d_q} \sigma_i^2(W_q^\top W_k)$ quantifies how much of the query-key product's energy sits in its top $s$ singular directions. The paper computes $\partial \mathrm{vec}(P)/\partial \mathrm{vec}(W_q^\top W_k)=X^\top \otimes X^\top$ with $P=X^\top W_q^\top W_k X$, so a low-rank input $X$ makes the gradient of the query-key product low-rank and spectrally concentrated, driving $W_q^\top W_k$ toward low rank. Theorem 1 then turns low-rank-plus-large-singular-values of $W_q^\top W_k$ into a sparse, low-rank attention map. Finally, Weyl's inequality $\sigma_1(W_1+W_2)\le\sigma_1(W_1)+\sigma_1(W_2)$ supplies the cap $\alpha_t\le\tau\sigma_1(W_{t-1})/\sigma_1(\nabla W_t)$ that keeps successive weight updates smooth.

What would settle it

Train a small GPT with AdamW2 and log, at every step, the SEC index $\mathrm{SEC}(d_q,4)$ and the numerical rank of each attention map. The central claim predicts that every crash is preceded by the index crossing a high threshold (for instance, more than 99 percent of energy in four directions) and attention rank dropping to a small value; a single crash without that precursor, or a stable run in which the index stays high, would refute the claim.

Watch

Extended reading notes

Core claim

The central claim is that the object that breaks Transformer training is the product $W_q^\top W_k$: when its spectral energy concentrates on a few directions and its largest singular value is large, the logit matrix $X^\top W_q^\top W_k X$ becomes effectively low-rank, softmax makes the attention map both sparse and low-rank, and the block output collapses. This malignant entropy collapse leaves most tokens with negligible gradient and sends the loss to divergence. The paper's Theorem 1 states that if the input features $X$ are low-rank and $W_q^\top W_k$ is low-rank with dominant singular values above $C_0\sqrt{d_q}$, then the attention map is sparse and low-rank in high probability. AdamW2 is then proposed as the cure: whenever the scheduled learning rate would push $\sigma_1(W_{t-1}-\alpha_t\nabla W_t)$ above $(1+\tau)\sigma_1(W_{t-1})$, it truncates $\alpha_t$ to $\tau \sigma_1(W_{t-1})/\sigma_1(\nabla W_t)$, ensuring that the spectral norm of every weight matrix can grow by at most a factor $(1+\tau)$ per step. Experiments on ViT-B/L/g, Swin-S/B, GPT-2-scale, and Flatten-Swin show this rule gives accuracy or loss essentially matching AdamW with warmup, and outperforms AdamW without warmup, which crashes.

Load-bearing premise

The proof of the crash mechanism assumes that the softmax nonlinearity does not significantly change the rank of the logit matrix, and it draws a high-probability conclusion without specifying a probability model; if softmax can raise the rank, the claimed link from spectral concentration to a low-rank attention map is not established.

Editorial extensions

If this is right

  • AdamW2 is a drop-in replacement for learning-rate warmup: the same cosine schedule plus the spectral clipping rule trains ViT-B/L, Swin-S/B, GPT-2-scale, and 1B-parameter ViT-g models to accuracy or loss essentially matching AdamW with warmup.
  • The clipping rule acts as an automatic per-block schedule: actual learning rates are reduced mainly in the early phase and mainly in shallow layers, then coincide with the default schedule later.
  • Low attention entropy alone is not a crash signal; the dangerous state is low entropy combined with low rank, so the SEC index gives a sharper diagnostic than entropy collapse alone.
  • Any intervention that keeps the spectrum of $W_q^\top W_k$ diffuse, not just learning-rate clipping, should also stabilize warmup-free training if the causal story is correct.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension suggested by this logic is to regularize the SEC index directly, for example by penalizing the energy ratio of the top few singular values of $W_q^\top W_k$, and to compare that against AdamW2's clipping rule.
  • The theorem's unquantified high-probability claim and its rank-preservation assumption for softmax leave room for a sharper statement; a proof that bounds the spectral norm of the softmax Jacobian would put the causal chain from SEC to malignant collapse on firmer ground.
  • Because Rule 1 uses only spectral norms and applies per weight matrix, the same mechanism could transfer to other architectures with exploding-spectrum failure modes, such as very deep residual networks, though the paper does not test that.
  • The recorded per-layer learning-rate curves suggest that warmup is a global proxy for what AdamW2 does explicitly and locally; if so, warmup length could be tuned more cheaply using the SEC index rather than by trial and error.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper studies why Transformer training diverges without learning-rate warmup and proposes an optimizer-level remedy. It first monitors 15 quantities during failed and successful training of ViT and GPT, observing that failures coincide with a rapid growth of sigma_1(W_q^T W_k) and an attention map that becomes simultaneously sparse and low-rank (called malignant entropy collapse). The authors introduce a spectral energy concentration (SEC) index, state Theorem 1 claiming that low-rank X and low-rank W with dominant singular values imply a sparse and low-rank attention map with high probability, and then derive from Weyl's inequality a bound on the learning rate: if alpha_t sigma_1(nabla W_t) / sigma_1(W_{t-1}) > tau, the learning rate is clipped to tau sigma_1(W_{t-1}) / sigma_1(nabla W_t). This rule defines the AdamW2 optimizer. Experiments on ViT-B/L, Swin-S/B, GPT-S, a 1B ViT, a 774M nanoGPT, and Flatten-Swin report that AdamW2 without warmup matches or slightly exceeds baselines that use warmup, with an ablation showing mild sensitivity to tau.

Significance. If the mechanistic claim were established, the paper would make a significant conceptual contribution: it would explain warmup as a response to spectral energy concentration in the query-key product and offer a simple, architecture-agnostic optimizer fix. The empirical contribution is credible and useful: AdamW2 is a very simple modification, requires no architecture change, and is tested on several model families at scales up to 1B parameters, with consistent parity to warmup baselines and an ablation over tau. The derivation of the clipping rule from Weyl's inequality is transparent and the rule is parameter-free apart from tau. However, the central theoretical claim that SEC of W_q^T W_k is the cause of model crash rests entirely on Theorem 1, whose proof in Appendix D contains unsupported assumptions; as written, the causal link is not established.

major comments (4)
  1. [Appendix D, Step 2 (proof of Theorem 1)] The claim that 'the softmax function does not significantly change the rank of the matrix' and that 'the softmax operation only introduces nonlinearities that do not increase the rank' is false. Softmax is a nonlinear map and generally increases rank. For example, the rank-1 logit matrix [[1,2],[2,4]] has row 2 equal to twice row 1, but its column-wise softmax is [[1/(1+e), 1/(1+e^2)], [e/(1+e), e^2/(1+e^2)]], whose two columns are not proportional, so the softmax output has rank 2. Consequently, the low-rankness of A does not follow from the low-rankness of P as stated.
  2. [Appendix D, Step 1 (proof of Theorem 1)] The argument that low-rank P with a few dominant singular values implies softmax concentrates probability mass on a few entries per column is not valid as stated. For instance, P = c * 11^T is rank 1 with a single large singular value, yet its softmax is the uniform matrix, not a sparse matrix. The theorem gives no condition that prevents all entries of P from being comparable, and the conclusion 'in high probability' is unfalsifiable because no probability model over X and W is specified. These gaps are load-bearing: without a valid step from low-rank dominant-singular-value P to sparse A, Theorem 1 does not establish the claimed SEC-to-crash mechanism.
  3. [Sections 3.3, 5, and Abstract] The paper's central claim that spectral energy concentration of W_q^T W_k is 'the reason' for malignant entropy collapse and model crash depends entirely on Theorem 1. Since Theorem 1 is not rigorously proven, the observed correlations in Figures 1, 4, 8, and 10 could be epiphenomenal, and AdamW2 might succeed simply because it is a conservative learning-rate controller that prevents divergence for reasons unrelated to SEC. The authors should either supply a corrected proof with explicit probabilistic or deterministic conditions under which low-rank dominant-singular-value P yields sparse and low-rank softmax output, or explicitly reframe the SEC mechanism as an empirical hypothesis and remove the causal phrasing from the abstract and conclusion.
  4. [Appendix C (proof of benign entropy collapse)] The proof of benign collapse assumes that W is a real symmetric positive semi-definite matrix and that the entries of x are iid standard Gaussian, whereas the paper's stated setting for benign collapse is a non-symmetric positive quasi-definite W (Appendix O). The transition from the symmetric PSD assumption to the non-symmetric case is not justified. Moreover, the propositions only compare the expectations E[x_i^T W x_i] and E[x_i^T W x_j]; without concentration bounds or variance estimates, this does not imply that softmax outputs are close to an identity matrix with high probability. This weakens the claimed distinction between benign and malignant collapse.
minor comments (7)
  1. [Abstract] There is a typo: 'learning rate warump' should be 'learning rate warmup'.
  2. [Figures 1, 8, 10 and captions] Several axis labels and captions have garbled or missing symbols, for example 'Wq Wk' instead of 'W_q^T W_k' and labels such as '|| 1||2' instead of '||gamma_1||_2'; these should be cleaned up for readability.
  3. [Section 3.3] The phrase 'rigid theoritical analysis' contains two typos ('rigid' for 'rigorous', 'theoritical' for 'theoretical').
  4. [Algorithm 1 and Section 3.4] In Algorithm 1, line 5 defines the update direction as the Adam-normalized quantity (M_t hat divided by sqrt(V_t hat) + epsilon), but the surrounding text and Equation (5) use the same symbol nabla W_t for the raw gradient. This conflation should be clarified, since the spectral ratio in Rule 1 is applied to the preconditioned update, not to the raw gradient.
  5. [Appendix E] The proof of Weyl's inequality writes both SVDs as W_1 = U Sigma_1 V^T and W_2 = U Sigma_2 V^T with the same U and V and contains a duplicated 'V = [v_1,...,v_n], V = [v_1,...,v_n]' phrase; the notation should be corrected to use distinct singular vectors U_1, V_1 and U_2, V_2.
  6. [Appendix O] The term 'positive quasi-definite' is used in the main text (Section 3.3) but defined only in Appendix O; a brief definition or pointer in the main text would help.
  7. [Section 4 (experiments)] The paper does not report the additional computational overhead of the per-step power iterations for every parameter matrix, or a runtime comparison against standard AdamW; a sentence quantifying this cost would strengthen the practicality claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: AdamW2 follows from Weyl's inequality and the warmup-free results are independent empirical demonstrations.

full rationale

The paper's derivation chain is self-contained in the sense relevant to circularity. The optimizer rule (Rule 1 / Algorithm 1) is derived from Weyl's inequality as a sufficient condition: σ1(W_t) ≤ σ1(W_{t-1}) + α_t σ1(∇W_t), leading to α_t ≤ τ σ1(W_{t-1})/σ1(∇W_t), with τ a hand-set threshold rather than a parameter fitted to the target results. The warmup-free training results on ViT, Swin-Transformer, GPT, and the 1B ViT are out-of-sample empirical comparisons against AdamW baselines. The SEC index is a diagnostic defined on Wq^T Wk, while the algorithm bounds spectral-norm growth of the updated weights; this connection is mechanistically loose but not circular, because the rule is not obtained by fitting the SEC index and the success of AdamW2 is not logically entailed by the SEC claim. The paper's self-citations (Qi et al. 2023a,b) appear only as background or comparison and are not load-bearing. A genuine rigor concern exists: Theorem 1's proof in Appendix D assumes that softmax does not significantly increase rank and that low-rank logits concentrate softmax mass; these are unsupported and in general false premises, but they are correctness and rigor gaps rather than circular reductions, so they do not raise the circularity score.

Assumptions & free parameters 1 free parameters · 5 assumptions · 1 invented entities

The central derivation depends on standard linear algebra (Weyl, Kronecker rank), but the theorem connecting SEC to malignant collapse relies on an unproven rank-preservation assumption for softmax. The practical optimizer also has a hand-set threshold τ per model family. The benign collapse proof assumes symmetric positive definite W even though the actual matrix Wq^T Wk is non-symmetric, as the paper notes in Appendix O. No new physical entities are introduced; the SEC index is a diagnostic quantity.

free parameters (1)
  • τ (spectral-norm update threshold) = 0.004 (ViT, Swin; ViT also 0.003), 0.01 (GPT)
    Hyperparameter in Rule 1: clip α_t to τ σ1(W_{t-1})/σ1(∇W_t) when α_t σ1(∇W_t)/σ1(W_{t-1}) > τ. Chosen per model family; ablation in Fig. 6 shows robustness across a range but no principled selection rule is given.
assumptions (5)
  • standard math Weyl's inequality on singular values
    Used in Sec. 3.4 to derive σ1(W_t) ≤ σ1(W_{t-1}) + α_t σ1(∇W_t) and the clipping bound in Eq. 6.
  • standard math rank(X⊗X) = rank(X)^2
    Used in Sec. 3.2 and Appendix Q to argue that a low-rank X implies a low-rank Jacobian X^T⊗X^T.
  • ad hoc to paper Softmax approximately preserves matrix rank
    Appendix D Step 2: 'Assume that the softmax function does not significantly change the rank of the matrix.' This assumption is load-bearing for Theorem 1 and is not proven.
  • ad hoc to paper Wq^T Wk is positive quasi-definite, treated as symmetric PSD in the proof
    Appendix C proves benign collapse for a real symmetric positive semi-definite W, while Appendix O states the actual Wq^T Wk is non-symmetric positive quasi-definite; the gap is acknowledged but not bridged.
  • domain assumption Gaussian iid entries in feature vectors xi
    Appendix C, Props 3 and 4 assume each element of xi is iid N(0,1) to compute expectations; real features are not iid Gaussian.
invented entities (1)
  • Spectral Energy Concentration (SEC) index independent evidence
    purpose: Quantify how much of the squared spectral energy of Wq^T Wk sits in the top s singular values; used to diagnose malignant entropy collapse and to motivate LR clipping.
    Directly computable from model weights in any training run; Fig. 4 shows crashed runs concentrate energy in fewer than 10 directions while stable runs spread it, so the quantity makes falsifiable contact with experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Taming Transformer Without Using Learning Rate Warmup." pith.science (2026). https://pith.science/paper/I53ZPSLA

@misc{pith2026250521910,
  author       = {Pith},
  title        = {Pith review of: Taming Transformer Without Using Learning Rate Warmup},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I53ZPSLA}},
  note         = {Machine review of arXiv:2505.21910}
}
abstract

Scaling Transformer to a large scale without using some technical tricks such as learning rate warump and using an obviously lower learning rate is an extremely challenging task, and is increasingly gaining more attention. In this paper, we provide a theoretical analysis for the process of training Transformer and reveal the rationale behind the model crash phenomenon in the training process, termed \textit{spectral energy concentration} of ${\bW_q}^{\top} \bW_k$, which is the reason for a malignant entropy collapse, where ${\bW_q}$ and $\bW_k$ are the projection matrices for the query and the key in Transformer, respectively. To remedy this problem, motivated by \textit{Weyl's Inequality}, we present a novel optimization strategy, \ie, making the weight updating in successive steps smooth -- if the ratio $\frac{\sigma_{1}(\nabla \bW_t)}{\sigma_{1}(\bW_{t-1})}$ is larger than a threshold, we will automatically bound the learning rate to a weighted multiple of $\frac{\sigma_{1}(\bW_{t-1})}{\sigma_{1}(\nabla \bW_t)}$, where $\nabla \bW_t$ is the updating quantity in step $t$. Such an optimization strategy can prevent spectral energy concentration to only a few directions, and thus can avoid malignant entropy collapse which will trigger the model crash. We conduct extensive experiments using ViT, Swin-Transformer and GPT, showing that our optimization strategy can effectively and stably train these Transformers without using learning rate warmup.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 19 canonical work pages

  1. [1]

    Rezero is all you need: Fast convergence at large depth

    Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. Rezero is all you need: Fast convergence at large depth. In Uncertainty in Artificial Intelligence, pp.\ 1352--1361. PMLR, 2021

    work page 2021

  2. [2]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  3. [3]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

    2023

  4. [4]

    The road less scheduled

    Aaron Defazio, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, Ashok Cutkosky, et al. The road less scheduled. arXiv preprint arXiv:2405.15682, 2024

    arXiv 2024

  5. [5]

    Scaling vision transformers to 22 billion parameters

    Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pp.\ 7480--7512. PMLR, 2023

    2023

  6. [6]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.\ 248--255. Ieee, 2009

    2009

  7. [7]

    Attention is not all you need: Pure attention loses rank doubly exponentially with depth

    Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pp.\ 2793--2803. PMLR, 2021

    work page 2021

  8. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020

    2020

Show all 57 references
  1. [9]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  2. [10]

    Adaptive subgradient methods for online learning and stochastic optimization

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12 0 (7), 2011

    2011

  3. [11]

    Openwebtext corpus

    Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019

    2019

  4. [12]

    Matrix computations

    Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 2013

    2013

  5. [13]

    Kronecker products and matrix calculus with applications

    Alexander Graham. Kronecker products and matrix calculus with applications. Courier Dover Publications, 2018

    2018

  6. [14]

    Flatten transformer: Vision transformer using focused linear attention

    Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 5961--5971, 2023

    2023

  7. [15]

    Query-key normalization for transformers

    Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-key normalization for transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp.\ 4246--4253, 2020

    2020

  8. [16]

    Topics in matrix analysis, 1991

    Roger A Horn and Charles R Johnson. Topics in matrix analysis, 1991. Cambridge University Presss, Cambridge, 37: 0 39, 1991

    1991

  9. [17]

    Matrix analysis

    Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012

    2012

  10. [18]

    Andrej Karpathy. NanoGPT . https://github.com/karpathy/nanoGPT, 2022

    2022

  11. [19]

    The lipschitz constant of self-attention

    Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant of self-attention. In International Conference on Machine Learning, pp.\ 5562--5571. PMLR, 2021

    2021

  12. [20]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  13. [21]

    Rotational equilibrium: How weight decay balances learning across neural networks

    Atli Kosson, Bettina Messmer, and Martin Jaggi. Rotational equilibrium: How weight decay balances learning across neural networks. arXiv preprint arXiv:2305.17212, 2023

    2023 arXiv

  14. [22]

    Analyzing & reducing the need for learning rate warmup in gpt training

    Atli Kosson, Bettina Messmer, and Martin Jaggi. Analyzing & reducing the need for learning rate warmup in gpt training. Advances in Neural Information Processing Systems, 2024

    2024

  15. [23]

    Backpropagation applied to handwritten zip code recognition

    Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1 0 (4): 0 541--551, 1989

    1989

  16. [24]

    Gradient-based learning applied to document recognition

    Yann LeCun, L \'e on Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86 0 (11): 0 2278--2324, 1998

    1998

  17. [25]

    Efficient backprop

    Yann LeCun, L \'e on Bottou, Genevieve B Orr, and Klaus-Robert M \"u ller. Efficient backprop. In Neural networks: Tricks of the trade, pp.\ 9--50. Springer, 2002

    2002

  18. [26]

    Understanding the difficulty of training transformers

    Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.\ 5747--5763, 2020

    2020

  19. [27]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 10012--10022, 2021

    2021

  20. [28]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016

    2016 arXiv

  21. [29]

    Fixing weight decay regularization in adam

    Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. In International Conference on Learning Representations, 2019

    2019

  22. [30]

    Signal propagation in transformers: Theoretical perspectives and the role of rank collapse

    Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurelien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems, 35: 0 27198--27211, 2022

    2022

  23. [31]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4195--4205, 2023

    2023

  24. [32]

    The matrix cookbook

    Kaare Brandt Petersen, Michael Syskind Pedersen, et al. The matrix cookbook. Technical University of Denmark, 7 0 (15): 0 510, 2008

    2008

  25. [33]

    Lipsformer: Introducing lipschitz continuity to vision transformers

    Xianbiao Qi, Jianan Wang, Yihao Chen, Yukai Shi, and Lei Zhang. Lipsformer: Introducing lipschitz continuity to vision transformers. In The Eleventh International Conference on Learning Representations, 2023 a

    2023

  26. [34]

    Understanding optimization of deep learning via jacobian matrix and lipschitz constant

    Xianbiao Qi, Jianan Wang, and Lei Zhang. Understanding optimization of deep learning via jacobian matrix and lipschitz constant. arXiv preprint arXiv:2306.09338, 2023 b

    2023 arXiv

  27. [35]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018

    2018

  28. [36]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  29. [37]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp...

    2021

  30. [38]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp.\ 8821--8831. Pmlr, 2021

    2021

  31. [39]

    A stochastic approximation method

    Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp.\ 400--407, 1951

    1951

  32. [40]

    Learning representations by back-propagating errors

    David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature, 323 0 (6088): 0 533--536, 1986

    1986

  33. [41]

    Cyclical learning rates for training neural networks

    Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vision (WACV), pp.\ 464--472. IEEE, 2017

    2017

  34. [42]

    Scan and snap: Understanding training dynamics and token composition in 1-layer transformer

    Yuandong Tian, Yiping Wang, Beidi Chen, and Simon S Du. Scan and snap: Understanding training dynamics and token composition in 1-layer transformer. Advances in Neural Information Processing Systems, 36: 0 71911--71947, 2023 a

    2023

  35. [43]

    Joma: Demystifying multilayer transformers via joint dynamics of mlp and attention

    Yuandong Tian, Yiping Wang, Zhenyu Zhang, Beidi Chen, and Simon Du. Joma: Demystifying multilayer transformers via joint dynamics of mlp and attention. arXiv preprint arXiv:2310.00535, 2023 b

    2023 arXiv

  36. [44]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  37. [45]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  38. [46]

    High-dimensional probability: An introduction with applications in data science, volume 47

    Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018

    2018

  39. [47]

    Deepnet: Scaling transformers to 1,000 layers

    Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers. arXiv preprint arXiv:2203.00555, 2022

    2022 arXiv

  40. [48]

    Learning deep transformer models for machine translation

    Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, and Lidia S Chao. Learning deep transformer models for machine translation. arXiv preprint arXiv:1906.01787, 2019

    1906 arXiv

  41. [49]

    Pytorch image models

    Ross Wightman. Pytorch image models. https://github.com/rwightman/pytorch-image-models, 2019

    2019

  42. [50]

    High-dimensional data analysis with low-dimensional models: Principles, computation, and applications

    John Wright and Yi Ma. High-dimensional data analysis with low-dimensional models: Principles, computation, and applications. Cambridge University Press, 2022

    2022

  43. [51]

    On layer normalization in the transformer architecture

    Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pp.\ 10524--10533. PMLR, 2020

    2020

  44. [52]

    Stabilizing transformer training by preventing attention entropy collapse

    Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M Susskind. Stabilizing transformer training by preventing attention entropy collapse. In International Conference on Machine Learning, pp.\ 40770--40803. PMLR, 2023

    2023

  45. [53]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019

    2019

  46. [54]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  47. [55]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  48. [56]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  49. [57]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.