Pith. sign in

REVIEW 1 cited by

Why Do We Need Warm-up? A Theoretical Perspective

T0 review · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that learning-rate warm-up is a consequence of (H0,H1)-smoothness: when the squared Hessian norm is bounded by a linear function of the loss suboptimality, the step size that tracks this bound starts small and rises as the

arxiv 2510.03164 v2 pith:Q5M5YSZS submitted 2025-10-03 cs.LG math.OCstat.ML

Foivos Alimisis , Rustem Islamov , Aurelien Lucchi This is my paper
classification cs.LGmath.OCstat.ML
keywords learning-ratewarm-upadaptivestep-sizegeneralizedsmoothness(H0H1)-smoothnesslosssuboptimalityconvergencecomplexityboundsnon-convexoptimizationneuralnetworklandscape
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that learning-rate warm-up is not a heuristic patch but a consequence of a specific curvature property: the squared norm of the Hessian is bounded above by an affine function of the gap between the current loss and the minimum. Under that property, the step size that best tracks the local curvature naturally starts small and increases as the loss decreases—exactly the warm-up shape used in practice. The paper proves that gradient descent with this adaptive step size converges in fewer iterations than any constant step size whenever the initial loss gap is large relative to the desired accuracy, and it shows the curvature property holds for simple neural networks under balancedness or L2 regularization. A reader should care because this gives a convergence-based explanation for why warm-up helps, and it predicts when warm-up should be most beneficial: poor initialization or high-precision targets.

What carries the argument

The central object is (H0,H1)-smoothness: a function satisfies ||∇²f(w)||² ≤ H0 + H1(f(w)−f*), so curvature grows at most linearly with the loss gap. Unlike the older (L0,L1)-smoothness, which bounds curvature by a linear function of the gradient norm, the loss gap decays monotonically during training, so this bound converts directly into a rising step-size schedule. The matching step-size rule η_k = 1/(10H0+20H1(f(w_k)−f*)) is the mechanism: as suboptimality falls, step size rises—warm-up. The convergence proofs assume either the Aiming condition (gradient points toward the minimizer set with strength proportional to the loss gap) or the µ-PL condition (gradient norm squared at least 2µ tim

What would settle it

Measure along the first training iterations of a real network the ratio (||∇²f(w)||² − H0)/(f(w)−f*) using Hessian-vector products rather than finite differences of gradients. If this ratio grows without bound as the loss decreases, or if the estimated local smoothness versus loss plot shows superlinear growth in the warm-up phase, then the finite (H0,H1) constants do not exist on the trajectory and the theorem's regime split—and with it the claimed speed-up—does not apply to that model. A second, purely theoretical check: exhibit an (H0,H1)-smooth function where a tuned constant step size rea

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that warm-up is prescribed, not optional, inside the class of (H0,H1)-smooth functions, where the squared Hessian norm is bounded by H0 + H1(f(w)−f*). Gradient descent with step η_k = θ/(10H0+20H1(f(w_k)−f*)) has a step size that rises as the loss gap falls, and the paper proves O(H0 dist²/(θ²ε) + H1 dist²/θ²) iterations under the Aiming condition and O((H1/µ)(f(w0)−f*) + (H0/µ) log(H0/(H1ε))) under µ-PL, while constant-step GD needs Ω(H1(f(w0)−f*) log(1/ε)/ε) iterations in the convex case. Hence warm-up is provably faster when H1(f(w0)−f*)/ε is large, meaning poor initialization or high precision. The condition is closed under sums and affi

Load-bearing premise

The load-bearing premise is that along the actual training trajectory the squared Hessian norm really is bounded by H0 + H1 times the loss gap with finite constants; the paper proves this only for simple networks under balancedness or L2 regularization, and its empirical measurements—a finite-difference proxy on slow SGD runs—visibly deviate from linear in the early phase, which is exactly the phase warm-up targets.

Editorial extensions

If this is right

  • If the claim is right, warm-up is not an add-on: it is the schedule that tracks the allowed step size under a loss-gap curvature bound, and its benefit is largest exactly when training starts far from the optimum or targets high accuracy.
  • The theory yields explicit complexity comparisons: warm-up GD removes the 1/ε dependence on H1, leaving the ε-term controlled by H0 alone under the Aiming condition, and gives a linear-then-log phase under PL.
  • Under (H0,H1)-smoothness, the condition's closure under finite sums and affine transformations means the bound is preserved when losses are averaged over batches, which underpins the paper's stochastic extension under an interpolation condition.
  • In practice, a one-parameter schedule of the form η_k/max{1, f_k/C} is a direct translation of the theoretical rule and matches tuned linear warm-up while outperforming no warm-up on language and vision models.
  • The theory predicts that warm-up should be shorter for optimizers that leave the linear curvature regime quickly, which the paper observes for adaptive optimizers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the sharpest place to test the story is the very first iterations: the paper's own curvature measurements deviate from the linear fit in that phase, and that is precisely the phase warm-up exploits. If the deviation reflects a superlinear curvature-in-loss relationship, the warm-up schedule would still be justified but with a different functional form.
  • Editorial inference: the formal theorems use the adaptive rule with known H0, H1 and f*, while the deployed schedule uses a tuned threshold C with clamping; the quantitative constants in the upper bounds therefore do not directly govern the experiments. A direct run of the un-clamped theoretical rule, with H0 and H1 estimated once, would tell whether the benefit is the mechanism or the tuning.
  • Editorial inference: the (H0,H1) bound with power ρ>1 is a natural next object; deep leaky-ReLU networks already require f(w)−f* raised to a depth-dependent power, so extending the warm-up proof to power laws would cover deeper architectures than the paper's Propositions 3.1–3.4.
  • Editorial inference: if the curvature-loss bound holds for a given network, warm-up length can in principle be set without search: it is the time until the loss gap crosses H0/(2H1), the regime boundary in the proofs, so estimating H0 and H1 once at initialization would turn the tuned parameter C into a prediction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the convergence analysis is a conditional derivation from the stated (H0,H1)-smoothness assumption, not a prediction that reduces to its inputs.

full rationale

The central derivation chain is genuinely conditional: Definition 3.1 postulates a bound on the Hessian in terms of loss suboptimality, and the adaptive step-size η_k = 1/(10H0+20H1(f(w_k)-f*)) is chosen to exploit that bound. Theorems 4.2 and 4.3 then prove convergence rates using the Aiming/PL conditions with proofs deferred to the appendix; the step-size is an adaptive optimization choice, not a fitted value renamed as a prediction. Theorem 4.1 is a lower-bound proof for constant-step GD within the same function class, and it does not rely on the authors' prior work. The empirical verification in §3.2 and Figures 1-2 fits a line to estimated local smoothness versus training loss; this is in-sample validation rather than an out-of-sample prediction, and the paper does not claim those fitted constants are used to predict convergence. The practical schedule in §5 uses a separately tuned threshold C, so no fitted parameter is recycled as the theoretical prediction. Self-citations (e.g., Islamov et al. 2024, Compagnoni et al. 2025) appear only as background and are not load-bearing. The paper's own stated limitations—that the bound deviates in the very initial phase and deteriorates after warm-up—are candid and weaken the empirical bridge but do not make the derivation circular. Overall, the proof structure is self-contained and the central claims have independent mathematical content.

Assumptions & free parameters 2 free parameters · 8 assumptions · 0 invented entities

Everything the central claim rests on beyond the observed data: the novel (H0,H1)-smoothness assumption, the structural conditions (Aiming/PL/interpolation) used in the convergence theorems, and the balancedness/L2-regularization settings under which the condition is proved for networks. The empirical verification adds a measurement assumption (finite-difference proxy) and the practical schedule adds a tuned constant C. Count of free parameters is modest, but the load-bearing assumption — that (H0,H1)-smoothness holds along real training trajectories — is only loosely supported.

free parameters (2)
  • C (warm-up threshold in the (H0,H1) schedule) = C ∈ {3.5, 4, 4.5} for LMs, C ∈ {3, 3.5, 4} for ViT; representative C=4
    §5 and §I.4: the one-parameter warm-up is η_k^base/max{1, f_Sk(w_k)/C}. C determines when warm-up ends and is tuned per dataset/model on validation performance; the claim of matching linear warm-up depends on this tuned value.
  • Empirical H0/H1 (slope and intercept of the best linear fit in Figs 1–2) = Not reported numerically; least-squares line through (estimated smoothness, training loss)
    The figures fit a line through the measured (smoothness, loss) pairs as evidence for the (H0,H1) condition. The fit is visual; no R², confidence intervals, or residual analysis are given, and the fitted constants are not used in the theoretical rates or the practical schedule.
assumptions (8)
  • ad hoc to paper (H0,H1)-smoothness: ‖∇²f(w)‖₂ ≤ H0 + H1(f(w)−f*) for all w.
    Definition 3.1 is the central assumption of the paper; the adaptive warm-up step-size (1) is derived from it, and all convergence theorems (4.2–4.4) and lemmas assume it. Its validity for real networks is the load-bearing bridge.
  • domain assumption Aiming condition: ⟨∇h(w), w−π_X(w)⟩ ≥ θ(h(w)−h*) for all w.
    Invoked in Definition 4.1 and Theorem 4.2 (also Theorem 4.4 per component) to convert distance descent into suboptimality descent; cited from Liu et al. [2023].
  • domain assumption µ-PL condition: ‖∇h(w)‖² ≥ 2µ(h(w)−h*).
    Used in Theorem 4.3 for linear convergence of the adaptive schedule; cited from Polyak [1963]/Karimi et al. [2016].
  • domain assumption Interpolation condition: f_i* = f* for all components i.
    Assumed in Theorem 4.4 for the stochastic setting; standard for over-parameterized networks [Ma et al., 2018].
  • domain assumption Balancedness: strong balance W_iᵀW_i = W_{i+1}W_{i+1}ᵀ (or weak ‖W_i‖_F = ‖W_{i+1}‖_F), plus λ_min(W₁ᵀW₁) ≥ h > 0 during GD.
    Props 3.1, 3.2 and C.1 establish (H0,H1)-smoothness for deep networks only on these constrained weight sets; the Transformers/ViTs of §5 do not satisfy them. The lower bound h enters H0,H1 and can blow up as h→0.
  • domain assumption Activation regularity: |φ(x)| ≤ C1|x|, |φ'(x)| ≤ C2, |φ''(x)| ≤ C3 (or |φ| ≤ C0+C1|x|).
    Props 3.3–3.4 rely on these bounds for two-layer networks with L2 regularization and MSE/CE losses.
  • domain assumption Finite-difference smoothness estimator with successive mini-batches approximates the local Hessian norm.
    §3.2 defines estimated smoothness as ‖∇f_{S_k}(w_{k+1})−∇f_{S_{k-1}}(w_k)‖/‖w_{k+1}−w_k‖ and interprets the resulting scatter as evidence for the (H0,H1) condition; mixing batches and iterates adds noise, and no error bars are given.
  • ad hoc to paper Practical schedule equivalence: f* ≈ 0 and the constant term 10H0 is negligible, so the theoretical step 1/(10H0+20H1(f−f*)) is implemented as baseLR/max{1, f_Sk(w_k)/C}.
    §5 uses η_k^base/max{1,f_Sk(w_k)/C} and calls it a 'hard counterpart' of (1); this equivalence is not derived from the theory, and C is tuned validationally.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Why Do We Need Warm-up? A Theoretical Perspective." pith.science (2026). https://pith.science/paper/Q5M5YSZS

@misc{pith2026251003164,
  author       = {Pith},
  title        = {Pith review of: Why Do We Need Warm-up? A Theoretical Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q5M5YSZS}},
  note         = {Machine review of arXiv:2510.03164}
}
abstract

Learning rate warm-up -- increasing the learning rate at the beginning of training -- has become a ubiquitous heuristic in modern deep learning, yet its theoretical foundations remain poorly understood. In this work, we provide a principled explanation for why warm-up improves training. We rely on a generalization of the $(L_0, L_1)$-smoothness condition, which bounds local curvature as a linear function of the loss suboptimality and exhibits desirable closure properties. We show -- both theoretically and empirically -- that this condition is satisfied by common neural architectures and accurately captures the curvature of the optimization landscape early in training. Adapting the learning rate in response to this curvature condition naturally induces a warm-up-like schedule, and we show that this choice yields provably faster convergence guarantees than using a fixed learning rate. Experiments on language and vision models show that the resulting one-parameter warm-up schedule can match tuned linear warm-up and improve over no warm-up.

Figures

Figures reproduced from arXiv: 2510.03164 by the authors.

Figure 1
Figure 1. Local smoothness approximation versus training loss for language models of varying sizes [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Local smoothness approximation against train loss during training a ResNet50 (left) and [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Performance of Adam (for 70M and 160M) and AdamW (for 410M with weight decay λ = 0.1) when training language models with three warm-up strategies: (H0, H1) warm-up with tuned C, tuned linear warm-up, and no warm-up. The last 20% of iterations is a linear decay from the peak LR to 10−5 in all cases. To derive a tighter convergence rate, we split the iterations into two parts – small and large function values – and an… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Effective LR with (H0, H1) warm-up when training language models on the FineWeb dataset for the peak LR 10−3 , varying parameter in (H0, H1) warm-up. need an interpolation condition, which is typically satisfied for over-parameterized networks [Ma et al., 2018]. Analyt…
Figure 5
Figure 5. Figure 5: Performance of AdamW with weight decay λ = 0.05 when training ViT model on the ImageNet32 dataset with three warm-up strategies: (H0, H1) warm-up with tuned C, tuned linear warm-up, and no warm-up. All LR schedules follow cosine decay after the warm-up phase. 10−4 , 10…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Avoiding Bias in Clipped SGD for Overparameterized Models under Generalized Smoothness

    math.OC 2026-05 unverdicted novelty 7.0 of 10

    Clipped and normalized SGD converge without bias in overparameterized interpolating models under (L0,L1)-smoothness, with improved rates and extensions to heavy-tailed noise and weaker smoothness.

Reference graph

Works this paper leans on

76 extracted references · 31 linked inside Pith · cited by 1 Pith paper

  1. [1]

    plainlm: Language model pretraining in pytorch

    Niccolò Ajroldi. plainlm: Language model pretraining in pytorch. https://github.com/Niccolo-Ajroldi/plainLM, 2024

    2024

  2. [2]

    vision: Vision model pretraining in pytorch

    Niccolò Ajroldi. vision: Vision model pretraining in pytorch. https://github.com/Niccolo-Ajroldi/vision, 2025

    2025

  3. [3]

    Benefits of learning rate annealing for tuning-robustness in stochastic optimization

    Amit Attia and Tomer Koren. Benefits of learning rate annealing for tuning-robustness in stochastic optimization. arXiv preprint arXiv:2503.09411, 2025

    arXiv 2025

  4. [4]

    Layer normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016

    arXiv 2016

  5. [5]

    A downsampled variant of imagenet as an alternative to the cifar datasets

    Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:1707.08819, 2017

    arXiv 2017

  6. [6]

    On the interaction of noise, compression role, and adaptivity under (l\_0, l\_1) -smoothness: An sde-based approach

    Enea Monzio Compagnoni, Rustem Islamov, Antonio Orvieto, and Eduard Gorbunov. On the interaction of noise, compression role, and adaptivity under (l\_0, l\_1) -smoothness: An sde-based approach. arXiv preprint arXiv:2506.00181, 2025

    arXiv 2025

  7. [7]

    Why gradients rapidly increase near the end of training

    Aaron Defazio. Why gradients rapidly increase near the end of training. arXiv preprint arXiv:2506.02285, 2025

    arXiv 2025

  8. [8]

    Optimal linear decay learning rate schedules and further refinements

    Aaron Defazio, Ashok Cutkosky, Harsh Mehta, and Konstantin Mishchenko. Optimal linear decay learning rate schedules and further refinements. arXiv preprint arXiv:2310.07831, 2023

    arXiv 2023

Show all 76 references
  1. [9]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2010 arXiv

  2. [10]

    Training dynamics of the cooldown stage in warmup-stable-decay learning rate scheduler

    Aleksandr Dremov, Alexander H \"a gele, Atli Kosson, and Martin Jaggi. Training dynamics of the cooldown stage in warmup-stable-decay learning rate scheduler. arXiv preprint arXiv:2508.01483, 2025

    2025 arXiv

  3. [11]

    Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced

    Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Advances in neural information processing systems, 2018

    2018

  4. [12]

    Beyond uniform smoothness: A stopped analysis of adaptive sgd

    Matthew Faw, Litu Rout, Constantine Caramanis, and Sanjay Shakkottai. Beyond uniform smoothness: A stopped analysis of adaptive sgd. In The Thirty Sixth Annual Conference on Learning Theory, 2023

    2023

  5. [13]

    Accelerated stochastic optimization methods under quasar-convexity

    Qiang Fu, Dongchu Xu, and Ashia Camage Wilson. Accelerated stochastic optimization methods under quasar-convexity. In International Conference on Machine Learning. PMLR, 2023

    2023

  6. [14]

    Convergence of clipped sgd on convex (l\_0, l\_1) -smooth functions

    Ofir Gaash, Kfir Yehuda Levy, and Yair Carmon. Convergence of clipped sgd on convex (l\_0, l\_1) -smooth functions. arXiv preprint arXiv:2502.16492, 2025

    2025 arXiv

  7. [15]

    A loss curvature perspective on training instability in deep learning

    Justin Gilmer, Behrooz Ghorbani, Ankush Garg, Sneha Kudugunta, Behnam Neyshabur, David Cardoze, George Dahl, Zachary Nado, and Orhan Firat. A loss curvature perspective on training instability in deep learning. arXiv preprint arXiv:2110.04369, 2021

    2021 arXiv

  8. [16]

    Methods for convex (l\_0, l\_1) -smooth optimization: Clipping, acceleration, and adaptivity

    Eduard Gorbunov, Nazarii Tupitsa, Sayantan Choudhury, Alen Aliev, Peter Richt \'a rik, Samuel Horv \'a th, and Martin Tak \'a c . Methods for convex (l\_0, l\_1) -smooth optimization: Clipping, acceleration, and adaptivity. arXiv preprint arXiv:2409.14989, 2024

    2024 arXiv

  9. [17]

    A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation

    Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. arXiv preprint arXiv:1810.13243, 2018

    2018 arXiv

  10. [18]

    Sgd for structured nonconvex functions: Learning rates, minibatching and interpolation

    Robert Gower, Othmane Sebbouh, and Nicolas Loizou. Sgd for structured nonconvex functions: Learning rates, minibatching and interpolation. In International Conference on Artificial Intelligence and Statistics, 2021

    2021

  11. [19]

    Accurate, large minibatch sgd: Training imagenet in 1 hour

    Priya Goyal, Piotr Doll \'a r, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017

    2017 arXiv

  12. [20]

    No wrong turns: The simple geometry of neural networks optimization paths

    Charles Guille-Escuret, Hiroki Naganuma, Kilian Fatras, and Ioannis Mitliagkas. No wrong turns: The simple geometry of neural networks optimization paths. arXiv preprint arXiv:2306.11922, 2023

    2023 arXiv

  13. [21]

    Scaling laws and compute-optimal training beyond fixed training durations

    Alex H \"a gele, Elie Bakouch, Atli Kosson, Leandro Von Werra, Martin Jaggi, et al. Scaling laws and compute-optimal training beyond fixed training durations. Advances in Neural Information Processing Systems, 2024

    2024

  14. [22]

    Gradient descent learns linear dynamical systems

    Moritz Hardt, Tengyu Ma, and Benjamin Recht. Gradient descent learns linear dynamical systems. Journal of Machine Learning Research, 2018

    2018

  15. [23]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016

    2016

  16. [24]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  17. [25]

    Near-optimal methods for minimizing star-convex functions and beyond

    Oliver Hinder, Aaron Sidford, and Nimit Sohoni. Near-optimal methods for minimizing star-convex functions and beyond. In Conference on learning theory, 2020

    2020

  18. [26]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022 arXiv

  19. [27]

    An empirical analysis of compute-optimal large language model training

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. An empirical analysis of compute-optimal large language model training. Advances in neural informat...

    2022

  20. [28]

    Minicpm: Unveiling the potential of small language models with scalable training strategies

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024

    2024 arXiv

  21. [29]

    Improving transformer optimization through better initialization

    Xiao Shi Huang, Felipe Perez, Jimmy Ba, and Maksims Volkovs. Improving transformer optimization through better initialization. In International Conference on Machine Learning, 2020

    2020

  22. [30]

    Loss landscape characterization of neural networks without over-parametrization

    Rustem Islamov, Niccol \`o Ajroldi, Antonio Orvieto, and Aurelien Lucchi. Loss landscape characterization of neural networks without over-parametrization. Advances in Neural Information Processing Systems, 2024

    2024

  23. [31]

    Why warmup the learning rate? underlying mechanisms and improvements

    Dayal Singh Kalra and Maissam Barkeshli. Why warmup the learning rate? underlying mechanisms and improvements. Advances in Neural Information Processing Systems, 2024

    2024

  24. [32]

    Linear convergence of gradient and proximal-gradient methods under the polyak- ojasiewicz condition

    Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal-gradient methods under the polyak- ojasiewicz condition. In Joint European conference on machine learning and knowledge discovery in databases, 2016

    2016

  25. [33]

    Andrej Karpathy. NanoGPT . https://github.com/karpathy/nanoGPT, 2022

    2022

  26. [34]

    Deep learning without poor local minima

    Kenji Kawaguchi. Deep learning without poor local minima. Advances in neural information processing systems, 2016

    2016

  27. [35]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  28. [36]

    An alternative view: When does sgd escape local minima? In International conference on machine learning, 2018

    Bobby Kleinberg, Yuanzhi Li, and Yang Yuan. An alternative view: When does sgd escape local minima? In International conference on machine learning, 2018

    2018

  29. [37]

    Accelerating sgdm via learning rate and batch size schedules: A lyapunov-based analysis

    Yuichi Kondo and Hideaki Iiduka. Accelerating sgdm via learning rate and batch size schedules: A lyapunov-based analysis. arXiv preprint arXiv:2508.03105, 2025

    2025 arXiv

  30. [38]

    Analyzing & reducing the need for learning rate warmup in gpt training

    Atli Kosson, Bettina Messmer, and Martin Jaggi. Analyzing & reducing the need for learning rate warmup in gpt training. Advances in Neural Information Processing Systems, 2024

    2024

  31. [39]

    Convex and non-convex optimization under generalized smoothness

    Haochuan Li, Jian Qian, Yi Tian, Alexander Rakhlin, and Ali Jadbabaie. Convex and non-convex optimization under generalized smoothness. Advances in Neural Information Processing Systems, 2023

    2023

  32. [40]

    Loss landscapes and optimization in over-parameterized non-linear systems and neural networks

    Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over-parameterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis, 2022

    2022

  33. [41]

    Aiming towards the minimizers: fast convergence of sgd for overparametrized problems

    Chaoyue Liu, Dmitriy Drusvyatskiy, Misha Belkin, Damek Davis, and Yian Ma. Aiming towards the minimizers: fast convergence of sgd for overparametrized problems. Advances in neural information processing systems, 2023

    2023

  34. [42]

    On the variance of the adaptive learning rate and beyond

    Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908.03265, 2019

    1908 arXiv

  35. [43]

    Theoretical analysis on how learning rate warmup accelerates convergence

    Yuxing Liu, Yuze Ge, Rui Pan, An Kang, and Tong Zhang. Theoretical analysis on how learning rate warmup accelerates convergence. arXiv preprint arXiv:2509.07972, 2025

    2025 arXiv

  36. [44]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016

    2016 arXiv

  37. [45]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  38. [46]

    The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning

    Siyuan Ma, Raef Bassily, and Mikhail Belkin. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning, 2018

    2018

  39. [47]

    Matrix differential calculus with applications to simple, hadamard, and kronecker products

    Jan R Magnus. Matrix differential calculus with applications to simple, hadamard, and kronecker products. Journal of Mathematical Psychology, 1985

    1985

  40. [48]

    An empirical model of large-batch training

    Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162, 2018

    2018 arXiv

  41. [49]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydl \' c ek, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 2024

    2024

  42. [50]

    Gradient methods for minimizing functionals

    Boris Teodorovich Polyak. Gradient methods for minimizing functionals. Zhurnal vychislitel'noi matematiki i matematicheskoi fiziki, 1963

    1963

  43. [51]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 2019

    2019

  44. [52]

    Gluon: Making muon & scion great again!(bridging theory and practice of lmo-based optimizers for llms)

    Artem Riabinin, Egor Shulgin, Kaja Gruntkowska, and Peter Richt \'a rik. Gluon: Making muon & scion great again!(bridging theory and practice of lmo-based optimizers for llms). arXiv preprint arXiv:2505.13416, 2025

    2025 arXiv

  45. [53]

    Stepping on the edge: Curvature aware learning rate tuners

    Vincent Roulet, Atish Agarwala, Jean-Bastien Grill, Grzegorz Swirszcz, Mathieu Blondel, and Fabian Pedregosa. Stepping on the edge: Curvature aware learning rate tuners. Advances in Neural Information Processing Systems, 2024

    2024

  46. [54]

    The surprising agreement between convex optimization theory and learning-rate scheduling for large model training

    Fabian Schaipp, Alexander H \"a gele, Adrien Taylor, Umut Simsekli, and Francis Bach. The surprising agreement between convex optimization theory and learning-rate scheduling for large model training. arXiv preprint arXiv:2501.18965, 2025

    2025 arXiv

  47. [55]

    Glu variants improve transformer

    Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  48. [56]

    On the generalization benefit of noise in stochastic gradient descent

    Samuel Smith, Erich Elsen, and Soham De. On the generalization benefit of noise in stochastic gradient descent. In International Conference on Machine Learning, 2020

    2020

  49. [57]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 2024

    2024

  50. [58]

    On the importance of initialization and momentum in deep learning

    Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, 2013

    2013

  51. [59]

    Fast convergence in learning two-layer neural networks with separable data

    Hossein Taheri and Christos Thrampoulidis. Fast convergence in learning two-layer neural networks with separable data. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023

    2023

  52. [60]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  53. [61]

    Empirical tests of optimization assumptions in deep learning

    Hoang Tran, Qinzi Zhang, and Ashok Cutkosky. Empirical tests of optimization assumptions in deep learning. arXiv preprint arXiv:2407.01825, 2024

    2024

  54. [62]

    Optimizing (l\_0, l\_1) -smooth functions by gradient methods

    Daniil Vankov, Anton Rodomanov, Angelia Nedich, Lalitha Sankar, and Sebastian U Stich. Optimizing (l\_0, l\_1) -smooth functions by gradient methods. arXiv preprint arXiv:2410.10800, 2024

    2024 arXiv

  55. [63]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 2017

    2017

  56. [65]

    Convergence of adagrad for non-convex objectives: Simple proofs and relaxed assumptions

    Bohan Wang, Huishuai Zhang, Zhiming Ma, and Wei Chen. Convergence of adagrad for non-convex objectives: Simple proofs and relaxed assumptions. In The Thirty Sixth Annual Conference on Learning Theory, 2023

    2023

  57. [66]

    Understanding warmup-stable-decay learning rates: A river valley loss landscape perspective

    Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, and Tengyu Ma. Understanding warmup-stable-decay learning rates: A river valley loss landscape perspective. arXiv preprint arXiv:2410.05192, 2024

    2024 arXiv

  58. [67]

    Small-scale proxies for large-scale transformer training instabilities

    Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXiv:2309.14322, 2023

    2023 arXiv

  59. [68]

    On the overlooked pitfalls of weight decay and how to mitigate them: A gradient-norm perspective

    Zeke Xie, Zhiqiang Xu, Jingzhao Zhang, Issei Sato, and Masashi Sugiyama. On the overlooked pitfalls of weight decay and how to mitigate them: A gradient-norm perspective. Advances in Neural Information Processing Systems, 2023

    2023

  60. [69]

    On layer normalization in the transformer architecture

    Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International conference on machine learning, 2020

    2020

  61. [70]

    Dive into deep learning

    Aston Zhang, Zachary C Lipton, Mu Li, and Alexander J Smola. Dive into deep learning. Cambridge University Press, 2023

    2023

  62. [71]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in neural information processing systems, 2019

    2019

  63. [72]

    Improved analysis of clipping algorithms for non-convex optimization

    Bohang Zhang, Jikai Jin, Cong Fang, and Liwei Wang. Improved analysis of clipping algorithms for non-convex optimization. Advances in Neural Information Processing Systems, 2020

    2020

  64. [73]

    Why gradient clipping accelerates training: A theoretical justification for adaptivity

    Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. arXiv preprint arXiv:1905.11881, 2019

    1905 arXiv

  65. [74]

    On the convergence and improvement of stochastic normalized gradient descent

    Shen-Yi Zhao, Yin-Peng Xie, and Wu-Jun Li. On the convergence and improvement of stochastic normalized gradient descent. Science China Information Sciences, 2021

    2021

  66. [75]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  67. [76]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  68. [77]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    1989

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.