Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

Learning-rate warmup provably accelerates gradient descent under a new gap-based smoothness assumption, by up to a factor of the horizon T.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 21:24 UTC pith:WOGODOQ6

load-bearing objection New smoothness assumption and a provable speedup for an adaptive increasing-rate schedule, but the schedule needs the unknown f*, so the paper overclaims on practical warmup. the 4 major comments →

arxiv 2509.07972 v1 pith:WOGODOQ6 submitted 2025-09-09 cs.LG math.OC

Theoretical Analysis on how Learning Rate Warmup Accelerates Convergence

Yuxing Liu , Yuze Ge , Rui Pan , An Kang , Tong Zhang This is my paper
classification cs.LG math.OC MSC 90C2690C1568T07
keywords learning rate warmupgeneralized smoothnesssuboptimality gapgradient descentstochastic gradient descentconvergence analysisnonconvex optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to give a rigorous explanation of why learning-rate warmup—starting training with a small learning rate and increasing it—helps train deep networks. It introduces a new smoothness condition, (rho, K0, Krho)-smoothness, which bounds the local curvature of the loss by K0 + Krho (f(w) - f*)^rho, so that regions with a smaller remaining loss gap are provably less curved. Under this condition, gradient descent with a learning rate that increases as the gap shrinks is proved to converge to a stationary point at a rate no worse than constant-rate GD, and in some cases up to a factor of T faster; the stochastic version gives up to sqrt(T) faster rates. The paper thus turns a widely used but poorly understood heuristic into a provable acceleration, and identifies the initial loss gap as the quantity that determines how much warmup can help. The price is that the analyzed schedule depends on knowing the optimal loss at each step.

Core claim

The central claim, Theorem 1, is that for any twice-differentiable function with ||Hessian(w)|| <= K0 + Krho (f(w) - f*)^rho, gradient descent with the increasing schedule eta_t = (1/(4*sqrt(2)+4)) min{1/K0, 1/(3*rho*Krho*Delta_t^rho)} achieves min_t ||grad f(w_t)||^2 <= O(K0*Delta0/T + Krho*Delta0 * sum_{t=0}^{T-1} Delta_t^rho / T^2), where Delta_t = f(w_t) - f*. A constant learning rate chosen from the initial gap gives O((K0*Delta0 + Krho*Delta0^{rho+1})/T). Because Delta_t is nonincreasing, the warmup bound is always at least as good and is better whenever the Krho term matters; Example 4, a river-valley function approximating neural loss landscapes, realizes a Theta(Delta0) speedup up t

What carries the argument

The load-bearing object is (rho, K0, Krho)-smoothness: the spectral norm of the Hessian at w is bounded by K0 + Krho (f(w) - f*)^rho, a condition that ties local curvature to the remaining loss gap. Its work is to make the descent lemma hold with constants that depend on the current gap: if consecutive iterates stay inside radius r(Delta_t) = min{C1 Delta_t^{-(rho-1)/2}, C2}, then the usual smooth-descent inequality applies locally. The learning-rate schedule is chosen so that eta_t ||grad f(w_t)|| stays inside that radius; since Delta_t decreases along the trajectory, the permitted step size increases, which is the warmup phenomenon the theorem isolates.

Load-bearing premise

The load-bearing premise is that local curvature is polynomially bounded by the remaining loss gap, and the schedule that then provably accelerates also requires knowing the optimal loss f* at each step, so without an estimate of the gap the theory is an oracle analysis.

What would settle it

On a one-dimensional function from the paper's lower-bound construction, or on the river-valley Example 4 with f* known, count the gradient steps needed to reach ||grad f(w)|| <= epsilon under a constant learning rate tuned for Delta0 and under the paper's increasing schedule; if the ratio does not grow with Delta0 as predicted, the claimed Theta(T) speedup is falsified. Alternatively, along a real neural-network training run, record Hessian spectral norm and f(w)-f*; if the envelope is not bounded by K0 + Krho Delta^rho for fixed rho, the new smoothness assumption fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • For any (rho, K0, Krho)-smooth objective, the increasing schedule is never worse than constant-rate GD, and the improvement is controlled by how much the average of Delta_t^rho is smaller than Delta_0^rho.
  • The acceleration grows with the initial loss gap: in the river-valley example it is Theta(Delta0) up to log factors, so warmup matters most when initialization is poor.
  • The rho=1 lower bound for non-increasing schedules implies that the warmup gain is not a proof artifact: within that function class, no non-increasing schedule matches the warmup rate.
  • Under bounded noise and under the ABC inequality, SGD with warmup keeps the optimal O(1/sqrt(T)) rate while shrinking the gap- and noise-dependent constants, giving up to Theta(sqrt(T)) speedup.
  • Because the new assumption is strictly weaker than gradient-norm-based generalized smoothness for rho<2, the convergence results cover functions, including some simple neural-network losses, where the older condition fails or only holds for rho>=2.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that a practical warmup schedule needs either an online estimate of f* or a proof that a fixed-duration linear warmup is comparable to the gap-adaptive schedule; the connection to linear warmup is currently empirical only.
  • The same gap-based smoothness condition could in principle design adaptive learning rates for momentum or Adam, but the paper's proofs do not cover those optimizers; testing whether Adam's warmup phase follows the predicted Delta_t dependence is a direct empirical check.
  • The lower-bound construction is trigonometric and specific to rho=1; extending it to other rho would show whether the Theta(T) separation is a general property of the assumption class rather than a special case.
  • The theory predicts warmup helps most when Delta0 is large, which is testable by varying initialization quality or batch size while keeping the architecture fixed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces a novel (ρ, K0, Kρ)-smoothness assumption that upper-bounds the Hessian spectral norm by K0 + Kρ (f(w) − f*)^ρ, i.e., by a power of the suboptimality gap rather than by the gradient norm. It proves that this family is strictly weaker than the standard (ρ, L0, Lρ)-smoothness for 0 ≤ ρ < 2, gives neural-network examples satisfying the new condition, and provides empirical log-log plots of estimated smoothness versus loss gap for ResNet and NanoGPT. The main theoretical results are convergence bounds for GD and SGD under gap-dependent adaptive learning rates, with a comparison to constant learning rates. The paper claims that these adaptive/increasing schedules, viewed as learning-rate warmup, can accelerate convergence by up to Θ(T) for GD and Θ(√T) for SGD. It also includes a river-valley example showing a large speedup and a lower bound for non-increasing schedules under (1, K0, K1)-smoothness.

Significance. If the central claim were established, the paper would offer a useful theoretical perspective on warmup: local smoothness controlled by the suboptimality gap, and step sizes that grow as the gap shrinks. The proposed smoothness family is genuinely different from existing gradient-norm-based generalized smoothness, and the neural-network examples and the lower bound are valuable independent contributions. In particular, Theorem 3's lower bound for non-increasing schedules matching the constant-rate upper bound is a strong, concrete result. However, the load-bearing claim that this theory explains practical warmup is not proven as stated. The accelerating schedule is an oracle schedule that depends on the unknown optimal loss f* at every iteration, and the worst-case comparison in Theorem 1 does not actually guarantee a speedup over constant learning rates. These gaps need to be addressed before the paper can support its title and abstract claims.

major comments (4)
  1. [4.1, Theorem 1 (Eqs. 2–3)] The schedule η_t = (1/(4√2+4)) min{1/K0, 1/(3ρKρ Δ_t^ρ)} is not a practical or even pre-determined warmup schedule: it requires knowledge of f* at every step through Δ_t = f(w_t) − f*. More importantly, the claimed improvement over constant learning rate is not a worst-case speedup. Since Σ_{t=0}^{T-1} Δ_t^ρ ≤ T Δ_0^ρ, the bound in (2) is at most O((K0Δ0 + KρΔ0^{ρ+1})/T), which is the same order as the constant-rate bound in (3). The conclusion that warmup is 'better' relies on favorable trajectory behavior of Δ_t that is not guaranteed by the theorem's assumptions. The paper should either prove a bound on ΣΔ_t^ρ under additional conditions or explicitly frame the result as an oracle adaptive-schedule advantage conditional on the trajectory.
  2. [4.2, Figure 2 and Table 1] The empirical bridge between the theoretical schedule and standard warmup is not supported by any theorem. The ResNet experiment uses mini-batch SGD with cosine decay, while Theorem 1 is for deterministic full-batch GD. The theoretical schedule is implemented using f* = 0 and tuned K0, Kρ, so it does not instantiate the theorem's assumptions. The claim that 'the theoretical schedule employed in Theorem 1 can be considered a valid representative of the warmup schedules' is therefore only an empirical observation, and Figure 2 is a single synthetic example. To support the central claim that warmup accelerates convergence, the paper needs at least one convergence theorem for a fixed, implementable increasing schedule (e.g., linear warmup), or the claims must be substantially narrowed.
  3. [5.1–5.2, Theorems 4–7] The stochastic results inherit the same oracle limitation: the step sizes in (19) and in Theorem 4 depend on Δ_t and hence on f* at each step. Moreover, the statement that 'η_t is approximately increasing' is heuristic because SGD does not ensure Δ_t is monotone; the proofs only establish Δ_t ≤ 4Δ0 (or 8Δ0) with high probability. Thus the claimed Θ(√T) stochastic acceleration over constant learning rates is not proven for any actual warmup schedule. The section would need a theorem for a pre-determined increasing sequence, or a careful reformulation in terms of adaptive gap-based step sizes rather than warmup.
  4. [4.3, Theorem 3] The lower bound is an important contribution, but it only bounds non-increasing schedules. It does not by itself establish that warmup is necessary to obtain faster rates; it shows tightness of the constant-rate upper bound. The paper should state this relationship more carefully to avoid the impression that the lower bound directly supports the warmup acceleration claim.
minor comments (4)
  1. [5.1, Theorem 5] The text says 'Under the same assumptions as Theorem 5' in the statement of Theorem 5; this should presumably be 'Theorem 4'.
  2. [5.1] Typo: 'Assuption 4' should be 'Assumption 4'.
  3. [1, Introduction] The phrase 'consistently accelerates GD' in the abstract and introduction is too strong given the oracle-schedule caveat; consider replacing with 'can accelerate under a gap-dependent adaptive schedule and favorable trajectories'.
  4. [3.3, Figure 1] The empirical smoothness estimate uses only six direction samples and no error bars or multiple seeds; the figure caption should note this limitation, and the text should not overstate the validation.

Circularity Check

0 steps flagged

No circularity: the warmup theorem is derived from the (ρ,K0,Kρ)-smoothness assumption and an explicit oracle schedule; the speedup term is left as a trajectory quantity and the lower bound is independently constructed.

full rationale

The paper's central derivation is self-contained. Theorem 1 posits an explicit increasing schedule η_t = (1/(4√2+4)) min{1/K0, 1/(3ρKρ Δ_t^ρ)} and proves by induction, via Lemma 2 and the verification η_t ≤ 1/L_t with L_t = 2K0 + Kρ(2Δ_t)^ρ, that Δ_t is nonincreasing and that Σ η_t ||∇f(w_t)||² ≤ 2Δ0. The warmup rate then follows from the AM-HM bound on Σ η_t; it is not obtained by assuming the conclusion. The improvement over the constant-Δ0 schedule is expressed through the explicit trajectory term Σ Δ_t^ρ, which is not assumed small in the theorem except through the already-proved monotonicity; Example 4 supplies an independent construction where it is small. Theorem 3's lower bound is an adversary construction with trigonometric functions and does not import the target result. The main caveat is that the schedule is an oracle schedule requiring f* at each step, and the paper's connection to practical linear warmup is empirical; this is an applicability gap, not a circular reduction. Self-citations (e.g., Liu et al. 2024) appear only in related work and are not load-bearing. The conclusion's own limitation statement covers momentum/Adam and the restricted lower bound. No step in the derivation reduces by construction to its inputs.

Axiom & Free-Parameter Ledger

2 free parameters · 6 axioms · 0 invented entities

The central theory rests on the new smoothness assumption, which is not proven for general neural networks, plus the oracle availability of the loss gap. The parameters K0, Krho, rho are assumed, not fitted, in the derivation, but in the experiments they are tuned to the data. No new physical or mathematical entities are introduced.

free parameters (2)
  • Learning rate scale multiplier = tuned per schedule, not reported
    In Section 4.2, the authors state they tune an optimal constant multiplier for all schedules to achieve fastest convergence without divergence; the fitted values are not listed.
  • K0, Krho in ResNet experiment = tuned from K0 in {1,4,8,16}, Krho in {1/4,1/2,1,4}, final values not reported
    The theoretical warmup schedule in the ResNet experiment uses these smoothness constants, tuned to the data; the specific chosen values are not given, affecting reproducibility.
axioms (6)
  • domain assumption (rho, K0, Krho)-smoothness: ||Hess f(w)|| <= K0 + Krho (f(w) - f*)^rho
    Definition 1 is the core modeling assumption. Its validity for neural networks is supported only by qualitative empirical plots and two toy examples, not by a general proof.
  • domain assumption The loss gap Delta_t = f(w_t) - f* is known or measurable at each step for the warmup schedule
    The accelerating schedule in Theorem 1 is defined using Delta_t, requiring f*. This is an oracle assumption not available in practice.
  • standard math Existence of f* > -infinity and finite initial gap Delta0
    Assumption 1; standard in nonconvex analysis.
  • domain assumption Convexity in Theorem 2
    Theorem 2 assumes f is convex to obtain the cocoercivity-type inequality; the nonconvex GD result in Theorem 1 does not use convexity.
  • domain assumption Noise assumptions: bounded noise (Assumption 3) and ABC inequality (Assumption 4)
    Used for SGD theorems; they are standard or recently introduced in the cited literature.
  • standard math Gronwall's inequality used in Lemma 4
    Lemma 3 is quoted from Li et al. 2023a as a standard tool.

pith-pipeline@v1.3.0-alltime-deepseek · 36232 in / 12723 out tokens · 151079 ms · 2026-08-04T21:24:33.028416+00:00 · methodology

0 comments
read the original abstract

Learning rate warmup is a popular and practical technique in training large-scale deep neural networks. Despite the huge success in practice, the theoretical advantages of this strategy of gradually increasing the learning rate at the beginning of the training process have not been fully understood. To resolve this gap between theory and practice, we first propose a novel family of generalized smoothness assumptions, and validate its applicability both theoretically and empirically. Under the novel smoothness assumption, we study the convergence properties of gradient descent (GD) in both deterministic and stochastic settings. It is shown that learning rate warmup consistently accelerates GD, and GD with warmup can converge at most $\Theta(T)$ times faster than with a non-increasing learning rate schedule in some specific cases, providing insights into the benefits of this strategy from an optimization theory perspective.

Figures

Figures reproduced from arXiv: 2509.07972 by An Kang, Rui Pan, Tong Zhang, Yuxing Liu, Yuze Ge.

Figure 1
Figure 1. Figure 1: Local smoothness vs. function suboptimality gap on training (a) ResNet18 on CIFAR-10 (b) [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An empirical experiment based on the synthetic problem setting in Example [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: A comparison between warmup learning rate schedules in ResNet training. The blue line [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Avoiding Bias in Clipped SGD for Overparameterized Models under Generalized Smoothness

    math.OC 2026-05 unverdicted novelty 7.0

    Clipped and normalized SGD converge without bias in overparameterized interpolating models under (L0,L1)-smoothness, with improved rates and extensions to heavy-tailed noise and weaker smoothness.

  2. A Provably Robust Multi-Jet Framework applied to Active Flow Control of an Airfoil in Weakly Compressible Flow

    physics.flu-dyn 2026-04 unverdicted novelty 7.0

    A new injective multi-jet framework for RL flow control provides jet-count-independent running cost upper bounds and enables superior coordinated jet strategies, achieving drag suppression beyond symmetric ideals on c...

  3. Why Do We Need Warm-up? A Theoretical Perspective

    cs.LG 2025-10 conditional novelty 5.0

    Under the proposed (H0,H1)-smoothness condition, gradient descent with a warm-up-style adaptive step-size provably converges faster than with any fixed step-size.

Reference graph

Works this paper leans on

37 extracted references · 23 canonical work pages · cited by 3 Pith papers · 1 internal anchor

  1. [1]

    Qsgd: Communication- Efficient SGD via Gradient Quantization and Encoding

    Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication- Efficient SGD via Gradient Quantization and Encoding . In Conference on Neural Information Processing Systems ( NeurIPS ) , pages 1709--1720, 2017

    work page 2017

  2. [2]

    Duchi, Dylan J

    Yossi Arjevani, Yair Carmon, John C. Duchi, Dylan J. Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming, 199 0 (1-2): 0 165--214, 2023

    work page 2023

  3. [3]

    Curtis, and Jorge Nocedal

    L \' e on Bottou, Frank E. Curtis, and Jorge Nocedal. Optimization Methods for Large - Scale Machine Learning . SIAM Review, 60 0 (2): 0 223--311, 2018

    work page 2018

  4. [4]

    Convex optimization

    Stephen P Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004

    2004

  5. [5]

    Gradient descent on neural networks typically occurs at the edge of stability

    Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=jh-rTtvkGeM

    work page 2021

  6. [6]

    Robustness to unbounded smoothness of generalized signsgd

    Michael Crawshaw, Mingrui Liu, Francesco Orabona, Wei Zhang, and Zhenxun Zhuang. Robustness to unbounded smoothness of generalized signsgd. Advances in neural information processing systems, 35: 0 9955--9968, 2022

    work page 2022

  7. [7]

    A loss curvature perspective on training instabilities of deep learning models

    Justin Gilmer, Behrooz Ghorbani, Ankush Garg, Sneha Kudugunta, Behnam Neyshabur, David Cardoze, George Edward Dahl, Zachary Nado, and Orhan Firat. A loss curvature perspective on training instabilities of deep learning models. In International Conference on Learning Representations, 2022

    work page 2022

  8. [8]

    A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation

    Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. arXiv preprint arXiv:1810.13243, 2018

    Pith/arXiv arXiv 2018

  9. [9]

    A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation

    Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=r14EOsCqKX

    work page 2019

  10. [10]

    Sgd: General Analysis and Improved Rates

    Robert Mansel Gower, Nicolas Loizou, Xun Qian, Alibek Sailanbayev, Egor Shulgin, and Peter Richt \' a rik. Sgd: General Analysis and Improved Rates . In International Conference on Machine Learning , volume abs/1901.09401, 2019

    Pith/arXiv arXiv 1901

  11. [11]

    Accurate, large minibatch sgd: Training imagenet in 1 hour

    Priya Goyal, Piotr Doll \'a r, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017

    Pith/arXiv arXiv 2017

  12. [12]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770--778, 2016

    2016

  13. [13]

    Three factors influencing minima in SGD , 2018

    Stanisław Jastrzębski, Zac Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Amos Storkey, and Yoshua Bengio. Three factors influencing minima in SGD , 2018. URL https://openreview.net/forum?id=rJma2bZCW

    work page 2018

  14. [14]

    Why warmup the learning rate? underlying mechanisms and improvements

    Dayal Singh Kalra and Maissam Barkeshli. Why warmup the learning rate? underlying mechanisms and improvements. Advances in Neural Information Processing Systems, 37: 0 111760--111801, 2024

    work page 2024

  15. [15]

    Better Theory for SGD in the Nonconvex World

    Ahmed Khaled and Peter Richt \' a rik. Better Theory for SGD in the Nonconvex World . Transactions on Machine Learning Research (TMLR), 2023, 2023

    work page 2023

  16. [16]

    Distributed learning with compressed gradients

    Sarit Khirirat, Hamid Reza Feyzmahdavian, and Mikael Johansson. Distributed learning with compressed gradients. arXiv preprint arXiv:1806.06573, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  17. [17]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    Pith/arXiv arXiv 2014

  18. [18]

    Analyzing & reducing the need for learning rate warmup in gpt training

    Atli Kosson, Bettina Messmer, and Martin Jaggi. Analyzing & reducing the need for learning rate warmup in gpt training. Advances in Neural Information Processing Systems, 37: 0 2914--2942, 2024

    work page 2024

  19. [19]

    Convex and Non -convex Optimization Under Generalized Smoothness

    Haochuan Li, Jian Qian, Yi Tian, Alexander Rakhlin, and Ali Jadbabaie. Convex and Non -convex Optimization Under Generalized Smoothness . In Conference on Neural Information Processing Systems ( NeurIPS ) , 2023 a

    work page 2023

  20. [20]

    Convergence of Adam Under Relaxed Assumptions

    Haochuan Li, Alexander Rakhlin, and Ali Jadbabaie. Convergence of Adam Under Relaxed Assumptions . In Conference on Neural Information Processing Systems ( NeurIPS ) , 2023 b

    work page 2023

  21. [21]

    On the Variance of the Adaptive Learning Rate and Beyond

    Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the Variance of the Adaptive Learning Rate and Beyond . In International Conference on Learning Representations ( ICLR ) , 2020

    work page 2020

  22. [22]

    Adagrad under anisotropic smoothness

    Yuxing Liu, Rui Pan, and Tong Zhang. Adagrad under anisotropic smoothness. arXiv preprint arXiv:2406.15244, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Revisiting the last-iterate convergence of stochastic gradient methods

    Zijian Liu and Zhengyuan Zhou. Revisiting the last-iterate convergence of stochastic gradient methods. arXiv preprint arXiv:2312.08531, 2023

    arXiv 2023

  24. [24]

    SGDR : Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. SGDR : Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=Skq89Scxx

    2017

  25. [25]

    Adaptive Gradient Descent without Descent

    Yura Malitsky and Konstantin Mishchenko. Adaptive Gradient Descent without Descent . In International Conference on Machine Learning ( ICML ) , pages 6702--6712, 2020

    work page 2020

  26. [26]

    Lectures on convex optimization, volume 137

    Yurii Nesterov et al. Lectures on convex optimization, volume 137. Springer, 2018

    2018

  27. [27]

    Global Convergence and Stability of Stochastic Gradient Descent

    Vivak Patel, Shushu Zhang, and Bowen Tian. Global Convergence and Stability of Stochastic Gradient Descent . In Conference on Neural Information Processing Systems ( NeurIPS ) , 2022

    work page 2022

  28. [28]

    Understanding Gradient Clipping In Incremental Gradient Methods

    Jiang Qian, Yuren Wu, Bojin Zhuang, Shaojun Wang, and Jing Xiao. Understanding Gradient Clipping In Incremental Gradient Methods . In International Conference on Artificial Intelligence and Statistics ( AISTATS ) , pages 1504--1512, 2021

    work page 2021

  29. [29]

    Smith, Pieter-Jan Kindermans, and Quoc V

    Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V. Le. Don't decay the learning rate, increase the batch size. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=B1Yy1BxCZ

    work page 2018

  30. [30]

    An elementary approach to tight worst case complexity analysis of gradient based methods

    Marc Teboulle and Yakov Vaisbourd. An elementary approach to tight worst case complexity analysis of gradient based methods. Mathematical Programming, 201 0 (1): 0 63--96, 2023

    work page 2023

  31. [31]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    Toward a Unified Theory of Gradient Descent under Generalized Smoothness

    Alexander Tyurin. Toward a Unified Theory of Gradient Descent under Generalized Smoothness . In Forty-second International Conference on Machine Learning , 2025

    work page 2025

  33. [33]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  34. [34]

    Understanding warmup-stable-decay learning rates: A river valley loss landscape perspective

    Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, and Tengyu Ma. Understanding warmup-stable-decay learning rates: A river valley loss landscape perspective. arXiv preprint arXiv:2410.05192, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Improved analysis of clipping algorithms for non-convex optimization

    Bohang Zhang, Jikai Jin, Cong Fang, and Liwei Wang. Improved analysis of clipping algorithms for non-convex optimization. Advances in Neural Information Processing Systems, 33: 0 15511--15521, 2020 a

    2020

  36. [36]

    Why Gradient Clipping Accelerates Training : A Theoretical Justification for Adaptivity

    Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why Gradient Clipping Accelerates Training : A Theoretical Justification for Adaptivity . In International Conference on Learning Representations ( ICLR ) , 2020 b

    work page 2020

  37. [37]

    On the convergence and improvement of stochastic normalized gradient descent

    Shen-Yi Zhao, Yin-Peng Xie, and Wu-Jun Li. On the convergence and improvement of stochastic normalized gradient descent. Science China Information Sciences, 64: 0 1--13, 2021

    work page 2021