Pith. sign in

REVIEW 3 major objections 4 minor 54 references

LightSAM: Parameter-Agnostic Sharpness-Aware Minimization

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that replacing SAM's SGD updates with adaptive optimizers removes the need to tune the perturbation radius and learning rate, giving $E\|\nabla f(x_t)\| \le O(\ln T / T^{1/4})$ for any positive choices.

desk verdict LightSAM's 'any ρ, η' claim is overstated — Theorem 3 schedules hyperparameters for the Adam variant, and Lemma 13's statement exceeds its proof — but the adaptive-perturbation idea and most of the machinery still warrant a major revision. read the letter →

arxiv 2505.24399 v1 pith:4ENDEG2K submitted 2025-05-30 cs.LG

Yifei Cheng , Li Shen , Hao Sun , Nan Yin , Xiaochun Cao , Enhong Chen This is my paper
classification cs.LG
keywords sharpness-awareminimizationparameter-agnosticadaptiveoptimizationnon-convexAdaGradAdamconvergencerateflatminima
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sharpness-Aware Minimization (SAM) improves generalization by perturbing weights, but it forces users to tune two hyperparameters—the perturbation radius and the learning rate—and existing theory ties both to unknowable problem constants. LightSAM makes both self-adapting by using an adaptive optimizer (AdaGrad-Norm, AdaGrad, or Adam) for both the perturbation step and the update step. The paper claims that under only L-smoothness and affine noise variance, LightSAM converges at $E\|\nabla f(x_t)\| \le O(\ln T / T^{1/4})$ for any positive $\rho$ and $\eta$, without the gradient-bounded assumption common in adaptive-optimizer proofs. If this holds, practitioners could skip hyperparameter search for SAM while keeping its flat-minima generalization benefit.

What carries the argument

The central object is the pair of accumulators $u_t = u_{t-1} + \|s_t\|^2$ and $v_t = v_{t-1} + \|g_t\|^2$, where $s_t = \nabla f(x_t, \xi_t)$ and $g_t = \nabla f(w_t, \xi_t)$ are the two stochastic gradients of each SAM iteration. The perturbation step uses $s_t/\sqrt{u_t}$ and the descent step uses $g_t/\sqrt{v_t}$, making the perturbation radius and learning rate adaptive schedules rather than constants. The proof machinery is the virtual sequence $\hat{w}_t = x_t + \rho \nabla f(x_t)/\sqrt{\hat{u}_t}$, which removes stochasticity from the perturbation, together with two lemmas that bound sums of $\|\nabla f(\hat{w}_t)\|^2$ against accumulator differences and a final comparison lemma $v_t \le (\rho L/\epsilon + 1)u_t$ that transfers the bound to the desired gradient norm.

What would settle it

Run Algorithm 1 on a one-dimensional quadratic $f(x) = (L/2)x^2$ with a fixed stochastic gradient, choose $\rho \gg \epsilon$ (for example $\rho=10$, $\epsilon=0.1$), and record $v_t/u_t$ across iterations. If $v_t/u_t$ ever exceeds $\rho L/\epsilon + 1$ while staying below $(\rho L/\epsilon + 1)^2$, the central Lemma 3 is false and the proof of Corollary 1 collapses.

Watch

Extended reading notes

Core claim

LightSAM's central claim is that the two-timescale SAM update—perturbation ascent $w_t = x_t + \rho s_t/\sqrt{u_t}$ followed by descent $x_{t+1} = x_t - \eta g_t/\sqrt{v_t}$—converges to a stationary point for any positive radius and learning rate once both step sizes are normalized by square roots of accumulated gradient norms. The paper proves this for three instantiations: scalar AdaGrad-Norm, coordinate-wise AdaGrad, and Adam-style momentum, with the Adam variant stated under order conditions $1-\beta_2 = O(T^{-1})$, $\eta = O(T^{-1/2})$, $\rho = O(T^{-1/2})$. The argument constructs a virtual sequence $\hat{w}_t$ using the true gradient in the perturbation, then two lemmas control the misalignment between gradient norms at $x_t$ and $w_t$, and a comparison bound converts the perturbed-weight accumulator into a rate on $E\|\nabla f(x_t)\|$. The resulting bound is $O(\ln T / T^{1/4})$, matching known adaptive-optimizer rates without the bounded-gradient assumption.

Load-bearing premise

The proof's final rate depends on the comparison between the two gradient accumulators, and the appendix only proves a weaker squared version of that comparison; if the linear version fails, the claimed rate does not follow.

Editorial extensions

If this is right

  • Users can run SAM without a separate tuning phase: the perturbation radius and learning rate adapt from gradient history in every iteration.
  • The same design transfers to three popular base optimizers, so practitioners can choose AdaGrad-Norm for simplicity, AdaGrad for coordinate-wise scaling, or Adam for transformer-style models.
  • The $O(\ln T / T^{1/4})$ rate matches prior adaptive-optimizer results while dropping the bounded-gradient assumption, extending SAM-style training to settings where gradient norms grow.
  • Fine-tuning experiments on ViT and RoBERTa show LightSAM matching or exceeding SAM and AdaSAM without hand-tuned schedules, suggesting immediate practical use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Theorem 3's order conditions mean the strictly unrestricted 'any values' claim applies to the AdaGrad-Norm and AdaGrad variants; the Adam version still asks for mild schedule conditions, a nuance the abstract flattens.
  • If the comparison between $v_t$ and $u_t$ can be made airtight, the same two-accumulator design could make other two-timescale optimizers—lookahead or extra-step methods—parameter-free by giving each scale its own adaptive accumulator.
  • A direct experiment could test whether adaptive perturbation preserves flat minima: compare the sharpness of LightSAM and SAM at matched test accuracy on the same checkpoint.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes LightSAM, a class of SAM variants that replace the SGD perturbation/update steps with adaptive optimizers: AdaGrad-Norm (LightSAM-I), AdaGrad (LightSAM-II), and Adam (LightSAM-III). The central claim is that, under L-smoothness and affine noise variance, all three variants converge at rate O(ln T / T^{1/4}) without any restrictions on the perturbation radius ρ or learning rate η, i.e., they are parameter-agnostic. Theorems 1 and 2 state such rates for LightSAM-I and II; Theorem 3 states a rate for LightSAM-III under additional scaling conditions. Experiments on MNIST, ImageNet fine-tuning, and GLUE compare LightSAM with SGD, SAM, ASAM, and AdaSAM, reporting competitive accuracy and lower sensitivity to hyperparameter changes.

Significance. If the main claim held, the paper would make a useful contribution: removing the need to tune the perturbation radius and learning rate in SAM would reduce a practical burden, and the analysis avoids the common bounded-gradient assumption. The paper also provides explicit convergence-rate statements and preliminary experiments across vision and NLP tasks. However, the advertised parameter-agnostic property is contradicted by the paper's own Theorem 3 for the Adam variant, and the proof for the AdaGrad variants relies on an inequality (Lemma 3 / Lemma 13) whose derivation is not supported as stated. These are load-bearing issues for the paper's central claim, so the work cannot be accepted in its present form.

major comments (3)
  1. [Section III-D, Theorem 3] Theorem 3 does not establish the claimed parameter-agnostic property. It proves convergence only under the explicit restrictions 1−β2 = O(T^{-1}), η = O(T^{-1/2}), ρ = O(T^{-1/2}), and β2 ≥ (√(D3²+4D3)−D3)/2, where D3 contains the problem-dependent constants D1 and C1 as well as ρ, η, β1, β2. Thus LightSAM-III's perturbation radius and learning rate must be scheduled as functions of T and must satisfy a problem-dependent lower bound on β2. This directly contradicts the abstract's statement that LightSAM converges with 'any choices of perturbation radius and learning rate,' and it contradicts the definition of parameter-agnostic given in footnote 1. Remark 3 acknowledges the constraints but does not resolve the contradiction; the theorem should be either removed from the parameter-agnostic claim or the claim must be revised to cover only LightSAM-I and II.
  2. [Section III-B, Lemma 3 / Appendix B, Lemma 13] Lemma 3 states v_t ≤ (ρL/ϵ + 1)u_t, and the proof of Theorem 4 uses this linear bound to replace 1/√v_t by a factor of √(ϵ/(ρL+ϵ)) times 1/√u_t. However, the proof of Lemma 13 in Appendix B ends with the inequality (ρL/√u_t + 1)² ≤ (ρL/ϵ + 1)², which supports only v_t ≤ (ρL/ϵ + 1)²u_t. The displayed linear inequality is not derived. Since this bound is used to convert the accumulated-gradient terms from the v-accumulator to the u-accumulator and to control logarithmic terms, the proof of the claimed rate is incomplete. The authors must either prove the linear bound or propagate the squared bound and show that the final O(ln T / T^{1/4}) rate still follows.
  3. [Section III-A, Assumption 2 and filtration definition] The filtration is defined as F_t = σ{s_1, g_1, ..., s_t, g_t}, which includes the current stochastic gradients. Under this definition, E_{F_t}∥∇f(x, ξ_t)∥² is simply the realized squared gradient, not a quantity satisfying the affine noise-variance bound D0 + D1∥∇f(x)∥² in general. Yet such conditional expectations are used throughout, for example in Lemma 10 and in the proof of Theorem 4. If the intended conditioning is on the history before sampling the current minibatch, the sigma algebra must be defined as F_{t-1} (or the proofs must use a different conditioning). This is a foundational issue affecting the stochasticity of all three theorems and must be corrected and re-verified.
minor comments (4)
  1. [Abstract] The phrase 'validate the the effectiveness' contains a duplicated article; please proofread the manuscript carefully.
  2. [Figure 1 caption] The caption contains corrupted Unicode tokens such as '/uni00000013' in place of axis labels; the figure and caption need to be regenerated.
  3. [Tables V and VII] Diverged runs are marked with '−', but the caption does not explain how divergence is defined or how the reported averages and standard deviations are computed over the remaining runs; please clarify.
  4. [Appendix B, proof of Theorem 2] The proof of Theorem 2 is stated to be 'almost the same' as that of Theorem 1 and is not repeated; given the coordinate-wise nature of LightSAM-II, the vector versions of Lemma 13 and the Cauchy-Schwarz steps should be stated explicitly rather than left to the reader.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the convergence proof is self-contained given the stated assumptions, with the only self-citation used for comparison rather than as load-bearing evidence.

full rationale

The central derivation does not reduce to its own inputs. Theorem 1 first bounds the weighted gradient sum via L-smoothness of f and Lemma 1 and Lemma 2, both proved in Appendix B from Assumptions 1 and 2; the target quantity is not assumed. The conversion from v-accumulator bounds to u-accumulator bounds relies on Lemma 3, and although Appendix B's Lemma 13 actually proves the square version v_t ≤ (1 + ρL/ϵ)^2 u_t rather than the linear version stated in the main text, this is a proof error about the constant, not a circular step: the inequality is derived from the perturbed-weight definition and the smoothness assumption, not imported from the conclusion. The final logarithmic rate is obtained by applying external lemmas from Wang et al. [15] and [16], which are non-self citations with independently stated assumptions. The paper's self-citation to AdaSAM [7] appears only in the related-work positioning and experimental comparison, not in any proof step that carries the parameter-agnostic claim. The main text claims hold for arbitrary ρ and η in Theorems 1 and 2, while Theorem 3 explicitly requires 1−β_2 = O(T^{-1}), η = O(T^{-1/2}), ρ = O(T^{-1/2}) and a problem-dependent lower bound on β_2; this contradicts the abstract's 'any choices' wording and is a serious correctness/consistency concern, but it is not circular because the restrictions are added conditions, not the theorem being reused as its own premise. Similarly, the unsupported linear Lemma 3 creates a gap in the stated constants for LightSAM-I and II, but the rate derivation would still follow with a modified constant if the square version were used, so this is not an instance of a prediction being equivalent to a fitted input or a self-citation chain forcing the result.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The algorithm introduces no new physical or mathematical entities. The proof relies on standard smoothness and noise assumptions plus one problematic technical inequality (Lemma 13) that is not justified by the derivation.

free parameters (3)
  • perturbation radius ρ
    Base radius in Algorithms 1-3; the theorem claims convergence for any positive ρ for variants I and II, and for ρ = O(T^{-1/2}) for variant III. Not fitted to data, but the constants in the bound depend on it.
  • learning rate η
    Base learning rate in the update step; the theorem claims convergence for any positive η for variants I and II, and for η = O(T^{-1/2}) for variant III. Not fitted to data.
  • stabilizer epsilon = unspecified small value
    Used as u0 = v0 = epsilon^2 to avoid division by zero; appears in denominators throughout the bounds. The paper only says it is 'a very small value' without a precise choice.
assumptions (4)
  • domain assumption f(·,ξ) is L-smooth for every ξ (Assumption 1).
    Used throughout the proofs to bound the difference between gradients at x_t and w_t.
  • domain assumption E[||∇f(x,ξ)||^2] ≤ D0 + D1||∇f(x)||^2 (Assumption 2).
    Affine noise variance assumption needed to control stochastic gradient norms without requiring a bounded gradient.
  • domain assumption Coordinate-wise L-smoothness and affine noise variance (Assumptions 3 and 4).
    Used for LightSAM-II and LightSAM-III.
  • ad hoc to paper The inequality v_t ≤ (ρL/epsilon + 1)u_t (Lemma 3 and Lemma 13).
    This bridge between the two adaptive accumulators is used to convert sums over v_t to sums over u_t. The derivation in Appendix B only justifies v_t ≤ (ρL/epsilon + 1)^2 u_t, so this axiom is unsupported and appears false.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LightSAM: Parameter-Agnostic Sharpness-Aware Minimization." pith.science (2026). https://pith.science/paper/4ENDEG2K

@misc{pith2026250524399,
  author       = {Pith},
  title        = {Pith review of: LightSAM: Parameter-Agnostic Sharpness-Aware Minimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4ENDEG2K}},
  note         = {Machine review of arXiv:2505.24399}
}
read the original abstract

Sharpness-Aware Minimization (SAM) optimizer enhances the generalization ability of the machine learning model by exploring the flat minima landscape through weight perturbations. Despite its empirical success, SAM introduces an additional hyper-parameter, the perturbation radius, which causes the sensitivity of SAM to it. Moreover, it has been proved that the perturbation radius and learning rate of SAM are constrained by problem-dependent parameters to guarantee convergence. These limitations indicate the requirement of parameter-tuning in practical applications. In this paper, we propose the algorithm LightSAM which sets the perturbation radius and learning rate of SAM adaptively, thus extending the application scope of SAM. LightSAM employs three popular adaptive optimizers, including AdaGrad-Norm, AdaGrad and Adam, to replace the SGD optimizer for weight perturbation and model updating, reducing sensitivity to parameters. Theoretical results show that under weak assumptions, LightSAM could converge ideally with any choices of perturbation radius and learning rate, thus achieving parameter-agnostic. We conduct preliminary experiments on several deep learning tasks, which together with the theoretical findings validate the the effectiveness of LightSAM.

Figures

Figures reproduced from arXiv: 2505.24399 by the authors.

Figure 1
Figure 1. Experimental results of fine-tuning ViT models on Imagenet. (a): Test accuracy w.r.t. epochs for ViT-Tiny; (b): Train loss w.r.t. epochs for ViT-Tiny; [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 40 canonical work pages

  1. [15]

    Convergence of adagrad for non-convex objectives: Simple proofs and relaxed assumptions,

    B. Wang, H. Zhang, Z. Ma, and W. Chen, “Convergence of adagrad for non-convex objectives: Simple proofs and relaxed assumptions,” in The Thirty Sixth Annual Conference on Learning Theory . PMLR, 2023, pp. 161–190

    work page 2023

  2. [16]

    Closing the gap be- tween the upper bound and lower bound of adam’s iteration complexity,

    B. Wang, J. Fu, H. Zhang, N. Zheng, and W. Chen, “Closing the gap be- tween the upper bound and lower bound of adam’s iteration complexity,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023

    work page 2023

  3. [1]

    On large-batch training for deep learning: Generalization gap and sharp minima,

    N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang, “On large-batch training for deep learning: Generalization gap and sharp minima,” arXiv preprint arXiv:1609.04836 , 2016

    arXiv 2016

  4. [2]

    Explor- ing generalization in deep learning,

    B. Neyshabur, S. Bhojanapalli, D. McAllester, and N. Srebro, “Explor- ing generalization in deep learning,” Advances in neural information processing systems, vol. 30, 2017

    work page 2017

  5. [3]

    Sharpness-aware minimization for efficiently improving generalization,

    P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur, “Sharpness-aware minimization for efficiently improving generalization,” in International Conference on Learning Representations , 2020

    work page 2020

  6. [4]

    Towards understanding sharpness-aware minimization,

    M. Andriushchenko and N. Flammarion, “Towards understanding sharpness-aware minimization,” in International Conference on Machine Learning. PMLR, 2022, pp. 639–668

    2022

  7. [5]

    Make sharpness-aware minimization stronger: A sparsified perturbation ap- proach,

    P. Mi, L. Shen, T. Ren, Y . Zhou, X. Sun, R. Ji, and D. Tao, “Make sharpness-aware minimization stronger: A sparsified perturbation ap- proach,” Advances in Neural Information Processing Systems , vol. 35, pp. 30 950–30 962, 2022

    work page 2022

  8. [6]

    Critical Influence of Overparameterization on Sharpness-aware Minimization

    S. Shin, D. Lee, M. Andriushchenko, and N. Lee, “The effects of overparameterization on sharpness-aware minimization: An empirical and theoretical analysis,” arXiv preprint arXiv:2311.17539 , 2023

    work page Pith review arXiv 2023

Show all 54 references
  1. [7]

    Adasam: Boosting sharpness-aware minimization with adaptive learning rate and momentum for training deep neural networks,

    H. Sun, L. Shen, Q. Zhong, L. Ding, S. Chen, J. Sun, J. Li, G. Sun, and D. Tao, “Adasam: Boosting sharpness-aware minimization with adaptive learning rate and momentum for training deep neural networks,” Neural Networks, vol. 169, pp. 506–519, 2024

    2024

  2. [8]

    The crucial role of normalization in sharpness-aware minimization,

    Y . Dai, K. Ahn, and S. Sra, “The crucial role of normalization in sharpness-aware minimization,” arXiv preprint arXiv:2305.15287, 2023

    2023 arXiv

  3. [9]

    On the convergence of adam under non-uniform smoothness: Separability from sgdm and beyond,

    B. Wang, H. Zhang, Q. Meng, R. Sun, Z.-M. Ma, and W. Chen, “On the convergence of adam under non-uniform smoothness: Separability from sgdm and beyond,” arXiv preprint arXiv:2403.15146 , 2024

    2024 arXiv

  4. [10]

    Parameter-agnostic optimization under relaxed smoothness,

    F. H ¨ubler, J. Yang, X. Li, and N. He, “Parameter-agnostic optimization under relaxed smoothness,” in International Conference on Artificial Intelligence and Statistics . PMLR, 2024, pp. 4861–4869

    2024

  5. [11]

    Simultaneous model selection and optimization through parameter-free stochastic learning,

    F. Orabona, “Simultaneous model selection and optimization through parameter-free stochastic learning,” Advances in Neural Information Processing Systems, vol. 27, 2014

    2014

  6. [12]

    Online learning without prior informa- tion,

    A. Cutkosky and K. Boahen, “Online learning without prior informa- tion,” in Conference on learning theory . PMLR, 2017, pp. 643–677

    2017

  7. [13]

    Training deep networks without learning rates through coin betting,

    F. Orabona and T. Tommasi, “Training deep networks without learning rates through coin betting,” Advances in Neural Information Processing Systems, vol. 30, 2017

    2017

  8. [14]

    Learning-rate-free learning by d- adaptation,

    A. Defazio and K. Mishchenko, “Learning-rate-free learning by d- adaptation,” arXiv preprint arXiv:2301.07733 , 2023

    2023 arXiv

  9. [17]

    How sharpness-aware minimization mini- mizes sharpness?

    K. Wen, T. Ma, and Z. Li, “How sharpness-aware minimization mini- mizes sharpness?” in The eleventh international conference on learning representations, 2023

    2023

  10. [18]

    Asam: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural net- works,

    J. Kwon, J. Kim, H. Park, and I. K. Choi, “Asam: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural net- works,” in International Conference on Machine Learning . PMLR, 2021, pp. 5905–5914

    2021

  11. [19]

    Ran- dom sharpness-aware minimization,

    Y . Liu, S. Mai, M. Cheng, X. Chen, C.-J. Hsieh, and Y . You, “Ran- dom sharpness-aware minimization,” Advances in Neural Information Processing Systems, vol. 35, pp. 24 543–24 556, 2022

    2022

  12. [20]

    Sharpness-aware training for free,

    J. Du, D. Zhou, J. Feng, V . Tan, and J. T. Zhou, “Sharpness-aware training for free,” Advances in Neural Information Processing Systems , vol. 35, pp. 23 439–23 451, 2022

    2022

  13. [21]

    Surrogate gap minimization improves sharpness-aware training,

    J. Zhuang, B. Gong, L. Yuan, Y . Cui, H. Adam, N. Dvornek, S. Tatikonda, J. Duncan, and T. Liu, “Surrogate gap minimization improves sharpness-aware training,” arXiv preprint arXiv:2203.08065 , 2022

    2022 arXiv

  14. [22]

    Sharpness-aware minimization with adaptive regularization for training deep neural networks,

    J. Zou, X. Deng, and T. Sun, “Sharpness-aware minimization with adaptive regularization for training deep neural networks,”arXiv preprint arXiv:2412.16854, 2024

    2024 arXiv

  15. [23]

    Sampa: Sharpness-aware minimiza- tion parallelized,

    W. Xie, T. Pethick, and V . Cevher, “Sampa: Sharpness-aware minimiza- tion parallelized,” arXiv preprint arXiv:2410.10683 , 2024

    2024 arXiv

  16. [24]

    Sharpness-aware lookahead for accelerating convergence and improving generalization,

    C. Tan, J. Zhang, J. Liu, and Y . Gong, “Sharpness-aware lookahead for accelerating convergence and improving generalization,” IEEE Transac- tions on Pattern Analysis and Machine Intelligence , 2024

    2024

  17. [25]

    On the convergence of adam and beyond,

    S. J. Reddi, S. Kale, and S. Kumar, “On the convergence of adam and beyond,” arXiv preprint arXiv:1904.09237 , 2019

    1904 arXiv

  18. [26]

    Adaptive subgradient methods for online learning and stochastic optimization

    J. Duchi, E. Hazan, and Y . Singer, “Adaptive subgradient methods for online learning and stochastic optimization.” Journal of machine learning research, vol. 12, no. 7, 2011. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 11

    2011

  19. [27]

    Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude,

    T. Tieleman, “Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude,” COURSERA: Neural networks for machine learning, vol. 4, no. 2, p. 26, 2012

    2012

  20. [28]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014

    2014 arXiv

  21. [29]

    On the convergence of stochastic gradient descent with adaptive stepsizes,

    X. Li and F. Orabona, “On the convergence of stochastic gradient descent with adaptive stepsizes,” in The 22nd international conference on artificial intelligence and statistics . PMLR, 2019, pp. 983–992

    2019

  22. [30]

    On the convergence of a class of adam-type algorithms for non-convex optimization,

    X. Chen, S. Liu, R. Sun, and M. Hong, “On the convergence of a class of adam-type algorithms for non-convex optimization,” arXiv preprint arXiv:1808.02941, 2018

    2018 arXiv

  23. [31]

    On the convergence of adaptive gradient methods for nonconvex optimization,

    D. Zhou, J. Chen, Y . Cao, Y . Tang, Z. Yang, and Q. Gu, “On the convergence of adaptive gradient methods for nonconvex optimization,” arXiv preprint arXiv:1808.05671 , 2018

    2018 arXiv

  24. [32]

    A simple convergence proof of adam and adagrad,

    A. D ´efossez, L. Bottou, F. Bach, and N. Usunier, “A simple convergence proof of adam and adagrad,” arXiv preprint arXiv:2003.02395 , 2020

    2003 arXiv

  25. [33]

    A unified analysis of adagrad with weighted aggregation and momentum acceleration,

    L. Shen, C. Chen, F. Zou, Z. Jie, J. Sun, and W. Liu, “A unified analysis of adagrad with weighted aggregation and momentum acceleration,” IEEE Transactions on Neural Networks and Learning Systems , 2023

    2023

  26. [34]

    Rmsprop converges with proper hyperparameter,

    N. Shi and D. Li, “Rmsprop converges with proper hyperparameter,” in International conference on learning representation , 2021

    2021

  27. [35]

    Adam can converge without any modification on update rules,

    Y . Zhang, C. Chen, N. Shi, R. Sun, and Z.-Q. Luo, “Adam can converge without any modification on update rules,” Advances in Neural Information Processing Systems , vol. 35, pp. 28 386–28 399, 2022

    2022

  28. [36]

    Dimension-free exponentiated gradient,

    F. Orabona, “Dimension-free exponentiated gradient,” Advances in Neu- ral Information Processing Systems , vol. 26, 2013

    2013

  29. [37]

    Unconstrained online linear learning in hilbert spaces: Minimax algorithms and normal approximations,

    H. B. McMahan and F. Orabona, “Unconstrained online linear learning in hilbert spaces: Minimax algorithms and normal approximations,” in Conference on Learning Theory . PMLR, 2014, pp. 1020–1039

    2014

  30. [38]

    Coin betting and parameter-free online learn- ing,

    F. Orabona and D. P ´al, “Coin betting and parameter-free online learn- ing,” Advances in Neural Information Processing Systems, vol. 29, 2016

    2016

  31. [39]

    Making sgd parameter-free,

    Y . Carmon and O. Hinder, “Making sgd parameter-free,” in Conference on Learning Theory . PMLR, 2022, pp. 2360–2389

    2022

  32. [40]

    Dog is sgd’s best friend: A parameter-free dynamic step size schedule,

    M. Ivgi, O. Hinder, and Y . Carmon, “Dog is sgd’s best friend: A parameter-free dynamic step size schedule,” in International Conference on Machine Learning . PMLR, 2023, pp. 14 465–14 499

    2023

  33. [41]

    Dowg unleashed: An efficient universal parameter-free gradient descent method,

    A. Khaled, K. Mishchenko, and C. Jin, “Dowg unleashed: An efficient universal parameter-free gradient descent method,” Advances in Neural Information Processing Systems , vol. 36, pp. 6748–6769, 2023

    2023

  34. [42]

    Towards simple and provable parameter-free adaptive gradient methods,

    Y . Tao, H. Yuan, X. Zhou, Y . Cao, and Q. Gu, “Towards simple and provable parameter-free adaptive gradient methods,” arXiv preprint arXiv:2412.19444, 2024

    2024 arXiv

  35. [43]

    Sgd and hogwild! convergence without the bounded gra- dients assumption,

    L. Nguyen, P. H. Nguyen, M. Dijk, P. Richt ´arik, K. Scheinberg, and M. Tak ´ac, “Sgd and hogwild! convergence without the bounded gra- dients assumption,” in International Conference on Machine Learning . PMLR, 2018, pp. 3750–3758

    2018

  36. [44]

    Smoothness- adaptive sharpness-aware minimization for finding flatter minima,

    H. Naganuma, J. L. Kim, A. Kyrillidis, and I. Mitliagkas, “Smoothness- adaptive sharpness-aware minimization for finding flatter minima,” in 5th Workshop on practical ML for limited/low resource settings

  37. [45]

    Online to offline conversions, universality and adaptive minibatch sizes,

    K. Levy, “Online to offline conversions, universality and adaptive minibatch sizes,” Advances in Neural Information Processing Systems , vol. 30, 2017

    2017

  38. [46]

    Adagrad stepsizes: Sharp conver- gence over nonconvex landscapes,

    R. Ward, X. Wu, and L. Bottou, “Adagrad stepsizes: Sharp conver- gence over nonconvex landscapes,” The Journal of Machine Learning Research, vol. 21, no. 1, pp. 9047–9076, 2020

    2020

  39. [47]

    A sufficient condition for convergences of adam and rmsprop,

    F. Zou, L. Shen, Z. Jie, W. Zhang, and W. Liu, “A sufficient condition for convergences of adam and rmsprop,” in Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , 2019, pp. 11 127–11 135

    2019

  40. [48]

    Iteration complexity of randomized block- coordinate descent methods for minimizing a composite function,

    P. Richt ´arik and M. Tak ´aˇc, “Iteration complexity of randomized block- coordinate descent methods for minimizing a composite function,” Mathematical Programming, vol. 144, no. 1-2, pp. 1–38, 2014

    2014

  41. [49]

    Towards quantifying the preconditioning effect of adam,

    R. Das, N. Agarwal, S. Sanghavi, and I. S. Dhillon, “Towards quantifying the preconditioning effect of adam,” arXiv preprint arXiv:2402.07114, 2024

    2024 arXiv

  42. [50]

    Robust- ness to unbounded smoothness of generalized signsgd,

    M. Crawshaw, M. Liu, F. Orabona, W. Zhang, and Z. Zhuang, “Robust- ness to unbounded smoothness of generalized signsgd,” Advances in Neural Information Processing Systems , vol. 35, pp. 9955–9968, 2022

    2022

  43. [51]

    Gradient-based learning applied to document recognition,

    Y . LeCun, L. Bottou, Y . Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE , vol. 86, no. 11, pp. 2278–2324, 1998

    1998

  44. [52]

    Training data-efficient image transformers & distillation through attention,

    H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. J ´egou, “Training data-efficient image transformers & distillation through attention,” in International conference on machine learning . PMLR, 2021, pp. 10 347–10 357

    2021

  45. [53]

    Why are adaptive methods good for attention models?

    J. Zhang, S. P. Karimireddy, A. Veit, S. Kim, S. Reddi, S. Kumar, and S. Sra, “Why are adaptive methods good for attention models?” Ad- vances in Neural Information Processing Systems , vol. 33, pp. 15 383– 15 393, 2020

    2020

  46. [54]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, vol. 364, 2019. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 12 APPENDIX A USEFUL INEQUALITIES We first show some inequalities which are useful for our analysis. L...

    1907 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.