Pith. sign in

REVIEW 4 minor 72 references

Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks

T0 review · 0 major / 4 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single unified iteration covers Adam and a dozen other DNN optimizers and forces every bounded trajectory to a critical point at an explicit polynomial rate.

desk verdict Solid unified strong-convergence theory for Adam and a dozen practical optimizers under KL + bounded trajectories, free of the usual momentum-step-size coupling. read the letter →

arxiv 2607.04233 v1 pith:PHSYUZK7 submitted 2026-07-05 math.OC cs.LG

Shokhrukh Ibragimov , Arnulf Jentzen This is my paper
classification math.OCcs.LG MSC 90C2668T0765K05
keywords unifiedgradientdescentKurdyka-ŁojasiewiczinequalityAdamoptimizerdeepneuralnetworksstrongconvergenceratesanalyticactivationsadaptiveoptimizers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper shows that the optimizers practitioners actually use for deep networks—Adam, RMSprop, NAG, Adan, AdaBelief, AMSGrad, Yogi and several variants—are all special cases of one abstract update rule called the unified gradient-descent (UGD) method. Under the single geometric assumption that the loss is a Kurdyka-Łojasiewicz function with locally Lipschitz gradient (satisfied by any network with analytic activations such as softplus or GeLU), every bounded UGD trajectory converges to a critical point. The convergence is quantitative: the gradient, the function-value gap and a power of the distance to the limit all decay like the reciprocal of the partial sums of the step-size sequence. Because the same proof covers both classical momentum methods and modern adaptive methods without coupling their hyperparameters to the learning rate, the result supplies the first strong-convergence guarantee for Adam that works under the natural, decoupled schedules used in practice.

What carries the argument

The UGD recursion: a single momentum-style update with an abstract positive-definite matrix A_n and a perturbation sequence µ_n that simultaneously encodes every listed adaptive or accelerated method; once the trajectory is bounded, a KL inequality upgrades weak (function-value) convergence into strong (iterate) convergence with rates.

What would settle it

Exhibit a concrete analytic DNN loss (softplus or GeLU network) and a set of Adam hyperparameters for which a bounded Adam trajectory either fails to approach a critical point or decays slower than any multiple of 1/∑ j^{- u} for u>3/4.

Watch

Extended reading notes

Core claim

Every bounded trajectory of the UGD iteration (which specialises to Adam, RMSprop, NAG, Adan, AdaBelief, AMSGrad, Yogi, …) applied to a Kurdyka-Łojasiewicz objective with locally Lipschitz gradient converges to a critical point at the explicit rate O(1/∑_{j=1}^n j^{- u}) for any u∈(3/4,1].

Load-bearing premise

The entire optimisation trajectory, together with the auxiliary adaptive matrices and momentum terms, must remain bounded for all time; without that a-priori bound the rates disappear.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 4 minor

Summary. The paper develops a unified gradient-descent (UGD) framework (Setting 2.1, eqs. (2.1)–(2.2)) that encompasses classical GD, momentum, NAG, RMSprop, Adam, Adamax, Nadam, Nadamax, Adan, AdaBelief, AMSGrad, Yogi and the explicit midpoint method. Under the standing hypothesis that the iterates and auxiliary sequences remain bounded, and for any KL objective with locally Lipschitz gradient, Theorem 3.9 (and its special case Theorem 1.1) proves that every such trajectory converges to a critical point at the explicit rate O(1/∑_{j=1}^n j^{- u}) for u∈(3/4,1]. The argument proceeds by a weak-convergence step (Corollary 2.9) based on a Taylor remainder and monotone-convergence estimates, followed by a quantitative KL descent lemma (Proposition 3.2) that yields the rates. The framework is then specialized to analytic DNN training losses (Corollaries 3.13–3.14, 4.5) and to each of the listed optimizers (Section 4).

Significance. If the result holds, it supplies the first fully rigorous, rate-equipped strong-convergence theory that covers the entire practical family of adaptive and accelerated first-order methods under a single set of hypotheses. The proofs are self-contained real-analysis arguments that stay inside the classical KL toolkit; the only modelling assumption that is not automatic is boundedness of trajectories, which the authors correctly flag and for which they cite companion works that give sufficient conditions. The explicit polynomial rates and the clean specialization to analytic DNN losses (softplus, GeLU, etc.) make the contribution immediately usable for the analysis of modern AI optimizers. The machine-readable structure of the proofs and the absence of circular definitions further strengthen the paper’s value as a reference result.

minor comments (4)
  1. The bounded-trajectory hypothesis is stated clearly from Setting 2.1 onward and is essential; a short remark in the introduction (or after Theorem 1.1) that points the reader more explicitly to the companion papers [14,17] for concrete sufficient conditions would improve accessibility without changing any claim.
  2. In several places (e.g., the statement of Theorem 1.1 and Corollary 3.11) the matrix process A_n is required only to satisfy that A_n-I is positive semi-definite; a one-sentence clarification that this covers both constant learning-rate matrices and the usual diagonal adaptive scalings would help non-specialist readers.
  3. Typographical consistency: the symbol for the objective is sometimes L and sometimes script L; standardizing to one notation throughout would remove a minor source of visual friction.
  4. The literature overview (Subsection 1.3) is thorough; a brief sentence distinguishing the present deterministic, hyperparameter-decoupled setting from the stochastic, learning-rate-coupled analyses of Barakat–Bianchi and others would make the novelty claim even sharper.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the unified UGD convergence rates are derived directly from the KL inequality and elementary estimates under an explicit bounded-trajectory hypothesis.

full rationale

The paper's central claim (Theorem 3.9 / Theorem 1.1) is an implication: if L is a KL function with locally Lipschitz gradient and the UGD iterates (together with the auxiliary sequences A_n, p_n, µ_n) remain bounded, then every such trajectory converges to a critical point at the explicit rate O(1/∑ j^{-ν}). Boundedness is stated as a standing hypothesis from Setting 2.1 onward and is used only to obtain local Hölder/Lipschitz constants and to apply the KL inequality on a compact set; the authors explicitly flag companion works that supply sufficient conditions for it. The weak-convergence step (Corollary 2.9, via Propositions 2.3–2.4 and 2.7) and the subsequent strong-convergence argument (Proposition 3.2 through Theorem 3.9) consist of self-contained Taylor remainders, geometric-series bounds, and standard KL descent estimates; none of these steps defines a quantity in terms of the claimed rate or imports a uniqueness theorem that forces the result. Self-citations appear only as background or as optional sufficient conditions for the boundedness hypothesis and are not load-bearing for the derivation itself. Consequently the derivation chain does not reduce to its inputs by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

The central claim rests on three standard analytic facts (analytic functions are KL; local Lipschitz of gradients of analytic maps; elementary Taylor remainder) plus the modelling hypotheses that the trajectory stays bounded and that the adaptive matrices A_n dominate the identity. No free parameters are fitted; the decay exponent u > 3/4 is an explicit hypothesis on the learning-rate schedule.

assumptions (4)
  • standard math Every real-analytic function R^d o R is a Kurdyka-Łojasiewicz function (Łojasiewicz 1965 / Bolte et al.).
    Invoked in §3.1 and Cor. 3.13–3.14 to transfer the abstract KL result to DNN empirical risk with softplus/GeLU activations.
  • standard math The gradient of a C^1 function with locally Hölder continuous derivative admits a standard Taylor remainder bound of order 1+δ.
    Lemma 2.2; used throughout the weak-convergence arguments.
  • domain assumption The optimization trajectory ( heta_n) and the auxiliary sequences (A_n, m_n, ho_n) remain bounded for all n.
    Standing hypothesis from Setting 2.1 onward; without it both weak and strong convergence fail.
  • domain assumption A_n - I_d is symmetric positive semi-definite for every n (or a positive multiple after rescaling by the learning rate).
    Ensures the descent inequality abla L · (A_n abla L) ≥ || abla L||^{2}; appears in every main theorem.
invented entities (1)
  • Unified Gradient Descent (UGD) iteration
    purpose: Single abstract recursion that simultaneously specialises to GD, momentum, NAG, RMSprop, Adam, Adamax, Nadam, Nadamax, Adan, AdaBelief, AMSGrad and Yogi by choice of the matrix sequence A_n and the momentum weights.
    The whole paper is organised around this abstraction; it is not claimed to be a new practical algorithm but a proof device.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks." pith.science (2026). https://pith.science/paper/PHSYUZK7

@misc{pith2026260704233,
  author       = {Pith},
  title        = {Pith review of: Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PHSYUZK7}},
  note         = {Machine review of arXiv:2607.04233}
}
read the original abstract

Gradient based optimization methods are nowadays the methods of choice for training deep neural networks (DNNs) in artificial intelligence (AI) systems. In practically relevant DNN training problems, one does usually not apply the standard gradient descent (GD) optimization method but instead one employs suitable sophisticated GD optimization methods, which incorporate adaptivity and/or acceleration techniques, such as the famous Adam optimizer. It is a key contribution of this work to provide a general unified convergence analysis for GD optimization methods in the training of DNNs with analytic activations such as the softplus and the popular Gaussian error linear unit (GeLU) activation. Our general unified convergence result applies to a large class of gradient based optimization methods such as the standard GD, the momentum, the Nesterov accelerated gradient (NAG), the RMSprop, the Adam, the Adamax, the Nadam, the Nadamax, the Adan, the AdaBelief, the AMSGrad, and the Yogi optimizers. Our analysis employs the theory of Kurdyka-{\L}ojasiewicz (KL) inequalities to establish convergence to critical points in the training of DNNs. To the best of our knowledge, the generality of our convergence analysis is also just in the special situation of the Adam optimizer a new contribution to the literature on the analysis of AI optimization algorithms.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

72 extracted references · 41 linked inside Pith

  1. [1]

    Convergence of the iterates of descent methods for analytic cost functions

    Absil, P.-A., Mahony, R., and Andrews, B. Convergence of the iterates of descent methods for analytic cost functions. SIAM J. Optim. 16 , 2 (2005), 531--547

    2005

  2. [2]

    J., Fleming, R

    Arag\'on Artacho, F. J., Fleming, R. M. T., and Vuong, P. T. Accelerating the DC algorithm for smooth functions. Math. Program. 169 , 1 (2018), 95--118

    2018

  3. [3]

    Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization

    Barakat, A., and Bianchi, P. Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization . arXiv:1911.07596 https://arxiv.org/abs/1911.07596\/ (2020)

    arXiv 1911

  4. [4]

    Convergence and dynamical behavior of the A dam algorithm for nonconvex stochastic optimization

    Barakat, A., and Bianchi, P. Convergence and dynamical behavior of the A dam algorithm for nonconvex stochastic optimization. SIAM J. Optim. 31 , 1 (2021), 244--274

    2021

  5. [5]

    Stochastic optimization with momentum: convergence, fluctuations, and traps avoidance

    Barakat, A., Bianchi, P., Hachem, W., and Schechtman, S. Stochastic optimization with momentum: convergence, fluctuations, and traps avoidance . Electron. J. Stat. 15 , 2 (2021), 3892--3947

    2021

  6. [6]

    Adaptive algorithms and stochastic approximations , vol

    Benveniste, A., M\'etivier, M., and Priouret, P. Adaptive algorithms and stochastic approximations , vol. 22 of Applications of Mathematics (New York) . Springer-Verlag, Berlin, 1990. Translated from the French by Stephen S. Wilson

    1990

  7. [7]

    signSGD: Compressed Optimisation for Non-Convex Problems

    Bernstein, J., Wang, Y.-X., Azizzadenesheli, K., and Anandkumar, A. signSGD: Compressed Optimisation for Non-Convex Problems . arXiv:1802.04434 https://arxiv.org/abs/1802.04434\/ (2018)

    arXiv 2018

  8. [8]

    Bierstone, E., and Milman, P. D. Semianalytic and subanalytic sets. Inst. Hautes \'Etudes Sci. Publ. Math. 67\/ (1988), 5--42

    1988

Show all 72 references
  1. [9]

    A Proof of Local Convergence for the Adam Optimizer

    Bock, S., and Weiß, M. A Proof of Local Convergence for the Adam Optimizer . In 2019 International Joint Conference on Neural Networks (IJCNN)\/ (2019), pp. 1--8

    2019

  2. [10]

    The ojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems

    Bolte, J., Daniilidis, A., and Lewis, A. The ojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems . SIAM J. Optim. 17 , 4 (2006), 1205--1223

    2006

  3. [11]

    On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization

    Chen, X., Liu, S., Sun, R., and Hong, M. On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization . arXiv:1808.02941 https://arxiv.org/abs/1808.02941\/ (2018)

    2018 arXiv

  4. [12]

    Robustness to Unbounded Smoothness of Generalized SignSGD

    Crawshaw, M., Liu, M., Orabona, F., Zhang, W., and Zhuang, Z. Robustness to Unbounded Smoothness of Generalized SignSGD . arXiv:2208.11195 https://arxiv.org/abs/2208.11195\/ (2022)

    2022 arXiv

  5. [13]

    A Simple Convergence Proof of Adam and Adagrad

    D \'e fossez, A., Bottou, L., Bach, F., and Usunier, N. A Simple Convergence Proof of Adam and Adagrad . Transactions on Machine Learning Research\/ (2022)

    2022

  6. [14]

    Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method

    Dereich, S., Do, T., and Jentzen, A. Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method . arXiv:2603.18899 https://arxiv.org/abs/2603.18899\/ (2026)

    2026

  7. [15]

    Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer

    Dereich, S., Do, T., Jentzen, A., and von Wurstemberger , P. Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer . arXiv:2511.06675 https://arxiv.org/abs/2511.06675\/ (2025)

    2025 arXiv

  8. [16]

    Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rates

    Dereich, S., Graeber, R., and Jentzen, A. Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rates . arXiv:2407.08100 https://arxiv.org/abs/2407.08100\/ (2024)

    2024 arXiv

  9. [17]

    Asymptotic stability properties and a priori bounds for Adam and other gradient descent optimization methods

    Dereich, S., Graeber, R., Jentzen, A., and Riekert, A. Asymptotic stability properties and a priori bounds for Adam and other gradient descent optimization methods . arXiv:2509.10476 https://arxiv.org/abs/2509.10476\/ (2025)

    2025

  10. [18]

    Convergence rates for the Adam optimizer

    Dereich, S., and Jentzen, A. Convergence rates for the Adam optimizer . arXiv:2407.21078 https://arxiv.org/abs/2407.21078\/ (2024)

    2024 arXiv

  11. [19]

    ODE approximation for the Adam algorithm: General and overparametrized setting

    Dereich, S., Jentzen, A., and Kassing, S. ODE approximation for the Adam algorithm: General and overparametrized setting . arXiv:2511.04622 https://arxiv.org/abs/2511.04622\/ (2025)

    2025

  12. [20]

    Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses

    Dereich, S., Jentzen, A., and Riekert, A. Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses . arXiv:2406.14340 https://arxiv.org/abs/2406.14340\/ (2024)

    2024 arXiv

  13. [21]

    Sharp higher order convergence rates for the Adam optimizer

    Dereich, S., Jentzen, A., and Riekert, A. Sharp higher order convergence rates for the Adam optimizer . arXiv:2504.19426 https://arxiv.org/abs/2504.19426\/ (2025)

    2025 arXiv

  14. [22]

    Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapes

    Dereich, S., and Kassing, S. Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapes . arXiv:2102.09385 https://arxiv.org/abs/2102.09385\/ (2024)

    2024 arXiv

  15. [23]

    Incorporating Nesterov Momentum into Adam

    Dozat, T. Incorporating Nesterov Momentum into Adam . https://openreview.net/forum?id=OM0jvwB8jIp57ZJjtNEZ, 2016. [Accessed 14-November-2025]

    2016

  16. [24]

    R., Singh, S

    Dubey, S. R., Singh, S. K., and Chaudhuri, B. B. AdaNorm: Adaptive Gradient Norm Correction based Optimizer for CNNs . arXiv:2210.06364 https://arxiv.org/abs/2210.06364\/ (2022)

    2022 arXiv

  17. [25]

    The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variance

    Faw, M., Tziotis, I., Caramanis, C., Mokhtari, A., Shakkottai, S., and Ward, R. The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variance . arXiv:2202.05791 https://arxiv.org/abs/2202.05791\/ (2022)

    2022 arXiv

  18. [26]

    Garrigos, G., and Gower, R. M. Handbook of Convergence Theorems for (Stochastic) Gradient Methods . arXiv:2301.11235 https://arxiv.org/abs/2301.11235\/ (2023)

    2023 arXiv

  19. [27]

    Convergence rates for momentum stochastic gradient descent with noise of machine learning type

    Gess, B., and Kassing, S. Convergence rates for momentum stochastic gradient descent with noise of machine learning type . arXiv:2302.03550 https://arxiv.org/abs/2302.03550\/ (2023)

    2023 arXiv

  20. [28]

    R., and Johansson, M

    Ghadimi, E., Feyzmahdavian, H. R., and Johansson, M. Global convergence of the Heavy-ball method for convex optimization . In 2015 European Control Conference (ECC)\/ (2015), pp. 310--315

    2015

  21. [29]

    Non asymptotic analysis of Adaptive stochastic gradient algorithms and applications

    Godichon-Baggioni, A., and Tarrago, P. Non asymptotic analysis of Adaptive stochastic gradient algorithms and applications . arXiv:2303.01370 https://arxiv.org/abs/2303.01370\/ (2023)

    2023 arXiv

  22. [30]

    Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case

    He, M., Liang, Y., Liu, J., and Xu, D. Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case . arXiv:2307.11782 https://arxiv.org/abs/2307.11782\/ (2023)

    2023 arXiv

  23. [31]

    Lecture 6e: RMSprop: Divide the gradient by a running average of its recent magnitude

    Hinton, G., Srivastava, N., and Swersky, K. Lecture 6e: RMSprop: Divide the gradient by a running average of its recent magnitude . https://www.cs.toronto.edu/ tijmen/csc321/slides/lecture_slides_lec6.pdf. [Accessed 14-November-2025]

    2025

  24. [32]

    Revisiting Convergence of AdaGrad with Relaxed Assumptions

    Hong, Y., and Lin, J. Revisiting Convergence of AdaGrad with Relaxed Assumptions . arXiv:2402.13794 https://arxiv.org/abs/2402.13794\/ (2024)

    2024 arXiv

  25. [33]

    Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

    Jentzen, A., Kuckuck, B., and von Wurstemberger, P. Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory . arXiv:2310.20360 https://arxiv.org/abs/2310.20360\/ (2023)

    2023 arXiv

  26. [34]

    On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks

    Jentzen, A., and Riekert, A. On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks. J. Mach. Learn. 1 , 2 (2022), 141--246

    2022

  27. [35]

    Lower error bounds for the stochastic gradient descent optimization algorithm: sharp convergence rates for slowly and fast decaying learning rates

    Jentzen, A., and von Wurstemberger, P. Lower error bounds for the stochastic gradient descent optimization algorithm: sharp convergence rates for slowly and fast decaying learning rates. J. Complexity 57\/ (2020), 101438, 16

    2020

  28. [36]

    Jiang, Y., Liu, J., Xu, D., and Mandic, D. P. UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization . arXiv:2305.05675 https://arxiv.org/abs/2305.05675\/ (2023)

    2023 arXiv

  29. [37]

    Convergence of the momentum method for semialgebraic functions with locally L ipschitz gradients

    Josz, C., Lai, L., and Li, X. Convergence of the momentum method for semialgebraic functions with locally L ipschitz gradients. SIAM J. Optim. 33 , 4 (2023), 3012--3037

    2023

  30. [38]

    P., and Ba, J

    Kingma, D. P., and Ba, J. Adam: A Method for Stochastic Optimization . arXiv: 1412.6980 https://arxiv.org/abs/1412.6980\/ (2014)

    2014 arXiv

  31. [39]

    Some useful LATEX commands

    Kuckuck, B. Some useful LATEX commands . https://latex.bennokuckuck.de https://latex.bennokuckuck.de [Accessed April-2026]\/ (2025)

    2026

  32. [40]

    Convergence of Adam Under Relaxed Assumptions

    Li, H., Rakhlin, A., and Jadbabaie, A. Convergence of Adam Under Relaxed Assumptions . arXiv:2304.13972 https://arxiv.org/abs/2304.13972\/ (2023)

    2023 arXiv

  33. [41]

    Convergence analysis of AdaBound with relaxed bound functions for non-convex optimization

    Liu, J., Kong, J., Xu, D., Qi, M., and Lu, Y. Convergence analysis of AdaBound with relaxed bound functions for non-convex optimization . Neural Networks 145\/ (2022), 300--307

    2022

  34. [42]

    On Almost Sure Convergence Rates of Stochastic Gradient Methods

    Liu, J., and Yuan, Y. On Almost Sure Convergence Rates of Stochastic Gradient Methods . arXiv:2202.04295 https://arxiv.org/abs/2202.04295\/ (2022)

    2022 arXiv

  35. [43]

    On the Variance of the Adaptive Learning Rate and Beyond

    Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J. On the Variance of the Adaptive Learning Rate and Beyond . arXiv:1908.03265 https://arxiv.org/abs/1908.03265\/ (2019)

    1908 arXiv

  36. [44]

    An Improved Analysis of Stochastic Gradient Descent with Momentum

    Liu, Y., Gao, Y., and Yin, W. An Improved Analysis of Stochastic Gradient Descent with Momentum . arXiv:2007.07989 https://arxiv.org/abs/2007.07989\/ (2020)

    2007 arXiv

  37. [45]

    Linearly convergent stochastic heavy ball method for minimizing generalization error

    Loizou, N., and Richt\' a rik, P. Linearly convergent stochastic heavy ball method for minimizing generalization error . arXiv:1710.10737 https://arxiv.org/abs/1710.10737\/ (2017)

    2017 arXiv

  38. [46]

    Momentum and stochastic momentum for stochastic gradient, N ewton, proximal point and subspace descent methods

    Loizou, N., and Richt\' a rik, P. Momentum and stochastic momentum for stochastic gradient, N ewton, proximal point and subspace descent methods. Comput. Optim. Appl. 77 , 3 (2020), 653--710

    2020

  39. [47]

    Ensembles semi-analytiques

    ojasiewicz, S. Ensembles semi-analytiques. Unpublished lecture notes. Institut des Hautes Études Scientifiques. 1965 . url: https://perso.univ-rennes1.fr/michel.coste/Lojasiewicz.pdf

    1965

  40. [48]

    Adaptive Gradient Methods with Dynamic Bound of Learning Rate

    Luo, L., Xiong, Y., Liu, Y., and Sun, X. Adaptive Gradient Methods with Dynamic Bound of Learning Rate . arXiv:1902.09843 https://arxiv.org/abs/1902.09843\/ (2019)

    1902 arXiv

  41. [49]

    A method of solving a convex programming problem with convergence rate O(1/k^2)

    Nesterov, Y. A method of solving a convex programming problem with convergence rate O(1/k^2) . Soviet Mathematics Doklady 27\/ (1983), 372--376

    1983

  42. [50]

    Some methods of speeding up the convergence of iteration methods

    Polyak, B. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics 4 , 5 (1964), 1--17

    1964

  43. [51]

    T., and Juditsky, A

    Polyak, B. T., and Juditsky, A. B. Acceleration of stochastic approximation by averaging. SIAM J. Control Optim. 30 , 4 (1992), 838--855

    1992

  44. [52]

    Convergence of SGD with momentum in the nonconvex case: A time window-based analysis

    Qiu, J., Ma, B., and Milzarek, A. Convergence of SGD with momentum in the nonconvex case: A time window-based analysis . arXiv:2405.16954 https://arxiv.org/abs/2405.16954\/ (2024)

    2024 arXiv

  45. [53]

    J., Kale, S., and Kumar, S

    Reddi, S. J., Kale, S., and Kumar, S. On the Convergence of Adam and Beyond . arXiv:1904.09237 https://arxiv.org/abs/1904.09237\/ (2019)

    1904 arXiv

  46. [54]

    An overview of gradient descent optimization algorithms

    Ruder, S. An overview of gradient descent optimization algorithms . arXiv:1609.04747 https://arxiv.org/abs/1609.04747\/ (2016)

    2016 arXiv

  47. [55]

    M., and Defazio, A

    Sebbouh, O., Gower, R. M., and Defazio, A. Almost sure convergence rates for Stochastic Gradient Descent and Stochastic Heavy Ball . In Proceedings of Thirty Fourth Conference on Learning Theory\/ (15--19 Aug 2021), M. Belkin and S. Kpotufe, Eds., vol. 134 of Proceedings of Ma...

    2021

  48. [56]

    Non-ergodic Convergence Analysis of Heavy-Ball Algorithms

    Sun, T., Yin, P., Li, D., Huang, C., Guan, L., and Jiang, H. Non-ergodic Convergence Analysis of Heavy-Ball Algorithms . arXiv:1811.01777 https://arxiv.org/abs/1811.01777\/ (2018)

    2018 arXiv

  49. [57]

    Tadi\'c, V. B. Convergence and convergence rate of stochastic gradient search in the case of multiple and non-isolated extrema . Stochastic Process. Appl. 125 , 5 (2015), 1715--1755

    2015

  50. [58]

    Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptions

    Wang, B., Zhang, H., Ma, Z., and Chen, W. Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptions . arXiv:2305.18471 https://arxiv.org/abs/2305.18471\/ (2023)

    2023 arXiv

  51. [59]

    SAdam: A Variant of Adam for Strongly Convex Functions

    Wang, G., Lu, S., Tu, W., and Zhang, L. SAdam: A Variant of Adam for Strongly Convex Functions . arXiv:1905.02957 https://arxiv.org/abs/1905.02957\/ (2019)

    1905 arXiv

  52. [60]

    AdaLoss: A computationally-efficient and provably convergent adaptive gradient method

    Wu, X., Xie, Y., Du, S., and Ward, R. AdaLoss: A computationally-efficient and provably convergent adaptive gradient method . arXiv:2109.08282 https://arxiv.org/abs/2109.08282\/ (2021)

    2021 arXiv

  53. [61]

    AdamL: A fast adaptive gradient method incorporating loss function

    Xia, L., and Massei, S. AdamL: A fast adaptive gradient method incorporating loss function . arXiv:2312.15295 https://arxiv.org/abs/2312.15295\/ (2023)

    2023 arXiv

  54. [62]

    Adam-family methods for nonsmooth optimization with convergence guarantees

    Xiao, N., Hu, X., Liu, X., and Toh, K.-C. Adam-family methods for nonsmooth optimization with convergence guarantees. J. Mach. Learn. Res. 25\/ (2024), Paper No. [48], 53

    2024

  55. [63]

    Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models

    Xie, X., Zhou, P., Li, H., Lin, Z., and Yan, S. Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models . arXiv:2208.06677 https://arxiv.org/abs/2208.06677\/ (2022)

    2022 arXiv

  56. [64]

    Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization

    Yang, T., Lin, Q., and Li, Z. Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization . arXiv:1604.03257 https://arxiv.org/abs/1604.03257\/ (2016)

    2016 arXiv

  57. [65]

    EAdam Optimizer: How Impact Adam

    Yuan, W., and Gao, K.-X. EAdam Optimizer: How Impact Adam . arXiv:2011.02150 https://arxiv.org/abs/2011.02150\/ (2020)

    2011 arXiv

  58. [66]

    A new non-adaptive optimization method: Stochastic gradient descent with momentum and difference

    Yuan, W., Hu, F., and Lu, L. A new non-adaptive optimization method: Stochastic gradient descent with momentum and difference . Applied Intelligence 52 , 4 (2022), 3939--3953

    2022

  59. [67]

    Adaptive methods for nonconvex optimization

    Zaheer, M., Reddi, S., Sachan, D., Kale, S., and Kumar, S. Adaptive methods for nonconvex optimization. In Advances in Neural Information Processing Systems\/ (2018), S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31, Curran Assoc...

    2018

  60. [68]

    Improved Analysis of Clipping Algorithms for Non-convex Optimization

    Zhang, B., Jin, J., Fang, C., and Wang, L. Improved Analysis of Clipping Algorithms for Non-convex Optimization . arXiv:2010.02519 https://arxiv.org/abs/2010.02519\/ (2020)

    2010 arXiv

  61. [69]

    Adam Can Converge Without Any Modification On Update Rules

    Zhang, Y., Chen, C., Shi, N., Sun, R., and Luo, Z.-Q. Adam Can Converge Without Any Modification On Update Rules . arXiv:2208.09632 https://arxiv.org/abs/2208.09632\/ (2022)

    2022 arXiv

  62. [70]

    Fast A da B elief: improving convergence rate for belief-based adaptive optimizers by exploiting strong convexity

    Zhou, Y., Huang, K., Cheng, C., Wang, X., Hussain, A., and Liu, X. Fast A da B elief: improving convergence rate for belief-based adaptive optimizers by exploiting strong convexity. IEEE Trans. Neural Netw. Learn. Syst. 34 , 9 (2023), 6515--6529

    2023

  63. [71]

    Zhuang, J., Tang, T., Ding, Y., Tatikonda, S., Dvornek, N., Papademetris, X., and Duncan, J. S. AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients . arXiv:2010.07468 https://arxiv.org/abs/2010.07468\/ (2020)

    2010 arXiv

  64. [72]

    A Sufficient Condition for Convergences of Adam and RMSProp

    Zou, F., Shen, L., Jie, Z., Zhang, W., and Liu, W. A Sufficient Condition for Convergences of Adam and RMSProp . arXiv:1811.09358 https://arxiv.org/abs/1811.09358\/ (2018)

    2018 arXiv

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.