Pith. sign in

REVIEW 4 minor 72 references

A single unified iteration covers Adam and a dozen other DNN optimizers and forces every bounded trajectory to a critical point at an explicit polynomial rate.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 20:45 UTC pith:PHSYUZK7

load-bearing objection Solid unified strong-convergence theory for Adam and a dozen practical optimizers under KL + bounded trajectories, free of the usual momentum-step-size coupling.

arxiv 2607.04233 v1 pith:PHSYUZK7 submitted 2026-07-05 math.OC cs.LG

Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks

Shokhrukh Ibragimov , Arnulf Jentzen This is my paper
classification math.OC cs.LG MSC 90C2668T0765K05
keywords unified gradient descentKurdyka-Łojasiewicz inequalityAdam optimizerdeep neural networksstrong convergence ratesanalytic activationsadaptive optimizers
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that the optimizers practitioners actually use for deep networks—Adam, RMSprop, NAG, Adan, AdaBelief, AMSGrad, Yogi and several variants—are all special cases of one abstract update rule called the unified gradient-descent (UGD) method. Under the single geometric assumption that the loss is a Kurdyka-Łojasiewicz function with locally Lipschitz gradient (satisfied by any network with analytic activations such as softplus or GeLU), every bounded UGD trajectory converges to a critical point. The convergence is quantitative: the gradient, the function-value gap and a power of the distance to the limit all decay like the reciprocal of the partial sums of the step-size sequence. Because the same proof covers both classical momentum methods and modern adaptive methods without coupling their hyperparameters to the learning rate, the result supplies the first strong-convergence guarantee for Adam that works under the natural, decoupled schedules used in practice.

Core claim

Every bounded trajectory of the UGD iteration (which specialises to Adam, RMSprop, NAG, Adan, AdaBelief, AMSGrad, Yogi, …) applied to a Kurdyka-Łojasiewicz objective with locally Lipschitz gradient converges to a critical point at the explicit rate O(1/∑_{j=1}^n j^{- u}) for any u∈(3/4,1].

What carries the argument

The UGD recursion: a single momentum-style update with an abstract positive-definite matrix A_n and a perturbation sequence µ_n that simultaneously encodes every listed adaptive or accelerated method; once the trajectory is bounded, a KL inequality upgrades weak (function-value) convergence into strong (iterate) convergence with rates.

Load-bearing premise

The entire optimisation trajectory, together with the auxiliary adaptive matrices and momentum terms, must remain bounded for all time; without that a-priori bound the rates disappear.

What would settle it

Exhibit a concrete analytic DNN loss (softplus or GeLU network) and a set of Adam hyperparameters for which a bounded Adam trajectory either fails to approach a critical point or decays slower than any multiple of 1/∑ j^{- u} for u>3/4.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 4 minor

Summary. The paper develops a unified gradient-descent (UGD) framework (Setting 2.1, eqs. (2.1)–(2.2)) that encompasses classical GD, momentum, NAG, RMSprop, Adam, Adamax, Nadam, Nadamax, Adan, AdaBelief, AMSGrad, Yogi and the explicit midpoint method. Under the standing hypothesis that the iterates and auxiliary sequences remain bounded, and for any KL objective with locally Lipschitz gradient, Theorem 3.9 (and its special case Theorem 1.1) proves that every such trajectory converges to a critical point at the explicit rate O(1/∑_{j=1}^n j^{- u}) for u∈(3/4,1]. The argument proceeds by a weak-convergence step (Corollary 2.9) based on a Taylor remainder and monotone-convergence estimates, followed by a quantitative KL descent lemma (Proposition 3.2) that yields the rates. The framework is then specialized to analytic DNN training losses (Corollaries 3.13–3.14, 4.5) and to each of the listed optimizers (Section 4).

Significance. If the result holds, it supplies the first fully rigorous, rate-equipped strong-convergence theory that covers the entire practical family of adaptive and accelerated first-order methods under a single set of hypotheses. The proofs are self-contained real-analysis arguments that stay inside the classical KL toolkit; the only modelling assumption that is not automatic is boundedness of trajectories, which the authors correctly flag and for which they cite companion works that give sufficient conditions. The explicit polynomial rates and the clean specialization to analytic DNN losses (softplus, GeLU, etc.) make the contribution immediately usable for the analysis of modern AI optimizers. The machine-readable structure of the proofs and the absence of circular definitions further strengthen the paper’s value as a reference result.

minor comments (4)
  1. The bounded-trajectory hypothesis is stated clearly from Setting 2.1 onward and is essential; a short remark in the introduction (or after Theorem 1.1) that points the reader more explicitly to the companion papers [14,17] for concrete sufficient conditions would improve accessibility without changing any claim.
  2. In several places (e.g., the statement of Theorem 1.1 and Corollary 3.11) the matrix process A_n is required only to satisfy that A_n-I is positive semi-definite; a one-sentence clarification that this covers both constant learning-rate matrices and the usual diagonal adaptive scalings would help non-specialist readers.
  3. Typographical consistency: the symbol for the objective is sometimes L and sometimes script L; standardizing to one notation throughout would remove a minor source of visual friction.
  4. The literature overview (Subsection 1.3) is thorough; a brief sentence distinguishing the present deterministic, hyperparameter-decoupled setting from the stochastic, learning-rate-coupled analyses of Barakat–Bianchi and others would make the novelty claim even sharper.

Circularity Check

0 steps flagged

No circularity: the unified UGD convergence rates are derived directly from the KL inequality and elementary estimates under an explicit bounded-trajectory hypothesis.

full rationale

The paper's central claim (Theorem 3.9 / Theorem 1.1) is an implication: if L is a KL function with locally Lipschitz gradient and the UGD iterates (together with the auxiliary sequences A_n, p_n, µ_n) remain bounded, then every such trajectory converges to a critical point at the explicit rate O(1/∑ j^{-ν}). Boundedness is stated as a standing hypothesis from Setting 2.1 onward and is used only to obtain local Hölder/Lipschitz constants and to apply the KL inequality on a compact set; the authors explicitly flag companion works that supply sufficient conditions for it. The weak-convergence step (Corollary 2.9, via Propositions 2.3–2.4 and 2.7) and the subsequent strong-convergence argument (Proposition 3.2 through Theorem 3.9) consist of self-contained Taylor remainders, geometric-series bounds, and standard KL descent estimates; none of these steps defines a quantity in terms of the claimed rate or imports a uniqueness theorem that forces the result. Self-citations appear only as background or as optional sufficient conditions for the boundedness hypothesis and are not load-bearing for the derivation itself. Consequently the derivation chain does not reduce to its inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 1 invented entities

The central claim rests on three standard analytic facts (analytic functions are KL; local Lipschitz of gradients of analytic maps; elementary Taylor remainder) plus the modelling hypotheses that the trajectory stays bounded and that the adaptive matrices A_n dominate the identity. No free parameters are fitted; the decay exponent u > 3/4 is an explicit hypothesis on the learning-rate schedule.

axioms (4)
  • standard math Every real-analytic function R^d o R is a Kurdyka-Łojasiewicz function (Łojasiewicz 1965 / Bolte et al.).
    Invoked in §3.1 and Cor. 3.13–3.14 to transfer the abstract KL result to DNN empirical risk with softplus/GeLU activations.
  • standard math The gradient of a C^1 function with locally Hölder continuous derivative admits a standard Taylor remainder bound of order 1+δ.
    Lemma 2.2; used throughout the weak-convergence arguments.
  • domain assumption The optimization trajectory ( heta_n) and the auxiliary sequences (A_n, m_n, ho_n) remain bounded for all n.
    Standing hypothesis from Setting 2.1 onward; without it both weak and strong convergence fail.
  • domain assumption A_n - I_d is symmetric positive semi-definite for every n (or a positive multiple after rescaling by the learning rate).
    Ensures the descent inequality abla L · (A_n abla L) ≥ || abla L||^{2}; appears in every main theorem.
invented entities (1)
  • Unified Gradient Descent (UGD) iteration no independent evidence
    purpose: Single abstract recursion that simultaneously specialises to GD, momentum, NAG, RMSprop, Adam, Adamax, Nadam, Nadamax, Adan, AdaBelief, AMSGrad and Yogi by choice of the matrix sequence A_n and the momentum weights.
    The whole paper is organised around this abstraction; it is not claimed to be a new practical algorithm but a proof device.

pith-pipeline@v1.1.0-grok45 · 60878 in / 2738 out tokens · 32215 ms · 2026-07-11T20:45:15.438980+00:00 · methodology

0 comments
read the original abstract

Gradient based optimization methods are nowadays the methods of choice for training deep neural networks (DNNs) in artificial intelligence (AI) systems. In practically relevant DNN training problems, one does usually not apply the standard gradient descent (GD) optimization method but instead one employs suitable sophisticated GD optimization methods, which incorporate adaptivity and/or acceleration techniques, such as the famous Adam optimizer. It is a key contribution of this work to provide a general unified convergence analysis for GD optimization methods in the training of DNNs with analytic activations such as the softplus and the popular Gaussian error linear unit (GeLU) activation. Our general unified convergence result applies to a large class of gradient based optimization methods such as the standard GD, the momentum, the Nesterov accelerated gradient (NAG), the RMSprop, the Adam, the Adamax, the Nadam, the Nadamax, the Adan, the AdaBelief, the AMSGrad, and the Yogi optimizers. Our analysis employs the theory of Kurdyka-{\L}ojasiewicz (KL) inequalities to establish convergence to critical points in the training of DNNs. To the best of our knowledge, the generality of our convergence analysis is also just in the special situation of the Adam optimizer a new contribution to the literature on the analysis of AI optimization algorithms.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

72 extracted references · 41 linked inside Pith

  1. [1]

    Convergence of the iterates of descent methods for analytic cost functions

    Absil, P.-A., Mahony, R., and Andrews, B. Convergence of the iterates of descent methods for analytic cost functions. SIAM J. Optim. 16 , 2 (2005), 531--547

    2005

  2. [2]

    J., Fleming, R

    Arag\'on Artacho, F. J., Fleming, R. M. T., and Vuong, P. T. Accelerating the DC algorithm for smooth functions. Math. Program. 169 , 1 (2018), 95--118

    2018

  3. [3]

    Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization

    Barakat, A., and Bianchi, P. Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization . arXiv:1911.07596 https://arxiv.org/abs/1911.07596\/ (2020)

    Pith/arXiv arXiv 1911

  4. [4]

    Convergence and dynamical behavior of the A dam algorithm for nonconvex stochastic optimization

    Barakat, A., and Bianchi, P. Convergence and dynamical behavior of the A dam algorithm for nonconvex stochastic optimization. SIAM J. Optim. 31 , 1 (2021), 244--274

    2021

  5. [5]

    Stochastic optimization with momentum: convergence, fluctuations, and traps avoidance

    Barakat, A., Bianchi, P., Hachem, W., and Schechtman, S. Stochastic optimization with momentum: convergence, fluctuations, and traps avoidance . Electron. J. Stat. 15 , 2 (2021), 3892--3947

    2021

  6. [6]

    Adaptive algorithms and stochastic approximations , vol

    Benveniste, A., M\'etivier, M., and Priouret, P. Adaptive algorithms and stochastic approximations , vol. 22 of Applications of Mathematics (New York) . Springer-Verlag, Berlin, 1990. Translated from the French by Stephen S. Wilson

    1990

  7. [7]

    signSGD: Compressed Optimisation for Non-Convex Problems

    Bernstein, J., Wang, Y.-X., Azizzadenesheli, K., and Anandkumar, A. signSGD: Compressed Optimisation for Non-Convex Problems . arXiv:1802.04434 https://arxiv.org/abs/1802.04434\/ (2018)

    Pith/arXiv arXiv 2018

  8. [8]

    Bierstone, E., and Milman, P. D. Semianalytic and subanalytic sets. Inst. Hautes \'Etudes Sci. Publ. Math. 67\/ (1988), 5--42

    1988

  9. [9]

    A Proof of Local Convergence for the Adam Optimizer

    Bock, S., and Weiß, M. A Proof of Local Convergence for the Adam Optimizer . In 2019 International Joint Conference on Neural Networks (IJCNN)\/ (2019), pp. 1--8

    2019

  10. [10]

    The ojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems

    Bolte, J., Daniilidis, A., and Lewis, A. The ojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems . SIAM J. Optim. 17 , 4 (2006), 1205--1223

    2006

  11. [11]

    On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization

    Chen, X., Liu, S., Sun, R., and Hong, M. On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization . arXiv:1808.02941 https://arxiv.org/abs/1808.02941\/ (2018)

    Pith/arXiv arXiv 2018

  12. [12]

    Robustness to Unbounded Smoothness of Generalized SignSGD

    Crawshaw, M., Liu, M., Orabona, F., Zhang, W., and Zhuang, Z. Robustness to Unbounded Smoothness of Generalized SignSGD . arXiv:2208.11195 https://arxiv.org/abs/2208.11195\/ (2022)

    Pith/arXiv arXiv 2022

  13. [13]

    A Simple Convergence Proof of Adam and Adagrad

    D \'e fossez, A., Bottou, L., Bach, F., and Usunier, N. A Simple Convergence Proof of Adam and Adagrad . Transactions on Machine Learning Research\/ (2022)

    2022

  14. [14]

    Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method

    Dereich, S., Do, T., and Jentzen, A. Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method . arXiv:2603.18899 https://arxiv.org/abs/2603.18899\/ (2026)

    arXiv 2026

  15. [15]

    Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer

    Dereich, S., Do, T., Jentzen, A., and von Wurstemberger , P. Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer . arXiv:2511.06675 https://arxiv.org/abs/2511.06675\/ (2025)

    Pith/arXiv arXiv 2025

  16. [16]

    Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rates

    Dereich, S., Graeber, R., and Jentzen, A. Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rates . arXiv:2407.08100 https://arxiv.org/abs/2407.08100\/ (2024)

    Pith/arXiv arXiv 2024

  17. [17]

    Asymptotic stability properties and a priori bounds for Adam and other gradient descent optimization methods

    Dereich, S., Graeber, R., Jentzen, A., and Riekert, A. Asymptotic stability properties and a priori bounds for Adam and other gradient descent optimization methods . arXiv:2509.10476 https://arxiv.org/abs/2509.10476\/ (2025)

    arXiv 2025

  18. [18]

    Convergence rates for the Adam optimizer

    Dereich, S., and Jentzen, A. Convergence rates for the Adam optimizer . arXiv:2407.21078 https://arxiv.org/abs/2407.21078\/ (2024)

    Pith/arXiv arXiv 2024

  19. [19]

    ODE approximation for the Adam algorithm: General and overparametrized setting

    Dereich, S., Jentzen, A., and Kassing, S. ODE approximation for the Adam algorithm: General and overparametrized setting . arXiv:2511.04622 https://arxiv.org/abs/2511.04622\/ (2025)

    arXiv 2025

  20. [20]

    Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses

    Dereich, S., Jentzen, A., and Riekert, A. Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses . arXiv:2406.14340 https://arxiv.org/abs/2406.14340\/ (2024)

    Pith/arXiv arXiv 2024

  21. [21]

    Sharp higher order convergence rates for the Adam optimizer

    Dereich, S., Jentzen, A., and Riekert, A. Sharp higher order convergence rates for the Adam optimizer . arXiv:2504.19426 https://arxiv.org/abs/2504.19426\/ (2025)

    Pith/arXiv arXiv 2025

  22. [22]

    Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapes

    Dereich, S., and Kassing, S. Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapes . arXiv:2102.09385 https://arxiv.org/abs/2102.09385\/ (2024)

    Pith/arXiv arXiv 2024

  23. [23]

    Incorporating Nesterov Momentum into Adam

    Dozat, T. Incorporating Nesterov Momentum into Adam . https://openreview.net/forum?id=OM0jvwB8jIp57ZJjtNEZ, 2016. [Accessed 14-November-2025]

    2016

  24. [24]

    R., Singh, S

    Dubey, S. R., Singh, S. K., and Chaudhuri, B. B. AdaNorm: Adaptive Gradient Norm Correction based Optimizer for CNNs . arXiv:2210.06364 https://arxiv.org/abs/2210.06364\/ (2022)

    Pith/arXiv arXiv 2022

  25. [25]

    The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variance

    Faw, M., Tziotis, I., Caramanis, C., Mokhtari, A., Shakkottai, S., and Ward, R. The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variance . arXiv:2202.05791 https://arxiv.org/abs/2202.05791\/ (2022)

    Pith/arXiv arXiv 2022

  26. [26]

    Garrigos, G., and Gower, R. M. Handbook of Convergence Theorems for (Stochastic) Gradient Methods . arXiv:2301.11235 https://arxiv.org/abs/2301.11235\/ (2023)

    Pith/arXiv arXiv 2023

  27. [27]

    Convergence rates for momentum stochastic gradient descent with noise of machine learning type

    Gess, B., and Kassing, S. Convergence rates for momentum stochastic gradient descent with noise of machine learning type . arXiv:2302.03550 https://arxiv.org/abs/2302.03550\/ (2023)

    Pith/arXiv arXiv 2023

  28. [28]

    R., and Johansson, M

    Ghadimi, E., Feyzmahdavian, H. R., and Johansson, M. Global convergence of the Heavy-ball method for convex optimization . In 2015 European Control Conference (ECC)\/ (2015), pp. 310--315

    2015

  29. [29]

    Non asymptotic analysis of Adaptive stochastic gradient algorithms and applications

    Godichon-Baggioni, A., and Tarrago, P. Non asymptotic analysis of Adaptive stochastic gradient algorithms and applications . arXiv:2303.01370 https://arxiv.org/abs/2303.01370\/ (2023)

    Pith/arXiv arXiv 2023

  30. [30]

    Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case

    He, M., Liang, Y., Liu, J., and Xu, D. Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case . arXiv:2307.11782 https://arxiv.org/abs/2307.11782\/ (2023)

    Pith/arXiv arXiv 2023

  31. [31]

    Lecture 6e: RMSprop: Divide the gradient by a running average of its recent magnitude

    Hinton, G., Srivastava, N., and Swersky, K. Lecture 6e: RMSprop: Divide the gradient by a running average of its recent magnitude . https://www.cs.toronto.edu/ tijmen/csc321/slides/lecture_slides_lec6.pdf. [Accessed 14-November-2025]

    2025

  32. [32]

    Revisiting Convergence of AdaGrad with Relaxed Assumptions

    Hong, Y., and Lin, J. Revisiting Convergence of AdaGrad with Relaxed Assumptions . arXiv:2402.13794 https://arxiv.org/abs/2402.13794\/ (2024)

    Pith/arXiv arXiv 2024

  33. [33]

    Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

    Jentzen, A., Kuckuck, B., and von Wurstemberger, P. Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory . arXiv:2310.20360 https://arxiv.org/abs/2310.20360\/ (2023)

    Pith/arXiv arXiv 2023

  34. [34]

    On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks

    Jentzen, A., and Riekert, A. On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks. J. Mach. Learn. 1 , 2 (2022), 141--246

    2022

  35. [35]

    Lower error bounds for the stochastic gradient descent optimization algorithm: sharp convergence rates for slowly and fast decaying learning rates

    Jentzen, A., and von Wurstemberger, P. Lower error bounds for the stochastic gradient descent optimization algorithm: sharp convergence rates for slowly and fast decaying learning rates. J. Complexity 57\/ (2020), 101438, 16

    2020

  36. [36]

    Jiang, Y., Liu, J., Xu, D., and Mandic, D. P. UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization . arXiv:2305.05675 https://arxiv.org/abs/2305.05675\/ (2023)

    Pith/arXiv arXiv 2023

  37. [37]

    Convergence of the momentum method for semialgebraic functions with locally L ipschitz gradients

    Josz, C., Lai, L., and Li, X. Convergence of the momentum method for semialgebraic functions with locally L ipschitz gradients. SIAM J. Optim. 33 , 4 (2023), 3012--3037

    2023

  38. [38]

    P., and Ba, J

    Kingma, D. P., and Ba, J. Adam: A Method for Stochastic Optimization . arXiv: 1412.6980 https://arxiv.org/abs/1412.6980\/ (2014)

    Pith/arXiv arXiv 2014

  39. [39]

    Some useful LATEX commands

    Kuckuck, B. Some useful LATEX commands . https://latex.bennokuckuck.de https://latex.bennokuckuck.de [Accessed April-2026]\/ (2025)

    2026

  40. [40]

    Convergence of Adam Under Relaxed Assumptions

    Li, H., Rakhlin, A., and Jadbabaie, A. Convergence of Adam Under Relaxed Assumptions . arXiv:2304.13972 https://arxiv.org/abs/2304.13972\/ (2023)

    Pith/arXiv arXiv 2023

  41. [41]

    Convergence analysis of AdaBound with relaxed bound functions for non-convex optimization

    Liu, J., Kong, J., Xu, D., Qi, M., and Lu, Y. Convergence analysis of AdaBound with relaxed bound functions for non-convex optimization . Neural Networks 145\/ (2022), 300--307

    2022

  42. [42]

    On Almost Sure Convergence Rates of Stochastic Gradient Methods

    Liu, J., and Yuan, Y. On Almost Sure Convergence Rates of Stochastic Gradient Methods . arXiv:2202.04295 https://arxiv.org/abs/2202.04295\/ (2022)

    Pith/arXiv arXiv 2022

  43. [43]

    On the Variance of the Adaptive Learning Rate and Beyond

    Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J. On the Variance of the Adaptive Learning Rate and Beyond . arXiv:1908.03265 https://arxiv.org/abs/1908.03265\/ (2019)

    Pith/arXiv arXiv 1908

  44. [44]

    An Improved Analysis of Stochastic Gradient Descent with Momentum

    Liu, Y., Gao, Y., and Yin, W. An Improved Analysis of Stochastic Gradient Descent with Momentum . arXiv:2007.07989 https://arxiv.org/abs/2007.07989\/ (2020)

    Pith/arXiv arXiv 2007

  45. [45]

    Linearly convergent stochastic heavy ball method for minimizing generalization error

    Loizou, N., and Richt\' a rik, P. Linearly convergent stochastic heavy ball method for minimizing generalization error . arXiv:1710.10737 https://arxiv.org/abs/1710.10737\/ (2017)

    Pith/arXiv arXiv 2017

  46. [46]

    Momentum and stochastic momentum for stochastic gradient, N ewton, proximal point and subspace descent methods

    Loizou, N., and Richt\' a rik, P. Momentum and stochastic momentum for stochastic gradient, N ewton, proximal point and subspace descent methods. Comput. Optim. Appl. 77 , 3 (2020), 653--710

    2020

  47. [47]

    Ensembles semi-analytiques

    ojasiewicz, S. Ensembles semi-analytiques. Unpublished lecture notes. Institut des Hautes Études Scientifiques. 1965 . url: https://perso.univ-rennes1.fr/michel.coste/Lojasiewicz.pdf

    1965

  48. [48]

    Adaptive Gradient Methods with Dynamic Bound of Learning Rate

    Luo, L., Xiong, Y., Liu, Y., and Sun, X. Adaptive Gradient Methods with Dynamic Bound of Learning Rate . arXiv:1902.09843 https://arxiv.org/abs/1902.09843\/ (2019)

    Pith/arXiv arXiv 1902

  49. [49]

    A method of solving a convex programming problem with convergence rate O(1/k^2)

    Nesterov, Y. A method of solving a convex programming problem with convergence rate O(1/k^2) . Soviet Mathematics Doklady 27\/ (1983), 372--376

    1983

  50. [50]

    Some methods of speeding up the convergence of iteration methods

    Polyak, B. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics 4 , 5 (1964), 1--17

    1964

  51. [51]

    T., and Juditsky, A

    Polyak, B. T., and Juditsky, A. B. Acceleration of stochastic approximation by averaging. SIAM J. Control Optim. 30 , 4 (1992), 838--855

    1992

  52. [52]

    Convergence of SGD with momentum in the nonconvex case: A time window-based analysis

    Qiu, J., Ma, B., and Milzarek, A. Convergence of SGD with momentum in the nonconvex case: A time window-based analysis . arXiv:2405.16954 https://arxiv.org/abs/2405.16954\/ (2024)

    Pith/arXiv arXiv 2024

  53. [53]

    J., Kale, S., and Kumar, S

    Reddi, S. J., Kale, S., and Kumar, S. On the Convergence of Adam and Beyond . arXiv:1904.09237 https://arxiv.org/abs/1904.09237\/ (2019)

    Pith/arXiv arXiv 1904

  54. [54]

    An overview of gradient descent optimization algorithms

    Ruder, S. An overview of gradient descent optimization algorithms . arXiv:1609.04747 https://arxiv.org/abs/1609.04747\/ (2016)

    Pith/arXiv arXiv 2016

  55. [55]

    M., and Defazio, A

    Sebbouh, O., Gower, R. M., and Defazio, A. Almost sure convergence rates for Stochastic Gradient Descent and Stochastic Heavy Ball . In Proceedings of Thirty Fourth Conference on Learning Theory\/ (15--19 Aug 2021), M. Belkin and S. Kpotufe, Eds., vol. 134 of Proceedings of Machine Learning Research , PMLR, pp. 3935--3971

    2021

  56. [56]

    Non-ergodic Convergence Analysis of Heavy-Ball Algorithms

    Sun, T., Yin, P., Li, D., Huang, C., Guan, L., and Jiang, H. Non-ergodic Convergence Analysis of Heavy-Ball Algorithms . arXiv:1811.01777 https://arxiv.org/abs/1811.01777\/ (2018)

    Pith/arXiv arXiv 2018

  57. [57]

    Tadi\'c, V. B. Convergence and convergence rate of stochastic gradient search in the case of multiple and non-isolated extrema . Stochastic Process. Appl. 125 , 5 (2015), 1715--1755

    2015

  58. [58]

    Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptions

    Wang, B., Zhang, H., Ma, Z., and Chen, W. Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptions . arXiv:2305.18471 https://arxiv.org/abs/2305.18471\/ (2023)

    Pith/arXiv arXiv 2023

  59. [59]

    SAdam: A Variant of Adam for Strongly Convex Functions

    Wang, G., Lu, S., Tu, W., and Zhang, L. SAdam: A Variant of Adam for Strongly Convex Functions . arXiv:1905.02957 https://arxiv.org/abs/1905.02957\/ (2019)

    Pith/arXiv arXiv 1905

  60. [60]

    AdaLoss: A computationally-efficient and provably convergent adaptive gradient method

    Wu, X., Xie, Y., Du, S., and Ward, R. AdaLoss: A computationally-efficient and provably convergent adaptive gradient method . arXiv:2109.08282 https://arxiv.org/abs/2109.08282\/ (2021)

    Pith/arXiv arXiv 2021

  61. [61]

    AdamL: A fast adaptive gradient method incorporating loss function

    Xia, L., and Massei, S. AdamL: A fast adaptive gradient method incorporating loss function . arXiv:2312.15295 https://arxiv.org/abs/2312.15295\/ (2023)

    Pith/arXiv arXiv 2023

  62. [62]

    Adam-family methods for nonsmooth optimization with convergence guarantees

    Xiao, N., Hu, X., Liu, X., and Toh, K.-C. Adam-family methods for nonsmooth optimization with convergence guarantees. J. Mach. Learn. Res. 25\/ (2024), Paper No. [48], 53

    2024

  63. [63]

    Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models

    Xie, X., Zhou, P., Li, H., Lin, Z., and Yan, S. Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models . arXiv:2208.06677 https://arxiv.org/abs/2208.06677\/ (2022)

    Pith/arXiv arXiv 2022

  64. [64]

    Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization

    Yang, T., Lin, Q., and Li, Z. Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization . arXiv:1604.03257 https://arxiv.org/abs/1604.03257\/ (2016)

    Pith/arXiv arXiv 2016

  65. [65]

    EAdam Optimizer: How Impact Adam

    Yuan, W., and Gao, K.-X. EAdam Optimizer: How Impact Adam . arXiv:2011.02150 https://arxiv.org/abs/2011.02150\/ (2020)

    Pith/arXiv arXiv 2011

  66. [66]

    A new non-adaptive optimization method: Stochastic gradient descent with momentum and difference

    Yuan, W., Hu, F., and Lu, L. A new non-adaptive optimization method: Stochastic gradient descent with momentum and difference . Applied Intelligence 52 , 4 (2022), 3939--3953

    2022

  67. [67]

    Adaptive methods for nonconvex optimization

    Zaheer, M., Reddi, S., Sachan, D., Kale, S., and Kumar, S. Adaptive methods for nonconvex optimization. In Advances in Neural Information Processing Systems\/ (2018), S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31, Curran Associates, Inc

    2018

  68. [68]

    Improved Analysis of Clipping Algorithms for Non-convex Optimization

    Zhang, B., Jin, J., Fang, C., and Wang, L. Improved Analysis of Clipping Algorithms for Non-convex Optimization . arXiv:2010.02519 https://arxiv.org/abs/2010.02519\/ (2020)

    Pith/arXiv arXiv 2010

  69. [69]

    Adam Can Converge Without Any Modification On Update Rules

    Zhang, Y., Chen, C., Shi, N., Sun, R., and Luo, Z.-Q. Adam Can Converge Without Any Modification On Update Rules . arXiv:2208.09632 https://arxiv.org/abs/2208.09632\/ (2022)

    Pith/arXiv arXiv 2022

  70. [70]

    Fast A da B elief: improving convergence rate for belief-based adaptive optimizers by exploiting strong convexity

    Zhou, Y., Huang, K., Cheng, C., Wang, X., Hussain, A., and Liu, X. Fast A da B elief: improving convergence rate for belief-based adaptive optimizers by exploiting strong convexity. IEEE Trans. Neural Netw. Learn. Syst. 34 , 9 (2023), 6515--6529

    2023

  71. [71]

    Zhuang, J., Tang, T., Ding, Y., Tatikonda, S., Dvornek, N., Papademetris, X., and Duncan, J. S. AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients . arXiv:2010.07468 https://arxiv.org/abs/2010.07468\/ (2020)

    Pith/arXiv arXiv 2010

  72. [72]

    A Sufficient Condition for Convergences of Adam and RMSProp

    Zou, F., Shen, L., Jie, Z., Zhang, W., and Liu, W. A Sufficient Condition for Convergences of Adam and RMSProp . arXiv:1811.09358 https://arxiv.org/abs/1811.09358\/ (2018)

    Pith/arXiv arXiv 2018