Sharp higher order convergence rates for the Adam optimizer

Adrian Riekert; Arnulf Jentzen; Steffen Dereich

REVIEW 1 major objections 5 minor 2 cited by

Sharp higher order convergence rates for the Adam optimizer

T0 review · 1 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adam with Hessian-tuned hyperparameters provably converges at the same rate as momentum, while RMSprop stays at gradient-descent speed.

desk verdict Important result on Adam's local rate, but the proof of Theorem 1.2 instantiates the wrong momentum coefficient—easy fix, deserves refereeing. read the letter →

arxiv 2504.19426 v1 pith:I5TRNYLZ submitted 2025-04-28 math.OC cs.AI

Steffen Dereich , Arnulf Jentzen , Adrian Riekert This is my paper · ORCID

classification math.OCcs.AI MSC 68W4090C25

keywords AdamoptimizerRMSpropmomentummethodconvergencerateconditionnumberadaptivelearningspectralradiuslocalminimizer

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proves a sharp local convergence-rate theorem for the Adam optimizer: if the momentum coefficient and step size are set using the extreme eigenvalues of the Hessian at a local minimizer, then Adam converges in a neighborhood of that minimizer with rate $(\sqrt{\chi}-1)/(\sqrt{\chi}+1)$, where $\chi$ is the Hessian's condition number. This is the same rate as the momentum method and strictly faster than gradient descent's classical rate $(\chi-1)/(\chi+1)$. The paper also proves that RMSprop, Adam's adaptive-rate cousin without momentum, reaches only the slower gradient-descent rate. The result applies to deterministic $C^2$ objectives whose Hessian at the minimizer lies between $rI$ and $r\chi I$.

What carries the argument

The proof treats Adam as a momentum iteration with adaptive per-coordinate step sizes. The load-bearing object is the linear iteration in the joint state $(\Theta_n,m_n)$ whose coefficient matrices $A_n$ converge, as the moment estimates settle, to the constant block matrix of the momentum method; its spectral radius is $\sqrt{\alpha}$. A generalized Gelfand spectral-radius lemma, together with a perturbation lemma for linear iterations, transfers that limiting spectral radius to the time-varying system, and a local-stability argument shows that the second-moment estimate and the bias-correction factors converge so that the adaptive step-size schedule settles at the fixed step $\gamma/\varepsilon$. The same machinery yields the RMSprop rate by viewing RMSprop as plain gradient descent with learning rates converging to $2/(\kappa+K)$.

What would settle it

Simulate the exact quadratic $L(\theta)=\frac12\sum_i\lambda_i(\theta_i-\psi_i)^2$ with known $\kappa$ and $K$, initialize near $\psi$, and run Adam with the paper's $\alpha=((1-(\gamma/\varepsilon)\kappa)/(1+(\gamma/\varepsilon)\kappa))^2$ and $\gamma=\varepsilon/\sqrt{\kappa K}$, with any $\beta\in(0,1)$. If the empirical contraction rate of $\|\Theta_n-\psi\|$ is not bounded by $(\sqrt{\chi}-1)/(\sqrt{\chi}+1)+\delta$ for all sufficiently small $\delta>0$, the theorem's rate is false. Running the same experiment with default $\alpha=0.9$ would not falsify the theorem, because that setting lies outside its assumptions.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is Theorem 1.2(iv) via Corollary 4.11: for every dimension, every condition number $\chi\in(1,\infty)$, and every $C^2$ objective $L$ with $(\nabla L)(\psi)=0$ and $rI\preceq(\mathrm{Hess}\,L)(\psi)\preceq r\chi I$, there exists a momentum coefficient $\alpha\in(0,1)$ such that the $\alpha$-$\beta$-$\varepsilon$-Adam optimizer converges to $\psi$ with rate $(\sqrt{\chi}-1)/(\sqrt{\chi}+1)$. The theorem exhibits the specific tuning $\alpha=((1-(\gamma/\varepsilon)\kappa)/(1+(\gamma/\varepsilon)\kappa))^2$ with $\gamma\le\varepsilon/\sqrt{\kappa K}$, where $\kappa=r$ and $K=r\chi$ are the smallest and largest Hessian eigenvalues. The companion claim is that every RMSprop optimizer converges with the slower rate $(\chi-1)/(\chi+1)$ (Corollary 2.9). Concretely, the rate means that for every $\rho$ larger than the stated rate, the distance $\|\Theta_n-\psi\|$ after $n$ steps stays bounded by a constant times $\rho^n$; in short, Adam's adaptivity costs nothing in local speed when the hyperparameters know the local curvature.

Load-bearing premise

The accelerated rate holds only when the optimizer's momentum coefficient and step size are set from the smallest and largest Hessian eigenvalues at the minimizer; if those numbers are unknown and a fixed default momentum such as 0.9 is used, the theorem's optimality claim is not asserted.

Editorial extensions

If this is right

For every deterministic $C^2$ objective whose Hessian at the minimizer lies between $rI$ and $r\chi I$, Adam with the tuned parameters converges locally with rate $(\sqrt{\chi}-1)/(\sqrt{\chi}+1)$: for every $\rho$ above that rate, $\sup_n \rho^{-n}\|\Theta_n-\psi\|$ is finite.
RMSprop, with any $\beta\in[0,1)$ and $\varepsilon>0$, is shown to converge with the slower rate $(\chi-1)/(\chi+1)$, exactly the gradient-descent rate.
Because $(\sqrt{\chi}-1)/(\sqrt{\chi}+1)<(\chi-1)/(\chi+1)$ for every $\chi>1$, Adam is strictly faster than gradient descent in every sufficiently ill-conditioned quadratic neighborhood.
The weak optimization error $|L(\Theta_n)-L(\psi)|$ inherits the squared rates $(\sqrt{\chi}-1)^2(\sqrt{\chi}+1)^{-2}$ for Adam and $(\chi-1)^2(\chi+1)^{-2}$ for RMSprop.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An immediate consequence the authors do not state: if a practitioner keeps a fixed momentum such as $\alpha=0.9$ instead of the Hessian-tuned value, the theorem's accelerated rate is not in force; nothing in the proof bounds that case.
Because the mechanism only uses that Adam's step-size factors converge to $1$ and the momentum block's spectral radius is $\sqrt{\alpha}$, similar local rate statements should hold for debiased Adam variants and for weight-decoupled Adam, as long as the same tuning is used.
A stochastic version of the local rate should appear before noise dominates: with small-variance gradient noise, the deterministic contraction should dominate early iterations, so finite-time bounds should show a window where the accelerated exponent is visible.
The RMSprop/Adam separation predicts that the acceleration in Adam comes entirely from momentum; a minimal experiment would compare Adam with momentum coefficient $\alpha=0$ against the theorem's tuned $\alpha$ on a quadratic.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing you should know: the paper is worth reading—the main result is real and important—but the proof of Theorem 1.2 as written has a parameter slip that needs a small fix before the headline rate is actually established. The slip is easy to repair, so this is a conditional accept, not a reject.

What's new: they give explicit local exponential rates for RMSprop and Adam. The Adam result, matching the momentum rate (√χ−1)/(√χ+1), was not in the cited predecessors. The mechanism is clean: near a strict minimizer the adaptive scaling factor in Adam converges to a constant, so the analysis reduces to a momentum iteration with converging step sizes; RMSprop similarly reduces to gradient descent. The spectral-radius arguments in Propositions 2.8, 3.3, 3.4, 4.9 are coherent and the rates follow. They also correctly credit Polyak for the GD and momentum items.

Now the soft spots, in proportion. The stress-test is right: Theorem 1.2's proof applies Corollary 4.11 with α set to the desired rate R=(√χ−1)/(√χ+1), but Corollary 4.11's hypothesis is α = ((1−γε^{-1}κ)/(1+γε^{-1}κ))^2, and with their choice of γ the right side is R^2. The corollary's conclusion is a rate √α, so plugging in α=R gives only √R, not R. The same mistake appears in item (ii). The fix is simply to set α=R^2. This is a genuine gap in the write-up, but it is localized and I would not call the underlying claim into question.

Two more caveats. The abstract says RMSprop 'only' converges with the GD rate, but the theorem is an upper bound on the rate; there is no lower bound here, so 'only' is too strong. And the Adam rate requires α and γ tuned to the Hessian bounds at the minimizer; the abstract does not say that. Neither changes the mathematics, but they matter for how the result is advertised. Also, the gradient-convergence lemma 4.2 is imported from Barakat–Bianchi; that is legitimate, but the local-rate proof is not fully self-contained.

The paper is for optimization theorists working on adaptive methods or local convergence. It deserves a serious referee. My recommendation: send it out, but ask the authors to fix the α↦α^2 slip and soften the abstract.

Referee Report

1 major / 5 minor

Summary. The paper studies local exponential convergence rates of RMSprop and Adam for smooth deterministic objectives near a strict local minimizer. The technical core is a set of perturbation results for linear iterations whose matrices converge (Lemmas 2.3–2.5), used to reduce the local dynamics of adaptive optimizers to the spectral radius of a limiting linearization. For a condition number χ, the authors prove that RMSprop attains the same rate as gradient descent, (χ−1)/(χ+1), and that Adam attains the accelerated momentum rate (√χ−1)/(√χ+1). The latter is obtained by combining a local contraction argument for a class of momentum iterations with convergent adaptive step sizes (Proposition 4.9) with the observation that Adam's bias-corrected prefactors converge to 1. The paper also contains a global convergence result for smooth strongly convex objectives with small step sizes and a comparison theorem showing that Adam asymptotically beats GD.

Significance. If the main result stands, this is a significant contribution: it gives the first explicit sharp local convergence rate for Adam matching the optimal momentum rate, with a clean and largely self-contained proof. The rates are derived from the spectral radius of the limiting linearization, not fitted to simulations, and no constants are tuned to data. The authors are honest that the GD and momentum parts are classical, and the external input from [2] is confined to a preparatory gradient-vanishing lemma that is not needed for the sharp local-rate proof. The main caveat is that the rate is an existence statement: it requires the momentum coefficient and step size to be chosen using the Hessian bounds, so the unqualified abstract sentence overstates what is proven for default hyperparameters.

major comments (1)

[Theorem 1.2 (Section 1.2) and Corollaries 4.10–4.11] The proof of Theorem 1.2(ii) and (iv) applies Corollary 4.10 and Corollary 4.11 with the wrong momentum coefficient. Corollary 4.10 requires α = ((1−γκ)/(1+γκ))^2, and Corollary 4.11 requires α = ((1−γε^{-1}κ)/(1+γε^{-1}κ))^2. In Theorem 1.2(ii), the substitution γ = r^{-1/2}(rχ)^{-1/2} gives γκ = 1/√χ, so the corollary requires α = R^2 with R = (√χ−1)/(√χ+1), whereas the proof sets α = R. With α = R, Corollary 4.10 only yields the rate √R, which is slower than the claimed rate R. The identical mismatch occurs in item (iv): substituting γ = r^{-1/2}(rχ)^{-1/2}ε gives γε^{-1}κ = 1/√χ, so Corollary 4.11 requires α = R^2, not α = R. Thus the cited implications, as written, do not prove the headline rate. The repair is immediate: set α ↦ R^2 in both applications; since α is existential in Theorem 1.2, the statement of the theorem remains valid after this correction.

minor comments (5)

[Section 3.4, proof of Proposition 3.4] The text states 'Lemma 3.2 therefore demonstrates that ρ(A) ≤ α'; this should be ρ(A) = √α (or ≤ √α). The subsequent conclusion uses the exponent √α + δ, so the typo does not affect the result, but the displayed inequality is incorrect.
[Section 4.5, Theorem 4.6] The notation in Theorem 4.6 is confusing: γ is used both for the uniform upper bound γ ∈ (0, 1/(4K)) and for a step-size variable γ ∈ [0, γ], and the expression Θ^{α,β,ε}_γ_n in the conclusion is not defined. The proof uses η = εγ, so the statement should be aligned with the proof.
[Abstract and Section 1.2] The abstract's sentence 'Adam also converges with the strictly faster rate' is unqualified. The theorem itself is an existence result: the accelerated rate requires the momentum coefficient α = R^2 and a step size γ of order ε/√(κK), both depending on the Hessian bounds. The abstract should indicate this qualification.
[Equation (1.3)] In the chain of inequalities, the denominator 'κ + 2√χ + 1' should read 'χ + 2√χ + 1'.
[Section 4.6, proof of Proposition 4.9] In the display after (4.40), the perturbation term is written as h_{Γ_θ_n}(Θ_θ_n, m_θ_n); it should be evaluated at the previous iterate, h_{Γ_θ_n}(Θ_θ_{n−1}, m_θ_{n−1}), as the preceding line and the bound require.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the sharp local Adam rate is derived from the spectral radius of a linearized momentum iteration and does not assume its own conclusion.

full rationale

The central claim, Theorem 1.2(iv), is obtained via Corollary 4.11, whose proof (Proposition 4.9) linearizes the Adam update around the minimizer, reduces the dynamics to the momentum iteration of Proposition 3.4, and controls the perturbation terms using local smoothness and the Hessian bounds. The exponential rate is then read off from the spectral radius of the limiting linear iteration via Lemma 3.2 and Lemma 2.4/2.5. This is a derivation, not a restatement: the rate (√chi−1)/(√chi+1) is not inserted into the assumptions; it is computed as sqrt(alpha) from the explicitly chosen hyperparameters and the eigenvalue bounds of Hess L(psi). No parameter is fitted to data and no fitted quantity is renamed as a prediction. The only substantial external input, [2], is used for Lemma 4.2 (gradient convergence to zero), and that lemma is not needed for the sharp local-rate result, which either assumes the needed convergence in Corollary 4.4 or establishes it locally in Proposition 4.9. [2] is an independent work by Barakat and Bianchi, not a self-citation, and the authors' own earlier papers [11]-[13] appear in the literature review and in non-convergence contextual remarks but do not carry the derivation. The observation that the theorem is an existence statement for hyperparameters tuned to the unknown Hessian bounds does not indicate circularity; likewise, the reviewer-flagged mismatch between alpha=R and the corollary's hypothesis alpha=R^2 is a proof-correctness issue, not an instance of the conclusion being assumed as an input.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

No data fitting is involved. The only problem-dependent choices are the optimizer hyperparameters alpha and gamma, which the theorems set from the Hessian bounds kappa and K. The proof reduces Adam to a constant-effective-step momentum method after showing the adaptive factor converges; the external theorem [2] is used for a preparatory gradient-convergence result, not for the sharp local rate.

free parameters (2)

momentum coefficient alpha = ((1-(gamma/epsilon)kappa)/(1+(gamma/epsilon)kappa))^2; for the sharp rate gamma/epsilon = 1/sqrt(kappa K)
Chosen as a function of the Hessian condition number to make the linearized momentum iteration have spectral radius sqrt(alpha). The theorem only asserts existence of such alpha.
step size gamma = epsilon/sqrt(kappa K) for the sharp local rate; gamma <= epsilon/sqrt(kappa K) in Corollary 4.11
Chosen so that the effective step gamma/epsilon equals the optimal momentum step 1/sqrt(kappa K).

assumptions (5)

domain assumption C^2 objective with Hessian bounds kappa I <= Hess L(psi) <= K I at the minimizer, or global strong convexity and smoothness for global results.
Definition 1.1 and the proofs in Sections 2 through 4 assume these bounds; the rate depends on chi = K/kappa.
ad hoc to paper The optimizer hyperparameters alpha and gamma are tuned using kappa and K.
Corollaries 4.10 and 4.11 set alpha and gamma from kappa and K; without this tuning the accelerated rate is not guaranteed by the paper's theorems.
standard math Barakat and Bianchi [2, Theorem 2 and Proposition 14] conditions for gradient convergence.
Lemma 4.2 is proved by verifying the conditions of [2] and invoking its results; these external results are not re-derived in this paper.
standard math Classical spectral radius and perturbation lemmas, including Gelfand's formula and norm equivalence.
Lemmas 2.3, 2.4, and 2.5 rely on standard matrix analysis; the paper reproduces the needed arguments.
standard math Momentum linearization eigenvalue computation in Lemma 3.2.
The proofs of Proposition 3.3 and Proposition 4.9 depend on the spectral radius of the 2d by 2d momentum matrix being sqrt(alpha); the paper proves this in detail.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Sharp higher order convergence rates for the Adam optimizer." pith.science (2026). https://pith.science/paper/I5TRNYLZ

@misc{pith2026250419426,
  author       = {Pith},
  title        = {Pith review of: Sharp higher order convergence rates for the Adam optimizer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I5TRNYLZ}},
  note         = {Machine review of arXiv:2504.19426}
}

read the original abstract

Gradient descent based optimization methods are the methods of choice to train deep neural networks in machine learning. Beyond the standard gradient descent method, also suitable modified variants of standard gradient descent involving acceleration techniques such as the momentum method and/or adaptivity techniques such as the RMSprop method are frequently considered optimization methods. These days the most popular of such sophisticated optimization schemes is presumably the Adam optimizer that has been proposed in 2014 by Kingma and Ba. A highly relevant topic of research is to investigate the speed of convergence of such optimization methods. In particular, in 1964 Polyak showed that the standard gradient descent method converges in a neighborhood of a strict local minimizer with rate (x - 1)(x + 1)^{-1} while momentum achieves the (optimal) strictly faster convergence rate (\sqrt{x} - 1)(\sqrt{x} + 1)^{-1} where x \in (1,\infty) is the condition number (the ratio of the largest and the smallest eigenvalue) of the Hessian of the objective function at the local minimizer. It is the key contribution of this work to reveal that Adam also converges with the strictly faster convergence rate (\sqrt{x} - 1)(\sqrt{x} + 1)^{-1} while RMSprop only converges with the convergence rate (x - 1)(x + 1)^{-1}.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks
math.OC 2026-07 accept novelty 7.0 of 10

Bounded trajectories of a broad class of GD optimizers (Adam, RMSprop, NAG, Adan, etc.) converge with polynomial rates to critical points of KL objectives with locally Lipschitz gradients, covering analytic-activation...
Adaptive Preconditioners Trigger Loss Spikes in Adam
cs.LG 2025-06 conditional novelty 6.0 of 10

Loss spikes in Adam occur when its second-moment memory decays faster than gradients grow, briefly removing the adaptive brake; a single Hessian-vector product along the gradient direction can flag the onset.

Reference graph

Works this paper leans on

41 extracted references · 20 canonical work pages · cited by 2 Pith papers

[2]

Convergence rates of a momentum algorithm with bounded adaptive step size for nonconvex optimization

Barakat, A., and Bianchi, P. Convergence rates of a momentum algorithm with bounded adaptive step size for nonconvex optimization. arXiv:1911.07596 (2020), 28 pages

arXiv 2020
[1]

Learning Theory from First Principles , ﬁrst edition ed

Bach, F. Learning Theory from First Principles , ﬁrst edition ed. The MIT Press, Cambridge, MA, 2024

work page 2024
[3]

Convergence and dynamical behavior of the Adam algorithm for nonconvex stochastic optimization

Barakat, A., and Bianchi, P. Convergence and dynamical behavior of the Adam algorithm for nonconvex stochastic optimization. SIAM J. Optim. 31 , 1 (2021), 244– 274

work page 2021
[4]

Stochastic optimization with momentum: convergence, ﬂuctuations, an d traps avoidance

Barakat, A., Bianchi, P., Hachem, W., and Schechtman, S. Stochastic optimization with momentum: convergence, ﬂuctuations, an d traps avoidance. Elec- tron. J. Stat. 15 , 2 (2021), 3892–3947

work page 2021
[5]

S., and von Wurstemberge r, P

Becker, S., Jentzen, A., Müller, M. S., and von Wurstemberge r, P. Learning the random variables in Monte Carlo simulations wi th stochastic gradient descent: machine learning for parametric PDEs and ﬁnancial derivative pricing. Math. Finance 34 , 1 (2024), 90–150. 24

work page 2024
[6]

A Proof of Local Convergence for the Adam Optimizer

Bock, S., and Weiß, M. A Proof of Local Convergence for the Adam Optimizer. In 2019 International Joint Conference on Neural Networks (IJ CNN) (2019), pp. 1– 8

work page 2019
[7]

Méthode générale pour la résolution des systèmes d’équatio ns si- multanées

Cauchy, A.-L. Méthode générale pour la résolution des systèmes d’équatio ns si- multanées. C. R. Acad. Sci. 25 (1847), 536–538

work page
[8]

On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization

Chen, X., Liu, S., Sun, R., and Hong, M. On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization. arXiv:1808.02941 (2019), 30 pages

arXiv 2019

Show all 41 references

[9]

Non-convergence of stochas- tic gradient descent in the training of deep neural networks

Cheridito, P., Jentzen, A., and Rossmannek, F. Non-convergence of stochas- tic gradient descent in the training of deep neural networks . Journal of Complexity 64 (2021), 101540

2021
[10]

A Simple Convergence Proof of Adam and Adagrad

Défossez, A., Bottou, L., Bach, F., and Usunier, N. A Simple Convergence Proof of Adam and Adagrad. Transactions on Machine Learning Research (2022)

2022
[11]

Non-convergence of Adam and other adaptive stochastic gradient descent optimization m ethods for non-vanishing learning rates

Dereich, S., Graeber, R., and Jentzen, A. Non-convergence of Adam and other adaptive stochastic gradient descent optimization m ethods for non-vanishing learning rates. arXiv:2407.08100 (2024), 54 pages

2024 arXiv
[12]

Convergence rates for the Adam optimizer

Dereich, S., and Jentzen, A. Convergence rates for the Adam optimizer. arXiv:2407.21078 (2024), 43 pages

2024 arXiv
[13]

Dereich, S., Jentzen, A., and Riekert, A. Learning rate adaptive stochastic gradient descent optimization methods: numeri cal simulations for deep learning methods for partial diﬀerential equations and con vergence analyses. arXiv:2406.14340 (2024), 68 pages

2024 arXiv
[14]

Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapes

Dereich, S., and Kassing, S. Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapes. arXiv:2102.09385 (2021), 24 pages

2021 arXiv
[15]

Do, T., Hannibal, S., and Jentzen, A. Non-convergence to global minimiz- ers in data driven supervised deep learning: Adam and stocha stic gradient descent optimization provably fail to converge to global minimizer s in the training of deep neural networks with ReLU activation...

2024 arXiv
[16]

On the oracle complexity of smooth strongly convex minimization

Drori, Y., and Taylor, A. On the oracle complexity of smooth strongly convex minimization. J. Complexity 68 (2022), Paper No. 101590, 20

2022
[17]

Adaptive subgradient methods for online learning and stochastic optimization

Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res. 12 (2011), 2121–2159

2011
[18]

Towards a Mathematical Understanding of Neural Network-Based Machine Learning: w hat we know and what we don’t

E, W., Ma, C., Wojtowytsch, S., and Wu, L. Towards a Mathematical Understanding of Neural Network-Based Machine Learning: w hat we know and what we don’t. arXiv:2009.10713 (2020), 56 pages

2020 arXiv
[19]

Blow up phenomena for gra- dient descent optimization methods in the training of artiﬁ cial neural networks

Gallon, D., Jentzen, A., and Lindner, F. Blow up phenomena for gra- dient descent optimization methods in the training of artiﬁ cial neural networks. arXiv:2211.15641 (2022), 84 pages. 25

2022 arXiv
[20]

Garrigos, G., and Gower, R. M. Handbook of Convergence Theorems for (Stochastic) Gradient Methods. arXiv:2301.11235 (2023), 59 pages

2023 arXiv
[21]

Non asymptotic analysis of Adaptive stochastic gradient algorithms and applications

Godichon-Baggioni, A., and Tarrago, P. Non asymptotic analysis of Adaptive stochastic gradient algorithms and applications. arXiv:2303.01370 (2023), 59 pages

2023 arXiv
[22]

Convergence of Adam for Non-convex Ob- jectives: Relaxed Hyperparameters and Non-ergodic Case

He, M., Liang, Y., Liu, J., and Xu, D. Convergence of Adam for Non-convex Ob- jectives: Relaxed Hyperparameters and Non-ergodic Case. arXiv:2307.11782 (2023), 39 pages

2023 arXiv
[23]

Lecture 6e: Rm- sprop: Divide the gradient by a running average of its recent magnitude

Hinton, G., Sriv asta v a, N., and Swersky, K. Lecture 6e: Rm- sprop: Divide the gradient by a running average of its recent magnitude. https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf. [Accessed 01-December-2017]

2017
[24]

Revisiting Convergence of AdaGrad with Relaxed Assump- tions

Hong, Y., and Lin, J. Revisiting Convergence of AdaGrad with Relaxed Assump- tions. arXiv:2402.13794 (2024), 30 pages

2024 arXiv
[25]

A., and Johnson, C

Horn, R. A., and Johnson, C. R. Matrix analysis , second edition, corrected reprint ed. Cambridge University Press, New York, NY, 2017

2017
[26]

Strong error analysis for stochastic gradient descent opti mization algorithms

Jentzen, A., Kuckuck, B., Neufeld, A., and von Wurstemberge r, P. Strong error analysis for stochastic gradient descent opti mization algorithms. IMA J. Numer. Anal. 41 , 1 (2021), 455–492

2021
[27]

Mathemat- ical Introduction to Deep Learning: Methods, Implementati ons, and Theory

Jentzen, A., Kuckuck, B., and von Wurstemberger, P. Mathemat- ical Introduction to Deep Learning: Methods, Implementati ons, and Theory. arXiv:2310.20360 (2023), 714 pages

2023 arXiv
[28]

Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and construc tions of local minimizers in the training of artiﬁcial neural networks

Jentzen, A., and Riekert, A. Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and construc tions of local minimizers in the training of artiﬁcial neural networks. arXiv:2402.05155 (2024), 36 pages

2024 arXiv
[29]

Lower error bounds for the stochas- tic gradient descent optimization algorithm: sharp conver gence rates for slowly and fast decaying learning rates

Jentzen, A., and von Wurstemberger, P. Lower error bounds for the stochas- tic gradient descent optimization algorithm: sharp conver gence rates for slowly and fast decaying learning rates. J. Complexity 57 (2020), 101438, 16

2020
[30]

P., and Ba, J

Kingma, D. P., and Ba, J. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 (2014), 15 pages

2014 arXiv
[31]

Convergence of Adam Under Relaxed Assumptions

Li, H., Rakhlin, A., and Jadbabaie, A. Convergence of Adam Under Relaxed Assumptions. arXiv:2304.13972 (2023), 35 pages

2023 arXiv
[32]

Dying ReLU and Initialization: Theory and Numerical Exampl es

Lu, L. Dying ReLU and Initialization: Theory and Numerical Exampl es. Commu- nications in Computational Physics 28 , 5 (June 2020), 1671–1706

2020
[33]

A method of solving a convex programming problem with conver - gence rate o(1/k2)

Nesterov, Y. A method of solving a convex programming problem with conver - gence rate o(1/k2). In Soviet Mathematics Doklady (1983), vol. 27, pp. 372–376

1983
[34]

Introductory lectures on convex optimization , vol

Nesterov, Y. Introductory lectures on convex optimization , vol. 87 of Applied Optimization. Kluwer Academic Publishers, Boston, MA, 2004. A basic cour se. 26

2004
[35]

Gradient methods for the minimisation of functionals

Polyak, B. Gradient methods for the minimisation of functionals. USSR Compu- tational Mathematics and Mathematical Physics 3 , 4 (1963), 864–878

1963
[36]

Some methods of speeding up the convergence of iteration met hods

Polyak, B. Some methods of speeding up the convergence of iteration met hods. USSR Computational Mathematics and Mathematical Physics 4 , 5 (1964), 1–17

1964
[37]

J., Kale, S., and Kumar, S

Reddi, S. J., Kale, S., and Kumar, S. On the Convergence of Adam and Beyond. arXiv:1904.09237 (2019), 23 pages

2019 arXiv
[38]

An overview of gradient descent optimization algorithms

Ruder, S. An overview of gradient descent optimization algorithms. arXiv:1609.04747 (2017), 14 pages

2017 arXiv
[39]

Optimization for deep learning: theory and algorithms

Sun, R. Optimization for deep learning: theory and algorithms. arXiv:1912.08957 (2019), 60 pages

2019 arXiv
[40]

Adam Can Converge Without Any Modiﬁcation On Update Rules

Zhang, Y., Chen, C., Shi, N., Sun, R., and Luo, Z.-Q. Adam Can Converge Without Any Modiﬁcation On Update Rules. arXiv:2208.09632 (2022), 68 pages

2022 arXiv
[41]

A Suﬃcient Condition for Convergences of Adam and RMSProp

Zou, F., Shen, L., Jie, Z., Zhang, W., and Liu, W. A Suﬃcient Condition for Convergences of Adam and RMSProp. arXiv:1811.09358 (2018), 24 pages. 27

2018 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [2]

Convergence rates of a momentum algorithm with bounded adaptive step size for nonconvex optimization

Barakat, A., and Bianchi, P. Convergence rates of a momentum algorithm with bounded adaptive step size for nonconvex optimization. arXiv:1911.07596 (2020), 28 pages

arXiv 2020

[2] [1]

Learning Theory from First Principles , ﬁrst edition ed

Bach, F. Learning Theory from First Principles , ﬁrst edition ed. The MIT Press, Cambridge, MA, 2024

work page 2024

[3] [3]

Convergence and dynamical behavior of the Adam algorithm for nonconvex stochastic optimization

Barakat, A., and Bianchi, P. Convergence and dynamical behavior of the Adam algorithm for nonconvex stochastic optimization. SIAM J. Optim. 31 , 1 (2021), 244– 274

work page 2021

[4] [4]

Stochastic optimization with momentum: convergence, ﬂuctuations, an d traps avoidance

Barakat, A., Bianchi, P., Hachem, W., and Schechtman, S. Stochastic optimization with momentum: convergence, ﬂuctuations, an d traps avoidance. Elec- tron. J. Stat. 15 , 2 (2021), 3892–3947

work page 2021

[5] [5]

S., and von Wurstemberge r, P

Becker, S., Jentzen, A., Müller, M. S., and von Wurstemberge r, P. Learning the random variables in Monte Carlo simulations wi th stochastic gradient descent: machine learning for parametric PDEs and ﬁnancial derivative pricing. Math. Finance 34 , 1 (2024), 90–150. 24

work page 2024

[6] [6]

A Proof of Local Convergence for the Adam Optimizer

Bock, S., and Weiß, M. A Proof of Local Convergence for the Adam Optimizer. In 2019 International Joint Conference on Neural Networks (IJ CNN) (2019), pp. 1– 8

work page 2019

[7] [7]

Méthode générale pour la résolution des systèmes d’équatio ns si- multanées

Cauchy, A.-L. Méthode générale pour la résolution des systèmes d’équatio ns si- multanées. C. R. Acad. Sci. 25 (1847), 536–538

work page

[8] [8]

On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization

Chen, X., Liu, S., Sun, R., and Hong, M. On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization. arXiv:1808.02941 (2019), 30 pages

arXiv 2019