Pith. sign in

REVIEW 4 major objections 5 minor 55 references

Simple Convergence Proof of Adam From a Sign-like Descent Perspective

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Vanilla Adam, unmodified, is proved to converge at the optimal $O(1/T^{1/4})$ rate for non-convex stochastic objectives, with a bound free of dimension and epsilon.

desk verdict The sign-descent reformulation is fresh and simplifies part of the analysis, but the advertised O(1/T^{1/4}) gradient-norm rate rests on an unstated independence assumption and an unproven positive lower bound, so Corollary 3.3 as written does not follow. read the letter →

arxiv 2507.05966 v1 pith:MSQA637E submitted 2025-07-08 cs.LG cs.AIcs.ITmath.IT

Hanyang Peng , Shuang Qin , Yue Yu , Fangqing Jiang , Hui Wang , Zhouchen Lin This is my paper
classification cs.LGcs.AIcs.ITmath.IT MSC 90C2690C1568T0768W40
keywords Adamoptimizernon-convexstochasticoptimizationsigndescentconvergencerate(L0L1q)-smoothnessp-affinevariancemomentumadaptive
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the vanilla version of Adam, without any modification to the update rule, converges in expectation at the rate $O(1/T^{1/4})$ for non-convex stochastic objectives. The rate is proved under generalized $(L_0,L_1,q)$-smoothness and generalized $p$-affine variance, and the constant does not depend on the model dimension $d$ or the numerical stability parameter $\epsilon$. Earlier proofs of Adam-type methods reached only $O(\ln T/T^{1/4})$ or needed bounded gradients, bounded variance, or large $\epsilon$, so this would be the first rate for plain Adam that matches the lower bound known for stochastic non-convex optimization. The authors' key move is to read Adam as a sign-like descent update, $x_{t+1}=x_t-\gamma_t (|m_t|/(\sqrt{v_t}+\epsilon))\circ\mathrm{Sign}(m_t)$, which turns the proof into a short decomposition of a weighted descent term and a momentum-mismatch error. A sympathetic reader would care because Adam is the default optimizer for large neural networks, and the result says the plain algorithm can be justified at the optimal rate without extra stabilization tricks.

What carries the argument

The central object is the normalized momentum vector $u_t:=|m_t|/(\sqrt{v_t}+\epsilon)$, treated as a single random variable rather than as the inverse of a preconditioner. Writing the update as $x_{t+1}=x_t-\gamma_t u_t\circ\mathrm{Sign}(m_t)$ makes the main inner product equal to the weighted $\ell^1$ norm $\|u_t\circ\nabla F(x_t)\|_1$, which is direct descent progress. The boundedness lemma gives $u_t^{(j)}\le R$ with $R=(1-\beta_1)/\sqrt{(1-\beta_2)(1-\beta_1^2/\beta_2)}$ under $\beta_1^2<\beta_2$, so the magnitude factor is uniformly controlled; a Markov-type lemma bounds the sign mismatch $\mathbb{E}[|\mathrm{Sign}(m_t)-\mathrm{Sign}(\nabla F)|]$ by $2\mathbb{E}[|m_t-\nabla F|]/|\nabla F|$. The proof's division of labor is that the sign term carries the descent, while a recursion on $m_t-\nabla F(x_t)$ carries the error.

What would settle it

Run vanilla Adam on a one-dimensional quadratic objective with slowly decaying gradient noise, and record the per-coordinate values $u_t=|m_t|/(\sqrt{v_t}+\epsilon)$. If $\min_t \mathbb{E}[u_t]$ approaches 0 while the objective value keeps decreasing, then the positive constant $\bar v$ required by Corollary 3.3 is absent, and the proof does not produce its claimed $O(1/T^{1/4})$ gradient-norm bound in that regime.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that Adam's update separates into a magnitude factor and a sign factor, $u_t=|m_t|/(\sqrt{v_t}+\epsilon)$ and $\mathrm{Sign}(m_t)$, and that this separation makes the existing convergence machinery unnecessary. Under Assumptions A, B.3 and C.3, with Conditions 1--3, Corollary 3.3 gives $(1/T)\sum_{t=0}^{T-1}\mathbb{E}[\|\nabla F(x_t)\|_2]\le (C_1/\bar v)\,(2\Delta_0/(C_2 T^{1/4} d^{1/2}) + 4R\|\nabla F(x_0)\|_2/(C_3 T^{1/2}) + 4C_3 R\hat\sigma/T^{1/4} + 4C_2 R^2\hat L/(C_3 T^{1/4}) + C_2 R^2\hat L/T^{7/4})$ in the general case, and a sharpened leading constant $512^{1/4} R \hat\sigma^{1/2}\hat L^{1/4}\Delta_0/T^{1/4}$ when the hyperparameters are tuned with oracle knowledge. The proof bounds the momentum-mismatch term $\mathbb{E}[\|m_t-\nabla F(x_t)\|_2]$ by a combination of the gradient norm, the noise level, and the learning rate; Condition 2 lets the weighted $\ell^1$ progress term factor into $\mathbb{E}[u_t^{(j)}]\,\mathbb{E}[\|\nabla F(x_t)\|_1]$, and Condition 3 converts the $\ell^1$ norm to the $\ell^2$ norm. The required positivity gap $\bar v\le \min_t\mathbb{E}[u_t^{(j)}]$ is what converts the weighted norm into the plain gradient-norm rate. The paper's own limitations note concedes that the weaker $p$-affine variance case is bought at the price of Conditions 1--3; the bounded-variance version drops those conditions but keeps a $d^{1/2}$ dependence.

Load-bearing premise

The whole argument turns on the expected ratio $|m_t|/(\sqrt{v_t}+\epsilon)$ staying strictly above a fixed positive constant in every coordinate; if that lower bound fails, only a weighted norm is shown to converge and the $O(1/T^{1/4})$ gradient-norm claim is not established.

Editorial extensions

If this is right

  • Vanilla Adam, not a corrected variant, is proved to converge at the one-over-fourth-root-T rate; the earlier logarithmic factor disappears.
  • The bound is independent of model dimension and epsilon, so the theory covers large-width, small-epsilon training regimes.
  • The allowed hyperparameter choice uses 1 minus beta1 proportional to 1 over the square root of T, so Adam keeps its adaptive character; earlier proofs pushed beta2 so close to 1 that Adam became AdaGrad-like.
  • Momentum is the deciding factor: without it, the same analysis gives only convergence to a noise-limited neighborhood, matching known signSGD results.
  • With oracle-tuned constants, the learning rate scales as 1 over T^{3/4} times 1 over the square root of d, so larger models call for smaller learning rates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not establish a positive lower bound on the expected normalized momentum; monitoring that quantity during real Adam runs is a direct way to see whether the theorem's regime holds in practice.
  • A natural extension would replace the coordinate-wise i.i.d. assumption with a weaker permutation-invariance condition, since the proof only needs the weighted L1 term to factor.
  • The sign-descent decomposition is likely to transfer to other adaptive methods whose updates can be written as magnitude times sign of a momentum, though the paper does not make that transfer.
  • If the rate is sharp, improvements to Adam in this setting must come from reducing the constants, not from changing the exponent.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a sign-like interpretation of Adam, rewriting the update as x_{t+1}=x_t-γ_t (|m_t|/(√v_t+ε))∘Sign(m_t), and develops a convergence proof for vanilla Adam under generalized (L0,L1,q)-smoothness and p-affine variance. The main result (Theorem 3.1) bounds a weighted gradient norm E[||u_t∘∇F(x_t)||_1], and Corollaries 3.2 and 3.3 convert this into bounds on the plain gradient norm E[||∇F(x_t)||_2], claiming the first O(1/T^{1/4}) rate for vanilla Adam without explicit dependence on model dimensionality d or the numerical stability parameter ε. The proof relies on three conditions on the realized gradients and on a positive lower bound for the weights u_t^(j). The paper is largely self-contained and includes empirical validation of two of the conditions.

Significance. If the central claim were correct, the sign-like perspective would be a substantial simplification and improvement over existing analyses of Adam, replacing the previous O(ln T/T^{1/4}) bounds with O(1/T^{1/4}) under weak smoothness and variance assumptions. The proof is self-contained and the sign-like reformulation is elegant, potentially opening a new route for the theory of adaptive optimizers. However, the advertised plain-gradient-norm convergence is not established: the derivation contains a false independence step and an unjustified positivity assumption, and both are load-bearing for the main claim. The empirical checks for Conditions 2 and 3 also do not support the form in which the conditions are used.

major comments (4)
  1. [Appendix C, Eq. (55)(i)] The factorization E[u_t^(j)|∇F(x_t)^(j)|] = E[u_t^(j)] E[|∇F(x_t)^(j)|] is asserted by claiming that u_t^(j) and |∇F(x_t)^(j)| are 'mutually independent'. Condition 2 only states that the coordinates of u_t are i.i.d. across j at each time t; it does not imply independence between u_t and the current true gradient. Since u_t is a function of g_1,...,g_t and x_t is a function of g_1,...,g_{t-1}, the two generally share randomness through the past trajectory, and E[u_t | x_t] is not constant in general. This step is the only bridge from the weighted-norm bound in Theorem 3.1 to the plain gradient-norm bounds in Corollary 3.3 and Finding 1, so Corollary 3.3 as stated does not follow from the stated assumptions.
  2. [Corollaries 3.2 and 3.3, statements of v and v-bar] Both corollaries assume a positive lower bound: 0 < v ≤ min_{t,j} u_t^(j) in Corollary 3.2 and 0 < v-bar ≤ min_t E[u_t^(j)] in Corollary 3.3. This assumption is not implied by Conditions 1-3 or by the smoothness/variance assumptions. For example, if the deterministic gradient decays to zero, then m_t decays to zero while √v_t also decays, and because ε is fixed, u_t^(j) = |m_t^(j)|/(√v_t^(j)+ε) can tend to zero; thus E[u_t^(j)] may become arbitrarily small or zero. Since the final bounds divide by v or v-bar, the advertised O(1/T^{1/4}) rate collapses unless such a positive lower bound is separately guaranteed, which the paper does not establish.
  3. [Lemma B.3 and Eq. (41)(i)] Lemma B.3 states E[|Sign(Z)-Sign(C)|] ≤ 2E[|Z-C|]/|C|, which is undefined when C=0. In the proof of Theorem 3.1, this lemma is applied coordinate-wise with C=∇F(x_t)^(j), which can be zero for coordinates where the true gradient component vanishes. The proof does not address this case, so the inequality chain leading to Eq. (41) is not valid as written for functions with zero gradient components on the trajectory.
  4. [Condition 1 and Eqs. (48)(x)-(51)] Condition 1 is stated as a condition on the realized gradient norms, with no expectation: sqrt((1/T)Σ||∇F(x_t)||_2^2) ≤ (C0/T)Σ||∇F(x_t)||_2. In the proof of Theorem 3.1, this realized quantity is used inside the derivation of the bound on E[||m_t-∇F(x_t)||_2] (Eq. (48)(x)), and then appears in Eq. (50) without an expectation, while the theorem statement and the final inequality in Eq. (51) contain expectations of ||∇F(x_t)||_2. The proof does not justify replacing the realized sum by its expectation, so the theorem statement does not follow from the assumptions as written; either Condition 1 must be stated in expectation form or an additional argument is needed.
minor comments (5)
  1. [Lemma B.4] The tightness statement in Lemma B.4 is not rigorously justified: the claim that s_- = (a+b^{1/γ})/2 does not satisfy Eq. (37) is false in some cases, e.g., when a=b^{1/γ}. The lemma may still be true with a different argument, but the proof as written is not fully correct.
  2. [Finding 1 and Corollary 3.3] The claim that convergence is 'dimension-free' is overstated: in Case 1 of Corollary 3.3, the bound contains a factor C1 that is defined through Condition 3 using √d, and the first term in Eq. (12) contains d^{-1/2}. Even in Case 2, the factor C1/v-bar may implicitly depend on d unless additional assumptions are imposed.
  3. [Section 3, Condition 2 and Figure 1] The two-sample Kolmogorov-Smirnov test used to validate Condition 2 checks whether two groups of coordinates have the same marginal distribution; it does not test whether the coordinates are independent across j. The caption's conclusion that the coordinates are 'independently drawn from the identical distribution' is therefore not supported by the reported experiment.
  4. [Notation, Theorem 3.1 and Corollary 3.2] The definition of R is inconsistent between Theorem 3.1 and Corollary 3.2: Theorem 3.1 has R=(1-β1)/sqrt((1-β2)(1-β1^2/β2)), while Corollary 3.2 displays R=1-β1/√... without parentheses, which appears to be a typo. Please correct the notation.
  5. [Abstract and Introduction] There is a typo 'condidtions' in the Introduction, and the phrase 'optimal rate' is used without a formal lower-bound citation at that point; the lower bound in [1] is cited later, but the abstract should be precise about the sense in which O(1/T^{1/4}) is optimal.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: derivation is a self-contained proof from stated assumptions; the flagged gaps are soundness concerns, not circular reasoning.

full rationale

The paper's central claim—that vanilla Adam converges at O(T^{-1/4}) under generalized (L0,L1,q)-smoothness and p-affine variance—is obtained by a self-contained proof chain: Theorem 3.1 controls the weighted quantity (1/T) Σ E[‖u_t ∘ ∇F(x_t)‖_1], and Corollaries 3.2 and 3.3 convert this to plain gradient norms using Conditions 1–3 and the explicit lower-bound choices 0 < v ≤ min_{t,j} u_t^{(j)} and 0 < v̄ ≤ min_t E[u_t^{(j)}]. These conditions and the constants C2, C3 are assumptions or free hyperparameter choices, not quantities fitted to any data or derived from the desired rate. No parameter is tuned to force the final bound, and no prediction is produced by renaming an input. The apparent independence step in Eq. (55)(i)—asserting that u_t^{(j)} and |∇F^{(j)}(x_t)| are mutually independent—is not implied by Condition 2's coordinate-wise i.i.d. statement, and the positivity of v̄ is not guaranteed near stationarity; but these are correctness and assumption-strength gaps, not circularity. The paper itself discloses in its Limitations section that Conditions 1–3 are additional constraints needed for the weaker noise assumption, which further confirms these are openly stated hypotheses rather than hidden equivalents of the conclusion. There are also no load-bearing self-citations: references to prior empirical sign-descent observations motivate the reinterpretation but do not carry the proof. The derivation therefore does not reduce to its own inputs, and no circular step can be exhibited from the text.

Assumptions & free parameters 2 free parameters · 7 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. It relies on standard smoothness and variance assumptions, plus three conditions that are tailored to the proof (Conditions 1-3) and a positivity lower bound v-bar. Conditions 2 and 3 and the v-bar bound are the most restrictive and are only weakly supported empirically.

free parameters (2)
  • C2 (learning-rate schedule constant) = chosen by user
    Appears in gamma = C2/(T^{3/4} d^{1/2}). The theorem holds for any positive C2, but the bound depends on it. It is not fitted to data.
  • C3 (momentum schedule constant) = chosen by user
    Appears in 1 - beta1 = C3/T^{1/2}. The bound depends on C3, and the convergence rate is valid for any positive C3 above the stated threshold.
assumptions (7)
  • domain assumption Assumption A: the objective F is bounded below by F*.
    Standard in non-convex optimization; invoked throughout the proof.
  • domain assumption Assumption B.3: (L0, L1, q)-smoothness, ||grad F(y) - grad F(x)|| <= (L0 + L1||grad F(x)||^q)||y - x||.
    Assumed for the descent lemma (Lemma B.1), imported from prior work [30, 49].
  • domain assumption Assumption C.3: p-affine variance, E||grad f(x; zeta) - grad F(x)||^2 <= sigma0^2 + sigma1^2 ||grad F(x)||^p.
    Used to bound the noise term in the momentum error (Eq. 48).
  • ad hoc to paper Condition 1: sqrt((1/T) sum ||grad F(x_t)||^2) <= (C0/T) sum ||grad F(x_t)||.
    Introduced to bound the p-affine noise term linearly. Not standard, though argued to hold for polynomially decaying gradients.
  • ad hoc to paper Condition 2: the coordinates u_t^(j) = |m_t^(j)|/(sqrt(v_t^(j)) + epsilon) are i.i.d. across j.
    Used in Eq. (55) to factor expectations and replace a weighted norm with a plain norm. Independence is not empirically verified.
  • ad hoc to paper Condition 3: ||grad F(x_t)||_1 = (sqrt(d)/C1) ||grad F(x_t)||_2 with 1 <= C1 << sqrt(d).
    Converts l1 convergence into l2 convergence. Empirically suggested for two layers, but not generally guaranteed.
  • ad hoc to paper Positive lower bound: there exists v-bar > 0 with v-bar <= min_t E[u_t^(j)] for all j (and v <= min_{t,j} u_t^(j) in Corollary 3.2).
    Essential to pass from the weighted-norm bound to the unweighted gradient-norm bound. Not guaranteed by the algorithm; can fail near stationarity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Simple Convergence Proof of Adam From a Sign-like Descent Perspective." pith.science (2026). https://pith.science/paper/MSQA637E

@misc{pith2026250705966,
  author       = {Pith},
  title        = {Pith review of: Simple Convergence Proof of Adam From a Sign-like Descent Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MSQA637E}},
  note         = {Machine review of arXiv:2507.05966}
}
abstract

Adam is widely recognized as one of the most effective optimizers for training deep neural networks (DNNs). Despite its remarkable empirical success, its theoretical convergence analysis remains unsatisfactory. Existing works predominantly interpret Adam as a preconditioned stochastic gradient descent with momentum (SGDM), formulated as $\bm{x}_{t+1} = \bm{x}_t - \frac{\gamma_t}{{\sqrt{\bm{v}_t}+\epsilon}} \circ \bm{m}_t$. This perspective necessitates strong assumptions and intricate techniques, resulting in lengthy and opaque convergence proofs that are difficult to verify and extend. In contrast, we propose a novel interpretation by treating Adam as a sign-like optimizer, expressed as $\bm{x}_{t+1} = \bm{x}_t - \gamma_t \frac{|\bm{m}_t|}{{\sqrt{\bm{v}_t}+\epsilon}} \circ {\rm Sign}(\bm{m}_t)$. This reformulation significantly simplifies the convergence analysis. For the first time, with some mild conditions, we prove that Adam achieves the optimal rate of ${\cal O}(\frac{1}{T^{\sfrac{1}{4}}})$ rather than the previous ${\cal O} \left(\frac{\ln T}{T^{\sfrac{1}{4}}}\right)$ under weak assumptions of the generalized $p$-affine variance and $(L_0, L_1, q)$-smoothness, without dependence on the model dimensionality or the numerical stability parameter $\epsilon$. Additionally, our theoretical analysis provides new insights into the role of momentum as a key factor ensuring convergence and offers practical guidelines for tuning learning rates in Adam, further bridging the gap between theory and practice.

Figures

Figures reproduced from arXiv: 2507.05966 by the authors.

Figure 1
Figure 1. The distribution an the two-sample Kolmogorov-Smirnov test for m(j) t / q v (j) t across coordinates of (a) Layer#32.conv.weight in ResNet-50 during training with Adam on ImageNet for 150 epochs, and (b) Layer#3.self-attention.in-proj-weight in GPT-2 (350M) during training with Adam on OpenWebText for 5,000 iterations. In this test, two groups of 10,000 samples were uniformly drawn from all coordinates of the layer,… view at source ↗
Figure 2
Figure 2. The distribution and C1 = √ d∥∇f(xt)∥2 ∥∇f(xt)∥1 for gradients across coordinates of (a) Layer#32.conv.weight in ResNet-50 during training with Adam on ImageNet for 150 epochs, and (b) Layer#3.self-attention.in-proj￾weight in GPT-2 (350M) during training with Adam on OpenWebText for 50,000 iterations. Throughout training, C1 remains consistently below 3, which is significantly smaller than √ d, where d represents th… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

55 extracted references · 32 canonical work pages

  1. [1]

    Lower bounds for non-convex stochastic optimization.Mathematical Programming, 199(1-2):165–214, 2023

    Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Wood- worth. Lower bounds for non-convex stochastic optimization.Mathematical Programming, 199(1-2):165–214, 2023. 9

    work page 2023

  2. [2]

    SGD with AdaGrad stepsizes: Full adaptivity with high probability to unknown parameters, unbounded gradients and affine variance

    Amit Attia and Tomer Koren. SGD with AdaGrad stepsizes: Full adaptivity with high probability to unknown parameters, unbounded gradients and affine variance. InInternational Conference on Machine Learning, pages 1147–1171. PMLR, 2023

    work page 2023

  3. [3]

    Dissecting Adam: The sign, magnitude and variance of stochastic gradients

    Lukas Balles and Philipp Hennig. Dissecting Adam: The sign, magnitude and variance of stochastic gradients. InInternational Conference on Machine Learning, pages 404–413, 2018

    work page 2018

  4. [4]

    signSGD: Compressed optimisation for non-convex problems

    Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signSGD: Compressed optimisation for non-convex problems. InInternational Conference on Machine Learning, pages 560–569, 2018

    work page 2018

  5. [5]

    Gradient convergence in gradient methods with errors.SIAM Journal on Optimization, 10(3):627–642, 2000

    Dimitri P Bertsekas and John N Tsitsiklis. Gradient convergence in gradient methods with errors.SIAM Journal on Optimization, 10(3):627–642, 2000

    2000

  6. [6]

    Optimization methods for large-scale machine learning.SIAM review, 60(2):223–311, 2018

    Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning.SIAM review, 60(2):223–311, 2018

    2018

  7. [7]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  8. [8]

    Towards practical Adam: Non-convexity, convergence theory, and mini-batch acceleration.Journal of Machine Learning Research, 23(229):1–47, 2022

    Congliang Chen, Li Shen, Fangyu Zou, and Wei Liu. Towards practical Adam: Non-convexity, convergence theory, and mini-batch acceleration.Journal of Machine Learning Research, 23(229):1–47, 2022

    work page 2022

Show all 55 references
  1. [9]

    Lion secretly solves constrained optimization: As lyapunov predicts.arXiv preprint arXiv:2310.05898, 2023

    Lizhang Chen, Bo Liu, Kaizhao Liang, and Qiang Liu. Lion secretly solves constrained optimization: As lyapunov predicts.arXiv preprint arXiv:2310.05898, 2023

    2023 arXiv

  2. [10]

    Symbolic discovery of optimization algo- rithms.arXiv preprint arXiv:2302.06675, 2023

    Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, et al. Symbolic discovery of optimization algo- rithms.arXiv preprint arXiv:2302.06675, 2023

    2023 arXiv

  3. [11]

    On the convergence of a class of Adam-type algorithms for non-convex optimization.arXiv preprint arXiv:1808.02941, 2018

    Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of a class of Adam-type algorithms for non-convex optimization.arXiv preprint arXiv:1808.02941, 2018

    2018 arXiv

  4. [12]

    PaLM: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1– 113, 2023

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1– 113, 2023

    2023

  5. [13]

    Ro- bustness to unbounded smoothness of generalized signSGD

    Michael Crawshaw, Mingrui Liu, Francesco Orabona, Wei Zhang, and Zhenxun Zhuang. Ro- bustness to unbounded smoothness of generalized signSGD. InAdvances in neural information processing systems, pages 9955–9968, 2022

    2022

  6. [14]

    A simple convergence proof of Adam and AdaGrad.arXiv preprint arXiv:2003.02395, 2020

    Alexandre Défossez, Léon Bottou, Francis Bach, and Nicolas Usunier. A simple convergence proof of Adam and AdaGrad.arXiv preprint arXiv:2003.02395, 2020

    2003 arXiv

  7. [15]

    The Llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  8. [16]

    Beyond uniform smoothness: A stopped analysis of adaptive SGD

    Matthew Faw, Litu Rout, Constantine Caramanis, and Sanjay Shakkottai. Beyond uniform smoothness: A stopped analysis of adaptive SGD. InThe Thirty Sixth Annual Conference on Learning Theory, pages 89–160. PMLR, 2023

    2023

  9. [17]

    The power of adaptivity in SGD: Self-tuning step sizes with unbounded gradients and affine variance

    Matthew Faw, Isidoros Tziotis, Constantine Caramanis, Aryan Mokhtari, Sanjay Shakkottai, and Rachel Ward. The power of adaptivity in SGD: Self-tuning step sizes with unbounded gradients and affine variance. InConference on Learning Theory, pages 313–355. PMLR, 2022

    2022

  10. [18]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InIEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016

    2016

  11. [19]

    Neural networks for machine learning lecture 6a overview of mini-batch gradient descent.Cited on, 14(8):2, 2012

    Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent.Cited on, 14(8):2, 2012. 10

    2012

  12. [20]

    High probability convergence of Adam under unbounded gradients and affine variance noise.arXiv preprint arXiv:2311.02000, 2023

    Yusu Hong and Junhong Lin. High probability convergence of Adam under unbounded gradients and affine variance noise.arXiv preprint arXiv:2311.02000, 2023

    2023 arXiv

  13. [21]

    On convergence of Adam for stochastic optimization under relaxed assumptions.arXiv preprint arXiv:2402.03982v2, 2025

    Yusu Hong and Junhong Lin. On convergence of Adam for stochastic optimization under relaxed assumptions.arXiv preprint arXiv:2402.03982v2, 2025

    2025 arXiv

  14. [22]

    Parameter-agnostic optimization under relaxed smoothness

    Florian Hübler, Junchi Yang, Xiang Li, and Niao He. Parameter-agnostic optimization under relaxed smoothness. InInternational Conference on Artificial Intelligence and Statistics, pages 4861–4869. PMLR, 2024

    2024

  15. [23]

    Non-convex distributionally robust optimization: Non-asymptotic analysis

    Jikai Jin, Bohang Zhang, Haiyang Wang, and Liwei Wang. Non-convex distributionally robust optimization: Non-asymptotic analysis. InAdvances in Neural Information Processing Systems, volume 34, pages 2771–2782, 2021

    2021

  16. [24]

    Linear convergence of gradient and proximal- gradient methods under the polyak-łojasiewicz condition

    Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal- gradient methods under the polyak-łojasiewicz condition. InMachine Learning and Knowledge Discovery in Databases: European Conference, ECML-PKDD-2016, pages 795–811, 2016

    2016

  17. [25]

    Adam: A method for stochastic optimization

    Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InInternational Conference on Learning Representations (ICLR), 2015

    2015

  18. [26]

    Segment anything.arXiv preprint arXiv:2304.02643, 2023

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything.arXiv preprint arXiv:2304.02643, 2023

    2023 arXiv

  19. [27]

    Imagenet classification with deep convolutional neural networks.Communications of the ACM, 60(6):84–90, 2017

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks.Communications of the ACM, 60(6):84–90, 2017

    2017

  20. [28]

    Noise is not the main factor behind the gap between SGD and Adam on transformers, but sign descent might be.arXiv preprint arXiv:2304.13960, 2023

    Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise is not the main factor behind the gap between SGD and Adam on transformers, but sign descent might be.arXiv preprint arXiv:2304.13960, 2023

    2023 arXiv

  21. [29]

    Heavy- tailed class imbalance and why Adam outperforms gradient descent on language models.arXiv preprint arXiv:2402.19449, 2024

    Frederik Kunstner, Robin Yadav, Alan Milligan, Mark Schmidt, and Alberto Bietti. Heavy- tailed class imbalance and why Adam outperforms gradient descent on language models.arXiv preprint arXiv:2402.19449, 2024

    2024 arXiv

  22. [30]

    Convergence of Adam under relaxed assumptions.Advances in Neural Information Processing Systems, 36, 2023

    Haochuan Li, Alexander Rakhlin, and Ali Jadbabaie. Convergence of Adam under relaxed assumptions.Advances in Neural Information Processing Systems, 36, 2023

    2023

  23. [31]

    An improved analysis of stochastic gradient descent with momentum.Advances in Neural Information Processing Systems, 33:18261–18271, 2020

    Yanli Liu, Yuan Gao, and Wotao Yin. An improved analysis of stochastic gradient descent with momentum.Advances in Neural Information Processing Systems, 33:18261–18271, 2020

    2020

  24. [32]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976–11986, 2022

    2022

  25. [33]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  26. [34]

    On the convergence of Adam and beyond

    Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of Adam and beyond. In International Conference on Learning Representations, 2018

    2018

  27. [35]

    A direct adaptive method for faster backpropagation learning: The rprop algorithm

    Martin Riedmiller and Heinrich Braun. A direct adaptive method for faster backpropagation learning: The rprop algorithm. InIEEE international conference on neural networks, pages 586–591. IEEE, 1993

    1993

  28. [36]

    1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs

    Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs. InConference of the International Speech Communication Association, volume 2014, pages 1058–1062. Singapore, 2014

    2014

  29. [37]

    RMSProp converges with proper hyperparameter

    Naichen Shi, Dawei Li, Mingyi Hong, and Ruoyu Sun. RMSProp converges with proper hyperparameter. InInternational Conference on Learning Representation, 2021. 11

    2021

  30. [38]

    Scalable distributed DNN training using commodity GPU cloud computing

    Nikko Ström. Scalable distributed DNN training using commodity GPU cloud computing. In Conference of the International Speech Communication Association, 2015

    2015

  31. [39]

    Momentum ensures convergence of signSGD under weaker assumptions

    Tao Sun, Qingsong Wang, Dongsheng Li, and Bao Wang. Momentum ensures convergence of signSGD under weaker assumptions. InInternational Conference on Machine Learning, pages 33077–33099, 2023

    2023

  32. [40]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  33. [41]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  34. [42]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  35. [43]

    Convergence of AdaGrad for non-convex objectives: Simple proofs and relaxed assumptions

    Bohan Wang, Huishuai Zhang, Zhiming Ma, and Wei Chen. Convergence of AdaGrad for non-convex objectives: Simple proofs and relaxed assumptions. InConference on Learning Theory, pages 161–190. PMLR, 2023

    2023

  36. [44]

    Convergence of AdaGrad for non-convex objectives: Simple proofs and relaxed assumptions

    Bohan Wang, Huishuai Zhang, Zhiming Ma, and Wei Chen. Convergence of AdaGrad for non-convex objectives: Simple proofs and relaxed assumptions. InThe Thirty Sixth Annual Conference on Learning Theory, pages 161–190. PMLR, 2023

    2023

  37. [45]

    On the convergence of Adam under non-uniform smoothness: Separability from sgdm and beyond

    Bohan Wang, Huishuai Zhang, Qi Meng, Ruoyu Sun, Zhi-Ming Ma, and Wei Chen. On the convergence of Adam under non-uniform smoothness: Separability from sgdm and beyond. arXiv preprint arXiv:2403.15146, 2024

    2024 arXiv

  38. [46]

    Provable adaptivity of Adam under non-uniform smoothness

    Bohan Wang, Yushun Zhang, Huishuai Zhang, Qi Meng, Ruoyu Sun, Zhi-Ming Ma, Tie-Yan Liu, Zhi-Quan Luo, and Wei Chen. Provable adaptivity of Adam under non-uniform smoothness. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2960–2969, 2023

    2023

  39. [47]

    Convnext v2: Co-designing and scaling convnets with masked autoencoders

    Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16133–16142, 2023

    2023

  40. [48]

    Tensor programs V: Tuning large neural networks via zero-shot hyperparameter transfer.arXiv preprint arXiv:2203.03466, 2022

    Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs V: Tuning large neural networks via zero-shot hyperparameter transfer.arXiv preprint arXiv:2203.03466, 2022

    2022 arXiv

  41. [49]

    Improved analysis of clipping algorithms for non-convex optimization

    Bohang Zhang, Jikai Jin, Cong Fang, and Liwei Wang. Improved analysis of clipping algorithms for non-convex optimization. InAdvances in Neural Information Processing Systems, volume 33, pages 15511–15521, 2020

    2020

  42. [50]

    Why gradient clipping accelerates training: A theoretical justification for adaptivity

    Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. InInternational Conference on Learning Representations, 2019

    2019

  43. [51]

    Adam can converge without any modification on update rules.Advances in neural information processing systems, 35:28386–28399, 2022

    Yushun Zhang, Congliang Chen, Naichen Shi, Ruoyu Sun, and Zhi-Quan Luo. Adam can converge without any modification on update rules.Advances in neural information processing systems, 35:28386–28399, 2022. 12 Appendix Limitations.While the proposed novel route for the convergenc...

    2022

  44. [52]

    Recently, [20] provably demonstrate the convergence rate of vanilla Adam in high probability perspective, but it only works with the stronger coordinate-wise affine variance

    respectively proved random-shuffled AMSProp and Adam will converge to the neighbourhood of stationary points with the rate O poly(lnE) E1/4 +σ 0 where E is the number of epoches rather than iterations under the affine growth condition that is equivalent to the affine variance ...

  45. [53]

    [49] posits that it is also equivalent to an affine form of the gradient norm for the first-order differentiable function

    first introduced the unciform (L0, L1)-smooth condition to theoretically explain why Clipped- SGD converges faster than vanilla SGD, and they also empirically verified that local smoothness indeed varies with the norm of gradients during DNN training. [49] posits that it is al...

  46. [54]

    [30] further extended the linear (L0, L1)-smooth to the generalized polynomial version, and proved that Adam will converged to O poly(lnT) T 1/4 with the weaker assumption

    theoretically analyzing random-shuffled Adam under this condition, but its convergence rate is provable O poly(lnE) E1/4 where E is the number of epoch, just like [51]. [30] further extended the linear (L0, L1)-smooth to the generalized polynomial version, and proved that Adam...

  47. [55]

    tX k=1 βt−k 1 (gk − ∇F(xk)) 2 # | {z } T2 + 1 T T−1X t=0 E

    Hence, whenT≥8, we have 1 T PT−1 t=0 ∥∇F(x t)∥2 2 1 T PT−1 t=0 ∥∇F(x t)∥2 2 ≤ O 2(1−α) 2 1−2α .(26) 14 B.2 Useful Lemmas Lemma B.1Under Assumption B.3, for anyx,y∈R d, the function obeys F(y)≤F(x) +⟨∇F(x),y−x⟩+ L0 +L 1∥∇F(x)∥ q 2 2 ∥y−x∥ 2 2.(27) Proof.For anyx,y∈R d, we have ...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.