Pith. sign in

REVIEW 2 major objections 4 minor 3 cited by

SGD with Adaptive Preconditioning: Unified Analysis and Momentum Acceleration

T0 review · 2 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single convergence proof covers the main AdaGrad-type optimizers and shows that adding Nesterov momentum provably accelerates the diagonal ones.

desk verdict Solid unified framework for non-accelerated AdaGrad-type methods, but the accelerated AdaGrad/DASGO theorem is not proven as written due to a broken projection argument. read the letter →

arxiv 2506.23803 v1 pith:Y5NNMHSP submitted 2025-06-30 cs.LG math.OC

Dmitry Kovalev This is my paper
classification cs.LGmath.OC MSC 90C2590C1568W40
keywords adaptivepreconditioningAdaGradstochasticgradientdescentmatrixHoldersmoothnessNesterovmomentumunifiedconvergenceanalysisDASGOanisotropic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to show that many adaptive stochastic gradient methods, including AdaGrad, AdaGrad-Norm, and the matrix-preconditioned ASGO/One-sided Shampoo, are instances of one generic algorithm and can be analyzed with one convergence proof. The paper claims that under matrix or anisotropic Holder smoothness and a relaxed bounded-variance noise condition, this unified proof recovers the best known rates for those methods and supplies the first convergence guarantees for DASGO. It then claims that when the curvature and noise operators commute with the preconditioners, which holds automatically for diagonal preconditioning, Nesterov momentum accelerates AdaGrad and DASGO beyond their non-accelerated rates. If correct, this is the first theoretical justification that diagonal preconditioning and momentum can work together, which the author argues may explain the practical efficiency of Adam.

What carries the argument

The load-bearing object is the preconditioning subspace $\mathcal{H} \subset \mathcal{S}$ of self-adjoint operators, together with the potential $\phi(h) = \delta h + \eta^2/h$. The preconditioner is the explicit operator $H_k = \eta(\delta I + \mathrm{proj}_{\mathcal{H}}(S_k))^{-1/2}$, and the FTL-BTL lemma converts the cumulative gradient sum into a single inequality that drives the regret bound. For the accelerated version, the commutativity assumption on the smoothness operator $L$ and noise operator $\Sigma$ is what lets the square $H_k^2$ be characterized as the solution of another optimization problem, producing the logarithmic bound that yields the improved rate.

What would settle it

On a sparse anisotropic convex quadratic satisfying the diagonal assumptions, compare $\log \mathbb{E}[f(x_K)-f(x^*)]$ against $\log K$ for AdaGrad with and without momentum; the paper predicts exponents $(1+3\nu)/2$ and $(1+\nu)/2$ respectively, so similar empirical slopes would refute the acceleration claim.

Watch

Extended reading notes

Core claim

The central discovery is a single convergence theory for the update $x_{k+1} = \arg\min_x \langle g_k, x\rangle + \tfrac12 \|x - x_k\|^2_{H_k^{-1}}$, where the preconditioner $H_k$ is chosen by a Follow-the-Leader optimization over a subspace of self-adjoint operators with potential $\phi(h) = \delta h + \eta^2/h$, giving the explicit form $H_k = \eta(\delta I + \mathrm{proj}_{\mathcal{H}}(S_k))^{-1/2}$. Under convexity and $(\|L\|_{\mathrm{tr}}^{(1-\nu)/2}, \nu)$-Holder smoothness with respect to $\|\cdot\|_L$, plus a noise bound measured through an operator $\Sigma$, the paper derives a regret bound that becomes the state-of-the-art rate for AdaGrad-Norm, AdaGrad, and ASGO/One-sided Shampoo when specialized, and a first rate for DASGO. When $L$ and $\Sigma$ commute with every allowed preconditioner, the squared preconditioner $H_k^2$ solves a second optimization problem, which lets the same machinery produce a Nesterov-accelerated bound: $\mathbb{E}[f(x_{K+1}) - f(x^*)] \le C_K \|L\|_{\mathrm{tr}} R^{1+\nu}/(K+2)^{(1+3\nu)/2} + C_K \|\Sigma\|_{\mathrm{tr}} R/\sqrt{K+2} + 4\sqrt{\delta} R \dim(\mathcal{X})/(K+2)^2$, with $C_K = O(1 + \ln K + \cdots)$. The paper states this as the first theoretical demonstration that AdaGrad-type diagonal preconditioning and momentum can combine productively.

Load-bearing premise

The accelerated result stands on the assumption that the smoothness operator $L$ and the noise operator $\Sigma$ commute with every admissible preconditioner; the momentum proof collapses without this commutativity, though it holds automatically for the diagonal preconditioners of AdaGrad and DASGO.

Editorial extensions

If this is right

  • AdaGrad and DASGO with Nesterov momentum converge at rate $\tilde O(\|l\|_1 \|X^*\|^{1+\nu}_{2\to\infty}/K^{(1+3\nu)/2} + \|\sigma\|_1 \|X^*\|_{2\to\infty}/\sqrt{K})$, improving on their non-accelerated $O(1/K^{(1+\nu)/2})$ rate.
  • DASGO, previously proposed without guarantees, now has a convergence theorem, and the paper identifies it as the diagonal counterpart of Scion.
  • The unified proof covers the non-smooth case ($\nu=0$) and the full Holder range $0<\nu<1$, which prior structured-preconditioner analyses did not cover.
  • In the smooth diagonal case, the accelerated bound beats the best known scalar-stepsize accelerated AdaGrad results when the curvature and noise are sparse and the optimum is dense.
  • The momentum plus diagonal preconditioning result, combined with existing EMA analyses, offers a theoretical route to explaining Adam's practical success.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If commutativity is the only obstruction, the acceleration theorem should extend to any commutative subalgebra of preconditioners, such as block-diagonal or Kronecker-structured ones, so the practical reach may be wider than the examples listed.
  • The DASGO-Scion connection suggests a direct empirical test: on row-sparse convex losses, DASGO with momentum should outperform both Scion and scalar AdaGrad, matching the predicted rate advantage.
  • The paper's reliance on projection to enforce the almost-sure iterate bound hints that a parameter-free, tuning-free diagonal preconditioned algorithm is the next natural target, since current parameter-free results are limited to scalar step sizes.
  • A practical design consequence the author does not state: mixing the exponential moving average with the accelerated diagonal analysis likely yields the first end-to-end theory for Adam itself, not just for AdaGrad.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes a unified convergence analysis for AdaGrad-type preconditioned stochastic gradient methods under matrix/anisotropic Hölder smoothness and a relaxed bounded-variance noise assumption. The non-accelerated framework (Algorithm 1) is shown to cover AdaGrad-Norm, AdaGrad, ASGO/One-sided Shampoo, and DASGO, and yields a single proof that recovers state-of-the-art guarantees in the smooth case while also covering Hölder and non-smooth regimes. The paper then introduces an accelerated variant (Algorithm 2) using Nesterov momentum, whose analysis relies on a commutativity assumption (Assumption 4) between the smoothness/noise operators and the preconditioning space. The claimed result is that diagonal-preconditioned methods such as AdaGrad and DASGO can simultaneously benefit from preconditioning and momentum, with rates O~(1/K^{(1+3ν)/2}) in the smoothness term. The paper also establishes a connection between DASGO and the Scion algorithm. Conditional Theorem 2 and the projection-based Theorem 4 are presented as the main accelerated guarantees.

Significance. If correct, the paper's contributions are significant. The unified non-accelerated analysis is elegant: it derives the preconditioner formula from a single FTL-BTL argument, relaxes the noise assumption used in prior work, and covers a whole Hölder-smoothness spectrum, while providing the first convergence guarantees for DASGO and clarifying its relation to Scion. These parts of the paper are well-structured and appear sound. The accelerated part, if established, would be the first theoretical justification that diagonal AdaGrad-type preconditioning and Nesterov momentum can be combined productively. However, the central accelerated claim is currently not proven because the appendix's projection fix does not restore the required iterate bound for Algorithm 4. Since this gap concerns the paper's headline contribution, the manuscript is not yet acceptable in its present form, although the non-accelerated framework is valuable and likely repairable.

major comments (2)
  1. [Section 4 and Appendix B (Theorem 2, Theorem 4, Algorithm 4)] The projection fix in Appendix B does not justify the almost-sure bound R(x_k - x*) ≤ R (eq. (16)) for the accelerated iterates. In Algorithm 4, line 6 computes both the projected point x_{k+1} and the unprojected point x_{k+1/2} using eq. (31); line 7 then sets x_{k+1} = alpha_k x_{k+1/2} + (1 - alpha_k)x_k, so the final momentum iterate is a convex combination involving the unprojected point and need not lie in Q_R. The proof of Lemma 10 relies on eq. (16) at eq. (55), where the terms 1/2 sum_k <R_k, H_k^{-1} - H_{k-1}^{-1}> are bounded by 1/2 R^2 <I, H_K^{-1}>; that step is unjustified for Algorithm 4. If one instead replaced x_{k+1/2} by the projected x_{k+1} in line 7, the iterates would stay in Q_R, but then the descent inequality in eq. (34) is stated for f_k(x_{k+1/2}), and convexity does not imply f_k(proj(y)) ≤ f_k(y) for the non-Euclidean projection in eq. (31). The appendix's claim that only 'trivial changes' to Lemma 7 are needed is therefore unsupported. Consequently, Theorem 4, and hence the advertised unconditional accelerated rates for AdaGrad and DASGO in Section 4.2 (eq. (29)), are not established; Theorem 2 remains a conditional statement under the a priori bound eq. (16). This is a load-bearing issue because the acceleration result is the paper's main novelty.
  2. [Section 4.1 (Assumption 4)] The accelerated framework is substantially narrower than the non-accelerated one. Assumption 4 requires the smoothness operator L and noise operator Sigma to commute with every admissible preconditioner H, and this holds automatically only when H consists of diagonal (or scalar) operators. Lemma 8, Lemma 9, Lemma 11, and Lemma 12 all depend on this commutativity. Thus the unified accelerated analysis does not apply to ASGO/One-sided Shampoo or general matrix preconditioning, even though the title and abstract emphasize a unified treatment of preconditioning. The paper does state this limitation in Section 4.1, but the scope should be made explicit in the abstract and introduction, because the accelerated claim is effectively restricted to diagonal preconditioning.
minor comments (4)
  1. [Algorithm 4, lines 6-7] The reuse of the symbol x_{k+1} on the left-hand side of line 7, after line 6 has already computed a projected x_{k+1} via eq. (31), is confusing. Renaming the projected point (e.g., x_{k+1}^{proj}) would make the overwriting explicit and would also underscore the issue raised in the major comment.
  2. [Assumption 3 and proof of Lemma 6] The proof of Lemma 6 uses the fact that the gradient estimator at iteration k is independent of the current iterate x_k when taking the expectation (step (b) of the displayed calculation). This independence is not explicitly stated in Assumption 3; adding it to the assumption or to a preliminary remark would make the proof self-contained.
  3. [Section 4.2, eqs. (29)-(30)] The comparison between the claimed DASGO/AdaGrad rate in eq. (29) and the state-of-the-art scalar-stepsize rate in eq. (30) uses different norms (∥l∥_1 vs. ∥l∥_∞, ∥X*∥_{2→∞} vs. ∥X*∥). A brief remark on how the constants are matched or where the comparison is informal would help the reader assess the claimed improvement.
  4. [Throughout] There are several minor typographical and notational inconsistencies, such as inconsistent capitalization of 'AdaGrad' versus 'Adagrad' and the somewhat cryptic phrasing of the 'no extra assumptions' statement in Section 1.4. A careful proofreading pass is recommended.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the convergence rates are derived from the stated smoothness, variance, and commutation assumptions, and the self-citation to Kovalev and Borodich (2024) is not load-bearing because the needed momentum lemma is re-proven in the appendix.

full rationale

The derivation chain is self-contained. The preconditioner form H_k = eta(δI + proj_H(S_k))^{-1/2} (Lemma 2) is obtained by first-order optimality from the potential phi(h) = δh + eta^2/h, and eta = R is chosen a priori as the standard AdaGrad universal parameter, not fitted to the target rates. The regret bounds in Lemmas 5 and 6 combine FTL-BTL, Hölder smoothness (Assumption 2), and the variance bound (Assumption 3); Theorem 1 follows by algebra and Jensen's inequality, with the iterate bound in eq. (16) stated explicitly as a condition. The accelerated analysis defines f_k via alpha_k and proves Lemma 7 directly using convexity and the Nesterov combination; the statement that this interpretation is due to Kovalev and Borodich (2024) is not load-bearing because the needed inequality is re-derived in Appendix E.1. Lemma 9 relies on Lemmas 8, 11 and 12, all proven from Assumption 4 and Property A1.2; no step imports the paper's conclusions from the cited prior work. The 'connection' between DASGO and Scion is a stated update-rule equivalence (choosing S_k = g_k⟨g_k,·⟩ in eq. (5)), not a derived quantitative prediction that reduces to its own input. The Appendix B projection discussion raises a possible correctness gap for Algorithm 4: line 7 averages the unprojected x_{k+1/2}, so eq. (16) is not enforced for the momentum iterates, and the claimed 'trivial changes to Lemma 7' are not demonstrated. That is a validity concern, not a circularity, because no bound is assumed in terms of the bound being proved and no equation is equivalent to its own conclusion by construction. Therefore the paper does not exhibit self-definitional, fitted-input, or self-citation-based circularity in its central claims.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on three modeling assumptions (Assumptions 1-3) plus Assumption 4 for acceleration. No constants are fitted to the target rates: eta and delta are algorithm inputs. The framework introduces no new physical or algorithmic entities; the time-varying function f_k is a proof technique based on the author's prior work.

free parameters (2)
  • eta (preconditioner scaling) = R (or 2R in accelerated version)
    Standard AdaGrad-universality choice proportional to the iterate-distance bound R; not fitted to data, but a hand-set parameter.
  • delta (potential regularization) = small positive constant
    Regularizer in potential phi(h)=delta*h+eta^2/h; appears in log terms and the negligible delta-term of the rates.
assumptions (5)
  • domain assumption Assumption 1: projection onto H is order-preserving and H is closed under operator functions
    Defines admissible preconditioner sets; needed to compute H_k in Lemma 2 and for trace identities.
  • domain assumption Assumption 2: f is convex and (||L||_tr^{(1-nu)/2}, nu)-Holder smooth w.r.t. ||.||_L with L in H
    Central smoothness model tying the analysis to the operator norm; includes non-smooth nu=0.
  • domain assumption Assumption 3: unbiased stochastic gradients with bounded variance E||n||^2_{Sigma^{-1}} <= ||Sigma||_tr and Sigma in H
    Noise model; more general than prior works' E[n n^T] <= Sigma^2.
  • domain assumption Assumption 4: L and Sigma commute with all H in H
    Key to accelerated Lemmas 8-12; automatically true for diagonal H, hence for AdaGrad/DASGO.
  • standard math Standard operator inequalities: FTL-BTL, Loewner-Heinz theorem, concavity of trace sqrt, Jensen, Young, Holder, Cauchy-Schwarz, Weierstrass approximation
    Used throughout the proofs; not derived in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SGD with Adaptive Preconditioning: Unified Analysis and Momentum Acceleration." pith.science (2026). https://pith.science/paper/Y5NNMHSP

@misc{pith2026250623803,
  author       = {Pith},
  title        = {Pith review of: SGD with Adaptive Preconditioning: Unified Analysis and Momentum Acceleration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y5NNMHSP}},
  note         = {Machine review of arXiv:2506.23803}
}
read the original abstract

In this paper, we revisit stochastic gradient descent (SGD) with AdaGrad-type preconditioning. Our contributions are twofold. First, we develop a unified convergence analysis of SGD with adaptive preconditioning under anisotropic or matrix smoothness and noise assumptions. This allows us to recover state-of-the-art convergence results for several popular adaptive gradient methods, including AdaGrad-Norm, AdaGrad, and ASGO/One-sided Shampoo. In addition, we establish the fundamental connection between two recently proposed algorithms, Scion and DASGO, and provide the first theoretical guarantees for the latter. Second, we show that the convergence of methods like AdaGrad and DASGO can be provably accelerated beyond the best-known rates using Nesterov momentum. Consequently, we obtain the first theoretical justification that AdaGrad-type algorithms can simultaneously benefit from both diagonal preconditioning and momentum, which may provide an ultimate explanation for the practical efficiency of Adam.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Nesterov Finds GRAAL: Optimal and Adaptive Gradient Method for Convex Optimization

    math.OC 2025-07 conditional novelty 7.0 of 10

    Accelerated GRAAL is the first adaptive first-order method that proves near-optimal accelerated complexity for convex L-smooth and (L0,L1)-smooth functions with geometric stepsize growth.

  2. Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Rates

    math.OC 2025-11 conditional novelty 5.0 of 10

    Non-Euclidean SGD variants (SignSGD, Muon) provably match adaptive optimizers' convergence rates under structured smoothness and noise assumptions.

  3. AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updates

    cs.LG 2025-09 conditional novelty 5.0 of 10

    The paper proposes AdaGO, a Muon variant with a scalar AdaGrad-Norm step size, and proves optimal nonconvex convergence rates while reporting empirical gains on regression and CIFAR-10.

Reference graph

Works this paper leans on

51 extracted references · 11 canonical work pages · cited by 3 Pith papers

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    An, K., Liu, Y., Pan, R., Ma, S., Goldfarb, D., and Zhang, T. (2025). Asgo: Adaptive structured gradient optimization. arXiv preprint arXiv:2503.20762

    arXiv 2025

  3. [3]

    M., Hauth, A., Millican, K., et al

    Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023). Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    arXiv 2023

  4. [4]

    and Newhouse, L

    Bernstein, J. and Newhouse, L. (2024). Old optimizer, new norm: An anthology. arXiv preprint arXiv:2409.20325

    arXiv 2024

  5. [5]

    Carlen, E. (2010). Trace inequalities and quantum entropy: an introductory course. Entropy and the quantum , 529(73-140):146

    work page 2010

  6. [6]

    C., Hinder, O., and Sidford, A

    Carmon, Y., Duchi, J. C., Hinder, O., and Sidford, A. (2020). Lower bounds for finding stationary points i. Mathematical Programming , 184(1):71--120

    work page 2020

  7. [7]

    Cutkosky, A. (2019). Anytime online-to-batch, optimism and acceleration. In International conference on machine learning , pages 1446--1454. PMLR

    2019

  8. [8]

    and Orabona, F

    Cutkosky, A. and Orabona, F. (2018). Black-box reductions for parameter-free online learning in banach spaces. In Conference On Learning Theory , pages 1493--1529. PMLR

    2018

Show all 51 references
  1. [9]

    and Mishchenko, K

    Defazio, A. and Mishchenko, K. (2023). Learning-rate-free learning by d-adaptation. In International Conference on Machine Learning , pages 7449--7479. PMLR

    2023

  2. [10]

    D \'e fossez, A., Bottou, L., Bach, F., and Usunier, N. (2020). A simple convergence proof of adam and adagrad. arXiv preprint arXiv:2003.02395

    2020 arXiv

  3. [11]

    Duchi, J., Hazan, E., and Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research , 12(7)

    2011

  4. [12]

    Fatkhullin, I., Etesami, J., He, N., and Kiyavash, N. (2022). Sharp analysis of stochastic optimization under global kurdyka-lojasiewicz inequality. Advances in Neural Information Processing Systems , 35:15836--15848

    2022

  5. [13]

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  6. [14]

    Gupta, V., Koren, T., and Singer, Y. (2017). A unified approach to adaptive regularization in online and stochastic optimization. arXiv preprint arXiv:1706.06569

    2017 arXiv

  7. [15]

    Gupta, V., Koren, T., and Singer, Y. (2018). Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning , pages 1842--1850. PMLR

    2018

  8. [16]

    Hinder, O., Sidford, A., and Sohoni, N. (2020). Near-optimal methods for minimizing star-convex functions and beyond. In Conference on learning theory , pages 1894--1938. PMLR

    2020

  9. [17]

    Ivgi, M., Hinder, O., and Carmon, Y. (2023). Dog is sgd’s best friend: A parameter-free dynamic step size schedule. In International Conference on Machine Learning , pages 14465--14499. PMLR

    2023

  10. [18]

    Jiang, R., Maladkar, D., and Mokhtari, A. (2024). Convergence analysis of adaptive gradient methods under refined smoothness and noise assumptions. arXiv preprint arXiv:2406.04592

    2024 arXiv

  11. [19]

    Jordan, K., Jin, Y., Boza, V., You, J., Cesista, F., Newhouse, L., and Bernstein, J. (2024). Muon: An optimizer for hidden layers in neural networks. Cited on , page 10

    2024

  12. [20]

    and Vempala, S

    Kalai, A. and Vempala, S. (2005). Efficient algorithms for online decision problems. Journal of Computer and System Sciences , 71(3):291--307

    2005

  13. [21]

    Y., Bach, F., and Cevher, V

    Kavis, A., Levy, K. Y., Bach, F., and Cevher, V. (2019). Unixgrad: A universal, adaptive algorithm with optimal guarantees for constrained optimization. Advances in neural information processing systems , 32

    2019

  14. [22]

    Khaled, A., Mishchenko, K., and Jin, C. (2023). Dowg unleashed: An efficient universal parameter-free gradient descent method. Advances in Neural Information Processing Systems , 36:6748--6769

    2023

  15. [23]

    Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    2014 arXiv

  16. [24]

    Kleinberg, B., Li, Y., and Yuan, Y. (2018). An alternative view: When does sgd escape local minima? In International conference on machine learning , pages 2698--2707. PMLR

    2018

  17. [25]

    Kovalev, D. (2025). Understanding gradient orthogonalization for deep learning via non-euclidean trust-region optimization. arXiv preprint arXiv:2503.12645

    2025 arXiv

  18. [26]

    and Borodich, E

    Kovalev, D. and Borodich, E. (2024). On linear convergence in smooth convex-concave bilinearly-coupled saddle-point optimization: Lower bounds and optimal algorithms. arXiv preprint arXiv:2411.14601

    2024 arXiv

  19. [27]

    Kreisler, I., Ivgi, M., Hinder, O., and Carmon, Y. (2024). Accelerated parameter-free stochastic optimization. In The Thirty Seventh Annual Conference on Learning Theory , pages 3257--3324. PMLR

    2024

  20. [28]

    LeCun, Y., Bengio, Y., and Hinton, G. (2015). Deep learning. nature , 521(7553):436--444

    2015

  21. [29]

    Y., Yurtsever, A., and Cevher, V

    Levy, K. Y., Yurtsever, A., and Cevher, V. (2018). Online adaptive methods, universality and acceleration. Advances in neural information processing systems , 31

    2018

  22. [30]

    and Orabona, F

    Li, X. and Orabona, F. (2019). On the convergence of stochastic gradient descent with adaptive stepsizes. In The 22nd international conference on artificial intelligence and statistics , pages 983--992. PMLR

    2019

  23. [31]

    Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. (2024a). Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  24. [32]

    Liu, Y., Pan, R., and Zhang, T. (2024b). Adagrad under anisotropic smoothness. arXiv preprint arXiv:2406.15244

    2024 arXiv

  25. [33]

    and Hutter, F

    Loshchilov, I. and Hutter, F. (2017). Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101

    2017 arXiv

  26. [34]

    McMahan, H. B. and Streeter, M. (2010). Adaptive bound optimization for online convex optimization. arXiv preprint arXiv:1002.4908

    2010 arXiv

  27. [35]

    and Defazio, A

    Mishchenko, K. and Defazio, A. (2023). Prodigy: An expeditiously adaptive parameter-free learner. arXiv preprint arXiv:2306.06101

    2023 arXiv

  28. [36]

    Mishchenko, K., Malinovsky, G., Stich, S., and Richt \'a rik, P. (2022). Proxskip: Yes! local gradient steps provably lead to communication acceleration! finally! In International Conference on Machine Learning , pages 15750--15769. PMLR

    2022

  29. [37]

    Nesterov, Y. (1983). A method for unconstrained convex minimization problem with the rate of convergence O (1/k^2) . Dokl. Akad. Nauk. SSSR , 269(3):543

    1983

  30. [38]

    Nesterov, Y. (2013). Introductory lectures on convex optimization: A basic course , volume 87. Springer Science & Business Media

    2013

  31. [39]

    Nesterov, Y. (2015). Universal gradient methods for convex optimization problems. Mathematical Programming , 152(1):381--404

    2015

  32. [40]

    Orabona, F. (2023). Normalized gradients for all. arXiv preprint arXiv:2308.05621

    2023 arXiv

  33. [41]

    and P \'a l, D

    Orabona, F. and P \'a l, D. (2021). Parameter-free stochastic optimization of variationally coherent functions. arXiv preprint arXiv:2102.00236

    2021 arXiv

  34. [42]

    Pethick, T., Xie, W., Antonakopoulos, K., Zhu, Z., Silveti-Falls, A., and Cevher, V. (2025). Training deep learning models with norm-constrained lmos. arXiv preprint arXiv:2502.07529

    2025 arXiv

  35. [43]

    J., Kale, S., and Kumar, S

    Reddi, S. J., Kale, S., and Kumar, S. (2019). On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237

    2019 arXiv

  36. [44]

    Riabinin, A., Shulgin, E., Gruntkowska, K., and Richt \'a rik, P. (2025). Gluon: Making muon & scion great again!(bridging theory and practice of lmo-based optimizers for llms). arXiv preprint arXiv:2505.13416

    2025 arXiv

  37. [45]

    Rodomanov, A., Jiang, X., and Stich, S. U. (2024). Universality of adagrad stepsizes for stochastic optimization: Inexact oracle, acceleration and variance reduction. Advances in Neural Information Processing Systems , 37:26770--26813

    2024

  38. [46]

    and McMahan, H

    Streeter, M. and McMahan, H. B. (2010). Less regret via online conditioning. arXiv preprint arXiv:1002.4862

    2010 arXiv

  39. [47]

    Tieleman, T. (2012). Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning , 4(2):26

    2012

  40. [48]

    Trifonov, S., Levin, L., Chezhegov, S., and Beznosikov, A. (2025). Incorporating preconditioning into accelerated approaches: Theoretical guarantees and practical improvement. arXiv preprint arXiv:2505.23510

    2025

  41. [49]

    Vyas, N., Morwani, D., Zhao, R., Kwun, M., Shapira, I., Brandfonbrener, D., Janson, L., and Kakade, S. (2024). Soap: Improving and stabilizing shampoo using adam. arXiv preprint arXiv:2409.11321

    2024 arXiv

  42. [50]

    Xie, S., Wang, T., Reddi, S., Kumar, S., and Li, Z. (2025). Structured preconditioners in adaptive optimization: A unified analysis. arXiv preprint arXiv:2503.10537

    2025 arXiv

  43. [51]

    Zhou, Y., Yang, J., Zhang, H., Liang, Y., and Tarokh, V. (2019). Sgd converges to global minimum in deep learning via star-convex path. arXiv preprint arXiv:1901.00451

    2019 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.