Pith. sign in

REVIEW 3 major objections 6 minor 14 cited by

AdaGO shows that a clamped norm-based AdaGrad stepsize can adapt Muon's orthogonalized momentum updates while preserving their direction, with optimal worst-case convergence rates for nonconvex optimization.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

The paper proposes AdaGO, a Muon variant with a scalar AdaGrad-Norm step size, and proves optimal nonconvex convergence rates while reporting empirical gains on regression and CIFAR-10.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A clean, cheap adaptive step size for Muon with solid rates for the exact-orthogonalization variant; the gap to the implemented Newton–Schulz version and the thin experiments are the main caveats. the 3 major comments →

arxiv 2509.02981 v2 pith:6KV3STAR submitted 2025-09-03 cs.LG math.OC

AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updates

Minxin Zhang , Yuxuan Liu , Hayden Schaeffer This is my paper
classification cs.LG math.OC
keywords AdaGrad-NormMuon optimizerorthogonalized momentumadaptive stepsizesnonconvex stochastic optimizationconvergence ratesspectral descentmatrix preconditioning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AdaGO asks whether a Muon-style orthogonalized update direction can be paired with a scalar AdaGrad stepsize without sacrificing worst-case convergence. The paper's answer is yes: with a clamped norm accumulator and a step size of the form max(epsilon, eta times the ratio of current to accumulated gradient norms), AdaGO converges at O(T^{-1/4}) in stochastic nonconvex optimization and O(1/sqrt(T)) in the deterministic full-batch setting, rates that match known lower bounds for first-order methods. The stepsize is scalar, so it rescales the update without rotating it, keeping the direction exactly orthogonalized momentum, which is a spectral descent direction. The change to Muon is essentially one extra scalar accumulator, making the adaptivity cheap. Small-scale experiments on CIFAR-10 and function regression show lower training loss and higher test accuracy than Muon and Adam.

Core claim

The paper's central claim is that AdaGrad's norm-based adaptive stepsize works for orthogonalized momentum updates, provided the accumulated gradient norms are clamped by a constant gamma and the step is also scaled by the current clamped gradient norm. Under standard smoothness and unbiased bounded-variance noise assumptions, the average nuclear norm of the true gradient converges at O(T^{-1/4}) in the stochastic nonconvex setting and at O(1/sqrt(T)) deterministically, both optimal in the worst case. The proof's engine is that the inner product of the momentum matrix with its orthogonalization equals the nuclear norm of the momentum matrix, so the orthogonalized direction acts as a subgradi

What carries the argument

The carrying object is the pair (O_t, alpha_t): O_t = Orth(M_t) is the polar factor of the momentum matrix M_t, the closest matrix with orthonormal rows or columns, and alpha_t = max(epsilon, eta min(||G_t||, gamma)/v_t) with v_t^2 accumulating the clamped squared gradient norms. Because Orth(M_t) lies in the subgradient of the nuclear norm at M_t, the identity <M_t, Orth(M_t)> = ||M_t||_* converts the descent inequality into a bound on the average nuclear norm of the true gradient. The stepsize is a single scalar, so it rescales the orthogonal direction without distorting it, gamma removes the need for uniformly bounded gradients, and the epsilon floor keeps the iterates from stalling.

Load-bearing premise

The convergence proof treats every update direction as the exact orthogonalization of the momentum matrix, but the practical algorithm uses Newton-Schulz iterations to approximate it, and that approximation error never appears in the assumptions or the bounds.

What would settle it

Run AdaGO on an ill-conditioned smooth nonconvex problem using exact SVD orthogonalization versus the Newton-Schulz approximation with identical stepsizes, and compare average norm of the true gradient. If the approximate version does not follow the O(T^{-1/4}) bound or diverges while the exact version converges, the missing orthogonalization error is the cause.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • AdaGO adds exactly one scalar accumulator to Muon, so the per-step cost and memory footprint remain essentially those of Muon.
  • The stochastic O(T^{-1/4}) and deterministic O(1/sqrt(T)) rates match first-order lower bounds, meaning the adaptivity costs nothing in worst-case convergence.
  • The epsilon floor guarantees a minimum step size, so AdaGO cannot freeze near a stationary point and inherits the convergence of small-constant-step Muon.
  • Clamping the accumulated gradient norms with gamma extends AdaGrad-Norm-style analysis to nonconvex settings without requiring uniformly bounded gradients.
  • Because the step size uses min(||G_t||,gamma), updates decay to zero near stationarity, a null-gradient-consistency property the analysis relies on.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the exact orthogonalization assumption were replaced by a bound on the Newton-Schulz approximation error, the proof would likely pay an extra term proportional to that error; measuring that error on ill-conditioned gradients would show when the theory stops matching practice.
  • Since v0 and gamma enter only logarithmically in the bounds, AdaGO should be nearly insensitive to those hyperparameters, a claim worth testing by sweeping gamma over several orders of magnitude.
  • The empirical heuristic epsilon < eta^2, combined with the theory's T-dependent epsilon choices, suggests an annealed epsilon schedule could improve late-stage convergence in practical implementations.
  • Because the step size is a ratio of current to historical gradient norms, AdaGO is a natural candidate for replacing hand-tuned Muon learning-rate schedules in large-scale language-model training, though the paper only demonstrates smaller tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. AdaGO combines Muon's orthogonalized momentum with an AdaGrad-Norm-style scalar stepsize. Algorithm 2 maintains a momentum matrix M_t, accumulates clamped squared gradient norms in a scalar v_t^2, and updates parameters as Θ_t = Θ_{t-1} - max{ε, η min(||G_t||,γ)/v_t} Orth(M_t). The paper proves convergence rates under smoothness and bounded-variance assumptions: O(T^{-1/4}) in the stochastic nonconvex setting (Theorem 3.3), O(1/√T) in the deterministic full-batch setting (Theorem 3.4), and noise-adaptive rates with increasing batch sizes when momentum is disabled (Theorem 3.5). The proofs use a descent lemma, a momentum error recursion, and logarithmic sum bounds. Experiments compare AdaGO with Muon and Adam on a synthetic regression task and CIFAR-10 classification.

Significance. If the rates are correct, the paper makes a useful contribution: it shows that a scalar norm-based AdaGrad stepsize can be attached to an orthogonalized update direction without sacrificing worst-case nonconvex rates, and it does so at negligible computational overhead. The proof structure is standard and mostly self-contained, and the claimed rates match known lower bounds. The significance is tempered, however, by the gap between the exact Orth(M_t) used in the theory and the Newton-Schulz approximation used in practice, and by an empirical section that reports no variance information. The exact-Orth idealized result is plausible and worth publishing after substantial revision; the current manuscript overclaims the practical optimizer whose rates are proved.

major comments (3)
  1. [Section 2 / Algorithm 2, Line 6; Appendix C, Eq. (7)] The proof relies on the exact polar identity <M_t,O_t> = ||M_t||_* to convert the inner product with the gradient into a descent term. Section 1 itself notes that practical Muon implementations replace exact Orth by Newton-Schulz iterations and that theoretical analyses assume exact orthogonalization. For an approximate polar factor O_t, the descent inequality in Eq. (7) acquires an additional nonnegative error term ||M_t||_* - <M_t,O_t> at every iteration, and no assumption or lemma bounds its cumulative sum. Thus Theorems 3.3 and 3.4 are proved only for the idealized Algorithm 2 with exact Orth, not for the 'minimal modification to Muon' implementation described in the paper and used in the experiments. This is load-bearing because the practical optimizer being advocated is not the one whose optimal rates are established. The authors should either provide an approximation-error analysi
  2. [Appendix C, Eq. (10)-(11)] With M_0=0 and M_1=(1-µ)G_1, the initial momentum error is E_1 = M_1 - ∇L(Θ_0) = (1-µ)\tilde E_1 - µ∇L(Θ_0). The proof uses E||E_1||_* ≤ κ√r/√b and E||E_1||_F^2 ≤ κ^2/b in Eq. (10) and Eq. (11), respectively. These bounds are false unless ∇L(Θ_0)=0 or an additional assumption is imposed. The omitted µ-scaled initial-gradient terms appear to enter the final bound only at lower order (roughly O(T^{-1/2}) and O(T^{-(3/8+q)}) after normalization), so the claimed rate may survive a repair, but the proof as written is not rigorous. Please correct the initial-condition handling or state an extra assumption.
  3. [Section 4, Figures 1-2 and Table 1] The empirical claim that AdaGO 'consistently outperforms' Muon and Adam is based on single training curves with no seeds, standard deviations, or repeated-run statistics reported. With only one CIFAR-10 run and one synthetic regression run, the observed gaps could plausibly be within run-to-run variation. The authors should report multiple seeds with error bars or confidence bands, or soften the empirical conclusion. Additionally, since AdaMuon, COSMOS, ASGO and PolarGrad are discussed as related adaptive variants of Muon, a comparison with at least one such method would strengthen the practical claim that AdaGO is preferable among adaptive orthogonal-update optimizers.
minor comments (6)
  1. [Appendix D] In the displayed descent inequality, 'max{ϵ, α_t}' appears instead of 'max{ϵ, ηα_t}' in several places. The missing η is carried implicitly in later bounds, but the notation should be consistent.
  2. [Appendix A] The text refers to 'Figure 3a' and 'Figure 3b' for the motivating GD-vs-OGD experiment, but these figures are not included in the manuscript text I reviewed. Please ensure all cited figures are present.
  3. [Section 2] The statement that scaling by the clamped gradient norm ensures the per-iteration update 'decays to zero' is only true when ε is scheduled to vanish with T. In Algorithm 2 as written with constant ε>0, the update norm is bounded below by ε. Please clarify that null gradient consistency holds for the theoretical schedule, not for a fixed ε implementation.
  4. [Theorem 3.5] The batch size b_t = √t should be read as ceil(√t) to be an integer; please state this explicitly.
  5. [Appendix C, Eq. (13)] There is a typographical oddity: a factor (ln(...))^{1/4} appears inside a square root in the last term of the bound. Please verify the expression and simplify if possible.
  6. [Throughout] Minor typos and formatting issues: 'demostate' in Section 1.2, 'CIF AR-10' spacing in headings, and inconsistent spacing in Algorithm 2. These do not affect the technical content.

Circularity Check

0 steps flagged

No circularity: the convergence proof is self-contained given explicit hyperparameter schedules; the Newton-Schulz approximation gap is a validity concern, not a circularity.

full rationale

The paper's derivation chain does not reduce to its own inputs. Theorem 3.3 and 3.4 are proven from Assumptions 3.1–3.2 using an explicit polar/SVD property of exact orthogonalization, namely that Orth(M_t) = UV^T implies <M_t, Orth(M_t)> = ||M_t||_*, which is a mathematical identity rather than an assumed conclusion. The hyperparameters epsilon, eta, mu, and batch sizes are set as explicit functions of T in the theorems, not fitted to any dataset and then renamed as predictions. The momentum error analysis in Appendix C follows the recursion in Cutkosky and Mehta [8], an external result, and the AdaGrad summation bound is adapted from Ward et al. [34]'s Lemma 3.2, also external. The 'optimal rate' claims cite Arjevani et al. [2] and Carmon et al. [6], external lower bounds rather than self-citations, and these citations are not used to derive the algorithm's update rule. No load-bearing premise is justified solely by a citation to the present authors' prior work; indeed, the paper contains no self-citations. The unresolved gap between exact Orth(M_t) assumed in Algorithm 2 Line 6 and the Newton-Schulz approximation mentioned in Section 1 is a correctness/robustness limitation of the theory relative to the implemented version, not a circularity, since the theorem explicitly analyzes the exact-orthogonalization algorithm and makes no claim bounding the approximation error. Overall, the central convergence claims have independent mathematical content and are not forced by definition or by self-citation.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claim rests on two standard optimization assumptions, plus the exact-orthogonalization idealization and a known lower-bound benchmark. The only fitted numbers are ordinary hyperparameters (eta, epsilon, momentum); no explanatory entity is introduced.

free parameters (5)
  • eta (base learning rate) = regression 0.5, classification 0.05; theory T^{-(3/8+q)} or T^{-q}
    Tuned per optimizer and task; controls overall step scale and appears in every convergence bound.
  • epsilon (stepsize floor) = regression 5e-3, classification 5e-4; theory T^{-3/4} or T^{-1/2}
    Lower bound on the adaptive step; prevents vanishing updates and is tuned in the experiments despite being set by T in the analysis.
  • gamma (gradient norm clipping constant) = not reported; claimed robust over a wide range
    Clips gradient norms in the accumulator v_t; appears only logarithmically in theory but its practical value is not given, leaving a reproducibility gap.
  • mu (momentum coefficient) = 0.95 in experiments; 1 - T^{-1/2} in stochastic theory
    Momentum for the orthogonalized update; the theorem requires a specific schedule while the experiments pick a constant.
  • v0 (initial accumulator) = unspecified >0
    Initializes the squared-norm accumulator for numerical stability; theory shows only logarithmic dependence, but the exact value is not disclosed.
axioms (5)
  • domain assumption L is L-smooth in the dual spectral norm: ||grad L(Theta) - grad L(Theta')||_* <= L ||Theta - Theta'||_2 (Assumption 3.1)
    Underlies the descent lemma and the smoothness penalty in all three convergence proofs.
  • domain assumption Stochastic gradients are unbiased with E||G_t - grad L(Theta_{t-1})||_F^2 <= kappa^2/b_t (Assumption 3.2)
    Used to control the noise in the momentum error recursion, which is the main source of the stochastic rate.
  • domain assumption Exact orthogonalization Orth(M_t) = U V^T is computed every iteration (Algorithm 2, line 6)
    The theory does not cover the Newton-Schulz approximate orthogonalization used in practice; the approximation error is absent from the analysis.
  • standard math Independent minibatches and linear momentum yield the recursion in Eq. (9) for the momentum error
    The derivation in Appendix C sums geometric series and applies Cauchy-Schwarz to control noise terms.
  • standard math Known lower bounds [2] and [6] are valid benchmarks for this algorithm class
    Used to label the rates optimal; the lower bounds are for standard first-order stochastic optimization, not specifically for spectral-norm orthogonalized dynamics.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updates." pith.science (2026). https://pith.science/paper/6KV3STAR

@misc{pith2026250902981,
  author       = {Pith},
  title        = {Pith review of: AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updates},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6KV3STAR}},
  note         = {Machine review of arXiv:2509.02981}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The recently proposed Muon optimizer updates weight matrices via orthogonalized momentum and has demonstrated strong empirical success in large language model training. However, it remains unclear how to determine the learning rates for such orthogonalized updates. AdaGrad, by contrast, is a widely used adaptive method that scales stochastic gradients by accumulated past gradients. We propose a new algorithm, AdaGO, which combines a norm-based AdaGrad-type stepsize with an orthogonalized update direction, bringing together the benefits of both approaches. Unlike other adaptive variants of Muon, AdaGO preserves the orthogonality of the update direction, which can be interpreted as a spectral descent direction, while adapting the stepsizes to the optimization landscape by scaling the direction with accumulated past gradient norms. The implementation of AdaGO requires only minimal modification to Muon, with a single additional scalar variable, the accumulated squared gradient norms, to be computed, making it computationally and memory efficient. Optimal theoretical convergence rates are established for nonconvex functions in both stochastic and deterministic settings under standard smoothness and unbiased bounded-variance noise assumptions. Empirical results on CIFAR-10 classification and function regression demonstrate that AdaGO outperforms Muon and Adam.

Figures

Figures reproduced from arXiv: 2509.02981 by Hayden Schaeffer, Minxin Zhang, Yuxuan Liu.

Figure 1
Figure 1. Figure 1: Comparing optimizer performance for the regression task. [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparing the performance of optimizers on the CIFAR-10 image classification task. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: GD vs OGD with varied learning rates (lr) [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\ell_1$-norm Lower Bounds

    cs.LG 2026-05 unverdicted novelty 8.0

    SignSGD provably beats SGD by a factor of d under sparse noise via matched ℓ1-norm upper and lower bounds, with an equivalent result for Muon on matrices, and this predicts faster GPT-2 pretraining.

  2. OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality

    math.OC 2026-06 unverdicted novelty 7.0

    OptMuon combines orthogonalized momentum with closed-loop adaptation to achieve noise-adaptive convergence rates that automatically become near-optimal deterministic first-order rates without retuning when noise vanishes.

  3. Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers

    math.OC 2026-05 conditional novelty 7.0

    Proposes equivariant optimizers matched to the symmetry groups of embeddings, SwiGLU projections and MoE routers, with experiments showing consistent gains over AdamW on language model pre-training.

  4. A unified convergence theory for adaptive first-order methods in the nonconvex case, including AdaNorm, full and diagonal AdaGrad, Shampoo and Muo

    cs.LG 2026-04 unverdicted novelty 7.0

    A unified stochastic convergence theory is developed for adaptive preconditioned first-order methods including AdaGrad variants, Shampoo, and Muon in nonconvex optimization.

  5. Muse: Representation Geometry of Muon Beyond Normalized Momentum

    cs.LG 2026-07 conditional novelty 6.0

    The matrix shape given to Muon's orthonormalization step is a genuine optimizer axis: square-ish reshapes match native Muon, skinnier reshapes interpolate toward normalized SGD with momentum.

  6. OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

    cs.LG 2026-07 conditional novelty 6.0

    A meta-pipeline plus LMO four-axis view yields a dual taxonomy of 108 optimizers, and a multi-objective LLM/vision benchmark shows no single family dominates the quality–cost–memory frontier.

  7. OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality

    math.OC 2026-06 unverdicted novelty 6.0

    OptMuon combines orthogonalized momentum with trajectory-dependent AdaGrad-Norm adaptation to obtain expected-stationarity rates of order T^{-1/2} + sigma^{1/2}T^{-1/4} or T^{-1/2} + sigma^{1/3}T^{-1/3} that reduce to...

  8. Stochastic convergence of parallel asynchronous adaptive first-order methods

    cs.AI 2026-06 unverdicted novelty 6.0

    Introduces a class of asynchronous adaptive first-order methods and establishes O(1/sqrt t) convergence (up to logs) for non-convex stochastic optimization under reasonable assumptions.

  9. Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers

    math.OC 2026-05 unverdicted novelty 6.0

    Proposes equivariant optimizer updates matched to layer symmetries for embeddings, SwiGLU MLPs, and MoE routers, with reported gains in validation loss and training stability on several language model architectures.

  10. Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimization

    cs.LG 2026-02 conditional novelty 6.0

    DeVA_S8 reweights Muon's matrix-sign update in the matrix's eigenbasis with a singular-value signal-to-noise ratio, reaching target LLM validation perplexity with ~6.6% fewer tokens than Muon.

  11. Convergence of Spectral Descent for Non-smooth Optimization

    cs.LG 2026-05 unverdicted novelty 5.0

    Proves linear convergence of Spectral Descent (SD) and Truncated SD for non-smooth convex problems under stated conditions, sublinear rates for regularized versions via Frank-Wolfe, and recovery guarantees for robust ...

  12. Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectives

    math.OC 2026-05 unverdicted novelty 5.0

    Proximal stochastic spectral preconditioning converges for nonconvex constrained objectives under heavy-tailed noise, with a variance-reduced version achieving faster rates and a refined analysis of Muon iterations.

  13. A Muon-Accelerated Algorithm for Low Separation Rank Tensor Generalized Linear Models

    stat.ML 2026-04 unverdicted novelty 5.0

    LSRTR-M integrates Muon updates into the LSRTR algorithm for tensor GLMs, achieving faster convergence, lower estimation errors on synthetic linear/logistic/Poisson models, and competitive performance with better effi...

  14. Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Rates

    math.OC 2025-11 conditional novelty 5.0

    Non-Euclidean SGD variants (SignSGD, Muon) provably match adaptive optimizers' convergence rates under structured smoothness and noise assumptions.

Reference graph

Works this paper leans on

37 extracted references · 13 canonical work pages · cited by 12 Pith papers · 1 internal anchor

  1. [1]

    K. An, Y. Liu, R. Pan, Y. Ren, S. Ma, D. Goldfarb, and T. Zhang. Asgo: Adaptive structured gradient optimization.arXiv preprint arXiv:2503.20762, 2025

    arXiv 2025

  2. [2]

    Arjevani, Y

    Y. Arjevani, Y. Carmon, J. C. Duchi, D. J. Foster, N. Srebro, and B. Woodworth. Lower bounds for non-convex stochastic optimization.Mathematical Programming, 199(1):165–214, 2023

    work page 2023

  3. [3]

    Bernstein

    J. Bernstein. The modula docs, 2025. URLhttps://docs.modula.systems/

    work page 2025

  4. [4]

    Bernstein and L

    J. Bernstein and L. Newhouse. Old optimizer, new norm: An anthology.arXiv preprint arXiv:2409.20325, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    D. E. Carlson, E. Collins, Y.-P. Hsieh, L. Carin, and V. Cevher. Preconditioned spectral descent for deep learning.Advances in neural information processing systems, 28, 2015

    work page 2015

  6. [6]

    Carmon, J

    Y. Carmon, J. C. Duchi, O. Hinder, and A. Sidford. Lower bounds for finding stationary points i.Mathematical Programming, 184(1):71–120, 2020

    work page 2020

  7. [7]

    L. Chen, J. Li, and Q. Liu. Muon optimizes under spectral norm constraints.arXiv preprint arXiv:2506.15054, 2025

    arXiv 2025

  8. [8]

    Cutkosky and H

    A. Cutkosky and H. Mehta. Momentum improves normalized sgd. InInternational conference on machine learning, pages 2260–2268. PMLR, 2020

    work page 2020

  9. [9]

    D´ efossez, L

    A. D´ efossez, L. Bottou, F. Bach, and N. Usunier. A simple convergence proof of adam and adagrad.arXiv preprint arXiv:2003.02395, 2020

    Pith/arXiv arXiv 2003

  10. [10]

    Duchi, E

    J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization.Journal of machine learning research, 12(7), 2011

    work page 2011

  11. [11]

    C. Fan, M. Schmidt, and C. Thrampoulidis. Implicit bias of spectral descent and muon on multiclass separable data.arXiv preprint arXiv:2502.04664, 2025

    arXiv 2025

  12. [12]

    M. Faw, I. Tziotis, C. Caramanis, A. Mokhtari, S. Shakkottai, and R. Ward. The power of adaptivity in sgd: Self-tuning step sizes with unbounded gradients and affine variance. In Conference on Learning Theory, pages 313–355. PMLR, 2022

    work page 2022

  13. [13]

    Gupta, T

    V. Gupta, T. Koren, and Y. Singer. Shampoo: Preconditioned stochastic tensor optimization. InInternational Conference on Machine Learning, pages 1842–1850. PMLR, 2018

    work page 2018

  14. [14]

    N. J. Higham. Computing the polar decomposition—with applications.SIAM Journal on Scientific and Statistical Computing, 7(4):1160–1174, 1986

    1986

  15. [15]

    Jordan, Y

    K. Jordan, Y. Jin, V. Boza, Y. Jiacheng, F. Cecista, L. Newhouse, and J. Bernstein. Muon: An optimizer for hidden layers in neural networks.URL https://kellerjordan. github. io/posts/muon, 2024

    work page 2024

  16. [16]

    Jordan, Y

    K. Jordan, Y. Jin, V. Boza, Y. Jiacheng, F. Cesista, L. Newhouse, and J. Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URLhttps://kellerjordan.github. io/posts/muon/. 9

    work page 2024

  17. [17]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    Pith/arXiv arXiv 2014

  18. [18]

    D. Kovalev. Sgd with adaptive preconditioning: Unified analysis and momentum acceleration. arXiv preprint arXiv:2506.23803, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    D. Kovalev. Understanding gradient orthogonalization for deep learning via non-euclidean trust-region optimization.arXiv preprint arXiv:2503.12645, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    T. T.-K. Lau, Q. Long, and W. Su. Polargrad: A class of matrix-gradient optimizers from a unifying preconditioning perspective.arXiv preprint arXiv:2505.21799, 2025

    arXiv 2025

  21. [21]

    Li and M

    J. Li and M. Hong. A note on the convergence of muon.arXiv preprint arXiv:2502.02900, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Li and F

    X. Li and F. Orabona. On the convergence of stochastic gradient descent with adaptive stepsizes. InThe 22nd international conference on artificial intelligence and statistics, pages 983–992. PMLR, 2019

    work page 2019

  23. [23]

    J. Liu, J. Su, X. Yao, Z. Jiang, G. Lai, Y. Du, Y. Qin, W. Xu, E. Lu, J. Yan, et al. Muon is scalable for llm training.arXiv preprint arXiv:2502.16982, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    L. Liu, Z. Xu, Z. Zhang, H. Kang, Z. Li, C. Liang, W. Chen, and T. Zhao. Cosmos: A hybrid adaptive optimizer for memory-efficient training of llms.arXiv preprint arXiv:2502.17410, 2025

    arXiv 2025

  25. [25]

    F. Orabona. Normalized gradients for all.arXiv preprint arXiv:2308.05621, 2023

    Pith/arXiv arXiv 2023

  26. [26]

    Pethick, W

    T. Pethick, W. Xie, K. Antonakopoulos, Z. Zhu, A. Silveti-Falls, and V. Cevher. Training deep learning models with norm-constrained lmos.arXiv preprint arXiv:2502.07529, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    N. Sato, H. Naganuma, and H. Iiduka. Analysis of muon’s convergence and critical batch size. arXiv preprint arXiv:2507.01598, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    I. Shah, A. M. Polloreno, K. Stratos, P. Monk, A. Chaluvaraju, A. Hojel, A. Ma, A. Thomas, A. Tanwer, D. J. Shah, et al. Practical efficiency of muon for pretraining.arXiv preprint arXiv:2505.02222, 2025

    Pith/arXiv arXiv 2025

  29. [29]

    W. Shen, R. Huang, M. Huang, C. Shen, and J. Zhang. On the convergence analysis of muon. arXiv preprint arXiv:2505.23737, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    C. Si, D. Zhang, and W. Shen. AdaMuon: Adaptive muon optimizer.arXiv preprint arXiv:2507.11005, 2025

    arXiv 2025

  31. [31]

    Streeter and H

    M. Streeter and H. B. McMahan. Less regret via online conditioning.arXiv preprint arXiv:1002.4862, 2010

    Pith/arXiv arXiv 2010

  32. [32]

    N. Vyas, D. Morwani, R. Zhao, M. Kwun, I. Shapira, D. Brandfonbrener, L. Janson, and S. Kakade. Soap: Improving and stabilizing shampoo using adam.arXiv preprint arXiv:2409.11321, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    B. Wang, H. Zhang, Z. Ma, and W. Chen. Convergence of adagrad for non-convex objectives: Simple proofs and relaxed assumptions. InThe Thirty Sixth Annual Conference on Learning Theory, pages 161–190. PMLR, 2023. 10

    work page 2023

  34. [34]

    R. Ward, X. Wu, and L. Bottou. Adagrad stepsizes: Sharp convergence over nonconvex landscapes.Journal of Machine Learning Research, 21(219):1–30, 2020

    2020

  35. [35]

    S. Xie, T. Wang, S. Reddi, S. Kumar, and Z. Li. Structured preconditioners in adaptive optimization: A unified analysis.arXiv preprint arXiv:2503.10537, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    T. Zhang. Sharp perturbation bounds on the frobenius norm of subunitary and positive polar factor.arXiv preprint arXiv:2507.14940, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  37. [37]

    t−1X τ=0 µτ ˜Et+1−τ ∗ # +µL t−1X τ=0 µτ E[max{ϵ, ηαt−τ }] ≤ µtκ√r√ b + (1−µ) √rE

    D. Zhou, J. Chen, Y. Cao, Z. Yang, and Q. Gu. On the convergence of adaptive gradient methods for nonconvex optimization.arXiv preprint arXiv:1808.05671, 2018. Appendix A. Motivating Example: GD vs. OGD in a Linear Case As a motivating example, we compare GD and OGD in training a one-layer linear neural network. The loss function is given by L(W) := 1 2 J...

    Pith/arXiv arXiv 2018

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.