Pith. sign in

REVIEW 3 major objections 5 minor 116 references

Muon's reported edge over AdamW vanishes under per-optimizer tuning in low-rank matrix factorization.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 05:44 UTC pith:MRDNH4AO

load-bearing objection A useful, carefully controlled negative result: Muon is not uniformly better than AdamW on matrix factorization, but the claim rests on LR-only tuning and deserves a referee who will push on Muon's fixed hyperparameters. the 3 major comments →

arxiv 2607.13246 v1 pith:MRDNH4AO submitted 2026-07-14 cs.LG cs.AI

Reassessing Muon for Matrix Factorization

Ali Parviz , Gal Mishne , Alex Cloninger This is my paper
classification cs.LG cs.AI MSC 68T0790C2665K10
keywords Muon optimizermatrix factorizationAdamWspectral orthogonalizationoptimizer comparisonnonnegative matrix factorizationlow-rank matrix recoveryhyperparameter tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Muon reshapes gradient updates by approximately orthogonalizing them through a Newton–Schulz polar factor, and it has been reported to speed up large-scale language-model training. This paper asks whether that advantage belongs to the update rule itself or to the scale, architecture, and data of modern networks, by testing Muon on low-rank matrix factorization with each optimizer individually tuned over a learning-rate grid. The central finding is that under per-optimizer tuning no method dominates: tuned AdamW and gradient descent match or beat Muon on plain low-rank factorization, Muon and AdamW are comparable on matrix completion, and Muon wins clearly on nonnegative matrix factorization and on deep tensor-train factorization. A sympathetic reader should take away that Muon's benefit is problem-dependent and concentrated in nonnegativity-constrained and deep-product regimes, and that single-learning-rate comparisons are not reliable evidence about optimizer quality.

Core claim

On the paper's own terms: across five matrix-recovery problems, with a controlled protocol that sweeps only the learning rate (20 or 25 log-spaced points) and holds all other hyperparameters fixed, Muon does not consistently outperform AdamW. At matched tuning, AdamW and plain gradient descent reach near machine precision on low-rank factorization while Muon plateaus several orders higher; on matrix completion the two adaptive methods are indistinguishable; on nonnegative factorization Muon wins clearly, reaching losses of 1e-9 to 1e-6 where AdamW stalls near 1e-3. An analytical derivation shows why: Muon's update is the polar factor of the negative gradient (the orthonormal-column matrix cl

What carries the argument

The comparison centers on Muon's Newton–Schulz orthogonalization, an approximate polar factor that discards singular-value magnitudes and acts as spectral-norm steepest descent, versus AdamW's coordinate-wise rescaling by the square root of a second moment. The experimental machinery is a systematic learning-rate sweep (25 or 20 log-spaced points per problem–optimizer–condition triple) with momentum, Nesterov flag, Newton–Schulz count, and orthogonalization coefficients held at standard values, plus a patience-based decay schedule and three seeds. This protocol exposes each method's stable learning-rate range and reveals that losses vary by roughly eleven orders of magnitude across the grid,

Load-bearing premise

The conclusion that Muon does not consistently beat AdamW rests on treating a learning-rate-only sweep (20 or 25 log-spaced values, all other hyperparameters fixed at standard values) as equal tuning; if Muon's momentum, Nesterov flag, Newton–Schulz count, or orthogonalization coefficients were tuned per problem, its reported losses could shift enough to change several rankings.

What would settle it

Re-run the paper's Table 1 protocol with per-optimizer sweeps over all free hyperparameters (momentum, Nesterov flag, Newton–Schulz count, orthogonalization coefficients for Muon; beta_1, beta_2, and weight decay for AdamW). The central claim fails if Muon then wins on plain low-rank factorization, or if AdamW wins on nonnegative matrix factorization.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Muon's reported gains on large-language-model training should not be attributed to a generic spectral-preconditioning benefit; on simple matrix factorization a well-tuned AdamW matches or beats it.
  • On nonnegative matrix factorization, Muon's orthogonalized updates consistently win across ranks and spectral shapes, suggesting its value there comes from discouraging redundant, collinear factors.
  • In tensor-train factorizations of increasing depth, Muon overtakes AdamW by one to two orders of magnitude at depth 6, so its advantage appears tied to deep, ill-conditioned product parameterizations rather than shallow ones.
  • Because loss-minimizing learning rates differ across optimizers by 1.5–3.1 decades and tuning reverses default-rate rankings in nearly all nineteen settings, any comparison at a single learning rate is an artifact of that choice.
  • At a fixed extreme condition number, AdamW adapts to clustered or heavy-tailed spectra while Muon does not; spectrum shape, not just condition number, determines which optimizer wins.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: if full hyperparameter tuning (momentum, Nesterov, Newton–Schulz count, orthogonalization coefficients) were included, Muon's plain-factorization gap might shrink, but the paper's structural story predicts it would not reverse; the decisive experiment is to re-run Table 1 with per-optimizer sweeps over all knobs.
  • The derived sqrt(N) scaling between AdamW and Muon step norms implies that when transferring Muon across matrix sizes, retuning should start at eta_muon ≈ sqrt(N) * eta_adamw rather than a fresh sweep; this is directly testable and, if confirmed, would give practical guidance the paper mostly leaves implicit.
  • The subspace-alignment dynamics suggest Muon's implicit bias is more isotropic, distributing capacity across directions; on tasks where balanced recovery matters more than final training loss, such as fair or robust representation learning, Muon might generalize better even where its training loss is worse. This follows from the paper's mechanism but is not claimed in it.
  • If the NMF advantage is truly about discouraging redundant factors, a prediction follows for tabular or sparse data with near-collinear features: Muon should show a similar edge on nonnegativity-constrained dictionary learning, a neighboring problem the paper does not test.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper revisits Muon on low-rank matrix factorization. It formulates symmetric factorization, matrix completion, NMF, tensor-train, and kernel factorization problems; for each, it sweeps learning rate over 20-30 log-spaced points while fixing all other optimizer hyperparameters, runs 3 seeds, and reports tuned final NMSE for Muon, AdamW, GD, and SignGD. The main empirical claim is that under this per-optimizer tuning, no method dominates: AdamW/GD win plain low-rank factorization, Muon and AdamW are comparable on completion, and Muon clearly wins on both NMF variants; conditioning and spectral shape change rankings. The paper includes a spectral-subspace analysis and an analytical appendix claiming to derive AdamW and Muon updates from trust-region and Procrustes proxies. The authors conclude that Muon's large-scale advantages are problem-dependent and argue for controlled benchmarks.

Significance. The controlled problem suite and the distinction between default-rate and tuned-rate rankings are useful; the paper makes a plausible case that Muon's advantage in LLM pretraining does not transfer automatically to simple spectral problems. If the empirical claims survive a broader hyperparameter sweep, the paper would be a valuable cautionary contribution to optimizer evaluation methodology. The learning-rate sweeps are extensive, the baselines are standard, and the NMF/tensor-train results provide a contrasting regime where orthogonalization helps. However, the headline negative claim is more conditional than the abstract states, for the reasons detailed below.

major comments (3)
  1. [Section 3.2 and Table 1] The protocol defines 'tuning' as a learning-rate sweep only: 25 or 20 log-spaced LR points with 'all other hyperparameters (momentum, and for Muon the Newton-Schulz count J and the orthogonalization coefficients) at standard values.' The Muon update (Eqs. 6-8 and 11) depends on mu, the Nesterov flag, J, and (a,b,c), and AdamW depends on beta1, beta2, and epsilon; none of these are varied. Therefore the central claim 'under per-optimizer tuning no method dominates universally' (Section 4, Table 1) is actually a claim about LR-tuned defaults. Since a non-representative default configuration could conceal a Muon advantage (or create one), the headline needs either a sensitivity analysis (at least over J and the Newton-Schulz coefficients, and preferably over mu) or a consistent rephrasing as 'under learning-rate-only tuning.' This concern directly bears on the paper's main conclusion.
  2. [Appendix L, Proposition 1 and Eq. (22)] Proposition 1 states that the AdamW update is the analytical solution of the linearized-loss/trust-region proxy. However, Eq. (22) uses the raw gradient G_t in the numerator, whereas the actual AdamW update uses the bias-corrected first moment m_t (with m_t = beta1 m_{t-1} + (1-beta1) G_t). The object solved in Eqs. (21)-(22) is a normalized-gradient method, not AdamW. Consequently the scale comparison in Eq. (37) and the explanation of why the Muon learning rate should be scaled by sqrt(N) are not valid as a derivation for AdamW. This does not invalidate the empirical Table 1, but it is a real inaccuracy in the theoretical framing and should be corrected or explicitly labeled as a heuristic for a gradient-normalization variant.
  3. [Table 1] Table 1, the main quantitative support for the 'no universal winner' claim, reports only geometric means over 3 seeds, with no per-seed values or confidence intervals. Some adjacent entries are extremely close (e.g., factorization kappa=1: AdamW 1.9e-13 vs GD 1.6e-13; completion kappa=1: Muon 4.6e-16 vs AdamW 4.8e-16), so the bold-faced 'best per row' is not robust to seed variation. The paper should report the spread (e.g., per-seed losses or log-std intervals in Table 1) or soften the ranking claims where differences are within noise.
minor comments (5)
  1. [Appendices E and J] Appendix E and Appendix J contain identical text, with the alignment matrix defined as Eq. (15) and Eq. (16). Please consolidate or remove one duplicate.
  2. [Figure 4 caption vs. text] The Figure 4 caption reports diagonal energy values 0.505 and 0.578, while the text reports 0.586 vs 0.643 in the caption/body. Please verify which numbers are correct.
  3. [Abstract and Section 3.2] The abstract says 'carefully tuned comparison' and the text says 'per-optimizer tuning,' but the protocol is a learning-rate-only sweep. The wording should be aligned with what was actually done.
  4. [Appendix L, Eq. (22)] The denominator in the AdamW-style update is written as sqrt(v_t) without the epsilon term used in the standard algorithm (sqrt(v_hat) + epsilon). Add epsilon or state that it is omitted.
  5. [Section 2 and Eq. (37)] The sentence 'We analyze the exact softplus...' has a stray capital 'We' after a comma. Also, Eq. (37) introduces N=1000 for the step-size ratio while the main experiments use d=100; clarify whether N refers to a different setting.

Circularity Check

0 steps flagged

No significant circularity: direct empirical comparisons against external baselines, no self-citations, and no fitted inputs renamed as predictions.

full rationale

The paper's central claims—that Muon does not consistently outperform AdamW on low-rank factorization under per-optimizer tuning, and that results are sensitive to learning rate and conditioning—are direct empirical comparisons against external baselines (AdamW, GD, SignGD). Table 1 reports minima over explicit learning-rate grids, not fitted predictions of a related quantity, so no fitted-input-called-prediction pattern appears. The reference list contains no self-citations by the authors (Parviz, Mishne, Cloninger), so there is no load-bearing self-citation or imported uniqueness theorem. Appendix L derives update-norm scaling from a local proxy problem; although Eq. 22 replaces AdamW's first moment m_t with the raw gradient G_t, making the derivation unfaithful to actual AdamW, this is a correctness/faithfulness issue rather than circularity: the derived sqrt(N) scaling follows from the stated proxy assumptions and is not an input to the empirical comparison. The main limitation—that Section 3.2 defines 'per-optimizer tuning' as a learning-rate sweep only, holding Muon's momentum, Nesterov flag, Newton–Schulz count J, and orthogonalization coefficients fixed—conditions the generality of the negative conclusion but does not make any result reduce by construction to its own inputs. No derivation chain in the paper is equivalent to its inputs, and no prediction is forced by a self-citation or by definition.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper's empirical contribution is largely self-contained; its central claims depend on experimental design choices (LR-only tuning, standard hyperparameters, small random init) rather than on fitted constants. The only 'derivation' content, Appendix L, is parameter-free but contains a proxy mismatch for AdamW.

free parameters (4)
  • learning rate α (per optimizer, per setting) = best point on logarithmic grid; exact values not tabulated (grid [1e-4,5e-1] or [1e-4,3.2])
    The entire 'tuned comparison' is defined by choosing the best LR per method; this is the only hyperparameter swept, so the reported rankings are LR-optimal, not jointly optimal.
  • Muon momentum μ = 0.95
    Fixed at Jordan's standard value; not swept, although momentum is a core component of the Muon direction and could change its basin on NMF/completion.
  • Newton-Schulz iteration count J = 5
    Fixed at standard value; more/fewer iterations change orthogonality approximation quality and could affect comparisons.
  • AdamW betas/epsilon/weight decay = not stated ('standard values')
    These defaults determine AdamW's adaptive scaling and basin shape; omitting them makes 'equal tuning' not fully specified.
axioms (4)
  • standard math Thin SVD and polar decomposition of the gradient exist and satisfy the trace/bound identities used in Appendix L.
    Used in Propositions 1–2 and norm scaling; standard linear algebra, no threat.
  • domain assumption Small random initialization is used for all problems.
    Section 3.1 states 'small random initialization' but does not specify distribution/scale; matrix factorization is known to be initialization-sensitive, and conclusions could depend on this choice.
  • domain assumption The factorized-matrix objective is a faithful minimal model of deep learning's product parameterizations.
    Appendix H argues this; it is the premise that lets the authors generalize from d=100 factorization to Muon's LLM behavior. If the proxy is not faithful, the broader implications are unsupported.
  • ad hoc to paper Muon's NMF advantage is caused by orthogonalized updates discouraging redundant factors.
    Section 1 Findings and Section 5 assert this mechanism; the paper provides no controlled ablation proving it, so it is an interpretation rather than a measured result.

pith-pipeline@v1.3.0-alltime-deepseek · 20626 in / 14371 out tokens · 151496 ms · 2026-08-02T05:44:23.409474+00:00 · methodology

0 comments
read the original abstract

Muon has recently emerged as a strong optimizer for large-scale deep learning, where it reshapes gradient updates through approximate orthogonalization and has been reported to outperform Adam and AdamW in large language model training. Its empirical success has motivated a growing body of theoretical work that interprets Muon as steepest descent under the spectral norm. Yet it remains unclear which of Muon's advantages stem from its update rule itself and which are artifacts of the scale, architecture, and data of modern deep networks. In this work, we isolate the optimizer from these confounding factors by studying Muon on a simple, well-understood, and spectrally structured problem: low-rank matrix factorization. Through a controlled comparison against carefully tuned adaptive baselines, we find that Muon does not consistently outperform AdamW in this setting and that several previously reported advantages are sensitive to hyperparameter choices. Our results provide a more nuanced picture of when spectrum-aware orthogonalization is beneficial and argue for evaluating modern optimizers on controlled problems in addition to end-to-end benchmarks.

Figures

Figures reproduced from arXiv: 2607.13246 by Alex Cloninger, Ali Parviz, Gal Mishne.

Figure 1
Figure 1. Figure 1: Low-rank factorization, conditioning sweep (target rank r=15, matched search rank k=15, dimension d=100). Tuned final loss vs. learning rate for each condition number κ, from well-conditioned (κ=1) to strongly ill-conditioned (κ=625) (geometric mean over 3 seeds, ±log-std band). The target’s 15 singular values are spaced linearly from κ down to 1, so κ is exactly the condition number. When tuned, AdamW and… view at source ↗
Figure 2
Figure 2. Figure 2 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: AdamW outperforms Muon variants across spectrum shapes (matrix factorization, target m×n = 150×150, rank r=5). Tuned final loss for each of the seven spectral profiles at fixed κ = 104 (mean over 3 seeds). Both Muon and Muon without Nesterov-style momentum exhibit degraded relative performance on most spectral shapes, whereas AdamW remains effective despite the extreme ill-conditioning. 0 20 40 60 80 0 20 … view at source ↗
Figure 4
Figure 4. Figure 4: Spectral subspace recovery (Gaussian-kernel target K ∈ R 100×100, full rank k=100, tuned LR, 12 inits). Alignment Mij = |⟨ui , ej ⟩| (Eq. 16) between learned singular vectors and true eigenvectors of K. (a) AdamW and (b) Muon both recover the top eigen￾vectors, diverging in the near-degenerate mid-band. (c) The difference isolates it: AdamW holds a tighter ordered diagonal (blue), Muon disperses onto neigh… view at source ↗
Figure 5
Figure 5. Figure 5: Tensor-Train Noiseless regime (true rank r ⋆ = 4). Final loss versus learning rate for each optimizer, across search ranks r ∈ {2, 4, 8, 20} spanning under-, exactly- (r = r ⋆ , bold panel), and over-parameterized settings. Solid lines are the median over 3 seeds; shaded bands the min–max. 3.2. Evaluation protocol We evaluate recovery with the normalized mean-squared error (NMSE), the squared recon￾structi… view at source ↗
Figure 6
Figure 6. Figure 6: Spectral subspace recovery: Muon vs. AdamW. Absolute alignment Mij = |⟨ui , ej ⟩| (Eq. 16) between learned singular vectors ui and true eigenvectors ej of K, every 500 steps (AdamW top, Muon bottom). A bright, correctly-ordered diagonal indicates faithful recovery; off-diagonal mass reflects mixing. AdamW sharpens its diagonal steadily, while Muon stays near-random for ∼1500 steps, then snaps into a more d… view at source ↗
Figure 7
Figure 7. Figure 7: Matrix completion, conditioning sweep. Each panel plots the final re￾construction loss (y-axis, log scale; geometric mean over 3 seeds with a shaded ±one-log￾standard-deviation band) against the learning rate (x-axis, log scale), for condition number κ. Muon and AdamW are indistinguishable in the well-conditioned regime; all methods degrade and converge toward one another as κ grows. 1e-04 4e-04 2e-03 7e-0… view at source ↗
Figure 8
Figure 8. Figure 8: Matrix completion, search-rank sweep (true rank 4, κ=5). Over￾parameterization (rank 100) widens the band of effective learning rates and lets every method except GD reach machine precision. Implicit bias and generalization. A second line of work studies the implicit bias induced by Muon. Fan et al. (2025) show that in linear classification, idealized Muon converges to a solution that maximizes margin with… view at source ↗
Figure 9
Figure 9. Figure 9: Tensor-Train Noisy regime (true rank r ⋆ = 30, additive observation noise). Final loss versus learning rate for each optimizer, across search ranks r ∈ {5, 15, 30, 60} spanning under-, exactly- (r = r ⋆ , bold panel), and over-parameterized settings. Solid lines are the median over 3 seeds; shaded bands the min–max; the loss floors at the noise level rather than at zero. been drawn between Muon and second-… view at source ↗
Figure 10
Figure 10. Figure 10: Gaussian-kernel NMF at a principled bottleneck. Non-negative factoriza￾tion of a fixed Gaussian RBF kernel K ∈ R 100×100 at search rank r=24 (the smallest rank capturing 95% of K’s spectral energy). Tuned final loss vs. learning rate (geometric mean over 3 seeds, ±log-std band; patience-based decay schedule). On this smooth, well-behaved landscape Muon’s orthogonalized updates give no advantage: AdamW rea… view at source ↗
Figure 11
Figure 11. Figure 11: Best-tuned reconstruction loss versus tensor-train depth. Each curve shows the final MSE of one optimizer at its best learning rate (minimum over the grid) for a given depth, averaged over three seeds; shaded bands span the per-seed min–max. Left: clean, low-rank target. Right: noisy, higher-rank target. Search bond rank is fixed to the over-parameterized regime (R=12 clean, R=30 noisy). Lower is better; … view at source ↗
Figure 12
Figure 12. Figure 12: Learning-rate stability across activation and capacity. Final MSE (log) vs. learning rate (log) for four optimizers fitting the Gaussian kernel as Kˆ = ϕ(XW)V ⊤; mean over three seeds, bands show min–max, divergent runs capped at 105 . The activation ϕ is exact softplus or its second-order Taylor surrogate, and R is the search rank: (a) softplus, R=10; (b) Taylor, R=10; (c) softplus, R=100; (d) Taylor, R=… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

116 extracted references · 1 canonical work pages

  1. [1]

    2024 , note =

    Keller Jordan , title =. 2024 , note =

    2024

  2. [2]

    CoRR , volume =

    Jingyuan Liu and Jianlin Su and Xingcheng Yao and Zhejun Jiang and Guokun Lai and Yulun Du and Yidao Qin and Weixin Xu and Enzhe Lu and Junjie Yan and Yanru Chen and Huabin Zheng and Yibo Liu and Shaowei Liu and Bohong Yin and Weiran He and Han Zhu and Yuzhi Wang and Jianzhou Wang and Mengnan Dong and Zheng Zhang and Yongsheng Kang and Hao Zhang and Xinra...

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.16982 2025

  3. [3]

    Improved Convergence Rates of

    Shuntaro Nagashima and Hideaki Iiduka , year=. Improved Convergence Rates of. 2601.19400 , archivePrefix=

    arXiv

  4. [4]

    Training Deep Learning Models with Norm-Constrained LMOs , booktitle =

    Thomas Pethick and Wanyun Xie and Kimon Antonakopoulos and Zhenyu Zhu and Antonio Silveti. Training Deep Learning Models with Norm-Constrained LMOs , booktitle =. 2025 , url =

    2025

  5. [5]

    2022 , eprint=

    Orthogonalising gradients to speed up neural network optimisation , author=. 2022 , eprint=

    2022

  6. [6]

    Proceedings of the 35th International Conference on Machine Learning,

    Jeremy Bernstein and Yu. Proceedings of the 35th International Conference on Machine Learning,. 2018 , url =

    2018

  7. [7]

    Preconditioning Benefits of Spectral Orthogonalization in

    Jianhao Ma and Yu Huang and Yuejie Chi and Yuxin Chen , year=. Preconditioning Benefits of Spectral Orthogonalization in. 2601.13474 , archivePrefix=

    arXiv

  8. [8]

    International conference on machine learning , pages=

    On the optimization of deep networks: Implicit acceleration by overparameterization , author=. International conference on machine learning , pages=. 2018 , organization=

    2018

  9. [9]

    arXiv preprint arXiv:1711.00811 , year=

    Expressive power of recurrent neural networks , author=. arXiv preprint arXiv:1711.00811 , year=

    Pith/arXiv arXiv

  10. [10]

    Conference on learning theory , pages=

    On the expressive power of deep learning: A tensor analysis , author=. Conference on learning theory , pages=. 2016 , organization=

    2016

  11. [11]

    Advances in neural information processing systems , volume=

    Tensorizing neural networks , author=. Advances in neural information processing systems , volume=

  12. [12]

    A Note on the Convergence of

    Jiaxiang Li and Mingyi Hong , year=. A Note on the Convergence of. 2502.02900 , archivePrefix=

    Pith/arXiv arXiv

  13. [13]

    Gluon: Making

    Artem Riabinin and Egor Shulgin and Kaja Gruntkowska and Peter Richt. Gluon: Making. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.13416 , eprinttype =. 2505.13416 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.13416 2025

  14. [14]

    CoRR , volume =

    Wei Shen and Ruichuan Huang and Minhui Huang and Cong Shen and Jiawei Zhang , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.23737 , eprinttype =. 2505.23737 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.23737 2025

  15. [15]

    CoRR , volume =

    Lizhang Chen and Jonathan Li and Qiang Liu , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2506.15054 , eprinttype =. 2506.15054 , timestamp =

    doi:10.48550/arxiv.2506.15054 2025

  16. [16]

    Convergence Bound and Critical Batch Size of

    Naoki Sato and Hiroki Naganuma and Hideaki Iiduka , year=. Convergence Bound and Critical Batch Size of. 2507.01598 , archivePrefix=

    Pith/arXiv arXiv

  17. [17]

    Journal of Machine Learning Research , year =

    Vardan Papyan , title =. Journal of Machine Learning Research , year =

  18. [18]

    Proceedings of the 36th International Conference on Machine Learning , pages =

    An Investigation into Neural Net Optimization via Hessian Eigenvalue Density , author =. Proceedings of the 36th International Conference on Machine Learning , pages =. 2019 , editor =

    2019

  19. [19]

    Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond , journal =

    Levent Sagun and L. Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond , journal =. 2017 , url=

    2017

  20. [20]

    Levent Sagun and Utku Evci and V. Ugur G. Empirical Analysis of the Hessian of Over-Parametrized Neural Networks , booktitle =. 2018 , url =

    2018

  21. [21]

    Investigating the Overlooked Hessian Structure: From

    Qian-Yuan Tang and Yufei Gu and Yunfeng Cai and Mingming Sun and Ping Li and zhou Xun and Zeke Xie , booktitle=. Investigating the Overlooked Hessian Structure: From. 2025 , url=

    2025

  22. [22]

    2022 , isbn =

    Measure Theory, Probability, and Stochastic Processes , series =. 2022 , isbn =. doi:10.1007/978-3-031-14205-5 , url =

    work page doi:10.1007/978-3-031-14205-5 2022

  23. [23]

    Polloreno and Karl Stratos and Philip Monk and Adarsh Chaluvaraju and Andrew Hojel and Andrew Ma and Anil Thomas and Ashish Tanwer and Darsh J

    Ishaan Shah and Anthony M. Polloreno and Karl Stratos and Philip Monk and Adarsh Chaluvaraju and Andrew Hojel and Andrew Ma and Anil Thomas and Ashish Tanwer and Darsh J. Shah and Khoi Nguyen and Kurt Smith and Michael Callahan and Michael Pust and Mohit Parmar and Peter Rushton and Platon Mazarakis and Ritvik Kapila and Saurabh Srivastava and Somanshu Si...

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.02222 2025

  24. [24]

    Proceedings of the 39th International Conference on Machine Learning , pages =

    Only tails matter: Average-Case Universality and Robustness in the Convex Regime , author =. Proceedings of the 39th International Conference on Machine Learning , pages =. 2022 , volume =

    2022

  25. [25]

    2019 , eprint=

    Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization , author=. 2019 , eprint=

    2019

  26. [26]

    2019 , url=

    Gradient Descent Happens in a Tiny Subspace , author=. 2019 , url=

    2019

  27. [27]

    Minhak Song and Kwangjun Ahn and Chulhee Yun , booktitle=. Does. 2025 , url=

    2025

  28. [28]

    Proceedings of the 37th International Conference on Machine Learning , pages =

    Acceleration through spectral density estimation , author =. Proceedings of the 37th International Conference on Machine Learning , pages =. 2020 , volume =

    2020

  29. [29]

    CoRR , volume =

    Dmitry Kovalev , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2503.12645 , eprinttype =. 2503.12645 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.12645 2025

  30. [30]

    Beyond the Ideal: Analyzing the Inexact

    Egor Shulgin and Sultan AlRashed and Francesco Orabona and Peter Richt. Beyond the Ideal: Analyzing the Inexact. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2510.19933 , eprinttype =. 2510.19933 , timestamp =

    doi:10.48550/arxiv.2510.19933 2025

  31. [31]

    Error Feedback for

    Kaja Gruntkowska and Alexander Gaponov and Zhirayr Tovmasyan and Peter Richt. Error Feedback for. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2510.00643 , eprinttype =. 2510.00643 , timestamp =

    doi:10.48550/arxiv.2510.00643 2025

  32. [32]

    Convergence of

    Kim, Gyu Yeol and Oh,. Convergence of. The Fourteenth International Conference on Learning Representations , year=

  33. [33]

    CoRR , volume =

    Damek Davis and Dmitriy Drusvyatskiy , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2512.04299 , eprinttype =. 2512.04299 , timestamp =

    doi:10.48550/arxiv.2512.04299 2025

  34. [34]

    CoRR , volume =

    Weijie Su , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2511.00674 , eprinttype =. 2511.00674 , timestamp =

    doi:10.48550/arxiv.2511.00674 2025

  35. [35]

    CoRR , volume =

    Noah Amsel and David Persson and Christopher Musco and Robert Gower , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.16932 , eprinttype =. 2505.16932 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.16932 2025

  36. [36]

    2025 , eprint=

    Factorization-free Orthogonal Projection onto the Positive Semidefinite Cone with Composite Polynomial Filtering , author=. 2025 , eprint=

    2025

  37. [37]

    2024 , url =

    Keller Jordan and Jeremy Bernstein and Brendan Rappazzo and @fernbear.bsky.social and Boza Vlado and You Jiacheng and Franz Cesista and Braden Koszarsky and @Grad62304977 , title =. 2024 , url =

    2024

  38. [38]

    Rakhuba , title =

    Ekaterina Grishina and Matvey Smirnov and Maxim V. Rakhuba , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2506.10935 , eprinttype =. 2506.10935 , timestamp =

    doi:10.48550/arxiv.2506.10935 2025

  39. [39]

    2025 , month =

    Franz Louis Cesista and You Jiacheng and Keller Jordan , title =. 2025 , month =

    2025

  40. [40]

    PolarGrad:

    Tim Tsz. PolarGrad:. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.21799 , eprinttype =. 2505.21799 , timestamp =

    doi:10.48550/arxiv.2505.21799 2025

  41. [41]

    2025 , url =

    Boumal, Nicolas and Gonon, Antoine , title =. 2025 , url =

    2025

  42. [42]

    Advances in neural information processing systems , volume=

    Implicit regularization in deep matrix factorization , author=. Advances in neural information processing systems , volume=

  43. [43]

    Advances in neural information processing systems , volume=

    Implicit regularization in matrix factorization , author=. Advances in neural information processing systems , volume=

  44. [44]

    2025 , eprint=

    Dion: Distributed Orthonormalized Updates , author=. 2025 , eprint=

    2025

  45. [45]

    CoRR , volume =

    Chuan He and Zhanwang Deng and Zhaosong Lu , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2509.11983 , eprinttype =. 2509.11983 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2509.11983 2025

  46. [46]

    CoRR , volume =

    Feihu Huang and Yuning Luo and Songcan Chen , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2509.14562 , eprinttype =. 2509.14562 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2509.14562 2025

  47. [47]

    CoRR , volume =

    Ahmed Khaled and Kaan Ozkara and Tao Yu and Mingyi Hong and Youngsuk Park , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2510.16981 , eprinttype =. 2510.16981 , timestamp =

    doi:10.48550/arxiv.2510.16981 2025

  48. [48]

    CoRR , volume =

    Rui Pan and Yang Luo and Yuxing Liu and Yang You and Tong Zhang , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2510.17802 , eprinttype =. 2510.17802 , timestamp =

    doi:10.48550/arxiv.2510.17802 2025

  49. [49]

    Why transformers need

    Zhang, Yushun and Chen, Congliang and Ding, Tian and Li, Ziniu and Sun, Ruoyu and Luo, Zhiquan , journal=. Why transformers need

  50. [50]

    Provable acceleration of

    Xu, Zhenghao and Wang, Yuqing and Zhao, Tuo and Ward, Rachel and Tao, Molei , booktitle=. Provable acceleration of

  51. [51]

    Conference On Learning Theory , pages=

    Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations , author=. Conference On Learning Theory , pages=. 2018 , organization=

    2018

  52. [52]

    arXiv preprint arXiv:2409.20325 , year=

    Old optimizer, new norm: An anthology , author=. arXiv preprint arXiv:2409.20325 , year=

    Pith/arXiv arXiv

  53. [53]

    Advances in neural information processing systems , volume=

    Preconditioned spectral descent for deep learning , author=. Advances in neural information processing systems , volume=

  54. [54]

    Training deep learning models with norm-constrained

    Pethick, Thomas and Xie, Wanyun and Antonakopoulos, Kimon and Zhu, Zhenyu and Silveti-Falls, Antonio and Cevher, Volkan , journal=. Training deep learning models with norm-constrained

  55. [55]

    BIT Numerical Mathematics , volume=

    Perturbation bounds in connection with singular value decomposition , author=. BIT Numerical Mathematics , volume=. 1972 , publisher=

    1972

  56. [56]

    Proceedings of the International Congress of Mathematicians 2010 , pages=

    Non-asymptotic theory of random matrices: extreme singular values , author=. Proceedings of the International Congress of Mathematicians 2010 , pages=. 2010 , organization=

    2010

  57. [57]

    Numerical Linear Algebra with Applications , volume=

    Some remarks on the perturbation of polar decompositions for rectangular matrices , author=. Numerical Linear Algebra with Applications , volume=. 2006 , publisher=

    2006

  58. [58]

    International Conference on Machine Learning , pages=

    Shampoo: Preconditioned stochastic tensor optimization , author=. International Conference on Machine Learning , pages=. 2018 , organization=

    2018

  59. [59]

    2018 , organization=

    Bernstein, Jeremy and Wang, Yu-Xiang and Azizzadenesheli, Kamyar and Anandkumar, Animashree , booktitle=. 2018 , organization=

    2018

  60. [60]

    Stat , volume=

    On the non-asymptotic and sharp lower tail bounds of random variables , author=. Stat , volume=. 2020 , publisher=

    2020

  61. [61]

    SIAM journal on Matrix Analysis and Applications , volume=

    The geometry of algorithms with orthogonality constraints , author=. SIAM journal on Matrix Analysis and Applications , volume=. 1998 , publisher=

    1998

  62. [62]

    2013 , publisher=

    Matrix computations , author=. 2013 , publisher=

    2013

  63. [63]

    SIAM Journal on Matrix Analysis and Applications , volume=

    New perturbation bounds for the unitary polar factor , author=. SIAM Journal on Matrix Analysis and Applications , volume=. 1995 , publisher=

    1995

  64. [64]

    SIAM journal on matrix analysis and applications , volume=

    Perturbation bounds of unitary and subunitary polar factors , author=. SIAM journal on matrix analysis and applications , volume=. 2002 , publisher=

    2002

  65. [65]

    arXiv preprint arXiv:2410.21265 , year=

    Modular duality in deep learning , author=. arXiv preprint arXiv:2410.21265 , year=

    Pith/arXiv arXiv

  66. [66]

    IEEE Journal of Selected Topics in Signal Processing , volume=

    Stochastic spectral descent for discrete graphical models , author=. IEEE Journal of Selected Topics in Signal Processing , volume=. 2015 , publisher=

    2015

  67. [67]

    Transformers are

    Katharopoulos, Angelos and Vyas, Apoorv and Pappas, Nikolaos and Fleuret, Fran. Transformers are. International conference on machine learning , pages=. 2020 , organization=

    2020

  68. [68]

    2008 , publisher=

    Functions of matrices: theory and computation , author=. 2008 , publisher=

    2008

  69. [69]

    Stochastic spectral descent for restricted

    Carlson, David and Cevher, Volkan and Carin, Lawrence , booktitle=. Stochastic spectral descent for restricted. 2015 , organization=

    2015

  70. [70]

    arXiv preprint arXiv:2202.07052 , year=

    Orthogonalising gradients to speed up neural network optimisation , author=. arXiv preprint arXiv:2202.07052 , year=

    Pith/arXiv arXiv

  71. [71]

    arXiv preprint arXiv:2504.16041 , year=

    Muon Optimizer Accelerates Grokking , author=. arXiv preprint arXiv:2504.16041 , year=

    Pith/arXiv arXiv

  72. [72]

    arXiv preprint arXiv:2310.01769 , year=

    How over-parameterization slows down gradient descent in matrix sensing: The curses of symmetry and initialization , author=. arXiv preprint arXiv:2310.01769 , year=

    Pith/arXiv arXiv

  73. [73]

    The Polar Express: Optimal Matrix Sign Methods and Their Application to the

    Amsel, Noah and Persson, David and Musco, Christopher and Gower, Robert , journal=. The Polar Express: Optimal Matrix Sign Methods and Their Application to the

  74. [74]

    Practical efficiency of

    Shah, Ishaan and Polloreno, Anthony M and Stratos, Karl and Monk, Philip and Chaluvaraju, Adarsh and Hojel, Andrew and Ma, Andrew and Thomas, Anil and Tanwer, Ashish and Shah, Darsh J , journal=. Practical efficiency of

  75. [75]

    Understanding

    Maes, Lucas and Zhang, Tianyue H and Jolicoeur-Martineau, Alexia and Mitliagkas, Ioannis and Scieur, Damien and Lacoste-Julien, Simon and Guille-Escuret, Charles , journal=. Understanding

  76. [76]

    Towards quantifying the preconditioning effect of

    Das, Rudrajit and Agarwal, Naman and Sanghavi, Sujay and Dhillon, Inderjit S , journal=. Towards quantifying the preconditioning effect of

  77. [77]

    arXiv preprint arXiv:2502.01763 , year=

    On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning , author=. arXiv preprint arXiv:2502.01763 , year=

    Pith/arXiv arXiv

  78. [78]

    Muon is scalable for

    Liu, Jingyuan and Su, Jianlin and Yao, Xingcheng and Jiang, Zhejun and Lai, Guokun and Du, Yulun and Qin, Yidao and Xu, Weixin and Lu, Enzhe and Yan, Junjie and others , journal=. Muon is scalable for

  79. [79]

    Implicit Bias of Spectral Descent and

    Fan, Chen and Schmidt, Mark and Thrampoulidis, Christos , journal=. Implicit Bias of Spectral Descent and

  80. [80]

    2013 , publisher =

    Matrix Analysis , author =. 2013 , publisher =

    2013

Showing first 80 references.