Pith. sign in

REVIEW 3 major objections 5 minor 79 references

MALT: Lightweight Curvature-Aware Muon via Diagonal Preconditioning

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MALT claims that diagonally preconditioning the momentum before Newton-Schulz orthogonalization makes Muon converge to lower GPT-2 validation loss at nearly unchanged cost.

desk verdict A cheap and plausible Muon variant with consistent single-run empirical gains; the stated convergence theorem has a concrete exponent error and doesn't cover the implemented Newton-Schulz approximation. read the letter →

arxiv 2608.05088 v1 pith:I7SSWG62 submitted 2026-08-05 cs.LG

Tongle Wu , Huanyu Dong , Ying Sun , Ziye Ma This is my paper
classification cs.LG
keywords MALTMuondiagonalpreconditioningNewton-Schulzorthogonalizationnormgraftingcurvatureanisotropylanguagemodelpretrainingstochasticnonconvexoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Muon replaces AdamW-style updates for matrix parameters by orthogonalizing a momentum matrix with Newton-Schulz iterations, which flattens gradient anisotropy but leaves curvature anisotropy untouched. MALT (Muon Augmented by Lightweight Two-sided preconditioning) claims to fix that by scaling each matrix parameter's momentum on the left and right with diagonal preconditioners built from row and column squared-gradient norms, then orthogonalizing in the preconditioned space and mapping the direction back. A norm-grafting factor keeps the update magnitude aligned with Muon's, so the extra curvature awareness costs only $O(m+n)$ memory per $m\times n$ matrix. On GPT-2 Small, Medium, and Large pretraining, the paper reports that MALT consistently lowers validation loss relative to Muon, and MALTER, which adds a norm-based Adam-type scalar stepsize, improves further while adding only one scalar per matrix. If true, the central claim is that two-sided diagonal preconditioning and orthogonalization attack different anisotropies and can be combined cheaply.

What carries the argument

Two-sided diagonal preconditioners $L_t=\mathrm{Diag}((l_t+\epsilon)^{-1/8})$ and $R_t=\mathrm{Diag}((r_t+\epsilon)^{-1/8})$, where $l_t$ and $r_t$ are exponential moving averages of row-wise and column-wise squared gradient norms. They act as a tractable curvature surrogate: large accumulated row or column gradient norms suppress movement along those directions, and because the scaling is non-orthogonal it actually changes the Muon direction. The Newton-Schulz iterations approximate the polar factor $\mathrm{Orth}$; the norm-grafting ratio $\lVert O_t\rVert_F/(\lVert D_t\rVert_F+\epsilon)$ sets the step magnitude to match the orthogonalized preconditioned momentum; in MALTER the magnitude is instead set by $\alpha_t$, a norm-based Adam-style signal-to-noise estimate computed from $\lVert L_t G_t R_t\rVert_F^2$.

What would settle it

Substitute the printed $\eta$ from (21) into the balancing identity used to obtain (28): the terms balance only with $n^{-3/4}$, so the derivation as printed fails for any $n\neq 1$. For the approximation gap, train a fixed model under MALT while sweeping the number of Newton-Schulz iterations and computing the polar residual $\lVert O_t^\top O_t - I\rVert_F$; if validation loss moves systematically with the residual, the exact-polar assumption in Theorem 1 is material, and if not, the theorem's gap is immaterial to the empirical claim.

Watch

Extended reading notes

Core claim

Starting from a local quadratic model $f(X+\eta\Delta)\approx f(X)+\eta\langle\nabla f(X),\Delta\rangle+\frac{\eta^2}{2}\langle\Delta,B_t(\Delta)\rangle$ and a positive-definite curvature surrogate $B_t$, MALT defines the update by solving the quadratic in the preconditioned space, applying $\mathrm{Orth}$ there, and mapping back: $\Delta_t = B_t^{-1/2}\,\mathrm{Orth}(B_t^{-1/2}\nabla f)$. The lightweight implementation replaces $B_t$ with diagonal row/column factors $L_t=\mathrm{Diag}((l_t+\epsilon)^{-1/8})$ and $R_t=\mathrm{Diag}((r_t+\epsilon)^{-1/8})$, so the update is $L_t\,\mathrm{Orth}(L_t M_t R_t)\,R_t$ with the magnitude grafted from the orthogonalized momentum. The paper reports MALT validation-loss gains over Muon of about 0.011 on all three GPT-2 sizes, MALTER gains of 0.024, 0.028, and 0.016, and training/validation curves below Muon and AdamW throughout. An identity carries much of the conceptual weight: $\mathrm{Orth}(U^\top G V)=U^\top\mathrm{Orth}(G)V$ for orthogonal $U,V$, so rotating momentum into an orthogonal eigenbasis before orthogonalization cannot change Muon's direction; only a non-orthogonal preconditioner such as MALT's diagonal scaling can. Theorem 1 states a stochastic nonconvex convergence bound for MALT with high-accuracy complexity $O(n^{7/2}L\Delta_{0,1}\sigma^2/(B\delta^4))$, matching the $\delta^{-4}$ dependence of Muon analyses at the cost of a worse polynomial dimension factor.

Load-bearing premise

The convergence theorem is proved for the exact polar factor $\mathrm{Orth}$, while Algorithm 1 runs finitely many Newton-Schulz iterations and no bound on that approximation error is given; moreover the printed step size in (21) carries $n^{3/4}$ where the proof's balancing equality requires $n^{-3/4}$, so the theorem as printed is not derived.

Editorial extensions

If this is right

  • On GPT-2 Small, Medium, and Large, MALT's validation loss is lower than Muon's by roughly 0.011 to 0.013, and MALTER is lower still, with gains over Muon comparable to Muon's own gains over AdamW.
  • For an $m\times n$ matrix, MALT adds only $m+n$ scalars of optimizer state beyond Muon's $mn$ momentum, and measured per-iteration wall-clock time on GPT-2 Medium stays within about 1 percent of Muon.
  • Because $\mathrm{Orth}(U^\top G V)=U^\top\mathrm{Orth}(G)V$, rotating the gradient into a learned orthogonal eigenbasis before orthogonalization cannot change the Muon direction, which makes MALT's cheap non-orthogonal diagonal scaling the actual source of its directional change relative to Muon.
  • Theorem 1 claims a stochastic nonconvex stationarity rate for MALT with iteration complexity $\delta^{-4}$ in the high-accuracy regime, matching Muon-type analyses while incurring a worse polynomial dependence on matrix dimension $n^{7/2}$.
  • The paper interprets MALTER's additional gains as evidence that noise-adaptive scalar stepsizes and curvature-aware preconditioned orthogonalization are complementary, not competing, mechanisms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the equivariance identity suggests a design rule: for any optimizer that orthogonalizes, only non-orthogonal preconditioning can change the update direction, so other cheap non-orthogonal maps, such as block-diagonal, permuted, or Kronecker factors, are natural candidates to test in place of MALT's diagonal pair.
  • Beyond the paper, the preconditioner condition numbers reported in Figure 6, which reach values like $10^8$, are a testable signature of the claimed mechanism; measuring whether layers with larger preconditioner condition numbers show larger MALT-versus-Muon gains would directly probe the curvature-anisotropy story.
  • Beyond the paper, MALTER's norm-based adaptive scalar is applied after grafting and does not depend on the exact orthogonalization subroutine, so transplanting it to other Muon variants, such as row-normalized or low-rank orthogonalization methods, is a natural way to test whether the noise-adaptive component transfers.
  • Beyond the paper, the exact-polar assumption in Theorem 1 is an open gap; bounding the finite Newton-Schulz error in the descent lemma, or measuring the polar residual during real runs, would tell whether Algorithm 1 inherits the stated guarantee or only an approximate version of it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MALT, a Muon-style optimizer that applies lightweight left and right diagonal preconditioners, built from exponential moving averages of row-wise and column-wise squared gradient norms, to the momentum matrix before Newton-Schulz orthogonalization, and then grafts the update magnitude using the Frobenius norm of the orthogonalized matrix. MALTER extends MALT with a norm-based adaptive scalar stepsize. The paper claims a stochastic nonconvex convergence guarantee for MALT and reports GPT-2 Small, Medium, and Large pretraining experiments in which MALT and MALTER achieve lower training and validation losses than Muon while maintaining nearly the same memory and wall-clock time.

Significance. If the convergence theorem were established and the empirical gains were robust, MALT would be a practically appealing addition to the growing family of Muon variants: the O(m+n) extra optimizer state per matrix parameter is negligible, and the paper includes concrete memory and wall-clock measurements on a 355M-parameter model. The idea of combining two-sided diagonal preconditioning with Muon-style orthogonalization is natural and the comparison against Muon at three model scales is a useful data point. The paper also makes a serious attempt at a stochastic nonconvex proof and is transparent about its hyperparameters. However, the theorem as stated is not supported by its proof, and the headline empirical claim rests on single runs without error bars, so the significance is conditional on substantial revision.

major comments (3)
  1. [§4, Eq. (21) and Appendix A] Theorem 1 states the step-size condition as eta = epsilon^{5/8} n^{3/4} sqrt(2 Delta_{0,1} (1-mu_1)/(5 L T)). In Appendix A, Eq. (28) balances 4 epsilon^{-1/4} Delta_{0,1}/(T eta) against 10 epsilon^{-3/2} L n^{3/2} eta/(1-mu_1). Minimizing this sum over eta gives eta* proportional to epsilon^{5/8} n^{-3/4} sqrt(Delta_{0,1}(1-mu_1)/(L T)), with a negative exponent on n. The displayed theorem therefore uses the opposite n dependence from the step size actually required in the proof; as printed, the equality in Eq. (28) does not hold, and the bounds in Eqs. (29)-(31) and (22) do not follow. This must be corrected and propagated through the derived rate.
  2. [Algorithm 1 and Appendix A] The convergence proof uses the exact-polar identity <L_t M_t R_t, Orth(L_t M_t R_t)> = ||L_t M_t R_t||_* in step (i) of Eq. (23), together with the assumption that Orth returns the exact polar factor. Algorithm 1, however, computes Orth with a finite number of Newton-Schulz iterations, and the paper supplies no iteration count, no approximation-error bound, and no perturbation argument showing that the finite-iteration output is close enough to the polar factor for the proof to carry over. As a result, Theorem 1 does not currently cover the algorithm whose experiments are reported. Please either add an explicit bound on the Newton-Schulz approximation error under the preconditioned matrices encountered by MALT, or state Theorem 1 for an idealized exact-Orth variant and provide separate evidence that the approximation error is negligible in practice.
  3. [§5, Table 3 and Table 2] The central empirical claim that MALT and MALTER 'consistently outperform' Muon is based on single training runs without seeds, error bars, or variance estimates. The reported validation-loss improvements over Muon for Small and Medium are about 0.011 and 0.019, respectively, which are small enough to be comparable to run-to-run variation in large-scale pretraining. In addition, MALTER is tuned on a separate learning-rate grid (Table 2), and its optimal learning rates are roughly an order of magnitude larger than those for Muon and MALT; this makes it difficult to separate the effect of the adaptive stepsize from the effect of a different hyperparameter sweep. Please add multiple seeds and error bars for the headline comparisons, and either run the baselines on the same grid as MALTER or explicitly justify why a distinct grid is necessary for a fair comparison.
minor comments (5)
  1. [§3.5] The sentence 'The eighth square root of lt, Rt can make the update too aggressive' is garbled; it should refer to the eighth-root factors L_t and R_t, not to 'the eighth square root of lt, Rt'.
  2. [§5.5] The text 'Compared to Muon, MALT and APDMuon introduce mild additional memory overheads' refers to an undefined optimizer 'APDMuon'; this appears to be a leftover and should read 'MALT and MALTER'.
  3. [Throughout] The spelling of the orthogonalization method is inconsistent: 'NewtonSchulz' appears in the abstract and elsewhere, while 'Newton-Schulz' appears in other places; please unify the spelling.
  4. [Figure 1 and Appendix B] The motivating experiment in Figure 1(b) uses exact-Hessian preconditioners P=(A^T A)^{-1/2}, Q=(B^T B)^{-1/2} as stated in Eq. (35), not the diagonal row/column preconditioners used by MALT. The caption and text should make clear that this is a proof of concept for curvature-aware preconditioning in general, and does not by itself demonstrate that MALT's specific diagonal preconditioner captures curvature.
  5. [§4, Eq. (22)] The stationarity measure in Theorem 1 is a minimum of E[||nabla f||_F^{1/2}] and a normalized E^2[||nabla f||_F^{1/2}] term, which is non-standard; after fixing the step-size error, please add a sentence explaining how this measure translates into a standard gradient-norm complexity bound.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the convergence theorem is a self-contained derivation under stated assumptions, and the empirical comparisons tune learning rates for all optimizers rather than fitting any prediction to the reported validation-loss targets.

full rationale

I walked the claimed derivation chain: the motivation in Section 3.2 derives a preconditioned Muon update from a local quadratic surrogate, Algorithm 1 defines MALT concretely, Theorem 1 in Section 4 states a stochastic nonconvex bound, and Appendix A supplies the proof. No step reduces by construction to its own input. The preconditioners L_t and R_t are computed from row and column squared-gradient moving averages; they are not fitted to validation loss or to the convergence metric. Norm grafting in Algorithm 1 uses the Frobenius norm of O_t to set the update magnitude, but this is an explicit design rule, not a disguised prediction of the experiments. The convergence proof uses only Assumptions 1-3 together with inequalities; it does not fit constants to observed data. The empirical section tunes learning rates on a grid for every optimizer, which is standard practice and does not make the comparison circular. The paper contains no load-bearing self-citation: the author list does not overlap with the cited proofs or prior optimizer work, and the proof of Lemma 1 follows the framework of reference [61] but is written out in the appendix. Two reviewer concerns belong to correctness, not circularity: the displayed step size in Eq. (21) has n^{3/4} while the balancing equality in Eq. (28) requires n^{-3/4}, and the proof assumes an exact polar factor although Algorithm 1 runs finite Newton-Schulz iterations with no error bound. These are unsupported-theorem issues, not cases where an output is equivalent to an input by definition. The curvature-aware interpretation of row/column gradient statistics is a modeling assumption rather than a derived fact, and asserting a motivation is not circularity. Overall, the paper's central claims have independent content and are externally testable.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests on hand-chosen optimizer hyperparameters and tuned learning rates, plus the assumption that row and column gradient EMAs approximate curvature. The convergence theorem rests on standard smoothness and variance assumptions and on exact polar orthogonalization, while the algorithm uses Newton-Schulz iterations. No new physical entities are introduced.

free parameters (4)
  • Learning rate per model and optimizer = Small: MALT 0.0013, MALTER 0.012; Medium: MALT 0.0009, MALTER 0.009; Large: MALT 0.0007, MALTER 0.006
    Selected by grid search on 10K-step validation loss (Tables 1 and 2); the relative ranking of optimizers depends on this tuning.
  • Second-moment EMA decay mu_2 for preconditioners = 0.99
    Hand-chosen in Table 6; controls how fast row and column gradient statistics adapt.
  • First-moment momentum mu_1 = 0.95 for Muon, MALT, and MALTER
    Hand-chosen in Table 6; affects momentum noise and update scale.
  • Damping constant epsilon = 1e-8
    Hand-chosen for numerical stability; appears in the preconditioners and in the convergence bound of Theorem 1.
assumptions (4)
  • standard math Assumptions 1-3: f is bounded below, L-smooth, and stochastic gradients are unbiased with bounded variance.
    Stated in Section 4 before Theorem 1; standard for stochastic nonconvex optimization.
  • domain assumption The proof assumes exact polar orthogonalization, while Algorithm 1 uses Newton-Schulz iterations.
    Appendix A uses <A, Orth(A)> = nuclear norm and orthogonal equivariance, which hold only for the exact polar factor; the approximation error is not bounded.
  • ad hoc to paper Row and column EMA squared gradient norms serve as a curvature surrogate.
    Section 3.3 asserts this approximation and explicitly notes that the preconditioners do not estimate the Hessian; no derivation connects these statistics to curvature.
  • domain assumption The curvature surrogate B_t is positive definite and has a square root.
    Required for the preconditioned-space derivation in Section 3.2; heuristically justified by prior adaptive and Shampoo-style optimizers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MALT: Lightweight Curvature-Aware Muon via Diagonal Preconditioning." pith.science (2026). https://pith.science/paper/I7SSWG62

@misc{pith2026260805088,
  author       = {Pith},
  title        = {Pith review of: MALT: Lightweight Curvature-Aware Muon via Diagonal Preconditioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I7SSWG62}},
  note         = {Machine review of arXiv:2608.05088}
}
read the original abstract

Muon has recently emerged as a promising alternative to AdamW for language model pretraining by orthogonalizing momentum matrices using Newton-Schulz iterations. Although Muon mitigates gradient anisotropy, it does not explicitly account for the curvature geometry of the loss landscape and may therefore remain sensitive to curvature anisotropy. We bridge this gap by proposing MALT (Muon Augmented by Lightweight Two-sided Preconditioning), which uses lightweight diagonal preconditioners to reduce the sensitivity of Muon to curvature anisotropy. Specifically, MALT uses two-sided diagonal preconditioners with low memory and computational overhead to approximately capture the curvature geometry of the loss landscape. It orthogonalizes the preconditioned momentum using Newton-Schulz iterations and maps the result back to define the update direction, while norm grafting controls the update magnitude. To improve the robustness of MALT to stochastic gradient noise, we further propose MALTER (MALT with Adaptive stEpsize Rescaling). Convergence guarantees are provided for MALT in the stochastic non-convex setting. Experiments on GPT-2 Small, Medium, and Large pretraining show that the proposed methods outperform Muon while maintaining nearly the same memory footprint and wall-clock time.

Figures

Figures reproduced from arXiv: 2608.05088 by the authors.

Figure 1
Figure 1. Comparison of Muon with and without curvature-aware preconditioning under the same [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. GPT-2 small (124M) pretraining. The small model is trained under learning rates [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. GPT-2 medium (355 M) pretraining. The medium model is trained under learning rates [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: GPT-2 large (774 M) pretraining. The large model is trained at learning rates reported [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Final validation loss of pretraining GPT-2 Small, Medium and Large for 10K steps. The [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Condition numbers of the diagonal preconditioners in MALT for representative trans [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Additional step size settings for Muon-style updates. (a) Nuclear-norm scaled polar [PITH_FULL_IMAGE:figures/full_fig_p029_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

79 extracted references · 40 canonical work pages

  1. [1]

    Disentangling adaptive gradient methods from learning rates

    Naman Agarwal, Rohan Anil, Elad Hazan, Tomer Koren, and Cyril Zhang. Disentangling adaptive gradient methods from learning rates. arXiv preprint arXiv:2002.11803 , 2020

    arXiv 2002

  2. [2]

    Natural gradient works efficiently in learning

    Shun-ichi Amari. Natural gradient works efficiently in learning. Neural Computation , 10(2):251–276, 1998

    work page 1998

  3. [3]

    Noah Amsel, David Persson, Christopher Musco, and Robert M. Gower. The polar express: Optimal matrix sign methods and their application to the muon algorithm. arXiv preprint arXiv:2505.16932, 2025

    arXiv 2025

  4. [4]

    Scalable second order optimization for deep learning

    Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018 , 2020

    arXiv 2002

  5. [5]

    Memory efficient adaptive optimization

    Rohan Anil, Vineet Gupta, Tomer Koren, and Yoram Singer. Memory efficient adaptive optimization. In Advances in Neural Information Processing Systems , 2019

    work page 2019

  6. [6]

    High- dimensional sgd aligns with emerging outlier eigenspaces

    Gerard Ben Arous, Reza Gheissari, Jiaoyang Huang, and Aukosh Jagannath. High- dimensional sgd aligns with emerging outlier eigenspaces. In International Conference on Learning Representations, volume 2024, pages 47732–47778, 2024. 18

    work page 2024

  7. [7]

    signsgd: Compressed optimisation for non-convex problems

    Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Anima Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning , pages 560–569. PMLR, 2018

    work page 2018

  8. [8]

    Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S

    Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021

    arXiv 2021

Show all 79 references
  1. [9]

    Practical gauss-newton optimisation for deep learning

    Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical gauss-newton optimisation for deep learning. In International Conference on Machine Learning , pages 557–565. PMLR, 2017

    2017

  2. [10]

    Muoneq: Balancing before orthogonalization with lightweight equilibration

    Da Chang, Qiankun Shi, Lvgang Zhang, Yu Li, Ruijie Zhang, Yao Lu, Yongxiang Liu, and Ganzhao Yuan. Muoneq: Balancing before orthogonalization with lightweight equilibration. arXiv preprint arXiv:2603.28254 , 2026

    2026 arXiv

  3. [11]

    Trasmuon: Trust-region adaptive scaling for orthogonalized momentum optimizers

    Peng Cheng, Jiucheng Zang, Qingnan Li, Liheng Ma, Yufei Cui, Yingxue Zhang, Boxing Chen, Ming Jian, and Wen Tong. Trasmuon: Trust-region adaptive scaling for orthogonalized momentum optimizers. arXiv preprint arXiv:2602.13498 , 2026

    2026

  4. [12]

    Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio

    Yann N. Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non- convex optimization. In Advances in Neural Information Processing Systems , volume 27, 2014

    2014

  5. [13]

    DeepSeek-V3.2: Pushing the frontier of open large language models

    DeepSeek-AI. DeepSeek-V3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556 , 2025

    2025 arXiv

  6. [14]

    Rmnp: Row-momentum normalized preconditioning for scalable matrix-based optimization

    Shenyang Deng, Zhuoli Ouyang, Tianyu Pang, Zihang Liu, Ruochen Jin, Shuhua Yu, and Yaoqing Yang. Rmnp: Row-momentum normalized preconditioning for scalable matrix-based optimization. arXiv preprint arXiv:2603.20527 , 2026

    2026 arXiv

  7. [15]

    Aurora: A leverage-aware optimizer for rectangular matrices, 2026

    Alec Dewulf, Dhruv Pai, Li Yang, Ashley Zhang, and Ben Keigwin. Aurora: A leverage-aware optimizer for rectangular matrices, 2026. URL https://blog.tilderesearch.com/blog/aurora, 2026

    2026

  8. [16]

    Towards quantifying the hessian structure of neural networks

    Zhaorui Dong, Yushun Zhang, Jianfeng Yao, and Ruoyu Sun. Towards quantifying the hessian structure of neural networks. arXiv preprint arXiv:2505.02809 , 2025

    2025

  9. [17]

    The newton-muon optimizer

    Zhehang Du and Weijie Su. The newton-muon optimizer. arXiv preprint arXiv:2604.01472 , 2026

    2026

  10. [18]

    Adaptive subgradient methods for online learning and stochastic optimization

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research , 12(61):2121–2159, 2011

    2011

  11. [19]

    Sai Surya Duvvuri, Fnu Devvrit, Rohan Anil, Cho-Jui Hsieh, and Inderjit S. Dhillon. Combin- ing axes preconditioners through kronecker approximation for deep learning. In International Conference on Learning Representations , 2024

    2024

  12. [20]

    What really matters in matrix-whitening optimizers? arXiv preprint arXiv:2510.25000 , 2025

    Kevin Frans, Pieter Abbeel, and Sergey Levine. What really matters in matrix-whitening optimizers? arXiv preprint arXiv:2510.25000 , 2025. 19

    2025

  13. [21]

    A stable whitening optimizer for efficient neural network training

    Kevin Frans, Sergey Levine, and Pieter Abbeel. A stable whitening optimizer for efficient neural network training. Advances in Neural Information Processing Systems , 38:174086– 174110, 2026

    2026

  14. [22]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodal- ity, long context, and next generation agentic capabilities

    Gemini Team, Google. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodal- ity, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 , 2025

    2025 arXiv

  15. [23]

    An investigation into neural net opti- mization via hessian eigenvalue density

    Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net opti- mization via hessian eigenvalue density. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 2232–

  16. [24]

    Openwebtext corpus, 2019

    Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus, 2019

    2019

  17. [25]

    Accelerating newton-schulz itera- tion for orthogonalization via chebyshev-type polynomials

    Ekaterina Grishina, Matvey Smirnov, and Maxim Rakhuba. Accelerating newton-schulz itera- tion for orthogonalization via chebyshev-type polynomials. arXiv preprint arXiv:2506.10935 , 2025

    2025

  18. [26]

    Shampoo: Preconditioned stochastic tensor optimization

    Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1842–1850. PMLR, 2018

    2018

  19. [27]

    Low-rank orthogonalization for large- scale matrix optimization with applications to foundation model training

    Chuan He, Zhanwang Deng, and Zhaosong Lu. Low-rank orthogonalization for large- scale matrix optimization with applications to foundation model training. arXiv preprint arXiv:2509.11983, 2025

    2025 arXiv

  20. [28]

    Spectra: Rethinking optimizers for llms under spectral anisotropy

    Zhendong Huang, Hengjie Cao, Fang Dong, Ruijun Huang, Mengyi Chen, Yifeng Yang, Xin Zhang, Anrui Chen, Mingzhi Dong, Yujiang Wang, et al. Spectra: Rethinking optimizers for llms under spectral anisotropy. arXiv preprint arXiv:2602.11185 , 2026

    2026

  21. [29]

    Richter, Quentin Anthony, Timo- thée Lesort, Eugene Belilovsky, and Irina Rish

    Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L. Richter, Quentin Anthony, Timo- thée Lesort, Eugene Belilovsky, and Irina Rish. Simple and scalable strategies to continually pre-train large language models. In International Conference on Learning Representations , 2024

    2024

  22. [30]

    Muon: An optimizer for hidden layers in neural networks, 2024

    Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan. github. io/posts/muon , 6(3):4, 2024

    2024

  23. [31]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020

    2001 arXiv

  24. [32]

    Andrej Karpathy. nanogpt. https://github.com/karpathy/nanoGPT, 2022. GitHub repos- itory

    2022

  25. [33]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Inter- national Conference on Learning Representations , 2015. 20

    2015

  26. [34]

    Polargrad: A class of matrix-gradient optimizers from a unifying preconditioning perspective

    Tim Tsz-Kit Lau, Qi Long, and Weijie Su. Polargrad: A class of matrix-gradient optimizers from a unifying preconditioning perspective. arXiv preprint arXiv:2505.21799 , 2025

    2025

  27. [35]

    Variance-adaptive muon: Accelerating llm pretraining with nsr-modulated and variance-scaled momentum

    Jingru Li, Yibo Fan, and Huan Li. Variance-adaptive muon: Accelerating llm pretraining with nsr-modulated and variance-scaled momentum. arXiv preprint arXiv:2601.14603 , 2026

    2026

  28. [36]

    Normuon: Making muon more efficient and scalable

    Zichong Li, Liming Liu, Chen Liang, Weizhu Chen, and Tuo Zhao. Normuon: Making muon more efficient and scalable. arXiv preprint arXiv:2510.05491 , 2025

    2025

  29. [37]

    Muown: Row-norm control for muon optimization

    Kai Lion, Florian Hübler, Bingcong Li, Antonio Orvieto, and Niao He. Muown: Row-norm control for muon optimization. arXiv preprint arXiv:2605.10797 , 2026

    2026 arXiv

  30. [38]

    Sophia: A scalable stochas- tic second-order optimizer for language model pre-training

    Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochas- tic second-order optimizer for language model pre-training. In International Conference on Learning Representations, 2024

    2024

  31. [39]

    Muon is scalable for llm training

    Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for llm training. arXiv preprint arXiv:2502.16982, 2025

    2025 arXiv

  32. [40]

    Muon 2: Boosting muon via adaptive second-moment preconditioning

    Ziyue Liu, Ruijie Zhang, Zhengyang Wang, Yequan Zhao, Yupeng Su, Zi Yang, and Zheng Zhang. Muon 2: Boosting muon via adaptive second-moment preconditioning. arXiv preprint arXiv:2604.09967, 2026

    2026 arXiv

  33. [41]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019

    2019

  34. [42]

    Deep learning via hessian-free optimization

    James Martens. Deep learning via hessian-free optimization. In Proceedings of the 27th International Conference on Machine Learning , pages 735–742, 2010

    2010

  35. [43]

    Optimizing neural networks with kronecker-factored ap- proximate curvature

    James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored ap- proximate curvature. In Proceedings of the 32nd International Conference on Machine Learn- ing, volume 37 of Proceedings of Machine Learning Research, pages 2408–2417. PMLR, 2015

    2015

  36. [44]

    The Llama 4 herd: The beginning of a new era of natively multimodal ai innovation

    Meta AI. The Llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https://ai.meta.com/blog/llama-4-multimodal-intelligence/ , 2025. Accessed: 2026- 06-07

    2025

  37. [45]

    Large language models: A survey

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey. arXiv preprint arXiv:2402.06196, 2024

    2024 arXiv

  38. [46]

    Improved convergence rates of muon optimizer for nonconvex optimization

    Shuntaro Nagashima and Hideaki Iiduka. Improved convergence rates of muon optimizer for nonconvex optimization. arXiv preprint arXiv:2601.19400 , 2026

    2026

  39. [47]

    Efficient large-scale language model train- ing on GPU clusters using Megatron-LM

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model train- ing on GPU clus...

    2021 arXiv

  40. [48]

    Lectures on Convex Optimization

    Yurii Nesterov. Lectures on Convex Optimization . Springer, 2018. 21

    2018

  41. [49]

    Spectral flattening is all muon needs: How orthogonalization controls learning rate and convergence

    Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong, Tuc Nguyen, James Bailey, and Trung Le. Spectral flattening is all muon needs: How orthogonalization controls learning rate and convergence. arXiv preprint arXiv:2605.13079 , 2026

    2026 arXiv

  42. [50]

    Jorge Nocedal and Stephen J. Wright. Numerical Optimization . Springer, 2 edition, 2006

    2006

  43. [51]

    GPT-5.5 System Card

    OpenAI. GPT-5.5 System Card. https://openai.com/index/gpt-5-5-system-card/ ,

  44. [52]

    Carbon emissions and large neural network training

    David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350 , 2021

    2021 arXiv

  45. [53]

    ZeRO: Memory op- timizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory op- timizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , 2020

    2020

  46. [54]

    Navigating LLM valley: From AdamW to memory-efficient and matrix- based optimizers

    Aditya Ranganath. Navigating LLM valley: From AdamW to memory-efficient and matrix- based optimizers. arXiv preprint arXiv:2605.09176 , 2026

    2026 arXiv

  47. [55]

    Sumo: Subspace- aware moment-orthogonalization for accelerating memory-efficient llm training

    Yehonathan Refael, Guy Smorodinsky, Tom Tirer, and Ofir Lindenbaum. Sumo: Subspace- aware moment-orthogonalization for accelerating memory-efficient llm training. Advances in Neural Information Processing Systems , 38:147250–147281, 2026

    2026

  48. [56]

    Eigenvalues of the hessian in deep learning: Singularity and beyond

    Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the hessian in deep learning: Singularity and beyond. arXiv preprint arXiv:1611.07476 , 2016

    2016 arXiv

  49. [57]

    Ugur Guney, Yann Dauphin, and Leon Bottou

    Levent Sagun, Utku Evci, V. Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over-parametrized neural networks. arXiv preprint arXiv:1706.04454, 2017

    2017 arXiv

  50. [58]

    Uğur Güney, Yann Dauphin, and Léon Bottou

    Levent Sagun, Utku Evci, V. Uğur Güney, Yann Dauphin, and Léon Bottou. Empirical analysis of the hessian of over-parametrized neural networks. In International Conference on Learning Representations Workshop, 2018

    2018

  51. [59]

    Schraudolph

    Nicol N. Schraudolph. Fast curvature matrix-vector products for second-order gradient de- scent. Neural Computation , 14(7):1723–1738, 2002

    2002

  52. [60]

    Adafactor: Adaptive learning rates with sublinear memory cost

    Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International conference on machine learning , pages 4596–4604. PMLR, 2018

    2018

  53. [61]

    On the conver- gence analysis of muon

    Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, and Jiawei Zhang. On the conver- gence analysis of muon. arXiv preprint arXiv:2505.23737 , 2025

    2025 arXiv

  54. [62]

    Adamuon: Adaptive muon optimizer

    Chongjie Si, Debing Zhang, and Wei Shen. Adamuon: Adaptive muon optimizer. arXiv preprint arXiv:2507.11005 , 2025

    2025

  55. [63]

    Decoupling variance and scale-invariant updates in adaptive gradient descent for unified vector and matrix optimization

    Zitao Song, Cedar Site Bai, Zhe Zhang, Brian Bullins, and David F Gleich. Decoupling variance and scale-invariant updates in adaptive gradient descent for unified vector and matrix optimization. arXiv preprint arXiv:2602.06880 , 2026. 22

    2026 arXiv

  56. [64]

    Q. Y. Tang et al. Investigating the overlooked hessian structure. In International Conference on Machine Learning , 2025

    2025

  57. [65]

    Lecture 6.5—rmsprop: Divide the gradient by a running average of its recent magnitude

    Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5—rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012

    2012

  58. [66]

    Nikhil Vyas, Depen Morwani, Rosie Zhao, Mujin Kwun, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham M. Kakade. Soap: Improving and stabilizing shampoo using adam for language modeling. In International Conference on Learning Representations , 2025

    2025

  59. [67]

    Under- standing warmup-stable-decay learning rates: A river valley loss landscape perspective

    Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, and Tengyu Ma. Under- standing warmup-stable-decay learning rates: A river valley loss landscape perspective. In International Conference on Learning Representations , 2025

    2025

  60. [68]

    Fismo: Fisher-structured momentum-orthogonalized optimizer

    Chenrui Xu, Wenjing Yan, and Ying-Jun Angela Zhang. Fismo: Fisher-structured momentum-orthogonalized optimizer. arXiv preprint arXiv:2601.21750 , 2026

    2026

  61. [69]

    Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and Michael W. Mahoney. Adahessian: An adaptive second order optimizer for machine learning. In Pro- ceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 10665–10673, 2021

    2021

  62. [70]

    Preconditioning for accelerated gradient descent optimization and regularization

    Qiang Ye. Preconditioning for accelerated gradient descent optimization and regularization. arXiv preprint arXiv:2410.00232 , 2024

    2024

  63. [71]

    Nora: Normalized orthogonal row alignment for scalable matrix optimizer

    Jinghui Yuan, Jiaxuan Zou, Shuo Wang, Yong Liu, and Feiping Nie. Nora: Normalized orthogonal row alignment for scalable matrix optimizer. arXiv preprint arXiv:2605.03769 , 2026

    2026 arXiv

  64. [72]

    Gram newton–schulz: A fast, hardware-aware newton–schulz algorithm for muon

    Jack Zhang, Noah Amsel, Berlin Chen, and Tri Dao. Gram newton–schulz: A fast, hardware-aware newton–schulz algorithm for muon. https://tridao.me/blog/2026/ gram-newton-schulz/, 2026

    2026

  65. [73]

    Adam improves muon: Adaptive moment estimation with orthogonalized momentum

    Minxin Zhang, Yuxuan Liu, and Hayden Schaeffer. Adam improves muon: Adaptive moment estimation with orthogonalized momentum. arXiv preprint arXiv:2602.17080 , 2026

    2026

  66. [74]

    Muon+: To- wards better muon via one additional normalization step

    Ruijie Zhang, Yequan Zhao, Ziyue Liu, Zhengyang Wang, and Zheng Zhang. Muon+: To- wards better muon via one additional normalization step. arXiv preprint arXiv:2602.21545 , 2026

    2026 arXiv

  67. [75]

    Mousse: Rectifying the geometry of muon with curvature-aware precon- ditioning

    Yechen Zhang, Shuhao Xing, Junhao Huang, Kai Lv, Yunhua Zhou, Xipeng Qiu, Qipeng Guo, and Kai Chen. Mousse: Rectifying the geometry of muon with curvature-aware precon- ditioning. arXiv preprint arXiv:2603.09697 , 2026

    2026

  68. [76]

    Why transformers need adam: A hessian perspective

    Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhi-Quan Luo. Why transformers need adam: A hessian perspective. Advances in neural information processing systems, 37:131786–131823, 2024

    2024

  69. [77]

    Kingma, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun

    Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Diederik P. Kingma, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates to gain more. In International Conference on Learning Representations , 2025. 23

    2025

  70. [78]

    A survey of large language models

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223 , 2023. 24 A Proof of Theorem 1 Based on Assumption 2, there is E [f (W t...

    2023 arXiv

  71. [2026]

    Accessed: 2026-06-07

    2026

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.