Pith. sign in

REVIEW 3 major objections 5 minor 52 references

For the Muon optimizer, the shape you fold each parameter block into before orthogonalizing it is part of the geometry: shorter shapes lose singular channels, while a square reshape matches the native layout in pretraining.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 01:49 UTC pith:TTWLJTCI

load-bearing objection The representation axis is a real theoretical contribution, but the main empirical claim is undercut by an effective-learning-rate confound the paper never addresses. the 3 major comments →

arxiv 2607.14536 v1 pith:TTWLJTCI submitted 2026-07-16 cs.LG

Muse: Representation Geometry of Muon Beyond Normalized Momentum

Da Chang , Qiankun Shi , Lvgang Zhang , Di He , Yaoshuai Ma , Ganzhao Yuan , Yongxiang Liu This is my paper
classification cs.LG
keywords Muon optimizerpolar mapmatrix representation geometryshort-side ranknuclear norm supportMarchenko-Pastur profileoptimizer geometryLLM pretraining
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Muon-style optimizers apply a polar (orthogonal-factor) map to matrix-valued momentum, but the same parameter vector can be folded into many matrix shapes before that map. This paper argues that the representation choice is a real design axis: each Frobenius-isometric reshape induces a distinct steepest-descent direction over a spectral ball, and the shorter side of the reshaped matrix controls the update rank, nuclear-norm support, pulled-back scaling, and convergence constants. A teacher–student analysis with collapsed curvature and a random-matrix spectral profile predicts that balanced shapes such as nearest-square match the native layout, while skinnier shapes degrade toward normalized SGD with momentum. Pretraining LLaMA2-130M and LLaMA2-600M confirm the ordering: nearest-square sits within 0.003 validation loss of native, the skinny ladder falls off, and the vector endpoint is clearly worse. The claim matters because it locates Muon's advantage in multi-channel nuclear support rather than in coordinate-wise adaptivity, and it makes reshape choice a practical accuracy–cost knob.

Core claim

The central claim is that the matrixization supplied to Muon's polar map is an integral component of its geometry. For any Frobenius-isometric reshaping T, the pulled-back polar direction D_T(G) = T^{-1}(Orth(T(G))) satisfies the steepest-descent identity ⟨G, D_T(G)⟩ = ‖T(G)‖_* over the spectral ball, so different reshapes of the same block yield different update directions. The short side p_T bounds the rank of the polar update, the nuclear support, the pullback's Frobenius norm, and the constants in a stochastic nonconvex stationarity guarantee of order Õ(K^{-1/4}). Under a local curvature-collapse model in which the ReLU teacher–student Hessian becomes a scalar Frobenius quadratic and the

What carries the argument

The central object is the family of Frobenius-isometric flat representations T_p: R^d → R^{p × d/p}, ranging from the native shape through a nearest-square reshape, skinny-θ reshapes, down to the 1×d vector endpoint. The polar map Orth(A)=UV^T (the orthogonal factor from the SVD) is applied to T_p(M_t), and the direction is pulled back by T_p^{-1}. The designed identity is ⟨G, D_T(G)⟩ = ‖T(G)‖_*, which makes each reshape a distinct steepest-descent geometry under the spectral/nuclear norm, with the short side p_T acting as the channel budget. In the dissipation analysis, the scalar shape factor β(γ) γ^{1/4} — the MP-mean of the singular values times the fourth root of the aspect ratio — orde

Load-bearing premise

The quantitative dissipation ordering assumes that the singular values of the represented perturbation follow a Marchenko–Pastur distribution with the corresponding aspect ratio; if real optimizer-state spectra deviate from that isotropic model, the predicted shape-factor curve and learning-rate scalings can fail even though the rank-bound ordering is forced.

What would settle it

Train a Transformer twice at fixed budget, once with a square reshape and once with a vector reshape, and measure whether the vector's best validation loss can beat or match the square's; the theory says the square's Θ(d^{1/4}) nuclear-support advantage should make that impossible at any learning rate. More directly, compute the empirical singular-value profile of saved momentum blocks and re-derive the dissipation ordering with β(γ) replaced by the measured nuclear-to-Frobenius ratio; if the observed ranking of representations reverses, the MP-based mechanism is wrong.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Nearest-square reshaping matches native Muon in LLaMA2 pretraining (600M: 2.671 vs 2.668 validation loss), so the native row–column layout is not required for Muon's gains.
  • Reducing the temporary short side degrades performance monotonically toward normalized SGD with momentum (600M: Skinny-25 2.710, Vector 3.080), consistent with loss of singular-channel support.
  • All fixed representations share the same Õ(K^{-1/4}) stochastic stationarity rate, with worst-case constants controlled only by the short side p_T; same-short-side representations have identical bounds.
  • Under the curvature-collapse/MP model, the square shape's leading normalized dissipation exceeds the vector's by Θ(d^{1/4}), which predicts a large-learning-rate separation between matrix and vector endpoints.
  • The representation family also modulates Newton–Schulz cost: square reshapes cost more than native (relative NS cost 1.5–1.75×), while skinnier and vector reshapes cost far less, making representation a compute–accuracy trade-off axis.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the short side alone controls worst-case constants, reshaping different modules to different short sides (e.g., skinny for attention, square for MLP) may recover most of Muon's gain at lower orthogonalization cost — a per-module schema the paper does not sweep.
  • The MP-based ordering is an isotropic-model prediction; on real momentum states, one could replace the theoretical β(γ) with the measured nuclear-to-Frobenius ratio per module and test whether the resulting ranking predicts the observed pretraining loss ordering more sharply than the shape ladder does.
  • Since the vector endpoint provably collapses the polar update to one singular channel, Muon's empirical separation from normalized SGD may be largely explained as nuclear-support preservation; this suggests hybrid interpolations (e.g., keeping only the top-k channels of the polar factor) as a cheap approximation with a controlled loss of geometry.
  • The theory fixes the learning-rate rule η_T ∼ τ_d σ_0 √q_T / β(γ_T); a direct transfer test would be to train the same model with two representations whose β(γ)γ^{1/4} values predict a specific LR ratio and check that the matched-loss operating points satisfy it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies how the choice of matrix representation (native, nearest-square, skinny, vector) supplied to the polar/orthogonalization step of Muon-style optimizers affects the resulting update geometry. It proves that any Frobenius-isometric representation induces a polar steepest-descent direction in the represented spectral norm (Theorem 3.1), that the represented short side p_T controls the rank, norm, and stationarity constants in a stochastic nonconvex bound (Theorem 3.2), and that, under a curvature-collapse teacher–student model and a Marchenko–Pastur spectral assumption, the early normalized dissipation scales with β(γ_T)γ_T^{1/4} (Theorem 3.5, Corollary 3.6). Empirically, LLaMA2-130M and LLaMA2-600M pretraining sweeps show Native and Square achieving the best validation loss, with progressively skinnier representations degrading toward normalized SGD with momentum.

Significance. If the central claim holds, the paper establishes a genuinely new design axis for polar-type optimizers: the matrixization passed to the polar map is not a harmless relabeling but determines rank support, update norm, and learning-rate calibration. The theoretical core is largely sound: the spectral-duality argument for Theorem 3.1 is clean, the short-side dependence in Theorem 3.2 is correctly derived through the rank bound on the polar direction, and the MP calculation leading to β(1)=8/(3π) and the Θ(d^{1/4}) square-to-vector separation is internally consistent. The paper also provides falsifiable predictions — the shape-factor curve and the representation-dependent LR scaling — and includes detailed appendices with proofs, which is a strength. However, the empirical confirmation of the central ladder is currently vulnerable to a learning-rate multiplier confound, and the quantitative spectral prediction is validated only indirectly on momentum blocks rather than the perturbation-from-initialization object analyzed by the theory. These issues are fixable, but they are load-bearing for the paper's empirical conclusions.

major comments (3)
  1. [Appendix G / Section 4.1, Tables 2–3] If the multiplier is computed from the represented shape, the reported LR grids cover only a tiny fraction of the range predicted by Corollary 3.6; the resulting loss ladder does not distinguish the geometric mechanism from a step-size mismatch. Please clarify and, if necessary, rerun or add a matched-effective-LR analysis.
  2. [Section 3.3, Assumption 3.3; Figure 1(b); Appendix H] This concern does not invalidate Theorem 3.5 as a theoretical statement; it affects how much empirical weight the quantitative MP prediction should carry.
  3. [Section 4.1 / Figure 4] This is secondary if the LR confound is resolved, but it is needed to support the strength of the empirical ordering.
minor comments (5)
  1. [Figure 1(b)] The legend uses small background points for actual γ_T and connected markers for weighted averages, but the weighting is not defined in the caption. Please state how the averages are weighted (e.g., by parameter count or nuclear mass).
  2. [Section 3.3, Eq. after Theorem 3.5] The notation 'g(γ)=β(γ)γ^{1/4}' is used in Figure 1(b) but introduced in the text only as 'β(γ_T)γ_T^{1/4}'. Define g(γ) explicitly at first use to make the figure self-contained.
  3. [Appendix G, 'standard Muon multiplier'] Even if the multiplier is native-based in the experiments, the paper should state this explicitly in the protocol, since a reader cannot infer it from Algorithm 1, which uses a single scalar η_t.
  4. [Section 2.2] The definition of 'skinny-θ' via argmin_{a|d, a≤s} |a−θs| can select different actual fractions for different divisors; reporting the nearest divisor would help reproducibility. Table 1 gives representative shapes, but the exact chosen short sides are not listed for every block.
  5. [Abstract / Introduction] The phrase 'beyond normalized momentum' is evocative, but it may mislead: the vector endpoint is exactly nSGDM, so Muon is shown to contain nSGDM as a special case, not to go 'beyond' it in all aspects. Consider rephrasing to 'from normalized momentum to multi-channel polar geometry'.

Circularity Check

0 steps flagged

No significant circularity: the representation geometry is proven self-contained, and the empirical loss ladder is measured after independent per-representation learning-rate tuning.

full rationale

The paper's central derivation is not circular. Theorems 3.1 and 3.2 are proven in the appendix directly from Frobenius isometry, polar-factor identities, and standard smoothness/oracle assumptions; the representation enters only through the defined pulled-back direction D_T and the short side p_T, and no fitted parameter is later relabeled as a prediction. The dissipation ordering in Theorem 3.5 is a conditional statement: under the curvature-collapse theorem and the explicitly stated isotropic Marchenko–Pastur baseline (Assumption 3.3), the represented nuclear-to-squared-Frobenius ratio evaluates to β(γ)γ^{1/4}/(σ0 d^{1/4}). The MP profile is an assumption, not a fit to LLaMA losses, and Section 6 explicitly says it is only an isotropic baseline and that the result depends on the represented spectral quantity itself. Figure 1(b) compares the theory's g(γ) with a separately defined empirical shape factor measured from saved momentum blocks; this is an external moment check that could fail, not a tautology. The pretraining ladder is obtained by sweeping and selecting a learning rate for each representation (Tables 2–3), so the loss ordering is observed, not computed from the theory. Self-citations appear only in contextual statements about shared backends and related convergence analyses; they are not invoked as a load-bearing uniqueness theorem or as the sole justification of the central claim. The possible ambiguity about whether the standard Muon multiplier 0.2/√max(m,n) uses the native or represented shape is a correctness/confound risk, but the paper does not specify the represented-shape reading, and without a quoted reduction it cannot be scored as circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The central claim rests on standard duality and MP results plus two paper-specific postulates (MP spectral profile of represented state; fixed-mask curvature-collapse conditions). The former is partially tested; the latter's failure at depth is disclosed. The theory contributes a derived quantity (shape factor) without fitting constants to training loss; the only fitted knobs in the empirical chain are per-method learning rates. No new physical or mathematical entities are postulated.

free parameters (2)
  • per-method learning rate = 130M: N 0.01, Sq 0.0075, V 0.015; 600M: N 0.0015, Sq 0.0015, Sk-75/50/25 0.001 (grid edge), V 0.003
    Each representation's LR is chosen via a per-method grid sweep minimizing 3-seed final validation loss (Tables 2–3). Extreme representations select at grid edges, so part of the loss ladder is confounded with tuning completeness.
  • σ_0 (initial perturbation scale) = measured from saved teacher–student initialization; cancels in leading ratios
    Assumption 3.3 sets the scale of T(E_0)'s singular values; it is measured from the initialization, not fit, and enters the final gain β(γ)γ^{1/4}/(σ_0 d^{1/4}).
axioms (5)
  • ad hoc to paper Assumption 3.3: normalized singular values of the represented perturbation satisfy the Marchenko–Pastur profile with mean β(γ_T) and unit second moment
    The paper's own spectral-model postulate; the dissipation ordering β(γ)γ^{1/4} is computed from it. Tested only at trend level in Figure 1(b) on saved momentum blocks.
  • ad hoc to paper Assumptions 3.1–3.2: nonzero hidden rows over a fixed radius plus strong incoherence/moment conditions on teacher-output columns
    These define the curvature-collapse regime of Theorem 3.3; the paper admits in Section 6 and Figure 8(d) that the endpoint β_{1,C,N} can go negative at depth.
  • standard math Assumptions 2.1–2.3: L-smoothness, unbiased stochastic gradients, bounded variance
    Standard stochastic-optimization assumptions used by Theorem 3.2; ubiquitous in the cited literature.
  • standard math Spectral–nuclear norm duality and polar-factor identities
    Basis of Theorem 3.1's steepest-descent identity; classical (Bhatia 2009).
  • standard math Marchenko–Pastur law, including β(1) = 8/(3π) quarter-circle mean
    Used in Remarks 3.7 and F.2 for the square-to-vector Θ(d^{1/4}) separation; the constant is correctly computed.

pith-pipeline@v1.3.0-alltime-deepseek · 39959 in / 23481 out tokens · 238981 ms · 2026-08-02T01:49:15.390369+00:00 · methodology

0 comments
read the original abstract

Muon-style optimizers apply a polar map to matrix momentum, but their updates also depend on the representation of each parameter block before orthogonalization. We study this representation choice as a form of optimizer geometry and introduce {\method}, a family of Muon-style optimizers that shares the same momentum rule and Newton--Schulz backend across native, nearest-square, skinny, and vector representations. Each Frobenius-isometric representation induces a distinct polar steepest-descent geometry, in which the shorter matrix dimension determines the number of supported singular channels, the pullback scaling, and the constants in stochastic nonconvex convergence bounds. In a teacher--student model, curvature collapse and an isotropic Marchenko--Pastur spectral profile connect early-stage dissipation to the represented nuclear-to-squared-Frobenius norm ratio. Pretraining experiments on LLaMA2-130M and LLaMA2-600M, together with fixed-momentum diagnostics, show that balanced non-native representations can match the performance of the native representation, whereas reducing the shorter dimension weakens the scaling and singular-channel support, leading to behavior that increasingly resembles normalized momentum.

Figures

Figures reproduced from arXiv: 2607.14536 by Da Chang, Di He, Ganzhao Yuan, Lvgang Zhang, Qiankun Shi, Yaoshuai Ma, Yongxiang Liu.

Figure 1
Figure 1. Figure 1: Matrix representations create a nuclear-support gap relative to normalized SGD with momentum (nSGDM) (Cutkosky and Mehta, 2020; Zhao et al., 2024). (a) For saved LLaMA2 momentum blocks, Vector (nSGDM) lies on the Frobenius–nuclear identity line, whereas matrix representations exhibit greater nuclear support at comparable Frobenius scales. (b) Both the MP shape factor g(γ) = β(γ)γ 1/4 and its empirical coun… view at source ↗
Figure 2
Figure 2. Figure 2: The controlled teacher–student model exhibits hidden-unit block dominance in the local W-curvature. (a) Empirical W-block Hessian arranged by hidden unit. (b) Quantiles of rWW i := Ri/λmin(Hb WW ii ), where Ri = P k̸=i ∥Hb WW ik ∥op; values below one indicate diagonal hidden-unit blocks dominating their off-block coupling. (c) Coordinate lower envelope α1 and hidden-unit block lower endpoint β1 = mini{λmin… view at source ↗
Figure 3
Figure 3. Figure 3: Short-side reduction changes the learning-rate calibration and becomes more costly with depth. (a) Largest single-hidden-layer dimension sweep (m, n, C) = (96, 192, 768); selected points move to larger learning rates for narrower or vectorized representations even when the best final losses are comparable. (b) Best final normalized loss across hidden depth after tuning the constant learning rate for each r… view at source ↗
Figure 4
Figure 4. Figure 4: shows a consistent ordering at both scales: Native and Square remain at the low-loss end of the polar family, short-side reductions form a higher-loss ladder, and Vector is clearly separated. At 600M, Square is within 0.003 validation loss of Native. 0.002 0.003 0.005 0.008 0.01 0.015 0.02 Matrix update LR 3.20 3.25 3.30 3.35 3.40 3.45 3.50 3.55 Final validation loss AdamW Native Square Sk-75 Sk-50 Sk-25 S… view at source ↗
Figure 5
Figure 5. Figure 5: summarizes module geometry at the final checkpoint without duplicating the LLaMA2-130M Frobenius–nuclear panel in [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Additional single-hidden-layer dimension sweeps for Figure [PITH_FULL_IMAGE:figures/full_fig_p037_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Ten-hidden-layer 10000-step supplement to [PITH_FULL_IMAGE:figures/full_fig_p037_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Finite-(C, N) and depth diagnostics for the fixed-mask curvature window in Theorem 3.3. Panels (a)–(c) use the one-hidden-layer hidden-row diagnostic dimensions (m, n) = (24, 48): panel (a) plots the normalized lower endpoint, panel (b) plots the relative curvature-window width ρC,N = (β2,C,N − β1,C,N )/(β1,C,N + β2,C,N ), and panel (c) plots the off-block radius RC,N,i(τE) = P k̸=i ∥HWW C,N,ik(τE)∥op. Pan… view at source ↗
Figure 9
Figure 9. Figure 9: Full fixed-mask Hessian magnitude maps for the depth diagnostic at [PITH_FULL_IMAGE:figures/full_fig_p039_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Layerwise represented nuclear norm across checkpoints. Each bar is the parameter-count [PITH_FULL_IMAGE:figures/full_fig_p040_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Module-resolved supplement to Figure [PITH_FULL_IMAGE:figures/full_fig_p040_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Layerwise represented nuclear norm across LLaMA2-600M checkpoints. Each bar is the [PITH_FULL_IMAGE:figures/full_fig_p041_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Module-resolved supplement to Figure [PITH_FULL_IMAGE:figures/full_fig_p041_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 29 linked inside Pith

  1. [1]

    Dion: Distributed orthonormalized updates.arXiv preprint arXiv:2504.05295,

    Kwangjun Ahn, Byron Xu, Natalie Abreu, Ying Fan, Gagik Magakyan, Pratyusha Sharma, Zheng Zhan, and John Langford. Dion: Distributed orthonormalized updates.arXiv preprint arXiv:2504.05295,

    arXiv

  2. [3]

    Asgo: Adaptive structured gradient optimization.arXiv preprint arXiv:2503.20762,

    Kang An, Yuxing Liu, Rui Pan, Yi Ren, Shiqian Ma, Donald Goldfarb, and Tong Zhang. Asgo: Adaptive structured gradient optimization.arXiv preprint arXiv:2503.20762,

    arXiv

  3. [6]

    Towards understanding of orthogonalization in muon

    Valentyn Boreiko, Zhiqi Bu, and Sheng Zha. Towards understanding of orthogonalization in muon. In High-dimensional Learning Dynamics 2025,

    2025

  4. [8]

    A note on stability for orthogonalized matrix momentum with client sampling.arXiv preprint arXiv:2606.01720, 2026a

    Da Chang, Qiankun Shi, Lvgang Zhang, Yu Li, and Ruijie Zhang. A note on stability for orthogonalized matrix momentum with client sampling.arXiv preprint arXiv:2606.01720, 2026a. Da Chang, Qiankun Shi, Lvgang Zhang, Yu Li, Ruijie Zhang, Yao Lu, Yongxiang Liu, and Ganzhao Yuan. Muoneq: Balancing before orthogonalization with lightweight equilibration.arXiv ...

    Pith/arXiv arXiv

  5. [9]

    Momentum improves normalized sgd.arXiv preprint arXiv:2002.03305,

    Ashok Cutkosky and Harsh Mehta. Momentum improves normalized sgd.arXiv preprint arXiv:2002.03305,

    Pith/arXiv arXiv 2002

  6. [11]

    Rmnp: Row-momentum normalized preconditioning for scalable matrix-based optimization.arXiv preprint arXiv:2603.20527,

    Shenyang Deng, Zhuoli Ouyang, Tianyu Pang, Zihang Liu, Ruochen Jin, Shuhua Yu, and Yaoqing Yang. Rmnp: Row-momentum normalized preconditioning for scalable matrix-based optimization.arXiv preprint arXiv:2603.20527,

    Pith/arXiv arXiv

  7. [12]

    Towards quantifying the hes- sian structure of neural networks.ArXiv, abs/2505.02809,

    Zhaorui Dong, Yushun Zhang, Zhi-Quan Luo, Jianfeng Yao, and Ruoyu Sun. Towards quantifying the hes- sian structure of neural networks.ArXiv, abs/2505.02809,

    arXiv

  8. [14]

    A stable whitening optimizer for efficient neural network training.arXiv preprint arXiv:2506.07254,

    Kevin Frans, Sergey Levine, and Pieter Abbeel. A stable whitening optimizer for efficient neural network training.arXiv preprint arXiv:2506.07254,

    arXiv

  9. [15]

    Accelerating newton-schulz iteration for orthogonalization via chebyshev-type polynomials.arXiv preprint arXiv:2506.10935,

    Ekaterina Grishina, Matvey Smirnov, and Maxim Rakhuba. Accelerating newton-schulz iteration for orthogonalization via chebyshev-type polynomials.arXiv preprint arXiv:2506.10935,

    arXiv

  10. [16]

    Error feedback for muon and friends.arXiv preprint arXiv:2510.00643,

    Kaja Gruntkowska, Alexander Gaponov, Zhirayr Tovmasyan, and Peter Richtárik. Error feedback for muon and friends.arXiv preprint arXiv:2510.00643,

    arXiv

  11. [17]

    Effective quantization of muon optimizer states.arXiv preprint arXiv:2509.23106,

    Aman Gupta, Rafael Celente, Abhishek Shivanna, DT Braithwaite, Gregory Dexter, Shao Tang, Hiroto Udagawa, Daniel Silva, Rohan Ramanath, and S Sathiya Keerthi. Effective quantization of muon optimizer states.arXiv preprint arXiv:2509.23106,

    arXiv

  12. [18]

    Di He, Songjun Tu, Ajay Jaiswal, Li Shen, Ganzhao Yuan, Shiwei Liu, and Lu Yin

    URLhttps://openreview.net/forum?id= lJSfxtLpLm. Di He, Songjun Tu, Ajay Jaiswal, Li Shen, Ganzhao Yuan, Shiwei Liu, and Lu Yin. Alphadecay: Module- wise weight decay for heavy-tailed balancing in llms.Advances in Neural Information Processing Systems, 38:29745–29768, 2026a. Di He, Songjun Tu, Keyu Wang, Lu Yin, and Shiwei Liu. One lr doesn’t fit all: Heav...

    Pith/arXiv arXiv

  13. [20]

    Muon: An optimizer for hidden layers in neural networks, 2024.URL https://kellerjordan

    Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cecista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024.URL https://kellerjordan. github. io/posts/muon, 6,

    2024

  14. [22]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors,3rd International Conference on Learning Representations, ICLR 2015,

    2015

  15. [24]

    cc/paper_files/paper/2023/hash/a3cc50126338b175e56bb3cad134db0b-Abstract-Conference

    URLhttp://papers.nips. cc/paper_files/paper/2023/hash/a3cc50126338b175e56bb3cad134db0b-Abstract-Conference. html. Jiaxiang Li and Mingyi Hong. A note on the convergence of muon.arXiv preprint arXiv:2502.02900,

    Pith/arXiv arXiv 2023

  16. [25]

    Muon is scalable for llm training.arXiv preprint arXiv:2502.16982,

    Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for llm training.arXiv preprint arXiv:2502.16982,

    Pith/arXiv arXiv

  17. [27]

    Improved convergence rates of muon optimizer for nonconvex optimization.arXiv preprint arXiv:2601.19400,

    Shuntaro Nagashima and Hideaki Iiduka. Improved convergence rates of muon optimizer for nonconvex optimization.arXiv preprint arXiv:2601.19400,

    arXiv

  18. [28]

    Spectral flattening is all muon needs: How orthogonalization controls learning rate and convergence

    Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong, Tuc Nguyen, James Bailey, and Trung Le. Spectral flattening is all muon needs: How orthogonalization controls learning rate and convergence. arXiv preprint arXiv:2605.13079,

    Pith/arXiv arXiv

  19. [29]

    Training deep learning models with norm-constrained lmos.arXiv preprint arXiv:2502.07529,

    Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained lmos.arXiv preprint arXiv:2502.07529,

    Pith/arXiv arXiv

  20. [31]

    Lions and muons: Optimization via stochastic frank-wolfe

    Maria-Eleni Sfyraki and Jun-Kun Wang. Lions and muons: Optimization via stochastic frank-wolfe. arXiv preprint arXiv:2506.04192,

    Pith/arXiv arXiv

  21. [32]

    Practical efficiency of muon for pretraining.arXiv preprint arXiv:2505.02222,

    Ishaan Shah, Anthony M Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J Shah, et al. Practical efficiency of muon for pretraining.arXiv preprint arXiv:2505.02222,

    Pith/arXiv arXiv

  22. [33]

    Adafactor: Adaptive learning rates with sublinear memory cost.ArXiv, abs/1804.04235,

    Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost.ArXiv, abs/1804.04235,

    Pith/arXiv arXiv

  23. [35]

    A distributed data-parallel pytorch imple- mentation of the distributed shampoo optimizer for training neural networks at-scale.arXiv preprint arXiv:2309.06497,

    Hao-Jun Michael Shi, Tsung-Hsien Lee, Shintaro Iwasaki, Jose Gallego-Posada, Zhijing Li, Kaushik Rangadurai, Dheevatsa Mudigere, and Michael Rabbat. A distributed data-parallel pytorch imple- mentation of the distributed shampoo optimizer for training neural networks at-scale.arXiv preprint arXiv:2309.06497,

    Pith/arXiv arXiv

  24. [36]

    Beyond the ideal: Analyzing the inexact muon update.arXiv preprint arXiv:2510.19933,

    Egor Shulgin, Sultan AlRashed, Francesco Orabona, and Peter Richtárik. Beyond the ideal: Analyzing the inexact muon update.arXiv preprint arXiv:2510.19933,

    arXiv

  25. [37]

    Muon is not that special: Random or inverted spectra work just as well.arXiv preprint arXiv:2605.11181,

    12 Zakhar Shumaylov, Nathaël Da Costa, Peter Zaika, Bálint Mucsányi, Alex Massucco, Yoav Gelberg, Carola-Bibiane Schönlieb, Yarin Gal, and Philipp Hennig. Muon is not that special: Random or inverted spectra work just as well.arXiv preprint arXiv:2605.11181,

    Pith/arXiv arXiv

  26. [38]

    Adamuon: Adaptive muon optimizer.arXiv preprint arXiv:2507.11005,

    Chongjie Si, Debing Zhang, and Wei Shen. Adamuon: Adaptive muon optimizer.arXiv preprint arXiv:2507.11005,

    arXiv

  27. [39]

    Hugo Touvron, Louis Martin, Kevin R

    URLhttps://api.semanticscholar.org/CorpusID:276094438. Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Bria...

    Pith/arXiv arXiv

  28. [40]

    Soap: Improving and stabilizing shampoo using adam.arXiv preprint arXiv:2409.11321,

    Nikhil Vyas, Depen Morwani, Rosie Zhao, Mujin Kwun, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. Soap: Improving and stabilizing shampoo using adam.arXiv preprint arXiv:2409.11321,

    Pith/arXiv arXiv

  29. [41]

    Ziqing Wen, Jiahuan Wang, Ping Luo, Dongsheng Li, and Tao Sun

    URLhttp://papers.nips.cc/paper_files/paper/2023/hash/ 7ac19fdcdf4f311f3e3ef2e7ef4784d7-Abstract-Conference.html. Ziqing Wen, Jiahuan Wang, Ping Luo, Dongsheng Li, and Tao Sun. Foam: Blocked state folding for memory-efficient llm training.arXiv preprint arXiv:2512.07112,

    Pith/arXiv arXiv 2023

  30. [42]

    On the width scaling of neural optimizers under matrix operator norms i: Row/column normalization and hyperparameter transfer.arXiv preprint arXiv:2603.09952,

    Ruihan Xu, Jiajin Li, and Yiping Lu. On the width scaling of neural optimizers under matrix operator norms i: Row/column normalization and hyperparameter transfer.arXiv preprint arXiv:2603.09952,

    arXiv

  31. [43]

    Mars: Unleashing the power of variance reduction for training large models.arXiv preprint arXiv:2411.10438,

    Huizhuo Yuan, Yifeng Liu, Shuang Wu, Xun Zhou, and Quanquan Gu. Mars: Unleashing the power of variance reduction for training large models.arXiv preprint arXiv:2411.10438,

    Pith/arXiv arXiv

  32. [44]

    Adagrad meets muon: Adaptive stepsizes for orthogonal updates.arXiv preprint arXiv:2509.02981,

    Minxin Zhang, Yuxuan Liu, and Hayden Schaeffer. Adagrad meets muon: Adaptive stepsizes for orthogonal updates.arXiv preprint arXiv:2509.02981,

    Pith/arXiv arXiv

  33. [45]

    Adam improves muon: Adaptive moment estimation with orthogonalized momentum.arXiv preprint arXiv:2602.17080,

    Minxin Zhang, Yuxuan Liu, and Hayden Scheaffer. Adam improves muon: Adaptive moment estimation with orthogonalized momentum.arXiv preprint arXiv:2602.17080,

    arXiv

  34. [46]

    Why transformers need adam: A hessian perspective.ArXiv, abs/2402.16788,

    13 Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhimin Luo. Why transformers need adam: A hessian perspective.ArXiv, abs/2402.16788,

    Pith/arXiv arXiv

  35. [47]

    Dongruo Zhou, Pan Xu, and Quanquan Gu

    doi: 10.1007/s11432-022-3892-8. Dongruo Zhou, Pan Xu, and Quanquan Gu. Stochastic nested variance reduction for nonconvex optimiza- tion.Journal of machine learning research, 21(103):1–63,

    doi:10.1007/s11432-022-3892-8

  36. [48]

    14 Appendix A Related Work ▶ Adaptive and structured preconditioning.Coordinate-wise adaptive methods estimate diagonal or factored geometry from gradient history, whereas natural-gradient methods, K-FAC, Shampoo, SOAP, and related tensor preconditioners organize geometry at the layer, matrix, or tensor level (Duchi et al., 2011; Kingma and Ba, 2015; Losh...

    2011

  37. [49]

    The model has123 .59M trainable parameters

    with12layers, hidden width768,12attention heads of dimension64, MLP hidden width 2048, RMSNorm with ϵ = 10 −6, tied token embeddings, no bias parameters, and no dropout. The model has123 .59M trainable parameters. Training uses the FineWeb10B stream (Penedo et al., 2024), which contains10 ,255,324,043training tokens and100 ,000,000validation tokens, and a...

    2048

  38. [50]

    with vocabulary size50,304. We use sequence length4096, global batch size128,4960training steps,496warmup steps, cosine learning-rate decay, weight decay0 .1, gradient clipping at norm1 .0, and bfloat16 training. These runs use four NVIDIA RTX PRO 6000 Blackwell GPUs (96GB). Muon/Muse updates the matrix modules summarized in Table 1 with momentum0.95, no ...

    2024

  39. [51]

    with hidden width1024, MLP hidden width4096, RMSNorm, tied token embeddings, no bias parameters, and no dropout. The run uses the C4 stream with T5-base tokenization (Raffel et al., 2020), sequence length4096, global batch size320, and9600training steps, corresponding to 12,582,912,000training tokens. The warmup–stable–decay schedule uses800warmup steps a...

    2020

  40. [52]

    In these settings, the selected markers for narrower or vectorized representations occur at larger learning rates than the native and square markers

    These panels use the same protocol as the main-text sweep and cover the remaining settingsm∈ {32, 48, 64}, with n = 2m and C = 8m. In these settings, the selected markers for narrower or vectorized representations occur at larger learning rates than the native and square markers. 0 2000 4000 6000 8000 10000 Stochastic steps 10−5 10−4 10−3 10−2 10−1 100 No...

    2000

  41. [2001]

    Convergence bound and critical batch size of muon optimizer.arXiv preprint arXiv:2507.01598,

    Naoki Sato, Hiroki Naganuma, and Hideaki Iiduka. Convergence bound and critical batch size of muon optimizer.arXiv preprint arXiv:2507.01598,

    Pith/arXiv arXiv

  42. [2009]

    Turbo-muon: Accelerating orthogonality-based optimization with pre-conditioning.arXiv preprint arXiv:2512.04632,

    Thibaut Boissin, Thomas Massena, Franck Mamalet, and Mathieu Serrurier. Turbo-muon: Accelerating orthogonality-based optimization with pre-conditioning.arXiv preprint arXiv:2512.04632,

    Pith/arXiv arXiv

  43. [2010]

    Old optimizer, new norm: An anthology.arXiv preprint arXiv:2409.20325,

    Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology.arXiv preprint arXiv:2409.20325,

    Pith/arXiv arXiv

  44. [2015]

    Tim Tsz-Kit Lau, Qi Long, and Weijie Su

    URLhttp://arxiv.org/abs/1412.6980. Tim Tsz-Kit Lau, Qi Long, and Weijie Su. Polargrad: A class of matrix-gradient optimizers from a unifying preconditioning perspective.arXiv preprint arXiv:2505.21799,

    Pith/arXiv arXiv

  45. [2018]

    Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, and Jiawei Zhang

    URLhttps://api.semanticscholar.org/CorpusID:4786918. Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, and Jiawei Zhang. On the convergence analysis of muon.arXiv preprint arXiv:2505.23737,

    Pith/arXiv arXiv

  46. [2019]

    Chao Ma, Wenbo Gong, Meyer Scetbon, and Edward Meeds

    URLhttps://openreview.net/ forum?id=Bkg6RiCqY7. Chao Ma, Wenbo Gong, Meyer Scetbon, and Edward Meeds. Swan: Sgd with normalization and whitening enables stateless llm training.ArXiv, abs/2412.13148,

    Pith/arXiv arXiv

  47. [2020]

    Momentum-based variance reduction in non-convex sgd.ArXiv, abs/1905.10018,

    Ashok Cutkosky and Francesco Orabona. Momentum-based variance reduction in non-convex sgd.ArXiv, abs/1905.10018,

    Pith/arXiv arXiv 1905

  48. [2021]

    Florian Hübler, Thomas Pethick, and Suvrit Sra

    URL https://api.semanticscholar.org/CorpusID:235436027. Florian Hübler, Thomas Pethick, and Suvrit Sra. Free heavy-tailed lunch for muon: A theoretical justification of empirical success.arXiv preprint arXiv:2606.14560,

    arXiv

  49. [2023]

    10 Cong Fang, Chris Junchi Li, Zhouchen Lin, and T. Zhang. Spider: Near-optimal non-convex optimization via stochastic path integrated differential estimator.ArXiv, abs/1807.01695,

    Pith/arXiv arXiv

  50. [2024]

    Muonbp: Faster muon via block-periodic orthogonalization.arXiv preprint arXiv:2510.16981,

    Ahmed Khaled, Kaan Ozkara, Tao Yu, Mingyi Hong, and Youngsuk Park. Muonbp: Faster muon via block-periodic orthogonalization.arXiv preprint arXiv:2510.16981,

    arXiv

  51. [2025]

    The polar express: Optimal matrix sign methods and their application to the muon algorithm.arXiv preprint arXiv:2505.16932,

    Noah Amsel, David Persson, Christopher Musco, and Robert M Gower. The polar express: Optimal matrix sign methods and their application to the muon algorithm.arXiv preprint arXiv:2505.16932,

    Pith/arXiv arXiv

  52. [2026]

    On the convergence of muon and beyond.arXiv preprint arXiv:2509.15816,

    Da Chang, Yongxiang Liu, and Ganzhao Yuan. On the convergence of muon and beyond.arXiv preprint arXiv:2509.15816,

    Pith/arXiv arXiv