Reassessing Muon for Matrix Factorization

Alex Cloninger; Ali Parviz; Gal Mishne

Muon's reported edge over AdamW vanishes under per-optimizer tuning in low-rank matrix factorization.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 05:44 UTC pith:MRDNH4AO

load-bearing objection A useful, carefully controlled negative result: Muon is not uniformly better than AdamW on matrix factorization, but the claim rests on LR-only tuning and deserves a referee who will push on Muon's fixed hyperparameters. the 3 major comments →

arxiv 2607.13246 v2 pith:MRDNH4AO submitted 2026-07-14 cs.LG cs.AI

Reassessing Muon for Matrix Factorization

Ali Parviz , Gal Mishne , Alex Cloninger This is my paper

classification cs.LG cs.AI MSC 68T0790C2665K10

keywords Muon optimizermatrix factorizationAdamWspectral orthogonalizationoptimizer comparisonnonnegative matrix factorizationlow-rank matrix recoveryhyperparameter tuning

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Muon reshapes gradient updates by approximately orthogonalizing them through a Newton–Schulz polar factor, and it has been reported to speed up large-scale language-model training. This paper asks whether that advantage belongs to the update rule itself or to the scale, architecture, and data of modern networks, by testing Muon on low-rank matrix factorization with each optimizer individually tuned over a learning-rate grid. The central finding is that under per-optimizer tuning no method dominates: tuned AdamW and gradient descent match or beat Muon on plain low-rank factorization, Muon and AdamW are comparable on matrix completion, and Muon wins clearly on nonnegative matrix factorization and on deep tensor-train factorization. A sympathetic reader should take away that Muon's benefit is problem-dependent and concentrated in nonnegativity-constrained and deep-product regimes, and that single-learning-rate comparisons are not reliable evidence about optimizer quality.

Core claim

On the paper's own terms: across five matrix-recovery problems, with a controlled protocol that sweeps only the learning rate (20 or 25 log-spaced points) and holds all other hyperparameters fixed, Muon does not consistently outperform AdamW. At matched tuning, AdamW and plain gradient descent reach near machine precision on low-rank factorization while Muon plateaus several orders higher; on matrix completion the two adaptive methods are indistinguishable; on nonnegative factorization Muon wins clearly, reaching losses of 1e-9 to 1e-6 where AdamW stalls near 1e-3. An analytical derivation shows why: Muon's update is the polar factor of the negative gradient (the orthonormal-column matrix cl

What carries the argument

The comparison centers on Muon's Newton–Schulz orthogonalization, an approximate polar factor that discards singular-value magnitudes and acts as spectral-norm steepest descent, versus AdamW's coordinate-wise rescaling by the square root of a second moment. The experimental machinery is a systematic learning-rate sweep (25 or 20 log-spaced points per problem–optimizer–condition triple) with momentum, Nesterov flag, Newton–Schulz count, and orthogonalization coefficients held at standard values, plus a patience-based decay schedule and three seeds. This protocol exposes each method's stable learning-rate range and reveals that losses vary by roughly eleven orders of magnitude across the grid,

Load-bearing premise

The conclusion that Muon does not consistently beat AdamW rests on treating a learning-rate-only sweep (20 or 25 log-spaced values, all other hyperparameters fixed at standard values) as equal tuning; if Muon's momentum, Nesterov flag, Newton–Schulz count, or orthogonalization coefficients were tuned per problem, its reported losses could shift enough to change several rankings.

What would settle it

Re-run the paper's Table 1 protocol with per-optimizer sweeps over all free hyperparameters (momentum, Nesterov flag, Newton–Schulz count, orthogonalization coefficients for Muon; beta_1, beta_2, and weight decay for AdamW). The central claim fails if Muon then wins on plain low-rank factorization, or if AdamW wins on nonnegative matrix factorization.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Muon's reported gains on large-language-model training should not be attributed to a generic spectral-preconditioning benefit; on simple matrix factorization a well-tuned AdamW matches or beats it.
On nonnegative matrix factorization, Muon's orthogonalized updates consistently win across ranks and spectral shapes, suggesting its value there comes from discouraging redundant, collinear factors.
In tensor-train factorizations of increasing depth, Muon overtakes AdamW by one to two orders of magnitude at depth 6, so its advantage appears tied to deep, ill-conditioned product parameterizations rather than shallow ones.
Because loss-minimizing learning rates differ across optimizers by 1.5–3.1 decades and tuning reverses default-rate rankings in nearly all nineteen settings, any comparison at a single learning rate is an artifact of that choice.
At a fixed extreme condition number, AdamW adapts to clustered or heavy-tailed spectra while Muon does not; spectrum shape, not just condition number, determines which optimizer wins.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

A testable extension: if full hyperparameter tuning (momentum, Nesterov, Newton–Schulz count, orthogonalization coefficients) were included, Muon's plain-factorization gap might shrink, but the paper's structural story predicts it would not reverse; the decisive experiment is to re-run Table 1 with per-optimizer sweeps over all knobs.
The derived sqrt(N) scaling between AdamW and Muon step norms implies that when transferring Muon across matrix sizes, retuning should start at eta_muon ≈ sqrt(N) * eta_adamw rather than a fresh sweep; this is directly testable and, if confirmed, would give practical guidance the paper mostly leaves implicit.
The subspace-alignment dynamics suggest Muon's implicit bias is more isotropic, distributing capacity across directions; on tasks where balanced recovery matters more than final training loss, such as fair or robust representation learning, Muon might generalize better even where its training loss is worse. This follows from the paper's mechanism but is not claimed in it.
If the NMF advantage is truly about discouraging redundant factors, a prediction follows for tabular or sparse data with near-collinear features: Muon should show a similar edge on nonnegativity-constrained dictionary learning, a neighboring problem the paper does not test.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A useful, carefully controlled negative result: Muon is not uniformly better than AdamW on matrix factorization, but the claim rests on LR-only tuning and deserves a referee who will push on Muon's fixed hyperparameters.

read the letter

Short version: this is a competent empirical study with a genuinely useful negative result. Under learning-rate-tuned comparison on low-rank matrix factorization, Muon does not consistently beat AdamW or even GD. That is worth knowing before the field settles into the habit of treating Muon as a default improvement. The paper supports a problem-dependent picture, not a uniform verdict.

What is actually new: this is the first systematic LR-swept comparison I know of across five factorization settings—plain factorization, completion, NMF, tensor-train, and a kernel target. The headline result is that well-tuned AdamW and GD win on plain factorization, Muon and AdamW are comparable on completion, and Muon wins clearly on NMF. The spectral-shape experiment at fixed conditioning is a nice addition, and the appendices are unusually transparent about limitations (for example, they note that increasing tensor-train depth changes the problem itself, and that the AdamW proxy is approximate). The paper earns credit for isolating optimizer behavior from scale and architecture confounders.

Soft spots, in proportion. The main one is that "per-optimizer tuning" is really learning-rate-only tuning. Momentum, the Nesterov flag, Newton–Schulz count J, and the orthogonalization coefficients are fixed at standard values and never swept. Muon is sensitive to those choices, so the reported plateau on plain factorization could shift if those were tuned problem-by-problem. The conclusion is conditional on that protocol. I would not call it fatal—the paper is explicit about the protocol—but the claim is oversold slightly. Three seeds is modest, and Table 1 gives no error bars (the figures do show bands). Separately, Appendix L's analytical derivation for AdamW substitutes the raw gradient for the first moment and then marks the result approximate; that is honest, but it weakens the theoretical framing. It does not undermine Table 1.

Who this is for: optimizer benchmarking folks and anyone deciding whether to use Muon on matrix/tensor problems. The negative result is directly relevant to the current wave of Muon enthusiasm. I would send this to a serious referee. The request should be to extend the hyperparameter sweeps for Muon, report error bars in the main table, and fix or soften the AdamW derivation. If those survive, the paper is publishable and worth citing.

Referee Report

3 major / 5 minor

Summary. This paper revisits Muon on low-rank matrix factorization. It formulates symmetric factorization, matrix completion, NMF, tensor-train, and kernel factorization problems; for each, it sweeps learning rate over 20-30 log-spaced points while fixing all other optimizer hyperparameters, runs 3 seeds, and reports tuned final NMSE for Muon, AdamW, GD, and SignGD. The main empirical claim is that under this per-optimizer tuning, no method dominates: AdamW/GD win plain low-rank factorization, Muon and AdamW are comparable on completion, and Muon clearly wins on both NMF variants; conditioning and spectral shape change rankings. The paper includes a spectral-subspace analysis and an analytical appendix claiming to derive AdamW and Muon updates from trust-region and Procrustes proxies. The authors conclude that Muon's large-scale advantages are problem-dependent and argue for controlled benchmarks.

Significance. The controlled problem suite and the distinction between default-rate and tuned-rate rankings are useful; the paper makes a plausible case that Muon's advantage in LLM pretraining does not transfer automatically to simple spectral problems. If the empirical claims survive a broader hyperparameter sweep, the paper would be a valuable cautionary contribution to optimizer evaluation methodology. The learning-rate sweeps are extensive, the baselines are standard, and the NMF/tensor-train results provide a contrasting regime where orthogonalization helps. However, the headline negative claim is more conditional than the abstract states, for the reasons detailed below.

major comments (3)

[Section 3.2 and Table 1] The protocol defines 'tuning' as a learning-rate sweep only: 25 or 20 log-spaced LR points with 'all other hyperparameters (momentum, and for Muon the Newton-Schulz count J and the orthogonalization coefficients) at standard values.' The Muon update (Eqs. 6-8 and 11) depends on mu, the Nesterov flag, J, and (a,b,c), and AdamW depends on beta1, beta2, and epsilon; none of these are varied. Therefore the central claim 'under per-optimizer tuning no method dominates universally' (Section 4, Table 1) is actually a claim about LR-tuned defaults. Since a non-representative default configuration could conceal a Muon advantage (or create one), the headline needs either a sensitivity analysis (at least over J and the Newton-Schulz coefficients, and preferably over mu) or a consistent rephrasing as 'under learning-rate-only tuning.' This concern directly bears on the paper's main conclusion.
[Appendix L, Proposition 1 and Eq. (22)] Proposition 1 states that the AdamW update is the analytical solution of the linearized-loss/trust-region proxy. However, Eq. (22) uses the raw gradient G_t in the numerator, whereas the actual AdamW update uses the bias-corrected first moment m_t (with m_t = beta1 m_{t-1} + (1-beta1) G_t). The object solved in Eqs. (21)-(22) is a normalized-gradient method, not AdamW. Consequently the scale comparison in Eq. (37) and the explanation of why the Muon learning rate should be scaled by sqrt(N) are not valid as a derivation for AdamW. This does not invalidate the empirical Table 1, but it is a real inaccuracy in the theoretical framing and should be corrected or explicitly labeled as a heuristic for a gradient-normalization variant.
[Table 1] Table 1, the main quantitative support for the 'no universal winner' claim, reports only geometric means over 3 seeds, with no per-seed values or confidence intervals. Some adjacent entries are extremely close (e.g., factorization kappa=1: AdamW 1.9e-13 vs GD 1.6e-13; completion kappa=1: Muon 4.6e-16 vs AdamW 4.8e-16), so the bold-faced 'best per row' is not robust to seed variation. The paper should report the spread (e.g., per-seed losses or log-std intervals in Table 1) or soften the ranking claims where differences are within noise.

minor comments (5)

[Appendices E and J] Appendix E and Appendix J contain identical text, with the alignment matrix defined as Eq. (15) and Eq. (16). Please consolidate or remove one duplicate.
[Figure 4 caption vs. text] The Figure 4 caption reports diagonal energy values 0.505 and 0.578, while the text reports 0.586 vs 0.643 in the caption/body. Please verify which numbers are correct.
[Abstract and Section 3.2] The abstract says 'carefully tuned comparison' and the text says 'per-optimizer tuning,' but the protocol is a learning-rate-only sweep. The wording should be aligned with what was actually done.
[Appendix L, Eq. (22)] The denominator in the AdamW-style update is written as sqrt(v_t) without the epsilon term used in the standard algorithm (sqrt(v_hat) + epsilon). Add epsilon or state that it is omitted.
[Section 2 and Eq. (37)] The sentence 'We analyze the exact softplus...' has a stray capital 'We' after a comma. Also, Eq. (37) introduces N=1000 for the step-size ratio while the main experiments use d=100; clarify whether N refers to a different setting.

Circularity Check

0 steps flagged

No significant circularity: direct empirical comparisons against external baselines, no self-citations, and no fitted inputs renamed as predictions.

full rationale

The paper's central claims—that Muon does not consistently outperform AdamW on low-rank factorization under per-optimizer tuning, and that results are sensitive to learning rate and conditioning—are direct empirical comparisons against external baselines (AdamW, GD, SignGD). Table 1 reports minima over explicit learning-rate grids, not fitted predictions of a related quantity, so no fitted-input-called-prediction pattern appears. The reference list contains no self-citations by the authors (Parviz, Mishne, Cloninger), so there is no load-bearing self-citation or imported uniqueness theorem. Appendix L derives update-norm scaling from a local proxy problem; although Eq. 22 replaces AdamW's first moment m_t with the raw gradient G_t, making the derivation unfaithful to actual AdamW, this is a correctness/faithfulness issue rather than circularity: the derived sqrt(N) scaling follows from the stated proxy assumptions and is not an input to the empirical comparison. The main limitation—that Section 3.2 defines 'per-optimizer tuning' as a learning-rate sweep only, holding Muon's momentum, Nesterov flag, Newton–Schulz count J, and orthogonalization coefficients fixed—conditions the generality of the negative conclusion but does not make any result reduce by construction to its own inputs. No derivation chain in the paper is equivalent to its inputs, and no prediction is forced by a self-citation or by definition.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper's empirical contribution is largely self-contained; its central claims depend on experimental design choices (LR-only tuning, standard hyperparameters, small random init) rather than on fitted constants. The only 'derivation' content, Appendix L, is parameter-free but contains a proxy mismatch for AdamW.

free parameters (4)

learning rate α (per optimizer, per setting) = best point on logarithmic grid; exact values not tabulated (grid [1e-4,5e-1] or [1e-4,3.2])
The entire 'tuned comparison' is defined by choosing the best LR per method; this is the only hyperparameter swept, so the reported rankings are LR-optimal, not jointly optimal.
Muon momentum μ = 0.95
Fixed at Jordan's standard value; not swept, although momentum is a core component of the Muon direction and could change its basin on NMF/completion.
Newton-Schulz iteration count J = 5
Fixed at standard value; more/fewer iterations change orthogonality approximation quality and could affect comparisons.
AdamW betas/epsilon/weight decay = not stated ('standard values')
These defaults determine AdamW's adaptive scaling and basin shape; omitting them makes 'equal tuning' not fully specified.

axioms (4)

standard math Thin SVD and polar decomposition of the gradient exist and satisfy the trace/bound identities used in Appendix L.
Used in Propositions 1–2 and norm scaling; standard linear algebra, no threat.
domain assumption Small random initialization is used for all problems.
Section 3.1 states 'small random initialization' but does not specify distribution/scale; matrix factorization is known to be initialization-sensitive, and conclusions could depend on this choice.
domain assumption The factorized-matrix objective is a faithful minimal model of deep learning's product parameterizations.
Appendix H argues this; it is the premise that lets the authors generalize from d=100 factorization to Muon's LLM behavior. If the proxy is not faithful, the broader implications are unsupported.
ad hoc to paper Muon's NMF advantage is caused by orthogonalized updates discouraging redundant factors.
Section 1 Findings and Section 5 assert this mechanism; the paper provides no controlled ablation proving it, so it is an interpretation rather than a measured result.

pith-pipeline@v1.3.0-alltime-deepseek · 20626 in / 14371 out tokens · 151496 ms · 2026-08-02T05:44:23.409474+00:00 · methodology

0 comments

read the original abstract

Muon has recently emerged as a strong optimizer for large-scale deep learning, where it reshapes gradient updates through approximate orthogonalization and has been reported to outperform Adam and AdamW in large language model training. Its empirical success has motivated a growing body of theoretical work that interprets Muon as steepest descent under the spectral norm. Yet it remains unclear which of Muon's advantages stem from its update rule itself and which are artifacts of the scale, architecture, and data of modern deep networks. In this work, we isolate the optimizer from these confounding factors by studying Muon on a simple, well-understood, and spectrally structured problem: low-rank matrix factorization. Through a controlled comparison against carefully tuned adaptive baselines, we find that Muon does not consistently outperform AdamW in this setting and that several previously reported advantages are sensitive to hyperparameter choices. Our results provide a more nuanced picture of when spectrum-aware orthogonalization is beneficial and argue for evaluating modern optimizers on controlled problems in addition to end-to-end benchmarks.