An Isotropy-Preserving Spectral Cap for Muon: Theory and Three Case Studies

Jiachun Li

REVIEW 4 major objections 4 minor 4 references

Muon removes the 1/||W|| brake on weight growth, and a spectral cap that projects out the top singular direction restores control of the spectrum without freezing learning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 11:48 UTC pith:BGT73LMP

load-bearing objection A clever, honestly-presented theory for a spectral cap on Muon, with one clean proof-of-concept and two case studies whose causal claims don't yet hold up; worth refereeing but not as settled evidence. the 4 major comments →

arxiv 2607.19771 v1 pith:BGT73LMP submitted 2026-07-22 cs.LG cs.AI

An Isotropy-Preserving Spectral Cap for Muon: Theory and Three Case Studies

Jiachun Li This is my paper

classification cs.LG cs.AI

keywords Muonspectral capisotropyscale invariancematrix sign optimizersingular value spectrummixture-of-experts routerlow-precision attention

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Muon, a matrix-sign optimizer, removes the 1/||W|| brake that keeps plain SGD updates small, so weight norms drift outward faster (t^{1/2} vs t^{1/4} under exact scale invariance). The paper's central claim is that a lightweight spectral cap — projecting out only the first-order growth of the top singular direction of the data-coupled covariance W K_X W^T — controls this drift and preserves isotropy without hurting training. The key mechanism is that the second-order perturbation of the spectral norm is non-negative: after capping the top mode, the update still raises lower singular values, rotates the top direction, or switches it. Three case studies show concrete value: participation rank of a transformer feed-forward layer rises from 34 to 96 with no validation-loss change; a mixture-of-experts router's top-share drops from 0.995 to 0.12-0.52 across layers; and a bf16 attention head that would have blown up is stabilized through the crash window. The whole theory rests on an exact scale-invariance assumption the paper itself flags as strong and only approximately true.

Core claim

Under the idealization that the loss is exactly scale invariant, L(cW) = L(W), the gradient of a weight matrix is orthogonal to W and its Frobenius norm decays like 1/||W||_F; plain SGD inherits this brake, while Muon's matrix-sign step msign(G) is scale-invariant and annihilates the 1/||W||_F factor, so its update norm is O(eta) and the weight norm grows as t^{1/2} rather than t^{1/4}. The paper's second discovery is that the second-order perturbation of the spectral norm is always non-negative: after projecting out the first-order top-mode component of an update, the residual evolution still raises lower singular values, rotates the top singular vectors, or allows top switching. Hence a sp

What carries the argument

The spectral cap is the central device: at each step, estimate the top eigenpair (q, λ1) of the data-coupled covariance KY = W KX W^T (or its per-head attention analogue) by a few power iterations; form the rank-1 gradient direction B = 2 q g^T that increases λ1 fastest; then from the candidate Muon update H subtract max(0, ⟨B, H⟩ - ρλ1) · B/||B||^2. This kills first-order λ1 growth while leaving the orthogonal complement of the update free. The proof that the cap does not freeze training is the non-negative second-order spectral perturbation (derived via the symmetric dilation of W): every second-order term (aj + bj)^2/(4(σ1 - σj)) + (aj - bj)^2/(4(σ1 + σj)) is non-negative, so the cap conv

Load-bearing premise

The load-bearing premise is exact scale invariance of the loss under weight rescaling, L(cW)=L(W) for all c>0; if real networks deviate — through biases, weight decay, residual connections, or unnormalized layers — the predicted norm-growth rates and the claim that Muon is special versus SGD or Adam are no longer guaranteed to hold.

What would settle it

Test the scale-invariance assumption across a training run: measure the cosine alignment c_G = ⟨W, ∇L⟩/(||W|| ||∇L||) at many checkpoints and layers; if c_G systematically exceeds ~0.01, or if ||∇L||·||W|| is not roughly constant, the theory's quantitative predictions fail. More directly, train a Muon run without a cap in a small residual network with exact normalization and measure the exponent of ||W||_F growth; if it deviates from t^{1/2} (and the SGD control from t^{1/4}), the brake-removal mechanism is refuted.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

If the theory holds, Muon-trained networks will drift weight norms and spectral norms faster than SGD-trained ones; the spectral cap is a cheap (sub-1% of FLOPs) way to control that drift and keep output covariances isotropic.
Because the second-order term is non-negative, capping only the top singular direction automatically improves participation rank (H2) and Shannon entropy (H1) of the singular-value spectrum, at no extra cost.
In mixture-of-experts routers, the cap prevents the rank-1 collapse where all tokens route to the same expert — a failure that loss-free bias cannot fix once the selection-score geometry is one-dimensional.
In low-precision (bf16) attention, capping the per-head Q/K covariance keeps attention scores below the bf16 failure threshold, preventing the softmax biased-rounding crash without touching attention internals, and it provides a real-time counterfactual diagnostic (excess) that a patched softmax cannot.
Under approximate scale invariance, the same cap should apply to other matrix-sign optimizers with the same orthogonality property, since the second-order non-negativity argument is optimizer-agnostic.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The cap's excess metric — the amount of first-order λ1 growth removed per step — is itself a real-time health diagnostic; one could schedule the cap tolerance ρ to relax when excess is small and tighten when excess spikes, turning the cap into an adaptive safety valve.
The theory suggests a broader design space: any optimizer that keeps updates roughly orthogonal to W and has an O(1) update norm should be equally cap-able; AdamW without weight decay may also exhibit the drift-then-collapse pattern, and the same cap could be applied to it.
A testable extension is to apply the cap to the input-covariance-normalized object A = W K_X^{1/2} for arbitrary K_X, as the paper sketches, and to compare against caps on W; the nanoGPT experiment already predicts the data-coupled cap is more on-target.
The strict scale-invariance assumption could be relaxed: if one can bound the radial gradient component ⟨W, ∇L⟩/||W||, the rates become approximate but the cap mechanism and its second-order argument survive, so the cap is a robust practical device even where the theory's rates fail.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A clever, honestly-presented theory for a spectral cap on Muon, with one clean proof-of-concept and two case studies whose causal claims don't yet hold up; worth refereeing but not as settled evidence.

read the letter

Jim,

You should know two things. First, the core idea is real: projecting the top singular component out of each Muon update, justified by the non-negative second-order perturbation, is a neat and novel mechanism for improving isotropy without freezing training. The paper also ships actual derivations I could redo: the SGD t^{1/4} vs Muon t^{1/2} norm-growth rates follow from exact scale invariance, and the second-order formula in §1.5.1 checks out. Second, the empirical sections are not at the same standard as the theory. The nanoGPT demonstration is the cleanest: val loss unchanged within noise, participation rank up from 34 to 96. That part gives real support to the 'H∞ cap improves H2' claim. But the MoE and FlashAttention stories have holes. In the MoE, G3 directly caps the selection-score covariance and ends up with the best top-share (0.272 vs G1's 0.479), yet its hard-max load collapse is 0.219, three times worse than G1's 0.071. The paper's own phase-transition explanation would predict G3 to be the best balancer, not the middle. That inversion is not addressed. In the FlashAttention case, the cap does not stop λmax from growing 150× (873 → 133K); it only slows it. The rescue is compared against a single uncapped log, no seeds, and the paper itself attributes part of the recovery to attention peaking that is 'not designed into the cap.' The counterfactual excess_mean metric is clever, but it predicts an uncapped rise of ~200K while the capped λ actually falls by 56K — so the first-order model is not what is saving you. Also, the t^{1/4} vs t^{1/2} rates are never checked against any actual norm-vs-time curve; the only empirical test is a single checkpoint measurement of ⟨W,G⟩, which is an input, not an output prediction. Assumption 1 is honestly flagged as the main load-bearing assumption, and the paper admits residual connections, biases, weight decay, and the last layer all break it. Given that, the theory is a clean idealized limit, not yet a description of real Muon runs.

The paper is clearly written, unusually candid, and the limitations are stated rather than hidden. It deserves a serious referee — the idea is novel and the nanoGPT result is solid — but it should not be published as-is; the MoE inversion and the FA single-baseline issue need to be dealt with before anyone should rely on the practical claims.

I'd bring it to the reading group and cite it if I write about optimizer geometry. Accept for peer review, yes.

Best.

Referee Report

4 major / 4 minor

Summary. This preliminary report proposes a unified theory of Muon training dynamics and a spectral cap, then applies the cap to three systems. Under Assumption 1 (exact scale invariance of the loss under positive weight rescaling), the paper derives that SGD updates scale as O(eta/||W||_F) while the Muon msign update is O(eta), giving norm growth rates t^{1/4} vs t^{1/2} (Section 1.4). It then observes that the second-order spectral perturbation is non-negative, so removing the first-order top-direction component of an update still permits learning through non-top directions, top-vector rotation, and top switching (Section 1.5). The cap is applied to data-coupled covariances: KY for the nanoGPT FFN projection, K_{z+b} for a 64-expert MoE router, and per-head Q/K covariances for a bf16 FlashAttention block. Reported results include loss-neutral increases in participation rank, prevention of a MoE rank-1 collapse, and stable traversal of the FA crash window. The paper explicitly labels the scale-invariance assumption as idealizing and the empirical results as preliminary.

Significance. The paper's strength is its explicit, partially verified mathematical framework: the perturbation calculation in Section 1.5.1 is concrete, the per-step cost estimates are transparent, and the norm-growth rates are falsifiable predictions. If the mechanism holds, the spectral cap is a cheap and attractive intervention for Muon training. However, the central theoretical claims rest on an idealization whose empirical support is a single checkpoint (Section 1.4.4), and the three case studies are small-scale, mostly single-seed, and measure isotropy partly on the very object the cap modifies. The paper is therefore a useful preliminary characterization rather than a settled demonstration; its practical claims need substantially more controlled evidence before they can be regarded as established.

major comments (4)

[Sections 1.2, 1.4.2, 1.4.4] Assumption 1 is load-bearing for the 1/||W|| brake, the t^{1/4} vs t^{1/2} rates, and the claim that Muon is special. The empirical support is a single checkpoint at step 48K, with c_G and c_P averaged over 24 weight matrices; this does not verify the constancy of C=||H(Theta_t)|| over a trajectory, nor the near-zero <W,P> needed for the Muon radial term. In addition, the ODE reduction R-dot=a/R treats C as a constant, which does not follow from exact scale invariance. If Assumption 1 fails in real runs—biases, weight decay, residuals, last unnormalized layer—the quantitative predictions no longer describe actual Muon training. The caveats are acknowledged, but the load-bearing status of the assumption should be matched by broader validation or by rephrasing the rates as conditional statements.
[Sections 3.3, 3.4, 3.7] The L0 hard-load results contradict the paper's phase-transition story. G3, which caps the selection-score covariance K_{z+b} directly, has the lowest top-share at L0 (0.272 vs G1's 0.479) but a hard-max of 0.219, roughly three times worse than G1's 0.071. If top-share is the control variable, G3 should be the best load balancer, not an intermediate one. The paper states that G3 is 'not as clean as G1' but offers no mechanism; this undermines the claim that the cap prevents the rank-1 collapse by controlling top-share. The discrepancy should be resolved, e.g., by characterizing the full distribution of K_{z+b} eigenvalues or identifying a different quantity that predicts hard-max.
[Sections 4.4, 4.5, 4.8] The FA case does not establish a causal role for the cap. The cap is claimed to 'stop lambda_max from reaching the failure threshold,' but lambda_max(K_K) grows from 873 at 60K to 133,208 at 86K (150x), and score.max reaches 27,264, about half the bf16 ceiling. The rescue is partly attributed to attention peaking that 'is not designed into the cap' (Section 4.4.3), and the comparison uses a single uncapped baseline with no seeds or error bars. The excess_mean counterfactual predicts an uncapped first-order rise of roughly 200K over 14K steps, while the capped lambda actually falls by 56K; this indicates the first-order projection is not the operative mechanism. The paper should either provide a matched ensemble, a direct ablation (cap on/off at the same seed), or a causal decomposition showing that the cap, not attention dynamics, prevents the crash.
[Sections 2.5, 3.3] Isotropy improvements are partially by construction. In the nanoGPT and MoE applications, the cap directly removes first-order growth of lambda_1(KY) or lambda_1(K_{z+b}), and the reported metrics (top-share, participation rank) are functions of the same covariance. A lower top-share is expected if the cap holds lambda_1 fixed while tr(K) grows; this does not by itself demonstrate that the model learns better or more diverse features. The meaningful evidence is loss-neutrality and failure avoidance, which are present, but the paper should avoid presenting the covariance metrics as independent evidence of improved isotropy. An independent probe (e.g., downstream task performance, gradient alignment, or a control where the cap is applied to a different direction) would strengthen the claim.

minor comments (4)

[Section 3.3] The claim that G3 wins 16 of 18 cells and that the other two are 'PR at L2 / L1' appears inconsistent with the table: G3 actually wins PR at L1 (3.3 vs 2.2) and loses only PR at L2 (5.6 vs 6.6). Please correct the count and the cited cells.
[Section 4.4.1] The index notation is confusing: the text says 'paper's L2 is our L1' but the table and figure captions use both 'L1 H2' and 'paper's L2'. Use a single indexing convention throughout, with a clear mapping to the reference paper's layer numbering.
[Section 1.4.4] The table formatting for c_G and c_P (e.g., '+0.0000 0.0052') should show signs and standard errors explicitly. As written, the reader cannot tell which values are means and which are absolute values.
[General] No code release or checkpoint availability is mentioned. Given the dependence on specific training details (RMS-matched Muon, bf16 attention, exact cap implementation), releasing the code used for the three case studies would materially improve reproducibility.

Circularity Check

2 steps flagged

Isotropy gains are partly by construction (the cap directly targets the eigenvalue used in the metrics), and the FA counterfactual excess_mean is computed from the capped run itself; the scale-invariance derivation and val-loss-neutrality findings are non-circular.

specific steps

self definitional [Section 1.6; Section 2.5; Section 3.3]
"Because the second order is ≥ 0 (Section 1.5), capping only σ1 lets the remaining update pour energy into non-top directions, so trK_Y grows while σ1 is held — p1 falls, H∞ rises indirectly, and H2 and H1 improve for free."

The cap is implemented by projecting out the first-order growth of λ1 (Section 2.1: B=∂λ1/∂W, H←H−max(0,⟨B,H⟩−ρλ1)B/‖B‖²), so λ1 is the direct control target. The headline 'isotropy' metrics are direct functions of that same λ1: top-share = λ1/trK and participation rank = (trK)²/‖K‖²_F = e^{H2}. Holding λ1 while trK continues to grow mechanically lowers top-share and raises PR; the paper itself states this as an automatic consequence ('p1 falls, H∞ rises indirectly, and H2 and H1 improve for free'). Thus the reported isotropy improvements (e.g., Section 2.5: PR 34→96) are partly by construction, not independent empirical predictions. The non-trivial residue is val-loss neutrality, which is measured independently.
other [Section 4.5 (excess_mean verification)]
"excess_mean=X⇔'if the cap did not act this step, the next step's λ1 would rise by X (LR included)'... Verification on L1 H2: at iter 86K, excess_mean=22. From 86K to 100K (14K steps), an uncapped expected rise of ∼10–22 per step (mean ∼15) would accumulate a first-order expected λ increase of ∼200,000+. Instead λfalls by 56K (133K→76K)."

The per-step excess is computed from the actual H and W of the capped run; integrating this local first-order quantity over 14K steps as 'what the uncapped run would do' assumes the capped trajectory's H sequence is the counterfactual uncapped trajectory, which it is not—the cap has already altered W at every previous step. This makes the accumulated counterfactual self-referential rather than an independent estimate of the baseline. The genuine evidence is the separately run uncapped baseline (Section 4.4.4), while the excess_mean arithmetic is a local diagnostic, not a valid long-horizon counterfactual.

full rationale

The central theoretical claims are self-contained derivations from Assumption 1 (Sections 1.2–1.5): the SGD 1/‖W‖ brake, Muon's removal of it via msign scale-invariance, and the non-negative second-order spectral perturbation are obtained by explicit calculation, not by presupposing the experimental results. The empirical check of cG/cP is a stated single-checkpoint sanity check and is not used as a circular proof. There is no load-bearing self-citation: references are external (Muon blog, nanoGPT, the FlashAttention paper). The partial circularity is confined to the empirical framing: because the cap is defined as a projection against the first-order growth of λ1, the metrics used as evidence of 'increased isotropy' (λmax, top-share=λ1/trK, PR=(trK)²/‖K‖²_F) are direct functions of the very eigenvalue being capped; improvements in those metrics follow in large part by construction. The paper's own statement that 'H2 and H1 improve for free' concedes this. The nontrivial, independently measured result is that val loss is essentially unchanged, which supports the 'second order still learns' mechanism. Additionally, the FA excess_mean counterfactual is computed from the capped run's own updates and then integrated over a long window, making it self-referential as a counterfactual; the separate uncapped baseline is the valid comparison. The MoE L0 result (G3 top-share 0.272 vs G1 0.479, but hard-max 0.219 vs 0.071) weakens the claimed geometry-to-load causal chain but is a correctness issue, not circularity. Overall, the derivation chain is not fundamentally circular; score 4 reflects the constructional isotropy metrics and the self-referential counterfactual diagnostic.

Axiom & Free-Parameter Ledger

2 free parameters · 7 axioms · 0 invented entities

Everything load-bearing: Assumption 1 (exact scale invariance) and the trajectory-level constancy claims (C, k, <W, P> ~ 0) that convert the ODE calculations into the t^{1/4}/t^{1/2} rates; the spectral-gap requirement for the perturbation expansions; the reference paper's bf16 failure mechanism imported for the FA narrative; and the K_X = I simplification (explicitly relaxed in Section 1.7, so not load-bearing for experiments). No constants are fitted to data anywhere in the paper; cap thresholds (rho = 0, npower = 4, M = 4096 subsampling) are design choices, not fitted parameters. No new physical entities (particles, forces, dimensions) are postulated; the cap is an update-projection rule and excess_mean is a counterfactual diagnostic estimator.

free parameters (2)

C = ||H(Theta_t)||_F (direction-only gradient norm)
Introduced in Section 1.4.2 as an 'O(1) direction-only quantity' and treated as constant to integrate R-dot = eta^2*C^2/R. Not fitted, but the constancy assumption is required for the SGD t^{1/4} rate.
k = rank(G_t) (Muon update energy)
Section 1.4.3: ||P_t||^2_F = k_t is called 'roughly a constant k', giving R(t) ~ b*t and the t^{1/2} rate. Constancy assumed, not measured.

axioms (7)

ad hoc to paper Exact scale invariance L(cW) = L(W) for all c > 0 (Assumption 1, Section 1.2)
The load-bearing idealization, explicitly called 'a strong idealisation' by the author. Only approximately true (biases, weight decay, residual connections, last unnormalized layer). Empirically probed once via c_G ~ 5e-3 at one checkpoint (Section 1.4.4).
domain assumption C = ||H(Theta_t)||_F stays O(1) along the trajectory
Needed in Section 1.4.2 for the ODE R-dot = a/R; never verified across training.
domain assumption <W_t, msign(G_t)>_F ~ 0 (Muon radial component vanishes)
Used in Section 1.4.3 to get Delta R_Muon ~ ||D||^2 and the t^{1/2} rate. Not a consequence of Assumption 1; empirically |c_P| ~ 0.010 at one checkpoint (Section 1.4.4).
domain assumption k_t = rank(G_t) is roughly constant
Used in Section 1.4.3 for the Muon norm-growth rate.
domain assumption Simple top singular value with spectral gap sigma_1 > sigma_2 throughout
Required for the first/second-order perturbation expansions in Sections 1.4.5 and 1.5.1; the paper notes sigma_1 - sigma_j > 0 is the gap but does not verify it on trajectories.
domain assumption Reference-paper bf16 failure mechanism applies (cross-block max-subtraction loses precision at large score scale with many near-max entries)
The FA rescue narrative imports this causal chain from ref [1] (Section 4.2) and extends it to this setup. The baseline's own lambda/tie diagnostics are never shown, so the chain is not independently confirmed here.
domain assumption K_X = I pedagogical simplification (Section 1.1), relaxed in Section 1.7
Used for the clean K_Y = W W^T statements. The experimental caps target data-coupled K_Y, K_{z+b}, and K_Q, so this simplification does not load the empirical claims.

pith-pipeline@v1.3.0-alltime-deepseek · 21061 in / 39412 out tokens · 369540 ms · 2026-08-01T11:48:26.374820+00:00 · methodology

0 comments

read the original abstract

Muon and related matrix-sign optimizers are increasingly used to pre-train large language models, but their effect on the internal geometry of individual weight matrices is not well understood. This preliminary report proposes a unified framework built on a single idealizing assumption -- exact scale invariance of the loss under weight rescaling, which holds approximately in normalization-heavy networks. Under this assumption, plain SGD carries a built-in 1/||W|| brake on its update size, whereas Muon's matrix-sign step removes that brake, so both the Frobenius and spectral norms drift outward faster (t^{1/2} versus t^{1/4}). We further observe that the spectral-norm perturbation has a non-negative second-order term. This implies that a lightweight "spectral cap" -- which projects out only the first-order growth of the single top singular direction from each update -- can control the output covariance W K_X W^T without freezing training: the weight keeps learning through non-top directions, top-direction rotation, and top switching. We relate this cap to the min-entropy (H-infinity) of the singular-value spectrum. We then study three systems trained with Muon: a nanoGPT feed-forward projection, a 64-expert mixture-of-experts router, and the query/key projections of a bf16 FlashAttention block. In each case the cap increases isotropy and, at the margins -- a router collapsing to a single expert, and the near-divergence of one attention head -- prevents a concrete failure, while leaving validation loss essentially unchanged. We emphasize that the scale-invariance assumption is strong and that these small-scale results are preliminary; comments are welcome.

Figures

Figures reproduced from arXiv: 2607.19771 by Jiachun Li.

**Figure 1.** Figure 1: nanoGPT 50K val-loss curves. D0/D1/D1d essentially overlap over the whole run, with a final difference |∆val| < 0.01 nats. This is the proof of concept: the spectral cap (projecting out first-order top-mode growth) does not hurt val loss, so the model keeps learning through the “update learns new directions in the orthogonal complement” mechanism of Section 1.5. Final val loss @ 50K: D0 = 3.8680, D1 = 3.87… view at source ↗

**Figure 2.** Figure 2: Participation rank tr(KY ) 2/∥KY ∥ 2 F (i.e. e H2 of Section 1.6, max = 384) of the FFN output KY across the six mlp.c_proj layers. D1d (green) raises the L1 PR from D0’s (blue) 34 to 96 (nearly 3×); D1 (red) is in between. L0 is the hardest — all three are near 1 (i.e. KY is essentially rank-1 at L0). 2.6 Part 2 summary • the cap does not hurt val loss ⇒ empirical support for the “second order still learn… view at source ↗

**Figure 3.** Figure 3: seed=42 val-loss trajectory, three methods overlaid. Key observation: during step 10K–30K, G0’s val is slightly below G1/G3 (the cap constrains the router’s directional freedom); during the lr decay at step 40K–50K (warmdown start marked by the dashed line), the capped methods G1/G3 overtake G0. This is the “slower early, more stable late” signature of the cap: it prevents G0’s fast early loss drop bought … view at source ↗

**Figure 4.** Figure 4: seed=42 final per-expert load histogram (3 methods × 6 layers). In each subplot the horizontal axis is the 64 experts (sorted by load), the vertical axis the load fraction, with the ideal 1/E = 0.0156 as a dashed line. L0 (top row): G0 (blue) has its first expert take 25% (the rank-1 collapse, visualised); G1 and G3 both flatten to near-ideal. L1–L4: the three methods are essentially the same. L5 (bottom r… view at source ↗

**Figure 5.** Figure 5: Per-layer load CV (E · std(F)/mean(F), lower is more balanced, ideal = 0). L0: G0 = 1.96 (very imbalanced), G1 = 0.30 (near perfect), G3 = 0.92 (in between). L1 is the reverse: G1 = 1.07 (side effect), G3 = 0.07 (repaired). L5: G3 = 1.50 < G0 = 1.88 < G1 = 2.86. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗

**Figure 6.** Figure 6: Loss trajectory: F3 vs uncapped vs patched [PITH_FULL_IMAGE:figures/full_fig_p022_6.png] view at source ↗

**Figure 7.** Figure 7: Global grad-norm trajectory q (log-y axis, up to 120K). The global grad norm ∥g∥2 = P p ∥∇pL∥ 2 F (over all parameters p, before clipping), one value per train step. Blue = F3 (continuous trace); red dots = the uncapped baseline (one summary point per 1000 iters). Dashed black = grad-clip max-norm = 1.0. The two overlap before 56K (grad norm ∼ 0.4–0.5); from ∼ 56K the uncapped run explodes — 64K=52, 72K=60… view at source ↗

**Figure 8.** Figure 8: The 86K crisis at L1 H2 (this figure is specifically for L1 H2). Left axis: λmax(K L1,H2 Q ) (blue) and λmax(K L1,H2 K ) (red) over the run; right axis: the corresponding q_excess (green) and k_excess (orange). Red shading = the 80K–95K crisis window. λK rises from 873 at 60K to a peak of 133K at 86K (150×), and the cap excess rises in step from 0.13 to 22/step (170×). The cap intercepts hardest exactly at… view at source ↗

**Figure 9.** Figure 9: Two bf16-safety quantities of the L1 H2 attention scores [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

4 extracted references · 2 linked inside Pith

[1]

Qiu and Q

H. Qiu and Q. Yao. Why low-precision transformer training fails: an analysis on Flash Atten- tion. InInternational Conference on Learning Representations (ICLR), 2026. arXiv preprint arXiv:2510.04212

Pith/arXiv arXiv 2026
[2]

Liu et al

J. Liu et al. Muon is scalable for LLM training. arXiv preprint arXiv:2502.16982, 2025

Pith/arXiv arXiv 2025
[3]

K. Jordan. Muon: an optimizer for hidden layers in neural networks.https://kellerjordan. github.io/posts/muon/, 2024

2024
[4]

Karpathy

A. Karpathy. nanoGPT.https://github.com/karpathy/nanoGPT, 2023. 26

2023

[1] [1]

Qiu and Q

H. Qiu and Q. Yao. Why low-precision transformer training fails: an analysis on Flash Atten- tion. InInternational Conference on Learning Representations (ICLR), 2026. arXiv preprint arXiv:2510.04212

Pith/arXiv arXiv 2026

[2] [2]

Liu et al

J. Liu et al. Muon is scalable for LLM training. arXiv preprint arXiv:2502.16982, 2025

Pith/arXiv arXiv 2025

[3] [3]

K. Jordan. Muon: an optimizer for hidden layers in neural networks.https://kellerjordan. github.io/posts/muon/, 2024

2024

[4] [4]

Karpathy

A. Karpathy. nanoGPT.https://github.com/karpathy/nanoGPT, 2023. 26

2023