Pith. sign in

REVIEW 3 major objections 3 minor 25 references

Muon Meets Mamba: Spectral Optimization for State Space Models

T0 review · 3 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Muon's benefit in Mamba-2 is localized to the output projection; better input conditioning does not explain it.

desk verdict A well-run empirical study with a genuinely new localization result—Muon on Mamba-2's output projection helps most—though the comparison is partly confounded by per-regime learning-rate choices and the downstream payoff is flat. read the letter →

arxiv 2608.03941 v1 pith:HSBOPVAG submitted 2026-08-04 cs.LG

Arslan Battalov , Karim Kramin , Alexander Markotenko , Sofia Sinitsina This is my paper
classification cs.LG
keywords MuonAdamWMamba-2statespacemodelsspectraloptimizationNewton-Schulztokenefficiencyoptimizerlocalization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks where, if anywhere, the Muon optimizer helps inside a Mamba-2 language model. By varying only which projection matrices receive Muon, it finds the benefit is concentrated in the output projection: Muon on out_proj alone beats Muon on the input projection or on both, and beats the pure AdamW baseline. The advantage is mainly token efficiency — the winning regime matches AdamW's final validation loss at about 58% of the tokens — and the ordering holds on two corpora and two token budgets, persisting well past the compute-optimal point. The paper also closes off the obvious explanation: Muon lowers the condition number of whichever matrix it trains, so the better-conditioned input projection is not the source of the gain. A careful reader would come away with a concrete recipe (apply Muon to the output projection, not the whole block) and a warning that pre-training loss gains at this scale did not carry to zero-shot downstream accuracy.

What carries the argument

The controlled experimental factor is the assignment of Muon to the two two-dimensional matrix groups in each Mamba-2 block: in_proj (G_in) and out_proj (G_out), giving four regimes k=0..3. Muon's update forms matrix-valued momentum with a Nesterov step, then applies a five-iteration Newton-Schulz polynomial that orthogonalizes the update direction, i.e. steepest descent under the spectral norm; AdamW handles all other parameters. The comparison measures final validation loss, validation-equivalent tokens, and spectral diagnostics (condition number, effective rank, spectral norm) of the two projection groups.

What would settle it

Rerun the four regimes on OpenWebText at 1e9 tokens with per-group learning rates for k=2 (separate rates for G_in and G_out) and a denser k=1 sweep around 1.5e-2 using more than two seeds; if k=2 or k=1 then ties or beats k=3 on final validation loss, the output-projection localization claim collapses.

Watch

Extended reading notes

Core claim

Muon's benefit in a Mamba-2 130M is localized to the output projection. Assigning Muon only to out_proj (k=3) yields the lowest final validation loss of four regimes: ΔL = −0.116 vs AdamW at 10^9 tokens on OpenWebText (3.3019 vs 3.4178), beating input-only (k=1) and joint (k=2). The ranking k=3 < k=2 < k=1 < k=0 is stable across two corpora and two budgets, and at 5×10^10 tokens k=3 still leads AdamW by 0.017. The gain is token efficiency: k=3 reaches AdamW's final 10^9-token loss in ≈5.8×10^8 tokens. Conditioning does not explain this: Muon conditions whichever projection it trains, yet the best-conditioned input projection does not lower loss, while the best-conditioned output projection d

Load-bearing premise

The ranking of the four regimes rests on learning rates picked from a two-seed 2e8-token pilot grid, with k=1 set to 1.0e-2 instead of the plateau minimum 1.5e-2 and k=2 using a single scalar rate for two heterogeneous groups; re-calibration could change which regime wins.

Editorial extensions

If this is right

  • For Mamba-2 training, the default should be Muon on out_proj alone; applying Muon to in_proj or to all eligible matrices is worse than the single-group assignment in every corpus-budget cell tested.
  • At 10^9 tokens, k=3 reaches AdamW's final validation loss in roughly 5.8×10^8 tokens, so localized Muon buys about 40% of the token budget at matched loss.
  • The k=3 advantage persists when training runs to 5×10^10 tokens (~19× Chinchilla), so it is not an artifact of under-training, though the gap narrows as the budget grows.
  • Spectral conditioning is not a reliable guide: Muon improves the condition number of whichever projection it trains, but input-projection conditioning does not improve loss, so optimizer gains should be evaluated on loss curves, not spectra.
  • At 130M, the pre-training loss gap does not transfer to zero-shot tasks (mean accuracy 41.7 vs 42.0), so localized optimizer gains need downstream validation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the proposed mechanism is right — Muon helps a uniform dense projection that writes to the residual stream and hurts a merged row-concatenation of heterogeneous sub-projections — then other architectures with fused QKV or gated projections should show the same pattern; this is testable by repeating the protocol on a Transformer with grouped projection matrices.
  • The k=2 result may be an artifact of a single learning rate applied across heterogeneous groups; the paper's own split-Muon experiment shows per-slice treatment changes the picture, so a per-group-rate k=2 might close the gap to k=3.
  • The token-efficiency saving suggests a practical rule for compute-constrained SSM training: spend Muon only where the matrix writes to the residual stream, and validate with a matched-loss token count rather than final loss alone.
  • Because the 130M zero-shot numbers are near chance, the downstream decoupling should not be taken as evidence that the loss gain is useless at scale; a larger-model replication could plausibly show both better loss and better tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper reports a controlled comparison of Muon and AdamW for training Mamba-2 130M, varying only which of the two matrix-valued projection groups (input projection G_in, output projection G_out, or both) are trained with Muon. Four regimes are compared: k=0 (all AdamW), k=1 (Muon on G_in), k=2 (Muon on G_in ∪ G_out), and k=3 (Muon on G_out). Learning rates for AdamW and for each Muon regime are selected from two-seed pilot sweeps at 2e8 tokens, then frozen for full runs at 1e9 and 2.6e9 tokens on OpenWebText and FineWeb-Edu. The main claims are: (i) k=3 achieves the lowest validation loss in all corpus–budget cells (e.g., ΔL = -0.116 vs AdamW at 1B tokens on OpenWebText); (ii) the ordering k=3 < k=2 < k=1 < k=0 is stable across corpora and budgets at the endpoints; (iii) the benefit is mainly token-efficiency, with k=3 matching AdamW's final 1B-token loss at ~5.8e8 tokens; (iv) spectral diagnostics show Muon lowers the condition number of whichever projection it is applied to, but the better-conditioned G_in does not yield the largest loss improvement; and (v) a single-seed run to 5e10 tokens on FineWeb-Edu shows the k=3 vs k=0 gap persists (2.8853 vs 2.9027). The paper is explicitly framed as empirical and includes several negative results (downstream evaluation, pilot-scale ablation reversal).

Significance. If the results are correct, this is a useful empirical data point for optimizer placement in state-space models: the Muon benefit is not uniform across Mamba-2's projection matrices, and the output projection is the most effective target. The paper's protocol is disciplined in several respects: learning rates were frozen before full runs, the AdamW baseline is an external reference, paired seeds are used, a fixed checkpoint grid is specified, and the authors report negative results (no downstream gain, pilot-scale artifacts) rather than only favorable outcomes. The spectral diagnostics, while interpretive, provide a concrete metric-based attempt to explain the effect. However, the study is narrow (130M-parameter model, two shared seeds, no significance tests, single-seed 50B run) and the central localization claim is partly confounded by per-regime learning-rate tuning. The contribution is therefore a plausible and honestly reported empirical finding rather than a definitive or theoretically grounded result.

major comments (3)
  1. [Section IV / Table VI] The paper states that the controlled experimental factor is the Muon assignment and that learning rates are 'held fixed across regimes' (Section IV), but Table VI shows a different Muon learning rate for each regime (η=1e-2 for k=1, 2e-2 for k=2, 1.5e-2 for k=3). The comparison k=3 vs k=2 vs k=1 is therefore a joint comparison of assignment and per-regime learning rate, not an assignment-only comparison. This is load-bearing for the central claim that 'out_proj alone beats both': the abstract and Section X concede for k=2 that a single scalar rate serving heterogeneous sub-blocks could explain its underperformance, but the same caveat applies to the entire ranking. I recommend either (a) adding sensitivity runs that evaluate each assignment at a common Muon learning rate (e.g., η=1.5e-2 for all k), or (b) explicitly reframing the abstract and conclusions to say the result is established
  2. [Appendix A / Section X] The pilot grid uses only two seeds, and the k=1 learning rate was deliberately set at 1.0e-2 instead of the plateau minimum 1.5e-2 because the plateau minimum showed higher seed variance (Appendix A). The difference between these two settings (ΔL=0.013) is smaller than the seed standard deviation at η=1.5e-2 (0.028), so the choice is defensible, but it means the reported k=1 loss is not the best this regime can achieve under the pilot's own criterion. More generally, the rank ordering of the intermediate regimes (k=1 vs k=2) is not stable: at 2.6B tokens the two coincide on OpenWebText and reverse on FineWeb-Edu (Table III). Since the paper's headline claim is the localization to G_out, at minimum the authors should report the full pilot table (all grid points, both seeds) and perform a simple paired significance check (e.g., a paired permutation test across seeds) for the k=3 vs k=2 dif
  3. [Section VIII] The 5e10-token run is single-seed, and no error bars or other uncertainty estimates are reported for the final gap of 0.017 (2.8853 vs 2.9027). At 130M scale, single-seed loss differences of this magnitude can be within run-to-run noise; the two-seed standard deviations in Table II (0.001–0.010) do not transfer to the much longer run because the LR schedule and data order differ. Additionally, this run only compares k=0 and k=3, so it does not test the localization claim at scale (k=2 and k=1 are absent). I suggest either adding a second seed for the 50B runs or, if that is computationally prohibitive, framing Section VIII explicitly as a single-seed exploratory observation rather than as confirmation of the localization claim.
minor comments (3)
  1. [Section III / Appendix A] The reference AdamW learning rate is given as α0=3e-3 in Section III, but the actual selected rate is α*=3.6e-3 (Appendix A). Clarify the relationship between the reference and the calibrated value to avoid confusion.
  2. [Table III / Section VI-C] The bottom row of Table III is labeled 'ΔL(k=0−k=3)' and reports positive numbers (+0.116, +0.063, ...), whereas Table II defines ΔL relative to k=0 and reports negative values for improvements. Use a consistent sign convention, e.g., report ΔL = L_k − L_0 for all rows.
  3. [Appendix D] The text 'splitG in andMuon onG out' appears to be missing a space; also consider defining 'split-Muon' formally before first use. Minor presentation issue.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical protocol is self-contained and no claimed result reduces to its inputs by construction.

full rationale

I walked the paper's derivation chain. The regimes M(k) are defined by assigning an externally defined optimizer (Muon) to predefined matrix groups Gin/Gout, with all other parameters on AdamW. Learning rates for each regime were selected in a pilot sweep and frozen before the full-budget runs; the final validation losses are measured outcomes, not algebraic consequences of the pilot objective. The token-efficiency claim is computed by linear interpolation of measured validation-loss curves, so it is an interpolation of data, not a fitted parameter renamed as a prediction. The spectral diagnostics are explicitly interpretive ('not primary success metrics') and are used to test, not to define, the loss ranking. There are no self-citations: references [2], [3], [16], [17], and [18] are external works by Jordan, Liu et al., and Bernstein/Newhouse, with no author overlap with this paper, so no load-bearing self-citation chain exists. The manuscript's own conceded limitation—that k=2's underperformance might reflect a single scalar Muon learning rate applied to heterogeneous sub-blocks (Section X) and that k=1's rate was set off the plateau minimum (Appendix A)—is a hyperparameter-calibration confound, not circularity: the reported regime ranking is not equivalent to the pilot loss by construction, and the k=3 advantage persists across two corpora, three budgets, and a 5e10-token run with rates frozen. No equation or definition makes any claimed result equal to its own input. Finding: no significant circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or theoretical entities are introduced; the paper only reuses existing optimizers and architectures. The main load-bearing inputs are the tuned learning rates and the adopted reference implementations.

free parameters (4)
  • AdamW learning rate alpha = 3.6e-3
    Selected on OWT pilot at 2e8 tokens over an 8-point grid (Appendix A); held fixed for all regimes and corpora.
  • Muon LR k=1 = 1.0e-2
    Chosen within a plateau rather than at the raw plateau minimum 1.5e-2 due to lower seed variance (Appendix A); affects the k=1 comparison.
  • Muon LR k=2 = 2.0e-2
    Unambiguous pilot minimum for the joint in_proj+out_proj assignment (Appendix A).
  • Muon LR k=3 = 1.5e-2
    Pilot minimum for out_proj-only assignment; lowest pilot loss across regimes (Appendix A).
assumptions (5)
  • domain assumption Newton-Schulz coefficients (a,b,c)=(3.4445,-4.7750,2.0315) with NNS=5 yield the orthogonal polar factor
    Taken from Jordan's Muon reference [2],[3]; not re-derived in this paper.
  • domain assumption Mamba-2 130M block structure and official implementation match the cited reference [1],[6]
    The eligible matrix groups Gin/Gout are defined relative to this implementation.
  • ad hoc to paper Linear interpolation between 11 checkpoints estimates validation-equivalent tokens
    Used to quantify token efficiency; assumes smooth monotone loss curves.
  • ad hoc to paper Pilot LR selection at 2e8 tokens transfers to 1B and 2.6B budgets and to FineWeb-Edu
    A reduced transfer check was run, but not full per-budget per-corpus sweeps.
  • ad hoc to paper Two shared seeds (42,1337) are treated as descriptive replicates, not statistical tests
    The paper explicitly describes its paired comparisons as descriptive rather than formal significance tests (Section V).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Muon Meets Mamba: Spectral Optimization for State Space Models." pith.science (2026). https://pith.science/paper/HSBOPVAG

@misc{pith2026260803941,
  author       = {Pith},
  title        = {Pith review of: Muon Meets Mamba: Spectral Optimization for State Space Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HSBOPVAG}},
  note         = {Machine review of arXiv:2608.03941}
}
read the original abstract

Muon is a recent optimizer that orthogonalizes the update to each weight matrix with a Newton-Schulz iteration, which performs steepest descent under the spectral norm. Almost all the evidence for it comes from Transformer models, and its behavior on state-space models is largely unreported. We compare Muon with AdamW on Mamba-2 130M under a controlled protocol that varies only which weight groups are trained with Muon. The benefit is localized. Muon on the output projection alone beats Muon on the input projection or on both. The advantage is mainly one of token efficiency. It holds on two corpora and two token budgets, and persists when training continues well past the compute-optimal point. Conditioning does not explain the gain. Muon lowers the condition number of whichever projection it trains, but the better-conditioned input projection is not the one that helps.

Figures

Figures reproduced from arXiv: 2608.03941 by the authors.

Figure 1
Figure 1. Mamba-2 block (single layer ℓ) with the two-dimensional, Muon-eligible projection matrices highlighted. Muon is assigned to blue Gin at k = 1 and k = 2, and to green Gout at k = 2 (jointly with Gin) and at k = 3 (alone). Gray cells are parameters always trained with AdamW: one-dimensional SSD and normalization parameters, three-dimensional depthwise convolution filters, embeddings, the language-modeling head, and bi… view at source ↗
Figure 4
Figure 4. Smoothed training loss over the late training phase ( [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 3
Figure 3. Smoothed training loss (rolling mean over 50 steps), averaged over [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (14 more)
Figure 7
Figure 7. Figure 7: Smoothed training loss (rolling mean over 50 steps) for the three [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 6
Figure 6. Figure 6: Validation loss at checkpoints Tck for the three remaining corpus– budget cells, averaged over seeds {42, 1337} (the primary OpenWebText 109 - token run is [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 10
Figure 10. Figure 10: Layer-averaged spectral metrics for Gout over training, seed 42. Panels as in [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 9
Figure 9. Figure 9: Layer-averaged spectral metrics for Gin over training, seed 42. The four panels show spectral norm σ1, Frobenius norm, effective rank, and condition number κ. X. Discussion Across our four assignment regimes, Muon on out_proj alone (k = 3) consistently outperforms the …
Figure 11
Figure 11. Figure 11: Training (left) and validation (right) loss for [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 13
Figure 13. Figure 13: Muon learning-rate sweep at the pilot budget [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]
Figure 14
Figure 14. Figure 14: Validation loss for Muon on in_proj sub-blocks added on top of k = 3 (late-phase zoom). Left: OWT pilot (2 × 108 tokens); only x, B, C falls below the k = 3 reference (dashed). Right: FineWeb-Edu at 2.6 × 109 tokens; the ordering inverts and + x, B, C ends above k = 3…
Figure 15
Figure 15. Figure 15: Smoothed training loss (rolling mean over 50 steps) for the sub [PITH_FULL_IMAGE:figures/full_fig_p012_15.png]
Figure 16
Figure 16. Figure 16: Per-layer spectral profile for Gin at Tmax, seed 42. Per-layer profiles.: The per-layer profiles are consistent across all 24 layers with no systematic depth dependence, and the half-budget profiles are visually indistinguishable from the final ones; the spectral stru…
Figure 19
Figure 19. Figure 19: Per-layer Gout spectral profile at the final checkpoint, OpenWebText at 2.6 × 109 tokens, seed 42. tokens seen (B) 0 5 10 spectral norm 1 tokens seen (B) 0 20 40 60 80 Frobenius norm W F 0.0 0.5 1.0 tokens seen (B) 625 650 675 700 effective rank 0.0 0.5 1.0 tokens see…
Figure 22
Figure 22. Figure 22: Layer-averaged Gout spectral metrics over training, FineWeb-Edu at 2.6 × 109 tokens, seed 42. Appendix D Localizing the input-projection effect The input projection in_proj (Gin) is a single 3352 × 768 matrix that row-stacks five functionally distinct sub￾matrices Wz,…
Figure 23
Figure 23. Figure 23: Per-layer Gout spectral profile at the final checkpoint, FineWeb-Edu at 2.6 × 109 tokens, seed 42. z x B C Δt in_proj sub-block 10 1 6 × 10 0 2 × 10 1 3 × 10 1 4 × 10 1 condition number κ OWT, 2.6B z x B C Δt in_proj sub-block FineWeb, 1B z x B C Δt in_proj sub-block …
Figure 24
Figure 24. Figure 24: Layer-averaged condition number κ of the five in_proj row-slices (z, x, B, C, ∆t) at the final checkpoint, by regime and corpus–budget cell (seed 42). Muon on in_proj (k = 1, k = 2) conditions the ill-posed z and x slices but not the already-isotropic B, C, ∆t. run, s…
Figure 25
Figure 25. Figure 25: Per-slice condition number κ (left) and effective-rank fraction (right) of the five in_proj slices under split-Muon (OWT, 2.6 × 109 tokens, mean over seeds). The gate Wz and the data channel Wx are the two worst-conditioned slices yet have opposite usefulness (removin…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 10 canonical work pages

  1. [1]

    state-spaces/mamba,

    state-spaces, “state-spaces/mamba,” https://github.com/state-spaces/ mamba, 2024, official Mamba implementation repository, accessed 2026-04-24

    work page 2024

  2. [2]

    Muon: An optimizer for hidden layers in neural networks,

    K. Jordan, “Muon: An optimizer for hidden layers in neural networks,” https://kellerjordan.github.io/posts/muon/, Dec. 2024, blog post. Ac- cessed: 2026-04-23

    work page 2024

  3. [3]

    Kellerjordan/Muon,

    ——, “Kellerjordan/Muon,” https://github.com/KellerJordan/Muon, 2024, public Muon implementation repository, accessed 2026-04-24

    work page 2024

  4. [4]

    Efficiently modeling long sequences with structured state spaces,

    A. Gu, K. Goel, and C. R ´e, “Efficiently modeling long sequences with structured state spaces,”arXiv preprint arXiv:2111.00396, 2021

    arXiv 2021

  5. [5]

    Mamba: Linear-time sequence modeling with selective state spaces,

    A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,”arXiv preprint arXiv:2312.00752, 2023

    arXiv 2023

  6. [6]

    Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality,

    T. Dao and A. Gu, “Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality,” inProceedings of the 41st International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol

    work page

  7. [7]

    An em- pirical study of mamba-based language models,

    R. Waleffe, W. Byeon, D. Riach, B. Norick, V. Korthikanti, T. Dao, A. Gu, A. Hatamizadeh, S. Singh, D. Narayananet al., “An em- pirical study of mamba-based language models,”arXiv preprint arXiv:2406.07887, 2024

    arXiv 2024

  8. [8]

    Can mamba learn how to learn? a comparative study on in-context learning tasks,

    J. Park, J. Park, Z. Xiong, N. Lee, J. Cho, S. Oymak, K. Lee, and D. Papailiopoulos, “Can mamba learn how to learn? a comparative study on in-context learning tasks,”arXiv preprint arXiv:2402.04248, 2024

    arXiv 2024

Show all 25 references
  1. [9]

    Jamba: A hybrid transformer-mamba language model,

    O. Lieber, B. Lenz, H. Bata, G. Cohen, J. Osin, I. Dalmedigos, E. Safahi, S. Meirom, Y. Belinkov, S. Shalev-Shwartzet al., “Jamba: A hybrid transformer-mamba language model,”arXiv preprint arXiv:2403.19887, 2024

    2024 arXiv

  2. [10]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in International Conference on Learning Representations, 2019. [Online]. Available: https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  3. [11]

    Why transformers need adam: A hessian perspective,

    Y. Zhang, C. Chen, T. Ding, Z. Li, R. Sun, and Z.-Q. Luo, “Why transformers need adam: A hessian perspective,”Advances in neural information processing systems, vol. 37, pp. 131 786–131 823, 2024

    2024

  4. [12]

    Heavy- tailed class imbalance and why adam outperforms gradient descent on language models,

    F. Kunstner, A. Milligan, R. Yadav, M. Schmidt, and A. Bietti, “Heavy- tailed class imbalance and why adam outperforms gradient descent on language models,”Advances in Neural Information Processing Systems, vol. 37, pp. 30 106–30 148, 2024. z x B C Δt in_proj slice 0 5 10 15co...

    2024

  5. [13]

    Deconstructing what makes a good optimizer for language models,

    R. Zhao, D. Morwani, D. Brandfonbrener, N. Vyas, and S. Kakade, “Deconstructing what makes a good optimizer for language models,” arXiv preprint arXiv:2407.07972, 2024

    2024 arXiv

  6. [14]

    Scalable second order optimization for deep learning,

    R. Anil, V. Gupta, T. Koren, K. Regan, and Y. Singer, “Scalable second order optimization for deep learning,”arXiv preprint arXiv:2002.09018, 2020

    2002 arXiv

  7. [15]

    A distributed data-parallel pytorch implementation of the distributed shampoo optimizer for training neural networks at-scale,

    H.-J. M. Shi, T.-H. Lee, S. Iwasaki, J. Gallego-Posada, Z. Li, K. Ran- gadurai, D. Mudigere, and M. Rabbat, “A distributed data-parallel pytorch implementation of the distributed shampoo optimizer for training neural networks at-scale,”arXiv preprint arXiv:2309.06497, 2023

    2023 arXiv

  8. [16]

    Muon is scalable for llm training,

    J. Liu, J. Su, X. Yao, Z. Jiang, G. Lai, Y. Du, Y. Qin, W. Xu, E. Lu, J. Yanet al., “Muon is scalable for llm training,”arXiv preprint arXiv:2502.16982, 2025

    2025 arXiv

  9. [17]

    Old optimizer, new norm: An anthol- ogy,

    J. Bernstein and L. Newhouse, “Old optimizer, new norm: An anthol- ogy,”arXiv preprint arXiv:2409.20325, 2024

    2024 arXiv

  10. [18]

    Modular duality in deep learning,

    ——, “Modular duality in deep learning,”arXiv preprint arXiv:2410.21265, 2024

    2024 arXiv

  11. [19]

    Exact solutions to the nonlinear dynamics of learning in deep linear neural networks,

    A. M. Saxe, J. L. McClelland, and S. Ganguli, “Exact solutions to the nonlinear dynamics of learning in deep linear neural networks,”arXiv preprint arXiv:1312.6120, 2013

    2013 arXiv

  12. [20]

    Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning,

    C. H. Martin and M. W. Mahoney, “Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning,”Journal of Machine Learning Research, vol. 22, no. 165, pp. 1–73, 2021

    2021

  13. [21]

    The effective rank: A measure of effective dimensionality,

    O. Roy and M. Vetterli, “The effective rank: A measure of effective dimensionality,” in2007 15th European signal processing conference. IEEE, 2007, pp. 606–610

    2007

  14. [22]

    OpenWebText cor- pus,

    A. Gokaslan, V. Cohen, E. Pavlick, and S. Tellex, “OpenWebText cor- pus,” https://skylion007.github.io/OpenWebTextCorpus/, 2019, dataset release

    2019

  15. [23]

    The FineWeb datasets: Decanting the web for the finest text data at scale,

    G. Penedo, H. Kydl ´ıˇcek, A. Lozhkov, M. Mitchell, C. Raffel, L. v. Werra, and T. Wolf, “The FineWeb datasets: Decanting the web for the finest text data at scale,”arXiv preprint arXiv:2406.17557, 2024

    2024 arXiv

  16. [24]

    Training compute-optimal large language models,

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Ruther- ford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clarket al., “Training compute-optimal large language models,”arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  17. [235]

    10 041–10 071

    PMLR, 2024, pp. 10 041–10 071. [Online]. Available: https: //proceedings.mlr.press/v235/dao24a.html

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.