Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Structure-aware optimizers Muon and SOAP make LLM pretraining stable at batch sizes up to 100M tokens, a regime where AdamW degrades.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 06:38 UTC pith:3OM7TVIE

load-bearing objection The SOAP stability fix and layer-wise distributed optimizer are real contributions, but the RMS-matching argument behind the headline performance claims has a math gap, so treat the benchmark comparisons as provisional. the 4 major comments →

arxiv 2607.20548 v1 pith:3OM7TVIE submitted 2026-07-13 cs.LG cs.AI

SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales

Mikail Khona , Aditya Vavre , Boxiang Wang , Deyu Fu , Hao Wu , Mike Chrzanowski , Bryan Catanzaro , Dheevatsa Mudigere
show 6 more authors
Jeff Pool Michael Lightstone Mohammad Shoeybi Mostofa Patwary Nima Tajbakhsh Tijmen Blankevoort
This is my paper
classification cs.LG cs.AI
keywords optimizerMuonSOAPAdamWlarge-batch trainingLLM pretrainingMixture-of-Expertspreconditioned gradient methods
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that structure-aware optimizers Muon and SOAP (and their KL-enhanced variant) are not just faster but more scalable than AdamW: at global batch sizes up to 100M tokens, they keep training stable and token-efficient while AdamW's loss degrades. The authors identify the cause of SOAP's large-batch instability as a stale preconditioner, fix it with per-step QR eigenbasis updates and KL-divergence covariance accumulation, and back the fix empirically on multi-billion-parameter dense and MoE models. They also contribute a layer-wise distributed optimizer that preserves full 2D weight matrices and hides communication, removing the systems objection to higher-order optimizers. A sympathetic reader would care because this shifts the practical frontier of LLM pretraining: larger batches, fewer communication bottlenecks, and a concrete replacement for AdamW at scale.

Core claim

The core discovery is that the limiting 'critical batch size' phenomenon is optimizer-specific: AdamW's element-wise second-moment estimator becomes poorly calibrated at large batch, whereas spectral and tensor preconditioners structurally normalize update directions. Using update-RMS matched learning rates, Muon and SOAP consistently beat AdamW in training loss and downstream benchmarks across dense 8B, 30B-A3B, and 72B-A8B hybrid MoE models, and remain stable up to 100M-token batches. For SOAP, the paper pinpoints the instability to the eigenbasis update omitting the current step's gradient; per-step QR with current-gradient inclusion plus KL-Shampoo's covariance update (with condition num

What carries the argument

The load-bearing identity is the SVD equivalence: with momentum off, Shampoo/SOAP's whitening reduces to U V^T, the polar factor that Muon approximates via Newton-Schulz iterations, so all three optimizers are doing spectral update shaping. The comparison protocol is carried by update-RMS matching: eta' = eta sqrt(B'/B) for batch scaling and a momentum damping factor sqrt((1-beta1)/(1+beta1)) applied to Muon so that all optimizers have matched update norms. Stability is carried by per-step QR recomputation of SOAP's eigenbasis using the current gradient, and by KL-divergence-regularized Kronecker factor accumulation, which lowers the condition number of the matrices being eigen-decomposed. T

Load-bearing premise

The comparison rests on update-RMS matching with square-root batch-size scaling, plus fixed epsilon, betas, and LR schedules across optimizers; if this transfer does not equate effective step quality, the Muon/SOAP advantage over AdamW could be a learning-rate artifact rather than a property of the optimizer.

What would settle it

Train the same 30B-A3B MoE model at 50M-100M token batch with AdamW after a per-optimizer grid search over epsilon, beta1, beta2, and LR; if a tuned AdamW matches Muon/SOAP's loss and stability, the claimed large-batch advantage collapses. Alternatively, compare with linear LR scaling instead of square-root; if the apparent Muon advantage is reversed under the alternative transfer rule, the protocol assumption is doing the work.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Large-batch pretraining becomes viable at 100M-token global batches without the usual quality cliff, cutting data-parallel communication overhead and enabling better GPU utilization.
  • KL-SOAP is the recommended default where memory is not the binding constraint; Muon remains the lower-memory alternative.
  • The stale-preconditioner fix generalizes: any Shampoo-family or preconditioner-based optimizer should update its basis every step with the current gradient when training at large batch.
  • Hybrid recipes that route only certain tensor types to AdamW (e.g., 1D convolution filters) and the rest to Muon/SOAP perform best for mixed architectures.
  • If AdamW's degradation is optimizer-inherent, these results imply the effective critical batch size frontier can be extended by swapping the optimizer rather than by changing the model.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's fairness protocol leaves AdamW's epsilon, betas, and LR schedule untuned at large batch; a tuned AdamW might narrow the gap, so the strongest defensible claim is 'untuned AdamW degrades' rather than 'AdamW cannot scale'.
  • The square-root batch-size scaling rule may itself favor optimizers that are insensitive to update noise; an alternative transfer rule (e.g., linear scaling or per-optimizer noise matching) could change the ranking.
  • Applying the same per-step-QR plus KL covariance recipe to other optimizers (e.g., Eigen-corrected Shampoo or DASH) is a cheap test with potentially large payoff for large-batch stability.
  • The 100M-token result suggests spectral optimizers may tolerate batch sizes far beyond what scaling laws for AdamW predict; measuring the critical batch size of Muon/SOAP across model scales is a natural next experiment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper adapts higher-order optimizers (Muon, SOAP, and a KL-Shampoo variant) to large-scale LLM pretraining, with three main contributions: (i) a diagnosis of SOAP instabilities at large batch sizes and algorithmic fixes (per-step QR eigenbasis updates and KL-Shampoo covariance estimation), (ii) an empirical comparison of Muon, SOAP, and AdamW under an update-RMS-matching protocol claiming superior stability and token efficiency up to 100M-token batches, and (iii) a layer-wise distributed optimizer implementation for Megatron-LM with overlapping communication. The paper reports experiments on 8B dense and 30B/72B MoE models up to 3T tokens, including MXFP8 training, and releases a codebase.

Significance. If the central claims hold, the paper would provide strong evidence that spectral/tensor optimizers can replace AdamW in large-batch LLM pretraining, with a practical distributed implementation. The paper's strengths include a clear empirical diagnosis of SOAP's stale-preconditioner instability (Figs. 4-6), the introduction of KL-SOAP as a stable variant, a substantial large-scale experimental effort, and an open-source implementation. The stability findings and the system contribution are valuable even if the head-to-head comparison is later qualified. However, the fairness of the optimizer comparison rests on a questionable RMS-matching argument, and the headline 100M-token claim lacks a direct AdamW baseline at that batch size.

major comments (4)
  1. [Sec. 5.2] The RMS-matching argument for SOAP vs. AdamW is incomplete. The displayed derivation only proves ||ΔW_SOAP||_F = ||N||_F, i.e., that rotation preserves the norm of SOAP's internal Adam update N. It does not prove N equals the AdamW update or that the two optimizers have equal update RMS. For a rectangular layer, after rotation to the eigenbasis the Adam operation acts on approximately r nonzero diagonal entries, so ||ΔW_SOAP||_F ≈ η√r whereas ||ΔW_AdamW||_F ≈ η√(mn); these differ by a factor depending on the layer shape. No compensating scaling is introduced in Algorithm 2. Thus the learning-rate transfer between SOAP and AdamW is not established, and the loss differences in Figs. 7 and 8 could be effective-LR artifacts. Please either prove the equality under the actual update rules, measure per-layer update RMS in the experiments, or adjust the protocol accordingly.
  2. [Abstract / Sec. 5.3 / Fig. 7] The claim that 'AdamW degrades' at batch sizes up to 100M tokens is not directly supported. The 4x GBS experiment in Fig. 7 (100M tokens) compares only Muon, SOAP, and MOP; no AdamW run at that batch size is shown. Figure 10 shows AdamW at 25M and 50M tokens for a different model (Nano-V3), not at 100M. To substantiate the headline claim, add an AdamW baseline at the largest batch size or weaken the claim to the batch sizes actually tested.
  3. [Figs. 1, 2, 7, 8 / Table 5] All optimizer comparisons are single runs without seeds, error bars, or other uncertainty quantification. Given the paper's claim that SOAP/Muon 'consistently outperform' AdamW, and several differences are small (e.g., Table 5, Commonsense Avg. gaps below 1 point), the absence of repeated runs makes the consistency claim fragile. At least for the smaller models (Qwen-3-30B-A3B, 621M) or for a subset of configurations, please report multiple seeds with mean/standard deviation, or explicitly state that the differences are from single runs and interpret them accordingly.
  4. [Appendix A.3, Eq. (13)] The condition-number reduction claim κ(S_KL-Shampoo) = sqrt(κ(S_Shampoo)) is derived under fixed-point and balanced-spectrum assumptions (Λ_a ≈ Λ_b ≈ Σ) that are stated but not justified for the actual training dynamics. This is presented as an explanatory result rather than a formal theorem, which is acceptable, but the paper should clearly label it as a consistency argument, not a proof, and note that the empirical stability benefit in Figs. 4-6 is the primary evidence. If the theoretical claim is meant to be load-bearing, a convergence or perturbation analysis would be needed.
minor comments (5)
  1. [Sec. 5.2] The notation ΔW is used both for the generic parameter update and for SOAP's rotated update; please define consistently. In Eq. (1), u_t and ΔW are conflated.
  2. [Table 1] The text says 'All models use 32 heads with head dimension 128,' but the GQA column lists values 2, 8, 8, 4. This is confusing; clarify whether 32 is the number of query heads and GQA is the number of KV groups.
  3. [Sec. 5.2] The Muon correction factor sqrt((1-β1)/(1+β1)) ≈ 0.2 is introduced without derivation or citation to the exact formula. Please state the underlying assumption or provide a reference to the derivation.
  4. [Sec. 5.4.1] The term 'slingshot instability' is used but not formally defined, and no comparison to previously reported slingshot phenomena is given. A brief definition or reference would help.
  5. [Table 7] The last column header 'MXFP8 (Lion)' is not explained in the text; clarify whether this is an additional baseline or a variant of the Muon recipe.

Circularity Check

0 steps flagged

No significant circularity: central claims are empirical comparisons; no prediction is equivalent to fitted inputs or self-citations by construction.

full rationale

The paper's load-bearing claims are empirical: Muon/SOAP stability and loss advantages are measured training trajectories and benchmark scores, not quantities derived from fitted parameters or from the authors' own prior results. The only derivation-like result, the KL-Shampoo condition-number reduction in App. A.3 (Eq. 4/13), is a consistency calculation under an explicit SVD/fixed-point assumption; it is not a prediction-from-fit and does not smuggle in the conclusion. The update-RMS matching protocol (Sec. 5.2) is a hyperparameter-transfer heuristic, not a circular step: even though the displayed norm identity only establishes ||ΔW_SOAP||_F = ||N||_F and the further equality with AdamW's update norm is questionable, the reported losses are not forced by that identity; any mismatch is a fairness/correctness concern rather than a reduction of the conclusion to its inputs. Citations such as Nemotron-H [46] and Kimi [13] are external empirical anchors/methods, not self-citation chains that substitute for derivation. The paper's own limitation statement (epsilon not tuned) is an honest disclaimer, not evidence of circularity. Therefore no circular step meets the required standard of exhibiting Eq. X = Eq. Y by construction or a fitted-input-called-prediction.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The empirical claims do not introduce invented entities. Free parameters are mostly standard hyperparameters, but epsilon and per-model LRs are explicitly admitted as untuned, so they carry weight in the comparison. The KL-Shampoo condition-number result is derived under an idealized fixed-point assumption, not a fitted prediction.

free parameters (6)
  • epsilon (SOAP/AdamW and Muon) = 1e-8 (SOAP/AdamW), 1e-7 (Muon)
    Numerical floors; authors explicitly state epsilon is not systematically tuned (Sec 5.5 'Limitations of our work'), yet it affects stability and optimizer comparisons.
  • Learning rates and min LR per model = e.g. 8e-4/8e-6, 1e-3/1e-5, 1.6e-3/1.6e-5
    Hand-selected or borrowed from baselines; central to fairness of optimizer comparison (Table 3).
  • Momentum/EMA coefficients beta1, beta2, beta_kron = 0.9, 0.95, 0.95
    Standard values chosen without tuning; affect update magnitude and stability (Tables 4 and 6).
  • Newton-Schulz iterations and coefficients = n=16; PolarExpress coefficients
    Chosen orthogonalization budget; quality and cost of Muon's orthogonalization depend on it (Table 4).
  • Batch-size ramp-up schedule = start 2M tokens, double every S steps until target; eta proportional to sqrt(B)
    Hand-designed and used only for large-batch Muon runs; improves final results (Sec 5.3, Fig 3).
  • KL-Shampoo exponent p = -1
    Algorithmic choice in the KL covariance update; not fitted, but central to the KL-SOAP variant.
axioms (5)
  • domain assumption Per-sample gradients are independent, so Cov(g_B) = Sigma / B (Sec 3.2).
    Basis of the square-root batch-size scaling rule; false when gradients within a batch are correlated.
  • domain assumption Update-RMS matching plus square-root LR scaling is a fair learning-rate transfer across AdamW, SOAP, and Muon.
    No validation that matched update RMS equates optimizer effectiveness; central to the comparison fairness (Sec 3.2, Sec 5.2).
  • domain assumption Training loss correlates with held-out validation and downstream quality.
    Stated in Sec 5.3; used to treat loss curves as evidence of quality.
  • ad hoc to paper In Appendix A.3, the KL-Shampoo fixed point has balanced spectra, Lambda_a approx Lambda_b approx Sigma.
    Needed for the condition-number reduction claim (Eq. 12-13); not independently verified and not a general property of training dynamics.
  • domain assumption Layer-wise distribution of full parameter matrices preserves optimizer math without approximations.
    Underlies the system contribution in Sec 3.3 and Sec 6; relies on variable-sized allgather correctness.

pith-pipeline@v1.3.0-alltime-deepseek · 20568 in / 13900 out tokens · 141684 ms · 2026-08-02T06:38:56.006516+00:00 · methodology

0 comments
read the original abstract

Higher-order optimizers such as Muon and SOAP offer faster convergence than AdamW, but their computational cost and numerical stability challenges have limited adoption at scale. In this work, we adapt and enhance preconditioned gradient methods to overcome the practical challenges of large-scale LLM pretraining. We first identify instabilities in SOAP at large batch sizes and propose algorithmic modifications including per-step QR orthogonalization and improved preconditioning strategies that eliminate loss spikes and enable stable training in these regimes. We then present a unified empirical study of SOAP, Muon, and AdamW using update-RMS matching to ensure fair learning rate transfer across optimizers. As part of this analysis, we empirically evaluate the orthogonalization quality of Muon. Our experiments on multi-billion-parameter models trained on trillions of tokens reveal that SOAP and Muon consistently outperform AdamW at the scales we tested. Notably, at batch sizes of up to 100M tokens for next-token prediction, these optimizers maintain training stability and quality while AdamW degrades. To enable efficient training at large scale, we introduce a layer-wise distributed optimizer compatible with Megatron-LM. Our implementation balances memory and hides communication while avoiding approximations to the optimizer computations, thus retaining their convergence benefits. Additionally, we identify and build specific system-level improvements to further accelerate our layer-wise implementation. To support the research community, we release a codebase that contains emerging algorithms for optimization: https://github.com/NVIDIA-NeMo/Emerging-Optimizers

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. When Does Muon Help Agentic Reinforcement Learning?

    cs.LG 2026-07 conditional novelty 6.0

    Under a shared KL/clipping recipe in agentic RL, fan-in Muon at 3e-5 delivers a larger stable update and improves late success over a fixed AdamW 1e-6 baseline, with the effect tied to update magnitude rather than spe...

Reference graph

Works this paper leans on

74 extracted references · 38 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    Pith/arXiv arXiv 2014

  2. [2]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Pith/arXiv arXiv 2017

  3. [3]

    Neural networks for machine learning lecture 6a overview of mini-batch gradient descent.Cited on, 14(8):2, 2012

    Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent.Cited on, 14(8):2, 2012

    2012

  4. [4]

    Laprop: Separating momentum and adaptivity in adam.arXiv preprint arXiv:2002.04839, 2020

    Liu Ziyin, Zhikang T Wang, and Masahito Ueda. Laprop: Separating momentum and adaptivity in adam.arXiv preprint arXiv:2002.04839, 2020

    Pith/arXiv arXiv 2002

  5. [5]

    Fast approximate natural gradient descent in a kronecker factored eigenbasis.Advances in neural information processing systems, 31, 2018

    Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent in a kronecker factored eigenbasis.Advances in neural information processing systems, 31, 2018

    2018

  6. [6]

    Optimizing neural networks with kronecker-factored approxi- mate curvature

    James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approxi- mate curvature. InInternational conference on machine learning, pages 2408–2417. PMLR, 2015

    2015

  7. [7]

    A progressive batching l-bfgs method for machine learning

    Raghu Bollapragada, Jorge Nocedal, Dheevatsa Mudigere, Hao-Jun Shi, and Ping Tak Peter Tang. A progressive batching l-bfgs method for machine learning. InInternational Conference on Machine Learning, pages 620–629. PMLR, 2018

    2018

  8. [8]

    Soap: Improving and stabilizing shampoo using adam.arXiv preprint arXiv:2409.11321, 2024

    Nikhil Vyas, Depen Morwani, Rosie Zhao, Mujin Kwun, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. Soap: Improving and stabilizing shampoo using adam.arXiv preprint arXiv:2409.11321, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Purifying shampoo: Investigating shampoo’s heuristics by decomposing its preconditioner

    Runa Eschenhagen, Aaron Defazio, Tsung-Hsien Lee, Richard E Turner, and Hao-Jun Michael Shi. Purifying shampoo: Investigating shampoo’s heuristics by decomposing its preconditioner. arXiv preprint arXiv:2506.03595, 2025

    arXiv 2025

  10. [10]

    Understanding and improving the shampoo optimizer via kullback-leibler minimization.arXiv e-prints, pages arXiv–2509, 2025

    Wu Lin, Scott C Lowe, Felix Dangel, Runa Eschenhagen, Zikun Xu, and Roger B Grosse. Understanding and improving the shampoo optimizer via kullback-leibler minimization.arXiv e-prints, pages arXiv–2509, 2025

    2025

  11. [11]

    Training deep learning models with norm-constrained lmos.arXiv preprint arXiv:2502.07529, 2025

    Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained lmos.arXiv preprint arXiv:2502.07529, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    Shampoo: Preconditioned stochastic tensor optimization

    Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. InInternational Conference on Machine Learning, pages 1842–1850. PMLR, 2018

    2018

  13. [13]

    Muon is scalable for llm training.arXiv preprint arXiv:2502.16982, 2025

    Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for llm training.arXiv preprint arXiv:2502.16982, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Practical efficiency of muon for pretraining.arXiv preprint arXiv:2505.02222, 2025

    Ishaan Shah, Anthony M Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J Shah, et al. Practical efficiency of muon for pretraining.arXiv preprint arXiv:2505.02222, 2025. 21 SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales

    Pith/arXiv arXiv 2025

  15. [15]

    The polar express: Optimal matrix sign methods and their application to the muon algorithm.arXiv preprint arXiv:2505.16932, 2025

    Noah Amsel, David Persson, Christopher Musco, and Robert M Gower. The polar express: Optimal matrix sign methods and their application to the muon algorithm.arXiv preprint arXiv:2505.16932, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Muon: An optimizer for hidden layers in neural networks, 2024b.URL https://kellerjordan

    K Jordan, Y Jin, V Boza, Y Jiacheng, F Cecista, L Newhouse, and J Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024b.URL https://kellerjordan. github. io/posts/muon

  17. [17]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Hunyuan-large: An open-source moe model with 52 billion activated parameters by tencent.arXiv preprint arXiv:2411.02265, 2024

    Xingwu Sun, Yanfeng Chen, Yiqing Huang, Ruobing Xie, Jiaqi Zhu, Kai Zhang, Shuaipeng Li, Zhen Yang, Jonny Han, Xiaobo Shu, et al. Hunyuan-large: An open-source moe model with 52 billion activated parameters by tencent.arXiv preprint arXiv:2411.02265, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Accurate, large minibatch sgd: Training imagenet in 1 hour.arXiv preprint arXiv:1706.02677, 2017

    Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour.arXiv preprint arXiv:1706.02677, 2017

    Pith/arXiv arXiv 2017

  20. [20]

    On the sdes and scaling rules for adaptive gradient algorithms.Advances in Neural Information Processing Systems, 35:7697–7711, 2022

    Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. On the sdes and scaling rules for adaptive gradient algorithms.Advances in Neural Information Processing Systems, 35:7697–7711, 2022

    2022

  21. [21]

    Pytorch fsdp: experiences on scaling fully sharded data parallel.arXiv preprint arXiv:2304.11277, 2023

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel.arXiv preprint arXiv:2304.11277, 2023

    Pith/arXiv arXiv 2023

  22. [22]

    Zero: Memory optimiza- tions toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models. InSC20: international conference for high performance computing, networking, storage and analysis, pages 1–16. IEEE, 2020

    2020

  23. [23]

    Scalable second order optimization for deep learning.arXiv preprint arXiv:2002.09018, 2020

    Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning.arXiv preprint arXiv:2002.09018, 2020

    Pith/arXiv arXiv 2002

  24. [24]

    Dash: Faster shampoo via batched block preconditioning and efficient inverse-root solvers.arXiv preprint arXiv:2602.02016, 2026

    Ionut-Vlad Modoranu, Philip Zmushko, Erik Schultheis, Mher Safaryan, and Dan Alistarh. Dash: Faster shampoo via batched block preconditioning and efficient inverse-root solvers.arXiv preprint arXiv:2602.02016, 2026

    Pith/arXiv arXiv 2026

  25. [25]

    Precondi- tioned spectral descent for deep learning.Advances in neural information processing systems, 28, 2015

    David E Carlson, Edo Collins, Ya-Ping Hsieh, Lawrence Carin, and Volkan Cevher. Precondi- tioned spectral descent for deep learning.Advances in neural information processing systems, 28, 2015

    2015

  26. [26]

    Stochastic spectral descent for restricted boltzmann machines

    David Carlson, Volkan Cevher, and Lawrence Carin. Stochastic spectral descent for restricted boltzmann machines. InArtificial intelligence and statistics, pages 111–119. PMLR, 2015

    2015

  27. [27]

    Stochastic spectral descent for discrete graphical models.IEEE Journal of Selected Topics in Signal Processing, 10(2):296–311, 2015

    David Carlson, Ya-Ping Hsieh, Edo Collins, Lawrence Carin, and Volkan Cevher. Stochastic spectral descent for discrete graphical models.IEEE Journal of Selected Topics in Signal Processing, 10(2):296–311, 2015

    2015

  28. [28]

    The duality structure gradient descent algorithm: analysis and applications to neural networks.arXiv preprint arXiv:1708.00523, 2017

    Thomas Flynn. The duality structure gradient descent algorithm: analysis and applications to neural networks.arXiv preprint arXiv:1708.00523, 2017

    Pith/arXiv arXiv 2017

  29. [29]

    Modular duality in deep learning.arXiv preprint arXiv:2410.21265, 2024

    Jeremy Bernstein and Laker Newhouse. Modular duality in deep learning.arXiv preprint arXiv:2410.21265, 2024. 22 SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales

    Pith/arXiv arXiv 2024

  30. [30]

    Old optimizer, new norm: An anthology.arXiv preprint arXiv:2409.20325, 2024

    Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology.arXiv preprint arXiv:2409.20325, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Scalable optimization in the modular norm.arXiv preprint arXiv:2405.14813, 2024

    Tim Large, Yang Liu, Minyoung Huh, Hyojin Bahng, Phillip Isola, and Jeremy Bernstein. Scalable optimization in the modular norm.arXiv preprint arXiv:2405.14813, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    Muon+: Towards better muon via one additional normalization step.arXiv preprint arXiv:2602.21545, 2026

    Ruijie Zhang, Yequan Zhao, Ziyue Liu, Zhengyang Wang, and Zheng Zhang. Muon+: Towards better muon via one additional normalization step.arXiv preprint arXiv:2602.21545, 2026

    Pith/arXiv arXiv 2026

  33. [33]

    Towards a principled muon under𝜇p: Ensuring spectral conditions throughout training.arXiv preprint arXiv:2601.01306, 2026

    John Zhao. Towards a principled muon under𝜇p: Ensuring spectral conditions throughout training.arXiv preprint arXiv:2601.01306, 2026

    arXiv 2026

  34. [34]

    Adamuon: Adaptive muon optimizer.arXiv preprint arXiv:2507.11005, 2025

    Chongjie Si, Debing Zhang, and Wei Shen. Adamuon: Adaptive muon optimizer.arXiv preprint arXiv:2507.11005, 2025

    arXiv 2025

  35. [35]

    Normuon: Making muon more efficient and scalable.arXiv preprint arXiv:2510.05491, 2025

    Zichong Li, Liming Liu, Chen Liang, Weizhu Chen, and Tuo Zhao. Normuon: Making muon more efficient and scalable.arXiv preprint arXiv:2510.05491, 2025

    arXiv 2025

  36. [36]

    Fantastic pretraining optimizers and where to find them ii: From weight decay to hyperball optimization, 12 2025

    Kaiyue Wen, Xingyu Dang, Kaifeng Lyu, Tengyu Ma, and Percy Liang. Fantastic pretraining optimizers and where to find them ii: From weight decay to hyperball optimization, 12 2025

    2025

  37. [37]

    Controlled llm training on spectral sphere.arXiv preprint arXiv:2601.08393, 2026

    Tian Xie, Haoming Luo, Haoyu Tang, Yiwen Hu, Jason Klein Liu, Qingnan Ren, Yang Wang, Wayne Xin Zhao, Rui Yan, Bing Su, et al. Controlled llm training on spectral sphere.arXiv preprint arXiv:2601.08393, 2026

    arXiv 2026

  38. [38]

    Adam improves muon: Adaptive moment estimation with orthogonalized momentum.arXiv preprint arXiv:2602.17080, 2026

    Minxin Zhang, Yuxuan Liu, and Hayden Schaeffer. Adam improves muon: Adaptive moment estimation with orthogonalized momentum.arXiv preprint arXiv:2602.17080, 2026

    arXiv 2026

  39. [39]

    Manifold constrained steepest descent.arXiv preprint arXiv:2601.21487, 2026

    Kaiwei Yang and Lexiao Lai. Manifold constrained steepest descent.arXiv preprint arXiv:2601.21487, 2026

    arXiv 2026

  40. [40]

    The newton-muon optimizer.arXiv preprint arXiv:2604.01472, 2026

    Zhehang Du and Weijie Su. The newton-muon optimizer.arXiv preprint arXiv:2604.01472, 2026

    arXiv 2026

  41. [41]

    Mousse: Rectifying the geometry of muon with curvature-aware preconditioning

    Yechen Zhang, Shuhao Xing, Junhao Huang, Kai Lv, Yunhua Zhou, Xipeng Qiu, Qipeng Guo, and Kai Chen. Mousse: Rectifying the geometry of muon with curvature-aware preconditioning. arXiv preprint arXiv:2603.09697, 2026

    arXiv 2026

  42. [42]

    vescale-fsdp: Flexible and high-performance fsdp at scale.arXiv preprint arXiv:2602.22437, 2026

    Zezhou Wang, Youjie Li, Zhiqi Lin, Jiacheng Yang, Cong Xie, Guanyu Feng, Zheng Zhong, Ziyue Huang, Hongyu Zhu, Zhi Zhang, et al. vescale-fsdp: Flexible and high-performance fsdp at scale.arXiv preprint arXiv:2602.22437, 2026

    Pith/arXiv arXiv 2026

  43. [43]

    Canzona: A unified, asynchronous, and load-balanced framework for distributed matrix-based optimizers.arXiv preprint arXiv:2602.06079, 2026

    Liangyu Wang, Siqi Zhang, Junjie Wang, Yiming Dong, Bo Zheng, Zihan Qiu, Shengkun Tang, Di Wang, Rui Men, and Dayiheng Liu. Canzona: A unified, asynchronous, and load-balanced framework for distributed matrix-based optimizers.arXiv preprint arXiv:2602.06079, 2026

    arXiv 2026

  44. [44]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  45. [45]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. 23 SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales

    Pith/arXiv arXiv 2025

  46. [46]

    Nemotron-h: A family of accurate and efficient hybrid mamba- transformer models.arXiv preprint arXiv:2504.03624, 2025

    Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich, Aleksander Ficek, Alexis Bjorlin, Ali Taghibakhshi, Amala Sanjay Deshmukh, Ameya Sunil Mahabaleshwarkar, et al. Nemotron-h: A family of accurate and efficient hybrid mamba- transformer models.arXiv preprint arXiv:2504.03624, 2025

    Pith/arXiv arXiv 2025

  47. [47]

    Scaling laws and compute-optimal training beyond fixed training durations.Advances in Neural Information Processing Systems, 37:76232–76264, 2024

    Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna B Allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations.Advances in Neural Information Processing Systems, 37:76232–76264, 2024

    2024

  48. [48]

    AdamW Weight RMS.https: // kexue

    Jianlin Su. AdamW Weight RMS.https: // kexue. fm/ archives/ 11307, Oct 2025

    2025

  49. [49]

    Stochastic hessian fittings with lie groups.arXiv preprint arXiv:2402.11858, 2024

    Xi-Lin Li. Stochastic hessian fittings with lie groups.arXiv preprint arXiv:2402.11858, 2024

    arXiv 2024

  50. [50]

    Rotational equilibrium: How weight decay balances learning across neural networks.arXiv preprint arXiv:2305.17212, 2023

    Atli Kosson, Bettina Messmer, and Martin Jaggi. Rotational equilibrium: How weight decay balances learning across neural networks.arXiv preprint arXiv:2305.17212, 2023

    Pith/arXiv arXiv 2023

  51. [51]

    On the importance of initialization and momentum in deep learning

    Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. InInternational conference on machine learning, pages 1139–1147. pmlr, 2013

    2013

  52. [52]

    Incorporating nesterov momentum into adam

    Timothy Dozat. Incorporating nesterov momentum into adam. 2016

    2016

  53. [53]

    An empirical model of large-batch training.arXiv preprint arXiv:1812.06162, 2018

    Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training.arXiv preprint arXiv:1812.06162, 2018

    Pith/arXiv arXiv 2018

  54. [54]

    Minimax-01: Scaling foundation models with lightning attention.arXiv preprint arXiv:2501.08313, 2025

    Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, et al. Minimax-01: Scaling foundation models with lightning attention.arXiv preprint arXiv:2501.08313, 2025

    Pith/arXiv arXiv 2025

  55. [55]

    Large batch optimization for deep learning: Training bert in 76 minutes

    Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. InInternational Conference on Learning Representations, 2020

    2020

  56. [56]

    Train longer, generalize better: closing the generalization gap in large batch training of neural networks

    Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. InProceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, Red Hook, NY, USA, 2017

    2017

  57. [57]

    Large batch training of convolutional networks

    Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888, 2017

    Pith/arXiv arXiv 2017

  58. [58]

    One weird trick for parallelizing convolutional neural networks.arXiv preprint arXiv:1404.5997, 2014

    Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks.arXiv preprint arXiv:1404.5997, 2014

    Pith/arXiv arXiv 2014

  59. [59]

    Better & faster large language models via multi-token prediction

    Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. InProceedings of the 41st International Conference on Machine Learning, ICML’24, 2024

    2024

  60. [60]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060, 2024

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060, 2024

    Pith/arXiv arXiv 2024

  61. [61]

    Scaling exponents across parameterizations and optimizers.arXiv preprint arXiv:2407.05872, 2024

    Katie Everett, Lechao Xiao, Mitchell Wortsman, Alexander A Alemi, Roman Novak, Peter J Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jaehoon Lee, et al. Scaling exponents across parameterizations and optimizers.arXiv preprint arXiv:2407.05872, 2024. 24 SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales

    Pith/arXiv arXiv 2024

  62. [62]

    Small-scale proxies for large-scale transformer training instabilities.arXiv preprint arXiv:2309.14322, 2023

    Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, et al. Small-scale proxies for large-scale transformer training instabilities.arXiv preprint arXiv:2309.14322, 2023

    Pith/arXiv arXiv 2023

  63. [63]

    Tensor programs ivb: Adaptive optimization in the infinite-width limit.arXiv preprint arXiv:2308.01814, 2023

    Greg Yang and Etai Littwin. Tensor programs ivb: Adaptive optimization in the infinite-width limit.arXiv preprint arXiv:2308.01814, 2023

    Pith/arXiv arXiv 2023

  64. [64]

    Zachary Nado.𝜖, a nuisance no more.https://zna.do/epsilon

  65. [65]

    Megatron-lm: Training multi-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019

    Pith/arXiv arXiv 1909

  66. [66]

    Stabilizing native low-rank llm pretrain- ing.arXiv preprint arXiv:2602.12429, 2026

    Paul Janson, Edouard Oyallon, and Eugene Belilovsky. Stabilizing native low-rank llm pretrain- ing.arXiv preprint arXiv:2602.12429, 2026

    Pith/arXiv arXiv 2026

  67. [67]

    Deepseek llm: Scaling open-source language models with longtermism.arXiv preprint arXiv:2401.02954, 2024

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism.arXiv preprint arXiv:2401.02954, 2024

    Pith/arXiv arXiv 2024

  68. [68]

    Predictable scale: Part i, step law– optimal hyperparameter scaling law in large language model pretraining.arXiv preprint arXiv:2503.04715, 2025

    Houyi Li, Wenzhen Zheng, Qiufeng Wang, Hanshan Zhang, Zili Wang, Shijie Xuyang, Yuantao Fan, Zhenyu Ding, Haoying Wang, Ning Ding, et al. Predictable scale: Part i, step law– optimal hyperparameter scaling law in large language model pretraining.arXiv preprint arXiv:2503.04715, 2025

    Pith/arXiv arXiv 2025

  69. [69]

    On the role of batch size in stochastic conditional gradient methods.arXiv preprint arXiv:2603.21191, 2026

    Rustem Islamov, Roman Machacek, Aurelien Lucchi, Antonio Silveti-Falls, Eduard Gorbunov, and Volkan Cevher. On the role of batch size in stochastic conditional gradient methods.arXiv preprint arXiv:2603.21191, 2026

    arXiv 2026

  70. [70]

    Spectral scaling laws of muon.arXiv preprint arXiv:2606.04058, 2026

    Gagik Magakyan, Pablo Parrilo, and Asuman Ozdaglar. Spectral scaling laws of muon.arXiv preprint arXiv:2606.04058, 2026

    Pith/arXiv arXiv 2026

  71. [71]

    Dissecting adam: The sign, magnitude and variance of stochastic gradients

    Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients. InInternational Conference on Machine Learning, pages 404–413. PMLR, 2018

    2018

  72. [72]

    A new perspective on shampoo’s preconditioner.arXiv preprint arXiv:2406.17748, 2024

    Depen Morwani, Itai Shapira, Nikhil Vyas, Eran Malach, Sham Kakade, and Lucas Janson. A new perspective on shampoo’s preconditioner.arXiv preprint arXiv:2406.17748, 2024

    Pith/arXiv arXiv 2024

  73. [73]

    Recipes for pre-training llms with mxfp8.arXiv preprint arXiv:2506.08027, 2025

    Asit Mishra, Dusan Stosic, Simon Layton, and Paulius Micikevicius. Recipes for pre-training llms with mxfp8.arXiv preprint arXiv:2506.08027, 2025

    Pith/arXiv arXiv 2025

  74. [74]

    Symbolic discovery of optimization algorithms.Advances in neural information processing systems, 36:49205–49233, 2023

    Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms.Advances in neural information processing systems, 36:49205–49233, 2023. 25 SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales A. Appendix A.1. An SVD perspective on SOAP’...

    arXiv 2023