Pith. sign in

REVIEW 3 major objections 4 minor 46 references

Dion3: Full-Stack Orthogonal Updates

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Dion3 shows a full-stack redesign of Muon's orthogonalization step can match or beat its loss while cutting optimizer step time up to 6x.

desk verdict The speed side of Dion3 is solid and worth taking seriously; the 'matches or improves loss' claim is a single-seed, single-dataset result that the abstract overstates. read the letter →

arxiv 2608.11612 v1 pith:2CCZWEKD submitted 2026-08-12 cs.LG cs.AI

Noah Amsel , Jack Zhang , Kwangjun Ahn , Ali Naeimi , Austin Feng , Berlin Chen , Tri Dao , John Langford This is my paper · ORCID
classification cs.LGcs.AI
keywords MuonoptimizerNewton-SchulzorthogonalizationGrammatrixfractionalrowselectionerrorfeedbackdistributedtrainingsymmetricGEMMkernelsLLMpretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Dion3 is a redesigned version of the Muon optimizer, which trains large language models by orthogonalizing momentum matrices. The paper tries to show that the expensive cubic-time Newton-Schulz orthogonalization step can be made cheap at every level: reformulating it on the small symmetric Gram matrix, accelerating that with custom symmetric GEMM kernels, and orthogonalizing only a fraction of the momentum matrix's rows via a simple selection rule with error feedback. If the paper is right, practitioners can keep Muon's convergence behavior in dense and sparse transformers while paying a small fraction of its optimizer overhead, with step time down by up to 6x. The paper also reports a mild quality gain: at 14B parameters, selecting 1/4 of rows each step lowers validation loss by 0.027 and raises downstream accuracy by 0.7 points relative to a tuned NorMuon baseline.

What carries the argument

The load-bearing mechanism is the identity polar(X) = (XX^T)^(-1/2)X combined with the factorization of odd polynomials p_t(x)=x h_t($x^{2}$). That factorization lets Newton-Schulz run on the small symmetric Gram matrix R_t = X_t X_t^T while keeping a cumulative Q_t, producing the same output as standard Newton-Schulz in exact arithmetic. The other half is the row-selection update rule: pick k=ceil(fn) rows of M with largest L1 norm, polarize the submatrix with Gram Newton-Schulz, update only those rows, and apply error feedback by decaying only selected momentum rows so unselected rows can build up and be chosen later. A restart after the second iteration stabilizes the Gram iteration against spurious negative eigenvalues introduced by half-precision arithmetic.

What would settle it

Train a 7B dense transformer on a second data mixture, such as FineWeb-Edu, with Dion3 at f=1/4 versus a tuned NorMuon, running several seeds; if Dion3's final validation loss is not at most NorMuon's, the claim that Dion3 matches or improves the loss fails to transfer outside the paper's setting.

Watch

Extended reading notes

Core claim

The central claim is that Muon's polarization step can be replaced by a mathematically equivalent Gram-matrix iteration plus a coarser, cheaper update rule without losing Muon's optimization quality. Writing polar(X)=(XX^T)^(-1/2)X, Gram Newton-Schulz iterates on the n by n Gram matrix and needs only two rectangular matrix multiplications instead of the usual ten, while the fractional update rule selects k=ceil(fn) rows of largest L1 norm, orthogonalizes just those rows, updates only those weights, and dampens only those momentum rows via error feedback. The paper's experiments show that for 1B-14B dense transformers trained on ClimbMix, tuned fractional updates track below NorMuon's loss throughout training, and the optimizer step time falls by up to 6x, with communication volume scaling as the fraction f. It further establishes a transfer rule eta' = eta/$\sqrt$(f) that keeps the Frobenius norm of the update matched to full Muon.

Load-bearing premise

The quality claim stands on the premise that updating only a fraction of momentum rows each step, with error feedback for the rest, is a harmless or slightly helpful approximation across architectures, data, and scales; the experiments cover one data mixture and dense transformers from 1B to 14B.

Editorial extensions

If this is right

  • Using Dion3 at f=1/4 as a drop-in replacement for Muon should give essentially the same or slightly lower training loss while cutting optimizer step time by about 3.7x on top of Gram Newton-Schulz and kernel gains, for large dense models.
  • The eta*sqrt(f) approximately 0.01 rule means users do not need a full learning-rate sweep for each new fraction; set eta' = eta/sqrt(f).
  • In communication-bound regimes, megabatching collapses communication rounds to a constant and fractional selection reduces communication volume by a factor of 1/f, so the benefits should be largest when many small matrices are sharded across many GPUs.
  • Gram Newton-Schulz applies to any Muon variant, including NorMuon and MoE training, so improvements to the orthogonalization step compose with other optimizers that use Newton-Schulz as a black box.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The improved final loss at f<1 suggests row selection acts as an implicit regularizer; if so, the same trick could be ported to other cubic-time update rules that orthogonalize or whiten momentum, but the paper does not test that.
  • Because the paper reports single-seed runs on one data mixture, the quality edge at 14B (-0.027) may be within run-to-run noise; a multi-seed replication on a second data mixture would settle whether the improvement is systematic.
  • Communication savings grow linearly in 1/f, so in settings where weights are sharded across data-center networks rather than local NVLink interconnects, the end-to-end speedup could exceed the 6x reported; the paper's benchmarks use local interconnects.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents Dion3, a revision of the Muon optimizer that targets the cost of the Newton–Schulz orthogonalization step at four levels: a Gram-matrix reformulation of Newton–Schulz (Gram Newton–Schulz), custom symmetric GEMM kernels written in CuteDSL, a fractional row-selection update rule with error feedback, and megabatched communication for distributed training. The authors prove an algebraic equivalence between standard Newton–Schulz and the Gram iteration (Theorem 2), provide a stability analysis with a restart schedule, and report experiments on 1B–14B dense transformers trained on ClimbMix. The headline claims are that Dion3 matches or improves the loss of Muon/NorMuon while reducing optimizer step time by up to 6x, and that the update rule improves on Dion in both speed and performance.

Significance. If the speed results hold, the practical significance is high: reducing Muon's optimizer-step overhead from roughly 26x AdamW to about 4x at 7B scale would make orthogonal optimizers substantially more accessible in distributed training. The paper has clear strengths: the Gram Newton–Schulz equivalence is a clean algebraic result; the FLOP accounting is careful (up to a typo noted below); the stability analysis, restart schedule, and benchmark methodology with interquartile ranges are well specified; and the authors provide open-source packages for the optimizer and kernels. The weak link is the loss-quality component of the central claim, which currently rests on single-seed, single-dataset experiments, and the abstract's claim of improvement over Dion is not backed by any direct Dion baseline experiment.

major comments (3)
  1. [§8.1, Fig. 3, Table 2] The claim that Dion3 'matches or improves on the loss achieved by Muon' is supported only by single-seed runs on a single data mixture (ClimbMix) with dense transformers at 1B–14B, and the hyperparameters (f, η) are selected from the same sweep whose best values are then reported (Figure 3). No repeated seeds or confidence intervals are given for the final losses in Figure 3, Figure 4, or Table 2, and the larger-scale runs in Table 2 use the learning rate from the 1B transfer rule η√f = 0.01 without per-scale validation. Section 9 itself states that 'Further work is needed to determine how widely this improvement generalizes.' Because this loss-quality claim is load-bearing for the abstract, the paper should either add repeated-seed or multi-dataset evidence, or explicitly restrict the claim to the tested setting.
  2. [Abstract, §3.2] The abstract states that the update rule 'improves on Dion ... in both speed and performance,' and Section 3.2 says the update rule is 'simpler and faster than Dion,' but no experiment in the paper compares Dion3 against the Dion baseline. The comparisons to NorMuon/Muon in Section 8 do not support a claim about Dion. Add a direct Dion baseline to the speed and loss experiments, or remove the performance claim from the abstract and intro.
  3. [§4.1] The FLOP comparison in the text says 'For a typical Muon application (T = 5, α = 43), it saves 55% of the FLOPs used by standard Newton-Schulz with symmetric GEMMs, or 68% compared to a typical implementation without symmetric GEMMs.' Plugging α = 43 into the paper's own formulas gives T(3α+1) = 650 and (4T+3α−3) = 146, i.e., a 77.5% saving, not 55%; the stated 55%/68% correspond to α = 4. Please correct the aspect-ratio typo and recompute any downstream percentages.
minor comments (4)
  1. [§8.2, Fig. 6] The text reports 'overall speedups of 3.6× and 6.5×,' while Figure 6 and the abstract say '6×.' Please harmonize these numbers so the abstract, figure, and body agree.
  2. [Algorithm 4, §8.1] Section 6 says the distributed implementation selects the top-f fraction of rows from each shard, with global selection as an option, but Section 8.1 does not state which selection mode was used in the reported runs. Please specify this for reproducibility.
  3. [Figure 3] The heatmap reports one number per (f, η) cell with no indication of the number of runs or run-to-run variability. With single runs, a bolded 'best' cell is not evidence of a significant difference from neighboring cells; please state the number of seeds per cell or add error bars.
  4. [Appendix B.2.1] The restart schedule is derived under the empirical bound that spurious negative eigenvalues are at least −4 × 10⁻⁴. Please add a sentence noting that this is an empirical assumption that should be re-validated if the precision, architecture, or coefficient set changes.

Circularity Check

0 steps flagged · score 2.0 of 10

No construction-level circularity: Gram Newton-Schulz is a proven algebraic equivalence and the learning-rate rule follows from Frobenius norm matching; only minor, non-load-bearing self-citations were found.

full rationale

The paper's central derivation chain is self-contained at the level of algebra and cost analysis. Theorem 2 proves the scalar iteration underlying Gram Newton-Schulz directly from the polynomial decomposition p_t(x) = x h_t(x^2), with no fitted quantity or target loss entering the equivalence; the same holds for the FLOP counts and the restart analysis, where the assumed spurious eigenvalue bound of about -4e-4 is an observed numerical input used to choose a schedule, not a prediction derived from the method itself. The learning-rate transfer rule eta' = eta/sqrt(f) is obtained by equating the Frobenius norms of the Muon and Dion3 updates, so it is not fitted to the validation losses that are later reported; using this rule at 3B-14B is extrapolation rather than circularity. The main caveats are evidential rather than circular: the claim that f<1 improves loss is based on a single-seed, single-dataset sweep in which the best f and eta are selected from the same grid that produces the reported minima, and the abstract asserts improvement over Dion without a direct benchmark against Dion. The paper also relies on two overlapping-author prior works, Ahn et al. for error feedback and Amsel et al. for Polar Express coefficients, but both are used as implemented components and are themselves tested or simulated within the present paper rather than invoked as uniqueness theorems or as substitutes for the experimental comparison. These self-citations are real prior-art inputs but are not load-bearing in a way that makes any result true by construction. Overall, the specific equations do not reduce to their own inputs, so the appropriate finding is no significant circularity, with a small score adjustment only for the presence of minor non-load-bearing self-citations.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims rest on two categories of uncharged premises: numerical assumptions about half-precision Gram matrices and the empirical approximation quality of row-selected updates. The free parameters show that the speed/quality tradeoff is controlled by a user-chosen f and by a stabilization schedule selected under an assumed perturbation bound, so part of Dion3's behavior is fitted, not derived.

free parameters (4)
  • selection fraction f = 1/4 recommended; 1/8 best at 1B
    User-controlled compression factor introduced in Algorithm 4, swept over 1/2, 1/4, 1/8, and 1/16 in Section 8.1; it determines the speed/quality tradeoff and the eta/sqrt(f) learning-rate scaling.
  • Gram-NS restart location = after iteration 2 of 5
    Chosen in Section 4.2 and Appendix B.2.1 by sweeping all restart schedules under an assumed spurious negative eigenvalue bound of -4e-4; it is a hand-picked stabilization point, not derived from first principles.
  • assumed spurious eigenvalue bound = -4e-4
    Observed in Appendix B.1 on one synthetic matrix and used to select the restart schedule; if real weight matrices produce larger spurious eigenvalues, the restart guarantee may not hold.
  • Polar Express safety factor = 1.05
    Section B.2.2 recommends 1.05 instead of the standard 1.02 as an empirical caution; it affects the convergence margin of Gram Newton-Schulz.
assumptions (5)
  • standard math polar(M) = (M M^T)^-1/2 M when M has full row rank.
    Used in Section 4 as the starting formula for Gram Newton-Schulz; it is the textbook polar decomposition identity.
  • domain assumption The Polar Express Newton-Schulz coefficients drive singular values in [0,1] to approximately 1.
    Borrowed from the authors' prior ICLR paper [3]; the paper inherits the convergence behavior without re-deriving it.
  • ad hoc to paper Half-precision spurious negative eigenvalues of the Gram matrix are bounded by -4e-4 and are reset by restarting every two iterations.
    Section 4.2 and Appendices B.1/B.2.1; the bound is measured on synthetic matrices and used to justify Algorithm 3, but is not proven for arbitrary weight matrices.
  • ad hoc to paper L1-norm row selection with error feedback approximates full Muon to within acceptable quality.
    Algorithm 4 and Sections 6 and 8.1; this is the central modeling assumption behind the fractional update rule and is validated only empirically on one dataset.
  • domain assumption Single-seed training comparisons are representative of run-to-run variance.
    Section 8.1 reports one run per configuration in Table 2 and Figures 3-5; the paper implicitly assumes variance is small enough to support the stated conclusions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dion3: Full-Stack Orthogonal Updates." pith.science (2026). https://pith.science/paper/2CCZWEKD

@misc{pith2026260811612,
  author       = {Pith},
  title        = {Pith review of: Dion3: Full-Stack Orthogonal Updates},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2CCZWEKD}},
  note         = {Machine review of arXiv:2608.11612}
}
read the original abstract

The Muon optimizer incurs a significant overhead cost due to its cubic-time Newton-Schulz orthogonalization step. When weights are sharded, communication overhead compounds this computational cost, eroding the benefits of Muon in many settings. We present Dion3, a revision of Muon that targets this overhead at every level of the stack. Our Gram Newton-Schulz algorithm reduces the FLOP cost of orthogonalization, our CuteDSL kernels accelerate it by exploiting symmetry, and our megabatching strategy reduces communication overhead. Moreover, we propose a simple change to the update rule that cuts costs even further: selecting only a fraction of the momentum matrix's rows to orthogonalize at each step. This update rule improves on Dion (another "compressed" version of Muon), in both speed and performance. Overall, Dion3 matches or improves on the loss achieved by Muon but reduces optimizer step time by up to 6x. Dion3 is available via the dion package (https://github.com/microsoft/dion) as a drop-in replacement for Muon.

Figures

Figures reproduced from arXiv: 2608.11612 by the authors.

Figure 1
Figure 1. Optimizer step time of Muon (excluding forward/backward pass) relative to AdamW for a 7B [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Left: Symmetric GEMM computes 256 × 256 tiles from the lower triangle and main diagonal, then transposes and copies each lower tile to the corresponding upper tile. Right: Our CuteDSL symmetric GEMM kernels benchmarked against cuBLAS GEMM kernels on Hopper and Blackwell GPUs. Input matrices A, B, C have dimensions n × n. For large enough n, our kernels achieve a ∼ 2× speedup over cuBLAS, both with and without an epi… view at source ↗
Figure 3
Figure 3. Left: Final validation loss for 1B-parameter models trained on 100B tokens of ClimbMix with NorMuon or Dion3, as a function of row-selection fraction f and learning rate η. NorMuon is f = 1. Bolded cell in each row shows best η. Optimal ηs track the line η √ f = 0.01, shown in red on a log-log scale. Not shown: f = 1, η = 0.03 (loss = 2.237) and f = 1/32, η = 0.04 (loss = 2.191). Right: The same data reduced to two … view at source ↗
Figures from the paper (20 more)
Figure 4
Figure 4. Figure 4: Validation loss curves for 1B-parameter models trained on 100B tokens of ClimbMix with NorMuon [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Validation loss over training at 14B for Dion3 ( [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Optimizer step time (excluding forward/backward pass) relative to standard Muon across model [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: When training with Muon on a Hopper GPU, switching from standard Newton-Schulz to Gram [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: Newton-Schulz time per model weight for (1) standard Newton-Schulz implemented in pure PyTorch, [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Optimizer step time for AdamW and Muon with three different Newton-Schulz routines on a single [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: Estimated exposed Newton-Schulz time for one step of Kimi K2 with pipeline parallelism. We [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Naive Gram Newton-Schulz used to train Llama-430M in half precision. Numerical instability [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Evolution of eigenvalues of Rt, Qt, and Xt in Naive Gram Newton-Schulz with (at, bt, ct) = ( 15/8, 10/8, 3/8) as a function of the corresponding singular value of X0. Top: float64. Bottom: bfloat16. Now we rerun the experiment in bfloat16 arithmetic ( [PITH_FULL_IMAG…
Figure 13
Figure 13. Figure 13: When Rt evolves according to (4), negative eigenvalues diverge to −∞. To prove that the tiny spurious negative eigenvalues of R0 suffice to cause divergence, we rerun the experiment in float64 precision, but cast R0 from float64 to bfloat16 and then back to float64 to…
Figure 14
Figure 14. Figure 14: Evolution of eigenvalues of Rt, Qt, and Xt when all operations use float64 except R0 = X0X⊤ 0 , which uses bfloat16. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]
Figure 15
Figure 15. Figure 15: As the eigenvectors drift, the spectral norms of [PITH_FULL_IMAGE:figures/full_fig_p027_15.png]
Figure 16
Figure 16. Figure 16: Restarting prevents the divergence of Rt in half-precision. 0 5 10 Step (t) 0.00 0.05 0.10 0.15 0.20 Relative Diagonalization Error Rt Qt Xt 0 5 10 Step (t) 1.0 1.1 1.2 1.3 1.4 1.5 Max Eigenvalue Rt Observed Theoretical 0 5 10 Step (t) 5 10 15 20 25 Max Eigenvalue Qt …
Figure 17
Figure 17. Figure 17: Restarting curbs eigenvector drift. B.2.1 When to Restart: Polar Express Coefficients for Muon We now derive the optimal restarting schedule for a given total number of iterations T. To prevent Xt = QtX0 from blowing up, we must control the condition number of Qt, eve…
Figure 18
Figure 18. Figure 18: Left: Min/max eigenvalue of Rt and Qt without restarts. R0 starts with a negative eigenvalue at −4 · 10−4 . Right: Minimum eigenvalue of Rt and condition number of Qt if restart is placed after iteration 1, 2, 3, or 4. Restarting after iteration 2 best controls the si…
Figure 19
Figure 19. Figure 19: Gram Newton-Schulz with Polar Express coefficients and a restart after [PITH_FULL_IMAGE:figures/full_fig_p029_19.png]
Figure 20
Figure 20. Figure 20: Theoretical behavior of both standard and Gram Newton-Schulz (Polar Express coefficients) when [PITH_FULL_IMAGE:figures/full_fig_p030_20.png]
Figure 21
Figure 21. Figure 21: shows the results. Both effects discussed in Section 7 are clearly visible. The latency floor is ∼25 µs (left panel), of which ∼15–17 µs is host-side dispatch (as measured separately with perf_counter and no device synchronization). Without megabatching, we pay this c…
Figure 22
Figure 22. Figure 22: Validation loss at 1B for NorMuon and for Dion3 at [PITH_FULL_IMAGE:figures/full_fig_p035_22.png]
Figure 23
Figure 23. Figure 23: Optimizer step time (excluding forward/backward pass) relative to standard Muon across model [PITH_FULL_IMAGE:figures/full_fig_p036_23.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 30 canonical work pages

  1. [1]

    Julien Abadji, Marah Abdin, Connor Adams, Eric Alcaide, Mustafa Altun, Michele Artoni, Junze Bao, Uday Barar, Vassilis Bekiaris, Arkadii Bessonov, et al.Laguna M.1/XS.2 Technical Report. 2026. arXiv: 2605.27605

    arXiv 2026

  2. [2]

    Kwangjun Ahn, Byron Xu, Natalie Abreu, Ying Fan, Gagik Magakyan, Pratyusha Sharma, Zheng Zhan, and John Langford.Dion: Distributed Orthonormalized Updates. 2025. arXiv:2504.05295

    arXiv 2025

  3. [3]

    The Polar Express: Optimal Matrix Sign Methods and their Application to the Muon Algorithm

    Noah Amsel, David Persson, Christopher Musco, and Robert M. Gower. “The Polar Express: Optimal Matrix Sign Methods and their Application to the Muon Algorithm”. In:The Fourteenth International Conference on Learning Representations. 2026. https://openreview.net/forum?id=yRtgZ1K8hO

    work page 2026

  4. [4]

    Google DeepMind, 2025.https://jax-ml.github.io/scaling-book/

    Jacob Austin, Sholto Douglas, Roy Frostig, Anselm Levskaya, Charlie Chen, Sharad Vikram, Federico Lebron, Peter Choy, Vinay Ramasesh, Albert Webson, and Reiner Pope.How to Scale Your Model. Google DeepMind, 2025.https://jax-ml.github.io/scaling-book/

    work page 2025

  5. [5]

    Deriving Muon

    Jeremy Bernstein. Deriving Muon. 2025. https://jeremybernste.in/writing/deriving-muon

    work page 2025

  6. [6]

    Thibaut Boissin, Thomas Massena, Franck Mamalet, and Mathieu Serrurier.Turbo-Muon: Accelerating Orthogonality-Based Optimization with Pre-Conditioning. 2025. arXiv:2512.04632

    arXiv 2025

  7. [7]

    Towards understanding of orthogonalization in Muon

    Valentyn Boreiko, Zhiqi Bu, and Sheng Zha. “Towards understanding of orthogonalization in Muon”. In: Tiny Titans: The next wave of On-Device Learning for Foundation Models (TTODLer-FM). 2025. https://openreview.net/forum?id=4vzhqq5hpX

    work page 2025

  8. [8]

    Franz Louis Cesista, Jiacheng You, and Keller Jordan.Squeezing 1-2% Efficiency Gains Out of Muon by Optimizing the Newton-Schulz Coefficients. 2025. http://leloykun.github.io/ponder/muon-opt- coeffs/. 17

    work page 2025

Show all 46 references
  1. [9]

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, et al.DeepSeek-V3 Technical Report. 2025. arXiv:2412.19437

    2025 arXiv

  2. [10]

    Nemotron-CLIMB: Clustering-based Iterative Data Mixture Bootstrapping for Language Model Pre- training

    Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, Mostofa Patwary, Yingyan Celine Lin, Jan Kautz, and Pavlo Molchanov. “Nemotron-CLIMB: Clustering-based Iterative Data Mixture Bootstrapping for Language Model...

    2025

  3. [11]

    Essential AI.Layer Sharding for Large-Scale Training with Muon. 2025. https://www.essential.ai/ research/infra

    2025

  4. [12]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al.Gemma 3 Technical Report. 2025. arXiv:2503.19786

    2025 arXiv

  5. [13]

    GLM-5 Team, Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, et al.GLM-5: from Vibe Coding to Agentic Engineering. 2026. arXiv: 2602.15763

    2026 arXiv

  6. [14]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.The Llama 3 Herd of Models. 2024. arXiv:2407.21783

    2024 arXiv

  7. [15]

    Ekaterina Grishina, Matvey Smirnov, and Maxim Rakhuba.Accelerating Newton-Schulz Iteration for Orthogonalization via Chebyshev-type Polynomials. 2026. arXiv:2506.10935

    2026

  8. [16]

    Wentao Guo, Mayank Mishra, Xinle Cheng, Ion Stoica, and Tri Dao.SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations. 2026. arXiv:2512.14080

    2026

  9. [17]

    Higham.Functions of Matrices: Theory and Computation

    Nicholas J. Higham.Functions of Matrices: Theory and Computation. Society for Industrial and Applied Mathematics, 2008. doi: 10.1137/1.9780898717778

    2008 doi

  10. [18]

    Taejong Joo, Wenhan Xia, Cheolmin Kim, Ming Zhang, and Eugene Ie.On Surprising Effectiveness of Masking Updates in Adaptive Optimizers. 2026. arXiv:2602.15322

    2026

  11. [19]

    Muon: An optimizer for hidden layers in neural networks

    Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. 2024. https://kellerjordan. github.io/posts/muon/

    2024

  12. [20]

    MuonBP: Faster Muon via Block-Periodic Orthogonalization

    Ahmed Khaled, Kaan Ozkara, Tao Yu, Mingyi Hong, and Youngsuk Park. “MuonBP: Faster Muon via Block-Periodic Orthogonalization”. In: The Fourteenth International Conference on Learning Representations. 2026. https://openreview.net/forum?id=mHouLSUQP5

    2026

  13. [21]

    Charles, Cheng Chen, Guanduo Chen, Haiting Chen, Huarong Chen, Jiahao Chen, Ningxin Chen, et al.Kimi K2: Open Agentic Intelligence

    Kimi Team, Yifan Bai, Yiping Bao, Y. Charles, Cheng Chen, Guanduo Chen, Haiting Chen, Huarong Chen, Jiahao Chen, Ningxin Chen, et al.Kimi K2: Open Agentic Intelligence. 2025. arXiv:2507.20534

    2025 arXiv

  14. [22]

    On the Computation of the Matrix k-th Root

    Slobodan Lakić. “On the Computation of the Matrix k-th Root”. In: Zeitschrift für Angewandte Mathematik und Mechanik78.3 (1998), pp. 167–172.https://onlinelibrary.wiley.com/doi/abs/ 10.1002/%28SICI%291521-4001%28199803%2978%3A3%3C167%3A%3AAID-ZAMM167%3E3.0.CO%3B2-R

    1998

  15. [23]

    NorMuon: Making Muon more efficient and scalable

    Zichong Li, Liming Liu, Chen Liang, Weizhu Chen, and Tuo Zhao. “NorMuon: Making Muon more efficient and scalable”. In:Forty-third International Conference on Machine Learning. 2026. https: //openreview.net/forum?id=m1IRWFAMsa

    2026

  16. [24]

    Junghwan Lim, Sungmin Lee, Dongseok Kim, Taehyun Kim, Eunhwan Park, Jeesoo Lee, Jeongdoo Lee, Junhyeok Lee, Wai Ting Cheung, Dahye Choi, et al.Motif 2 12.7B technical report. 2025. arXiv: 2511.07464

    2025

  17. [25]

    Tianyang Lin.Flash-Muon: An Efficient Implementation of Muon Optimizer. 2025. https://github. com/nil0x9/flash-muon

    2025

  18. [26]

    Pull Request #1428, NVIDIA/Megatron-LM

    Jingyuan Liu.A proof of concept for Distributed Muon. Pull Request #1428, NVIDIA/Megatron-LM. Accessed: 2026-07-10. 2025.https://github.com/NVIDIA/Megatron-LM/pull/1428

    2026

  19. [27]

    Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al.Muon is Scalable for LLM Training. 2025. arXiv:2502.16982

    2025 arXiv

  20. [28]

    Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf.FineWeb-Edu: the Finest Collection of Educational Content. 2024. doi: 10.57967/hf/2497. 18

    2024 doi

  21. [29]

    Trion: FFT-based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of LLMs

    Ionut-Vlad Modoranu, Mher Safaryan, Erik Schultheis, Max Ryabinin, Artem Chumachenko, and Dan Alistarh. “Trion: FFT-based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of LLMs”. In:The Fourteenth International Conference on Learning Representations. 2026. https...

    2026

  22. [30]

    Laker Newhouse, Dakota Goldberg, and Ricardo Ruiz.Faster symmetric matrix multiplication with ThunderKittens. 2024. https://www.lakernewhouse.com/assets/writing/faster-symmul-with- thunderkittens.pdf

    2024

  23. [31]

    Arora, Yu Bai, Bowen Baker, et al.gpt-oss-120b & gpt-oss-20b Model Card

    OpenAI, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, et al.gpt-oss-120b & gpt-oss-20b Model Card. 2025. arXiv: 2508.10925

    2025 arXiv

  24. [32]

    Ziyuan Tang, Tianshi Xu, Yousef Saad, and Yuanzhe Xi.Hierarchical Muon: Tiled Newton-Schulz Updates for Efficient Muon Optimization. 2026. arXiv:2606.27216

    2026 arXiv

  25. [33]

    Transpose one of the MLP matrices + add Triton kernel for symmetric matmul

    Byron Xu. Transpose one of the MLP matrices + add Triton kernel for symmetric matmul. Pull Request #109, KellerJordan/modded-nanogpt. Accessed: 2026-07-10. 2025.https://github.com/ KellerJordan/modded-nanogpt/pull/109

    2026

  26. [34]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.Qwen3 Technical Report. 2025. arXiv:2505.09388

    2025 arXiv

  27. [35]

    PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. “PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel”. In:Proceedings of the VLDB Endowment16.12 (2023), pp. 3848–3860. 19 Appen...

    2023

  28. [36]

    X←atX + BX Our symmetric GEMM kernel can compute matrix quadratics in one launch, fusing the addition ofatI by addingat to all diagonal entries of the output when they are at the register level. This optimization completely obviates any I/O operations needed for theatI additio...

  29. [37]

    Zt←atI +btRt−1 +ctR2 t−1 // Applyht(Rt−1)

  30. [38]

    (RZ)t← Rt−1Zt +atRt−1

  31. [39]

    Fused GEMM + Add

    Rt← Zt(RZ)t +at(RZ)t This change fixes all collected training examples in which symmetric GEMMs were less stable than non- symmetric GEMMs. C Kernel Implementation Details C.1 Symmetric GEMM Kernel Details We implement all of our symmetric GEMM kernels with square cluster work...

  32. [40]

    This is the size of the cluster used to train DeepSeek-V3, with upgraded hardware [9]

    The training cluster consists of256 nodes of eight H100s each (2048 GPUs in total), connected with NDR 400 Gb/s InfiniBand inter-node (8 NICs per node, 1:1 NIC-to-GPU ratio) and NVLink 4.0 intra-node. This is the size of the cluster used to train DeepSeek-V3, with upgraded har...

  33. [41]

    Training inbfloat16 attains 40% model flop utilization (MFU), which is typical for MoEs at this scale on H100s. 35 1536 2304 Model dimension 0.0 0.2 0.4 0.6 0.8 1.0 Runtime: Fraction of Baseline 1 GPU 1536 2304 4096 Model dimension 4 GPUs Standard NorMuon + Symmetric kernels +...

  34. [42]

    The optimizer steps of pipeline stages2 to 16 are fully hidden behind the backwards of stages1 to 15

    The only non-overlapped optimizer time is that of the last pipeline stage to complete its backward pass (i.e., pipeline stage1 of 16). The optimizer steps of pipeline stages2 to 16 are fully hidden behind the backwards of stages1 to 15

  35. [43]

    Under these assumptions, the optimal way to partition the Newton-Schulz work of pipeline stage1 is as follows

    Pipeline stage 1 contains the dense layer and3 MoE layers. Under these assumptions, the optimal way to partition the Newton-Schulz work of pipeline stage1 is as follows. Each of the16 GPUs in pipeline stage1’s expert parallel group gets 384 experts/layer× 3 MoE layers 16 GPUs ...

  36. [44]

    Training uses32 H100s across 4 nodes (8 GPUs per node)

  37. [45]

    Training inbfloat16 hits 40% MFU

  38. [46]

    Each layer has3 MLP weights (up, gate, down), and the attention weights WQ, WK, WV, and WO

    Weights are sharded evenly across GPUs using FSDP, and the exposed Newton-Schulz time is that of 80 layers/32 GPUs≈ 3 layers. Each layer has3 MLP weights (up, gate, down), and the attention weights WQ, WK, WV, and WO. According to our benchmarking, standard Newton-Schulz of • ...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.