Pith. sign in

REVIEW 2 major objections 2 cited by

Anytime Training with Schedule-Free Spectral Optimization

T0 review · 2 major / 0 minor · reviewed 2026-05-25 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Schedule-free spectral optimization matches tuned AdamW on 125M and 772M parameter models using one hyperparameter configuration across 1-8x Chinchilla horizons.

desk verdict SF-NorMuon delivers schedule-free training that matches tuned AdamW across horizons on 125M and 772M models with one fixed hyperparameter set. read the letter →

arxiv 2605.23061 v1 pith:CH6ZJZ5U submitted 2026-05-21 cs.LG cs.AImath.OCstat.ML

Anuj Apte , Pranav Deshpande , Niraj Kumar , Shouvanik Chakrabarti , Junhyung Lyle Kim This is my paper · ORCID
classification cs.LGcs.AImath.OCstat.ML
keywords schedule-freeoptimizationspectraloptimizerlanguagemodeltraininganytimeweightdecayAdamWChinchillahorizonsneuralnetworkoptimizers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard neural network training ties learning-rate schedules to a fixed horizon, creating path dependence and forcing costly re-tuning when data volume changes. Schedule-free methods remove explicit schedules to address this, but prior versions like SF-AdamW still fall short of well-tuned AdamW. The paper introduces SF-NorMuon, a schedule-free spectral optimizer that closes the gap by matching or exceeding tuned AdamW performance on 125M and 772M parameter language models with a single hyperparameter setup. This holds across training lengths from 1x to 8x Chinchilla horizons. The approach lets practitioners obtain high-quality checkpoints at any point without committing to a total horizon in advance.

What carries the argument

SF-NorMuon, a schedule-free spectral optimizer that applies weight decay at the fast iterate within schedule-free spectral dynamics.

What would settle it

Training runs of SF-NorMuon on 772M parameter models over 8x Chinchilla horizons that exhibit instability or performance drops when weight decay is removed from the fast iterate.

Watch

Extended reading notes

Core claim

SF-NorMuon is a schedule-free spectral optimizer that, with a single hyperparameter configuration, matches or exceeds tuned AdamW on 125M and 772M parameter language models across 1 to 8 times Chinchilla horizons. The work proves a stationarity guarantee for schedule-free spectral dynamics and shows that weight decay applied at the fast iterate is essential for long-horizon stability. This removes the need to commit to a training horizon upfront while preserving competitive performance.

Load-bearing premise

Weight decay applied at the fast iterate is essential for maintaining long-horizon stability in schedule-free spectral dynamics.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The manuscript proposes SF-NorMuon, a schedule-free spectral optimizer, claiming that a single hyperparameter configuration matches or exceeds per-horizon-tuned AdamW on 125M and 772M parameter language models across 1--8× Chinchilla horizons. It further states a stationarity guarantee for schedule-free spectral dynamics and identifies weight decay applied at the fast iterate as essential for long-horizon stability, enabling horizon-independent high-quality checkpoints.

Significance. If the empirical matching holds with the reported single-configuration robustness and the stationarity result is non-vacuous, the work would meaningfully advance practical anytime optimization by reducing horizon-dependent retuning costs and supporting continual learning. The theoretical identification of the weight-decay placement provides a concrete design principle that could guide further schedule-free methods.

major comments (2)
  1. [Abstract] Abstract: the central empirical claim (SF-NorMuon matches or exceeds tuned AdamW with one fixed hyperparameter set across two model sizes and multiple horizons) is stated without reference to any table, figure, ablation, or error analysis, preventing assessment of effect size, variance, or whether the result is load-bearing for the 'closes this gap' conclusion.
  2. [Abstract] Abstract (theoretical analysis section): the stationarity guarantee and the claim that weight decay at the fast iterate is 'essential for long-horizon stability' are asserted without any displayed equations, assumptions, or derivation outline, so it is impossible to verify whether the condition is derived or imposed by construction.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their comments on the abstract. We respond to each major comment below and indicate where revisions will be made.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central empirical claim (SF-NorMuon matches or exceeds tuned AdamW with one fixed hyperparameter set across two model sizes and multiple horizons) is stated without reference to any table, figure, ablation, or error analysis, preventing assessment of effect size, variance, or whether the result is load-bearing for the 'closes this gap' conclusion.

    Authors: We agree the abstract would be clearer with explicit pointers to the supporting evidence. The performance comparisons (including means and standard deviations over three random seeds) appear in Section 4, Tables 1–2 and Figures 2–4; hyperparameter robustness and weight-decay ablations are in Section 5. In the revision we will add concise parenthetical citations to these results within the abstract. revision: yes

  2. Referee: [Abstract] Abstract (theoretical analysis section): the stationarity guarantee and the claim that weight decay at the fast iterate is 'essential for long-horizon stability' are asserted without any displayed equations, assumptions, or derivation outline, so it is impossible to verify whether the condition is derived or imposed by construction.

    Authors: The stationarity guarantee is stated and proved as Theorem 3.1 (with the key assumption of weight decay applied to the fast iterate). The necessity of this placement is shown by a counter-example in Appendix B when weight decay is instead applied to the slow iterate. The abstract summarizes the result at high level; we will add a reference to Theorem 3.1 in the revised abstract. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity in derivation chain

full rationale

The paper's primary result is the empirical observation that a single fixed hyperparameter configuration of SF-NorMuon matches or exceeds per-horizon-tuned AdamW on 125M and 772M models across 1-8× Chinchilla horizons. The mentioned stationarity proof and weight-decay-at-fast-iterate condition are presented as supporting theoretical analysis, not as the load-bearing premise whose equations reduce to the performance numbers by construction. No self-definitional steps, fitted-input-called-prediction patterns, or load-bearing self-citation chains appear in the abstract or described material that would equate any claimed prediction to its own inputs. The work is therefore self-contained against external benchmarks.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Only abstract available; no explicit free parameters, axioms, or invented entities can be extracted or audited.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Anytime Training with Schedule-Free Spectral Optimization." pith.science (2026). https://pith.science/paper/CH6ZJZ5U

@misc{pith2026260523061,
  author       = {Pith},
  title        = {Pith review of: Anytime Training with Schedule-Free Spectral Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CH6ZJZ5U}},
  note         = {Machine review of arXiv:2605.23061}
}
abstract

Standard neural network training relies on learning-rate schedules tied to a fixed horizon, leading to strong path dependence and costly re-tuning as data availability changes. Schedule-Free (SF) methods address this by removing explicit schedules, yet SF-AdamW, the current state-of-the-art anytime optimizer, consistently underperforms well-tuned AdamW baselines. We propose SF-NorMuon, a schedule-free spectral optimizer that closes this gap: with a single hyperparameter configuration, SF-NorMuon matches or exceeds tuned AdamW on 125M and 772M parameter language models across $1$--$8\times$ Chinchilla horizons. On the theoretical side, we prove a stationarity guarantee for schedule-free spectral dynamics and identify weight decay at the fast iterate as essential for long-horizon stability. SF-NorMuon enables practitioners to obtain high-quality checkpoints at any point during training without committing to a horizon in advance. By closing the performance gap with tuned baselines, SF-NorMuon makes horizon-free optimization more practical, taking a step towards truly open-ended, continual learning.

Figures

Figures reproduced from arXiv: 2605.23061 by the authors.

Figure 1
Figure 1. Comparison of SF-NorMuon (this work), SF-AdamW, and tuned AdamW baselines for LLaMA-2-style transformers trained on FineWeb-100B. Left: 125M model with AdamW learning rate tuned per horizon with cosine schedule. Right: 772M model with a single optimized AdamW configuration across horizons. Dashed lines highlight that learning rate schedule for a long horizon is sub-optimal for a smaller token budget. The grid of hyp… view at source ↗
Figure 2
Figure 2. Weight decay strategies for schedule-free optimizers. Left: SF-AdamW with no decay (orange) diverges; decay at Y (blue) exhibits the best performance up to ∼30B tokens (c.f., [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Importance of explicit momentum and row-wise normalization for SF-NorMuon. Left: Ablating explicit momentum (µ = 0) significantly degrades performance, with final loss increasing from 3.14 to 3.28. This validates the importance of smoothing the gradient before computing the polar factor, as discussed in Section 2. Right: Ablating row-wise normalization leads to a smaller gap, with SF-NorMuon reaching the same loss a… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Quasi steady-state analysis for SF-NorMuon with decay at Z (averaged over layers). Left: The ratio |ρt+1 − ρt|/ρt remains below 1% after warmup, validating the quasi steady-state hypothesis (Hypothesis 3.1). Right: Alignment αt (purple) and RMS(Z) (red) over training. …
Figure 5
Figure 5. Figure 5: Learning rate sweep comparison be￾tween SF-AdamW and SF-NorMuon. Improvement across learning rates. A key practical advantage of SF-NorMuon is robustness to the choice of learning rate [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Learning rate sweep for AdamW with cosine scheduling on the 125M model across four training [PITH_FULL_IMAGE:figures/full_fig_p035_6.png]
Figure 7
Figure 7. Figure 7: Optimizer comparison on the 125M model. Scheduled methods (NorMuon, AdamW) are shown with [PITH_FULL_IMAGE:figures/full_fig_p036_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. WSqD: A Horizon-Free Learning Rate Schedule for Large Model Training

    cs.LG 2026-07 conditional novelty 6.0 of 10

    WSqD uses a horizon-independent shifted inverse-square-root base plus linear cooldown to get optimal last-iterate rates in convex stochastic optimization and match or beat tuned WSD on LLM pretraining with one reused ...

  2. Scale Weight Decay and Train Better

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Muon with weight decay scaled by η/η_max reaches the same MoE validation loss ~30% faster than constant-decay Muon while preserving asymptotic stationarity of the unregularized objective.

Reference graph

Works this paper leans on

87 extracted references · 87 canonical work pages · cited by 2 Pith papers

  1. [1]

    Springer New York, 2009.isbn: 9780387848587.doi: 10.1007/978-0-387-84858-7.url: http://dx.doi.org/ 10.1007/978-0-387-84858-7

    Trevor Hastie, Robert Tibshirani, and Jerome Friedman.The Elements of Statistical Learning. Springer New York, 2009.isbn: 9780387848587.doi: 10.1007/978-0-387-84858-7.url: http://dx.doi.org/ 10.1007/978-0-387-84858-7

    work page doi:10.1007/978-0-387-84858-7.url: 2009

  2. [2]

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville.Deep Learning.http://www.deeplearningbook. org. MIT Press, 2016

    work page 2016

  3. [3]

    M., & Bishop, H

    Christopher M. Bishop and Hugh Bishop.Deep Learning: Foundations and Concepts. Springer In- ternational Publishing, 2024.isbn: 9783031454684.doi: 10.1007/978-3-031-45468-4 .url: http: //dx.doi.org/10.1007/978-3-031-45468-4

    work page doi:10.1007/978-3-031-45468-4 2024

  4. [4]

    Language models are few-shot learners

    Tom Brown et al. “Language models are few-shot learners”. In:Advances in neural information processing systems33 (2020), pp. 1877–1901

    work page 2020

  5. [5]

    Hugo Touvron et al.LLaMA: Open and Efficient Foundation Language Models. 2023. arXiv:2302.13971 [cs.CL].url:https://arxiv.org/abs/2302.13971

    work page Pith review arXiv 2023

  6. [6]

    Wayne Xin Zhao et al.A Survey of Large Language Models. 2023. eprint:arXiv:2303.18223

    work page Pith review arXiv 2023

  7. [7]

    Shervin Minaee et al.Large Language Models: A Survey. 2024. eprint:arXiv:2402.06196

    work page Pith review arXiv 2024

  8. [8]

    Anthony Brohan et al.RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. 2023. eprint:arXiv:2307.15818

    work page Pith review arXiv 2023

Show all 87 references
  1. [9]

    Yueen Ma et al.A Survey on Vision-Language-Action Models for Embodied AI. 2024. eprint:arXiv: 2405.14093

    2024 arXiv

  2. [10]

    Moo Jin Kim et al.OpenVLA: An Open-Source Vision-Language-Action Model. 2024. eprint:arXiv: 2406.09246

    2024 arXiv

  3. [11]

    Continual lifelong learning with neural networks: A review

    German I. Parisi et al. “Continual lifelong learning with neural networks: A review”. In:Neural Networks113 (May 2019), pp. 54–71.issn: 0893-6080.doi: 10.1016/j.neunet.2019.01.012 .url: http://dx.doi.org/10.1016/j.neunet.2019.01.012

    2019 doi

  4. [12]

    Liyuan Wang et al.A Comprehensive Survey of Continual Learning: Theory, Method and Application

  5. [13]

    arXiv:2302.00487 [cs.LG].url:https://arxiv.org/abs/2302.00487

  6. [14]

    Overcoming catastrophic forgetting in neural networks

    James Kirkpatrick et al. “Overcoming catastrophic forgetting in neural networks”. In:Proceedings of the National Academy of Sciences114.13 (Mar. 2017), pp. 3521–3526.issn: 1091-6490.doi:10.1073/pnas. 1611835114.url:http://dx.doi.org/10.1073/pnas.1611835114

    2017 doi

  7. [15]

    A continual learning survey: Defying forgetting in classification tasks

    Matthias Delange et al. “A continual learning survey: Defying forgetting in classification tasks”. In: IEEE Transactions on Pattern Analysis and Machine Intelligence(2021).issn: 1939-3539.doi: 10. 1109/tpami.2021.3057446.url:http://dx.doi.org/10.1109/TPAMI.2021.3057446

    2021 doi

  8. [16]

    Step-size optimization for continual learning

    Thomas Degris et al. “Step-size optimization for continual learning”. In:arXiv preprint arXiv:2401.17401 (2024)

    2024

  9. [17]

    Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging

    Alexandru Meterez et al. “Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging”. In:arXiv preprint arXiv:2602.03702(2026)

    2026

  10. [18]

    The road less scheduled

    Aaron Defazio et al. “The road less scheduled”. In:Advances in Neural Information Processing Systems 37 (2024), pp. 9974–10007

    2024

  11. [19]

    How far away are truly hyperparameter-free learning algorithms?

    Priya Kasimbeg et al. “How far away are truly hyperparameter-free learning algorithms?” In:arXiv preprint arXiv:2505.24005(2025)

    2025

  12. [20]

    SGDR: Stochastic Gradient Descent with Warm Restarts

    Ilya Loshchilov and Frank Hutter. “SGDR: Stochastic Gradient Descent with Warm Restarts”. In: International Conference on Learning Representations. 2017.url:https://openreview.net/forum? id=Skq89Scxx

    2017

  13. [21]

    Jordan Hoffmann et al.Training Compute-Optimal Large Language Models. 2022. arXiv:2203.15556 [cs.CL].url:https://arxiv.org/abs/2203.15556

    2022 arXiv

  14. [22]

    Shengding Hu et al.MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies. 2024. arXiv:2404.06395 [cs.CL].url:https://arxiv.org/abs/2404.06395. 12

    2024 arXiv

  15. [23]

    A method for solving the convex programming problem with convergence rate O (1/k2)

    Yurii Nesterov. “A method for solving the convex programming problem with convergence rate O (1/k2)”. In:Dokl akad nauk Sssr. Vol. 269. 1983, p. 543

    1983

  16. [24]

    An optimal method for stochastic composite optimization

    Guanghui Lan. “An optimal method for stochastic composite optimization”. In:Mathematical Program- ming133.1 (2012), pp. 365–397

    2012

  17. [25]

    Alexander Hägele et al.Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations

  18. [26]

    arXiv:2405.18392 [cs.LG].url:https://arxiv.org/abs/2405.18392

  19. [27]

    Yunshui Li et al.Model Merging in Pre-training of Large Language Models. 2025. arXiv:2505.12082 [cs.CL].url:https://arxiv.org/abs/2505.12082

    2025

  20. [28]

    Averaging weights leads to wider optima and better generalization

    Pavel Izmailov et al. “Averaging weights leads to wider optima and better generalization”. In:arXiv preprint arXiv:1803.05407(2018)

    2018 arXiv

  21. [29]

    Accelerating neural network training: An analysis of the AlgoPerf competition

    Priya Kasimbeg et al. “Accelerating neural network training: An analysis of the AlgoPerf competition”. In: The Thirteenth International Conference on Learning Representations. 2025.url:https://openreview. net/forum?id=CtM5xjRSfm

    2025

  22. [30]

    Scaling laws and compute-optimal training beyond fixed training durations

    Alexander Hägele et al. “Scaling laws and compute-optimal training beyond fixed training durations”. In:Advances in Neural Information Processing Systems37 (2024), pp. 76232–76264

    2024

  23. [31]

    Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training

    Minhak Song et al. “Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training”. In:High-dimensional Learning Dynamics 2025. 2025.url: https : / / openreview.net/forum?id=b5HYeRzG9M

    2025

  24. [32]

    2026.url:https://openreview.net/forum?id=Jw7khYzYzl

    Andrei Semenov, Matteo Pagliardini, and Martin Jaggi.Benchmarking Optimizers for Large Language Model Pretraining. 2026.url:https://openreview.net/forum?id=Jw7khYzYzl

    2026

  25. [33]

    Adam: A Method for Stochastic Optimization

    Diederik P. Kingma and Jimmy Ba. “Adam: A Method for Stochastic Optimization”. In:International Conference on Learning Representations (ICLR). 2015.url:https://arxiv.org/abs/1412.6980

    2015 arXiv

  26. [34]

    Decoupled Weight Decay Regularization

    Ilya Loshchilov and Frank Hutter. “Decoupled Weight Decay Regularization”. In:International Confer- ence on Learning Representations. 2019.url:https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  27. [35]

    github.io/posts/muon/

    Keller Jordan et al.Muon: An optimizer for hidden layers in neural networks.https://kellerjordan. github.io/posts/muon/. Accessed: 2026-01-25. 2024

    2026

  28. [36]

    GitHub repository, master branch

    Keller Jordan et al.Muon (GitHub repository): An optimizer for hidden layers in neural networks. GitHub repository, master branch. 2024.url:https://github.com/KellerJordan/Muon (visited on 01/25/2026)

    2024

  29. [37]

    Chongjie Si, Debing Zhang, and Wei Shen.AdaMuon: Adaptive Muon Optimizer. 2025. arXiv:2507. 11005 [cs.LG].url:https://arxiv.org/abs/2507.11005

    2025

  30. [38]

    Zichong Li et al.NorMuon: Making Muon more efficient and scalable. 2025. arXiv:2510.05491 [cs.LG]. url:https://arxiv.org/abs/2510.05491

    2025

  31. [39]

    Dion: Distributed Orthonormalized Updates

    Kwangjun Ahn et al. “Dion: Distributed Orthonormalized Updates”. In:arXiv preprint: 2504.05295 (2025)

    2025

  32. [40]

    Kwangjun Ahn, Noah Amsel, and John Langford.Dion2: A Simple Method to Shrink Matrix in Muon

  33. [41]

    arXiv:2512.16928 [cs.LG].url:https://arxiv.org/abs/2512.16928

  34. [42]

    Kang An et al.ASGO: Adaptive Structured Gradient Optimization. 2025. arXiv:2503.20762 [cs.LG]. url:https://arxiv.org/abs/2503.20762

    2025

  35. [43]

    Liliang Ren et al.Rethinking Language Model Scaling under Transferable Hypersphere Optimization

  36. [44]

    arXiv:2603.28743 [cs.LG].url:https://arxiv.org/abs/2603.28743

    arXiv

  37. [45]

    Noah Amsel et al.The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm. 2026. arXiv:2505.16932 [cs.LG].url:https://arxiv.org/abs/2505.16932

    2026 arXiv

  38. [46]

    Ziyue Liu et al.Muon2: Boosting Muon via Adaptive Second-Moment Preconditioning. 2026. arXiv: 2604.09967 [cs.LG].url:https://arxiv.org/abs/2604.09967

    2026 arXiv

  39. [47]

    Ahmed Khaled et al.MuonBP: Faster Muon via Block-Periodic Orthogonalization. 2025. arXiv:2510. 16981 [cs.LG].url:https://arxiv.org/abs/2510.16981. 13

    2025

  40. [48]

    Alexey Kravatskiy et al.The Ky Fan Norms and Beyond: Dual Norms and Combinations for Matrix Optimization. 2025. arXiv:2512.09678 [math.OC].url:https://arxiv.org/abs/2512.09678

    2025

  41. [49]

    Jingyuan Liu et al.Muon is Scalable for LLM Training. 2025. arXiv:2502 . 16982 [cs.LG].url: https://arxiv.org/abs/2502.16982

    2025 arXiv

  42. [50]

    Kimi Team.Kimi K2.5: Visual Agentic Intelligence. 2026. arXiv:2602.02276 [cs.CL].url: https: //arxiv.org/abs/2602.02276

    2026 arXiv

  43. [51]

    GLM-5-Team.GLM-5: from Vibe Coding to Agentic Engineering. 2026. arXiv:2602.15763 [cs.LG]. url:https://arxiv.org/abs/2602.15763

    2026 arXiv

  44. [52]

    Varun Singh et al.Arcee Trinity Large Technical Report. 2026. arXiv:2602.17004 [cs.LG] .url: https://arxiv.org/abs/2602.17004

    2026

  45. [53]

    Levent Sagun et al.Empirical Analysis of the Hessian of Over-Parametrized Neural Networks. 2017. eprint:arXiv:1706.04454

    2017 arXiv

  46. [54]

    An Investigation into Neural Net Optimization via Hessian Eigenvalue Density

    Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. “An Investigation into Neural Net Optimization via Hessian Eigenvalue Density”. In:Proceedings of the 36th International Conference on Machine Learning. Ed. by Kamalika Chaudhuri and Ruslan Salakhutdinov. Vol. 97. Proceedings ...

    2019

  47. [55]

    arXiv:2512.04299 [cs.LG].url:https://arxiv.org/abs/2512.04299

    Damek Davis and Dmitriy Drusvyatskiy.When do spectral gradient updates help in deep learning?2026. arXiv:2512.04299 [cs.LG].url:https://arxiv.org/abs/2512.04299

    2026

  48. [56]

    The surprising agreement between convex optimization theory and learning-rate scheduling for large model training

    Fabian Schaipp et al. “The surprising agreement between convex optimization theory and learning-rate scheduling for large model training”. In:arXiv preprint arXiv:2501.18965(2025)

    2025

  49. [57]

    A simple weight decay can improve generalization

    Anders Krogh and John A. Hertz. “A simple weight decay can improve generalization”. In:Proceedings of the 5th International Conference on Neural Information Processing Systems. NIPS’91. Denver, Colorado: Morgan Kaufmann Publishers Inc., 1991, pp. 950–957.isbn: 1558602224

    1991

  50. [58]

    arXiv: 2310.04415 [cs.LG].url:https://arxiv.org/abs/2310.04415

    Francesco D’Angelo et al.Why Do We Need Weight Decay in Modern Deep Learning?2024. arXiv: 2310.04415 [cs.LG].url:https://arxiv.org/abs/2310.04415

    2024

  51. [59]

    Shikai Qiu et al.Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales. 2026. arXiv:2512.05620 [cs.LG].url:https://arxiv.org/abs/2512.05620

    2026

  52. [60]

    Da Chang, Yongxiang Liu, and Ganzhao Yuan.On the Convergence of Muon and Beyond. 2026. arXiv: 2509.15816 [cs.LG].url:https://arxiv.org/abs/2509.15816

    2026 arXiv

  53. [61]

    Wei Shen et al.On the Convergence Analysis of Muon. 2026. arXiv:2505.23737 [stat.ML] .url: https://arxiv.org/abs/2505.23737

    2026 arXiv

  54. [62]

    Naoki Sato, Hiroki Naganuma, and Hideaki Iiduka.Convergence Bound and Critical Batch Size of Muon Optimizer. 2025. arXiv:2507.01598 [cs.LG].url:https://arxiv.org/abs/2507.01598

    2025

  55. [63]

    Jiaxiang Li and Mingyi Hong.A Note on the Convergence of Muon. 2025. arXiv:2502.02900 [math.OC]. url:https://arxiv.org/abs/2502.02900

    2025

  56. [64]

    Jeremy Bernstein and Laker Newhouse.Old Optimizer, New Norm: An Anthology. 2024. arXiv:2409. 20325 [cs.LG].url:https://arxiv.org/abs/2409.20325

    2024 arXiv

  57. [65]

    Thomas Pethick et al.Training Deep Learning Models with Norm-Constrained LMOs. 2025. arXiv: 2502.07529 [cs.LG].url:https://arxiv.org/abs/2502.07529

    2025 arXiv

  58. [66]

    Symbolic discovery of optimization algorithms

    Xiangning Chen et al. “Symbolic discovery of optimization algorithms”. In:Advances in neural informa- tion processing systems36 (2023), pp. 49205–49233

    2023

  59. [67]

    Vineet Gupta, Tomer Koren, and Yoram Singer.Shampoo: Preconditioned Stochastic Tensor Optimiza- tion. 2018. arXiv:1802.09568 [cs.LG].url:https://arxiv.org/abs/1802.09568

    2018 arXiv

  60. [68]

    Accurate, large minibatch sgd: Training imagenet in 1 hour

    Priya Goyal et al. “Accurate, large minibatch sgd: Training imagenet in 1 hour”. In:arXiv preprint arXiv:1706.02677(2017)

    2017 arXiv

  61. [69]

    Why warmup the learning rate? underlying mechanisms and improvements

    Dayal Singh Kalra and Maissam Barkeshli. “Why warmup the learning rate? underlying mechanisms and improvements”. In:Advances in Neural Information Processing Systems37 (2024), pp. 111760–111801. 14

    2024

  62. [70]

    Learning-rate-free learning by d-adaptation

    Aaron Defazio and Konstantin Mishchenko. “Learning-rate-free learning by d-adaptation”. In:Interna- tional Conference on Machine Learning. PMLR. 2023, pp. 7449–7479

    2023

  63. [71]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer et al. “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer”. In:International Conference on Learning Representations. 2017

    2017

  64. [72]

    GShard: Scaling giant models with conditional computation and automatic sharding

    Dmitry Lepikhin et al. “GShard: Scaling giant models with conditional computation and automatic sharding”. In:International Conference on Learning Representations. 2021

    2021

  65. [73]

    Switch Transformers: scaling to trillion parameter models with simple and efficient sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. “Switch Transformers: scaling to trillion parameter models with simple and efficient sparsity”. In:Journal of Machine Learning Research23.120 (2022), pp. 1–39

    2022

  66. [74]

    Twan van Laarhoven.L2 Regularization versus Batch and Weight Normalization. 2017. arXiv:1706. 05350 [cs.LG].url:https://arxiv.org/abs/1706.05350

    2017 arXiv

  67. [75]

    Aaron Defazio.Why Gradients Rapidly Increase Near the End of Training. 2025. arXiv:2506.02285 [cs.LG].url:https://arxiv.org/abs/2506.02285

    2025

  68. [76]

    Hugo Touvron et al.Llama 2: Open Foundation and Fine-Tuned Chat Models. 2023. arXiv:2307.09288 [cs.CL].url:https://arxiv.org/abs/2307.09288

    2023 arXiv

  69. [77]

    Jianlin Su et al.RoFormer: Enhanced Transformer with Rotary Position Embedding. 2023. arXiv: 2104.09864 [cs.CL].url:https://arxiv.org/abs/2104.09864

    2023 arXiv

  70. [78]

    Biao Zhang and Rico Sennrich.Root Mean Square Layer Normalization. 2019. arXiv:1910.07467 [cs.LG].url:https://arxiv.org/abs/1910.07467

    2019 arXiv

  71. [79]

    Zhengyan Zhang et al.ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMs. 2024. arXiv:2402.03804 [cs.LG].url:https://arxiv.org/abs/2402.03804

    2024

  72. [80]

    Andrej Karpathy.NanoGPT.https://github.com/karpathy/nanoGPT. 2022

    2022

  73. [81]

    2024.url:https://github

    Keller Jordan et al.modded-nanogpt: Speedrunning the NanoGPT baseline. 2024.url:https://github. com/KellerJordan/modded-nanogpt

    2024

  74. [82]

    Guilherme Penedo et al.The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

  75. [83]

    arXiv:2406.17557 [cs.CL].url:https://arxiv.org/abs/2406.17557

    arXiv

  76. [84]

    Language models are unsupervised multitask learners

    Alec Radford et al. “Language models are unsupervised multitask learners”. In: (2019)

    2019

  77. [85]

    NVIDIA A100 Tensor Core GPU: Performance and Innovation

    Jack Choquette et al. “NVIDIA A100 Tensor Core GPU: Performance and Innovation”. In:IEEE Micro41.2 (Mar. 2021), pp. 29–35.issn: 1937-4143.doi: 10.1109/mm.2021.3061394 .url: http: //dx.doi.org/10.1109/MM.2021.3061394

    2021 doi

  78. [86]

    PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transforma- tion and Graph Compilation

    Jason Ansel et al. “PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transforma- tion and Graph Compilation”. In:Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. ASPLOS ’2...

    2024 doi

  79. [87]

    Anytime Training with Schedule-Free Spectral Optimization

    Kaichao You et al.How Does Learning Rate Decay Help Modern Neural Networks?2019. arXiv: 1908.01878 [cs.LG].url:https://arxiv.org/abs/1908.01878. 15 Technical Appendices and Supplementary Material for “Anytime Training with Schedule-Free Spectral Optimization” In this appendix,...

    2019

Pith tools

Reviewed May 25, 2026 · model on record in the stance chip above.