Pith. sign in

REVIEW 74 references

Scaling weight decay with the learning-rate schedule keeps the optimization target honest and trains MoE models about 30% faster.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 12:50 UTC pith:K3HGYBEO

load-bearing objection Clean Muon theory plus a real MoE speedup, but the headline experiment confounds η² scheduling with simply decaying less.

arxiv 2607.23777 v1 pith:K3HGYBEO submitted 2026-07-26 cs.LG cond-mat.dis-nncs.AImath.OC

Scale Weight Decay and Train Better

Anuj Apte This is my paper
classification cs.LG cond-mat.dis-nncs.AImath.OC
keywords weight decayscaled weight decayMuonMuon-SWRobbins-Monromixture-of-expertslearning-rate scheduleasymptotic stationarity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Modern large-model training applies constant decoupled weight decay, which steadily shrinks the weights and can pull the optimizer toward a biased fixed point that is not a minimizer of the original loss. This paper proposes scaled weight decay: multiply the usual decay by the fraction of peak learning rate, so the shrink term falls as η² rather than η. That change is small enough to preserve the asymptotic stationarity guarantees of unregularized SGD and of the spectral optimizer Muon, while still stabilizing training when the learning rate is high. A steady-state analysis shows why the weight norm collapses under constant decay but plateaus under scaled decay. Empirically, Muon with scaled weight decay reaches the same validation loss roughly 30% sooner than ordinary Muon on mixture-of-experts models from 72M to 930M parameters, with the gain largest at the biggest scale tested.

Core claim

Replacing constant decoupled weight decay λη_t by the scaled rule λη_t²/η_max preserves the asymptotic stationarity guarantees of unregularized SGD and Muon, eliminates the asymptotic bias that constant decay introduces on simple quadratics, keeps weight norms from collapsing, and yields roughly 30% fewer steps to the same validation loss when Muon trains MoE transformers at the largest scale studied.

What carries the argument

Scaled weight decay: the update multiplies the weight by (1 − λ η_t²/η_max) instead of (1 − λ η_t). Because the extra shrink is O(η_t²) it is summable under Robbins–Monro schedules and does not shift the stationary set of the original loss.

Load-bearing premise

That the speed-up, which grows from about 22% at 72M parameters to about 29% at 930M, will keep growing or at least hold when models reach frontier size and token counts.

What would settle it

Train a matched pair of MoE models at a substantially larger scale (or several independent seeds at the current largest scale) with identical hyperparameters except constant versus scaled weight decay; if scaled decay no longer reaches the same validation loss in meaningfully fewer steps, the central practical claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A one-line change to the decay coefficient can cut pre-training steps by tens of percent without retuning other hyperparameters.
  • Constant decoupled weight decay should not be treated as asymptotically faithful to the original loss; scaled decay restores that faithfulness for both SGD and Muon.
  • Weight-norm collapse under long cosine schedules is largely an artifact of constant decay and can be avoided by the scaled rule.
  • The same scaling principle is expected to transfer to other matrix optimizers (Shampoo, SOAP, Aurora, Muon variants) with only local code changes.
  • If the scale trend continues, frontier MoE pre-training budgets can be reduced while keeping the same target loss.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the method needs no extra compute and almost no new hyperparameters, it is a candidate default for any long-horizon run that already uses decoupled weight decay.
  • The same O(η²) shrink idea may stabilize other schedule-dependent regularizers that currently enter at order η and bias late training.
  • Post-training stages that inherit a collapsed weight norm from constant-decay pre-training may behave differently if the base model was trained with scaled decay; that interaction is untested.
  • Joint retuning of peak learning rate and decay coefficient under the scaled rule could enlarge the observed speed-up beyond the hyperparameter-matched comparison reported here.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Circularity Check

0 steps flagged

No significant circularity: theory targets the unregularized loss by construction of the O(η²) shrink, and the 30% speedup is a direct head-to-head measurement, not a fitted quantity renamed as prediction.

full rationale

The paper’s load-bearing chain does not reduce inputs to outputs by definition. Scaled weight decay is proposed from Robbins–Monro summability (shrink at O(η²_t) rather than O(η_t)); Theorems 3.1–3.4 are standard smooth/strongly-convex and Muon descent arguments showing the extra term is a summable perturbation, so stationarity is w.r.t. the original unregularized f—not a quantity defined from the method’s own fit. Propositions 3.1–3.2 exhibit constant-decay fixed points on explicit quadratics; those fixed points are derived from the update, not smuggled in. The empirical claim (Table 1, Fig. 1) is a direct Muon vs Muon-SW comparison at fixed nominal λ and schedule, not a prediction recovered from a fitted parameter. Section 5’s steady-state r★_t (Eq. 36) is obtained by setting r_{t+1}=r_t in the RMS recursion; the alignment coefficients k≈20.8 and a0≈7.4×10^{-3} are descriptive late-run fits used only to explain why norms plateau vs collapse, and are not used to manufacture the speedup number. Citation of Defazio [22] (same η/η_max rule, different motivation) and of the author’s prior schedule-free work is ordinary related-work attribution, not a load-bearing uniqueness theorem. Confounds about cumulative decay budget are experimental-design issues, not circularity. Score 0; steps empty.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 1 invented entities

The central empirical claim rests on standard smooth nonconvex / strongly convex stochastic approximation assumptions plus a modeling choice that peak LRs transferred by μP and cube-root horizon rules are fair for both decay schedules. Theory adds a bounded-iterate hypothesis in the nonconvex regime and treats Newton–Schulz polar approximation as close enough that the ideal polar analysis still guides practice. No new physical entities; free parameters are ordinary optimizer hyperparameters held fixed across arms.

free parameters (5)
  • weight decay coefficient λ = 0.1
    Fixed at 0.1 for all runs to match frontier recipes; not swept against scaled vs constant decay.
  • peak learning rate η_max (per width/horizon) = 0.02 base; 1e-2 to 4.8e-3 across widths in Table 1
    Set by μP transfer (η=0.02 at reference) plus cube-root horizon rescaling; validated on short sweeps but not re-optimized separately for Muon-SW.
  • Muon shape scale α = 0.2
    Fixed at 0.2 so Muon RMS matches AdamW; standard in Muon literature, not tuned here.
  • alignment fit coefficients k, a0 for steady-state norm = k≈20.8 (scaled); a0≈7.4e-3, k'≈40.5 (constant)
    Linear fits of weight–update alignment vs η in late training (Fig. 8); used only for explanatory steady-state limits, not for the speedup claim.
  • tokens per active parameter = ~600–650
    Training horizon chosen ~600–650 tok/active to mirror frontier MoE recipes; defines the regime of the empirical claim.
axioms (6)
  • standard math Robbins–Monro conditions: Σ η_t = ∞ and Σ η_t² < ∞ for asymptotic stationarity of stochastic approximation.
    Invoked throughout §3 as the justification for placing decay at O(η²).
  • domain assumption Objective is L-smooth and bounded below; stochastic gradients have bounded variance of the form E[||g||²|F] ≤ σ² + ρ||∇F||².
    Standard nonconvex SGD assumptions in Theorems 3.1–3.2.
  • domain assumption Iterates remain bounded in mean square (sup E||x_t||² < ∞) in the nonconvex scaled-decay analysis.
    Explicit hypothesis in Theorems 3.1 and 3.3; not automatic without strong convexity.
  • domain assumption Muon update direction is the polar factor (spectral-norm ball projection) of the momentum buffer, with ||O_t||_op ≤ 1.
    Defines Muon; used for the constant-decay spectral ball trap (Prop. 3.2) and nuclear-norm stationarity (Thm 3.3).
  • domain assumption μP-MoE width transfer and cube-root LR–horizon rule yield near-optimal peak LRs transferable across the compared methods.
    §4 and Appendix C; underwrites the claim that only the decay rule differs between arms.
  • ad hoc to paper Five Newton–Schulz iterations are a sufficient practical surrogate for exact polar orthogonalization.
    Algorithm 1 / Fig. 6 show NS5 yields ~0.88 of ideal Frobenius norm; theory assumes exact polar, empirics correct the steady-state formula with measured magnitude.
invented entities (1)
  • Muon-SW (Muon with scaled weight decay) independent evidence
    purpose: Name the optimizer variant that multiplies decoupled decay by η_t/η_max inside the Muon update.
    Not a new physical or mathematical object—just a named combination of existing Muon and the Defazio-style scaled decay rule.

pith-pipeline@v1.2.0-grok45-kimik3 · 34328 in / 4292 out tokens · 93740 ms · 2026-07-30T12:50:33.884415+00:00 · methodology

0 comments
read the original abstract

The discovery of scaling laws has motivated training neural networks on ever increasing quantities of data. This is typically done with a constant decoupled weight decay which causes the network weights to shrink steadily over the course of training. Taking inspiration from the Robbins--Monro conditions, we propose to scale weight decay by the fraction of the peak learning rate $\eta/\eta_{\max}$. We prove that this scaled weight decay preserves the asymptotic stationarity guarantees of the corresponding unregularized methods for both stochastic gradient descent and the non-Euclidean spectral optimizer Muon, thereby avoiding the additional asymptotic bias introduced by constant decoupled weight decay. This retains the stability benefits of weight decay without changing the asymptotic optimization target. Using a steady-state analysis, we explain why under standard weight decay the weight norm shrinks steadily as training proceeds, whereas under scaled weight decay it settles to a roughly constant value. When applied to the training of mixture-of-experts models, Muon with scaled weight decay (Muon-SW) consistently outpaces Muon with identical hyperparameters, reaching the same validation loss $\mathbf{30\%}$ faster at our largest scale across models from $72 - 930$ million parameters trained at $\sim 600$ tokens per active parameter. If this trend continues to hold, the method promises to substantially accelerate the pre-training of frontier models while requiring only a few lines of code to implement.

Figures

Figures reproduced from arXiv: 2607.23777 by Anuj Apte.

Figure 1
Figure 1. Figure 1: Scaled weight decay trains faster and keeps weights from collapsing (width-1024 MoE, ∼932M parameters). Left: validation loss versus optimizer steps for Muon with standard weight decay and Muon-SW with scaled weight decay (decay factor η 2 t λ/ηmax instead of ηtλ). Muon-SW reaches the best validation loss of the standard run (2.675) in 264k steps versus 374k, a ∼ 30% reduction in steps (dashed arrow). Righ… view at source ↗
Figure 2
Figure 2. Figure 2: Scaled weight decay converges to a minimizer of the original loss, while constant weight decay does [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Weight–update alignment and steady-state weight norm (width 1024). Left: the aggre￾gate alignment at between the weights and the Muon update is negative under both rules, with its magnitude decreasing through training as the learning rate and relative update size fall. Right: the observed RMS weight norm (solid) against the instantaneous steady-state prediction (dashed). The theory captures the separation … view at source ↗
Figure 4
Figure 4. Figure 4: µP-MoE learning-rate transfer across width. We train µP LLaMA-style sparse MoE models at widths 128, 256, and 512 with fixed MoE granularity. The y-axis shows excess validation loss relative to the best learning rate at each width, making the transfer shape comparable across model sizes. A broad shared low-loss basin appears around 10−2–3 × 10−2 , and we select η = 0.02 for subsequent experiments. To test … view at source ↗
Figure 5
Figure 5. Figure 5: Learning-rate transfer across training horizon. We train the µP LLaMA-style sparse MoE model at width 256 for horizons T ∈ {5000, 10000, 20000} optimizer steps and sweep the peak AdamW learning rate. The y-axis shows excess validation loss relative to the best learning rate at each horizon; the x-axis is the horizon-rescaled learning rate η (T /5000)1/3 . Under this rescaling the three curves share a commo… view at source ↗
Figure 6
Figure 6. Figure 6: Imperfect orthogonalization of the five-step Newton–Schulz polar map (width 1024). We plot the layer-averaged normalized Frobenius norm 1 L P ℓ ∥U (ℓ) t ∥F / p min(mℓ, nℓ) of the applied Muon update. An exact full-rank polar factor would give a ratio of one; the five-step Newton–Schulz output instead stays near 0.88 throughout training. 40 80 120 160 Tokens (B) 80k 160k 240k 320k Training steps -0.010 -0.0… view at source ↗
Figure 7
Figure 7. Figure 7: Weight–gradient and weight–momentum alignment (width 1024). Left: the raw gradient stays nearly orthogonal to the weights under both decay rules, ⟨Wt, Gt⟩ ≈ 0, as expected from the scale￾invariance induced by the normalization layers. Right: the momentum buffer which is an exponential average of past gradients, acquires an appreciably negative alignment with the weights. As the learning rate decays the new… view at source ↗
Figure 8
Figure 8. Figure 8: Late-run weight–update alignment versus learning rate (width 1024). Aggregate weight– update alignment a against the learning rate η over the final 50k steps (steps 326k–376k); the observed samples lie on the right, and the dashed fits are extended toward η = 0 to expose their asymptotics. Scaled weight decay is well described by the origin-constrained relation a ≈ −20.8 η, so that a/η approaches a finite … view at source ↗
Figure 9
Figure 9. Figure 9: Validation loss and weight norm at width 512. Left: validation loss for Muon (constant weight decay) and Muon-SW (scaled weight decay); the dashed arrow marks the reduction in steps for the scaled run to reach the best constant-decay loss of 2.971 (108,250 vs. 78,500 steps, a ∼27% reduction). Right: the scaled-decay weight norm rises quickly and remains near 0.20, while the constant-decay norm peaks early … view at source ↗
Figure 10
Figure 10. Figure 10: Validation loss and weight norm at width 768. Left: the best constant-decay loss of 2.799 is reached at 219,000 steps, and Muon-SW reaches it at 161,000 steps, a ∼26% reduction. Right: scaled weight decay holds the norm near 0.17 while the constant-decay norm turns over and decays to ∼0.04. Together with the width-512 result, this confirms that the loss improvement and contrasting norm dynamics are not sp… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

74 extracted references · 50 linked inside Pith

  1. [1]

    Guilherme Penedo et al.The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

  2. [2]

    Jared Kaplan et al.Scaling Laws for Neural Language Models. 2020. arXiv:2001.08361 [cs.LG].url: https://arxiv.org/abs/2001.08361

    Pith/arXiv arXiv 2020

  3. [3]

    Jordan Hoffmann et al.Training Compute-Optimal Large Language Models. 2022. arXiv:2203.15556 [cs.CL].url:https://arxiv.org/abs/2203.15556

    Pith/arXiv arXiv 2022

  4. [4]

    Explaining neural scaling laws

    Yasaman Bahri et al. “Explaining neural scaling laws”. In:Proceedings of the National Academy of Sciences121.27 (2024), e2311878121.doi:10.1073/pnas.2311878121. eprint:https://www.pnas. org/doi/pdf/10.1073/pnas.2311878121.url:https://www.pnas.org/doi/abs/10.1073/pnas. 2311878121

    doi:10.1073/pnas.2311878121 2024

  5. [5]

    Alexey Dosovitskiy et al.An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. 2021. arXiv:2010.11929 [cs.CV].url:https://arxiv.org/abs/2010.11929

    Pith/arXiv arXiv 2021

  6. [6]

    Xiaohua Zhai et al.Scaling Vision Transformers. 2022. arXiv:2106 . 04560 [cs.CV].url:https : //arxiv.org/abs/2106.04560

    Pith/arXiv arXiv 2022

  7. [7]

    Potsawee Manakul et al.Scaling Open Discrete Audio Foundation Models with Interleaved Semantic, Acoustic, and Text Tokens. 2026. arXiv:2602.16687 [cs.SD].url:https://arxiv.org/abs/2602. 16687

    arXiv 2026

  8. [8]

    Sebastian Sartor and Neil Thompson.Neural Scaling Laws in Robotics. 2025. arXiv:2405 . 14005 [cs.RO].url:https://arxiv.org/abs/2405.14005

    Pith/arXiv arXiv 2025

  9. [9]

    Kimi Team et al.Kimi K2: Open Agentic Intelligence. 2026. arXiv:2507.20534 [cs.LG].url:https: //arxiv.org/abs/2507.20534

    Pith/arXiv arXiv 2026

  10. [10]

    5 Team et al.GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models. 2025. arXiv: 2508.06471 [cs.CL].url:https://arxiv.org/abs/2508.06471

    Pith/arXiv arXiv 2025

  11. [11]

    DeepSeek-AI et al.DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. 2026. arXiv:2606.19348 [cs.CL].url:https://arxiv.org/abs/2606.19348

    arXiv 2026

  12. [12]

    A Simple Weight Decay Can Improve Generalization

    Anders Krogh and John Hertz. “A Simple Weight Decay Can Improve Generalization”. In:Advances in Neural Information Processing Systems. Ed. by J. Moody, S. Hanson, and R.P. Lippmann. Vol. 4. Morgan-Kaufmann, 1991.url:https : / / proceedings . neurips . cc / paper _ files / paper / 1991 / file/8eefcfdf5990e441f0fb6f3fad709e21-Paper.pdf

    1991

  13. [13]

    arXiv: 2310.04415 [cs.LG].url:https://arxiv.org/abs/2310.04415

    Francesco D’Angelo et al.Why Do We Need Weight Decay in Modern Deep Learning?2024. arXiv: 2310.04415 [cs.LG].url:https://arxiv.org/abs/2310.04415

    Pith/arXiv arXiv 2024

  14. [14]

    Ilya Loshchilov and Frank Hutter.Decoupled Weight Decay Regularization. 2019. arXiv:1711.05101 [cs.LG].url:https://arxiv.org/abs/1711.05101

    Pith/arXiv arXiv 2019

  15. [15]

    Dayal Singh Kalra and Maissam Barkeshli.Why Warmup the Learning Rate? Underlying Mechanisms and Improvements. 2024. arXiv:2406.09405 [cs.LG].url:https://arxiv.org/abs/2406.09405

    Pith/arXiv arXiv 2024

  16. [16]

    Ilya Loshchilov and Frank Hutter.SGDR: Stochastic Gradient Descent with Warm Restarts. 2017. arXiv:1608.03983 [cs.LG].url:https://arxiv.org/abs/1608.03983

    Pith/arXiv arXiv 2017

  17. [17]

    2024.url:https : //kellerjordan.github.io/posts/muon/

    Keller Jordan et al.Muon: An optimizer for hidden layers in neural networks. 2024.url:https : //kellerjordan.github.io/posts/muon/

    2024

  18. [18]

    Jingyuan Liu et al.Muon is Scalable for LLM Training. 2025. arXiv:2502 . 16982 [cs.LG].url: https://arxiv.org/abs/2502.16982

    Pith/arXiv arXiv 2025

  19. [19]

    A Stochastic Approximation Method

    Herbert Robbins and Sutton Monro. “A Stochastic Approximation Method”. In:The Annals of Math- ematical Statistics22.3 (1951), pp. 400–407.issn: 0003-4851.doi:10.1214/aoms/1177729586.url: http://dx.doi.org/10.1214/aoms/1177729586

    arXiv 1951

  20. [20]

    Noam Shazeer et al.Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. 2017. arXiv:1701.06538 [cs.LG].url:https://arxiv.org/abs/1701.06538. 14

    Pith/arXiv arXiv 2017

  21. [21]

    William Fedus, Barret Zoph, and Noam Shazeer.Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. 2022. arXiv:2101.03961 [cs.LG].url:https://arxiv. org/abs/2101.03961

    Pith/arXiv arXiv 2022

  22. [22]

    Aaron Defazio.Why Gradients Rapidly Increase Near the End of Training. 2025. arXiv:2506.02285 [cs.LG].url:https://arxiv.org/abs/2506.02285

    Pith/arXiv arXiv 2025

  23. [23]

    Vineet Gupta, Tomer Koren, and Yoram Singer.Shampoo: Preconditioned Stochastic Tensor Optimiza- tion. 2018. arXiv:1802.09568 [cs.LG].url:https://arxiv.org/abs/1802.09568

    Pith/arXiv arXiv 2018

  24. [24]

    Rohan Anil et al.Scalable Second Order Optimization for Deep Learning. 2021. arXiv:2002.09018 [cs.LG].url:https://arxiv.org/abs/2002.09018

    Pith/arXiv arXiv 2021

  25. [25]

    Nikhil Vyas et al.SOAP: Improving and Stabilizing Shampoo using Adam. 2025. arXiv:2409.11321 [cs.LG].url:https://arxiv.org/abs/2409.11321

    Pith/arXiv arXiv 2025

  26. [26]

    Alec Dewulf et al.Aurora: A Leverage-Aware Spectral Optimizer. 2026. arXiv:2606.27715 [cs.LG]. url:https://arxiv.org/abs/2606.27715

    Pith/arXiv arXiv 2026

  27. [27]

    Dion: Distributed Orthonormalized Updates

    Kwangjun Ahn et al. “Dion: Distributed Orthonormalized Updates”. In:arXiv preprint: 2504.05295 (2025)

    arXiv 2025

  28. [28]

    Kwangjun Ahn, Noah Amsel, and John Langford.Dion2: A Simple Method to Shrink Matrix in Muon

  29. [29]

    Noah Amsel et al.The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm. 2026. arXiv:2505.16932 [cs.LG].url:https://arxiv.org/abs/2505.16932

    Pith/arXiv arXiv 2026

  30. [30]

    Anuj Apte et al.Anytime Training with Schedule-Free Spectral Optimization. 2026. arXiv:2605.23061 [cs.LG].url:https://arxiv.org/abs/2605.23061

    Pith/arXiv arXiv 2026

  31. [31]

    Kang An et al.ASGO: Adaptive Structured Gradient Optimization. 2025. arXiv:2503.20762 [cs.LG]. url:https://arxiv.org/abs/2503.20762

    arXiv 2025

  32. [32]

    Ziyue Liu et al.Muon 2: Boosting Muon via Adaptive Second-Moment Preconditioning. 2026. arXiv: 2604.09967 [cs.LG].url:https://arxiv.org/abs/2604.09967

    Pith/arXiv arXiv 2026

  33. [33]

    Ahmed Khaled et al.MuonBP: Faster Muon via Block-Periodic Orthogonalization. 2025. arXiv:2510. 16981 [cs.LG].url:https://arxiv.org/abs/2510.16981

    arXiv 2025

  34. [34]

    Optimization Methods for Large-Scale Machine Learning

    L´ eon Bottou, Frank E. Curtis, and Jorge Nocedal. “Optimization Methods for Large-Scale Machine Learning”. In:SIAM Review60.2 (2018), pp. 223–311

    2018

  35. [35]

    Kai Lion et al.Muown: Row-Norm Control for Muon Optimization. 2026. arXiv:2605.10797 [cs.LG]. url:https://arxiv.org/abs/2605.10797

    Pith/arXiv arXiv 2026

  36. [36]

    Naoki Sato, Hiroki Naganuma, and Hideaki Iiduka.Convergence Bound and Critical Batch Size of Muon Optimizer. 2026. arXiv:2507.01598 [cs.LG].url:https://arxiv.org/abs/2507.01598

    Pith/arXiv arXiv 2026

  37. [37]

    On the Convergence Analysis of Muon

    Wei Shen et al. “On the Convergence Analysis of Muon”. In:arXiv preprint arXiv:2505.23737(2025)

    Pith/arXiv arXiv 2025

  38. [38]

    Da Chang, Yongxiang Liu, and Ganzhao Yuan.On the Convergence of Muon and Beyond. 2026. arXiv: 2509.15816 [cs.LG].url:https://arxiv.org/abs/2509.15816

    Pith/arXiv arXiv 2026

  39. [39]

    Language Models are Unsupervised Multitask Learners

    Alec Radford et al. “Language Models are Unsupervised Multitask Learners”. In: (2019)

    2019

  40. [40]

    Hugo Touvron et al.Llama 2: Open Foundation and Fine-Tuned Chat Models. 2023. arXiv:2307.09288 [cs.CL].url:https://arxiv.org/abs/2307.09288

    Pith/arXiv arXiv 2023

  41. [41]

    Greg Yang et al.Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer. 2022. arXiv:2203.03466 [cs.LG].url:https://arxiv.org/abs/2203.03466

    Pith/arXiv arXiv 2022

  42. [42]

    Jan Ma la´ snicki et al.µ-Parametrization for Mixture of Experts. 2025. arXiv:2508.09752 [cs.LG]. url:https://arxiv.org/abs/2508.09752

    arXiv 2025

  43. [43]

    Johan Bjorck et al.Scaling Optimal LR Across Token Horizons. 2025. arXiv:2409.19913 [cs.LG]. url:https://arxiv.org/abs/2409.19913

    Pith/arXiv arXiv 2025

  44. [45]

    Twan van Laarhoven.L2 Regularization versus Batch and Weight Normalization. 2017. arXiv:1706. 05350 [cs.LG].url:https://arxiv.org/abs/1706.05350

    Pith/arXiv arXiv 2017

  45. [46]

    Atli Kosson, Bettina Messmer, and Martin Jaggi.Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks. 2024. arXiv:2305.17212 [cs.LG].url:https://arxiv.org/abs/ 2305.17212

    Pith/arXiv arXiv 2024

  46. [47]

    Lizhang Chen et al.Cautious Weight Decay. 2026. arXiv:2510.12402 [cs.LG].url:https://arxiv. org/abs/2510.12402

    arXiv 2026

  47. [48]

    2022.url: https://openreview.net/forum?id=J7V_4aauV6B

    Zeke Xie, Issei Sato, and Masashi Sugiyama.Understanding and Scheduling Weight Decay. 2022.url: https://openreview.net/forum?id=J7V_4aauV6B

    2022

  48. [49]

    Kaiyue Wen et al.Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimiza- tion. 2026. arXiv:2606.16899 [cs.LG].url:https://arxiv.org/abs/2606.16899

    arXiv 2026

  49. [50]

    Thinking Machines Lab.url:https://thinkingmachines.ai/news/introducing-inkling/

  50. [51]

    Albert Gu and Tri Dao.Mamba: Linear-Time Sequence Modeling with Selective State Spaces. 2024. arXiv:2312.00752 [cs.LG].url:https://arxiv.org/abs/2312.00752

    Pith/arXiv arXiv 2024

  51. [52]

    Tri Dao and Albert Gu.Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. 2024. arXiv:2405.21060 [cs.LG].url:https://arxiv.org/abs/ 2405.21060

    Pith/arXiv arXiv 2024

  52. [53]

    Liliang Ren et al.Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling. 2025. arXiv:2406.07522 [cs.CL].url:https://arxiv.org/abs/2406.07522

    Pith/arXiv arXiv 2025

  53. [54]

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh.Gated Delta Networks: Improving Mamba2 with Delta Rule. 2025. arXiv:2412.06464 [cs.CL].url:https://arxiv.org/abs/2412.06464

    Pith/arXiv arXiv 2025

  54. [55]

    Yifei Zuo et al.Parallax: Parameterized Local Linear Attention for Language Modeling. 2026. arXiv: 2605.29157 [cs.LG].url:https://arxiv.org/abs/2605.29157

    Pith/arXiv arXiv 2026

  55. [56]

    Long Ouyang et al.Training language models to follow instructions with human feedback. 2022. arXiv: 2203.02155 [cs.CL].url:https://arxiv.org/abs/2203.02155

    Pith/arXiv arXiv 2022

  56. [57]

    Paul Christiano et al.Deep reinforcement learning from human preferences. 2023. arXiv:1706.03741 [stat.ML].url:https://arxiv.org/abs/1706.03741

    Pith/arXiv arXiv 2023

  57. [58]

    Rafael Rafailov et al.Direct Preference Optimization: Your Language Model is Secretly a Reward Model

  58. [59]

    Zhihong Shao et al.DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. 2024. arXiv:2402.03300 [cs.CL].url:https://arxiv.org/abs/2402.03300

    Pith/arXiv arXiv 2024

  59. [60]

    DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning

    Daya Guo et al. “DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning”. In: Nature645.8081 (2025), pp. 633–638.issn: 1476-4687.doi:10.1038/s41586- 025- 09422- z.url: http://dx.doi.org/10.1038/s41586-025-09422-z

    doi:10.1038/s41586- 2025

  60. [61]

    Jakub Krajewski et al.Scaling Laws for Fine-Grained Mixture of Experts. 2024. arXiv:2402.07871 [cs.LG].url:https://arxiv.org/abs/2402.07871

    Pith/arXiv arXiv 2024

  61. [62]

    arXiv:2305.18290 [cs.LG].url:https://arxiv.org/abs/2305.18290

    Pith/arXiv arXiv

  62. [63]

    Ege Erdil.Inference economics of language models. 2025. arXiv:2506.04645 [cs.LG].url:https: //arxiv.org/abs/2506.04645

    Pith/arXiv arXiv 2025

  63. [64]

    Root Mean Square Layer Normalization

    Biao Zhang and Rico Sennrich. “Root Mean Square Layer Normalization”. In:Advances in Neural Information Processing Systems. Ed. by H. Wallach et al. Vol. 32. Curran Associates, Inc., 2019.url: https://proceedings.neurips.cc/paper_files/paper/2019/file/1e8a19426224ca89e83cef47f1e7f53b- Paper.pdf

    2019

  64. [65]

    Jianlin Su et al.RoFormer: Enhanced Transformer with Rotary Position Embedding. 2023. arXiv: 2104.09864 [cs.CL].url:https://arxiv.org/abs/2104.09864

    Pith/arXiv arXiv 2023

  65. [66]

    Changxin Tian et al.Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models. 2025. arXiv:2507.17702 [cs.CL].url:https://arxiv.org/abs/2507.17702

    arXiv 2025

  66. [67]

    Benchmarking Optimizers for Large Language Model Pretraining

    Andrei Semenov, Matteo Pagliardini, and Martin Jaggi. “Benchmarking Optimizers for Large Language Model Pretraining”. In:arXiv preprint arXiv:2509.01440(2025).url:https : / / arxiv . org / abs / 2509.01440

    Pith/arXiv arXiv 2025

  67. [68]

    Simon, and Jeremy Bernstein.A Spectral Condition for Feature Learning

    Greg Yang, James B. Simon, and Jeremy Bernstein.A Spectral Condition for Feature Learning. 2024. arXiv:2310.17813 [cs.LG].url:https://arxiv.org/abs/2310.17813

    Pith/arXiv arXiv 2024

  68. [69]

    Greg Yang et al.Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks. 2023. arXiv:2310.02244 [cs.NE].url:https://arxiv.org/abs/2310.02244

    Pith/arXiv arXiv 2023

  69. [70]

    Noam Shazeer.GLU Variants Improve Transformer. 2020. arXiv:2002.05202 [cs.LG].url:https: //arxiv.org/abs/2002.05202. 16

    Pith/arXiv arXiv 2020

  70. [71]

    lr"], group[

    Adam Paszke et al.PyTorch: An Imperative Style, High-Performance Deep Learning Library. 2019. arXiv:1912.01703 [cs.LG].url:https://arxiv.org/abs/1912.01703. 17 Technical Appendices and Supplementary Material In this appendix, we provide additional background, experimental details, proofs, and implementation details that supplement the main text. This appe...

    Pith/arXiv arXiv 2019

  71. [74]

    Liliang Ren et al.Rethinking Language Model Scaling under Transferable Hypersphere Optimization

  72. [75]

    arXiv:2603.28743 [cs.LG].url:https://arxiv.org/abs/2603.28743

    Pith/arXiv arXiv

  73. [2024]

    arXiv:2406.17557 [cs.CL].url:https://arxiv.org/abs/2406.17557

    Pith/arXiv arXiv

  74. [2025]

    arXiv:2512.16928 [cs.LG].url:https://arxiv.org/abs/2512.16928

    arXiv