Pith. sign in

REVIEW 2 major objections 2 minor 1 cited by

LionMuon alternates Lion sign steps with Muon spectral steps on a fixed period to reach lower loss at lower compute than either alone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-30 18:28 UTC pith:AZK4VD3S

load-bearing objection LionMuon alternates Lion sign steps and Muon spectral steps with a shared dual-EMA buffer to match Lion memory while claiming better performance, but the buffer compatibility after switches is the untested assumption. the 2 major comments →

arxiv 2605.19811 v2 pith:AZK4VD3S submitted 2026-05-19 cs.LG

LionMuon: Alternating Spectral and Sign Descent for Efficient Training

Arman Bolatov , Artem Riabinin , Nikita Kornilov , Andrey Veprikov , Samuel Horv\'ath , Martin Tak\'a\v{c} , Aleksandr Beznosikov This is my paper
classification cs.LG
keywords LionMuonalternating optimizerMuonLionsign descentspectral descentheavy-tailed noisePareto dominance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces LionMuon, an optimizer that switches between the inexpensive sign-based Lion updates and the stronger but costlier spectral Muon updates every P steps while sharing one dual-EMA momentum buffer. This keeps total memory equal to Lion and half of AdamW while averaging the directional quality of both. Experiments at 124M scale show that P equals 2 yields lower validation loss than Muon, Lion, Signum or AdamW at lower total compute, with the same pattern holding at 355M and 720M. Separate theory derives sharp complexity bounds under heavy-tailed noise whose constants are period averages of the two methods and correctly predict the compute-optimal P.

Core claim

LionMuon alternates between Lion's sign steps and Muon's spectral matrix-sign steps using a shared dual-EMA buffer and fixed period P. At P=2 the method Pareto-dominates Muon, Lion, Signum and AdamW on every tested dataset and architecture at 124M model size, reaching lower validation loss at lower compute; the advantage continues at 355M and 720M. A single-EMA variant called SignMuon already outperforms pure Muon. Complexity bounds under heavy-tailed noise are controlled by period-averaged smoothness and noise constants that interpolate between the parent methods and forecast when LionMuon beats both.

What carries the argument

Fixed-period alternation between Lion sign steps and Muon spectral steps that share one dual-EMA momentum buffer.

Load-bearing premise

A single fixed alternation period together with one shared buffer preserves the directional quality of both Lion and Muon steps without introducing instability or requiring per-layer tuning.

What would settle it

At 720M scale LionMuon no longer reaches lower validation loss than Muon at equal or lower total compute.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Optimizer memory footprint equals Lion and is half of AdamW while still using Muon-quality directions on half the steps.
  • SignMuon alone already improves on pure Muon, showing the spectral direction can be approximated cheaply.
  • Complexity bounds under heavy-tailed noise predict the optimal period from averaged smoothness and noise constants.
  • The same Pareto dominance appears on multiple architectures and datasets at three different model scales.
  • Period-averaged constants govern whether the hybrid beats the pure methods.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The shared buffer may let other sign-spectral hybrids be built without doubling state.
  • If the fixed period works across scales, training pipelines could drop per-layer hyperparameter search for momentum.
  • Heavy-tailed noise analysis suggests the method could be tested on even larger models where tail behavior changes.
  • The approach might combine with other low-memory techniques such as 8-bit states to push efficiency further.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces LionMuon, an optimizer that alternates between Lion sign-based updates and Muon spectral matrix-sign updates on a fixed period P while sharing a single dual-EMA momentum buffer. It claims that at P=2 this yields Pareto dominance over Muon, Lion, Signum, and AdamW on every tested dataset and architecture at 124M scale (lower validation loss at lower compute), with the advantage persisting at 355M and 720M scales; a simpler SignMuon variant already beats pure Muon. On the theory side, sharp complexity bounds are derived under heavy-tailed noise using period-averaged smoothness and noise constants that interpolate between the Lion and Muon extremes, and these bounds are said to predict the compute-optimal period.

Significance. If the empirical dominance and the supporting theory both hold, the work supplies a practical route to hybrid sign/spectral optimization that cuts average per-step cost while retaining strong directional quality, together with explicit guidance on choosing P. The public code release is a clear strength that supports reproducibility.

major comments (2)
  1. [Experiments section (124M–720M results)] The central Pareto-dominance claim at P=2 (abstract and scaling experiments) rests on the shared dual-EMA buffer preserving directional quality across alternations without per-layer adaptation or instability. No ablation isolating the shared-buffer design versus separate buffers is reported, so it remains possible that the observed gains arise from hyperparameter search rather than the alternation mechanism itself.
  2. [Theory section on heavy-tailed complexity bounds] The complexity-bound derivation (theory section) obtains period-averaged constants by treating the buffer state as compatible after each Lion/Muon switch. If a sign update materially corrupts the spectral direction stored in the shared state (or vice versa), the averaged smoothness/noise quantities no longer govern the realized trajectory, undermining the prediction of when LionMuon outruns its parents.
minor comments (2)
  1. The abstract states that the bounds are 'sharp'; the manuscript should clarify whether this means they match known lower bounds or simply improve the leading constants relative to Lion and Muon.
  2. Notation for the dual-EMA buffer (how the Lion and Muon momentum states are interleaved within the single buffer) is introduced only briefly; a short pseudocode block in the method section would improve clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We respond to each major comment below, indicating planned revisions where appropriate.

read point-by-point responses

  1. Referee: [Experiments section (124M–720M results)] The central Pareto-dominance claim at P=2 (abstract and scaling experiments) rests on the shared dual-EMA buffer preserving directional quality across alternations without per-layer adaptation or instability. No ablation isolating the shared-buffer design versus separate buffers is reported, so it remains possible that the observed gains arise from hyperparameter search rather than the alternation mechanism itself.

    Authors: We agree that an ablation isolating the shared dual-EMA buffer from separate buffers would strengthen the attribution of gains to the alternation mechanism rather than hyperparameter choices. In the revised manuscript we will add this ablation at 124M scale, comparing the shared-buffer LionMuon against a separate-buffer variant while holding all other hyperparameters fixed. revision: yes

  2. Referee: [Theory section on heavy-tailed complexity bounds] The complexity-bound derivation (theory section) obtains period-averaged constants by treating the buffer state as compatible after each Lion/Muon switch. If a sign update materially corrupts the spectral direction stored in the shared state (or vice versa), the averaged smoothness/noise quantities no longer govern the realized trajectory, undermining the prediction of when LionMuon outruns its parents.

    Authors: The theory derives period-averaged constants under the modeling assumption that the shared buffer remains sufficiently compatible after each switch. While we do not supply a formal proof ruling out material corruption, the scaling experiments show that the derived bounds correctly predict the compute-optimal period. In revision we will add an explicit statement of this assumption together with a reference to the empirical validation in the theory section. revision: partial

Circularity Check

0 steps flagged

No circularity: theory derives independent bounds; experiments report empirical results without reduction to inputs

full rationale

The paper's central derivation consists of complexity bounds under heavy-tailed noise that use period-averaged smoothness and noise constants interpolating between Lion and Muon; these are presented as proven results rather than fitted or self-referential quantities. The alternation period P and shared dual-EMA buffer are introduced as algorithmic choices whose performance is validated empirically across model scales, with no equation or claim reducing a reported 'prediction' to a fitted input by construction. No self-citations are invoked as load-bearing uniqueness theorems, and no ansatz is smuggled via prior work. The derivation chain therefore remains self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

1 free parameters · 0 axioms · 0 invented entities

Abstract-only review yields limited visibility into free parameters or axioms; P appears to be the main tunable quantity chosen for reported performance.

free parameters (1)
  • period P
    Fixed integer controlling alternation frequency; value 2 is reported as optimal for the tested regimes.

pith-pipeline@v0.9.1-grok · 5811 in / 1069 out tokens · 31672 ms · 2026-06-30T18:28:20.284885+00:00 · methodology

0 comments
read the original abstract

In large-scale optimization, the cheapness and effectiveness of update steps are the most crucial factors for a successful optimizer. Sign-based optimizers like Lion or Signum produce cheap per-step updates, whereas Muon's spectral matrix-sign update gives a much stronger direction at a substantially higher per-step cost. In this work, we propose LionMuon, which retains the effectiveness of Muon steps while considerably cutting the averaged iteration cost, similar to sign-based methods. It alternates between Lion's and Muon's updates on a fixed period P, sharing a single dual-EMA momentum buffer between them. The optimizer state memory therefore matches Lion and is exactly half of AdamW's. A simpler single-EMA variant, SignMuon, by itself already outperforms pure Muon. At P = 2, LionMuon Pareto-dominates Muon, Lion, Signum, and AdamW on every dataset and architecture we tested at 124M model size, reaching lower validation loss at lower compute, and the same advantage persists at 355M and 720M scale. On the theory side, we prove sharp complexity bounds under heavy-tailed noise which are governed by period-averaged smoothness and noise that interpolate between Muon's and Lion's constants. These bounds predict the compute-optimal period and the conditions under which LionMuon outruns Muon and Lion. Code: https://github.com/brain-lab-research/lion-muon

Figures

Figures reproduced from arXiv: 2605.19811 by Aleksandr Beznosikov, Andrey Veprikov, Arman Bolatov, Artem Riabinin, Martin Tak\'a\v{c}, Nikita Kornilov, Samuel Horv\'ath.

Figure 1
Figure 1. Figure 1: The two LMO families and the trade￾off our methods target: cheap but weaker sign￾based steps, stronger but more expensive Muon steps, and our alternating methods. Within this family, sign-based methods sit at the cheap end: Signum updates with the sign of a single momentum buffer, while Lion uses two EMA timescales but keeps the same coordinate￾wise sign step. Muon sits at the opposite end, computing the m… view at source ↗
Figure 2
Figure 2. Figure 2: From left to right, then top to bottom: Gradient norms ratios [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Best validation loss vs. total training FLOPs for all optimizers across the three datasets (columns) and [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Best validation loss vs. total training FLOPs on FineWeb / GPT-2 at [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Hyperparameter tuning heatmap across all methods. [PITH_FULL_IMAGE:figures/full_fig_p033_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: FineWeb / GPT-2 (124M): validation loss vs. iterations (left) and vs. FLOPs (right). [PITH_FULL_IMAGE:figures/full_fig_p034_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: FineWeb / LLaMA: validation loss vs. iterations (left) and vs. FLOPs (right). [PITH_FULL_IMAGE:figures/full_fig_p035_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: SlimPajama / GPT-2: validation loss vs. iterations (left) and vs. FLOPs (right). [PITH_FULL_IMAGE:figures/full_fig_p035_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: SlimPajama / LLaMA: validation loss vs. iterations (left) and vs. FLOPs (right). [PITH_FULL_IMAGE:figures/full_fig_p036_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: WikiText-103 / GPT-2: validation loss vs. iterations (left) and vs. FLOPs (right). [PITH_FULL_IMAGE:figures/full_fig_p036_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: WikiText-103 / LLaMA: validation loss vs. iterations (left) and vs. FLOPs (right). [PITH_FULL_IMAGE:figures/full_fig_p037_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: FineWeb / GPT-2 (355M, ∼ 23 TPP, 1× Chinchilla): validation loss vs. iterations (left) and vs. FLOPs (right). 37 [PITH_FULL_IMAGE:figures/full_fig_p037_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: FineWeb / GPT-2 (720M, ∼ 5 TPP, 1/4 Chinchilla): validation loss vs. iterations (left) and vs. FLOPs (right). 38 [PITH_FULL_IMAGE:figures/full_fig_p038_13.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Sign compression for Muon: SignMuon, MuonSign, and the Limits of Error Feedback

    math.OC 2026-07 conditional novelty 7.0

    Sign-compressing Muon's update to one bit can make it ascend on linear objectives; error feedback works only on the gradient side, yet the divergent sign-after-the-LMO method wins in experiments.

Reference graph

Works this paper leans on

29 extracted references · 29 canonical work pages · cited by 1 Pith paper · 12 internal anchors

  1. [1]

    Kimi K2: Open Agentic Intelligence

    10 Figure 4: Best validation loss vs. total training FLOPs on FineWeb / GPT-2 at355M(1×Chinchilla,∼23TPP, left) and720M(1/4Chinchilla,∼5TPP,right). At 355M the alternating methods (LionMuonandSignMuonat smallP) Pareto-dominate pureMuon. At under-trained 720M,SignMuonP=2still beats pureMuon, the alternation effect survives the scale jump. Kimi Team. Kimi K...

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    George E. Dahl, Frank Schneider, Zachary Nado, Naman Agarwal, Chandramouli Shama Sastry, Philipp Hennig, Sourabh Medapati, Runa Eschenhagen, Priya Kasimbeg, Daniel Suo, Juhan Bae, Justin Gilmer, Abel L. Peirson, Bilal Khan, Rohan Anil, Mike Rabbat, Shankar Krishnan, Daniel Snider, Ehsan Amid, Kongtao Chen, Chris J. Maddison, Rakshith Vasudev, Michal Badur...

    work page arXiv

  3. [3]

    Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methods

    Andrey Veprikov, Arman Bolatov, Samuel Horv´ ath, Aleksandr Beznosikov, Martin Tak´ aˇ c, and Slavomir Hanzely. Preconditioned norms: A unified framework for steepest descent, quasi-newton and adaptive methods.arXiv preprint arXiv:2510.10777,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

    Priya Goyal, Piotr Doll´ ar, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour.arXiv preprint arXiv:1706.02677,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    Muon is Scalable for LLM Training

    Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is sca...

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Old Optimizer, New Norm: An Anthology

    URLhttps://arxiv.org/ abs/2409.20325. Lizhang Chen, Jonathan Li, and Qiang Liu. Muon optimizes under spectral norm constraints.arXiv preprint arXiv:2506.15054,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    signSGD with Majority Vote is Communication Efficient And Fault Tolerant

    Jeremy Bernstein, Jiawei Zhao, Kamyar Azizzadenesheli, and Anima Anandkumar. signsgd with majority vote is communication efficient and fault tolerant.arXiv preprint arXiv:1810.05291, 2018b. Egor Petrov, Grigoriy Evseev, Aleksey Antonov, Andrey Veprikov, Nikolay Bushkov, Stanislav Moiseev, and Alek- sandr Beznosikov. Leveraging coordinate momentum in signs...

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Sign operator for coping with heavy-tailed noise in non-convex optimization: High probability bounds under(l 0, l1)-smoothness.arXiv preprint arXiv:2502.07923,

    Nikita Kornilov, Philip Zmushko, Andrei Semenov, Mark Ikonnikov, Alexander Gasnikov, and Alexander Beznosikov. Sign operator for coping with heavy-tailed noise in non-convex optimization: High probability bounds under(l 0, l1)-smoothness.arXiv preprint arXiv:2502.07923,

    work page arXiv

  9. [9]

    Sign-Based Optimizers Are Effective Under Heavy-Tailed Noise

    Dingzhi Yu, Hongyi Tao, Yuanyu Wan, Luo Luo, and Lijun Zhang. Sign-based optimizers are effective under heavy-tailed noise.arXiv preprint arXiv:2602.07425,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    HTMuon: Improving Muon via Heavy-Tailed Spectral Correction

    URLhttps://arxiv.org/abs/2603.10067. Kwangjun Ahn, Byron Xu, Natalie Abreu, Ying Fan, Gagik Magakyan, Pratyusha Sharma, Zheng Zhan, and John Langford. Dion: Distributed orthonormalized updates.arXiv preprint arXiv:2504.05295,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    arXiv preprint arXiv:2510.16981 , year=

    Ahmed Khaled, Kaan Ozkara, Tao Yu, Mingyi Hong, and Youngsuk Park. MuonBP: Faster Muon via block-periodic orthogonalization.arXiv preprint arXiv:2510.16981,

    work page arXiv

  12. [12]

    LiMuon: Light and Fast Muon Optimizer for Large Models

    Feihu Huang, Yuning Luo, and Songcan Chen. Limuon: Light and fast muon optimizer for large models.arXiv preprint arXiv:2509.14562,

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [13]

    2602.01105 , archiveprefix =

    URLhttps://arxiv.org/abs/2602.01105. Noah Amsel, David Persson, Christopher Musco, and Robert M. Gower. The Polar Express: Optimal matrix sign methods and their application to the Muon algorithm.arXiv preprint arXiv:2505.16932,

    work page arXiv

  14. [14]

    Ac- celerating newton-schulz iteration for orthogonaliza- tion via chebyshev-type polynomials.arXiv preprint arXiv:2506.10935,

    URLhttps://arxiv.org/abs/2506.10935. Maria-Eleni Sfyraki and Jun-Kun Wang. Lions and muons: Optimization via stochastic frank-wolfe.arXiv preprint arXiv:2506.04192,

    work page arXiv

  15. [15]

    Improving Generalization Performance by Switching from Adam to SGD

    Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from Adam to SGD. arXiv preprint arXiv:1712.07628,

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    Adaptive Gradient Methods with Dynamic Bound of Learning Rate

    Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate.arXiv preprint arXiv:1902.09843,

    work page internal anchor Pith review Pith/arXiv arXiv 1902

  17. [17]

    From gradient clipping to normalization for heavy tailed sgd.arXiv preprint arXiv:2410.13849,

    Florian H¨ ubler, Ilyas Fatkhullin, and Niao He. From gradient clipping to normalization for heavy tailed sgd.arXiv preprint arXiv:2410.13849,

    work page arXiv

  18. [18]

    Li and M

    Jiaxiang Li and Mingyi Hong. A note on the convergence of muon.arXiv preprint arXiv:2502.02900,

    work page arXiv

  19. [19]

    On the Convergence Analysis of Muon

    13 Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, and Jiawei Zhang. On the convergence analysis of muon. arXiv preprint arXiv:2505.23737,

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    arXiv preprint arXiv:2503.20762 , year=

    Kang An, Yuxing Liu, Rui Pan, Yi Ren, Shiqian Ma, Donald Goldfarb, and Tong Zhang. Asgo: Adaptive structured gradient optimization.arXiv preprint arXiv:2503.20762,

    work page arXiv

  21. [21]

    Improved convergence rates of muon optimizer for nonconvex optimiza- tion.arXiv preprint arXiv:2601.19400,

    Shuntaro Nagashima and Hideaki Iiduka. Improved convergence rates of muon optimizer for nonconvex optimiza- tion.arXiv preprint arXiv:2601.19400,

    work page arXiv

  22. [22]

    Muon converges under heavy-tailed noise: Nonconvex h\"{o}lder-smooth empirical risk minimiza- tion.arXiv preprint arXiv:2603.15059,

    Hideaki Iiduka. Muon converges under heavy-tailed noise: Nonconvex h\"{o}lder-smooth empirical risk minimiza- tion.arXiv preprint arXiv:2603.15059,

    work page arXiv

  23. [23]

    Benchmarking optimizers for large lan- guage model pretraining.arXiv preprint arXiv:2509.01440,

    Andrei Semenov, Matteo Pagliardini, and Martin Jaggi. Benchmarking optimizers for large language model pre- training.arXiv preprint arXiv:2509.01440,

    work page arXiv

  24. [24]

    Accelerated stochastic ExtraGradient: Mixing hessian and gradient similarity to reduce communication in distributed and federated learning.arXiv preprint arXiv:2305.15938,

    Nikita Kornilov, Aleksandr Beznosikov, and Alexander Gasnikov. Accelerated stochastic ExtraGradient: Mixing hessian and gradient similarity to reduce communication in distributed and federated learning.arXiv preprint arXiv:2305.15938,

    work page arXiv

  25. [25]

    From gradient clipping to normalization for heavy tailed sgd

    Florian H¨ ubler, Ilyas Fatkhullin, and Niao He. From gradient clipping to normalization for heavy tailed sgd. In AISTATS 2025,

    work page 2025

  26. [26]

    t−1X j=0 βj 2 (1−β 2)St−j +β 2Rt−j ⋆ # ≤β t 2 · ∥E0∥⋆ + t−1X j=0 βj+1 2 E[∥Rt−j∥⋆] +E

    ∇f(W t)−G t . Further, we use the notationsSt =∇f(W t)−G t andR t =∇f(W t)− ∇f(W t−1)to unroll the recursion: Et =β 2Et−1 + (1−β 2)St +β 2Rt =β t 2E0 + t−1X j=0 βj 2 (1−β 2)St−j +β 2Rt−j . Now, we observe that ∥Rt−j∥⋆ =∥∇f(W t−j)− ∇f(W t−j−1)∥⋆ ≤L∥W t−j −W t−j−1∥=Lη t−1∥Ut−j−1∥ ≤LA. Therefore, we estimate using the norm equivalence Assumption 3: E[∥Et∥⋆]≤...

    work page 2023

  27. [27]

    Taking the maximum over these two cases forP∈(1,∞), we getA2 = max{ηM , α·η L}

    Using the norm equality (11), we have∥Uτ ∥2 ≈αand stepsizeη τ =η L. Taking the maximum over these two cases forP∈(1,∞), we getA2 = max{ηM , α·η L}. WhenP= 1, allτsteps belong only toSmuon andA 2 =η M . Step 2: Refined analysis of theLionSteps (t∈Slion).Fort∈S lion, the update utilizes the infinite norm∥·∥∞. To use Lemmas 1 and 2, we estimate the uniform u...

    work page 2024

  28. [28]

    The selected cells areSignMuonP=2:(η M , ηL)=(3×10−3,2×10 −5);SignMuonP=5:(5×10 −3,2×10 −5);SignMuon 31 T able 3:Full experimental configuration. Model architecture Number of layers 12 Number of heads 12 Embedding dim 768 Sequence length 512 Vocabulary size 50,304 (GPT-2 BPE) Architectures GPT-2 base; LLaMA (no biases, RoPE, SwiGLU) Training schedule Iter...

    work page 2025

  29. [29]

    feel right

    32 Figure 5: Hyperparameter tuning heatmap across all methods. Inductive step:assumingM ′ t−1 = (1−β)M t−1, we have M ′ t =βM ′ t−1 + (1−β)G t =β(1−β)M t−1 + (1−β)G t = (1−β) βMt−1 +G t = (1−β)M t. Since bothmsignandsignare positively homogeneous of degree zero (i.e.,msign(αX) = msign(X)for anyα >0), the update directionsmsign(M t)andmsign(M ′ t)are ident...

    work page arXiv