Muon moves faster along signal river directions early but converges slower or oscillates near optima than GD due to orthogonal updates removing scale information, supporting two-stage optimization.
arXiv preprint arXiv:2602.22681 , year=
5 Pith papers cite this work. Polarity classification is still indexing.
5
Pith papers citing it
fields
cs.LG 5years
2026 5representative citing papers
Muon outperforms Adam by reducing curvature penalty via lower Normalized Directional Sharpness, as shown via Taylor approximation on LLM training and proven on stylized quadratic problems with heterogeneous curvature.
AMUSE stabilizes Muon with time-varying schedule-free gradient evaluation, improving the performance-iteration Pareto frontier without learning-rate schedules.
Different Bregman divergences distribute unavoidable Kronecker approximation error differently across covariance spectrum, with top eigenspace better aligned to Hessian, motivating a subspace-aware optimizer using eigenvalue preconditioning on top subspace and adaptive isotropic acceleration on bott
Scale vectors in Pre-Norm LLMs aid optimization via preconditioning on linear layers rather than expressivity, and three lightweight modifications to them reduce terminal loss across model scales.
citing papers explorer
-
Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective
Muon moves faster along signal river directions early but converges slower or oscillates near optima than GD due to orthogonal updates removing scale information, supporting two-stage optimization.
-
Why Muon Outperforms Adam: A Curvature Perspective
Muon outperforms Adam by reducing curvature penalty via lower Normalized Directional Sharpness, as shown via Taylor approximation on LLM training and proven on stylized quadratic problems with heterogeneous curvature.
-
AMUSE: Anytime Muon with Stable Gradient Evaluation
AMUSE stabilizes Muon with time-varying schedule-free gradient evaluation, improving the performance-iteration Pareto frontier without learning-rate schedules.
-
Rethinking Bregman Divergences in Kronecker-Factored Optimizers
Different Bregman divergences distribute unavoidable Kronecker approximation error differently across covariance spectrum, with top eigenspace better aligned to Hessian, motivating a subspace-aware optimizer using eigenvalue preconditioning on top subspace and adaptive isotropic acceleration on bott
-
Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models
Scale vectors in Pre-Norm LLMs aid optimization via preconditioning on linear layers rather than expressivity, and three lightweight modifications to them reduce terminal loss across model scales.