AMUSE stabilizes Muon with time-varying schedule-free gradient evaluation, improving the performance-iteration Pareto frontier without learning-rate schedules.
arXiv preprint arXiv:2501.12243 , year=
3 Pith papers cite this work. Polarity classification is still indexing.
3
Pith papers citing it
citation-role summary
background 1
citation-polarity summary
fields
cs.LG 3roles
background 1polarities
background 1representative citing papers
Strong superposition causes neural loss to scale as the inverse of model dimension due to geometric feature overlaps, explaining scaling laws for broad frequency distributions.
GradPower applies sign-power to gradients before optimization and achieves lower terminal loss in language model pre-training across architectures, scales, datasets, and schedules.
citing papers explorer
-
AMUSE: Anytime Muon with Stable Gradient Evaluation
AMUSE stabilizes Muon with time-varying schedule-free gradient evaluation, improving the performance-iteration Pareto frontier without learning-rate schedules.
-
Superposition Yields Robust Neural Scaling
Strong superposition causes neural loss to scale as the inverse of model dimension due to geometric feature overlaps, explaining scaling laws for broad frequency distributions.
-
GradPower: Powering Gradients for Faster Language Model Pre-Training
GradPower applies sign-power to gradients before optimization and achieves lower terminal loss in language model pre-training across architectures, scales, datasets, and schedules.