Schedule-Free SGD and AdEMAMix reduce to accelerated SGD variants, and a simplified one-momentum AdEMAMix matches the original on a 150m transformer.
we fix β1, β2 to be the optimal values from the previous sweep
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
citation-role summary
background 1
citation-polarity summary
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants
Schedule-Free SGD and AdEMAMix reduce to accelerated SGD variants, and a simplified one-momentum AdEMAMix matches the original on a 150m transformer.