SWAN, a stateless optimizer combining gradient normalization and whitening, matches or beats Adam on LLaMA pretraining through 1.3B parameters with roughly half the memory and reported 2x token efficiency.
An Isometric Stochastic Optimizer
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
The Adam optimizer is the standard choice in deep learning applications. I propose a simple explanation of Adam's success: it makes each parameter's step size independent of the norms of the other parameters. Based on this principle I derive Iso, a new optimizer which makes the norm of a parameter's update invariant to the application of any linear transformation to its inputs and outputs. I develop a variant of Iso called IsoAdam that allows optimal hyperparameters to be transferred from Adam, and demonstrate that IsoAdam obtains a speedup over Adam when training a small Transformer.
citation-role summary
background 1
citation-polarity summary
fields
cs.LG 1years
2024 1verdicts
CONDITIONAL 1roles
background 1polarities
background 1representative citing papers
citing papers explorer
-
SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training
SWAN, a stateless optimizer combining gradient normalization and whitening, matches or beats Adam on LLaMA pretraining through 1.3B parameters with roughly half the memory and reported 2x token efficiency.