Optimizers such as Adam, Shampoo, and SOAP are unified as structured Fisher approximations, and two new derived optimizers, RACS and Alice, achieve faster LLaMA pre-training than Adam at lower memory.
Preconditioner on Matrix Lie Group for SGD
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
We study two types of preconditioners and preconditioned stochastic gradient descent (SGD) methods in a unified framework. We call the first one the Newton type due to its close relationship to the Newton method, and the second one the Fisher type as its preconditioner is closely related to the inverse of Fisher information matrix. Both preconditioners can be derived from one framework, and efficiently estimated on any matrix Lie groups designated by the user using natural or relative gradient descent minimizing certain preconditioner estimation criteria. Many existing preconditioners and methods, e.g., RMSProp, Adam, KFAC, equilibrated SGD, batch normalization, etc., are special cases of or closely related to either the Newton type or the Fisher type ones. Experimental results on relatively large scale machine learning problems are reported for performance study.
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension
Optimizers such as Adam, Shampoo, and SOAP are unified as structured Fisher approximations, and two new derived optimizers, RACS and Alice, achieve faster LLaMA pre-training than Adam at lower memory.