Reshaping momentum tensors into near-square matrices before rank-1 factorization, plus a binary sign matrix for the first momentum, cuts optimizer memory by up to 96% versus Adafactor, SM3, and CAME on the reported benchmarks.
Title resolution pending
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2024 1verdicts
REJECT 1representative citing papers
citing papers explorer
-
SMMF: Square-Matricized Momentum Factorization for Memory-Efficient Optimization
Reshaping momentum tensors into near-square matrices before rank-1 factorization, plus a binary sign matrix for the first momentum, cuts optimizer memory by up to 96% versus Adafactor, SM3, and CAME on the reported benchmarks.