The top-r gradient subspace GaLore refreshes is largely non-identifiable noise beyond ~39 of 128 directions; the apparent rotation is estimator noise, and optimizer state should be transported rather than carried blindly.
Among linear estimators Am, the minimum-mean-squared-error choice isA=U ⊤ newUold =R, i.e.ˆm=Rm
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2026 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training
The top-r gradient subspace GaLore refreshes is largely non-identifiable noise beyond ~39 of 128 directions; the apparent rotation is estimator noise, and optimizer state should be transported rather than carried blindly.