Weight matrix dynamics during training is modeled as Dyson Brownian motion, with the learning-rate-to-batch-size ratio controlling stochasticity; verified analytically in a Gaussian RBM and empirically in a nano-GPT.
Dyson, Statistical Theory of the Energy Levels of Complex Systems
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cond-mat.dis-nn 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Dyson Brownian motion and random matrix dynamics of weight matrices during learning
Weight matrix dynamics during training is modeled as Dyson Brownian motion, with the learning-rate-to-batch-size ratio controlling stochasticity; verified analytically in a Gaussian RBM and empirically in a nano-GPT.