During training, the effective feature rank of a network's last hidden layer grows in a staircase pattern alongside loss drops, and initializing for high rank accelerates training.
On the Inductive Bias of Neural Tangent Kernels
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
citation-role summary
background 1
citation-polarity summary
fields
cs.LG 1years
2024 1verdicts
CONDITIONAL 1roles
background 1polarities
background 1representative citing papers
citing papers explorer
-
$\epsilon$-rank and the Staircase Phenomenon: New Insights into Neural Network Training Dynamics
During training, the effective feature rank of a network's last hidden layer grows in a staircase pattern alongside loss drops, and initializing for high rank accelerates training.