A per-parameter decomposition of training loss change shows that learning is noisy, with only about half of parameters helping per step, some layers hurting overall, and learning spikes synchronized across layers.
Optimization methods for large-scale machine learning
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
citation-role summary
background 1
citation-polarity summary
fields
cs.LG 1years
2019 1verdicts
CONDITIONAL 1roles
background 1polarities
background 1representative citing papers
citing papers explorer
-
LCA: Loss Change Allocation for Neural Network Training
A per-parameter decomposition of training loss change shows that learning is noisy, with only about half of parameters helping per step, some layers hurting overall, and learning spikes synchronized across layers.