Trained ResNets on CIFAR-10 retain measurable dependence on initialization scale under low-LR SGD (26.5 pp test accuracy spread) but not under Adam, indicating that practical inductive bias is shaped by the forgetting time scale of the optimizer and regularizers.
On a continuous time model of gradient descent dynamics and instability in deep learning.arXiv preprint arXiv:2302.01952, 2023
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2026 1verdicts
UNVERDICTED 1representative citing papers
citing papers explorer
-
Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias
Trained ResNets on CIFAR-10 retain measurable dependence on initialization scale under low-LR SGD (26.5 pp test accuracy spread) but not under Adam, indicating that practical inductive bias is shaped by the forgetting time scale of the optimizer and regularizers.