REVIEW 5 cited by
Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Adaptive gradient methods such as Adam have gained increasing popularity in deep learning optimization. However, it has been observed that compared with (stochastic) gradient descent, Adam can converge to a different solution with a significantly worse test error in many deep learning applications such as image classification, even with a fine-tuned regularization. In this paper, we provide a theoretical explanation for this phenomenon: we show that in the nonconvex setting of learning over-parameterized two-layer convolutional neural networks starting from the same random initialization, for a class of data distributions (inspired from image data), Adam and gradient descent (GD) can converge to different global solutions of the training objective with provably different generalization errors, even with weight decay regularization. In contrast, we show that if the training objective is convex, and the weight decay regularization is employed, any optimization algorithms including Adam and GD will converge to the same solution if the training is successful. This suggests that the inferior generalization performance of Adam is fundamentally tied to the nonconvex landscape of deep learning optimization.
Forward citations
Cited by 5 Pith papers
-
Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
Weak-to-strong performance is governed by the overlap between the weak model's unlearnable error space and the strong model's principal-representation space, quantified by ||P_s(I-P_w)||.
-
Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)
The paper derives convergence rates and adaptive per-layer step sizes for Muon and Scion under a layer-wise (L0,L1)-smoothness assumption, and reports that these step sizes approximately match tuned values on NanoGPT.
-
GradAlign for Training-free Model Performance Inference
GradAlign scores architectures by per-sample gradient agreement at initialization and beats most existing training-free NAS metrics on NAS-Bench-101, NAS-Bench-201, and NDS.
-
A Method for Enhancing Generalization of Adam by Multiple Integrations
MIAdam adds an n-th order integral term to Adam's gradient update for the first ζ steps, then switches to Adam, and is reported to improve test accuracy and label-noise robustness.
-
Learning Causality for Modern Machine Learning
A thesis compiling six papers that use causal invariance to improve graph neural networks' out-of-distribution generalization, interpretability, and robustness.
Discussion (0). Continue with ORCID to comment.