Pith

open record

sign in

arxiv: 1705.08292 · v2 · pith:XZ7MLLMA · submitted 2017-05-23 · stat.ML · cs.LG

The Marginal Value of Adaptive Gradient Methods in Machine Learning

Ashia C. Wilson , Rebecca Roelofs , Mitchell Stern , Nathan Srebro , Benjamin Recht This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 reserved pith:XZ7MLLMArecord.jsonopen to challenge →

classification stat.ML cs.LG
keywords adaptivemethodsgradientsolutionsadagradadamdeepdescent
0
0 comments X
read the original abstract

Adaptive optimization methods, which perform local optimization with a metric constructed from the history of iterates, are becoming increasingly popular for training deep neural networks. Examples include AdaGrad, RMSProp, and Adam. We show that for simple overparameterized problems, adaptive methods often find drastically different solutions than gradient descent (GD) or stochastic gradient descent (SGD). We construct an illustrative binary classification problem where the data is linearly separable, GD and SGD achieve zero test error, and AdaGrad, Adam, and RMSProp attain test errors arbitrarily close to half. We additionally study the empirical generalization capability of adaptive methods on several state-of-the-art deep learning models. We observe that the solutions found by adaptive methods generalize worse (often significantly worse) than SGD, even when these solutions have better training performance. These results suggest that practitioners should reconsider the use of adaptive methods to train neural networks.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Learning Adaptive Coarse Spaces Using Transferable Neural Network Models for Linear and Nonlinear Overlapping Domain Decomposition Methods

    math.NA 2026-07 conditional novelty 6.0

    Neural networks trained on scalar diffusion data predict adaptive coarse basis functions for Schwarz methods, transferring without retraining to linear elasticity and nonlinear p-Laplace problems.

  2. Transfer learning-based method for automated ewaste recycling in smart cities

    cs.CV 2026-06 unverdicted novelty 2.0

    Transfer learning with fine-tuned AlexNet achieves 98% accuracy classifying smartphone e-waste into 12 classes on a small dataset via hyperparameter tuning and augmentation.

  3. Statistical Properties of Training & Generalization

    stat.ML 2026-06 unverdicted novelty 2.0

    Neural scaling laws in deep learning interact with physics constraints and inductive biases beyond classical statistics.

  4. Statistical Properties of Training & Generalization

    stat.ML 2026-06 unverdicted novelty 1.0

    Review of neural scaling laws and their relation to constraints and inductive biases when applying machine learning to physics problems.