REVIEW 7 cited by
Convergence of Adam Under Relaxed Assumptions
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
In this paper, we provide a rigorous proof of convergence of the Adaptive Moment Estimate (Adam) algorithm for a wide class of optimization objectives. Despite the popularity and efficiency of the Adam algorithm in training deep neural networks, its theoretical properties are not yet fully understood, and existing convergence proofs require unrealistically strong assumptions, such as globally bounded gradients, to show the convergence to stationary points. In this paper, we show that Adam provably converges to $\epsilon$-stationary points with ${O}(\epsilon^{-4})$ gradient complexity under far more realistic conditions. The key to our analysis is a new proof of boundedness of gradients along the optimization trajectory of Adam, under a generalized smoothness assumption according to which the local smoothness (i.e., Hessian norm when it exists) is bounded by a sub-quadratic function of the gradient norm. Moreover, we propose a variance-reduced version of Adam with an accelerated gradient complexity of ${O}(\epsilon^{-3})$.
Forward citations
Cited by 7 Pith papers
-
Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks
Bounded trajectories of a broad class of GD optimizers (Adam, RMSprop, NAG, Adan, etc.) converge with polynomial rates to critical points of KL objectives with locally Lipschitz gradients, covering analytic-activation...
-
Complexity Lower Bounds of Adaptive Gradient Algorithms for Non-convex Stochastic Optimization under Relaxed Smoothness
AdaGrad-type algorithms provably need a complexity quadratic in the initial gap and smoothness constants under relaxed smoothness, so they cannot match the optimal rate of clipped SGD.
-
A Nearly Optimal Single Loop Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness
SLIP is the first single-loop stochastic bilevel optimizer with eO(1/epsilon^4) oracle complexity under unbounded upper-level smoothness, both in expectation and with high probability.
-
PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learning
PADAM runs K differently averaged Adam trajectories in parallel, selects the one with the smallest test error, and achieves the best optimization error in nearly all of 13 tested scientific machine learning problems w...
-
AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training
AdamS replaces AdamW's second-moment storage with a momentum-and-gradient squared denominator, matching AdamW's loss curves with half the optimizer memory.
-
Sharp higher order convergence rates for the Adam optimizer
Adam can achieve the accelerated momentum convergence rate locally on smooth strongly convex problems when its momentum and step size are tuned to the condition number, while RMSprop is shown to converge at the slower...
-
Averaged Adam accelerates stochastic optimization in the training of deep neural network approximations for partial differential equation and optimal control problems
Averaged variants of Adam, using a sliding-window arithmetic average or an exponential moving average of iterates, are reported to beat Adam and SGD on the tested PDE, optimal control, and CIFAR-10 benchmarks.
Discussion (0). Continue with ORCID to comment.