REVIEW 3 cited by
First-order Methods Almost Always Avoid Saddle Points
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We establish that first-order methods avoid saddle points for almost all initializations. Our results apply to a wide variety of first-order methods, including gradient descent, block coordinate descent, mirror descent and variants thereof. The connecting thread is that such algorithms can be studied from a dynamical systems perspective in which appropriate instantiations of the Stable Manifold Theorem allow for a global stability analysis. Thus, neither access to second-order derivative information nor randomness beyond initialization is necessary to provably avoid saddle points.
Forward citations
Cited by 3 Pith papers
-
Extending the step-size restriction for gradient descent to avoid strict saddle points
Gradient descent almost surely avoids strict saddles for step sizes alpha < 2/L, provided the Hessian eigenvalue alpha^-1 occurs only on a measure-zero set.
-
Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective
A review that frames neural networks as dynamical systems, SGD as stochastic dynamics, and training as mean-field optimal control to unify deep learning theory.
-
Distributed Stochastic Gradient Method for Non-Convex Problems with Applications in Supervised Learning
A consensus-based distributed stochastic gradient algorithm is shown to converge in mean square to a critical point for non-convex problems, and it trains distributed neural networks on MNIST with accuracy comparable ...
Discussion (0). Continue with ORCID to comment.