REVIEW 5 cited by
On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Gradient descent (GD) and stochastic gradient descent (SGD) are the workhorses of large-scale machine learning. While classical theory focused on analyzing the performance of these methods in convex optimization problems, the most notable successes in machine learning have involved nonconvex optimization, and a gap has arisen between theory and practice. Indeed, traditional analyses of GD and SGD show that both algorithms converge to stationary points efficiently. But these analyses do not take into account the possibility of converging to saddle points. More recent theory has shown that GD and SGD can avoid saddle points, but the dependence on dimension in these analyses is polynomial. For modern machine learning, where the dimension can be in the millions, such dependence would be catastrophic. We analyze perturbed versions of GD and SGD and show that they are truly efficient---their dimension dependence is only polylogarithmic. Indeed, these algorithms converge to second-order stationary points in essentially the same time as they take to converge to classical first-order stationary points.
Forward citations
Cited by 5 Pith papers
-
Stochastic Saddle Avoidance Beyond Unit Excitation and Smoothness: A Pathwise Lyapunov-Perron Framework
A new pathwise Lyapunov-Perron framework proves almost sure saddle avoidance for stochastic recursions without unit excitation, covering SGD, mirror descent, proximal stochastic gradient, and random reshuffling.
-
Globally aware optimization with resurgence
SURGE proposes to find critical loss values from Borel singularities of a partition function and use them to scale learning rates during gradient descent.
-
Second-Order Guarantees of Stochastic Gradient Descent in Non-Convex Optimization
Under a relative gradient-noise bound plus a local noise condition at saddles, SGD reaches an approximate second-order stationary point in O(1/(μ²τ)) iterations.
-
Learning from Limited and Imperfect Data
A doctoral thesis compiling nine peer-reviewed papers on long-tailed image generation, long-tailed recognition, semi-supervised learning, and domain adaptation.
-
Distributed Stochastic Gradient Method for Non-Convex Problems with Applications in Supervised Learning
A consensus-based distributed stochastic gradient algorithm is shown to converge in mean square to a critical point for non-convex problems, and it trains distributed neural networks on MNIST with accuracy comparable ...
Discussion (0). Continue with ORCID to comment.