REVIEW 4 cited by
On exponential convergence of SGD in non-convex over-parametrized learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
On exponential convergence of SGD in non-convex over-parametrized learning
read the original abstract
Large over-parametrized models learned via stochastic gradient descent (SGD) methods have become a key element in modern machine learning. Although SGD methods are very effective in practice, most theoretical analyses of SGD suggest slower convergence than what is empirically observed. In our recent work [8] we analyzed how interpolation, common in modern over-parametrized learning, results in exponential convergence of SGD with constant step size for convex loss functions. In this note, we extend those results to a much broader non-convex function class satisfying the Polyak-Lojasiewicz (PL) condition. A number of important non-convex problems in machine learning, including some classes of neural networks, have been recently shown to satisfy the PL condition. We argue that the PL condition provides a relevant and attractive setting for many machine learning problems, particularly in the over-parametrized regime.
Forward citations
Cited by 4 Pith papers
-
Sharp First-Order Lower Bounds under Sublevel $\alpha$-Polyak-Lojasiewicz Conditions
Proves minimax lower bounds for first-order methods under sublevel alpha-PL conditions that match gradient descent and SGD upper bounds after showing global alpha-PL plus smoothness forces constant functions.
-
Sharp First-Order Lower Bounds under Sublevel $\alpha$-Polyak-Lojasiewicz Conditions
On the globally smooth sublevel-α-PŁ class, deterministic and bounded-variance first-order methods have matching minimax lower bounds of the form of GD and SGD rates.
-
Structure Before Collapse: Transient semantic geometry in next-token prediction
Semantic geometry emerges transiently early in next-token prediction training before collapsing to Neural Collapse symmetry in synthetic settings with latent semantic factors.
-
A Generalized Energy-Based Adaptive Gradient Method for Optimization
A generalized energy-based adaptive gradient method achieves unconditional energy stability and optimal O(1/ε) stationary-point convergence for any smooth concave energy function; the log-energy variant ALEGD converge...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.