REVIEW 7 cited by
Methods for Convex (L₀,L₁)-Smooth Optimization: Clipping, Acceleration, and Adaptivity
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Methods for Convex (L₀,L₁)-Smooth Optimization: Clipping, Acceleration, and Adaptivity
read the original abstract
Due to the non-smoothness of optimization problems in Machine Learning, generalized smoothness assumptions have been gaining a lot of attention in recent years. One of the most popular assumptions of this type is $(L_0,L_1)$-smoothness (Zhang et al., 2020). In this paper, we focus on the class of (strongly) convex $(L_0,L_1)$-smooth functions and derive new convergence guarantees for several existing methods. In particular, we derive improved convergence rates for Gradient Descent with (Smoothed) Gradient Clipping and for Gradient Descent with Polyak Stepsizes. In contrast to the existing results, our rates do not rely on the standard smoothness assumption and do not suffer from the exponential dependency from the initial distance to the solution. We also extend these results to the stochastic case under the over-parameterization assumption, propose a new accelerated method for convex $(L_0,L_1)$-smooth optimization, and derive new convergence rates for Adaptive Gradient Descent (Malitsky and Mishchenko, 2020).
Forward citations
Cited by 7 Pith papers
-
Stochastic Non-Smooth Convex Optimization with Unbounded Gradients
Introduces generalized Lipschitz class and shows clipped AdamW outperforms SGD and AdaGrad for stochastic convex optimization under this and related assumptions.
-
Stochastic Non-Smooth Convex Optimization with Unbounded Gradients
Clipped AdamW with exponentially weighted accumulation achieves superior global convergence rates for convex stochastic generalized Lipschitz optimization compared to SGD and AdaGrad.
-
Normalized First-Order Methods for Convex (L0, L1)-Smooth Optimization with Inexact Gradients
Comparison-oracle variants of NGD and Polyak GD converge for convex (L0, L1)-smooth objectives when the normalized-gradient error δ is bounded by explicit O(√ε)-scale thresholds.
-
Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Rates
Non-Euclidean SGD variants (SignSGD, Muon) provably match adaptive optimizers' convergence rates under structured smoothness and noise assumptions.
-
Frank-Wolfe Algorithms for (L0, L1)-smooth functions
Proposes (L0, L1)-Frank-Wolfe and adaptive variant claiming superior convergence rates for (L0, L1)-smooth objectives over classical Frank-Wolfe.
-
Frank-Wolfe Algorithms for (L0, L1)-smooth functions
A new (L0, L1)-Frank-Wolfe algorithm and its adaptive version are proposed for (L0, L1)-smooth optimization, with claims of better theoretical convergence rates and practical advantages over standard Frank-Wolfe methods.
-
Why Do We Need Warm-up? A Theoretical Perspective
Under the proposed (H0,H1)-smoothness condition, gradient descent with a warm-up-style adaptive step-size provably converges faster than with any fixed step-size.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.