REVIEW 7 cited by
On the Convergence of SGD with Biased Gradients
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We analyze the complexity of biased stochastic gradient methods (SGD), where individual updates are corrupted by deterministic, i.e. biased error terms. We derive convergence results for smooth (non-convex) functions and give improved rates under the Polyak-Lojasiewicz condition. We quantify how the magnitude of the bias impacts the attainable accuracy and the convergence rates (sometimes leading to divergence). Our framework covers many applications where either only biased gradient updates are available, or preferred, over unbiased ones for performance reasons. For instance, in the domain of distributed learning, biased gradient compression techniques such as top-k compression have been proposed as a tool to alleviate the communication bottleneck and in derivative-free optimization, only biased gradient estimators can be queried. We discuss a few guiding examples that show the broad applicability of our analysis.
Forward citations
Cited by 7 Pith papers
-
A Gradient Flow Perspective on Minimum MMD Estimation
Preconditioned gradient descent for min-MMD estimation converges asymptotically to a global minimizer under gradient-dominance and projection-residual conditions, and empirically beats standard GD.
-
CaliMatch: Adaptive Calibration for Improving Safe Semi-supervised Learning
CaliMatch adaptively calibrates classifier and OOD detector confidences in safe semi-supervised learning, improving classification accuracy on CIFAR-10/100, SVHN, TinyImageNet and ImageNet.
-
Optimization over Sparse Support-Preserving Sets: Two-Step Projection with Global Optimality Guarantees
An iterative hard-thresholding variant with a two-step projection offers global objective-value guarantees for sparse optimization with support-preserving convex constraints, including the first zeroth-order hard-thre...
-
Discrete State Diffusion Models: A Sample Complexity Perspective
Claims the first Õ(ε⁻²) sample-complexity bound for discrete-state diffusion, but the zero-approximation-error, optimization-error, and hardness lemmas carrying the proof are internally broken.
-
Neighbor-Sampling Based Momentum Stochastic Methods for Training Graph Neural Networks
The paper creates Adam-style optimizers that combine neighbor sampling and control variates for graph neural networks, with optimal convergence rates and better node-classification performance than control-variate SGD.
-
Stacey: Promoting Stochastic Steepest Descent via Accelerated $\ell_p$-Smooth Nonconvex Optimization
STACEY is a new ℓ_p steepest descent optimizer with primal-dual interpolation; its convergence theory covers only the unaccelerated base algorithm, and its empirical gains rely on grid-searched hyperparameters.
-
Stochastic Optimization and Data Science
The paper motivates stochastic optimization problems from statistical perspectives and describes offline and online approaches to solve expectation minimization problems.
Discussion (0). Sign in to comment.