Pith. sign in

REVIEW 3 cited by

Cutting Some Slack for SGD with Adaptive Polyak Stepsizes

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2202.12328 v2 pith:G5CKUREE submitted 2022-02-24 cs.LG math.OC

Cutting Some Slack for SGD with Adaptive Polyak Stepsizes

Robert M. Gower , Mathieu Blondel , Nidham Gazagnadou , Fabian Pedregosa This is my paper
classification cs.LG math.OC
keywords methodssizestepgradientslackvariantsadaptiveloss
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Tuning the step size of stochastic gradient descent is tedious and error prone. This has motivated the development of methods that automatically adapt the step size using readily available information. In this paper, we consider the family of SPS (Stochastic gradient with a Polyak Stepsize) adaptive methods. These are methods that make use of gradient and loss value at the sampled points to adaptively adjust the step size. We first show that SPS and its recent variants can all be seen as extensions of the Passive-Aggressive methods applied to nonlinear problems. We use this insight to develop new variants of the SPS method that are better suited to nonlinear models. Our new variants are based on introducing a slack variable into the interpolation equations. This single slack variable tracks the loss function across iterations and is used in setting a stable step size. We provide extensive numerical results supporting our new methods and a convergence theory.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Safeguarded Stochastic Polyak Step Sizes for Non-smooth Optimization: Robust Performance Without Small (Sub)Gradients

    math.OC 2025-12 conditional novelty 7.0

    A safeguarded stochastic Polyak step size, SPS_safe, yields O(1/√T) convergence to a neighborhood for convex non-smooth problems without interpolation or oracle loss values, with a momentum variant.

  2. Muon Does Not Converge on Convex Lipschitz Functions

    cs.LG 2026-05 unverdicted novelty 6.0

    Muon does not converge on convex Lipschitz functions regardless of learning rate, while error feedback restores theoretical convergence but degrades performance on CIFAR-10 and nanoGPT tasks.

  3. Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Scheduler

    math.OC 2026-06 unverdicted novelty 5.0

    Proposes Polyak schedulers for SAM with convergence proofs in deterministic and stochastic settings and empirical results showing reduced tuning needs.