pith. sign in

arxiv: 2606.01827 · v1 · pith:34UYGA4Snew · submitted 2026-06-01 · 🧮 math.OC · cs.LG· stat.ML

Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Scheduler

Dimitris Oikonomou , Nicolas Loizou This is my paper

Pith reviewed 2026-06-28 13:36 UTC · model grok-4.3

classification 🧮 math.OC cs.LGstat.ML
keywords sharpness-aware minimizationPolyak step sizeadaptive optimizationconvergence analysisstochastic gradient methodsmachine learning optimizers
0
0 comments X

The pith

Polyak schedulers adapted to SAM updates create learning-rate-free optimizers with proven linear convergence on smooth strongly convex problems.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper derives step-size rules based on the Polyak formula but applied directly to the sharpness-aware loss used inside SAM. These rules produce fully adaptive algorithms that require no separate learning-rate schedule. In the deterministic smooth setting the method converges linearly on strongly convex objectives and at an O(1/T) rate on convex objectives. The stochastic versions achieve the same rates up to a neighborhood whose size depends on gradient noise. Experiments show the new schedulers match or beat hand-tuned SAM while removing most learning-rate search.

Core claim

By tailoring the Polyak step-size formula to the SAM update, the resulting deterministic algorithms achieve linear convergence for smooth strongly convex objectives and an O(1/T) rate for convex objectives; the stochastic counterparts converge to a neighborhood of the optimum at analogous rates.

What carries the argument

The Polyak-type scheduler for SAM, which sets the step length at each iteration using the current sharpness-aware function value and gradient norm instead of a fixed learning rate.

If this is right

  • Linear convergence holds for smooth strongly convex deterministic problems without any learning-rate parameter.
  • O(1/T) sublinear rate holds for smooth convex deterministic problems.
  • Stochastic versions converge to a noise-dependent neighborhood at the same rates.
  • Empirical performance reaches or exceeds that of carefully tuned fixed-rate SAM on standard tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same Polyak construction could be applied to other sharpness-aware or gradient-penalty methods beyond the original SAM.
  • Removing the learning-rate hyperparameter may simplify large-scale distributed training pipelines where manual tuning is costly.
  • The neighborhood size in the stochastic analysis suggests a natural trade-off between step-size adaptivity and final accuracy that could be quantified further.

Load-bearing premise

The loss must be smooth and either strongly convex or convex, with bounded variance in the stochastic case.

What would settle it

On a smooth strongly convex quadratic, the distance to the minimizer after T steps fails to decrease at a linear rate.

Figures

Figures reproduced from arXiv: 2606.01827 by Dimitris Oikonomou, Nicolas Loizou.

Figure 1
Figure 1. Figure 1: Synthetic Ridge Regression. Deterministic regime. We compare USAM with Polyak Scheduler against representative constant step size USAM baselines with deterministic convergence guarantees. Specif￾ically, Khanh et al. (2024) assume ρ ∈ [0, 1/L) and γ ∈ [0, 4/(9L)), Andriushchenko & Flammarion (2022) assume ρ ∈ [0, 1/L) and γ ∈ [0, 1/L), and Oikonomou & Loizou (2025b) provide a linear convergence rate for ρ ∈… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison with other adaptive SAM optimizers. The results are shown in Section 4.1. In the determinis￾tic regime, our scheduler converges to the optimum in the fewest iterations; the AdaGrad-based LightSAM variants reach the same accuracy but require noticeably more itera￾tions, while the remaining baselines stall further from the optimum. In the stochastic regime, our scheduler is competi￾tive with the s… view at source ↗
read the original abstract

Sharpness-Aware Minimization (SAM) has established itself as a powerful and widely adopted optimizer for training machine learning models. By explicitly minimizing the sharpness of the loss landscape, SAM often improves generalization while delivering strong empirical performance. However, SAM and its variants, like most training algorithms, are sensitive to the choice of learning rate, which is typically selected through extensive hyperparameter tuning or predefined schedulers. In this work, motivated by recent advances on the effectiveness of stochastic Polyak step sizes for Stochastic Gradient Descent (SGD), we derive Polyak schedulers tailored to SAM-style updates, yielding novel adaptive algorithms in both deterministic and stochastic settings. In the smooth setting, we prove linear convergence for strongly convex objectives and an $\mathcal{O}(1/T)$ convergence rate for convex objectives in the deterministic case. In the stochastic setting, we establish analogous convergence guarantees up to a neighborhood of the optimum. Numerical experiments demonstrate that the proposed Polyak schedulers achieve performance comparable to or better than carefully tuned SAM baselines, while substantially reducing the need for learning-rate tuning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 3 minor

Summary. The manuscript derives Polyak-type step-size schedulers tailored to SAM-style updates, yielding adaptive algorithms in deterministic and stochastic regimes. Under standard smoothness assumptions, it proves linear convergence for strongly convex objectives and an O(1/T) rate for convex objectives in the deterministic case, with analogous neighborhood guarantees in the stochastic case. Experiments indicate that the proposed schedulers match or exceed the performance of carefully tuned SAM baselines while substantially reducing the need for learning-rate selection.

Significance. If the stated convergence results hold under the listed assumptions, the work supplies a theoretically grounded adaptive mechanism for SAM, an optimizer already in wide use. The explicit reduction in hyperparameter tuning burden, backed by both deterministic/stochastic rates and numerical comparisons, represents a concrete practical contribution to first-order methods.

minor comments (3)
  1. [§3] §3 (deterministic analysis): the statement of the Polyak step-size formula should explicitly reference the SAM perturbation radius ho to avoid ambiguity when readers compare to the original SAM update.
  2. [Table 1, §5] Table 1 and §5 (experiments): the reported test accuracies lack standard deviations across the listed random seeds; adding error bars or multiple-run statistics would strengthen the claim of 'comparable or better' performance.
  3. [Theorem 4.2] The stochastic convergence theorem (Theorem 4.2) invokes a bounded-variance assumption; a brief remark on whether this is verified or relaxed in the experiments would improve transparency.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive assessment of our work on Polyak-type schedulers for SAM, including the recognition of the convergence guarantees in deterministic and stochastic settings as well as the practical reduction in hyperparameter tuning. The report recommends minor revision but lists no specific major comments or requested changes.

Circularity Check

0 steps flagged

No significant circularity; derivation is self-contained

full rationale

The paper derives Polyak-type schedulers for SAM updates and proves standard convergence rates (linear for strongly convex, O(1/T) for convex) under explicit smoothness/convexity assumptions in deterministic and stochastic settings. No load-bearing self-citations, self-definitional steps, or fitted inputs renamed as predictions appear in the abstract or description; the central claims rest on independent analysis of the SAM update rule combined with Polyak step-size logic from prior SGD literature. The stochastic neighborhood result follows directly from variance bounds without circular reduction. This is the expected honest non-finding for a theory paper whose proofs are externally falsifiable.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no specific free parameters, axioms, or invented entities are detailed in the provided text.

pith-pipeline@v0.9.1-grok · 5725 in / 1100 out tokens · 33180 ms · 2026-06-28T13:36:06.335749+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 12 canonical work pages · 4 internal anchors

  1. [1]

    Lightsam: Parameter-agnostic sharpness-aware mini- mization.arXiv preprint arXiv:2505.24399,

    Cheng, Y ., Shen, L., Sun, H., Yin, N., Cao, X., and Chen, E. Lightsam: Parameter-agnostic sharpness-aware mini- mization.arXiv preprint arXiv:2505.24399,

    work page arXiv

  2. [2]

    R., and Usunier, N

    D´efossez, A., Bottou, L., Bach, F. R., and Usunier, N. A sim- ple convergence proof of adam and adagrad.Transactions on Machine Learning Research, 2022,

    2022

  3. [3]

    Improved Regularization of Convolutional Neural Networks with Cutout

    DeVries, T. Improved regularization of convolutional neural networks with cutout.arXiv preprint arXiv:1708.04552,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    M., and Schaipp, F

    Garrigos, G., Gower, R. M., and Schaipp, F. Function value learning: Adaptive learning rates based on the polyak stepsize and function splitting in erm.arXiv preprint arXiv:2307.14528,

    work page arXiv

  5. [5]

    M., Blondel, M., Gazagnadou, N., and Pedregosa, F

    Gower, R. M., Blondel, M., Gazagnadou, N., and Pedregosa, F. Cutting some slack for SGD with adaptive polyak stepsizes.arXiv preprint arXiv:2202.12328,

    work page arXiv

  6. [6]

    M., Garrigos, G., Loizou, N., Oikonomou, D., Mishchenko, K., and Schaipp, F

    Gower, R. M., Garrigos, G., Loizou, N., Oikonomou, D., Mishchenko, K., and Schaipp, F. Analysis of an idealized stochastic polyak method and its application to black- box model distillation.arXiv preprint arXiv:2504.01898,

    work page arXiv

  7. [7]

    arXiv preprint arXiv:1905.00313 , title =

    Hazan, E. and Kakade, S. Revisiting the polyak step size. arXiv preprint arXiv:1905.00313,

    work page arXiv 1905

  8. [8]

    D., Luong, H.-C., Mordukhovich, B

    Khanh, P. D., Luong, H.-C., Mordukhovich, B. S., and Tran, D. B. Fundamental convergence analysis of sharpness- aware minimization.arXiv preprint arXiv:2401.08060,

    work page arXiv

  9. [9]

    Prodigy: An expeditiously adaptive parameter-free learner.arXiv preprint arXiv:2306.06101, 2023

    Mishchenko, K. and Defazio, A. Prodigy: An expedi- tiously adaptive parameter-free learner.arXiv preprint arXiv:2306.06101,

    work page arXiv

  10. [10]

    Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks

    Neyshabur, B., Li, Z., Bhojanapalli, S., LeCun, Y ., and Srebro, N. Towards understanding the role of over- parametrization in generalization of neural networks. arXiv preprint arXiv:1805.12076,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Safeguarded Stochastic Polyak Step Sizes for Non-smooth Optimization: Robust Performance Without Small (Sub)Gradients

    Oikonomou, D. and Loizou, N. Safeguarded stochastic polyak step sizes for non-smooth optimization: Robust performance without small (sub) gradients.arXiv preprint arXiv:2512.02342, 2025a. Oikonomou, D. and Loizou, N. Sharpness-aware mini- mization: General analysis and improved rates. InICLR, 2025b. Oikonomou, D. and Loizou, N. Stochastic polyak step-size...

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    P., Mobahi, H., Agarwala, A., and Dauphin, Y

    Singh, S. P., Mobahi, H., Agarwala, A., and Dauphin, Y . Avoiding spurious sharpness minimization broadens ap- plicability of sam.arXiv preprint arXiv:2502.02407,

    work page arXiv

  13. [13]

    Understanding deep learning requires rethinking generalization

    Xie, W., Latorre, F., Antonakopoulos, K., Pethick, T., and Cevher, V . Improving sam requires rethinking its opti- mization formulation. InICML, 2024a. Xie, W., Pethick, T., and Cevher, V . Sampa: Sharpness- aware minimization parallelized. InNeurIPS, 2024b. Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires ...

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    Function Value Learning

    12 Adaptive Sharpness-Aware Minimization with a Polyak-type Step size Supplementary Material The Supplementary Material is organized as follows. Section A reviews additional related work on adaptive step-size methods. In Section B, we collect basic definitions and the auxiliary lemmas used throughout. Section C contains the proofs of the basic properties ...

    2022

  15. [15]

    and Adam/AdamW (Kingma & Ba, 2015; Loshchilov & Hutter,

    2015

  16. [16]

    replace this sum with exponentially weighted moving averages of the first and second gradient moments, and have become the de facto choice for training deep neural networks. The theoretical analysis of these methods has been refined over time, including corrections to the original Adam analysis and tighter conditions under which they match or fall short o...

    2018

  17. [17]

    13 Adaptive Sharpness-Aware Minimization with a Polyak-type Step size A more recent and conceptually distinct direction isparameter-freeorlearning-rate-freeoptimization

    combines stochastic recursive gradient estimators with adaptive and implicit step-size choices, reducing sensitivity to manually tuned learning-rate schedules while preserving the benefits of SARAH-type variance reduction. 13 Adaptive Sharpness-Aware Minimization with a Polyak-type Step size A more recent and conceptually distinct direction isparameter-fr...

    2023