Pith. sign in

REVIEW 3 major objections 4 minor 57 references

LSAM claims that smoothing SAM's loss landscape into a kernel-smoothed Boltzmann density and sampling it asynchronously lets distributed sharpness-aware optimization match SGD's convergence rate while beating data-parallel SAM empirically.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

LSAM combines SAM's sharpness-aware objective with an EASGD-style asynchronous sampling scheme and claims SGD-rate convergence plus better accuracy than data-parallel SAM.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Solid score identity and a coherent ESGD analysis, but the O(log T/√T) convergence result is proved for Algorithm 1, not the asynchronous SGLD algorithm that actually defines LSAM. the 3 major comments →

arxiv 2509.03110 v1 pith:SRGZ5SWA submitted 2025-09-03 cs.LG stat.ML

LSAM: Asynchronous Distributed Training with Landscape-Smoothed Sharpness-Aware Minimization

Yunfei Teng , Sixin Zhang This is my paper
classification cs.LG stat.ML MSC 68T0790C2668W15
keywords sharpness-aware minimizationasynchronous distributed trainingstochastic gradient Langevin dynamicslandscape smoothinglarge-batch trainingnonconvex optimizationconvergence rateflat minima
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LSAM's central claim is that sharpness-aware minimization can be made distributed-friendly by treating it as sampling. The paper reformulates SAM's worst-case loss as a Boltzmann density, convolves that density with a kernel to obtain a smoothed target, and then runs asynchronous Langevin sampling on each worker against a shared anchor. The paper proves that the analyzed update converges at the same O(log T / sqrt(T)) rate as SGD when the perturbation radius decays, and only to an O(rho^2) neighborhood when the radius is constant. Empirically, on SVHN, CIFAR-10, and CIFAR-100 with several CNN, ResNet, VGG, and WideResNet architectures, LSAM reports lower final test error and faster convergence than data-parallel SGD, data-parallel SAM, EASGD, and LSGD. If the claim holds, SAM can scale by adding sampling workers without inflating the global batch size, which is the trade-off that currently limits SAM in large-batch distributed training.

Core claim

The paper's core claim is an equivalence between SAM and a smoothed sampling target. The SAM perturbed loss f(T_{rho,gamma}(x)) induces a Boltzmann density pi_SAM(x) proportional to exp(-f_SAM(x)); convolving it with a kernel k gives pi_LSAM(y), whose score is the negative posterior mean of the kernel gradient under q(x|y). This identity turns SAM's adversarial step into a sampling problem, so flat-minima pressure and Gaussian smoothing's deep-minima pressure act together. The update alternates x_{t+1}=x_t - eta_t(g_t + lambda(x_t - y_t)) with y_{t+1}=alpha x_{t+1} + (1-alpha)y_t, coupling a fast track x to a slowly moving anchor y. Theorems 2-4 bound the average squared norm of the coupled

What carries the argument

The load-bearing mechanism is the score identity for the kernel-smoothed SAM Boltzmann density: the gradient of log pi_LSAM(y) equals the negative posterior mean of the kernel gradient under the conditional q(x|y), with q proportional to exp(-f_SAM(x) - k(x,y)). This identity converts optimization of a sharpness-aware nonconvex loss into sampling from a conditional distribution: workers generate Langevin samples along this score, and the global anchor y is updated from their aggregated displacements every n*tau iterations. The coupling term lambda(x_t - y_t) and mixing parameter alpha in the alternating update are what the convergence proofs exploit, via the Lyapunov function f(x_t) + (lambd

Load-bearing premise

The load-bearing premise is that the asynchronous sampler in Algorithm 2 produces an unbiased, bounded-variance stochastic gradient of the LSAM objective; the paper's SGD-rate proofs cover only Algorithm 1's explicit gradient, so the distributed algorithm's advertised convergence rate stands or falls on that unproved equivalence.

What would settle it

On a low-dimensional nonconvex target where the LSAM score can be computed exactly, run Algorithm 2's sampler and measure the bias and variance of the aggregated direction. If the bias does not go to zero as the number of iterations and tau grow, or the variance does not shrink with more workers, the claimed SGD-rate convergence for the distributed algorithm fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • SAM can be trained at scale without enlarging the global batch: workers keep their own batch size and communicate only sample-difference summaries every n*tau iterations.
  • With a decaying perturbation radius, the analyzed LSAM update converges at the same asymptotic rate as SGD, so the extra smoothing does not slow the rate.
  • A constant perturbation radius buys only an O(rho^2) neighborhood, making the radius schedule the key hyperparameter for exact convergence.
  • The smoothed target is claimed to combine SAM's wide minima with entropy smoothing's deep minima, so the resulting solutions should be both flat and low-loss.
  • LSAM keeps SAM's two-gradient communication volume but removes synchronization barriers, so its wall-clock advantage should grow with the number of workers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the asynchronous sampler's aggregated differences can be shown to be an unbiased, bounded-variance estimator of the LSAM score, the gap between Algorithm 1 and Algorithm 2 closes and the SGD-rate conclusion would then cover the distributed method as stated.
  • The theoretical claim makes a testable scaling prediction: LSAM's advantage over data-parallel SAM should widen as the number of workers grows, because per-worker batch stays fixed while data-parallel SAM must inflate the global batch; the reported experiments use only four workers.
  • Because Theorem 1 holds for general kernels, replacing the Gaussian with a heavier-tailed or Matérn kernel would change the smoothing geometry while preserving the score identity; the paper does not explore that direction.
  • The momentum-augmented surrogate direction is Adan-style look-ahead, so combining LSAM's sampling loop with adaptive optimizers is a natural, untested extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes LSAM, an asynchronous distributed variant of Sharpness-Aware Minimization (SAM). LSAM defines a kernel-smoothed target distribution π_LSAM built from SAM's perturbed objective, derives a score identity (Theorem 1), and claims that a distributed algorithm based on SGLD workers matches SGD's O(log T / sqrt(T)) convergence rate. The distributed algorithm (Algorithm 2) runs local SGLD samplers toward a conditional q(x|y), aggregates the sampled displacements every nτ steps, and applies a momentum-accelerated outer update. Experiments on SVHN, CIFAR-10, and CIFAR-100 compare LSAM with DP-SGD, DP-SAM, EASGD, and LSGD in terms of lowest test error and error-versus-epoch curves.

Significance. If the central claims were fully established, the sampling-parallel framework would be a notable contribution to scaling sharpness-aware training. The paper does contain some sound building blocks: Theorem 1 is an exact score identity for general kernels and extends prior Gaussian-kernel derivations, and the convergence proofs for the centralized ESGD/SAM recursion in Theorems 2–4 appear mathematically coherent. However, the manuscript's headline claim—that the proposed distributed algorithm LSAM converges at the SGD rate—is not proven, because the convergence theorems analyze Algorithm 1, not Algorithm 2. The efficiency claims are also not supported by any wall-clock or communication-volume measurement. The paper is therefore not publishable in its current form.

major comments (3)
  1. [§3.2 and §4.3 (Algorithm 1 vs Algorithm 2)] Theorems 2–4 and Corollary 6 analyze Algorithm 1, the recursion in Eqs. (6)–(7), under Assumption 3, where g_t is an unbiased stochastic gradient of f. Algorithm 2 is structurally different: workers run SGLD with score s = ∇_x log q(x|y) (line 6), then form g'_t = (1/nτ)∑_{i,s}(x_s − y) (line 13), add Nesterov momentum (line 14), and update y only every nτ steps. No result in the paper states or proves that this aggregate is an unbiased, bounded-variance estimator of ∇_y log π_LSAM(y_t) (up to a constant factor) for a finite-horizon SGLD trajectory with a slowly moving y. The proof gap is load-bearing: the claimed O(log T/√T) convergence rate for LSAM is not established for the algorithm actually proposed. The mismatch also includes the momentum term and the constant learning rates in Algorithm 2, neither of which appears in the analyzed recursion.
  2. [§3.2 Theorem 4 vs §4 and Appendix D] The only result giving convergence to zero at the SGD rate is Theorem 4, which assumes a decaying perturbation radius ρ_t = ρ0/√(t+1). Algorithm 2 and all experiments use a constant radius ρ = 0.1 (Algorithm 2 line 1; Appendix D.2). For constant ρ, Theorem 3 yields only E‖G_t‖² ≤ 4L²ρ² + O(log T/√T), i.e., a neighborhood of size O(ρ²), not convergence to a stationary point. Thus the statement in Contribution 3 that 'LSAM matches SGD convergence rates' is not supported for the implemented configuration.
  3. [Abstract, Conclusion, Remark 1, §5] The efficiency claims are not demonstrated. Remark 1 explicitly states that LSAM requires double the communication cost of SAM. Section 5 reports only test error versus training epoch; there is no wall-clock time, communication volume, or throughput measurement. The abstract's claim that LSAM 'eliminates synchronization bottlenecks' is also overstated, since Algorithm 2 still performs global synchronization every nτ iterations (line 12). The empirical results in Tables 1–2 do not compensate for this omission because they show only lowest test error, with no standard deviations, number of seeds, or wall-clock comparison.
minor comments (4)
  1. [Proof of Theorem 3] The proof contains stray superscript '2' symbols in the displayed inequalities after Eq. (21) ('the variance term' and subsequent display); the intended quantity is clear but should be cleaned up.
  2. [Proof of Proposition 7] The proof uses the symbol ε in 'Since ε > 0' and 'T_{ρ,ε}', while the statement and Assumption 1 use γ. This notation inconsistency should be fixed.
  3. [§4.2 and Algorithm 2] Equation (8) defines g'_t as ∇_y log π_LSAM(y_t) = −E_q[∇_y k(x, y_t)], while Algorithm 2 line 13 computes an average of (x_s − y). These are proportional only for a Gaussian kernel with a specific scaling. The scaling factor λ should be made explicit in the algorithm and in the text connecting Eq. (8) to line 13.
  4. [§4.3, Algorithm 2 line 12] The synchronization condition 'if ∑_i t^(i)_x mod nτ = 0' is ambiguous: t^(i)_x are local counters since the last reset, and ∑_i t^(i)_x will generally reach nτ only if workers advance in lockstep. The intended condition should be stated more precisely, e.g., 'when the total number of local steps across all workers since the last synchronization equals nτ'.

Circularity Check

0 steps flagged

No significant circularity: the score identity is an exact calculus identity; the convergence rates are self-contained analyses of Algorithm 1; self-citations are baselines, not load-bearing.

full rationale

The paper's central theoretical result, Theorem 1, is a direct differentiation-under-the-integral identity for the score of the convolved Gibbs density; it contains no fitted parameters and does not presuppose the convergence claims. The convergence proofs (Theorems 2–4, Lemma 5, Corollary 6) analyze Algorithm 1—an ESGD/SAM-type recursion with an unbiased stochastic-gradient oracle under Assumption 3—and are self-contained derivations of O(log T/sqrt(T)) rates for G_t and, via Corollary 6, for the gradient norm. These results do not rely on LSAM's own empirical outputs or on fitting any constants. The self-citations to Teng et al. (2019), Zhang et al. (2015), and Teng et al. (2022) are used as baselines and distributed-system references, not as justification for the convergence theorem. The main weakness is a correctness gap rather than circularity: Algorithm 2's asynchronous SGLD aggregation is not shown to supply the unbiased, bounded-variance gradient oracle that Theorems 2–4 require, so the rate claim is not formally connected to the proposed distributed algorithm. That is an unproven step, not a circular one, because Algorithm 2's gradient is defined as a Monte Carlo estimate of the score, not as the theorem's conclusion. No parameter is fitted and then renamed a prediction, and no uniqueness theorem is imported from the authors' prior work. Remark 1's admission of double communication cost weakens the efficiency claim but is not a circular dependency. The derivation chain is therefore free of self-definitional or fitted-input circularity, though the Algorithm 1-to-Algorithm 2 transfer remains an open theoretical issue.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central theory rests on standard stochastic approximation assumptions plus an unverified transfer from Algorithm 1 to Algorithm 2. Hyperparameters are chosen by grid search on the test sets, and several are central to the algorithm but not theoretically derived.

free parameters (6)
  • rho (SAM perturbation radius) = 0.1 (after grid search over 0.01, 0.05, 0.1)
    Controls the neighborhood used in the sharpness objective; appears directly in the Theorem 3 bias term O(rho^2); no theoretical selection rule is given.
  • lambda (pulling coefficient) via lambda0 = lambda0 selected from {0.1, 0.2, 0.5, 0.9}; schedule lambda = lambda0/(eta*tau)
    Couples each worker x to the global anchor y; central to the algorithm and to Lemma 5's bounded anchor gap.
  • initial learning rate eta = 0.02 for CNNs, 0.2 for ResNet/VGG/WRN
    Grid searched over {0.01, 0.02, 0.05, 0.1, 0.2, 0.3}; architecture-specific.
  • sync period tau = 16 for LSAM/LSGD/EASGD
    Controls how many inner sampling steps run before the synchronous outer update; not swept in the reported experiments.
  • momentum beta = 0.9, fixed
    Adan-style look-ahead term in the outer update.
  • optimization learning rate eta' = 1.0, fixed
    Constant learning rate for the outer optimization loop.
axioms (5)
  • domain assumption Assumption 3 (C1)-(C4): bounded variance, L-smoothness, stochastic L-smoothness, bounded expected gradient norm
    Standard oracle conditions used in Theorems 2 to 4 and Lemma 5; not verified for deep networks in the experiments.
  • domain assumption Assumption 1: f continuously differentiable, Z0 finite, and either convex or L-Lipschitz, with tail growth conditions
    Needed for the normalized Boltzmann density pi_SAM; may not hold for nonconvex neural network losses.
  • domain assumption Gaussian kernel smoothing is sufficient to realize the LSAM objective
    The implementation fixes a Gaussian kernel per prior work, but the score identity is for general kernels and no experiments compare kernel choices.
  • ad hoc to paper The asynchronous SGLD sampler in Algorithm 2 provides unbiased, bounded-variance gradient estimates of the outer score
    This bridge from Algorithm 1 to Algorithm 2 is never proven; it is the central derivation gap.
  • domain assumption Batch-norm statistics can be left unsynchronized without performance degradation
    Appendix C.2 asserts empirically that no degradation occurs, but provides no analysis and no error bars.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of LSAM: Asynchronous Distributed Training with Landscape-Smoothed Sharpness-Aware Minimization." pith.science (2026). https://pith.science/paper/SRGZ5SWA

@misc{pith2026250903110,
  author       = {Pith},
  title        = {Pith review of: LSAM: Asynchronous Distributed Training with Landscape-Smoothed Sharpness-Aware Minimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SRGZ5SWA}},
  note         = {Machine review of arXiv:2509.03110}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

While Sharpness-Aware Minimization (SAM) improves generalization in deep neural networks by minimizing both loss and sharpness, it suffers from inefficiency in distributed large-batch training. We present Landscape-Smoothed SAM (LSAM), a novel optimizer that preserves SAM's generalization advantages while offering superior efficiency. LSAM integrates SAM's adversarial steps with an asynchronous distributed sampling strategy, generating an asynchronous distributed sampling scheme, producing a smoothed sharpness-aware loss landscape for optimization. This design eliminates synchronization bottlenecks, accelerates large-batch convergence, and delivers higher final accuracy compared to data-parallel SAM.

Figures

Figures reproduced from arXiv: 2509.03110 by Sixin Zhang, Yunfei Teng.

Figure 1
Figure 1. Figure 1: Test error versus training epoch on SVHN. The zoomed plots are in [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Test error versus training epoch on CIFAR-10 and CIFAR [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Probability landscapes associated with π, πSAM, πESGD, and πLSAM. Among the three convergence modes—deep but sharp, wide but shallow, and both wide and deep—ESGD may converge to the first, SAM to the sec￾ond, whereas LSAM distinctively achieves the third. Motivation As shown in [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The global server establishes connections with each local server, while local servers [PITH_FULL_IMAGE:figures/full_fig_p019_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Test error versus train epoch on SVHN dataset. [PITH_FULL_IMAGE:figures/full_fig_p020_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Test error versus train epoch on CIFAR-10 and CIFAR-100 datasets. [PITH_FULL_IMAGE:figures/full_fig_p021_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 47 canonical work pages · 5 internal anchors

  1. [1]

    Towards understanding sharpness-aware minimization

    Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware minimization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 639--668. PMLR, 17--23...

    work page 2022

  2. [2]

    Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer

    Maksym Andriushchenko and Nicolas Flammarion. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. JMLR, 2023

    work page 2023

  3. [3]

    A fast iterative shrinkage-thresholding algorithm for linear inverse problems

    Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2 0 (1): 0 183--202, 2009

    work page 2009

  4. [4]

    mSAM: Micro-Batch-Averaged Sharpness-Aware Minimization

    Kayhan Behdin, Qingquan Song, Aman Gupta, Sathiya Keerthi, Ayan Acharya, Borja Ocejo, Gregory Dexter, Rajiv Khanna, David Durfee, and Rahul Mazumder. msam: Micro-batch-averaged sharpness-aware minimization, 2023. URL https://arxiv.org/abs/2302.09693

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  5. [5]

    L. Bottou. Online algorithms and stochastic approximations. In Online Learning and Neural Networks. Cambridge University Press, 1998

    work page 1998

  6. [6]

    Beyond local sharpness: Communication-efficient global sharpness-aware minimization for federated learning

    Debora Caldarola, Pietro Cagnasso, Barbara Caputo, and Marco Ciccone. Beyond local sharpness: Communication-efficient global sharpness-aware minimization for federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 25187--25197, June 2025

    work page 2025

  7. [7]

    Entropy- SGD : Biasing gradient descent into wide valleys

    Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy- SGD : Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838, 2016

    Pith/arXiv arXiv 2016

  8. [8]

    Parle: parallelizing stochastic gradient descent

    Pratik Chaudhari, Carlo Baldassi, Riccardo Zecchina, Stefano Soatto, Ameet Talwalkar, and Adam Oberman. Parle: parallelizing stochastic gradient descent. arXiv preprint arXiv:1707.00424, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  9. [9]

    Diffusive G ibbs sampling

    Wenlin Chen, Mingtian Zhang, Brooks Paige, Jos\' e Miguel Hern\' a ndez-Lobato, and David Barber. Diffusive G ibbs sampling. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings o...

    work page 2024

  10. [10]

    Improved analysis for a proximal algorithm for sampling

    Yongxin Chen, Sinho Chewi, Adil Salim, and Andre Wibisono. Improved analysis for a proximal algorithm for sampling. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pages 2984--3014. PMLR, 02--05 Jul 2022. URL https://proceedings.mlr.press/v178/ch...

    work page 2022

  11. [11]

    Convergence rate in a nonlinear two-time-scale stochastic approximation with state (time)-dependence

    Zixi Chen, Yumin Xu, and Ruixun Zhang. Convergence rate in a nonlinear two-time-scale stochastic approximation with state (time)-dependence. Proceedings of the AAAI Conference on Artificial Intelligence, 39 0 (15): 0 15993--16000, Apr. 2025. doi:10.1609/aaai.v39i15.33756. URL https://ojs.aaai.org/index.php/AAAI/article/view/33756

    work page doi:10.1609/aaai.v39i15.33756 2025

  12. [12]

    An iterative thresholding algorithm for linear inverse problems with a sparsity constraint

    Ingrid Daubechies, Michel Defrise, and Christine De Mol. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics, 57 0 (11): 0 1413--1457, 2004

    work page 2004

  13. [13]

    Grawa: Gradient-based weighted averaging for distributed training of deep learning models

    Tolga Dimlioglu and Anna Choromanska. Grawa: Gradient-based weighted averaging for distributed training of deep learning models. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 2251--2259. P...

    work page 2024

  14. [14]

    Thinh T. Doan. Nonlinear two-time-scale stochastic approximation: Convergence and finite-time performance. IEEE Transactions on Automatic Control, 68 0 (8): 0 4695--4705, 2023. doi:10.1109/TAC.2022.3210147

    work page arXiv 2023

  15. [15]

    Efficient sharpness-aware minimization for improved training of neural networks

    Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent Tan. Efficient sharpness-aware minimization for improved training of neural networks. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=n0OeTdNRG0Q

    work page 2022

  16. [16]

    Locally estimated global perturbations are better than local perturbations for federated sharpness-aware minimization

    Ziqing Fan, Shengchao Hu, Jiangchao Yao, Gang Niu, Ya Zhang, Masashi Sugiyama, and Yanfeng Wang. Locally estimated global perturbations are better than local perturbations for federated sharpness-aware minimization. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    work page 2024

  17. [17]

    Gerald B. Folland. Real Analysis: Modern Techniques and Their Applications. John Wiley & Sons, New York, NY, 2 edition, 1999. ISBN 978-0-471-25032-7

    work page 1999

  18. [18]

    Sharpness-aware minimization for efficiently improving generalization, 2020

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization, 2020. URL http://arxiv.org/abs/2010.01412. cite arxiv:2010.01412

    Pith/arXiv arXiv 2020

  19. [19]

    Willard Gibbs

    J. Willard Gibbs. Elementary Principles in Statistical Mechanics. Charles Scribner's Sons, 1902

    work page 1902

  20. [20]

    Ulf Grenander and Michael I. Miller. Representations of Knowledge in Complex Systems . Journal of the Royal Statistical Society. Series B (Methodological), 56 0 (4): 0 549--603, 1994. ISSN 00359246

    work page 1994

  21. [21]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770--778, 2016

    2016

  22. [22]

    Flat minima

    Sepp Hochreiter and J \"u rgen Schmidhuber. Flat minima. Neural Computation, 9 0 (1): 0 173--186, 1997

    work page 1997

  23. [23]

    Reverse diffusion monte carlo

    Xunpeng Huang, Hanze Dong, Yifan HAO, Yian Ma, and Tong Zhang. Reverse diffusion monte carlo. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=kIPEyMSdFV

    work page 2024

  24. [24]

    Asynchronous Sharpness-Aware Minimization For Fast and Accurate Deep Learning

    Junhyuk Jo, Jihyun Lim, and Sunwoo Lee. Asynchronous sharpness-aware minimization for fast and accurate deep learning, 2025. URL https://arxiv.org/abs/2503.11147

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  25. [25]

    On large-batch training for deep learning: Generalization gap and sharp minima

    Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In ICLR, 2017

    2017

  26. [26]

    Smooth minima: A convex relaxation framework for optimizing flatness

    Dongjun Kim, Jaeho Park, and Jinwoo Shin. Smooth minima: A convex relaxation framework for optimizing flatness. AISTATS, 2023

    work page 2023

  27. [27]

    Adam: A method for stochastic optimization

    Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015

    work page 2015

  28. [28]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. URL https://www.cs.toronto.edu/ kriz/

    work page 2009

  29. [29]

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, volume 25, pages 1097--1105, 2012

    work page 2012

  30. [30]

    Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks

    Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In ICML, pages 5905--5914, 2021

    work page 2021

  31. [31]

    Gradient-based learning applied to document recognition

    Yann LeCun, L \'e on Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86 0 (11): 0 2278--2324, 1998. doi:10.1109/5.726791

    doi:10.1109/5.726791 1998

  32. [32]

    Structured logconcave sampling with a restricted gaussian oracle

    Yin Tat Lee, Ruoqi Shen, and Kevin Tian. Structured logconcave sampling with a restricted gaussian oracle. In Mikhail Belkin and Samory Kpotufe, editors, Proceedings of Thirty Fourth Conference on Learning Theory, volume 134 of Proceedings of Machine Learning Research, pages 2993--3050. PMLR, 15--19 Aug 2021. URL https://proceedings.mlr.press/v134/lee21a.html

    work page 2021

  33. [33]

    Entropy- MCMC : Sampling from flat basins with ease

    Bolian Li and Ruqi Zhang. Entropy- MCMC : Sampling from flat basins with ease. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=oGNdBvymod

    work page 2024

  34. [34]

    Friendly sharpness-aware minimization

    Tao Li, Pan Zhou, Zhengbao He, Xinwen Cheng, and Xiaolin Huang. Friendly sharpness-aware minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5631--5640, June 2024

    work page 2024

  35. [35]

    Towards Efficient and Scalable Sharpness-Aware Minimization

    Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-aware minimization, 2022. URL https://arxiv.org/abs/2203.02714

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  36. [36]

    Rosenbluth, Marshall N

    Nicholas Metropolis, Arianna W. Rosenbluth, Marshall N. Rosenbluth, Augusta H. Teller, and Edward Teller. Equation of state calculations by fast computing machines. The Journal of Chemical Physics, 21 0 (6): 0 1087--1092, 1953

    work page 1953

  37. [37]

    Radford M. Neal. MCMC Using Hamiltonian Dynamics . May 2011. doi:10.1201/b10905

    doi:10.1201/b10905 2011

  38. [38]

    A method for solving a convex programming problem with convergence rate \( O (1/k^2)\)

    Yurii Nesterov. A method for solving a convex programming problem with convergence rate \( O (1/k^2)\). Soviet Mathematics Doklady, 27: 0 372--376, 1983

    work page 1983

  39. [39]

    Bissacco, Bo Wu, and A

    Yuval Netzer, Tao Wang, Adam Coates, A. Bissacco, Bo Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning. 2011. URL https://api.semanticscholar.org/CorpusID:16852518

    work page 2011

  40. [40]

    Exploring generalization in deep learning

    Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. NeurIPS, 30, 2017

    work page 2017

  41. [41]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024--8035, 2019. URL https://pytorch.org/

    work page 2019

  42. [42]

    Generalized federated learning via sharpness aware minimization

    Zhe Qu, Xingyu Li, Rui Duan, Yao Liu, Bo Tang, and Zhuo Lu. Generalized federated learning via sharpness aware minimization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, p...

    work page 2022

  43. [43]

    Flatsam: Federated learning with sharpness-aware minimization

    Zhe Qu, Xiang Li, and Peter Richt \'a rik. Flatsam: Federated learning with sharpness-aware minimization. In ICLR, 2023

    work page 2023

  44. [44]

    Practical sharpness-aware minimization cannot converge all the way to optima

    Dongkuk Si and Chulhee Yun. Practical sharpness-aware minimization cannot converge all the way to optima. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc

    work page 2023

  45. [45]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

    Pith/arXiv arXiv 2014

  46. [46]

    Adasam: Boosting sharpness-aware minimization with adaptive learning rate and momentum for training deep neural networks, 2023 a

    Hao Sun, Li Shen, Qihuang Zhong, Liang Ding, Shixiang Chen, Jingwei Sun, Jing Li, Guangzhong Sun, and Dacheng Tao. Adasam: Boosting sharpness-aware minimization with adaptive learning rate and momentum for training deep neural networks, 2023 a . URL https://arxiv.org/abs/2303.00565

    Pith/arXiv arXiv 2023

  47. [47]

    Dynamic regularized sharpness aware minimization in federated learning: approaching global consistency and smooth landscape

    Yan Sun, Li Shen, Shixiang Chen, Liang Ding, and Dacheng Tao. Dynamic regularized sharpness aware minimization in federated learning: approaching global consistency and smooth landscape. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023 b

    work page 2023

  48. [48]

    Y. Teng, W. Gao, F. Chalus, A. Choromanska, D. Goldfarb, and A. Weller. Leader stochastic gradient descent for distributed training of deep learning models. In NeurIPS, 2019

    work page 2019

  49. [49]

    Leader Stochastic Gradient Descent for Distributed Training of Deep Learning Models: Extension

    Yunfei Teng, Wenbo Gao, Francois Chalus, Anna Choromanska, Donald Goldfarb, and Adrian Weller. Leader stochastic gradient descent for distributed training of deep learning models: Extension, 2022. URL https://arxiv.org/abs/1905.10395

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  50. [50]

    Bayesian learning via stochastic gradient langevin dynamics

    Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML'11, page 681–688, Madison, WI, USA, 2011. Omnipress. ISBN 9781450306195

    work page 2011

  51. [51]

    How sharpness-aware minimization minimizes sharpness? In The Eleventh International Conference on Learning Representations, 2023 a

    Kaiyue Wen, Tengyu Ma, and Zhiyuan Li. How sharpness-aware minimization minimizes sharpness? In The Eleventh International Conference on Learning Representations, 2023 a . URL https://openreview.net/forum?id=5spDgWmpY6x

    work page 2023

  52. [52]

    Sharpness-aware minimization revisited: Weighted sharpness as a regularization term

    Yeming Wen, Kevin Luk, Max Gazeau, Guodong Zhang, Harris Chan, and Jimmy Ba. Sharpness-aware minimization revisited: Weighted sharpness as a regularization term. NeurIPS, 36, 2023 b

    work page 2023

  53. [53]

    Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models

    Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, and Shuicheng Yan. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    work page 2024

  54. [54]

    Wide residual networks

    Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016

    Pith/arXiv arXiv 2016

  55. [55]

    mixup: Beyond empirical risk minimization

    Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018

    2018

  56. [56]

    Zhang, A

    S. Zhang, A. Choromanska, and Y. LeCun. Deep learning with elastic averaging SGD . In NIPS, 2015

    work page 2015

  57. [57]

    Diffusion-based adversarial training produces robust models

    Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. Diffusion-based adversarial training produces robust models. ICML, 2022

    work page 2022

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.