Pith. sign in

REVIEW 3 major objections 3 minor 1 cited by

Preconditioned Sharpness-Aware Minimization: Unifying Analysis and a Novel Learning Algorithm

T0 review · 3 major / 3 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Preconditioning the sharpness-aware minimization subproblem unifies several SAM variants, and the derived infoSAM rule—scaling each perturbation coordinate by the inverse of an estimated gradient-noise variance—improves generalization on…

desk verdict The preSAM unification and D0 analysis are genuinely useful and theoretically sound, but infoSAM's variance heuristic is not yet convincing enough to support the headline empirical claims. read the letter →

arxiv 2501.06603 v1 pith:XQUGU2CC submitted 2025-01-11 cs.LG

Yilang Zhang , Bingcong Li , Georgios B. Giannakis This is my paper · ORCID
classification cs.LG
keywords sharpness-awareminimizationpreconditioninggeneralizationconvergenceanalysisstochasticgradientnoiseadversarialmodeldegradationdeepneuralnetworksimageclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sharpness-aware minimization (SAM) seeks flat minima by taking a gradient step at a point deliberately perturbed to maximize loss, and many variants have been proposed to make that perturbation smarter. This paper argues that all of these variants are instances of one preconditioned subproblem, $\max_{\|D_t\epsilon\|\le\rho} \langle C_t g_t, \epsilon\rangle$, in which the constraint preconditioner $D_t$ reshapes the allowed neighborhood and the objective preconditioner $C_t$ redirects the stochastic gradient. It proves an $O(1/\sqrt{T})$ convergence rate for the whole family under a bound on $\|D_t^{-1}\|$, covering methods like ASAM and FisherSAM that lacked such guarantees. On top of this framework the paper introduces infoSAM, an objective-preconditioned method whose perturbation scales each gradient coordinate by the inverse of an estimated noise variance, and reports that infoSAM outperforms SAM and ASAM on CIFAR-10, CIFAR-100, ImageNet, and under heavy label noise.

What carries the argument

The load-bearing object is the preconditioned SAM subproblem, $\max_{\epsilon}\, \langle C_t g_t, \epsilon\rangle$ subject to $\|D_t \epsilon\| \le \rho$, whose closed-form solution is $\epsilon_t = \rho D_t^{-2} C_t g_t / \|D_t^{-1} C_t g_t\|$. This subproblem carries the whole paper: the two preconditioners separate the design space into CP ($D_t$ reshapes the perturbation ball, $C_t = I$) and OP ($C_t$ redirects the objective, $D_t = I$), and the convergence theorem's only dependence on preconditioning is the uniform bound $\|D_t^{-1}\| \le D_0$. InfoSAM's mechanism is a specific OP choice, $C_t = \operatorname{diag}((m_t - g_t)^2)^{-1}$, with $m_t$ the exponentially moving average from (7a); this inverse-variance weighting is what redirects perturbations away from noisy coordinates.

What would settle it

Measure the true per-coordinate gradient-noise variance at a fixed checkpoint by drawing many independent minibatches, compare it with (7b), and check whether infoSAM's gains persist when the true variance replaces the estimate; also run infoSAM after zeroing noise in one coordinate, where the inverse-variance rule should produce an unbounded perturbation unless a safeguard exists.

Watch

Extended reading notes

Core claim

The central claim is that preconditioning is the common thread behind sharpness-aware minimization variants: fixing $C_t = I$ yields constraint preconditioning (CP), exemplified by ASAM and FisherSAM, while fixing $D_t = I$ yields objective preconditioning (OP), exemplified by $\ell_1$/ $\ell_\infty$-ball SAM, Lazy SAM, modified SSAM, and VaSSO. Unified convergence analysis then guarantees that every member of the family reaches $\mathbb{E}\|\nabla f(x_t)\|^2$ of order $O(1/\sqrt{T})$, with the caveat that the rate depends on the bound $D_0$ on $\|D_t^{-1}\|$; CP methods can therefore slow down or diverge if their inverse preconditioners are unbounded. The paper's new algorithm infoSAM instantiates OP with $C_t = \hat{\Sigma}_t^{-1}$, where $\hat{\Sigma}_t$ is diagonal with entries $(m_t - g_t)^2$ and $m_t$ is an exponentially moving average of past gradients, giving the perturbation $[\epsilon_t]_i \propto [g_t]_i / [\hat{\sigma}_t^2]_i$. The claimed effect is to suppress dimensions where stochastic gradient noise is large, fixing the adversarial model degradation induced by noise, and the reported experiments show consistent test-accuracy gains over SAM and ASAM.

Load-bearing premise

InfoSAM's perturbation rule assumes that the squared gap between a running average of past gradients and the current minibatch gradient is a faithful measure of per-coordinate gradient noise, and that dividing by that measure is safe; if the gap is biased or zero in some coordinate, the perturbation can amplify noise or divide by zero.

Editorial extensions

If this is right

  • ASAM and FisherSAM inherit a formal $O(1/\sqrt{T})$ convergence guarantee under bounded inverse preconditioners, which the paper notes they had lacked.
  • OP-based variants—sparse perturbation SAM, Lazy SAM, modified SSAM, VaSSO, and n-support-norm SAM—all share the same convergence rate as vanilla SAM, since their preconditioner does not enter the rate.
  • Choosing $\rho$ for CP methods is delicate: under the same $\rho$ as SAM, ASAM underperforms SAM in the reported experiments, consistent with the $D_0$-dependent rate.
  • InfoSAM's inverse-noise-variance perturbation improves top-1 accuracy over SAM and ASAM on CIFAR-10 and CIFAR-100 with four architectures, on ImageNet, and under 25–75% label noise.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is an automatic schedule for $\rho$ that adapts to measured gradient noise, rather than fixing a global perturbation radius.
  • The noise-variance estimator in (7b) is a heuristic borrowed from the AdaBelief step-size rule; one could test infoSAM against a version using an unbiased variance estimator to see whether the gains come from the inverse-variance weighting or from the EMA smoothing.
  • If the inverse-variance weighting is truly the mechanism, infoSAM should also help other perturbation-based methods such as adversarial training, where the attacker step is likewise corrupted by stochastic gradients.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes preSAM, a preconditioned formulation of the SAM inner maximization problem (Eq. 4), with the closed-form perturbation in Eq. (5), and uses it to categorize existing SAM variants into constraint preconditioning (CP) and objective preconditioning (OP). A unified nonconvex convergence theorem (Theorem 1) is stated under bounded \|D_t^{-1}\|. The paper then introduces infoSAM, an OP method whose perturbation scales each coordinate by the inverse of an estimated gradient-noise variance (Eq. 7), and reports experiments on CIFAR-10, CIFAR-100, ImageNet, and label-noise settings comparing infoSAM with SGD, SAM, and ASAM.

Significance. The preSAM framework is a clean and useful conceptual unification: several existing SAM variants, including ASAM, FisherSAM, ℓ∞/ℓ1-SAM, modified-SSAM, Lazy SAM, and VaSSO, are shown to fit as special cases, and the convergence analysis is self-contained under standard assumptions. The main strengths are the closed-form treatment of the preconditioned subproblem, the explicit separation of CP and OP effects on convergence, and the relatively broad set of architectures tested with repeated runs. However, the paper's central novel claim rests on the infoSAM perturbation rule, which is introduced as an unvalidated heuristic, and the empirical comparison omits the closest existing OP baseline, VaSSO. The theoretical contribution for infoSAM is also weaker than the abstract suggests, since Theorem 1 does not depend on C_t and therefore provides no support for the specific variance-inversion choice. If the step-size condition and variance-estimator issues are fixed, the framework contribution is solid; the infoSAM-specific claims need more support.

major comments (3)
  1. [Section III-A, Theorem 1] The stated step-size condition η ≤ 2/(3L) is insufficient for the proof as written. In the proof of Theorem 1, after dividing by η, the left-hand side carries the factor (1 − 3Lη/2); at η = 2/(3L) this factor is zero, so the inequality cannot be divided through to obtain the stated O(1/√T) bound on E‖∇f(x_t)‖². The condition should be strict, η < 2/(3L), or should be replaced by η ≤ c/(3L) for some c < 2, and the final bound should account for division by the positive constant (1 − 3Lη/2). This is fixable, but as stated the theorem's proof does not go through at the endpoint.
  2. [Section IV-B, Eqs. (7a)-(7c)] The variance estimator in Eq. (7b) is inverted without any safeguard in Eq. (7c). Concretely, σ̂_i² = (m_i − g_i)² can be zero whenever the EMA m_i equals the current gradient coordinate, for example on a noiseless or slowly varying coordinate; Eq. (7c) then divides by zero, and in finite precision the perturbation is dominated by rounding-level differences. No clamping or floor is specified in Algorithm 1 or in the hyperparameter tables in Appendix E. Moreover, the squared difference is a single-sample estimate whose expectation is proportional to the true variance plus a bias-squared term, and the required stationarity assumption on the gradient mean is never stated. Because Theorem 1 is independent of C_t, it provides no theoretical justification for this particular estimator, so the claimed empirical improvement rests entirely on an unvalidated heuristic. The authors should add a floor/regularization, prove an estimation-error bound under explicitly stated assumptions, or provide a sensitivity analysis.
  3. [Section V, Tables III and Fig. 1; Appendix B] VaSSO, which is listed in Table I as an OP instance and given a preSAM interpretation in Appendix B, is absent from all experiments. VaSSO is the closest existing method to infoSAM in spirit, since both aim to suppress the effect of gradient noise in the perturbation. Without this baseline, the claims in Section V-A that infoSAM "achieves the highest accuracy in all model setups" and in the abstract that it "leads to improved generalization" are not fully supported, because the comparison set is limited to SGD, SAM, and ASAM.
minor comments (3)
  1. [Appendix A, Eq. (9)] Equation (9) divides by ∥D_t^{-1} C_t g_t∥, but if this vector is zero the perturbation ϵ_t is undefined. The paper should either state a nondegeneracy condition for all t or define ϵ_t = 0 in that case.
  2. [Appendix E-B, Table VI] Table VI is headed ResNet-18, while the text in Section V-B says the ImageNet experiments use ResNet-50; one of these is a typo and should be corrected.
  3. [Appendix E-A, Table IV] In the PyramidNet-110 row of Table IV, the header mentions ASAM but the table body contains the label "ESAM," and infoSAM uses ρ = 0.2 while SAM and ASAM use ρ = 0.1, so that row's comparison is not controlled for the perturbation radius.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Theorem 1 is proved from stated assumptions without using infoSAM's variance estimator, and the framework's unifications are algebraic equivalences rather than derived predictions.

full rationale

The convergence analysis (Theorem 1) is self-contained: Appendix A proves the rate from Assumptions 1-3 and the bound ||D_t^{-1}|| <= D_0, and the proof never uses the particular form of C_t (or the infoSAM variance estimate), so the guarantee is not fitted to any experimental outcome. The claimed unifications are explicit algebraic reductions: e.g., VaSSO is shown in App. B to be recovered by a rank-one-plus-identity C_t, and Table II maps OP choices to equivalent norm constraints; these are reparameterizations of existing updates, not derivations of those algorithms' performance from the framework, so they do not smuggle in conclusions. infoSAM's perturbation rule (Eq. 7c) depends on the heuristic EMA-squared-difference estimator (Eq. 7b), which is unvalidated and could vanish or amplify noise; this is a correctness and robustness concern, but it is not circular because the estimator is not fitted to the reported test accuracies and no theorem assumes the rule. The self-citation [21] is used only to motivate the AMD phenomenon and to give VaSSO's OP form; the paper supplies its own SNR measurement and case studies, and no uniqueness or load-bearing claim is imported from [21]. Accordingly, none of the enumerated circularity patterns is exhibited.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The central convergence theorem rests on standard stochastic optimization assumptions plus a boundedness condition on D^{-1}; infoSAM additionally rests on a heuristic variance estimator with a tuned EMA coefficient. No new physical entities are introduced.

free parameters (2)
  • alpha (EMA coefficient in infoSAM) = 0.001, 0.005, 0.01, 0.025, or 0.05 depending on model and dataset
    Controls the variance estimate in Eq. (7); values in Tables IV-VI are tuned per architecture and dataset, and the paper gives no sensitivity analysis.
  • rho (perturbation radius) = 0.075, 0.1, or 0.2 per dataset
    Chosen per dataset for all SAM variants; CP methods such as ASAM perform poorly with the same rho as SAM, so this is a load-bearing tuning choice.
assumptions (6)
  • domain assumption Loss f is lower bounded (Assumption 1)
    Used in Theorem 1 to telescope the descent bound; standard for nonconvex stochastic optimization.
  • domain assumption Stochastic gradient g is L-Lipschitz (Assumption 2)
    Used in Lemmas 1-2 and Theorem 1; requires the minibatch gradient to satisfy the same Lipschitz condition as the full gradient.
  • domain assumption Unbiased gradient with bounded variance (Assumption 3)
    Standard in stochastic nonconvex optimization; needed for the O(1/sqrt T) rate and the sigma^2 term.
  • domain assumption Bounded inverse preconditioner norm ||D_t^{-1}|| <= D0
    Used to bound ||epsilon_t|| in Eq. (9); the authors note ASAM and FisherSAM may violate this, so the theorem's coverage of those variants requires an extra boundedness condition.
  • ad hoc to paper SAM perturbation is well approximated by first-order Taylor expansion with stochastic gradient (Eq. 2)
    The adversarial model interpretation that motivates infoSAM relies on this approximation; the convergence theorem itself does not need it.
  • ad hoc to paper Squared difference between EMA and current gradient estimates per-coordinate variance (Eq. 7b)
    No proof or bound is given that E[(m_t-g_t)^2] approximates Var(g_t); the entire infoSAM rescaling depends on this heuristic.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Preconditioned Sharpness-Aware Minimization: Unifying Analysis and a Novel Learning Algorithm." pith.science (2026). https://pith.science/paper/XQUGU2CC

@misc{pith2026250106603,
  author       = {Pith},
  title        = {Pith review of: Preconditioned Sharpness-Aware Minimization: Unifying Analysis and a Novel Learning Algorithm},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XQUGU2CC}},
  note         = {Machine review of arXiv:2501.06603}
}
read the original abstract

Targeting solutions over `flat' regions of the loss landscape, sharpness-aware minimization (SAM) has emerged as a powerful tool to improve generalizability of deep neural network based learning. While several SAM variants have been developed to this end, a unifying approach that also guides principled algorithm design has been elusive. This contribution leverages preconditioning (pre) to unify SAM variants and provide not only unifying convergence analysis, but also valuable insights. Building upon preSAM, a novel algorithm termed infoSAM is introduced to address the so-called adversarial model degradation issue in SAM by adjusting gradients depending on noise estimates. Extensive numerical tests demonstrate the superiority of infoSAM across various benchmarks.

Figures

Figures reproduced from arXiv: 2501.06603 by the authors.

Figure 1
Figure 1. (a) Top-1 and (b) top-5 accuracies on ImageNet. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Performance under different levels of label noise. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Behavior of SGD (left), ideal SAM (middle), and SAM with stochastic noise (right) near asymmetric valley. First row: [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison of the adversarial models in (a) SAM and [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Avoiding spurious sharpness minimization broadens applicability of SAM

    cs.LG 2025-02 conditional novelty 6.0 of 10

    SAM's failure in language modeling is traced to a dominant 'logit path' that minimizes sharpness spuriously, and the proposed Functional-SAM, which removes that path, improves validation loss over AdamW and SAM.

Reference graph

Works this paper leans on

36 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    Bert: Pre- training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre- training of deep bidirectional transformers for language understanding,” arXiv:1810.04805, 2018

    arXiv 2018

  2. [2]

    Ast: Audio spectrogram trans- former,

    Y . Gong, Y .-A. Chung, and J. Glass, “Ast: Audio spectrogram trans- former,” arXiv preprint arXiv:2104.01778 , 2021

    arXiv 2021

  3. [3]

    When vision transformers out- perform resnets without pre-training or strong data augmentations,

    X. Chen, C.-J. Hsieh, and B. Gong, “When vision transformers out- perform resnets without pre-training or strong data augmentations,” in Proc. Int. Conf. Learning Represention , 2022

    work page 2022

  4. [4]

    Un- derstanding deep learning (still) requires rethinking generalization,

    C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Un- derstanding deep learning (still) requires rethinking generalization,” Communications of the ACM , vol. 64, no. 3, pp. 107–115, 2021

    2021

  5. [5]

    Dropout: a simple way to prevent neural networks from overfitting,

    N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” J. Mach. Learn. Res. , vol. 15, pp. 1929–1958, 2014

    work page 1929

  6. [6]

    Au- toaugment: Learning augmentation strategies from data,

    E. D. Cubuk, B. Zoph, D. Mane, V . Vasudevan, and Q. V . Le, “Au- toaugment: Learning augmentation strategies from data,” in Proc. Conf. Computer Vision and Pattern Recognition , 2019, pp. 113–123

    work page 2019

  7. [7]

    The effects of regular- ization and data augmentation are class dependent,

    R. Balestriero, L. Bottou, and Y . LeCun, “The effects of regular- ization and data augmentation are class dependent,” arXiv preprint arXiv:2204.03632, 2022

    arXiv 2022

  8. [8]

    On large-batch training for deep learning: Generalization gap and sharp minima

    N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang, “On large-batch training for deep learning: Generalization gap and sharp minima.” in Proc. Int. Conf. Learning Represention , 2016

    work page 2016

Show all 36 references
  1. [9]

    Towards understanding sharpness-aware minimization

    M. Andriushchenko and N. Flammarion, “Towards understanding sharpness-aware minimization.” in Proc. Int. Conf. Machine Learning , 2022, pp. 639–668

    2022

  2. [10]

    How does sharpness-aware minimiza- tion minimizes sharpness,

    K. Wen, T. Ma, and Z. hiyuan Li, “How does sharpness-aware minimiza- tion minimizes sharpness,” in Proc. Int. Conf. Learning Represention , 2023

    2023

  3. [11]

    Fantas- tic generalization measures and where to find them,

    Y . Jiang, B. Neyshabur, D. Krishnan, H. Mobahi, and S. Bengio, “Fantas- tic generalization measures and where to find them,” arXiv:1912.02178, 2019

    1912 arXiv

  4. [12]

    Sharpness-aware minimization for efficiently improving generalization,

    P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur, “Sharpness-aware minimization for efficiently improving generalization,” in Proc. Int. Conf. Learning Represention , 2021

    2021

  5. [13]

    ASAM: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural net- works,

    J. Kwon, J. Kim, H. Park, and I. K. Choi, “ASAM: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural net- works,” in Proc. Int. Conf. Machine Learning , vol. 139, 2021, pp. 5905– 5914

    2021

  6. [14]

    Fisher SAM: Information geometry and sharpness aware minimisation

    M. Kim, D. Li, S. X. Hu, and T. M. Hospedales, “Fisher SAM: Information geometry and sharpness aware minimisation.” in Proc. Int. Conf. Machine Learning , 2022, pp. 11 148–11 161

    2022

  7. [15]

    Surrogate gap minimization improves sharpness-aware training

    J. Zhuang, B. Gong, L. Yuan, Y . Cui, H. Adam, N. Dvornek, S. Tatikonda, J. Duncan, and T. Liu, “Surrogate gap minimization improves sharpness-aware training.” in Proc. Int. Conf. Learning Rep- resention, 2022

    2022

  8. [16]

    Penalizing gradient norm for efficiently improving generalization in deep learning,

    Y . Zhao, H. Zhang, and X. Hu, “Penalizing gradient norm for efficiently improving generalization in deep learning,” in Proc. Int. Conf. Machine Learning, 2022, pp. 26 982–26 992

    2022

  9. [17]

    Implicit regularization of sharpness-aware minimization for scale-invariant problems,

    B. Li, L. Zhang, and N. He, “Implicit regularization of sharpness-aware minimization for scale-invariant problems,” in Proc. Adv. Neural Info. Processing Systems, 2024

    2024

  10. [18]

    Make sharpness-aware minimization stronger: A sparsified perturbation ap- proach,

    P. Mi, L. Shen, T. Ren, Y . Zhou, X. Sun, R. Ji, and D. Tao, “Make sharpness-aware minimization stronger: A sparsified perturbation ap- proach,” in Proc. Adv. Neural Info. Processing Systems , 2022

    2022

  11. [19]

    An adaptive policy to em- ploy sharpness-aware minimization,

    W. Jiang, H. Yang, Y . Zhang, and J. Kwok, “An adaptive policy to em- ploy sharpness-aware minimization,” arXiv preprint arXiv:2304.14647 , 2023

    2023 arXiv

  12. [20]

    SS-SAM: Stochastic scheduled sharpness-aware minimization for efficiently training deep neural net- works,

    Y . Zhao, H. Zhang, and X. Hu, “SS-SAM: Stochastic scheduled sharpness-aware minimization for efficiently training deep neural net- works,” arXiv:2203.09962, 2022

    2022 arXiv

  13. [21]

    Enhancing sharpness-aware optimization through variance suppression,

    B. Li and G. Giannakis, “Enhancing sharpness-aware optimization through variance suppression,” in Proc. Advances in Neural Info. Pro- cess. Syst. , vol. 36, 2023, pp. 70 861–70 879

    2023

  14. [22]

    Stochastic first-and zeroth-order methods for nonconvex stochastic programming,

    S. Ghadimi and G. Lan, “Stochastic first-and zeroth-order methods for nonconvex stochastic programming,” SIAM Journal on Optimization , vol. 23, no. 4, pp. 2341–2368, 2013

    2013

  15. [23]

    Optimization methods for large- scale machine learning,

    L. Bottou, F. E. Curtis, and J. Nocedal, “Optimization methods for large- scale machine learning,” SIAM Review , vol. 60, no. 2, pp. 223–311, 2018

    2018

  16. [24]

    Sharp minima can generalize for deep nets,

    L. Dinh, R. Pascanu, S. Bengio, and Y . Bengio, “Sharp minima can generalize for deep nets,” in Proc. Int. Conf. Machine Learning , 2017, pp. 1019–1028

    2017

  17. [25]

    Sparse prediction with the k- support norm,

    A. Argyriou, R. Foygel, and N. Srebro, “Sparse prediction with the k- support norm,” in Proc. Advances in Neural Info. Process. Syst. , 2012, pp. 1457–1465

    2012

  18. [26]

    AdaBelief optimizer: Adapting stepsizes by the belief in observed gradients,

    J. Zhuang, T. Tang, Y . Ding, S. C. Tatikonda, N. Dvornek, X. Pa- pademetris, and J. Duncan, “AdaBelief optimizer: Adapting stepsizes by the belief in observed gradients,” in Proc. Adv. Neural Info. Processing Systems, vol. 33, 2020, pp. 18 795–18 806

    2020

  19. [27]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, G. Hinton et al. , “Learning multiple layers of features from tiny images,” 2009

    2009

  20. [28]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. Conf. Computer Vision and Pattern Recognition , June 2016

    2016

  21. [29]

    Densely connected convolutional networks,

    G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proc. Conf. Computer Vision and Pattern Recognition, July 2017

    2017

  22. [30]

    Wide residual networks,

    S. Zagoruyko, “Wide residual networks,” arXiv preprint arXiv:1605.07146, 2016

    2016 arXiv

  23. [31]

    Deep pyramidal residual networks,

    D. Han, J. Kim, and J. Kim, “Deep pyramidal residual networks,” in Proc. Conf. Computer Vision and Pattern Recognition , July 2017

    2017

  24. [32]

    ImageNet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in Proc. Conf. Computer Vision and Pattern Recognition , 2009, pp. 248–255

    2009

  25. [33]

    Asymmetric valleys: Beyond sharp and flat local minima

    H. He, G. Huang, and Y . Yuan, “Asymmetric valleys: Beyond sharp and flat local minima.” in Proc. Adv. Neural Info. Processing Systems , vol. 32, 2019, pp. 2549–2560

    2019

  26. [34]

    Improved regularization of convolutional neural networks with cutout

    T. Devries and G. W. Taylor, “Improved regularization of convolutional neural networks with cutout.” vol. abs/1708.04552, 2017. APPENDIX A MISSING PROOFS For notational simplicity, we first rewrite Alg. 1 as xt+ 1 2 = xt + ϵt, where ϵt = ρ D−2 t Ctgt ∥D−1 t Ctgt∥ (8a) xt+1 = x...

    2017 arXiv

  27. [35]

    xt finishes the proof

    Taking expectation w.r.t. xt finishes the proof. Lemma 2. Alg. 1 (or equivalently iteration (8)) ensures that E ∥gt(xt+ 1 2 )∥2 ≤ 2L2D2 0ρ2 + 2E ∥∇f (xt)∥2 + 2σ2. Proof. The proof starts with bounding ∥gt(xt+ 1 2 )∥ via ∥gt(xt+ 1 2 )∥2 = ∥gt(xt+ 1 2 ) − gt(xt) +gt(xt)∥2 ≤ 2∥gt...

  28. [36]

    Hyperparameters used in our experiments are summarized in Tabs

    are leveraged. Hyperparameters used in our experiments are summarized in Tabs. IV and V. B. ImageNet ImageNet [32] has 1,281,167 images from 1000 classes for training and 50,000 images for validation. Due to the constraints on computational resources, we report the averaged re...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.