GCSAM: Gradient Centralized Sharpness Aware Minimization

Aleksandar Vakanski; Boyu Zhang; Min Xian; Mohamed Hassan

REVIEW 4 major objections 5 minor 53 references

GCSAM: Gradient Centralized Sharpness Aware Minimization

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GCSAM claims that centralizing the ascent gradient in SAM produces flatter minima, better test accuracy, and lower computational cost than SAM and Adam.

desk verdict Neat but incremental SAM variant; the appendix's PAC-Bayes argument does not survive scrutiny, and the single-run empirical results are too thin to carry the central claim as written. read the letter →

arxiv 2501.11584 v2 pith:CI2G3U3R submitted 2025-01-20 cs.LG

Mohamed Hassan , Aleksandar Vakanski , Boyu Zhang , Min Xian This is my paper · ORCID

classification cs.LG

keywords gradientcentralizationsharpness-awareminimizationflatminimageneralizationimageclassificationmedicalimagingPAC-Bayesboundoptimizer

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes GCSAM, an optimizer that inserts Gradient Centralization into the ascending step of Sharpness-Aware Minimization (SAM). The authors aim to establish that centering the ascent gradient before perturbing the weights both reduces sharpness more effectively and stabilizes training, so GCSAM attains higher test accuracy than SAM and Adam across general and medical image benchmarks while running faster than SAM. If true, GCSAM is a drop-in modification of an already widespread method, so the improvement would propagate to any SAM-based training pipeline. The supporting argument is a PAC-Bayes bound in which the centralized perturbation has smaller magnitude than the original gradient, plus empirical results on CIFAR-10, breast ultrasound, and COVID-19 chest X-ray datasets.

What carries the argument

The central object is the projection matrix P = I − ee^⊤ that subtracts the mean from each gradient vector, together with the perturbation rule ε_GC = ρ (P∇L_S(w))/||P∇L_S(w)||_p. The projection defines what the paper calls gradient centralization; applying it inside SAM's inner maximization yields a shorter ascent direction, which the paper uses to argue for a tighter sharpness bound (max over ε_GC of L_S(w+ε_GC) no larger than max over ε of L_S(w+ε)). The inequality ||P∇L_S(w)||^2 ≤ ||∇L_S(w)||^2 is the keystone of the proof and the algorithm.

What would settle it

Train a fixed architecture (e.g., ResNet-18) on CIFAR-10 with GCSAM, SAM, and Adam under identical schedules for at least 10 random seeds and compare test accuracy and time-to-target-accuracy; if GCSAM does not consistently beat or match SAM, the central claim is refuted. A targeted theoretical check would compute the PAC-Bayes bound of Theorem A.1 for a small two-layer network under both ε and ε_GC and verify whether the centralized bound actually holds.

Watch

Extended reading notes

Core claim

The central claim is that replacing the ascent gradient in SAM with its centralized version, ε_GC = ρ ∇_GC L_S(w)/||∇_GC L_S(w)||, does not sacrifice the sharpness-reduction guarantee and in fact tightens it, because ||∇_GC L_S(w)||^2 = ||∇L_S(w)||^2 − (e^⊤∇L_S(w))^2 ≤ ||∇L_S(w)||^2. From this inequality the paper derives a PAC-Bayes generalization bound for GCSAM and concludes that the method converges to flatter minima. Empirically, the paper reports that on four architectures GCSAM achieves higher test accuracy than Adam and SAM on CIFAR-10, breast ultrasound, and COVID-19 CXR-4 datasets, and that it trains faster than SAM on the medical benchmarks.

Load-bearing premise

The paper's theoretical tightening of the sharpness bound rests on assuming that the PAC-Bayes inequality from SAM remains valid when the perturbation is restricted to the centralized-gradient direction, a step the appendix passes over quickly.

Editorial extensions

If this is right

Any existing SAM-based pipeline can adopt GCSAM by changing only the ascent-gradient computation, so claimed gains would transfer across applications.
If the bound in Theorem A.1 holds, GCSAM inherits SAM's PAC-Bayes generalization guarantee with a sharper sharpness term.
The reduced gradient magnitude implies a built-in damping of gradient explosions during the ascent step, which may permit larger perturbation radii ρ.
Lower training time on vision transformers compared with SAM, if confirmed, would make sharpness-aware training more practical on large models.
GCSAM's accuracy gains over Adam, if replicated, offer a free accuracy improvement for image classifiers without extra hyperparameter tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The proof's transition from max over ε to max over ε_GC inside the PAC-Bayes bound is the step most likely to fail; if it does, GCSAM's theoretical advantage reduces to an empirical regularity rather than a proven guarantee.
Because the mean subtraction depends on layer width, GCSAM's effect should be stronger for wide layers and nearly vanish for narrow bottleneck layers; this is a testable prediction about where the optimizer helps most.
The reported speedups relative to SAM may be partly due to the early-stopping rule at 100% training accuracy; a time-to-target-accuracy comparison without early stopping would separate convergence speed from the stopping protocol.
GCSAM's principle is orthogonal to other SAM variants such as ASAM and MSAM, so combining centralization with those modifications is a natural next test.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Quick take on GCSAM. The algorithm is exactly what it says: centralize the gradient, use the centralized gradient to compute SAM's ascent perturbation, then update. Equations 6-7 and Algorithm 1 make this unambiguous. It is a legitimate composition, not a huge conceptual leap, but the paper is honest about that and the implementation is simple enough to reproduce. Credit where due: the medical imaging experiments cover BUS and COVIDx with four architectures, and they compare against several SAM variants as well as SAM and Adam. That is a reasonable empirical scope for a short optimizer paper.

The problem is the theory. The stress-test note holds up: Theorem A.1 does not prove what it claims. The proof copies Foret et al.'s PAC-Bayes inequality, then replaces max over the perturbation ball with L_S(w + epsilon_GC). Since epsilon_GC is one feasible point inside that ball, the max is greater than or equal to that value, so the inequality goes the wrong direction. The later norm comparison (||P grad|| <= ||grad||) does not rescue it, because epsilon_GC is normalized to have norm rho exactly. So the claimed tighter sharpness bound is unsupported. I looked for another argument in the text; there isn't one.

Empirically, there is a more basic issue: no seeds, no error bars, and no explicit statement of the base optimizer used for SAM and GCSAM in Tables 1-2. The pseudocode uses SGD, the prose says "such as SGD or Adam," and the baseline is labeled Adam. If SAM and GCSAM were run with Adam, that should be stated. Without repeats or seeds, differences of 0.3-2.4 points are within plausible run-to-run noise. The speed comparisons in Table 3 are also single-run and hard to interpret.

Overall, the algorithm is plausible and the experimental scope is broad, but the central theoretical claim collapses and the empirical support is too thin as it stands. A revision with the proof fixed or removed, proper seeds and error bars, and explicit base-optimizer settings could become a useful incremental contribution. I would not cite it in its current form, but the algorithmic idea is real enough that I would not desk-reject a revision.

Recommendation: send to peer review, expecting major revision.

Referee Report

4 major / 5 minor

Summary. The paper proposes Gradient Centralized Sharpness-Aware Minimization (GCSAM), which applies Gradient Centralization (GC) to the gradient used in SAM's inner ascent step. It claims that centralizing the gradient reduces gradient noise and variance, leads to tighter sharpness bounds, and yields better test accuracy and computational efficiency than SAM and Adam on CIFAR-10, breast ultrasound, and COVID-19 chest X-ray datasets across ResNet50, VGG16, ViT, and Swin Transformer. A PAC-Bayes generalization bound is stated in Theorem A.1 as theoretical support.

Significance. If the central claims were sound, GCSAM would be a simple, potentially useful drop-in modification of SAM for image classification. The paper has strengths: it releases code, evaluates on both general and medical imaging benchmarks, compares against several SAM variants, and includes loss-landscape visualizations. However, the theoretical mechanism in Appendix A.1 is invalid, and the experimental evidence consists of single-run accuracy numbers without error bars, seeds, or a reported base optimizer for SAM/GCSAM. As a result, the paper's main claims are currently unsupported.

major comments (4)

[Appendix A.1, Theorem A.1 proof] The proof of Theorem A.1 contains a direction error that invalidates the claimed bound. After copying the PAC-Bayes inequality from Foret et al., the manuscript replaces the expectation over Gaussian perturbations with max_{||ϵ||_p≤ρ} L_S(w+ϵ_GC), where ϵ_GC is a single perturbation along the centralized gradient. Since ϵ_GC is feasible for the original maximization, max_{||ϵ||_p≤ρ} L_S(w+ϵ) ≥ L_S(w+ϵ_GC), so this replacement gives a lower bound, not an upper bound. The subsequent norm comparison ||∇_GC L_S(w)||_2 ≤ ||∇L_S(w)||_2 does not reduce the perturbation radius because ϵ_GC is normalized to have norm exactly ρ. Thus the paper does not establish that GCSAM has a tighter sharpness bound than SAM, and the stated mechanism for GCSAM's generalization benefit is unsupported.
[§3, Tables 1–3] All experimental comparisons are single runs without seeds, error bars, or statistical tests. Table 1 reports differences as small as 0.03 percentage points (ViT: 84.05 vs. 84.08), and Table 3 shows comparable gaps; without variance estimates or multiple seeds, the claim that GCSAM 'consistently outperforms' SAM and Adam is not established. The manuscript also never states which base optimizer (e.g., SGD or Adam) is used for SAM and GCSAM in the experiments, so it is impossible to verify that the comparison is fair. These omissions concern the paper's central empirical claim and should be addressed.
[§3.1, experimental protocol] The experimental protocol is under-specified: the learning-rate and ρ grid values, number of epochs, weight decay, data augmentation, and early-stopping criterion are not reported. Training 'until each model achieves 100% accuracy on the training set' is an unusual stopping rule that can interact with optimizer differences, and the paper does not say whether the same stopping rule and hyperparameters were used for every optimizer. This prevents reproducibility of the results in Tables 1–3.
[§4, 'Better Computational Efficiency'] The computational-efficiency claim is not supported by the reported speed numbers. Table 3 lists GCSAM speeds of 2.10, 1.07, 1.18, and 1.87 relative to Adam, while SAM speeds are 2.27, 1.28, 1.32, and 2.04; these are single measurements with no stated measurement methodology (hardware, batch size, number of steps, warm-up, or number of runs). In the ResNet50 and Swin Transformer rows, GCSAM is actually slower than SAM, which contradicts the blanket claim of improved efficiency.

minor comments (5)

[Eq. (6)] The maximization in Eq. (6) is over ||ϵ_GC||_p ≤ ρ, but ϵ_GC is defined in Eq. (7) as a single normalized direction; it should be clarified whether the maximization is over all perturbations of that form or only the one computed at the current iterate.
[Algorithm 1] The line 'Compute Descent Step w = w − α(gGC(w + ϵGC))' is ambiguous: it is not defined how gGC applies centralization to gradients evaluated at the perturbed weights, and the parentheses are unbalanced. This should be rewritten with clear notation.
[Theorem A.1] The bound in Theorem A.1 introduces a parameter η that is not defined, uses ρ both as the perturbation radius and inside the Gaussian perturbation scaling, and the notation for the projection P in the theorem statement differs from the matrix P = I − ee^T used in the proof. These inconsistencies make the statement hard to parse.
[Fig. 3] The loss-landscape visualization is qualitative; the text claims GCSAM produces 'consistently flatter minima' but no quantitative sharpness measure (e.g., Hessian eigenvalue or the maximum loss in a ball of fixed radius) is reported.
[Throughout] There are numerous typos and grammatical issues, including 'an important area' in the Introduction, 'paramter update' in the Fig. 2 caption, 'fot' in the same caption, and 'Efficieny' in the Section 4 heading. A careful copyedit is needed.

Circularity Check

1 steps flagged · score 6.0 of 10

Theorem A.1's proof asserts the GCSAM bound verbatim as an intermediate step, so the claimed tighter sharpness bound is circularly assumed rather than derived.

other [Appendix A.1, Theorem A.1 proof, immediately after the SAM PAC-Bayes inequality]
"which can be bounded as: LD(w) ≤ max_{||ϵ||_p≤ρ} L_S(w+ϵ_GC) + sqrt(1/(n-1)(k log(1+||w||_2^2/(η^2 ρ^2)(1+sqrt(log n/k))^2)+4 log(n/δ)+O(1)))."

This line asserts the theorem's conclusion as an intermediate step. The preceding display from Foret et al. bounds an expectation over Gaussian perturbations, not a maximum over the single GCSAM perturbation ϵ_GC; no inequality connects one to the other. The next paragraph's norm comparison ||∇_GC L_S(w)||_2 ≤ ||∇L_S(w)||_2 cannot close the gap because ϵ_GC is normalized to have norm ρ regardless of the pre-normalization gradient norm. Thus the claimed tighter sharpness bound is not derived from the SAM bound; the proof assumes exactly the result it is meant to establish.

full rationale

GCSAM's headline empirical claim is self-contained: test accuracies in Tables 1-3 are measured, not derived from fitted constants, and the compared optimizers are external baselines. The minor self-citation [15] appears only in a related-work sentence alongside an independent citation [35] and is not load-bearing. The circularity lies in Appendix A.1. The proof of Theorem A.1 starts from Foret et al.'s PAC-Bayes bound and then writes 'which can be bounded as' followed by the theorem's own RHS (max over ϵ_GC). No derivation connects the Gaussian-expectation SAM bound to a maximum over the single centralized-gradient perturbation. The subsequent norm inequality is irrelevant because ϵ_GC = ρ ∇GC L_S(w)/||∇GC L_S(w)||_p is normalized to radius ρ, so the smaller pre-normalization norm does not reduce the perturbation radius. Thus the claimed tighter sharpness bound is assumed in its own proof. This is a genuine circular step for the theoretical mechanism, though the empirical benchmark results remain independent evidence. Score 6 reflects partial circularity: the paper's stated first-principles justification reduces to its conclusion, while the experimental comparisons do not.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim depends on tuned hyperparameters rho and alpha, on the unproved transfer of the SAM PAC-Bayes bound to the centralized-gradient perturbation, and on the assumption that one training run per setting is enough to compare optimizers. No new physical or conceptual entities are introduced.

free parameters (3)

Perturbation radius rho = not reported (grid-searched)
In Equations 6-7 and Algorithm 1, rho controls the ascent perturbation size. Section 3.1 says grid search identified optimal rho per model, but values are not listed.
Learning rate alpha = not reported (1e-4 on medical datasets)
Learning rate is a free hyperparameter. It was grid-searched on CIFAR-10 and fixed at 1e-4 for BUS and COVID-19, but no schedule or optimizer-specific values are reported.
Weight decay lambda = not reported
The objective in Equation 6 includes lambda over 2 times the squared weight norm divided by rho squared. The paper never specifies lambda, although it affects the solution.

assumptions (4)

domain assumption Gaussian perturbation of the weights does not improve the true risk: LD(w) <= E_{epsilon~N(0,sigma^2)}[LD(w+epsilon)] for some sigma > 0.
Invoked in Appendix A.1 as the starting point of the PAC-Bayes argument. It is the same assumption used by Foret et al.
ad hoc to paper The PAC-Bayes bound in Foret et al. remains valid when the ascent perturbation is restricted to the centralized-gradient direction.
Appendix A.1 substitutes epsilon_GC into the original SAM bound without proving that the bound applies to the GCSAM objective. This is the main unproved step.
domain assumption Training each model until 100 percent training accuracy with early stopping yields a fair basis for comparing generalization across optimizers.
Sections 3.1 and 3.2 set this protocol. It may favor optimizers that converge quickly and does not control for overfitting dynamics.
domain assumption A single run per optimizer is a sufficient estimator of test accuracy.
No seeds or repeats are reported. The claim of consistent improvement assumes the observed single-run accuracies are not noise.

how reviews work

0 comments

Cite this review

Pith. "Pith review of GCSAM: Gradient Centralized Sharpness Aware Minimization." pith.science (2026). https://pith.science/paper/CI2G3U3R

@misc{pith2026250111584,
  author       = {Pith},
  title        = {Pith review of: GCSAM: Gradient Centralized Sharpness Aware Minimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CI2G3U3R}},
  note         = {Machine review of arXiv:2501.11584}
}

read the original abstract

The generalization performance of deep neural networks (DNNs) is a critical factor in achieving robust model behavior on unseen data. Recent studies have highlighted the importance of sharpness-based measures in promoting generalization by encouraging convergence to flatter minima. Among these approaches, Sharpness-Aware Minimization (SAM) has emerged as an effective optimization technique for reducing the sharpness of the loss landscape, thereby improving generalization. However, SAM's computational overhead and sensitivity to noisy gradients limit its scalability and efficiency. To address these challenges, we propose Gradient-Centralized Sharpness-Aware Minimization (GCSAM), which incorporates Gradient Centralization (GC) to stabilize gradients and accelerate convergence. GCSAM normalizes gradients before the ascent step, reducing noise and variance, and improving stability during training. Our evaluations indicate that GCSAM consistently outperforms SAM and the Adam optimizer in terms of generalization and computational efficiency. These findings demonstrate GCSAM's effectiveness across diverse domains, including general and medical imaging tasks.

Figures

Figures reproduced from arXiv: 2501.11584 by the authors.

**Figure 2.** (a) GCSAM Algorithm. (b) Schematic of the GCSAM paramter update, where [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Comparison of loss landscapes for Adam, SAM, and GCSAM on the COVID-19 dataset. Adam demonstrates the sharpest [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 49 canonical work pages

[1]

Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang

Mart ´ın Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communica- tions Security, pages 308–318, 2016

work page 2016
[2]

Quantum con- volutional neural network for image classification

Mohammed Al-Yousif and Belal Al-Khateeb. Quantum con- volutional neural network for image classification. Fusion Practice and Applications, 15(2):61–72, 2024

work page 2024
[3]

Sharpness-aware minimization improves language model generalization

Dara Bahri, Hossein Mobahi, and Yi Tay. Sharpness-aware minimization improves language model generalization. Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Volume 1: Long Papers, 2022

work page 2022
[4]

Momentum-sam: Sharpness aware minimization without computational overhead

Marlon Becker, Rederick Altrock, and Benjamin Risse. Momentum-sam: Sharpness aware minimization without computational overhead. CoRR, 2024

work page 2024
[5]

Entropy-sgd: Biasing gradient descent into wide valleys

Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. International Conference on Learning, ICLR, 2017

work page 2017
[6]

When vision transformers outperform resnets without pre-training or strong data augmentations

Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resnets without pre-training or strong data augmentations. ICLR, 2022

work page 2022
[7]

Riemannian approach to batch normalization

Minhyung Cho and Jaehyung Lee. Riemannian approach to batch normalization. Advances in Neural Information Pro- cessing Systems, 30:5225–5235, 2017. 7

work page 2017
[8]

Sharp minima can generalize for deep nets

Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. Interna- tional Conference on Machine Learning, ICML 2017, 2017

work page 2017

Show all 53 references

[9]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

2021
[10]

Efficient sharpness-aware minimization for improved training of neural networks

Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Zhou, Liangli Zhen, Rick Goh, and Vincent Tan. Efficient sharpness-aware minimization for improved training of neural networks. In- ternational Conference on Learning Representations, 2022

2022
[11]

Gintare Karolina Dziugaite, Alexandre Drouin, Brady Neal, Nitarshan Rajkumar, Ethan Caballero, Linbo Wang, Ioannis Mitliagkas, and Daniel M. Roy. In search of robust measures of generalization. Annual Conference on Neural Information Processing Systems 2020, NeurIPS, 2020

2020
[12]

Sharpness-aware minimization for efficiently improving generalization

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. International Conference on Learning Representations, ICLR 2021 Spotlight, 2021

2021
[13]

Weight and gradi- ent centralization in deep neural networks

Wolfgang Fuhl and Enkelejda Kasneci. Weight and gradi- ent centralization in deep neural networks. Artificial Neu- ral Networks and Machine Learning – ICANN 2021, 12894: 227–239, 2019

2021
[14]

Cnn-based projected gradient descent for consistent image reconstruction

Harshit Gupta, Kyong Jin, Ha Nguyen, Michael McCann, and Michael Unser. Cnn-based projected gradient descent for consistent image reconstruction. IEEE transactions on medical imaging, 37(6):1440–1453, 2018

2018
[15]

Do sharpness-based optimizers improve generalization in medi- cal image analysis? arXiv preprint arXiv:2408.04065, 2024

Mohamed Hassan, Aleksandar Vakanski, and Min Xian. Do sharpness-based optimizers improve generalization in medi- cal image analysis? arXiv preprint arXiv:2408.04065, 2024

2024
[16]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Confer- ence on Computer Vision and Pattern Recognition, CVPR , pages 770–778, 2016

2016
[17]

Averaging weights leads to wider optima and better generalization

Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Wilson. Averaging weights leads to wider optima and better generalization. Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intel- ligence, 2018

2018
[18]

Fantastic generalization mea- sures and where to find them

Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization mea- sures and where to find them. 8th International Conference on Learning Representations, ICLR 2020, 2020

2020
[19]

Kus- ner

Jean Kaddour, Linqing Liu, Ricardo Silva, and Matt J. Kus- ner. When do flat minima optimizers work? Advances in Neural Information Processing Systems , 35:16577–16595, 2022

2022
[20]

Generalization in deep learning

Kenji Kawaguchi, Leslie Kaelbling, and Yoshua Bengio. Generalization in deep learning. Mathematical Aspects of Deep Learning, Cambridge University Press, 2022

2022
[21]

On large-batch training for deep learning: Generalization gap and sharp minima

Nitish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In- ternational Conference on Learning, ICLR, 2017

2017
[22]

Accurate image super-resolution using very deep convolutional net- works

Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional net- works. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1646–1654, 2016

2016
[23]

Fisher sam: Information geometry and sharpness aware min- imisation

Minyoung Kim, Da Li, Shell Hu, and Timothy Hospedales. Fisher sam: Information geometry and sharpness aware min- imisation. Proceedings of the 39th International Conference on Machine Learning, 162:11148–11161, 2022

2022
[24]

Adam: A method for stochastic optimization

Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014

2014 arXiv
[25]

Learning multiple layers of features from tiny images

Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 2009

2009
[26]

A simple weight decay can improve generalization

Anders Krogh and John Hertz. A simple weight decay can improve generalization. Advances in Neural Information Processing Systems, pages 950–957, 1992

1992
[27]

Anders Krogh and John A. Hertz. A simple weight decay can improve generalization. Advances in Neural Information Processing Systems, 4, 1991

1991
[28]

Asam: Adaptive sharpness-aware minimiza- tion for scale-invariant learning of deep neural networks

Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness-aware minimiza- tion for scale-invariant learning of deep neural networks. Proceedings of the 38th International Conference on Ma- chine Learning, 139:5905–5914, 2022

2022
[29]

A projected gradient descent method for crf inference allowing end-to-end training of arbitrary pairwise potentials

M ˚ans Larsson, Anurag Arnab, Fredrik Kahl, Shuai Zheng, and Philip Torr. A projected gradient descent method for crf inference allowing end-to-end training of arbitrary pairwise potentials. International Workshop on Energy Minimization Methods in Computer Vision and Pattern R...

2017
[30]

Visualizing the loss landscape of neural nets

Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Ad- vances in neural information processing systems, 2018

2018
[31]

Visualizing the loss landscape of neural nets

Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Ad- vances in Neural Information Processing Systems, NeurIPS, 2018

2018
[32]

Towards efficient and scalable sharpness-aware minimization

Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-aware minimization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022

2022
[33]

Swin trans- former: Hierarchical vision transformer using shifted win- dows

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin trans- former: Hierarchical vision transformer using shifted win- dows. IEEE/CVF International Conference on Computer 456 Vision, ICCV, pages 9992–10002, 2021

2021
[34]

Understanding the generalization benefit of normalization layers: Sharpness reduction

Kaifeng Lyu, Zhiyuan Li, and Sanjeev Arora. Understanding the generalization benefit of normalization layers: Sharpness reduction. Advances in Neural Information Processing Sys- tems, 35:34689–34708, 2022

2022
[35]

A retinal vessel segmentation method based on the sharpness-aware minimization model

Iqra Mariam, Xiaorong Xue, and Kaleb Gadson. A retinal vessel segmentation method based on the sharpness-aware minimization model. Sensors, 24(13), 2024

2024
[36]

Hover-trans: Anatomy-aware hover-transformer for roi-free breast cancer 8 diagnosis in ultrasound images

Yuhao Mo, Chu Han, Yu Liu, Min Liu, Zhenwei Shi, Jiatai Lin, Bingchao Zhao, Chunwang Huang, Bingjiang Qiu, Yan- fen Cui, Lei Wu, Xipeng Pan, Zeyan Xu, Xiaomei Huang, Zaiyi Liu, Ying Wang, and Changhong Liang. Hover-trans: Anatomy-aware hover-transformer for roi-free breast can...

2023
[37]

A method for solving a convex programming problem with convergence rate o(1/k2)

Yurii Nesterov. A method for solving a convex programming problem with convergence rate o(1/k2). 1983

1983
[38]

Un- derstanding the exploding gradient problem

Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. Un- derstanding the exploding gradient problem. CoRR, 2012

2012
[39]

On the momentum term in gradient descent learning algorithms

Ning Qian. On the momentum term in gradient descent learning algorithms. Neural networks, 12(1):145–151, 1999

1999
[40]

A benchmark for breast ultrasound image classification

Bryar Shareef, Min Xian, Shoukun Sun, Aleksandar Vakan- ski, Jianrui Ding, Chunping Ning, and A Benchmark for Breast Ultrasound Image Classification Heng-Da Cheng. A benchmark for breast ultrasound image classification. Available at SSRN: https://ssrn.com/abstract=4339660 or h...

doi
[41]

Very deep convo- lutional networks for large-scale image recognition

Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. 3rd In- ternational Conference on Learning Representations, ICLR, pages 1–14, 2015

2015
[42]

A bayesian perspective on gen- eralization and stochastic gradient descent

Samuel Smith and Quoc Le. A bayesian perspective on gen- eralization and stochastic gradient descent. International Conference on Learning, ICLR, 2017

2017
[43]

On the origin of implicit regularization in stochastic gradient descent

Samuel Smith, Benoit Dherin, David Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. International Conference on Learning Representa- tions, ICLR, 2021

2021
[44]

L2 regularization versus batch and weight normalization

Twan van Laarhoven. L2 regularization versus batch and weight normalization. arXiv:1706.05350, 2017

2017 arXiv
[45]

On orthogonality and learning recurrent networks with long term dependencies

Eugene V orontsov, Chiheb Trabelsi, Samuel Kadoury, and Chris Pal. On orthogonality and learning recurrent networks with long term dependencies. Proceedings of the 34th Inter- national Conference on Machine Learning , 70:3570–3578, 2017

2017
[46]

Cr-sam: Curva- ture regularized sharpness-aware minimization

Tao Wu, Tie Luo, and Donald Wunsch II. Cr-sam: Curva- ture regularized sharpness-aware minimization. Proceedings of the AAAI Conference on Artificial Intelligence , 38:6144– 6152, 2024

2024
[47]

Adver- sarial weight perturbation improves generalization in graph neural networks

Yihan Wu, Aleksandar Bojchevski, and Heng Huang. Adver- sarial weight perturbation improves generalization in graph neural networks. Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence , 1170:10417 – 10425, 2023

2023
[48]

Understanding and improving layer normaliza- tion

Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normaliza- tion. Advances in Neural Information Processing Systems , 32, 2019

2019
[49]

Gradient centralization: A new optimization tech- nique for deep neural networks

Hongwei Yong, Jianqiang Huang, Xiansheng Hua, and Lei Zhang. Gradient centralization: A new optimization tech- nique for deep neural networks. Computer Vision – ECCV
[50]

Understanding deep learning re- quires rethinking generalization

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning re- quires rethinking generalization. 5th International Confer- ence on Learning Representations, ICLR 2017, 2017

2017
[51]

Toward understanding the importance of noise in training neural networks

Mo Zhou, Tianyi Liu, Yan Li, Dachao Lin, Enlu Zhou, and Tuo Zhao. Toward understanding the importance of noise in training neural networks. Proceedings of the 36th Interna- tional Conference on Machine Learning, PMLR , 97:7594– 7602, 2019

2019
[52]

Surrogate gap minimization im- proves sharpness-aware training

Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha Dvornek, Sekhar Tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization im- proves sharpness-aware training. International Conference on Learning Representations, 2022. 9 A. Appendix A.1. Genera...

2022
[2020]

ECCV 2020, Springer, 2020

2020

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang

Mart ´ın Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communica- tions Security, pages 308–318, 2016

work page 2016

[2] [2]

Quantum con- volutional neural network for image classification

Mohammed Al-Yousif and Belal Al-Khateeb. Quantum con- volutional neural network for image classification. Fusion Practice and Applications, 15(2):61–72, 2024

work page 2024

[3] [3]

Sharpness-aware minimization improves language model generalization

Dara Bahri, Hossein Mobahi, and Yi Tay. Sharpness-aware minimization improves language model generalization. Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Volume 1: Long Papers, 2022

work page 2022

[4] [4]

Momentum-sam: Sharpness aware minimization without computational overhead

Marlon Becker, Rederick Altrock, and Benjamin Risse. Momentum-sam: Sharpness aware minimization without computational overhead. CoRR, 2024

work page 2024

[5] [5]

Entropy-sgd: Biasing gradient descent into wide valleys

Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. International Conference on Learning, ICLR, 2017

work page 2017

[6] [6]

When vision transformers outperform resnets without pre-training or strong data augmentations

Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resnets without pre-training or strong data augmentations. ICLR, 2022

work page 2022

[7] [7]

Riemannian approach to batch normalization

Minhyung Cho and Jaehyung Lee. Riemannian approach to batch normalization. Advances in Neural Information Pro- cessing Systems, 30:5225–5235, 2017. 7

work page 2017

[8] [8]

Sharp minima can generalize for deep nets

Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. Interna- tional Conference on Machine Learning, ICML 2017, 2017

work page 2017