Pith. sign in

REVIEW 4 major objections 5 minor 47 references

Towards Understanding the Role of Sharpness-Aware Minimization Algorithms for Out-of-Distribution Generalization

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Sharpness-aware minimizers beat Adam on shifted tasks by up to 8.01%.

desk verdict The zero-shot empirical headline is compromised by a learning-rate confound, but the GDA experiments and the eight-variant comparison are useful and the theory is honestly presented. read the letter →

arxiv 2412.05169 v1 pith:25CZO2UU submitted 2024-12-06 cs.LG cs.AI

Samuel Schapiro , Han Zhao This is my paper · ORCID
classification cs.LGcs.AI MSC 68T0768Q32
keywords sharpness-awareminimizationout-of-distributiongeneralizationgradualdomainadaptationzero-shotPAC-BayesboundsSAMvariantsdistributionshift
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sharpness-aware minimization (SAM) trains models to be stable under small adversarial weight perturbations, and this paper asks whether that stability pays off when the test distribution differs from training. Across four benchmarks, it reports that SAM beats the Adam baseline by 4.76% average zero-shot out-of-distribution accuracy, and the strongest SAM variants beat Adam by 8.01%; in gradual domain adaptation, where the model self-trains across intermediate domains, the reported gains are 0.82% and 1.52%. The paper also proves sharpness-aware bounds for both settings, bounding target error by source error plus sharpness, parameter drift, Wasserstein distance, and sample-complexity terms. The GDA bound is asymptotically no better than existing self-training bounds, so the theory does not yet explain why SAM helps; the empirical claim, if correct, would make SAM and its variants a simple, practical way to improve accuracy under distribution shift.

What carries the argument

The machinery is the sharpness-aware objective: instead of minimizing the loss at the current parameters, SAM seeks parameters whose worst-case loss over an $\ell^2$ ball of radius $\rho$ is minimized, $\theta^\star = \arg\min_{\theta} \max_{\|\beta\|\le\rho} E(\theta+\beta)$. The paper's theoretical engine is a "sharpness-aware error difference" lemma: for two distributions $\mu,\nu$ and two models $\theta_\mu,\theta_\nu$, $|E^\rho_\mu(\theta_\mu)-E_\nu(\theta_\nu)| \le S_\rho(\theta_\mu) + O(\|\theta_\mu-\theta_\nu\| + W_p(\mu,\nu))$, where $S_\rho$ is the $\rho$-sharpness. Combined with the PAC-Bayes bound, this lemma converts flatness into a distribution-shift guarantee; the GDA theorem applies it domain-by-domain along the intermediate chain, using the discrepancy measure and sequential Rademacher complexity from the online-learning view of self-training.

What would settle it

Train Adam and SAM on the four benchmarks with identical learning rates, schedules, and weight decays; if the 4.76% and 8.01% average margins shrink to noise, the reported advantage is a hyperparameter artifact rather than an effect of sharpness-aware minimization.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that minimizing flatness at train time transfers to new domains: the original SAM improves over Adam on every dataset considered for both zero-shot OOD generalization and gradual domain adaptation, and several SAM variants improve further. The paper reports exact numbers ($4.76\%$ zero-shot, $8.01\%$ for the strongest variants, $0.82\%$ GDA, $1.52\%$ for the strongest variants) and proposes that FisherSAM's use of an approximate Fisher/Hessian perturbation under cross-entropy loss may explain its zero-shot gains, while FriendlySAM's adaptive perturbation robustness may explain its GDA gains. The matching theoretical contribution is a pair of generalization bounds that place sharpness alongside distribution distance and parameter distance as the quantities governing OOD error, with the GDA bound matching the asymptotic rate of the standard self-training bound.

Load-bearing premise

The empirical comparison treats Adam as a fair control, but SAM was trained with a learning rate of $10^{-2}$ while Adam used $10^{-3}$, so the accuracy gap could come from the larger step size rather than from the sharpness-aware perturbation.

Editorial extensions

If this is right

  • Using SAM in place of Adam is a drop-in optimizer change that improved zero-shot OOD accuracy on all four benchmarks.
  • Stronger variants such as FisherSAM and FriendlySAM can add further gains, while cheaper variants like LookSAM and NoSAM still often match or beat original SAM.
  • The sharpness-aware bounds give a quantitative decomposition of OOD error into sharpness, parameter drift, and distribution distance, suggesting each is a separate lever for improving shifted-domain accuracy.
  • In gradual domain adaptation, SAM's consistent gain means the self-training loop can be improved during optimization without changing pseudo-labeling or architecture.
  • Because the GDA bound has the same asymptotic rate as standard self-training, any satisfactory theory of SAM's OOD advantage must go beyond flatness alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to run SAM and Adam with identical learning rates, schedules, and weight decays; if the headline margins shrink to noise, the reported gains are a hyperparameter artifact rather than an effect of sharpness.
  • FisherSAM's use of an approximate Fisher/Hessian perturbation is the paper's own explanation for its strong zero-shot results, so probing whether this variant scales to larger, more realistic distribution shifts would directly test that mechanism.
  • The sharpness-aware bound suggests a common yardstick for comparing flat-minima optimizers on OOD benchmarks: report sharpness, parameter drift, and Wasserstein shift alongside accuracy, and test whether lowering each component lowers target error.
  • The authors' conjecture of a localized analysis implies a concrete check: measure feature rank, denoising, and feature balance on the four benchmarks to see whether SAM's OOD benefit tracks those quantities rather than final flatness.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper studies sharpness-aware minimization (SAM) and eight of its variants in two out-of-distribution settings: zero-shot OOD generalization and gradual domain adaptation (GDA). On four datasets (Rotated MNIST, Color MNIST, Covertype, Portraits), it reports that SAM improves over an Adam baseline by 4.76% on average in the zero-shot setting and by 0.82% in GDA, with the strongest variants improving by 8.01% and 1.52% (abstract) or 1.42% (body). The theoretical part provides a sharpness-aware error-difference lemma, an OOD generalization bound (Theorem 1), and a GDA bound (Theorem 2); the authors state that the GDA bound is asymptotically no better than the existing bound of Wang et al. (2022). The paper is candid about this limitation and points to several directions for tighter analysis.

Significance. If the empirical claims were established, the paper would offer a useful practical comparison of SAM variants under distribution shift and a reasonable starting point for sharpness-based OOD theory. The strengths are the breadth of the variant survey, the computational-cost comparison in Table 1, the explicit proof appendix, and the honest discussion of the gap between theory and experiments. However, the zero-shot empirical headline is undermined by two confounds: SAM and the Adam baseline use different learning rates, and the reported accuracies are selected as the best over a hyperparameter grid on the test set. The theoretical contribution is modest because it reuses Foret et al.'s PAC-Bayes bound and Wang et al.'s discrepancy machinery, and the paper itself concedes that the GDA bound has no asymptotic advantage over prior work. Still, the derivations are mostly sound and the paper is clearly written; the empirical claims can in principle be repaired with better-controlled experiments.

major comments (4)
  1. [Section 3.1 (Model Setup), Table 2] The zero-shot comparison is confounded by learning rate: SAM uses Adam with learning rate 1e-2, while the Adam baseline uses learning rate 1e-3, and no learning-rate sweep is reported for Adam. Because the perturbation radius and the learning rate are never varied independently, the reported 4.76% and 8.01% average gains cannot be attributed to sharpness-aware minimization; they may be produced entirely by the tenfold larger learning rate. Please rerun the Adam baseline at the same learning rate as SAM, and ideally sweep the learning rate for both optimizers, before claiming that SAM improves zero-shot OOD generalization.
  2. [Appendix B, Table 2] The zero-shot results report the best accuracy over the rho grid and other variant-specific hyperparameters for each dataset, which is equivalent to selecting hyperparameters on the test set. This favors SAM variants relative to an Adam baseline that is not given the same selection procedure, and it makes the reported gains difficult to interpret. Please either fix the hyperparameters a priori or choose them on a validation split, and report the selected values for every optimizer and dataset.
  3. [Section 4.2, Table 3] For the GDA variant comparison, the number of intermediate domains T is chosen as the optimal T for SAM from Figure 1 and then used for all optimizers. If T is selected to favor SAM, the claim that SAM and FriendlySAM consistently outperform Adam in GDA is biased. Please compare over the full range of T, or choose T by a criterion that does not depend on the optimizer being evaluated.
  4. [Lemma 1, Appendix A.1] In the proof of Lemma 1, the symbol rho is redefined as the maximum Lipschitz constant, while in the statement of the lemma and in S_rho(theta_mu) it denotes the SAM perturbation radius. As written, the proof establishes the bound with sharpness measured at the Lipschitz-constant radius rather than at the algorithm's perturbation radius, which makes the sharpness term in Theorem 1 ambiguous. Please use a distinct symbol, such as L, for the Lipschitz constant and retain rho for the perturbation radius throughout.
minor comments (5)
  1. [Abstract, Section 4.3, Conclusion] The abstract and Section 1 state that the strongest SAM variants improve over Adam by 1.52% in GDA, while Section 4.3 and the Conclusion report 1.42%; please reconcile these numbers.
  2. [Appendix B] The sentence 'In Table 1, we report the best accuracy values obtained over all hyperparameter settings' should refer to Table 2, which contains the zero-shot accuracy results.
  3. [Section 1] There is a typo in the introduction: 'SAM remains remains understudied' should read 'SAM remains understudied.'
  4. [Lemma 2] Lemma 2 states an extra condition E(theta) <= E_{epsilon~N(0,rho^2 I)}[E(theta+epsilon)] that is not part of the standard Foret et al. PAC-Bayes bound; if this condition is needed, it should be stated as an explicit assumption in Theorem 1 and Theorem 2 and justified for the iterates of Algorithm 2 rather than only in a footnote.
  5. [Figure 1] Figure 1 does not show error bars or the number of seeds used; please add this information or state explicitly that the curves are single runs.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the sharpness-aware bounds are transparent decompositions built from cited PAC-Bayes and Rademacher lemmas, not from the paper's own empirical outputs; the main weakness is a learning-rate confound in the zero-shot experiment, which is an experimental-design issue rather than a circular derivation.

full rationale

The paper's theoretical results are assembled from independent published ingredients. Lemma 2 is the PAC-Bayes sharpness bound of Foret et al. (2021), Lemma 4 is the Rademacher generalization bound of Kumar et al. (2020), and Lemma 3 and Proposition 1 are taken from Wang et al. (2022). The sharpness term S_rho appears in Lemma 1 by adding and subtracting E_mu(theta_mu) and using the definition S_rho = E_rho - E; this is a valid decomposition rather than an import of the desired conclusion. Theorem 1 then combines these inequalities, and Theorem 2 applies Theorem 1 successively inside the GDA framework of Wang et al. (2022). Although Wang et al. is co-authored by one of the current authors, it is a prior published work with independent proofs, and the paper explicitly states that its GDA bound is asymptotically no better than Wang et al.'s, so the self-citation is not used to force the empirical claims. No uniqueness theorem or ansatz is smuggled in through self-citation; Section 4.6's 'only way' remark is conditional on the adopted discrepancy/sequential-Rademacher framework. The zero-shot empirical comparison does confound optimizer choice with learning rate: the text says SAM uses Adam with learning rate 10^-2 while the Adam baseline uses 10^-3, so the reported 4.76% and 8.01% gains may reflect a learning-rate artifact rather than sharpness-aware perturbation. That is a serious experimental criticism, but it is not circularity under the definitions used here because no fitted parameter is renamed as a prediction and no equation reduces to its own input. Overall, the derivation chain is self-contained with respect to its cited lemmas, and the empirical comparison flaw does not make the argument circular.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central empirical claims depend on several free hyperparameters selected by grid search per dataset, on two background assumptions about the loss and model class, and on the PAC-Bayes local-minimum condition. No new physical or mathematical entities are introduced.

free parameters (7)
  • Perturbation radius rho for SAM and variants = Grid {0.01, 0.02, 0.05, 0.1, 0.2}; ASAM uses {0.1, 0.2, 0.5, 1, 2}
    The paper reports best accuracy over this grid per dataset, so the reported SAM gains depend on test-set selection of rho (Appendix B).
  • SAM base learning rate = 1e-2
    Chosen for SAM while the Adam baseline uses 1e-3; this confound affects the zero-shot comparison (Section 3.1).
  • FisherSAM eta = Grid {0.01, 0.2, 0.5, 1}
    Variant hyperparameter selected per dataset to maximize test accuracy.
  • LookSAM alpha = Grid {0.5, 0.7, 0.1}
    Variant hyperparameter selected per dataset to maximize test accuracy.
  • FriendlySAM EMA momentum phi = Grid {0.6, 0.9, 0.95}
    Variant hyperparameter selected per dataset to maximize test accuracy.
  • ESAM stochastic weight parameter xi = Grid {0.5, 0.6}
    Variant hyperparameter selected per dataset to maximize test accuracy.
  • Number of intermediate domains T in GDA variant comparison = T=50 (Rotated MNIST), 10 (Color MNIST), 25 (Covertype), 10 (Portraits)
    Chosen as optimal for SAM from Figure 1, then used for all variants; variants might prefer a different T (Section 4.2).
assumptions (6)
  • domain assumption Lipschitz loss (Assumption 1): the loss is Lipschitz in label, input, and parameters with constants rho1, rho2, rho3.
    Used throughout Lemma 1 and Theorem 1 to bound error differences by Wasserstein distance and parameter distance (Section 3.3).
  • domain assumption Bounded Rademacher complexity (Assumption 2): there exists B such that R_mu(Theta) <= B/sqrt(n) for all distributions mu.
    Used in Lemma 4 to control the gap between empirical and population error in the proof of Theorem 1 (Appendix A.2).
  • ad hoc to paper PAC-Bayes local-minimum condition: E(theta) <= E_{epsilon~N(0,rho^2 I)}[E(theta+epsilon)] for the solutions theta_t.
    Assumed in Lemma 2 and Theorems 1 and 2; the paper says this 'should hold for local minima' but does not verify it for the trained networks.
  • standard math Foret et al. (2021) Lemma 2: PAC-Bayesian sharpness-aware generalization bound.
    Imported as an unproved background result and applied successively in the GDA analysis (Appendix A.3).
  • standard math Wang et al. (2022) Lemma 1 and Proposition 1: standard error difference over shifted domains and discrepancy bound.
    Used as black-box lemmas in the proof of Theorem 2 (Appendix A.5).
  • domain assumption GDA gradual shift assumption: consecutive domains have small average Wasserstein distance Delta.
    Defines the GDA setting and enters the bound as T*Delta (Definition 4 and Theorem 2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Understanding the Role of Sharpness-Aware Minimization Algorithms for Out-of-Distribution Generalization." pith.science (2026). https://pith.science/paper/25CZO2UU

@misc{pith2026241205169,
  author       = {Pith},
  title        = {Pith review of: Towards Understanding the Role of Sharpness-Aware Minimization Algorithms for Out-of-Distribution Generalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/25CZO2UU}},
  note         = {Machine review of arXiv:2412.05169}
}
abstract

Recently, sharpness-aware minimization (SAM) has emerged as a promising method to improve generalization by minimizing sharpness, which is known to correlate well with generalization ability. Since the original proposal of SAM, many variants of SAM have been proposed to improve its accuracy and efficiency, but comparisons have mainly been restricted to the i.i.d. setting. In this paper we study SAM for out-of-distribution (OOD) generalization. First, we perform a comprehensive comparison of eight SAM variants on zero-shot OOD generalization, finding that the original SAM outperforms the Adam baseline by $4.76\%$ and the strongest SAM variants outperform the Adam baseline by $8.01\%$ on average. We then provide an OOD generalization bound in terms of sharpness for this setting. Next, we extend our study of SAM to the related setting of gradual domain adaptation (GDA), another form of OOD generalization where intermediate domains are constructed between the source and target domains, and iterative self-training is done on intermediate domains, to improve the overall target domain error. In this setting, our experimental results demonstrate that the original SAM outperforms the baseline of Adam on each of the experimental datasets by $0.82\%$ on average and the strongest SAM variants outperform Adam by $1.52\%$ on average. We then provide a generalization bound for SAM in the GDA setting. Asymptotically, this generalization bound is no better than the one for self-training in the literature of GDA. This highlights a further disconnection between the theoretical justification for SAM versus its empirical performance, with recent work finding that low sharpness alone does not account for all of SAM's generalization benefits. For future work, we provide several potential avenues for obtaining a tighter analysis for SAM in the OOD setting.

Figures

Figures reproduced from arXiv: 2412.05169 by the authors.

Figure 1
Figure 1. A comparison of SAM with varying levels of perturbation strength [PITH_FULL_IMAGE:figures/full_fig_p012_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 33 canonical work pages

  1. [1]

    Sharpness-aware minimization leads to low-rank features

    Maksym Andriushchenko, Dara Bahri, Hossein Mobahi, and Nicolas Flammarion. Sharpness-aware minimization leads to low-rank features. Advances in Neural Information Processing Systems, 36: 0 47032--47051, 2023

    work page 2023

  2. [2]

    Invariant risk minimization, 2020

    Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization, 2020

    work page 2020

  3. [3]

    Why is sam robust to label noise? In International Conference on Learning Representations (ICLR), 2024

    Christina Baek, Zico Kolter, and Aditi Raghunathan. Why is sam robust to label noise? In International Conference on Learning Representations (ICLR), 2024

    work page 2024

  4. [4]

    A theory of learning from different domains

    Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Vaughan. A theory of learning from different domains. Machine Learning, 79: 0 151--175, 05 2010. doi:10.1007/s10994-009-5152-4

    doi:10.1007/s10994-009-5152-4 2010

  5. [5]

    Blackard and Denis J

    Jock A. Blackard and Denis J. Dean. Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables. Computers and Electronics in Agriculture, 24: 0 131--151, 1999. URL https://api.semanticscholar.org/CorpusID:13985407

    work page 1999

  6. [6]

    Optimization methods for large-scale machine learning

    L \'e on Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM review, 60 0 (2): 0 223--311, 2018

    2018

  7. [7]

    Swad: Domain generalization by seeking flat minima

    Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34: 0 22405--22418, 2021

    2021

  8. [8]

    Entropy-sgd: Biasing gradient descent into wide valleys

    Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019 0 (12): 0 124018, 2019

    2019

Show all 47 references
  1. [9]

    Why does sharpness-aware minimization generalize better than sgd? Advances in neural information processing systems, 36, 2024

    Zixiang Chen, Junkai Zhang, Yiwen Kou, Xiangning Chen, Cho-Jui Hsieh, and Quanquan Gu. Why does sharpness-aware minimization generalize better than sgd? Advances in neural information processing systems, 36, 2024

    2024

  2. [10]

    Sharp minima can generalize for deep nets

    Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp.\ 1...

    2017

  3. [11]

    Efficient sharpness-aware minimization for improved training of neural networks

    Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent YF Tan. Efficient sharpness-aware minimization for improved training of neural networks. In International Conference on Learning Representations (ICLR), 2022

    2022

  4. [12]

    Sharpness-aware minimization for efficiently improving generalization

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations (ICLR), 2021

    2021

  5. [13]

    Sachs, Brian Yin, Crystal Lee, Philipp Krähenbühl, and Alexei A

    Shiry Ginosar, Kate Rakelly, Sarah M. Sachs, Brian Yin, Crystal Lee, Philipp Krähenbühl, and Alexei A. Efros. A century of portraits: A visual historical record of american high school yearbooks. IEEE Transactions on Computational Imaging, 3 0 (3): 0 421--431, 2017. doi:10.110...

    2017

  6. [14]

    Gradual domain adaptation: Theory and algorithms

    Yifei He, Haoxiang Wang, Bo Li, and Han Zhao. Gradual domain adaptation: Theory and algorithms. In Journal of Machine Learning Research, 2024

    2024

  7. [15]

    Flat Minima

    Sepp Hochreiter and Jürgen Schmidhuber. Flat Minima . Neural Computation, 9 0 (1): 0 1--42, 01 1997. ISSN 0899-7667. doi:10.1162/neco.1997.9.1.1. URL https://doi.org/10.1162/neco.1997.9.1.1

    1997 doi

  8. [16]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015

    2015

  9. [17]

    Averaging weights leads to wider optima and better generalization

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In Conference on Uncertainty in Artificial Intelligence, 2018

    2018

  10. [18]

    Fantastic generalization measures and where to find them, 2020

    Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them, 2020

    2020

  11. [19]

    When do flat minima optimizers work? Advances in Neural Information Processing Systems, 35: 0 16577--16595, 2022

    Jean Kaddour, Linqing Liu, Ricardo Silva, and Matt J Kusner. When do flat minima optimizers work? Advances in Neural Information Processing Systems, 35: 0 16577--16595, 2022

    2022

  12. [20]

    On large-batch training for deep learning: Generalization gap and sharp minima

    Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations (ICLR), 2017

    2017

  13. [21]

    Fast and scalable bayesian deep learning by weight-perturbation in adam

    Mohammad Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava. Fast and scalable bayesian deep learning by weight-perturbation in adam. In International conference on machine learning, pp.\ 2611--2620. PMLR, 2018

    2018

  14. [22]

    F isher SAM : Information geometry and sharpness aware minimisation

    Minyoung Kim, Da Li, Shell X Hu, and Timothy Hospedales. F isher SAM : Information geometry and sharpness aware minimisation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference...

    2022

  15. [23]

    Understanding self-training for gradual domain adaptation

    Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self-training for gradual domain adaptation. In International conference on machine learning, pp.\ 5468--5479. PMLR, 2020

    2020

  16. [24]

    Kuznetsov and M

    V. Kuznetsov and M. Mohri. Discrepancy-based theory and algorithms for forecasting non-stationary time series. Annals of Mathematics and Artificial Intelligence, 2020 a . URL https://doi.org/10.1007/s10472-019-09683-1

    2020 doi

  17. [25]

    Discrepancy-based theory and algorithms for forecasting non-stationary time series, 2020 b

    Vitaly Kuznetsov and Mehryar Mohri. Discrepancy-based theory and algorithms for forecasting non-stationary time series, 2020 b

    2020

  18. [26]

    Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks

    Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning, pp.\ 5905--5914. PMLR, 2021

    2021

  19. [27]

    The mnist database of handwritten digits

    Yann LeCun and Corinna Cortes. The mnist database of handwritten digits. 1998. URL https://yann.lecun.com/exdb/mnist/

    1998

  20. [28]

    Friendly sharpness-aware minimization

    Tao Li, Pan Zhou, Zhengbao He, Xinwen Cheng, and Xiaolin Huang. Friendly sharpness-aware minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 5631--5640, 2024

    2024

  21. [29]

    Fisher-rao metric, geometry, and complexity of neural networks

    Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metric, geometry, and complexity of neural networks. In The 22nd international conference on artificial intelligence and statistics, pp.\ 888--896. PMLR, 2019

    2019

  22. [30]

    Towards efficient and scalable sharpness-aware minimization

    Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-aware minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 12360--12370, 2022

    2022

  23. [31]

    New insights and perspectives on the natural gradient method

    James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21 0 (146): 0 1--76, 2020. URL http://jmlr.org/papers/v21/17-678.html

    2020

  24. [32]

    Normalization layers are all that sharpness-aware minimization needs

    Maximilian Mueller, Tiffany Vlaar, David Rolnick, and Matthias Hein. Normalization layers are all that sharpness-aware minimization needs. Advances in Neural Information Processing Systems, 36, 2024

    2024

  25. [33]

    K-sam: Sharpness-aware minimization at the speed of sgd, 2022

    Renkun Ni, Ping yeh Chiang, Jonas Geiping, Micah Goldblum, Andrew Gordon Wilson, and Tom Goldstein. K-sam: Sharpness-aware minimization at the speed of sgd, 2022. URL https://arxiv.org/abs/2210.12864

    2022 arXiv

  26. [34]

    Online learning via sequential complexities

    Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Online learning via sequential complexities. Journal of Machine Learning Research, 16 0 (6): 0 155--186, 2015

    2015

  27. [35]

    David Samuel. sam. https://github.com/davda54/sam, 2020

    2020

  28. [36]

    Sharpness-aware minimization enhances feature quality via balanced learning

    Jacob Mitchell Springer, Vaishnavh Nagarajan, and Aditi Raghunathan. Sharpness-aware minimization enhances feature quality via balanced learning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=3xDaj4pRna

    2024

  29. [37]

    Dropout: A simple way to prevent neural networks from overfitting

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15 0 (56): 0 1929--1958, 2014. URL http://jmlr.org/papers/v15/srivastava14a.html

    1929

  30. [38]

    Understanding gradual domain adaptation: Improved analysis, optimal path and beyond

    Haoxiang Wang, Bo Li, and Han Zhao. Understanding gradual domain adaptation: Improved analysis, optimal path and beyond. In International Conference on Machine Learning, pp.\ 22784--22801. PMLR, 2022

    2022

  31. [39]

    Sharpness minimization algorithms do not only minimize sharpness to achieve better generalization

    Kaiyue Wen, Zhiyuan Li, and Tengyu Ma. Sharpness minimization algorithms do not only minimize sharpness to achieve better generalization. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36...

    2023

  32. [40]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing infe...

    2022

  33. [41]

    Towards a theoretical framework of out-of-distribution generalization

    Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. Towards a theoretical framework of out-of-distribution generalization. Advances in Neural Information Processing Systems, 34: 0 23519--23531, 2021

    2021

  34. [42]

    Flatness-aware minimization for domain generalization

    Xingxuan Zhang, Renzhe Xu, Han Yu, Yancheng Dong, Pengfei Tian, and Peng Cui. Flatness-aware minimization for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 5189--5202, 2023

    2023

  35. [43]

    On learning invariant representations for domain adaptation

    Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International conference on machine learning, pp.\ 7523--7532. PMLR, 2019

    2019

  36. [44]

    Fundamental limits and tradeoffs in invariant representation learning

    Han Zhao, Chen Dan, Bryon Aragam, Tommi S Jaakkola, Geoffrey J Gordon, and Pradeep Ravikumar. Fundamental limits and tradeoffs in invariant representation learning. Journal of machine learning research, 23 0 (340): 0 1--49, 2022

    2022

  37. [45]

    Gradual domain adaptation via gradient flow

    Zhan Zhuang, Yu Zhang, and Ying Wei. Gradual domain adaptation via gradient flow. In The Twelfth International Conference on Learning Representations, 2024

    2024

  38. [46]

    Towards robust out-of-distribution generalization bounds via sharpness

    Yingtian Zou, Kenji Kawaguchi, Yingnan Liu, Jiashuo Liu, Mong-Li Lee, and Wynne Hsu. Towards robust out-of-distribution generalization bounds via sharpness. In International Conference on Learning Representations (ICLR), 2024

    2024

  39. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.