Pith. sign in

REVIEW 3 major objections 6 minor 40 references

This paper shows that the inner perturbation geometry in Sharpness-Aware Minimization interacts with the outer optimizer: a layerwise spectral perturbation paired with Muon achieves the best ImageNet validation accuracy on ViT-Small/16 and

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 00:52 UTC pith:JR4FZD3V

load-bearing objection A clean, honest paper whose headline interaction is plausible but weakened by tuning on the same validation split and uneven search grids. the 3 major comments →

arxiv 2607.26001 v1 pith:JR4FZD3V submitted 2026-07-28 cs.LG stat.ML

Sharpness-Aware Minimization and Muon: Robustness under the Spectral Norm

Wenzhi Zhong , Edward Milsom , Michael Murray This is my paper
classification cs.LG stat.ML
keywords sharpness-aware minimizationspectral normMuon optimizermatrix-aware optimizationImageNet-1Kflat minimageneralizationlayerwise perturbation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether the geometry used inside Sharpness-Aware Minimization should be matched to the geometry of the outer optimizer. It introduces a layerwise spectral inner perturbation for matrix-valued weights, in which each layer is perturbed along a steepest-ascent direction under the spectral norm, and pairs it with either AdamW/SGDW or Muon. Across ImageNet-1K on ViT-Small/16 and ResNet-50, the spectral step gives only a modest gain with AdamW/SGDW but a large gain with Muon, reaching the best validation accuracy on both models (80.23 and 78.55). The paper argues that inner and outer geometries interact: Euclidean perturbations add a nearly constant gain regardless of outer optimizer, while spectral perturbations selectively benefit Muon. A reader should care because it suggests a practical recipe — spectral sharpness plus matrix-aware updates — and raises a question about why the two geometries couple.

Core claim

The paper claims that sharpness-aware training is geometry-dependent in a specific, exploitable way: replacing the flattened Euclidean inner perturbation with a layerwise spectral-norm perturbation, and pairing it with the Muon outer update, yields the best validation Top-1 on both ImageNet models tested. The claimed mechanism is an interaction between inner and outer geometries: a Euclidean inner step shifts both outer optimizers by roughly the same amount, whereas the spectral inner step amplifies Muon's advantage while barely helping (or slightly hurting) AdamW/SGDW. The paper's strongest numerical claim is that SPECSAM-MUON reaches 80.23 on ViT-Small/16 and 78.55 on ResNet-50, beating al

What carries the argument

The central object is the layerwise spectral inner perturbation: for each matrix-valued parameter block, the perturbation is the polar factor Ortho(G)=U_r V_r^T of the block gradient, a steepest-ascent direction under the spectral norm at unit radius (approximated by Newton–Schulz iterations in practice), while non-matrix parameters are perturbed along the normalized Euclidean gradient with a separate radius. The outer step is then a Muon update: momentum followed by approximate orthogonalization, applied only to matrix parameters. The machinery's role is to make both the sharpness probe and the descent direction respect the matrix structure of the weights, and the paper claims that this geo

Load-bearing premise

The central ranking rests on the assumption that hyperparameters selected by validation accuracy at half the training budget transfer correctly to the full 120-300 epoch runs; if the reduced-budget tuning favors different radius and learning-rate combinations for different methods, the spectral-plus-Muon advantage could be a selection artifact rather than a property of the geometries.

What would settle it

Train ResNet-50 and ViT-Small/16 on ImageNet with full-budget hyperparameter search for each method, or evaluate the final checkpoints on a held-out split such as ImageNet-v2 instead of the tuning validation split. If SPECSAM-MUON no longer beats SAM-MUON or SAM-AdamW/SGDW under full-budget tuning or on unseen data, the claimed spectral-inner/outer-coupling is not established.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, SAM-style training on vision models should use Muon as the outer optimizer together with a spectral inner perturbation; the gain over SAM-AdamW is over one point on ViT-Small/16 and over half a point on ResNet-50.
  • The additive behavior of Euclidean perturbations means that under standard SAM, switching from AdamW/SGDW to Muon does not change the benefits of sharpness-aware training; the spectral geometry is what unlocks Muon's advantage.
  • Since spectral perturbation lowers train accuracy relative to Euclidean variants while raising validation accuracy, the method appears to act as a stronger regularizer, consistent with sharpness control rather than pure fitting.
  • The interaction is observed on both a transformer and a CNN, suggesting it may transfer to other matrix-valued network components, though the paper does not test beyond image classification.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's explanation is empirical; a plausible mechanism left implicit is that Muon maintains higher effective rank of weights and features, giving it capacity to absorb spectral regularization without losing representational power.
  • The two-stage tuning protocol means part of the gain could come from the validation-based selection; a natural test is to evaluate on a held-out set or with full-budget tuning to confirm the interaction is not a selection artifact.
  • The result might extend to language modeling, where Muon is most widely used, but the paper does not test it; if the same coupling holds, spectral SAM could become a drop-in for LLM training.
  • The layerwise r=∞ aggregation and separate radii add hyperparameters; a simpler version with a single radius might capture most of the benefit, but the paper does not compare that simplification.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a norm-agnostic SAM template in which the inner perturbation norm and the outer optimizer are treated as independent choices. It introduces SPECSAM, a layerwise spectral-norm inner perturbation for matrix-valued parameters using r=infinity block aggregation and polar-factor (Newton-Schulz approximated) ascent, with a separate Euclidean radius for non-matrix parameters. The outer step is either AdamW/SGDW or Muon. On ImageNet-1K (ViT-Small/16, ResNet-50) the authors report that Euclidean SAM adds a roughly additive gain regardless of outer optimizer, whereas the spectral inner step selectively benefits Muon, with SPECSAM-MUON achieving the best validation accuracy (80.23 and 78.55). LAYERSAM controls for layerwise budgeting, and CIFAR-100 diagnostics examine sharpness, effective rank, and radius sensitivity.

Significance. The theoretical sections are a genuine strength: Eq. (3) correctly gives the first-order robust loss via the dual norm, Eq. (7)-(8) correctly derive the layerwise r=infinity spectral perturbation, and the polar-factor subgradient interpretation is appropriate. The LAYERSAM ablation in Table 11 is a good control for layerwise normalization, and the two-stage tuning protocol does control for base learning rate and weight decay. Three seeds plus ImageNet-ReaL/ImageNet-R attempts provide some robustness evidence. If the headline inner/outer geometry interaction is real, the paper offers a useful and transferable framework. However, the central empirical claim is currently under-supported because the same validation split is used for hyperparameter selection, checkpoint selection, and final reporting; see Major Comment 1.

major comments (3)
  1. [Appendix A.3 and A.5; Section 4.1] The central empirical claim that the spectral inner step selectively benefits Muon and that SPECSAM-MUON is best on both architectures rests on a selection protocol that can manufacture exactly this pattern. A.5 states that all hyperparameter searches use half the training budget and that final configurations are selected by validation Top-1 at those reduced budgets; A.3 states that all reported metrics are evaluated at the best-validation checkpoint. Thus the same ImageNet validation split is used to pick LR/WD, to pick rho/rho_m/rho_n, and to select the final checkpoint. The search grids are also unbalanced: for ViT, SAM variants search 4 rho values while SPECSAM variants search 4 rho_m x 6 rho_n = 24 combinations (Tables 5-6); ResNet has a similar imbalance. With selection on the evaluation split, a larger grid is more likely to land on a high-validation configuration by chance, and t
  2. [Section 4.1 and Appendix A.4] The 'spectral inner step selectively benefits Muon' comparison is not fully controlled for parameter routing. SPECSAM perturbs all matrix-valued parameters, including classification head, patch embeddings, and low-dimension convolutions, whereas the Muon outer group updates only hidden-layer backbone matrices; the remaining parameters are updated by AdamW/SGDW. Thus the inner perturbation is not applied to the same parameter set across the two outer optimizers. The LAYERSAM ablation in Table 11 controls for layerwise versus spectral geometry but not for this routing mismatch. Please report a version where the spectral inner perturbation is restricted to exactly the Muon group, or where the Muon group is expanded to all perturbed matrices.
  3. [Section 4.1, Tables 2 and 7] The claimed interaction is inferred from single selected configurations rather than from a matched-radius comparison on ImageNet. On ViT, SAM-MUON vs SAM-ADAMW (79.97 vs 79.17) and SPECSAM-MUON vs SPECSAM-ADAMW (80.23 vs 78.71) use different selected radii, chosen on the same validation split. The CIFAR-100 sweep in Figure 2 shows non-monotone radius sensitivity and suggests that a method's rank can change with rho. A matched-radius comparison, or the full ImageNet grid over rho/rho_m/rho_n, is needed before attributing the difference to geometry rather than to radius selection.
minor comments (6)
  1. [Section 2.1, Eq. (4)] The notation epsilon* is used in Eq. (4) before its formal definition and is non-unique for the spectral norm. Please state explicitly that any element of the subdifferential may be used, or fix a measurable selection.
  2. [Appendix A.4] The Moonlight-style update scaling for Muon is mentioned but no formula or coefficient schedule is given. Since the outer update scale directly affects the comparison with AdamW/SGDW, please provide the exact scaling used.
  3. [Eq. (9), Appendix A.4] Please specify the Newton-Schulz coefficient schedule and whether the NSk output is renormalized to unit spectral norm before forming the perturbation; this determines the effective radius.
  4. [Table 2] The Welch t-test bolding is computed on the same validation split used for selection. The caption and Section 4.1 should state that these p-values do not account for selection, and the phrase 'no statistically significant difference' should be treated as exploratory.
  5. [Conclusion] The phrase 'best validation accuracy' should be qualified as 'best among the methods and configurations evaluated under this protocol', since the paper does not claim a global optimum over all possible SAM variants.
  6. [References] Reference [39] is a workshop poster; please verify that the description of Spectral-SAM is accurate and appropriately caveated, and consider citing the full version if one exists.

Circularity Check

0 steps flagged

No significant circularity: spectral perturbation follows from dual-norm convex analysis; empirical claims are comparisons, not fitted predictions.

full rationale

The paper's derivation chain is self-contained: Eq. (8) is obtained from the norm-agnostic SAM template by computing the subdifferential of the dual norm, with Ortho(G) identified as a nuclear-norm subgradient (Eq. (5), citing Bhatia's standard duality result). This is a mathematical consequence of the definitions, not an assumption of the empirical conclusion. Muon is an existing external optimizer, and the paper makes no claim to derive its behavior from its own equations. The headline claim—that SPECSAM-MUON achieves the best validation accuracy—is an empirical comparison, not a quantity fitted to and then read back from the same data. The paper explicitly leaves the interaction mechanism unexplained ('Why the two geometries should interact this way is not settled by our experiments'), so no result is being presented as derived from its own premises. The main methodological caveat, stated in Appendix A.5, is that hyperparameters are selected by validation Top-1 at half budget and final metrics are reported on the same validation split at the best-validation checkpoint; this is a model-selection/statistical-validity concern (with unbalanced search-grid sizes across methods), but it does not make any stated equation or prediction equivalent to its inputs by construction. There are no load-bearing self-citations and no fitted parameter is relabeled as a prediction.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 0 invented entities

No physical entities are postulated. The theoretical skeleton is standard convex analysis; the cost is paid in tuned hyperparameters (radii, learning rates, Newton-Schulz counts) and in design choices such as r=infinity aggregation and the separate non-matrix radius alpha_g.

free parameters (6)
  • Euclidean SAM radius rho (SAM-* variants) = ViT-Small/16: 0.458 (SAM-AdamW), 0.7 (SAM-Muon); ResNet-50: 0.05 (SAM-SGDW), 0.2 (SAM-Muon)
    Selected by validation Top-1 on half-budget runs (Appendix A.5, Tables 7-8).
  • Spectral matrix radius rho_m (SPECSAM variants) = ViT: 6.56e-4 (SpecSAM-AdamW), 1.00e-3 (SpecSAM-Muon); ResNet: 5e-4 (SpecSAM-SGDW), 2e-2 (SpecSAM-Muon)
    Selected by the same validation protocol; the radii differ by orders of magnitude across methods, so tuning is load-bearing.
  • Non-matrix perturbation radius rho_n / alpha_g = ViT: 0 (SpecSAM-AdamW), 3.16e-3 (SpecSAM-Muon); ResNet: 1e-3, 3.16e-4
    alpha_g in Eq. (9) is set through rho_n; the ablation in Appendix B.1 shows no consistent benefit, but it is part of the tuned method.
  • Base learning rates and weight decays = ViT: AdamW LR 3e-3/WD 0.3, Muon 5.20e-3/WD 0.3; ResNet: SGDW LR 1.6/WD 1e-3, Muon 2e-2/WD 0.1
    Tuned before the SAM variants; SAM variants inherit them, which is fair across methods but still validation-fitted.
  • Newton-Schulz iteration counts = inner 3 (ImageNet) / 5 (CIFAR), outer 5
    Chosen by hand rather than ablated; approximation quality of Ortho affects both SPECSAM and Muon.
  • ViT-Tiny/4 output-layer radius multiplier = sqrt(1.92)
    Additional tuning knob for CIFAR diagnostics only (Appendix A.4).
axioms (6)
  • standard math Frechet differentiability of the loss and first-order expansion of the robust loss with uniform remainder on the unit ball
    Used to derive Eq. (3) and the SAM surrogate Eq. (4) in Section 2.1.
  • standard math Finite-dimensional dual-norm identities, subdifferential sum rule, dual of the spectral norm is the nuclear norm, and the polar factor is a nuclear-norm subgradient
    Justifies Ortho(G) as the steepest-ascent direction (Eq. 5) and the layerwise perturbation Eq. (8).
  • domain assumption The first-order SAM surrogate with stop-gradient is an adequate training objective
    Borrowed from Foret et al. [13]; the entire empirical comparison assumes this surrogate tracks the robust loss.
  • ad hoc to paper r=infinity (ell_infinity) aggregation over layer block norms is the right way to distribute perturbation budgets
    Motivated by Muon rather than derived; the product norm Eq. (6) with r=infinity is a design choice.
  • domain assumption Newton-Schulz with 3 or 5 iterations is a faithful approximation of Ortho
    Section 2.2 and Appendix A.4; if NSk fails on low-rank or tiny gradients, the effective perturbation differs from Eq. (8).
  • domain assumption Validation accuracy on ImageNet/CIFAR is the relevant generalization measure for the claims
    All conclusions use validation Top-1, ReaL, or ImageNet-R; other generalization measures might rank methods differently.

pith-pipeline@v1.3.0-alltime-deepseek · 19465 in / 16443 out tokens · 147181 ms · 2026-08-01T00:52:37.668941+00:00 · methodology

0 comments
read the original abstract

Sharpness-Aware Minimization (SAM) aims to improve generalization by encouraging insensitivity to small, worst-case parameter perturbations. However, the notion of a "small" perturbation is inherently geometry-dependent: while existing SAM variants have explored a wide range of choices, a clear perspective on which geometries are most effective in practice remains elusive. Recent work on matrix-aware optimization, particularly the Muon optimizer, suggests that respecting the matrix structure of hidden-layer weights can lead to strong empirical performance. Motivated by this, we study matrix-aware geometry in both stages of SAM: we introduce a layerwise spectral inner perturbation for matrix-valued hidden-layer parameters and combine it with either AdamW/SGDW or Muon in the outer update. Across ImageNet-1K experiments on ViT-Small/16 and ResNet-50, we find that the combination of a spectral inner step with a Muon outer step performs consistently strongly, achieving the best validation accuracy on both models among the evaluated methods.

Figures

Figures reproduced from arXiv: 2607.26001 by Edward Milsom, Michael Murray, Wenzhi Zhong.

Figure 1
Figure 1. Figure 1: Inner step geometry vs. outer optimizer on ImageNet-1K (left: ViT-Small/16, right: ResNet-50). [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Sensitivity to SAM training radius ρ on CIFAR-100 with ViT-Tiny/4. Left: best validation top-1 vs. ρ. Right: train top-1 vs. generalization gap, with ρ as an implicit variable along each curve; dashed diagonals are constant val-top-1 contours since val = train − gap. Stars mark the best-validation checkpoint per method. training accuracy for too little regularization in return. SAM-MUON and SPECSAM-MUON bo… view at source ↗
Figure 3
Figure 3. Figure 3: Effective-rank diagnostics for the CIFAR-100 ViT-Tiny/4 radius sweep. Within each transformer block, [PITH_FULL_IMAGE:figures/full_fig_p020_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Post-training sharpness analysis on the CIFAR-100 ViT-Tiny/4 radius sweep. The “sharpness” labels are [PITH_FULL_IMAGE:figures/full_fig_p021_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 15 linked inside Pith

  1. [1]

    Noah Amsel, David Persson, Christopher Musco, and Robert M. Gower. The polar express: Optimal matrix sign methods and their application to the Muon algorithm. InThe F ourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=yRtgZ1K8hO

    2026

  2. [2]

    Towards understanding sharpness-aware minimization, 2022

    Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware minimization, 2022. URL https://arxiv.org/abs/2206.06232

    Pith/arXiv arXiv 2022

  3. [3]

    Sharpness-aware min- imization leads to low-rank features

    Maksym Andriushchenko, Dara Bahri, Hossein Mobahi, and Nicolas Flammarion. Sharpness-aware min- imization leads to low-rank features. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, pages 47032– 47051. Curran Associates, Inc., 2023. URL https://proceedings.neurip...

    2023

  4. [4]

    Momentum-SAM: Sharpness aware minimization without computational overhead, 2025

    Marlon Becker, Frederick Altrock, and Benjamin Risse. Momentum-SAM: Sharpness aware minimization without computational overhead, 2025. URL https://arxiv.org/abs/2401.12033

    arXiv 2025

  5. [5]

    Modular duality in deep learning

    Jeremy Bernstein and Laker Newhouse. Modular duality in deep learning. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors,Proceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedings of Machine Learning Research, pages 3920–3930. PMLR, 13–...

    2025

  6. [6]

    Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord

    Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet?, 2020. URL https://arxiv.org/abs/2006.07159

    Pith/arXiv arXiv 2020

  7. [7]

    Better plain vit baselines for ImageNet-1K, 2022

    Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Better plain vit baselines for ImageNet-1K, 2022. URL https://arxiv.org/abs/2205.01580. 9

    Pith/arXiv arXiv 2022

  8. [8]

    Springer, 1997

    Rajendra Bhatia.Matrix Analysis, volume 169. Springer, 1997. ISBN 0387948465

    1997

  9. [9]

    When vision transformers outperform ResNets without pre-training or strong data augmentations, 2022

    Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform ResNets without pre-training or strong data augmentations, 2022. URL https://arxiv.org/abs/2106.01548

    Pith/arXiv arXiv 2022

  10. [10]

    DeepSeek-V4: Towards highly efficient million-token context intelligence

    DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence. Technical report, DeepSeek-AI, 2026. URL https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/resolve/main/DeepSeek_V4.pdf

    2026

  11. [11]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Pith/arXiv arXiv 2010

  12. [12]

    Efficient sharpness-aware minimization for improved training of neural networks

    Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent Tan. Efficient sharpness-aware minimization for improved training of neural networks. InInternational Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=n0OeTdNRG0Q

    2022

  13. [13]

    Sharpness-aware minimization for efficiently improving generalization.arXiv preprint arXiv:2010.01412, 2020

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization.arXiv preprint arXiv:2010.01412, 2020

    Pith/arXiv arXiv 2010

  14. [14]

    Glm-4.5: Agentic, reasoning, and coding (arc) foundation models, 2025

    GLM-4.5 Team. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models, 2025. URL https://arxiv.org/ abs/2508.06471

    Pith/arXiv arXiv 2025

  15. [15]

    Escaping the big data paradigm with compact transformers, 2022

    Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and Humphrey Shi. Escaping the big data paradigm with compact transformers, 2022. URL https://arxiv.org/abs/2104.05704

    Pith/arXiv arXiv 2022

  16. [16]

    Deep residual learning for image recognition, 2015

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL https://arxiv.org/abs/1512.03385

    Pith/arXiv arXiv 2015

  17. [17]

    The many faces of robustness: A critical analysis of out-of-distribution generalization, 2021

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization, 2021. URL https://arxiv.org/abs/2006.16241

    Pith/arXiv arXiv 2021

  18. [18]

    Flat minima.Neural Comput., 9(1):1–42, January 1997

    Sepp Hochreiter and Jürgen Schmidhuber. Flat minima.Neural Comput., 9(1):1–42, January 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.1.1. URL https://doi.org/10.1162/neco.1997.9.1.1

    doi:10.1162/neco.1997.9.1.1 1997

  19. [19]

    Muon: An optimizer for hidden layers in neural networks, 2024

    Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/

    2024

  20. [20]

    Fisher SAM: Information geometry and sharpness aware minimisation

    Minyoung Kim, Da Li, Shell X Hu, and Timothy Hospedales. Fisher SAM: Information geometry and sharpness aware minimisation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors,Proceedings of the 39th International Conference on Machine Learning, volume 162 ofProceedings of Machine Learning Research, page...

    2022

  21. [21]

    Kimi K2: Open agentic intelligence, 2025

    Kimi Team. Kimi K2: Open agentic intelligence, 2025. URL https://arxiv.org/abs/2507.20534

    Pith/arXiv arXiv 2025

  22. [22]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors,ICLR (Poster), 2015. URL http://dblp.uni-trier.de/db/conf/iclr/iclr2015.html#KingmaB14

    2015

  23. [23]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009. URL https://www.cs.toronto.edu/~kriz/learning-features-2009-TR. pdf

    2009

  24. [24]

    Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks

    Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In Marina Meila and Tong Zhang, editors,Proceedings of the 38th International Conference on Machine Learning, volume 139 ofProceedings of Machine Learning Research, pages 5905–5914. PMLR, 18–24 Jul ...

    2021

  25. [25]

    Friendly sharpness-aware minimization

    Tao Li, Pan Zhou, Zhengbao He, Xinwen Cheng, and Xiaolin Huang. Friendly sharpness-aware minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5631–5640, June 2024

    2024

  26. [26]

    Muon is scalable for LLM training, 2025

    Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is sca...

    Pith/arXiv arXiv 2025

  27. [27]

    Towards efficient and scalable sharpness- aware minimization

    Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness- aware minimization. In2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12350–12360, 2022. doi: 10.1109/CVPR52688.2022.01204

    arXiv 2022

  28. [28]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7. 10

    2019

  29. [29]

    Isambard-AI: a leadership class supercomputer optimised specifically for artificial intelligence, 2024

    Simon McIntosh-Smith, Sadaf R Alam, and Christopher Woods. Isambard-AI: a leadership class supercomputer optimised specifically for artificial intelligence, 2024. URL https://arxiv.org/abs/2410.11199

    Pith/arXiv arXiv 2024

  30. [30]

    Training deep learning models with norm-constrained LMOs, 2025

    Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and V olkan Cevher. Training deep learning models with norm-constrained LMOs, 2025. URL https://arxiv.org/abs/2502.07529

    Pith/arXiv arXiv 2025

  31. [31]

    The effective rank: A measure of effective dimensionality

    Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In2007 15th European Signal Processing Conference, pages 606–610, 2007

    2007

  32. [32]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge.International Journal of Computer Vision (IJCV), 115(3):211–252, 2015. doi: 10.1007/ s11263-015-0816-y

    2015

  33. [33]

    Avoiding spurious sharpness minimization broadens applicability of SAM

    Sidak Pal Singh, Hossein Mobahi, Atish Agarwala, and Yann Dauphin. Avoiding spurious sharpness minimization broadens applicability of SAM. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors,Proceedings of the 42nd International Conference on Machine Learning, volume 267 of...

    2025

  34. [34]

    Arcee trinity large technical report, 2026

    Varun Singh, Lucas Krauss, Sami Jaghouar, Matej Sirovatka, Charles Goddard, Fares Obied, Jack Min Ong, Jannik Straube, Fern, Aria Harley, Conner Stewart, Colin Kealty, Maziyar Panahi, Simon Kirsten, Anushka Deshpande, Anneketh Vij, Arthur Bresnu, Pranav Veldurthi, Raghav Ravishankar, Hardik Bishnoi, DatologyAI Team, Arcee AI Team, Prime Intellect Team, Ma...

    arXiv 2026

  35. [35]

    Rethinking the inception architecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2826, 2016. doi: 10.1109/CVPR.2016.308

    doi:10.1109/cvpr.2016.308 2016

  36. [36]

    How does sharpness-aware minimization minimize sharpness?, 2023

    Kaiyue Wen, Tengyu Ma, and Zhiyuan Li. How does sharpness-aware minimization minimize sharpness?, 2023. URL https://arxiv.org/abs/2211.05729

    Pith/arXiv arXiv 2023

  37. [37]

    PyTorch Image Models

    Ross Wightman. PyTorch Image Models. https://github.com/huggingface/pytorch-image-models, 2019

    2019

  38. [38]

    Riemannian SAM: Sharpness-aware minimization on riemannian manifolds

    Jihun Yun and Eunho Yang. Riemannian SAM: Sharpness-aware minimization on riemannian manifolds. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, pages 65784–65800. Curran Associates, Inc., 2023. URL https://proceedings. neurips.cc/paper_files/paper/2023/file/cf701...

    2023

  39. [39]

    Sharpness-aware minimization meets spectral norm

    Artur Zagitov, Roman Garaev, and Bader Rasheed. Sharpness-aware minimization meets spectral norm. Poster at the NeurIPS 2025 Workshop on Constrained Optimization for Machine Learning, 2025. URL https://www. researchgate.net/publication/400079982_Sharpness-Aware_Minimization_Meets_Spectral_Norm

    arXiv 2025

  40. [40]

    sharpness

    Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha C Dvornek, Sekhar Tatikonda, James S. Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-aware training. InInternational Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=edONMAnhLu-. 11 A Experiment details A.1 Architectures The main ...

    2022