Pith. sign in

REVIEW 3 major objections 5 minor 68 references

VASSO: Variance Suppression for Sharpness-Aware Minimization

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper's central claim: replacing SAM's minibatch-gradient adversary with an EMA-smoothed slope suppresses variance, provably stabilizes the stochastic linearization, and improves generalization on vision and language tasks.

desk verdict Solid extension of the authors' NeurIPS'23 VASSO: the efficient variant eVASSO is genuinely new and the theory is clean, but the stability-to-generalization link is asserted rather than proven. read the letter →

arxiv 2509.02433 v1 pith:TWWIHYSU submitted 2025-09-02 cs.LG

Bingcong Li , Yilang Zhang , Georgios B. Giannakis This is my paper
classification cs.LG
keywords sharpness-awareminimizationfriendlyadversaryvariancesuppressiongeneralizationstochasticlinearizationflatminimadeepneuralnetworksdelta-stability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sharpness-aware minimization (SAM) improves generalization by minimizing the worst-case loss over a small adversarial perturbation of the model parameters. In practice, SAM draws that perturbation from a single minibatch gradient, so the "adversary" is malicious only for that minibatch and can be friendly to the rest of the data; the paper argues this friendly-adversary problem can erase SAM's gains. The proposal, VASSO, replaces the raw minibatch gradient with an exponentially weighted moving average of recent gradients when choosing the perturbation. This provably shrinks the mean-square error of the slope and makes the stochastic linearization more stable by a factor of about sqrt(theta), and the experiments show the stabilized adversary improves accuracy across image classification, domain generalization, label-noise training, and machine translation while making cheap SAM variants more compute-efficient. If the paper is right, gradient variance—not just loss curvature—is a leading obstacle to SAM's generalization, and a nearly free averaging step addresses it.

What carries the argument

The load-bearing object is the EMA slope d_t=(1−θ)d_{t−1}+θ g_t(x_t); the perturbation is ε_t=ρ d_t/||d_t||. The EMA pools gradients across minibatches, so it estimates the full gradient with lower variance; Theorem 2 bounds its mean-square error by θσ^2+O((1−θ)^2σ^2/(θ^2√T)). Stability is measured by δ-stability, the expected gap between the quality of the linearization at slope d_t and at the true gradient. Theorem 3 is the carrier: under smoothness and bounded variance, VASSO's linearization is ρ(√θσ+O(σ/(θT^{1/4})))-stable versus ρσ for SAM, and Theorem 4 extends the guarantee to eVASSO.

What would settle it

Measure, on a real CIFAR-scale run, the δ-stability of SAM and VASSO perturbations against the full-batch-gradient perturbation at each checkpoint, and compare those values with hold-out accuracy. If VASSO's stability does not improve exactly when its test accuracy pulls ahead of SAM, the claimed mechanism fails; conversely, if δ-stability is high but accuracy does not follow across θ values, the link is incomplete.

Watch

Extended reading notes

Core claim

SAM's generalization shortfall, the paper argues, is a variance problem: the adversarial perturbation is only the normalized gradient of one minibatch, so noise makes it near-random relative to the full-batch gradient and "friendly" to other data. VASSO replaces that gradient with the EMA slope d_t=(1−θ)d_{t−1}+θ g_t(x_t), choosing the perturbation as ρ d_t/||d_t||. The paper proves this slope has mean-square error ≤θσ^2 plus a vanishing term, making VASSO ρ(√θσ+O(σ/(θT^{1/4})))-stable versus ρσ for SAM, and corroborates improved test accuracy on vision and language tasks, with eVASSO saving 57–70% of SAM's extra computation.

Load-bearing premise

The load-bearing premise is that reducing gradient variance and making the minibatch adversary's direction more stable, which the paper proves, is what drives the measured generalization gains; the theorems bound stability, and the generalization link is carried by the experiments.

Editorial extensions

If this is right

  • VASSO converges at the same rate as plain SGD up to constants, so the stability gain costs no asymptotic convergence speed.
  • The stability gain scales with √θ, but θ too small inflates the O((1−θ)^2/θ^2) correction; small θ values give the largest empirical gains, especially under label noise.
  • eVASSO obeys the same variance-suppression and stability bounds whenever it uses the perturbation, while computing only (1+p) gradients per iteration in expectation.
  • At matched accuracy, eVASSO saves 57% and 70% of eSAM's and SAM's extra computational overhead respectively.
  • Hessian-spectrum and loss-landscape tests show VASSO reaches flatter minima than SAM, supporting the claim that the stabilized adversary produces the intended sharpness effect.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A causal reading the paper invites but does not prove is that δ-stability is the channel through which generalization improves; a decisive test would hold δ-stability roughly fixed while varying gradient variance and check whether test accuracy still moves.
  • Because the proof only requires the perturbation to lie on the sphere, the same EMA-slope construction should transfer to other one-step stochastic-linearization trainers, such as multi-step ascent versions or scale-invariant variants, with the same stability bound up to constants.
  • Near a flat optimum in a large-data regime, raw minibatch perturbation directions become nearly uniform on the sphere while the EMA retains directional signal; a testable corollary is that VASSO's advantage over SAM grows as training approaches flatter optima or as batch size shrinks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes VASSO, a modification of SAM in which the adversarial perturbation is computed from an exponential moving average d_t of stochastic gradients instead of the single-minibatch gradient g_t(x_t), together with an efficient variant eVASSO that randomly skips the second forward-backward pass. The theoretical section proves (Theorem 1) convergence for SAM with any adversary confined to the sphere, (Theorem 2) an MSE bound for the EMA gradient estimate, (Theorem 3) a bound on the δ-stability of the induced stochastic linearization for VASSO, and (Theorem 4) the analogous stability bound for eVASSO. The empirical section reports accuracy/BLEU improvements over SAM on CIFAR-10/100, ImageNet, DomainBed, label-noise settings, and IWSLT-14, plus Hessian-spectrum and loss-landscape measurements intended to show that VASSO reaches flatter minima.

Significance. If the causal story of the paper were fully established, VASSO would be a simple, computationally cheap, and broadly applicable improvement to SAM. The paper has several genuine strengths: the proofs of Theorems 1–4 are included in Appendix D, the code is released, the experiments cover vision, domain generalization, label noise, and machine translation, and the δ-stability notion is clearly defined. The convergence result in Theorem 1 is clean and does not require bounded gradients. However, the paper's central conceptual claim — that variance suppression improves generalization by stabilizing adversaries — is not actually proven. The theory establishes stability of a first-order linearization metric, not a bound on sharpness, generalization gap, or test error. The empirical results are consistent with the proposed mechanism, but the controlled experiments do not isolate δ-stability from other effects of the EMA. The contribution is therefore better described as an empirical method with a stability guarantee, rather than a proven resolution of the 'friendly adversary' problem.

major comments (3)
  1. The load-bearing link between δ-stability and generalization is never established. Definition 1 bounds E|L_t(v) − L_t(∇f(x_t))|, which by the proof of Theorem 3 is at most ρ E‖v − ∇f(x_t)‖. This is a property of the gradient estimator, not of the loss landscape. Equation (4) calls L_t(v) − f(x_t) 'estimated sharpness', but for a nonlinear f the true sharpness max_{‖ϵ‖≤ρ} f(x+ϵ) − f(x) contains curvature terms of order ρ²λ_max that are invisible to the first-order quantity. Near a stationary point with small gradient noise, δ can be small even though the adversary direction carries no curvature information. Thus the statements in §3.1 that variance suppression 'promotes refined global sharpness estimates' and in §7 that 'stabilizing adversaries consolidates the generalization merits' are not consequences of Theorems 2–3. To make this claim load-bearing, the paper needs either (i) a theore
  2. The 'friendly adversary' diagnostics do not isolate the proposed causal mechanism. In Observation 1, SAM-db uses an independent batch for the adversary but the same batch for the update; this changes the correlation between the adversary and the descent direction, not merely the marginal variance of the adversary. The observed degradation could be due to a less informative update direction rather than to 'friendliness' in the sense of sharpness estimation. In Observation 2, adding Gaussian noise to g_t(x_t) changes the distribution of the perturbation direction and requires retuning ρ; it demonstrates that noise hurts, but not that δ-stability is the operative channel. A more direct test would compare VASSO against a baseline with the same EMA variance but no temporal correlation (e.g., using a larger minibatch for the adversary), or measure the empirical δ-stability on the actual traini
  3. The advertised √θ improvement in stability is an asymptotic statement, and the finite-T constants may spoil it for the practical choices of θ. From Eq. (6), the bound is θσ² + O((1−θ)²σ²/(θ²√T)). For θ = 0.2 and T = 200 (a typical epoch count in the experiments), the second term is of order (0.64/0.04)/√200 ≈ 1.13, i.e., comparable to θσ². Hence Theorem 3 does not actually certify that VASSO is more δ-stable than SAM for the hyperparameters used in the label-noise experiments, where the paper recommends small θ. The proof would be strengthened by stating the constants in Theorem 1 and by reporting the empirical δ-stability as a function of θ with confidence intervals, in addition to the qualitative Fig. 4.
minor comments (5)
  1. Calling L_t(v) − f(x_t) 'estimated sharpness' is misleading, since L_t is a linear upper envelope, not an estimate of the true maximization in (1). Suggest renaming it 'linearized loss surplus' or adding a caveat that it is only a first-order proxy.
  2. The statement that f_B(x_t+ϵ_t) − f_B(x_t) ≤ 0 whenever ⟨g_t(x_t), g_B(x_t)⟩ ≤ 0 is only true up to first-order Taylor error. Please state the 'for small ρ' qualification explicitly.
  3. The y-axis label is missing; the text explains that ∥ϵ_t − ϵ_{t−1}∥ is plotted, but the axis should be labeled and the units stated.
  4. Typographical issues: '(1.29x)' has a double parenthesis in the CIFAR-10 eVASSO (p=0.3) row, and 'eVaSSO' appears instead of 'eVASSO' in the caption text.
  5. The text reports θ tuned over {0.4, 0.9} and different optimal values per architecture, but there is no sensitivity analysis. A small table or paragraph on the effect of θ would help readers, especially because Theorem 3 gives conflicting guidance (small θ helps stability but hurts the bound's O term).

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the stability theorems are self-contained; the friendly-adversary-to-generalization link is an unproved empirical bridge, not a circular reduction.

full rationale

Theorem 2 and Theorem 3 are proved from Assumptions 1-3 in Appendix D; they do not assume the generalization improvement. Theorem 3 bounds the δ-stability metric of Definition 1, but that metric is a defined quantity, and the bound is a genuine derivation from the MSE bound E||d_t−∇f(x_t)||², not a restatement of the conclusion. The paper's own Sec. 2.5 introduces δ-stability as a characterization of 'friendly adversary' (a larger δ implies a more friendly adversary), so Theorem 3's conclusion that VASSO alleviates the friendly-adversary problem is partly definitional; however, the paper does not derive test error from δ-stability—the generalization claim rests on the numerical experiments in Sec. 5. The reference to the authors' prior work [23] appears only in the introduction ('This work broadens and innovates over [23]') and is not load-bearing for any theorem or algorithmic step. The self-citation is therefore not circular. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported, and no ansatz is smuggled in via citation. The main weakness is that no theorem connects δ-stability to generalization, but that is an evidential gap/correctness risk, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central stability results rest on standard stochastic optimization assumptions; the generalization benefit of a stable adversary is an unproven domain assumption supported only by experiments.

free parameters (3)
  • EMA weight theta = 0.2, 0.4, or 0.9 (tuned per task)
    Controls the stability-variance tradeoff in Theorem 3; selected from a small grid to maximize test accuracy.
  • perturbation radius rho = 0.01 to 0.5 (tuned per model)
    Standard SAM hyperparameter; tuned for SAM and reused for VASSO.
  • Bernoulli probability p (eVASSO) = 0.2 to 0.8
    Governs how often the second gradient is computed; sets the computation-generalization tradeoff.
assumptions (4)
  • standard math Assumptions 1-3: lower-bounded loss, L-smooth gradients, bounded stochastic gradient variance
    Common assumptions in nonconvex stochastic optimization, stated in Sec. 2.2.
  • domain assumption Stable stochastic linearization (small delta) translates into better generalization
    The paper asserts this link conceptually (Sec. 2.3, 3.1) but does not prove it; numerical results support it.
  • domain assumption Flat minima generalize better
    Inherited from SAM literature [5,8,9,10]; motivates the entire approach.
  • domain assumption Friendly adversary causes generalization degradation
    Motivated by SAM-db experiments in Sec. 2.3, but not a proven theorem.
invented entities (1)
  • friendly adversary
    purpose: Conceptual label for an adversary that does not align with global sharpness
    Defined via delta-stability (Definition 1); its existence is inferred from the paper's own variance experiments, so there is no independent falsifiable handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VASSO: Variance Suppression for Sharpness-Aware Minimization." pith.science (2026). https://pith.science/paper/TWWIHYSU

@misc{pith2026250902433,
  author       = {Pith},
  title        = {Pith review of: VASSO: Variance Suppression for Sharpness-Aware Minimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TWWIHYSU}},
  note         = {Machine review of arXiv:2509.02433}
}
read the original abstract

Sharpness-aware minimization (SAM) has well-documented merits in enhancing generalization of deep neural network models. Accounting for sharpness in the loss function geometry, where neighborhoods of `flat minima' heighten generalization ability, SAM seeks `flat valleys' by minimizing the maximum loss provoked by an adversarial perturbation within the neighborhood. Although critical to account for sharpness of the loss function, in practice SAM suffers from `over-friendly adversaries,' which can curtail the outmost level of generalization. To avoid such `friendliness,' the present contribution fosters stabilization of adversaries through variance suppression (VASSO). VASSO offers a general approach to provably stabilize adversaries. In particular, when integrating VASSO with SAM, improved generalizability is numerically validated on extensive vision and language tasks. Once applied on top of a computationally efficient SAM variant, VASSO offers a desirable generalization-computation tradeoff.

Figures

Figures reproduced from arXiv: 2509.02433 by the authors.

Figure 1
Figure 1. Intuition on why flat minima boost generalization. Solid (dotted) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) A friendly adversary diminishes the generalization ability of SAM; (b) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. (a) - (d) SAM’s adversaries are spread across the sphere; (e) SNR is in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The adversary of VASSO is more stable than SAM. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: eVASSO improves generalization-computation tradeoff relative to [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Visualization of loss landscapes. improves upon SAM. This confirms that the friendly adversary effect is indeed alleviated by variance suppression, which in turn boosts the generalization of ResNet18 as shown earlier in Section 5.1. Moreover, the flatness measure of eV…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 61 canonical work pages

  1. [23]

    Enhancing sharpness-aware optimization through variance suppression,

    B. Li and G. B. Giannakis, “Enhancing sharpness-aware optimization through variance suppression,” in Proc. Adv. Neural Info. Processing Systems, 2023

    work page 2023

  2. [1]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv:1810.04805, 2018

    arXiv 2018

  3. [2]

    Language models are few-shot learners,

    B. Tom, M. Benjamin, R. Nick, S. Melanie, K. Jared, D. Prafulla, N. Arvind, S. Pranav, S. Girish, A. Amanda, A. Sandhini, H.-V . Ariel, K. Gretchen, H. Tom, C. Rewon, R. Aditya, Z. D. M., W. Jeffrey, W. Clemens, H. Christopher, C. Mark, S. Eric, L. Mateusz, G. Scott, C. Benjamin, C. Jack, B. Christopher, M. Sam, R. Alec, S. Ilya, and A. Dario, “Language m...

    work page 2020

  4. [3]

    Understanding deep learning (still) requires rethinking gen- eralization,

    C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning (still) requires rethinking gen- eralization,” Communications of the ACM, vol. 64, no. 3, pp. 107–115, 2021

    work page 2021

  5. [4]

    Dropout: a simple way to prevent neural networks from overfitting,

    N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” J. Mach. Learn. Res., vol. 15, pp. 1929–1958, 2014

    work page 1929

  6. [5]

    When vision transform- ers outperform resnets without pre-training or strong data augmentations,

    X. Chen, C.-J. Hsieh, and B. Gong, “When vision transform- ers outperform resnets without pre-training or strong data augmentations,” in Proc. Int. Conf. Learning Represention , 2022

    work page 2022

  7. [6]

    The marginal value of adaptive gradient methods in machine learning

    A. C. Wilson, R. Roelofs, M. Stern, N. Srebro, B. Recht, and N. Srebro, “The marginal value of adaptive gradient methods in machine learning.” in Proc. Int. Conf. Machine Learning , vol. 30, 2017, pp. 4148–4158

    work page 2017

  8. [7]

    Decoupled weight decay regu- larization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regu- larization,” in Proc. Int. Conf. Learning Represention, 2017

    work page 2017

Show all 68 references
  1. [8]

    Sharpness-aware minimization for efficiently improving generalization,

    P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur, “Sharpness-aware minimization for efficiently improving generalization,” in Proc. Int. Conf. Learning Represention , 2021

    2021

  2. [9]

    Fantastic generalization measures and where to find them,

    Y . Jiang, B. Neyshabur, D. Krishnan, H. Mobahi, and S. Ben- gio, “Fantastic generalization measures and where to find them,” arXiv:1912.02178, 2019

    1912 arXiv

  3. [10]

    On large-batch training for deep learning: Generalization gap and sharp minima

    N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang, “On large-batch training for deep learning: Generalization gap and sharp minima.” in Proc. Int. Conf. Learning Represention, 2016

    2016

  4. [11]

    Entropy-SGD: Biasing gradient descent into wide valleys,

    P. Chaudhari, A. Choromanska, S. Soatto, Y . LeCun, C. Bal- dassi, C. Borgs, J. Chayes, L. Sagun, and R. Zecchina, “Entropy-SGD: Biasing gradient descent into wide valleys,” in Proc. Int. Conf. Learning Represention, 2017

    2017

  5. [12]

    GA-SAM: Gradient- strength based adaptive sharpness-aware minimization for improved generalization,

    Z. Zhang, R. Luo, Q. Su, and X. Sun, “GA-SAM: Gradient- strength based adaptive sharpness-aware minimization for improved generalization,” in Proc. Conf. Empirical Methods in Natural Language Processing, 2022

    2022

  6. [13]

    Towards under- standing sharpness-aware minimization

    M. Andriushchenko and N. Flammarion, “Towards under- standing sharpness-aware minimization.” in Proc. Int. Conf. Machine Learning, 2022, pp. 639–668

    2022

  7. [14]

    How does sharpness- aware minimization minimizes sharpness,

    K. Wen, T. Ma, and Z. hiyuan Li, “How does sharpness- aware minimization minimizes sharpness,” inProc. Int. Conf. Learning Represention, 2023

    2023

  8. [15]

    The dynamics of sharpness-aware minimization: Bouncing across ravines and drifting towards wide minima,

    P. L. Bartlett, P. M. Long, and O. Bousquet, “The dynamics of sharpness-aware minimization: Bouncing across ravines and drifting towards wide minima,”arXiv:2210.01513, 2022

    2022 arXiv

  9. [16]

    Surrogate gap min- imization improves sharpness-aware training

    J. Zhuang, B. Gong, L. Yuan, Y . Cui, H. Adam, N. Dvornek, S. Tatikonda, J. Duncan, and T. Liu, “Surrogate gap min- imization improves sharpness-aware training.” in Proc. Int. Conf. Learning Represention, 2022

    2022

  10. [17]

    Fisher SAM: Information geometry and sharpness aware minimi- sation

    M. Kim, D. Li, S. X. Hu, and T. M. Hospedales, “Fisher SAM: Information geometry and sharpness aware minimi- sation.” in Proc. Int. Conf. Machine Learning , 2022, pp. 11 148–11 161

    2022

  11. [18]

    To- wards efficient and scalable sharpness-aware minimization,

    Y . Liu, S. Mai, X. Chen, C.-J. Hsieh, and Y . You, “To- wards efficient and scalable sharpness-aware minimization,” in Proc. Conf. Computer Vision and Pattern Recognition, vol. 2022, 2022, pp. 12 350–12 360

    2022

  12. [19]

    Efficient sharpness-aware minimization for improved training of neural networks,

    J. Du, H. Yan, J. Feng, J. T. Zhou, L. Zhen, R. S. M. Goh, and V . Y . F. Tan, “Efficient sharpness-aware minimization for improved training of neural networks,” inProc. Int. Conf. Learning Represention, 2022

    2022

  13. [20]

    SS-SAM: Stochastic sched- uled sharpness-aware minimization for efficiently training deep neural networks,

    Y . Zhao, H. Zhang, and X. Hu, “SS-SAM: Stochastic sched- uled sharpness-aware minimization for efficiently training deep neural networks,” arXiv:2203.09962, 2022

    2022 arXiv

  14. [21]

    An adap- tive policy to employ sharpness-aware minimization,

    W. Jiang, H. Yang, Y . Zhang, and J. Kwok, “An adap- tive policy to employ sharpness-aware minimization,” arXiv:2304.14647, 2023

    2023 arXiv

  15. [22]

    Sharpness-aware training for free,

    J. Du, D. Zhou, J. Feng, V . Y . F. Tan, and J. T. Zhou, “Sharpness-aware training for free,” in Proc. Adv. Neural Info. Processing Systems, 2022

    2022

  16. [24]

    Stochastic first-and zeroth-order methods for nonconvex stochastic programming,

    S. Ghadimi and G. Lan, “Stochastic first-and zeroth-order methods for nonconvex stochastic programming,” SIAM Journal on Optimization , vol. 23, no. 4, pp. 2341–2368, 2013

    2013

  17. [25]

    Optimization methods for large-scale machine learning,

    L. Bottou, F. E. Curtis, and J. Nocedal, “Optimization methods for large-scale machine learning,” arXiv preprint arXiv:1606.04838, 2016

    2016 arXiv

  18. [26]

    Make sharpness-aware minimization stronger: A sparsified perturbation approach,

    P. Mi, L. Shen, T. Ren, Y . Zhou, X. Sun, R. Ji, and D. Tao, “Make sharpness-aware minimization stronger: A sparsified perturbation approach,” inProc. Adv. Neural Info. Processing Systems, 2022

    2022

  19. [27]

    Stochastic Frank-Wolfe methods for nonconvex optimization,

    S. J. Reddi, S. Sra, B. P ´oczos, and A. Smola, “Stochastic Frank-Wolfe methods for nonconvex optimization,” in Aller- ton conference on communication, control, and computing . IEEE, 2016, pp. 1244–1251

    2016

  20. [28]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, IEEE TRANSACTIONS ON PATTERN ANAL YSIS AND MACHINE INTELLIGENCE (SUBMITTED) 11 A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proc. Adv. Neural Info. Processing Systems , vol. 30, 2017

    2017

  21. [29]

    Accelerating stochastic gradient descent using predictive variance reduction,

    R. Johnson and T. Zhang, “Accelerating stochastic gradient descent using predictive variance reduction,” in Proc. Ad- vances in Neural Info. Process. Syst. , Lake Tahoe, Nevada, 2013, pp. 315–323

    2013

  22. [30]

    SARAH: A novel method for machine learning problems using stochastic recursive gradient,

    L. M. Nguyen, J. Liu, K. Scheinberg, and M. Tak ´aˇc, “SARAH: A novel method for machine learning problems using stochastic recursive gradient,” in Proc. Int. Conf. Ma- chine Learning, Sydney, Australia, 2017

    2017

  23. [31]

    Almost tune-free vari- ance reduction,

    B. Li, L. Wang, and G. B. Giannakis, “Almost tune-free vari- ance reduction,” in Proc. Intl. Conf. on Machine Learning , 2020

    2020

  24. [32]

    On the convergence of SARAH and beyond,

    B. Li, M. Ma, and G. B. Giannakis, “On the convergence of SARAH and beyond,” in Proc. Int. Conf. Artif. Intel. and Stats., 2019

    2019

  25. [33]

    K. V . Mardia and P. E. Jupp, Directional Statistics. Direc- tional statistics, 2000

    2000

  26. [34]

    ASAM: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks,

    J. Kwon, J. Kim, H. Park, and I. K. Choi, “ASAM: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks,” inProc. Int. Conf. Machine Learning, vol. 139, 2021, pp. 5905–5914

    2021

  27. [35]

    Improved regulariza- tion of convolutional neural networks with cutout

    T. Devries and G. W. Taylor, “Improved regulariza- tion of convolutional neural networks with cutout.” vol. abs/1708.04552, 2017

    2017 arXiv

  28. [36]

    ImageNet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in Proc. Conf. Computer Vision and Pattern Recognition, 2009, pp. 248–255

    2009

  29. [37]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. Int. Conf. Learning Represention , 2014

    2014

  30. [38]

    Domain generalization: A survey,

    K. Zhou, Z. Liu, Y . Qiao, T. Xiang, and C. C. Loy, “Domain generalization: A survey,” IEEE Trans. Pattern Analysis and Machine Intelligence, 2022

    2022

  31. [39]

    Unsupervised domain adaptation of object detectors: A survey,

    P. Oza, V . A. Sindagi, V . V . Sharmini, and V . M. Patel, “Unsupervised domain adaptation of object detectors: A survey,” IEEE Trans. Pattern Analysis and Machine Intel- ligence, 2023

    2023

  32. [40]

    Sharpness-aware gradient matching for domain generalization,

    P. Wang, Z. Zhang, Z. Lei, and L. Zhang, “Sharpness-aware gradient matching for domain generalization,” in Proc. Conf. Computer Vision and Pattern Recognition , 2023, pp. 3769– 3778

    2023

  33. [41]

    In search of lost domain generalization,

    I. Gulrajani and D. Lopez-Paz, “In search of lost domain generalization,” in Proc. Int. Conf. Learning Represention , 2020

    2020

  34. [42]

    Deeper, broader and artier domain generalization,

    D. Li, Y . Yang, Y .-Z. Song, and T. M. Hospedales, “Deeper, broader and artier domain generalization,” Oct 2017

    2017

  35. [43]

    Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias,

    C. Fang, Y . Xu, and D. N. Rockmore, “Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias,” December 2013

    2013

  36. [44]

    Deep hashing network for unsupervised domain adaptation,

    H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Pan- chanathan, “Deep hashing network for unsupervised domain adaptation,” in Proc. Conf. Computer Vision and Pattern Recognition, July 2017

    2017

  37. [45]

    Recognition in terra incognita,

    S. Beery, G. Van Horn, and P. Perona, “Recognition in terra incognita,” September 2018

    2018

  38. [46]

    Sharpness-aware gradient matching for domain generalization,

    P. Wang, Z. Zhang, Z. Lei, and L. Zhang, “Sharpness-aware gradient matching for domain generalization,” in Proc. Conf. Computer Vision and Pattern Recognition , June 2023, pp. 3769–3778

    2023

  39. [47]

    Dynamic loss for robust learning,

    S. Jiang, J. Li, J. Zhang, Y . Wang, and T. Xu, “Dynamic loss for robust learning,” IEEE Trans. Pattern Analysis and Machine Intelligence, 2023

    2023

  40. [48]

    Report on the 11th iwslt evaluation campaign, iwslt 2014,

    M. Cettolo, J. Niehues, S. Stker, L. Bentivogli, and M. Fed- erico, “Report on the 11th iwslt evaluation campaign, iwslt 2014,” 2014

    2014

  41. [49]

    The break-even point on optimiza- tion trajectories of deep neural networks

    S. Jastrzebski, M. Szymczak, S. Fort, D. Arpit, J. Tabor, K. Cho, and K. J. Geras, “The break-even point on optimiza- tion trajectories of deep neural networks.” in Proc. Int. Conf. Learning Represention, 2020

    2020

  42. [50]

    An investigation into neural net optimization via hessian eigenvalue density

    B. Ghorbani, S. Krishnan, and Y . Xiao, “An investigation into neural net optimization via hessian eigenvalue density.” in Proc. Int. Conf. Machine Learning, 2019, pp. 2232–2241

    2019

  43. [51]

    Visu- alizing the loss landscape of neural nets,

    H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein, “Visu- alizing the loss landscape of neural nets,” vol. 31, 2018

    2018

  44. [52]

    Three factors influencing min- ima in sgd,

    S. Jastrzebski, Z. Kenton, D. Arpit, N. Ballas, A. Fischer, Y . Bengio, and A. Storkey, “Three factors influencing min- ima in sgd,” arXiv:1711.04623, 2017

    2017 arXiv

  45. [53]

    Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data,

    G. K. Dziugaite and D. M. Roy, “Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data,” in Proc. Conf. Uncerntainty in Artif. Intel., 2017

    2017

  46. [54]

    Exploring generalization in deep learning

    B. Neyshabur, S. Bhojanapalli, D. Mcallester, N. Srebro, and N. Srebro, “Exploring generalization in deep learning.” in Proc. Adv. Neural Info. Processing Systems , vol. 30, 2017, pp. 5947–5956

    2017

  47. [55]

    On the generalization of models trained with sgd: Information-theoretic bounds and impli- cations,

    Z. Wang and Y . Mao, “On the generalization of models trained with sgd: Information-theoretic bounds and impli- cations,” in Proc. Int. Conf. Learning Represention, 2022

    2022

  48. [56]

    Averaging weights leads to wider optima and better generalization,

    P. Izmailov, D. Podoprikhin, T. Garipov, D. P. Vetrov, and A. G. Wilson, “Averaging weights leads to wider optima and better generalization,” in Proc. Conf. Uncerntainty in Artif. Intel., 2018, pp. 876–885

    2018

  49. [57]

    Adversarial weight per- turbation helps robust generalization,

    D. Wu, S.-T. Xia, and Y . Wang, “Adversarial weight per- turbation helps robust generalization,” in Proc. Adv. Neural Info. Processing Systems, vol. 33, 2020, pp. 2958–2969

    2020

  50. [58]

    Penalizing gradient norm for efficiently improving generalization in deep learning,

    Y . Zhao, H. Zhang, and X. Hu, “Penalizing gradient norm for efficiently improving generalization in deep learning,” in Proc. Int. Conf. Machine Learning, 2022, pp. 26 982–26 992

    2022

  51. [59]

    Implicit gradient regulariza- tion,

    D. G. Barrett and B. Dherin, “Implicit gradient regulariza- tion,” in Proc. Int. Conf. Learning Represention, 2021

    2021

  52. [60]

    Exploring the effect of multi-step ascent in sharpness-aware minimization,

    H. Kim, J. Park, Y . Choi, W. Lee, and J. Lee, “Exploring the effect of multi-step ascent in sharpness-aware minimization,” arXiv:2302.10181, 2023

    2023 arXiv

  53. [61]

    Stochastic con- ditional gradient methods: From convex minimization to submodular maximization,

    A. Mokhtari, H. Hassani, and A. Karbasi, “Stochastic con- ditional gradient methods: From convex minimization to submodular maximization,” Journal of Machine Learning Research, vol. 21, no. 1, pp. 4232–4280, 2020

    2020

  54. [62]

    Accelerating Frank- Wolfe with weighted average gradients,

    Y . Zhang, B. Li, and G. B. Giannakis, “Accelerating Frank- Wolfe with weighted average gradients,” in Proc. IEEE Int. Conf. Acoust., Speech, Sig. Process., 2021, pp. 5529–5533

    2021

  55. [63]

    Heavy ball momentum for conditional gradient,

    B. Li, A. Sadeghi, and G. Giannakis, “Heavy ball momentum for conditional gradient,” in Proc. Advances in Neural Info. Process. Syst., 2021

    2021

  56. [64]

    Understanding and increas- ing efficiency of Frank-Wolfe adversarial training,

    T. Tsiligkaridis and J. Roberts, “Understanding and increas- ing efficiency of Frank-Wolfe adversarial training,” in Proc. Conf. Computer Vision and Pattern Recognition , 2022, pp. 50–59

    2022

  57. [65]

    A momentum-guided Frank-Wolfe algorithm,

    B. Li, M. Coutino, G. B. Giannakis, and G. Leus, “A momentum-guided Frank-Wolfe algorithm,” IEEE Trans. on IEEE TRANSACTIONS ON PATTERN ANAL YSIS AND MACHINE INTELLIGENCE (SUBMITTED) 12 Signal Processing, vol. 69, pp. 3597–3611, 2021

    2021

  58. [66]

    Enhancing Frank-Wolfe with an extra subproblem,

    B. Li, L. Wang, G. B. Giannakis, and Z. Zhao, “Enhancing Frank-Wolfe with an extra subproblem,” in Proc. of 35th AAAI Conf. on Artificial Intelligence, 2021

    2021

  59. [67]

    Accelerated stochastic gradient-free and projection-free methods,

    F. Huang, L. Tao, and S. Chen, “Accelerated stochastic gradient-free and projection-free methods,” in Proc. Int. Conf. Machine Learning. PMLR, 2020, pp. 4519–4530. APPENDIX A SFW VIS -A-VIS SAM ADVERSARY The stochastic Frank-Wolfe (SFW) algorithm outlined in Alg. 3 solves the ...

    2020

  60. [68]

    Scaling is used to vary the SNR

    We use ∇f (x) = [0 .2, −0.1, 0.6], and stochastic noise ξ = [ ξ1, ξ2, ξ3], where ξ1, ξ2, ξ3 are iid Gaussian random variables with variance scaling with 0.2, 1, 2, respectively. Scaling is used to vary the SNR. We generate 100 adversaries by solving arg max∥ϵ∥≤ρ⟨∇f (x) +ξ, ϵ⟩ ...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.