VASSO replaces SAM's minibatch gradient with an exponential moving average of past gradients when computing the adversarial perturbation, improving generalization across vision and language tasks.
Exploring the Effect of Multi-step Ascent in Sharpness-Aware Minimization
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Recently, Sharpness-Aware Minimization (SAM) has shown state-of-the-art performance by seeking flat minima. To minimize the maximum loss within a neighborhood in the parameter space, SAM uses an ascent step, which perturbs the weights along the direction of gradient ascent with a given radius. While single-step or multi-step can be taken during ascent steps, previous studies have shown that multi-step ascent SAM rarely improves generalization performance. However, this phenomenon is particularly interesting because the multi-step ascent is expected to provide a better approximation of the maximum neighborhood loss. Therefore, in this paper, we analyze the effect of the number of ascent steps and investigate the difference between both single-step ascent SAM and multi-step ascent SAM. We identify the effect of the number of ascent on SAM optimization and reveal that single-step ascent SAM and multi-step ascent SAM exhibit distinct loss landscapes. Based on these observations, we finally suggest a simple modification that can mitigate the inefficiency of multi-step ascent SAM.
citation-role summary
background 1
citation-polarity summary
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1roles
background 1polarities
background 1representative citing papers
citing papers explorer
-
VASSO: Variance Suppression for Sharpness-Aware Minimization
VASSO replaces SAM's minibatch gradient with an exponential moving average of past gradients when computing the adversarial perturbation, improving generalization across vision and language tasks.