Gradient-similarity-regularized weight averaging and WA+SAM fine-tuning are tested on OOD and few-shot domain adaptation benchmarks, with mixed results that do not support the claimed improvements.
When Do Flat Minima Optimizers Work?
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Recently, flat-minima optimizers, which seek to find parameters in low-loss neighborhoods, have been shown to improve a neural network's generalization performance over stochastic and adaptive gradient-based optimizers. Two methods have received significant attention due to their scalability: 1. Stochastic Weight Averaging (SWA), and 2. Sharpness-Aware Minimization (SAM). However, there has been limited investigation into their properties and no systematic benchmarking of them across different domains. We fill this gap here by comparing the loss surfaces of the models trained with each method and through broad benchmarking across computer vision, natural language processing, and graph representation learning tasks. We discover several surprising findings from these results, which we hope will help researchers further improve deep learning optimizers, and practitioners identify the right optimizer for their problem.
citation-role summary
background 1
citation-polarity summary
fields
cs.CV 1years
2025 1verdicts
REJECT 1roles
background 1polarities
background 1representative citing papers
citing papers explorer
-
Weight Averaging for Out-of-Distribution Generalization and Few-Shot Domain Adaptation
Gradient-similarity-regularized weight averaging and WA+SAM fine-tuning are tested on OOD and few-shot domain adaptation benchmarks, with mixed results that do not support the claimed improvements.