Learning quantization-aware linear paths in weight space yields a midpoint whose direct quantization matches quantization-aware training performance without using straight-through estimators.
hub Mixed citations
Sharpness-Aware Minimization for Efficiently Improving Generalization
Mixed citation behavior. Most common role is background (57%).
54
Pith papers citing it
104
external citations · Pith
Background
57% of classified citations
abstract
In today's heavily overparameterized models, the value of the training loss provides few guarantees on model generalization ability. Indeed, optimizing only the training loss value, as is commonly done, can easily lead to suboptimal model quality. Motivated by prior work connecting the geometry of the loss landscape and generalization, we introduce a novel, effective procedure for instead simultaneously minimizing loss value and loss sharpness. In particular, our procedure, Sharpness-Aware Minimization (SAM), seeks parameters that lie in neighborhoods having uniformly low loss; this formulation results in a min-max optimization problem on which gradient descent can be performed efficiently. We present empirical results showing that SAM improves model generalization across a variety of benchmark datasets (e.g., CIFAR-10, CIFAR-100, ImageNet, finetuning tasks) and models, yielding novel state-of-the-art performance for several. Additionally, we find that SAM natively provides robustness to label noise on par with that provided by state-of-the-art procedures that specifically target learning with noisy labels. We open source our code at \url{https://github.com/google-research/sam}.
hub tools
citation-role summary
background 5
method 2
citation-polarity summary
representative citing papers
PoisonLoRA demonstrates ~100% attack success rates for stealthy LoRA poisoning via concept hijacking and task injection on real platforms, with robustness to base model transfer and multiple remixes.
Muon outperforms Adam by reducing curvature penalty via lower Normalized Directional Sharpness, as shown via Taylor approximation on LLM training and proven on stylized quadratic problems with heterogeneous curvature.
Face-Feature Tuning is a label-free logit remapping method that reduces FPR/TPR gaps across groups in deepfake detection while preserving overall accuracy.
Proposes pointwise Riemannian Dimension from feature eigenvalues to derive tighter, representation-aware generalization bounds for deep networks in the nonlinear regime.
Derives upper and lower generalization bounds for the student relative to the teacher using a new distillation divergence, plus a loss-sharpness-aware bound and a bias-variance-rank decomposition in the linear Gaussian case.
Gradient matching empirically recovers implicit regularization effects such as l2 penalties from early stopping and dropout in neural networks.
A fully differentiable TensorFlow gyrokinetic code allows approximate gradients of nonlinear turbulence quantities to be used for outer-loop tasks such as profile prediction despite stochasticity.
FP32-converged language models enter a post-convergence phase where INT4 quantization error explodes while FP32 perplexity remains stable, with onset tied to fine convergence rather than learning rate decay.
The FedSurg challenge benchmarks federated learning on appendectomy videos and finds only 26% F1 on unseen centers even with centralized data, plus extra penalties from decentralization, with spatiotemporal models performing best.
Neural loss landscapes contain flat channels to infinity along which gradient flow leads pairs of neurons to implement gated linear units.
The paper proposes Decoupled Performative Prediction showing that institutions achieve lower risk by using distinct internal decision models and disclosed models, with an algorithm that converges under standard assumptions and a deception cost metric that self-imposed constraints do not sufficiently
A hierarchical prior-decoder model using CLIP latents generates more diverse text-conditional images than direct methods while preserving photorealism and caption fidelity.
DP-NGD enables second-order optimization under differential privacy by decoupling curvature estimation onto public data, performing isotropic DP operations in a whitened space, and dynamically clamping curvature eigenvalues to prevent instability.
Hessian eigenvector displacement and inverse participation ratio metrics show SGD stabilizing leading curvature directions while Adam causes more reorganization and parameter localization in MLP training.
Derives closed-form gradient of WS upper bound on Hessian max eigenvalue for 3-layer cross-entropy NNs and proposes HSR regularization to steer toward flat minima.
Knowledge editing methods redistribute and suppress rather than overwrite facts in LLMs, creating narrow vulnerable regions in representation space that adversarial prompts can exploit.
LFPM mitigates backdoors in model merging by optimizing an anti-backdoor task vector in feature space under the Cross-Task Linearity framework to suppress backdoors without major clean-task degradation.
Flatness Preference Optimization (FlatPO) improves multimodal PEFT generalization by flattening a small set of sharp dimensions that dominate performance.
SFF smooths the non-convex loss landscape of pre-trained LTSMs by linear weight interpolation with a random model, enabling more effective fine-tuning while preserving pre-trained knowledge.
Patcher improves LLM robustness to malicious full-parameter finetuning by scaling train-time adversarial attacks in a bi-level optimization loop and supplies an efficient parallel implementation.
Casting Tent, EATA, SAR, DeYO, and COME into DP-TTA via per-sample clipping and Gaussian noise yields adequate privacy on ImageNet-C at modest accuracy and compute cost, with clipping sometimes improving stability.
For random MPS and Clifford+T circuits, increases in entanglement or T-count correlate with sharper loss minima and worse reconstruction under constrained neural capacity.
Optimizer-inspired Transformer architectures with momentum achieve lower validation loss than standard Transformers, with momentum identified as the key factor over preconditioning.
citing papers explorer
-
Neural Network Quantization by Learning Low-Loss Subspaces
Learning quantization-aware linear paths in weight space yields a midpoint whose direct quantization matches quantization-aware training performance without using straight-through estimators.
-
Customization under Fire: Plugin Poisoning in Text-to-Image Ecosystem
PoisonLoRA demonstrates ~100% attack success rates for stealthy LoRA poisoning via concept hijacking and task injection on real platforms, with robustness to base model transfer and multiple remixes.
-
Why Muon Outperforms Adam: A Curvature Perspective
Muon outperforms Adam by reducing curvature penalty via lower Normalized Directional Sharpness, as shown via Taylor approximation on LLM training and proven on stylized quadratic problems with heterogeneous curvature.
-
Toward Calibrated, Fair, and accurate Deepfake Detection
Face-Feature Tuning is a label-free logit remapping method that reduces FPR/TPR gaps across groups in deepfake detection while preserving overall accuracy.
-
Pointwise Generalization in Deep Neural Networks
Proposes pointwise Riemannian Dimension from feature eigenvalues to derive tighter, representation-aware generalization bounds for deep networks in the nonlinear regime.
-
On the Generalization of Knowledge Distillation: An Information-Theoretic View
Derives upper and lower generalization bounds for the student relative to the teacher using a new distillation divergence, plus a loss-sharpness-aware bound and a bias-variance-rank decomposition in the linear Gaussian case.
-
Estimating Implicit Regularization in Deep Learning
Gradient matching empirically recovers implicit regularization effects such as l2 penalties from early stopping and dropout in neural networks.
-
iGENE: A Differentiable Flux-Tube Gyrokinetic Code in TensorFlow
A fully differentiable TensorFlow gyrokinetic code allows approximate gradients of nonlinear turbulence quantities to be used for outer-loop tasks such as profile prediction despite stochasticity.
-
When Flat Minima Fail: Characterizing INT4 Quantization Collapse After FP32 Convergence
FP32-converged language models enter a post-convergence phase where INT4 quantization error explodes while FP32 perplexity remains stable, with onset tied to fine convergence rather than learning rate decay.
-
Federated Learning for Surgical Vision in Appendicitis Classification: Results of the FedSurg EndoVis 2024 Challenge
The FedSurg challenge benchmarks federated learning on appendectomy videos and finds only 26% F1 on unseen centers even with centralized data, plus extra penalties from decentralization, with spatiotemporal models performing best.
-
Flat Channels to Infinity in Neural Loss Landscapes
Neural loss landscapes contain flat channels to infinity along which gradient flow leads pairs of neurons to implement gated linear units.
-
Strategically Deceptive Model Deployment in Performative Prediction
The paper proposes Decoupled Performative Prediction showing that institutions achieve lower risk by using distinct internal decision models and disclosed models, with an algorithm that converges under standard assumptions and a deception cost metric that self-imposed constraints do not sufficiently
-
Hierarchical Text-Conditional Image Generation with CLIP Latents
A hierarchical prior-decoder model using CLIP latents generates more diverse text-conditional images than direct methods while preserving photorealism and caption fidelity.
-
Differentially Private Natural Gradient Descent
DP-NGD enables second-order optimization under differential privacy by decoupling curvature estimation onto public data, performing isotropic DP operations in a whitened space, and dynamically clamping curvature eigenvalues to prevent instability.
-
Characterizing Optimizer-Dependent Training Dynamics Through Hessian Eigenvector Displacement and Localization
Hessian eigenvector displacement and inverse participation ratio metrics show SGD stabilizing leading curvature directions while Adam causes more reorganization and parameter localization in MLP training.
-
Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks
Derives closed-form gradient of WS upper bound on Hessian max eigenvalue for 3-layer cross-entropy NNs and proposes HSR regularization to steer toward flat minima.
-
Exposing the Illusion of Erasure in Knowledge Editing for LLMs
Knowledge editing methods redistribute and suppress rather than overwrite facts in LLMs, creating narrow vulnerable regions in representation space that adversarial prompts can exploit.
-
From Parameters to Feature Space: Task Arithmetic for Backdoor Mitigation in Model Merging
LFPM mitigates backdoors in model merging by optimizing an anti-backdoor task vector in feature space under the Cross-Task Linearity framework to suppress backdoors without major clean-task degradation.
-
5% > 100%: Flatness Preference is All You Need for Multimodal Parameter-Efficient Fine-Tuning
Flatness Preference Optimization (FlatPO) improves multimodal PEFT generalization by flattening a small set of sharp dimensions that dominate performance.
-
Lost in the Non-convex Loss Landscape: How to Fine-tune the Large Time Series Model?
SFF smooths the non-convex loss landscape of pre-trained LTSMs by linear weight interpolation with a random model, enabling more effective fine-tuning while preserving pre-trained knowledge.
-
Defending Against Malicious Finetuning by Scaling Train-time Adversarial Attacks
Patcher improves LLM robustness to malicious full-parameter finetuning by scaling train-time adversarial attacks in a bi-level optimization loop and supplies an efficient parallel implementation.
-
Private and Stable Test-Time Adaptation with Differential Privacy
Casting Tent, EATA, SAR, DeYO, and COME into DP-TTA via per-sample clipping and Gaussian noise yields adequate privacy on ImageNet-C at modest accuracy and compute cost, with clipping sometimes improving stability.
-
Comparing Classical Simulation and Sample-Based Learning of Quantum Systems
For random MPS and Clifford+T circuits, increases in entanglement or T-count correlate with sharper loss minima and worse reconstruction under constrained neural capacity.
-
Momentum Streams for Optimizer-Inspired Transformers
Optimizer-inspired Transformer architectures with momentum achieve lower validation loss than standard Transformers, with momentum identified as the key factor over preconditioning.
-
Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning
Distinguishable Deletion unifies knowledge erasure and refusal for LLM unlearning via an energy index that enforces boundaries during training and enables refusal at inference.
-
Synthetic Data Generation for Long-Tail Medical Image Classification: A Case Study in Skin Lesions
A diffusion-based synthetic data pipeline using inpainting and OOD post-selection improves long-tail skin lesion classification on ISIC2019, delivering over 28% accuracy gain on the rarest class.
-
Geometric and Spectral Alignment for Deep Neural Network II
The work establishes margin-verified certificates for physical alignment of residual Jacobian chains by bounding truncation errors and decomposing the Physical Alignment Matrix orthogonally under fitted effective-rank windows.
-
Generalization at the Edge of Stability
Training at the edge of stability causes neural network optimizers to converge on fractal attractors whose effective dimension, measured via a new sharpness dimension from the Hessian spectrum, bounds generalization error in a way not captured by prior trace or norm measures.
-
Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima
Nexus matches standard pretraining loss while improving downstream and OOD performance by maximizing gradient similarity so task-specific minima stay close.
-
LAA-X: Unified Localized Artifact Attention for Quality-Agnostic and Generalizable Face Forgery Detection
LAA-X uses multi-task learning with explicit localized artifact attention and blending synthesis to build a deepfake detector that generalizes to high-quality and unseen manipulations after training only on real and pseudo-fake samples.
-
Robust Policy Optimization to Prevent Catastrophic Forgetting
FRPO applies a max-min robust optimization over KL-bounded policy neighborhoods during RLHF to reduce catastrophic forgetting of safety and accuracy under subsequent SFT or RL fine-tuning.
-
Downgrade to Upgrade: Optimizer Simplification Enhances Robustness in LLM Unlearning
Downgrading optimizers to lower-information variants during LLM unlearning yields more robust forgetting on MUSE and WMDP benchmarks by converging to harder-to-perturb loss basins.
-
Optimization Hyper-parameter Laws for Large Language Models
Opt-Laws predicts LLM final training loss from LR schedules via SDE-derived convergence and escape features, with 94% Top-2 hit rate on held-out schedules and F1=0.92 for divergence detection.
-
SAMPLe: SAM-based Optimizer for Prompt Learning in VLMs
SAMPLe adds dual gradient constraints (ERM alignment plus full-batch orthogonality) to SAM-style prompt learning and raises harmonic-mean base-to-new accuracy across CoOp, CoCoOp, MaPLe, TCP and CoPrompt.
-
Improving Sparse-View 3DGS Generalization via Flat Minima Optimization
Adapts flat minima optimization to 3DGS via anisotropy-aware perturbations and periodic reinitialization to improve generalization under sparse-view supervision.
-
Flatness Preserves Instruction Following in Vision-Language-Action Models
Sharpness-aware minimization during VLA finetuning preserves instruction following and yields over 60% gains across simulation and real-world tasks.
-
GRAIN: Group Aggregation via Min-Norm Objective
GRAIN is a gradient aggregation method using min-norm objectives to ensure non-negative inner products with group gradients, yielding tighter uniform stability bounds than SGD under smoothness assumptions.
-
CoughPhase-CLR: Designing an acoustics-informed foundation model for coughing sound classification
CoughPhase-CLR uses cough physiological phases to build contrastive positive pairs, outperforming random cropping on downstream tasks including COVID-19 detection and COPD classification.
-
SHAPO: Sharpness-Aware Policy Optimization for Safe Exploration
SHAPO adds a sharpness-aware adjustment to policy optimization that reweights gradients to favor conservative behavior in uncertain areas, yielding better safety-performance tradeoffs on continuous control tasks.
-
FADRW: A Feature-Aware Modulated and Dynamically Reweighted Loss for Few-Shot Linguistic Steganalysis
FADRW is a loss function framework that applies dynamic reweighting to counter decision bias and feature-aware modulation to improve separability of subtle steganographic features in few-shot settings.
-
Hallucination-Aware Diffusion Sampling for Inverse Problems via Robust Prior Updates
RPU stabilizes the prior update in diffusion inverse solvers to reduce measurement-conditioned hallucinations, with reported gains on FFHQ and ImageNet.
-
Re-M3Dr: Rebalanced MultiModal Mean Deviation Regression
Re-M3Dr is a multimodal regression framework using adaptive-margin supervised contrastive learning and sharpness-aware gradient modulation to stabilize training and reduce MSE by 29% versus SOTA multimodal methods on clinical eye-imaging datasets for mean deviation prediction.
-
UniAlign: A Model-Agnostic Framework for Robust Network Traffic Classification under Distribution Shifts
UniAlign improves robustness of deep learning NTC models under distribution shifts via domain alignment fine-tuning and stable ensembling, yielding 2.51% accuracy and 2.71% F1 gains over standard training on three public datasets.
-
Don't Stop Me Yet: Sampling Loss Minima via Dissipative Riemannian Mechanics
DiMS is a physics-inspired dynamical sampler guaranteed to exactly sample reparameterization-invariant minimum level sets in neural network loss landscapes.
-
MER-DG: Modality-Entropy Regularization for Multimodal Domain Generalization
MER-DG applies modality-entropy regularization to reduce fusion overfitting in multimodal domain generalization, reporting average gains of 5% over standard fusion and 2% over prior methods on EPIC-Kitchens and HAC benchmarks.
-
Secure and Privacy-Preserving Vertical Federated Learning
Three optimized MPC protocols for privacy-preserving vertical federated learning that support global and global-local updates while reducing computation versus naive full-MPC delegation.
-
A Faster Path to Continual Learning
C-Flat Turbo accelerates continual learning by skipping redundant flatness gradients via direction-invariance observations and linear adaptive scheduling, delivering 1-1.25x speedup with comparable accuracy.
-
Wolkowicz-Styan Upper Bound on the Hessian Eigenspectrum for Cross-Entropy Loss in Nonlinear Smooth Neural Networks
A closed-form upper bound on the maximum Hessian eigenvalue of cross-entropy loss is derived for smooth nonlinear neural networks.
-
MOMO: Mars Orbital Model Foundation Model for Mars Orbital Applications
MOMO merges sensor-specific models from three Mars orbital instruments at matched validation loss stages to form a foundation model that outperforms ImageNet, Earth observation, sensor-specific, and supervised baselines on nine Mars-Bench tasks.
-
HTMuon: Improving Muon via Heavy-Tailed Spectral Correction
HTMuon modifies Muon to produce heavier-tailed updates and weight spectra via HT-SR theory, yielding up to 0.98 lower perplexity on LLaMA pretraining and serving as a plug-in for other Muon variants.