Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefits
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:WD6D2JQTrecord.jsonopen to challenge →
read the original abstract
Weight averaging of Stochastic Gradient Descent (SGD) iterates is a popular method for training deep learning models. While it is often used as part of complex training pipelines to improve generalization or serve as a `teacher' model, weight averaging lacks proper evaluation on its own. In this work, we present a systematic study of the Exponential Moving Average (EMA) of weights. We first explore the training dynamics of EMA, give guidelines for hyperparameter tuning, and highlight its good early performance, partly explaining its success as a teacher. We also observe that EMA requires less learning rate decay compared to SGD since averaging naturally reduces noise, introducing a form of implicit regularization. Through extensive experiments, we show that EMA solutions differ from last-iterate solutions. EMA models not only generalize better but also exhibit improved i) robustness to noisy labels, ii) prediction consistency, iii) calibration and iv) transfer learning. Therefore, we suggest that an EMA of weights is a simple yet effective plug-in to improve the performance of deep learning models.
This paper has not been read by Pith yet.
Forward citations
Cited by 14 Pith papers
-
Exploring Line Bundle Standard Models with Transformers
A Transformer RL agent is trained to generate valid heterotic line bundle sums on CICYs that satisfy gauge embedding, anomaly cancellation, poly-stability, chirality, and no-exotics constraints.
-
Stabilizing distribution-free probabilistic forecasts
Neural network-parameterized regression splines enable joint optimization of forecast quality and stability in distribution-free probabilistic time series models by penalizing dissimilarities from forecast updates.
-
Vibrational infrared and Raman spectra of the methanol molecule with equivariant neural-network property surfaces
Equivariant neural networks produce dipole and polarizability surfaces for methanol that enable variational computation of vibrational IR and Raman spectra agreeing with experiment to 2.2 cm^{-1} RMSD on fundamentals.
-
Optimizing Visual Generative Models via Distribution-wise Rewards
Distribution-wise rewards with subset-replace strategy and post-hoc merging improve FID-50K on SiT (8.30 to 5.77) and EDM2 (3.74 to 3.52) while preserving diversity.
-
EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Games
EMAgnet replaces uniform-magnet regularization in PPO self-play with an EMA of last-iterate policy parameters and reports lower exploitability on most tested zero-sum benchmarks, especially those with dominated strategies.
-
A Unified Graph Language Model for Multi-Domain Multi-Task Graph Alignment Instruction Tuning
UniGraphLM uses a multi-domain multi-task GNN encoder and adaptive alignment to create unified graph tokens for LLMs across diverse domains and tasks.
-
Spatial Adapter: Structured Spatial Decomposition and Closed-Form Covariance for Frozen Predictors
The Spatial Adapter equips frozen predictors with a spatially regularized orthonormal basis for residuals and derives a closed-form low-rank-plus-noise covariance for spatial prediction and kriging.
-
\mathsf{VISTA}: Decentralized Machine Learning in Adversary Dominated Environments
VISTA adaptively tunes consistency thresholds in decentralized SGD so that the system converges asymptotically like standard SGD even when adversaries dominate the worker pool.
-
MC-GenRef: Annotation-free mammography microcalcification segmentation with generative posterior refinement
MC-GenRef performs annotation-free microcalcification segmentation via synthetic data from a lightweight image formation model plus test-time generative posterior refinement with a rectified-flow generator, yielding t...
-
LayerPipe2: Multistage Pipelining and Weight Recompute via Improved Exponential Moving Average for Training Neural Networks
LayerPipe2 derives per-layer delay assignments for multistage pipelined training and uses an improved moving average to recompute past weights without explicit storage.
-
FootsiesGym: A Fighting Game Benchmark for Two-Player Zero-Sum Imperfect-Information Games
FootsiesGym is an open-source, vectorized fighting-game benchmark for two-player zero-sum imperfect-information RL that isolates non-transitive neutral-game dynamics while remaining tractable on standard hardware.
-
Refined Analysis of Entropy-Regularized Actor-Critic
Exact critic in entropy-regularized actor-critic yields strong variance reduction, enabling Õ(log(1/ε)) sample complexity for ε-optimal regularized value.
-
Generation of Heterogeneous PET Images from Uniform Organ Activity Maps Using a Pretrained Domain-Adapted Diffusion Model
A domain-adapted diffusion model synthesizes heterogeneous PET images from uniform organ activity maps, achieving high quantitative accuracy (CCC > 0.92) and visual realism comparable to real scans.
-
No Adaptation Without Observation: Observability-Constrained Test-Time Prompt Tuning for LiDAR Semantic Segmentation
Observability-constrained test-time prompt tuning for LiDAR semantic segmentation reweights spatial supervision using per-location reliability estimates from beam terminations and neighborhood support, with prompt ada...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.