Neural network-parameterized regression splines enable joint optimization of forecast quality and stability in distribution-free probabilistic time series models by penalizing dissimilarities from forecast updates.
hub
Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefits
14 Pith papers cite this work, alongside 18 external citations. Polarity classification is still indexing.
14
Pith papers citing it
18
external citations · Pith
abstract
Weight averaging of Stochastic Gradient Descent (SGD) iterates is a popular method for training deep learning models. While it is often used as part of complex training pipelines to improve generalization or serve as a `teacher' model, weight averaging lacks proper evaluation on its own. In this work, we present a systematic study of the Exponential Moving Average (EMA) of weights. We first explore the training dynamics of EMA, give guidelines for hyperparameter tuning, and highlight its good early performance, partly explaining its success as a teacher. We also observe that EMA requires less learning rate decay compared to SGD since averaging naturally reduces noise, introducing a form of implicit regularization. Through extensive experiments, we show that EMA solutions differ from last-iterate solutions. EMA models not only generalize better but also exhibit improved i) robustness to noisy labels, ii) prediction consistency, iii) calibration and iv) transfer learning. Therefore, we suggest that an EMA of weights is a simple yet effective plug-in to improve the performance of deep learning models.
hub tools
citation-role summary
background 3
method 1
citation-polarity summary
representative citing papers
Equivariant neural networks produce dipole and polarizability surfaces for methanol that enable variational computation of vibrational IR and Raman spectra agreeing with experiment to 2.2 cm^{-1} RMSD on fundamentals.
Distribution-wise rewards with subset-replace strategy and post-hoc merging improve FID-50K on SiT (8.30 to 5.77) and EDM2 (3.74 to 3.52) while preserving diversity.
EMAgnet replaces uniform-magnet regularization in PPO self-play with an EMA of last-iterate policy parameters and reports lower exploitability on most tested zero-sum benchmarks, especially those with dominated strategies.
UniGraphLM uses a multi-domain multi-task GNN encoder and adaptive alignment to create unified graph tokens for LLMs across diverse domains and tasks.
The Spatial Adapter equips frozen predictors with a spatially regularized orthonormal basis for residuals and derives a closed-form low-rank-plus-noise covariance for spatial prediction and kriging.
VISTA adaptively tunes consistency thresholds in decentralized SGD so that the system converges asymptotically like standard SGD even when adversaries dominate the worker pool.
MC-GenRef performs annotation-free microcalcification segmentation via synthetic data from a lightweight image formation model plus test-time generative posterior refinement with a rectified-flow generator, yielding top Dice on INbreast and gains on an external cohort.
LayerPipe2 derives per-layer delay assignments for multistage pipelined training and uses an improved moving average to recompute past weights without explicit storage.
FootsiesGym is an open-source, vectorized fighting-game benchmark for two-player zero-sum imperfect-information RL that isolates non-transitive neutral-game dynamics while remaining tractable on standard hardware.
A Transformer trained by reinforcement learning generates heterotic line-bundle sums that satisfy anomaly-cancellation, stability, and chirality constraints, and its policy transfers usefully across Calabi-Yau geometries.
Exact critic in entropy-regularized actor-critic yields strong variance reduction, enabling Õ(log(1/ε)) sample complexity for ε-optimal regularized value.
A domain-adapted diffusion model synthesizes heterogeneous PET images from uniform organ activity maps, achieving high quantitative accuracy (CCC > 0.92) and visual realism comparable to real scans.
Observability-constrained test-time prompt tuning for LiDAR semantic segmentation reweights spatial supervision using per-location reliability estimates from beam terminations and neighborhood support, with prompt adapters and temporal prototype alignment.
citing papers explorer
-
Stabilizing distribution-free probabilistic forecasts
Neural network-parameterized regression splines enable joint optimization of forecast quality and stability in distribution-free probabilistic time series models by penalizing dissimilarities from forecast updates.
-
Vibrational infrared and Raman spectra of the methanol molecule with equivariant neural-network property surfaces
Equivariant neural networks produce dipole and polarizability surfaces for methanol that enable variational computation of vibrational IR and Raman spectra agreeing with experiment to 2.2 cm^{-1} RMSD on fundamentals.
-
Optimizing Visual Generative Models via Distribution-wise Rewards
Distribution-wise rewards with subset-replace strategy and post-hoc merging improve FID-50K on SiT (8.30 to 5.77) and EDM2 (3.74 to 3.52) while preserving diversity.
-
EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Games
EMAgnet replaces uniform-magnet regularization in PPO self-play with an EMA of last-iterate policy parameters and reports lower exploitability on most tested zero-sum benchmarks, especially those with dominated strategies.
-
A Unified Graph Language Model for Multi-Domain Multi-Task Graph Alignment Instruction Tuning
UniGraphLM uses a multi-domain multi-task GNN encoder and adaptive alignment to create unified graph tokens for LLMs across diverse domains and tasks.
-
Spatial Adapter: Structured Spatial Decomposition and Closed-Form Covariance for Frozen Predictors
The Spatial Adapter equips frozen predictors with a spatially regularized orthonormal basis for residuals and derives a closed-form low-rank-plus-noise covariance for spatial prediction and kriging.
-
\mathsf{VISTA}: Decentralized Machine Learning in Adversary Dominated Environments
VISTA adaptively tunes consistency thresholds in decentralized SGD so that the system converges asymptotically like standard SGD even when adversaries dominate the worker pool.
-
MC-GenRef: Annotation-free mammography microcalcification segmentation with generative posterior refinement
MC-GenRef performs annotation-free microcalcification segmentation via synthetic data from a lightweight image formation model plus test-time generative posterior refinement with a rectified-flow generator, yielding top Dice on INbreast and gains on an external cohort.
-
LayerPipe2: Multistage Pipelining and Weight Recompute via Improved Exponential Moving Average for Training Neural Networks
LayerPipe2 derives per-layer delay assignments for multistage pipelined training and uses an improved moving average to recompute past weights without explicit storage.
-
FootsiesGym: A Fighting Game Benchmark for Two-Player Zero-Sum Imperfect-Information Games
FootsiesGym is an open-source, vectorized fighting-game benchmark for two-player zero-sum imperfect-information RL that isolates non-transitive neutral-game dynamics while remaining tractable on standard hardware.
-
Exploring Line Bundle Standard Models with Transformers
A Transformer trained by reinforcement learning generates heterotic line-bundle sums that satisfy anomaly-cancellation, stability, and chirality constraints, and its policy transfers usefully across Calabi-Yau geometries.
-
Refined Analysis of Entropy-Regularized Actor-Critic
Exact critic in entropy-regularized actor-critic yields strong variance reduction, enabling Õ(log(1/ε)) sample complexity for ε-optimal regularized value.
-
Generation of Heterogeneous PET Images from Uniform Organ Activity Maps Using a Pretrained Domain-Adapted Diffusion Model
A domain-adapted diffusion model synthesizes heterogeneous PET images from uniform organ activity maps, achieving high quantitative accuracy (CCC > 0.92) and visual realism comparable to real scans.
-
No Adaptation Without Observation: Observability-Constrained Test-Time Prompt Tuning for LiDAR Semantic Segmentation
Observability-constrained test-time prompt tuning for LiDAR semantic segmentation reweights spatial supervision using per-location reliability estimates from beam terminations and neighborhood support, with prompt adapters and temporal prototype alignment.