Pith. sign in

REVIEW 4 major objections 5 minor 6 cited by

Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefits

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a single exponential moving average of SGD weights, evaluated outside the training loop, yields models that generalize better and are more robust to label noise, more consistent across runs, better calibrated, and…

desk verdict Useful empirical study of EMA, but the headline gains are confounded by early stopping and BN recomputation given only to the EMA. read the letter →

arxiv 2411.18704 v1 pith:WD6D2JQT submitted 2024-11-27 cs.LG

Daniel Morales-Brotons , Thijs Vogels , Hadrien Hendrikx This is my paper · ORCID
classification cs.LG
keywords exponentialmovingaverageweightaveragingstochasticgradientdescentimplicitregularizationlabelnoisemodelcalibrationpredictionchurntransferlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks what happens if, instead of reporting the last iterate of momentum SGD, one keeps an exponential moving average (EMA) of the weights during training and evaluates that average. Across image classification benchmarks the EMA model consistently matches or beats the last-iterate model on test accuracy and loss, and it also improves robustness to noisy labels, consistency of predictions across random runs, calibration, and transfer of learned features to other datasets. The authors argue that averaging reduces stochastic noise in the parameters, so training can keep a higher learning rate longer and thereby retain the implicit regularization that high-noise updates provide. A practical consequence is that the final, low-learning-rate phase of a cosine-annealed schedule becomes largely redundant: early stopping the EMA around 70–75% of the training budget recovers or improves on the full-budget last-iterate solution. The paper therefore proposes EMA as a simple plug-in that improves several quality axes at once.

What carries the argument

The central object is the exponential moving average of the SGD iterates, defined by $x^{\mathrm{EMA}}_{t+1} = \alpha x^{\mathrm{EMA}}_t + (1-\alpha) x_{t+1}$, kept outside the training loop so it does not affect the trajectory. The decay $\alpha$ controls the averaging window; the paper keeps five parallel EMAs with $\alpha \in \{0.968, 0.984, 0.992, 0.996, 0.998\}$ in one training run and selects the best epoch on a validation split, which turns the window size and the effective learning-rate stopping point into one-shot hyperparameters. The mechanism that carries the argument is the interaction between averaging and stochastic noise: averaging suppresses the noise that otherwise forces the learning rate to decay, so the iterate can keep a high learning rate and retain implicit regularization. The paper also identifies batch normalization statistics as a second mechanism: when they are recomputed, the averaged weights tolerate much slower decays, removing the main constraint on EMA use.

What would settle it

Run a controlled comparison where momentum SGD is early-stopped at its own best validation epoch, with batch-norm statistics recomputed at that checkpoint, and compare against the EMA early-stopped the same way. If the accuracy, calibration, and churn gaps vanish or reverse, the claim that EMA solutions are better than last-iterate solutions would be refuted for that setup.

Watch

Extended reading notes

Core claim

The central claim is that EMA solutions are genuinely different from last-iterate SGD solutions, and that the difference is useful. Averaging weights over a window of updates reduces parameter noise, which lets training continue at a relatively high learning rate; the stochastic gradient noise then acts as an implicit regularizer that biases the model away from sharp minima and from memorizing wrong labels. The paper reports consistent test-accuracy gains over momentum SGD on CIFAR-10, CIFAR-100, and Tiny-ImageNet with ResNet-18, WideResNet-28-10, and VGG-16, and much larger gains under 40% human label noise, where a plain EMA reaches 65.15% on CIFAR-100N versus 55.5% for the SGD baseline. The same EMA models cut prediction churn by roughly a third to a half across settings, reduce expected calibration error, and improve linear-evaluation transfer accuracy on frozen features. The authors further show that Batch Norm statistics—not the averaged weights themselves—are the limiting factor for large averaging windows: recomputing BN statistics once after training lets slower decays ($\alpha = 0.998$) be used and improves generalization further.

Load-bearing premise

The comparison gives the EMA model early stopping on a validation set and a post-training recomputation of batch-norm statistics, while the SGD baseline is evaluated at the final epoch of the full cosine-annealed schedule without those advantages; if the baseline were given the same early stopping and batch-norm recomputation, the reported margins could shrink or disappear.

Editorial extensions

If this is right

  • EMA can replace part of the learning-rate decay: early stopping around 70–75% of a cosine-annealed budget gives the reported accuracy while sparing the final low-learning-rate phase.
  • Plain EMA is competitive with specialized label-noise methods; the CIFAR-100N result of 65.15% beats several published robust-training baselines without extra augmentation or label refinement.
  • EMA reduces prediction churn: across runs it cuts disagreement roughly from 18.8% to 11.7% on CIFAR-100/ResNet-18 and from 29.4% to 15.3% on Tiny-ImageNet.
  • EMA improves calibration and combines with temperature scaling: ECE drops substantially and the best results use both.
  • EMA features transfer better: frozen backbones from EMA-trained models improve linear-evaluation accuracy on other datasets, e.g., 57.78% versus 52.77% from Tiny-ImageNet to CIFAR-100.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The explanation in terms of noise-driven implicit regularization is not tested outside image classification; a natural extension is to check whether EMA retains its calibration and churn benefits on language or speech models, where the paper acknowledges results may differ.
  • If the mechanism is really keeping the learning rate high rather than averaging per se, then an equivalent baseline that decays the learning rate to the same effective value early and stops there should reproduce part of the gain; this is a testable separation the paper does not run.
  • EMA's early-training advantage suggests a cheap recipe for semi-supervised and self-supervised teachers: one could start distillation from the EMA at a much earlier epoch than current practice, saving compute while keeping teacher quality.
  • Since BN recomputation removes the constraint on the averaging window, systems that cache or periodically refresh BN statistics could push $\alpha$ very close to 1 and possibly exceed the reported gains; this is a direct but untested extrapolation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper presents an empirical study of the exponential moving average (EMA) of SGD weights applied outside the training loop. The authors propose a pipeline that maintains five parallel EMA models with different decay rates, uses cosine annealing with validation-based early stopping, and (for BatchNorm networks) recomputes BN statistics after training. They report that EMA models outperform the last-iterate SGD baseline in test accuracy and loss (Table 1), in robustness to 40% label noise (Table 2, Fig. 2), in prediction consistency across independent runs (Table 3), in transfer via linear evaluation (Table 4), and in calibration (Table 5). They also analyze EMA training dynamics, arguing that averaging reduces parameter noise, permits higher learning rates, acts as implicit regularization, and performs particularly well early in training, which they propose as an explanation for the success of EMA teachers. The paper concludes that EMA is a simple, effective plug-in that improves multiple properties of deep learning models.

Significance. If the reported effects are due to the averaging mechanism itself, the paper is significant: a one-line change to an SGD pipeline would improve generalization and several robustness properties at negligible computational cost. The experimental coverage is broad (three datasets, four architectures, a real-world label-noise benchmark, five quality metrics, three seeds), and the protocol is careful in several respects: hyperparameters are selected on a held-out 20% validation split, the test set is used only once, and per-seed values are reported in the appendix. The paper is also honest in reporting null or negative results (App. A.1 step decay, App. C bootstrapping, App. E.2 continued training). However, the central comparisons are confounded: the EMA receives validation-based early stopping and BN recomputation while the SGD baseline does not, and the authors themselves attribute the transfer gains to early stopping in Sec. 4.5. The significance of the headline claims is therefore conditional on additional control experiments.

major comments (4)
  1. [Sec. 4.1, App. B] The comparison protocol is asymmetric. The EMA is early-stopped at the epoch of best validation accuracy or lowest validation loss and its BN statistics are recomputed once after training, while the baseline is the last-iterate SGD model at the final epoch of the cosine schedule without BN recomputation. Appendix B shows that the EMA's best epoch is systematically earlier than the end of training (e.g., CIFAR-100 ResNet-18: 146 vs 198 epochs; Tiny-ImageNet: 101 vs 148). This conflates three effects: the running average of weights, early stopping, and BN recomputation. Since all headline claims (generalization, label-noise robustness, prediction consistency, calibration, transfer) are measured under this protocol, the paper currently establishes the benefits of the full EMA pipeline rather than of EMA per se. The authors should add a control baseline that is early-stopped on the same validation criterion and, for BN networks, evaluated with recomputed BN statistics.
  2. [Sec. 4.3, Fig. 2] The label-noise result (+9.65 pp on CIFAR-100N) compares the EMA at its best early-stopped epoch with the SGD baseline at the final epoch, when the baseline has largely memorized the noisy labels (App. E.1). The paper does not report the SGD baseline early-stopped at its best validation epoch, so the margin cannot be attributed to averaging rather than to avoiding the late memorization phase. The claim of 'robustness to noisy labels' requires this control before it can be accepted.
  3. [Sec. 4.5] The transfer-learning section states that the improvement is 'likely because of early stopping, which is also known as an effective form of implicit regularization.' This is a direct concession that the reported transfer gains may not be caused by the EMA mechanism. A control with an early-stopped SGD baseline is needed to separate the effect of averaging from the effect of stopping early.
  4. [Sec. 4.2, Table 1] The claim that 'EMA performs consistently better than the baseline' is not supported by the paper's own data: in Table 1, the VGG-16 EMA early-stopped at lowest loss has accuracy 72.3 ± 0.19, below the baseline of 72.82 ± 0.17. Combined with the protocol asymmetry, the generalization claim 'always outperforming the SGD baseline' is overstated and should be re-scoped to the specific pipeline or supplemented with early-stopped baseline numbers.
minor comments (5)
  1. [Sec. 3.1] The sentence 'An EMA naturally avoids the need for (1)' is a broken cross-reference: (1) is the EMA update equation, not the first item in the numbered list of tuning overheads. The intended reference should be to the first bullet about choosing the averaging window.
  2. [Table 3] The header 'WRN-2810' should be 'WRN-28-10'.
  3. [Sec. 4.3, Fig. 2] The text says the EMA peaks at 65.15% accuracy at epoch 100, which matches the BN-recomputed value in Table 18, but the figures in App. A.2 with the same setup are stated to be without BN recomputation. The figure caption should clarify whether BN statistics are recomputed in the plotted curves.
  4. [Sec. 4.6, Table 5] On Tiny-ImageNet with ResNet-18, the ECE after temperature scaling is higher for EMA (3.57 ± 0.29) than for the baseline (3.35 ± 0.11); the claim that combining temperature scaling and EMA 'generally' yields the best calibration is technically hedged, but this exception deserves an explicit mention.
  5. [App. B] The detailed tables include SGD columns without recomputed BN statistics, so even a reader who wants to decompose the BN effect is missing the necessary baseline control.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: EMA benefits are direct held-out measurements; the early-stop/BN-recompute asymmetry is an experimental confound, not a circular reduction.

full rationale

This is an empirical measurement paper rather than a derivation. The central claims (EMA improves generalization, robustness to label noise, prediction consistency, calibration, and transfer) are supported by test-set metrics computed from held-out evaluation, with EMA hyperparameters (decay rate and early-stopping epoch) selected on an 80/20 validation split and then a fresh training run on the full data before test evaluation (Sec. 4.1, App. F). No reported number is obtained by fitting the same quantity it is said to predict; the proposed implicit-regularization mechanism is a post-hoc explanation and does not enter the reported accuracies, losses, churn values, or calibration errors. There are no load-bearing self-citations: the cited works (e.g., Izmailov et al. 2018, Cai et al. 2021, Wei et al. 2022) are prior independent methods, benchmarks, or BN-statistic handling techniques. The asymmetry in which the EMA is early-stopped and receives BN recomputation while the SGD baseline does not is a genuine experimental-design concern and a correctness risk, but it is not circularity: the EMA numbers are still direct measurements, not a reduction of the claim to its own inputs. Accordingly, no circular step can be exhibited, and the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No invented entities. The central claims rest on standard hyperparameters tuned on a validation split and on several domain assumptions about experimental fairness. The main load-bearing assumptions are that early stopping and BN recomputation for EMA but not for SGD is a fair comparison, and that the 80/20 hold-out selection transfers to full training.

free parameters (3)
  • EMA decay α = 0.998 (set of 5 decays 0.968-0.998 for dynamics)
    Chosen from prior practice; the paper shows BN limits large windows. Main results use α=0.998 with T=16, effective α^(1/16).
  • Early stopping epoch for EMA = Varies per dataset (e.g., ~143 for ResNet-18 CIFAR-100, ~100 for Tiny-ImageNet)
    Tuned on 80/20 validation split; this is a key hyperparameter for the reported gains.
  • Initial learning rate = e.g., 0.4-1.2 for ResNet-18, 0.1 for WRN, 0.05 for VGG
    Tuned on validation; the paper reports the same optimal LR for SGD and EMA.
assumptions (4)
  • domain assumption The validation-based early stopping epoch transfers from 80% to 100% training data.
    All main results use this protocol; if the optimal epoch shifts, results are not representative.
  • domain assumption BN recomputation for EMA only is a fair comparison.
    Baseline gets no extra forward pass; this could inflate EMA gains.
  • domain assumption Three seeds are sufficient.
    Mean/std over 3 runs; no statistical test or power analysis.
  • standard math EMA update formula (Eq. 1) is a standard convex combination of iterates.
    Used throughout; no new math.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefits." pith.science (2026). https://pith.science/paper/WD6D2JQT

@misc{pith2026241118704,
  author       = {Pith},
  title        = {Pith review of: Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefits},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WD6D2JQT}},
  note         = {Machine review of arXiv:2411.18704}
}
read the original abstract

Weight averaging of Stochastic Gradient Descent (SGD) iterates is a popular method for training deep learning models. While it is often used as part of complex training pipelines to improve generalization or serve as a `teacher' model, weight averaging lacks proper evaluation on its own. In this work, we present a systematic study of the Exponential Moving Average (EMA) of weights. We first explore the training dynamics of EMA, give guidelines for hyperparameter tuning, and highlight its good early performance, partly explaining its success as a teacher. We also observe that EMA requires less learning rate decay compared to SGD since averaging naturally reduces noise, introducing a form of implicit regularization. Through extensive experiments, we show that EMA solutions differ from last-iterate solutions. EMA models not only generalize better but also exhibit improved i) robustness to noisy labels, ii) prediction consistency, iii) calibration and iv) transfer learning. Therefore, we suggest that an EMA of weights is a simple yet effective plug-in to improve the performance of deep learning models.

Figures

Figures reproduced from arXiv: 2411.18704 by the authors.

Figure 1
Figure 1. CIFAR-100 on ResNet-18. Left: EMA vs SGD baseline, and learning rate (η). EMA is the best among the 5 EMA models at any given epoch, without recomputing BN stats (i.e. the maximum among EMA models plotted on 1b). We observe that EMA dominates momentum SGD and has a good performance since early on. EMA peaks at epoch 150, at the optimal η, and then deteriorates. Right: Breakdown of the 5 EMA models per decay (with an… view at source ↗
Figure 2
Figure 2. CIFAR-100N on ResNet-34. EMA vs SGD baseline, and learning rate η. EMA dominates SGD throughout training and peaks at epoch 100 (η = 0.4), greatly outperforming the best SGD model (+9.65 pp). Training on data with 40% of label noise, evaluating on clean test set, mean of 3 runs, α = 0.998. Method CIFAR-10N CIFAR-100N DivideMix 92.56 ± 0.42 71.13 ± 0.48 PES(semi) 92.68 ± 0.22 70.36 ± 0.33 ELR+ 91.09 ± 1.6 66.70 ± 0.0… view at source ↗
Figure 3
Figure 3. CIFAR-100 on ResNet-18, with step decay of the learning [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: EMA and momentum SGD training dynamics for the different datasets and models used in our [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 5
Figure 5. Figure 5: CIFAR-100 on ResNet-18, training on 80% split and evaluation on hold-out 20% split. The EMA [PITH_FULL_IMAGE:figures/full_fig_p023_5.png]
Figure 6
Figure 6. Figure 6: SGD and EMA validation accuracy for different initial values of learning rate [PITH_FULL_IMAGE:figures/full_fig_p024_6.png]
Figure 7
Figure 7. Figure 7: CIFAR-100N on ResNet-34. Accuracy on training data during training, split into Noisy (40% of data wrong labels) Clean (remaining 60%). Both models end up by memorizing all of the noisy labels, but the EMA model fits less noise relative the accuracy on the clean samples…
Figure 8
Figure 8. Figure 8: Test Accuracy with constant learning rate after stopping epoch (for EMA (acc.)) vs with cosine [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]
Figure 9
Figure 9. Figure 9: Breakdown of the 5 EMA models per decay (with and without BN recomputation after every [PITH_FULL_IMAGE:figures/full_fig_p027_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 18 citations worldwide. Full citation record

  1. Exploring Line Bundle Standard Models with Transformers

    hep-th 2026-06 unverdicted novelty 7.0 of 10

    A Transformer trained by reinforcement learning generates heterotic line-bundle sums that satisfy anomaly-cancellation, stability, and chirality constraints, and its policy transfers usefully across Calabi-Yau geometries.

  2. Beyond Token-Level Cross-Entropy: Fr\'echet Distributional Post-Training for Autoregressive Image Generation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    FD-loss post-training with detached rollout replay and a probability-level straight-through estimator improves FID and FD_r6 across eight ImageNet configurations.

  3. A Generalisable Generative Model for Multi-Detector Calorimeter Simulation

    physics.ins-det 2025-09 conditional novelty 6.0 of 10

    CaloDiT-2 demonstrates that pre-training a transformer-based diffusion model on multiple calorimeter detectors enables 25x less data and 20x less training time when adapting to a new detector.

  4. DMAF-Net: An Effective Modality Rebalancing Framework for Incomplete Multi-Modal Medical Image Segmentation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    DMAF-Net improves incomplete multi-modal MRI segmentation by dynamically masking missing modalities, aligning uni-modal and fused features through covariance, attention, and prototype losses, and reweighting training ...

  5. FootsiesGym: A Fighting Game Benchmark for Two-Player Zero-Sum Imperfect-Information Games

    cs.AI 2026-07 accept novelty 5.0 of 10

    FootsiesGym is an open-source, vectorized fighting-game benchmark for two-player zero-sum imperfect-information RL that isolates non-transitive neutral-game dynamics while remaining tractable on standard hardware.

  6. Demonstration of Efficient Predictive Surrogates for Large-scale Quantum Processors

    quant-ph 2025-07 conditional novelty 5.0 of 10

    Classical surrogates using truncated trigonometric expansions emulate noisy quantum processors and cut measurement overhead in VQE pre-training and Floquet phase identification.

Reference graph

Works this paper leans on

21 extracted references · 9 canonical work pages · cited by 6 Pith papers

  1. [1]

    There are many consistent explanations of unlabeled data: Why you should average

    Ben Athiwaratkun, Marc Finzi, Pavel Izmailov, and Andrew Gordon Wilson. There are many consistent explanations of unlabeled data: Why you should average. In7th International Conference on Learning Representations, ICLR 2019,

    work page 2019

  2. [6]

    Uncovering the limits of adversarial training against norm-bounded adversarial examples.arXiv preprint arXiv:2010.03593,

    Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. Uncovering the limits of adversarial training against norm-bounded adversarial examples.arXiv preprint arXiv:2010.03593,

    arXiv 2010

  3. [10]

    On large-batch training for deep learning: Generalization gap and sharp minima.arXiv preprint arXiv:1609.04836,

    Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima.arXiv preprint arXiv:1609.04836,

    arXiv

  4. [14]

    Trainable weight averaging for fast convergence and better generalization.arXiv preprint arXiv:2205.13104,

    Tao Li, Zhehao Huang, Qinghua Tao, Yingwen Wu, and Xiaolin Huang. Trainable weight averaging for fast convergence and better generalization.arXiv preprint arXiv:2205.13104,

    arXiv

  5. [15]

    On model stability as a function of random seed.arXiv preprint arXiv:1909.10447,

    Pranava Madhyastha and Rishabh Jain. On model stability as a function of random seed.arXiv preprint arXiv:1909.10447,

    arXiv 1909

  6. [16]

    Self: Learning to filter noisy labels with self-ensembling

    Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. Self: Learning to filter noisy labels with self-ensembling. arXiv preprint arXiv:1910.01842,

    arXiv 1910

  7. [17]

    Early weight averaging meets high learning rates for llm pre-training,

    14 Published in Transactions on Machine Learning Research (04/2024) Sunny Sanyal, Atula Neerkaje, Jean Kaddour, Abhishek Kumar, and Sujay Sanghavi. Early weight averaging meets high learning rates for llm pre-training,

    work page 2024

  8. [18]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556,

    arXiv

Show all 21 references
  1. [19]

    Wide residual networks.arXiv preprint arXiv:1605.07146,

    Sergey Zagoruyko and Nikos Komodakis. Wide residual networks.arXiv preprint arXiv:1605.07146,

    arXiv

  2. [20]

    15 Published in Transactions on Machine Learning Research (04/2024) A Additional examples of EMA training dynamics In this section, we provide additional examples of EMA training dynamics, always compared to the momentum SGD baseline, as described in Sec. 4.1. We first discuss...

    2024

  3. [21]

    26 Published in Transactions on Machine Learning Research (04/2024) G Sensitivity analysis to EMA decay rateα The decay rateα is a key hyperparameter in EMA models

    T = 1 T = 16 0.999875 0.998 0.99975 0.996 0.9995 0.992 0.999 0.984 0.998 0.968 Table 23: Equivalence of EMA decay rateα for different sampling periods. 26 Published in Transactions on Machine Learning Research (04/2024) G Sensitivity analysis to EMA decay rateα The decay rateα...

    2024

  4. [2014]

    Temporal ensembling for semi-supervised learning

    Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242,

    arXiv

  5. [2015]

    13 Published in Transactions on Machine Learning Research (04/2024) Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In International Conference on Learning Representations,

    2024

  6. [2016]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

    arXiv

  7. [2017]

    (s) gd over diagonal linear networks: Implicit regularisation, large stepsizes and edge of stability.arXiv preprint arXiv:2302.08982,

    Mathieu Even, Scott Pesme, Suriya Gunasekar, and Nicolas Flammarion. (s) gd over diagonal linear networks: Implicit regularisation, large stepsizes and edge of stability.arXiv preprint arXiv:2302.08982,

    arXiv

  8. [2018]

    Churn reduction via distillation.arXiv preprint arXiv:2106.02654, 2021a

    Heinrich Jiang, Harikrishna Narasimhan, Dara Bahri, Andrew Cotter, and Afshin Rostamizadeh. Churn reduction via distillation.arXiv preprint arXiv:2106.02654, 2021a. Yiding Jiang, Vaishnavh Nagarajan, Christina Baek, and J Zico Kolter. Assessing generalization of sgd via disagr...

    arXiv

  9. [2019]

    On the reproducibility of neural network predictions.arXiv preprint arXiv:2102.03349,

    Srinadh Bhojanapalli, Kimberly Wilber, Andreas Veit, Ankit Singh Rawat, Seungyeon Kim, Aditya Menon, and Sanjiv Kumar. On the reproducibility of neural network predictions.arXiv preprint arXiv:2102.03349,

    arXiv

  10. [2020]

    On calibration of modern neural networks

    12 Published in Transactions on Machine Learning Research (04/2024) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321–1330. PMLR,

    2024

  11. [2021]

    Large-scale machine learning with stochastic gradient descent

    Léon Bottou. Large-scale machine learning with stochastic gradient descent. InProceedings of COMP- STAT’2010: 19th International Conference on Computational StatisticsParis France, August 22-27, 2010 Keynote, Invited and Contributed Papers, pp. 177–186. Springer,

    2010

  12. [2022]

    Averaging weights leads to wider optima and better generalization

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, pp. 876–885. Association For Uncertainty ...

    2018

  13. [2023]

    Semi-supervised semantic segmentation needs strong, varied perturbations.arXiv preprint arXiv:1906.01916,

    Geoff French, Samuli Laine, Timo Aila, Michal Mackiewicz, and Graham Finlayson. Semi-supervised semantic segmentation needs strong, varied perturbations.arXiv preprint arXiv:1906.01916,

    1906 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.