{"id":"d60a85c7-9be3-4c37-ab38-c9e8aba188b3","arxiv_id":"2411.18704","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An exponential moving average of weights consistently improves generalization, label-noise robustness, prediction consistency, calibration, and transfer learning for image classifiers.","lead":"An empirical study shows that an exponential moving average of neural network weights, a simple plug-in, improves generalization, robustness to noisy labels, prediction consistency, calibration, and transfer learning compared with ordinary SGD. It maps EMA dynamics and hyperparameter choices, explaining why EMA works well as a teacher and as a final model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EMA's reported gains are confounded by early stopping and BN recomputation that only the EMA receives; without a baseline control for both, the central benefits claim is not yet established.","rationale":"The reader's conditional verdict and its identified weakest assumption match my own reading. The paper is careful in several respects: it uses an 80/20 validation split, reports three seeds, compares against SWA, and provides extensive appendix tables. Those strengths make the absence of the obvious control more conspicuous. The concern is not that the results are fabricated; they are probably reproducible as run. The issue is that the protocol varies two factors at once, EMA versus no EMA and early-stopped-plus-BN-recomputed versus full-budget-plus-running-BN-stats, so the causal attribution to EMA is not identified. The label-noise section is where the confound is most damaging, because cosine decay makes the late-phase SGD baseline overfit noisy labels in a way that early stopping would directly address. A single controlled rerun would settle the matter. I therefore recommend keeping the reader's CONDITIONAL verdict: the strong abstract claim should not be accepted at face value until the baseline is given the same early stopping and BN recomputation. If the controlled rerun erases the margins, the appropriate verdict would be REJECT or a substantially narrowed claim; if the margins persist, the paper's practical recommendation survives.","tokens_in":24842,"tokens_out":6965,"duration_ms":73072,"concrete_test":"Run the same experiments (Tables 1, 2, 4, 5) with an SGD baseline that is early-stopped at its best validation epoch under the identical cosine schedule and given the same one-time BN-statistic recomputation, with hyperparameters selected on the same 80/20 split. If the EMA-minus-SGD margins in accuracy, label-noise robustness, churn, transfer, and ECE do not survive this control, the central claim reduces to a comparison with a deliberately undertrained baseline. A minimal version for the most sensitive result: for CIFAR-100N, report the best epoch-wise validation accuracy of the SGD baseline; if it approaches the EMA peak of 65.15% at epoch 100, the robustness advantage is largely early stopping, not EMA.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that an EMA of SGD weights 'not only generalize[s] better but also exhibit[s] improved robustness to noisy labels, prediction consistency, calibration and transfer learning' (Abstract). The protocol that supports this claim (Sec. 4.1, App. B) is not symmetric. The EMA is early-stopped at the best validation accuracy/loss epoch, which App. B shows is always before about 75% of the budget, and it is then evaluated with recomputed Batch Norm statistics; the SGD baseline is evaluated at the final epoch of the cosine schedule without BN recomputation. This conflates the averaging mechanism with two other known regularizers. The paper itself concedes in Sec. 4.5 that its transfer improvements are 'likely because of early stopping, which is also known as an effective form of implicit regularization,' and App. A.1 shows that with step decay, EMA does not outperform the baseline. Appendix B confirms the asymmetry: for CIFAR-100 ResNet-18 the EMA best-accuracy epoch is around 146 while the baseline epoch is 198, and for Tiny-ImageNet the epochs are 101 versus 148. If the baseline were early-stopped at its best validation epoch and given the same BN recomputation, the reported margins, including the +9.65 pp on CIFAR-100N in Fig. 2, could shrink substantially. Since every headline benefit is measured under this asymmetric protocol, the most load-bearing assumption is the fairness of the comparison, not a modeling choice.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an empirical study of the exponential moving average (EMA) of SGD weights applied outside the training loop. The authors propose a pipeline that maintains five parallel EMA models with different decay rates, uses cosine annealing with validation-based early stopping, and (for BatchNorm networks) recomputes BN statistics after training. They report that EMA models outperform the last-iterate SGD baseline in test accuracy and loss (Table 1), in robustness to 40% label noise (Table 2, Fig. 2), in prediction consistency across independent runs (Table 3), in transfer via linear evaluation (Table 4), and in calibration (Table 5). They also analyze EMA training dynamics, arguing that averaging reduces parameter noise, permits higher learning rates, acts as implicit regularization, and performs particularly well early in training, which they propose as an explanation for the success of EMA teachers. The paper concludes that EMA is a simple, effective plug-in that improves multiple properties of deep learning models.","tokens_in":25124,"tokens_out":7180,"duration_ms":59245,"significance":"If the reported effects are due to the averaging mechanism itself, the paper is significant: a one-line change to an SGD pipeline would improve generalization and several robustness properties at negligible computational cost. The experimental coverage is broad (three datasets, four architectures, a real-world label-noise benchmark, five quality metrics, three seeds), and the protocol is careful in several respects: hyperparameters are selected on a held-out 20% validation split, the test set is used only once, and per-seed values are reported in the appendix. The paper is also honest in reporting null or negative results (App. A.1 step decay, App. C bootstrapping, App. E.2 continued training). However, the central comparisons are confounded: the EMA receives validation-based early stopping and BN recomputation while the SGD baseline does not, and the authors themselves attribute the transfer gains to early stopping in Sec. 4.5. The significance of the headline claims is therefore conditional on additional control experiments.","major_comments":[{"comment":"The comparison protocol is asymmetric. The EMA is early-stopped at the epoch of best validation accuracy or lowest validation loss and its BN statistics are recomputed once after training, while the baseline is the last-iterate SGD model at the final epoch of the cosine schedule without BN recomputation. Appendix B shows that the EMA's best epoch is systematically earlier than the end of training (e.g., CIFAR-100 ResNet-18: 146 vs 198 epochs; Tiny-ImageNet: 101 vs 148). This conflates three effects: the running average of weights, early stopping, and BN recomputation. Since all headline claims (generalization, label-noise robustness, prediction consistency, calibration, transfer) are measured under this protocol, the paper currently establishes the benefits of the full EMA pipeline rather than of EMA per se. The authors should add a control baseline that is early-stopped on the same validation criterion and, for BN networks, evaluated with recomputed BN statistics.","section":"Sec. 4.1, App. B"},{"comment":"The label-noise result (+9.65 pp on CIFAR-100N) compares the EMA at its best early-stopped epoch with the SGD baseline at the final epoch, when the baseline has largely memorized the noisy labels (App. E.1). The paper does not report the SGD baseline early-stopped at its best validation epoch, so the margin cannot be attributed to averaging rather than to avoiding the late memorization phase. The claim of 'robustness to noisy labels' requires this control before it can be accepted.","section":"Sec. 4.3, Fig. 2"},{"comment":"The transfer-learning section states that the improvement is 'likely because of early stopping, which is also known as an effective form of implicit regularization.' This is a direct concession that the reported transfer gains may not be caused by the EMA mechanism. A control with an early-stopped SGD baseline is needed to separate the effect of averaging from the effect of stopping early.","section":"Sec. 4.5"},{"comment":"The claim that 'EMA performs consistently better than the baseline' is not supported by the paper's own data: in Table 1, the VGG-16 EMA early-stopped at lowest loss has accuracy 72.3 ± 0.19, below the baseline of 72.82 ± 0.17. Combined with the protocol asymmetry, the generalization claim 'always outperforming the SGD baseline' is overstated and should be re-scoped to the specific pipeline or supplemented with early-stopped baseline numbers.","section":"Sec. 4.2, Table 1"}],"minor_comments":[{"comment":"The sentence 'An EMA naturally avoids the need for (1)' is a broken cross-reference: (1) is the EMA update equation, not the first item in the numbered list of tuning overheads. The intended reference should be to the first bullet about choosing the averaging window.","section":"Sec. 3.1"},{"comment":"The header 'WRN-2810' should be 'WRN-28-10'.","section":"Table 3"},{"comment":"The text says the EMA peaks at 65.15% accuracy at epoch 100, which matches the BN-recomputed value in Table 18, but the figures in App. A.2 with the same setup are stated to be without BN recomputation. The figure caption should clarify whether BN statistics are recomputed in the plotted curves.","section":"Sec. 4.3, Fig. 2"},{"comment":"On Tiny-ImageNet with ResNet-18, the ECE after temperature scaling is higher for EMA (3.57 ± 0.29) than for the baseline (3.35 ± 0.11); the claim that combining temperature scaling and EMA 'generally' yields the best calibration is technically hedged, but this exception deserves an explicit mention.","section":"Sec. 4.6, Table 5"},{"comment":"The detailed tables include SGD columns without recomputed BN statistics, so even a reader who wants to decompose the BN effect is missing the necessary baseline control.","section":"App. B"}],"recommendation":"major_revision","confidential_remarks":"The central problem is that the protocol asymmetry undermines all four headline benefits, and the authors' own Sec. 4.5 concession about early stopping is the clearest evidence of the confound. That said, the paper is not fundamentally unsound: the pipeline (EMA + early stopping + BN recomputation) may well be a useful practical recipe, and the authors report enough detail in the appendix to allow the missing control experiments to be run. I would not reject, but the claims need to be either re-scoped to the full pipeline or supported by early-stopped SGD baselines with identical BN treatment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a systematic empirical study of exponential moving averages of SGD weights, evaluated outside the training loop. It does several things well: it separates EMA from the student-teacher pipelines where it usually appears; it proposes a practical one-shot tuning scheme (run several decays in parallel, pick the best on a hold-out, early-stop); and it identifies Batch Norm statistics, rather than the weights, as the main constraint on long averaging windows (Fig. 1b and Table 24). The evaluation covers three datasets, three architectures, and four metrics beyond accuracy, with 3 seeds and a held-out validation set for hyperparameters. The related work is broad and fair, and the authors are upfront about the image-classification-only scope (Sec. 5) and about the step-decay case where EMA does not beat the baseline (App. A.1).\n\nThe soft spot is the comparison protocol. The EMA is early-stopped at its best validation epoch (about 75% of the budget) and gets BN stats recomputed; the SGD baseline runs to the end of the cosine schedule and does not get BN recomputation. So every headline benefit — generalization, label noise, churn, calibration, transfer — is measured under an asymmetric protocol. The transfer section explicitly attributes part of the gain to early stopping, and the label-noise gain of +9.65 pp is exactly the kind of number that could shrink if the baseline were also early-stopped. The dynamics plots also pick the best of five decays for the EMA curve rather than a single pre-registered decay, which flatters the EMA envelope.\n\nWhat is left standing? The BN finding is solid and useful, the early-training accuracy of EMA is a real observation, and the churn reduction, while probably not as large as claimed, points in a plausible direction because averaging over seeds naturally reduces variance. But the quantitative margins should be read as upper bounds.\n\nThis paper deserves a serious referee and would be a good reading-group piece precisely because the confound is instructive. My recommendation: engage with it, but require a symmetric baseline (early-stopped, BN-recomputed) before trusting the central 'EMA dominates SGD' claim. The qualitative dynamics and BN analysis are worth keeping regardless.","headline":"Useful empirical study of EMA, but the headline gains are confounded by early stopping and BN recomputation given only to the EMA.","tokens_in":25658,"tokens_out":3453,"would_cite":true,"duration_ms":32673,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a single exponential moving average of SGD weights, evaluated outside the training loop, yields models that generalize better and are more robust to label noise, more consistent across runs, better calibrated, and…","keywords":["exponential moving average","weight averaging","stochastic gradient descent","implicit regularization","label noise","model calibration","prediction churn","transfer learning"],"falsifier":"Run a controlled comparison where momentum SGD is early-stopped at its own best validation epoch, with batch-norm statistics recomputed at that checkpoint, and compare against the EMA early-stopped the same way. If the accuracy, calibration, and churn gaps vanish or reverse, the claim that EMA solutions are better than last-iterate solutions would be refuted for that setup.","tokens_in":24638,"feed_emoji":"🔁","tokens_out":6383,"duration_ms":53522,"temperature":0.7,"pith_summary":"This paper asks what happens if, instead of reporting the last iterate of momentum SGD, one keeps an exponential moving average (EMA) of the weights during training and evaluates that average. Across image classification benchmarks the EMA model consistently matches or beats the last-iterate model on test accuracy and loss, and it also improves robustness to noisy labels, consistency of predictions across random runs, calibration, and transfer of learned features to other datasets. The authors argue that averaging reduces stochastic noise in the parameters, so training can keep a higher learning rate longer and thereby retain the implicit regularization that high-noise updates provide. A practical consequence is that the final, low-learning-rate phase of a cosine-annealed schedule becomes largely redundant: early stopping the EMA around 70–75% of the training budget recovers or improves on the full-budget last-iterate solution. The paper therefore proposes EMA as a simple plug-in that improves several quality axes at once.","feed_headline":"EMA weights beat plain SGD on accuracy, noise, and calibration","feed_subtitle":"A simple running average of SGD weights adds robustness, consistency, calibration, and transfer on top of accuracy.","key_machinery":"The central object is the exponential moving average of the SGD iterates, defined by $x^{\\mathrm{EMA}}_{t+1} = \\alpha x^{\\mathrm{EMA}}_t + (1-\\alpha) x_{t+1}$, kept outside the training loop so it does not affect the trajectory. The decay $\\alpha$ controls the averaging window; the paper keeps five parallel EMAs with $\\alpha \\in \\{0.968, 0.984, 0.992, 0.996, 0.998\\}$ in one training run and selects the best epoch on a validation split, which turns the window size and the effective learning-rate stopping point into one-shot hyperparameters. The mechanism that carries the argument is the interaction between averaging and stochastic noise: averaging suppresses the noise that otherwise forces the learning rate to decay, so the iterate can keep a high learning rate and retain implicit regularization. The paper also identifies batch normalization statistics as a second mechanism: when they are recomputed, the averaged weights tolerate much slower decays, removing the main constraint on EMA use.","core_discovery":"The central claim is that EMA solutions are genuinely different from last-iterate SGD solutions, and that the difference is useful. Averaging weights over a window of updates reduces parameter noise, which lets training continue at a relatively high learning rate; the stochastic gradient noise then acts as an implicit regularizer that biases the model away from sharp minima and from memorizing wrong labels. The paper reports consistent test-accuracy gains over momentum SGD on CIFAR-10, CIFAR-100, and Tiny-ImageNet with ResNet-18, WideResNet-28-10, and VGG-16, and much larger gains under 40% human label noise, where a plain EMA reaches 65.15% on CIFAR-100N versus 55.5% for the SGD baseline. The same EMA models cut prediction churn by roughly a third to a half across settings, reduce expected calibration error, and improve linear-evaluation transfer accuracy on frozen features. The authors further show that Batch Norm statistics—not the averaged weights themselves—are the limiting factor for large averaging windows: recomputing BN statistics once after training lets slower decays ($\\alpha = 0.998$) be used and improves generalization further.","pith_inferences":["The explanation in terms of noise-driven implicit regularization is not tested outside image classification; a natural extension is to check whether EMA retains its calibration and churn benefits on language or speech models, where the paper acknowledges results may differ.","If the mechanism is really keeping the learning rate high rather than averaging per se, then an equivalent baseline that decays the learning rate to the same effective value early and stops there should reproduce part of the gain; this is a testable separation the paper does not run.","EMA's early-training advantage suggests a cheap recipe for semi-supervised and self-supervised teachers: one could start distillation from the EMA at a much earlier epoch than current practice, saving compute while keeping teacher quality.","Since BN recomputation removes the constraint on the averaging window, systems that cache or periodically refresh BN statistics could push $\\alpha$ very close to 1 and possibly exceed the reported gains; this is a direct but untested extrapolation."],"forward_implications":["EMA can replace part of the learning-rate decay: early stopping around 70–75% of a cosine-annealed budget gives the reported accuracy while sparing the final low-learning-rate phase.","Plain EMA is competitive with specialized label-noise methods; the CIFAR-100N result of 65.15% beats several published robust-training baselines without extra augmentation or label refinement.","EMA reduces prediction churn: across runs it cuts disagreement roughly from 18.8% to 11.7% on CIFAR-100/ResNet-18 and from 29.4% to 15.3% on Tiny-ImageNet.","EMA improves calibration and combines with temperature scaling: ECE drops substantially and the best results use both.","EMA features transfer better: frozen backbones from EMA-trained models improve linear-evaluation accuracy on other datasets, e.g., 57.78% versus 52.77% from Tiny-ImageNet to CIFAR-100."],"supporting_citations":[{"why":"Defines stochastic weight averaging and the flat-minima argument that this work compares EMA against.","marker":"Izmailov et al., 2018"},{"why":"Provides the classical theory showing iterate averaging reduces stochastic noise, the mechanism the paper relies on.","marker":"Polyak & Juditsky, 1992"},{"why":"Introduced the EMA teacher in Mean Teacher, the student-teacher setting the paper's early-performance result explains.","marker":"Tarvainen & Valpola, 2017"},{"why":"Uses an EMA teacher in BYOL; the paper cites this as a framework that motivated the study of EMA dynamics.","marker":"Grill et al., 2020"},{"why":"Supplies the moving-average batch-norm statistics procedure the paper adopts for its EMA models.","marker":"Cai et al., 2021"},{"why":"Provides the CIFAR-10N/CIFAR-100N noisy-label datasets and the leaderboard of specialized methods EMA is compared against.","marker":"Wei et al., 2022"},{"why":"The cosine annealing schedule the paper uses to sweep learning rate and early-stop the EMA.","marker":"Loshchilov & Hutter, 2017"},{"why":"The co-distillation method and churn measurements used as a baseline for prediction consistency.","marker":"Bhojanapalli et al., 2021"}],"fun_headline_variants":["EMA of weights: simple trick that beats last-iterate SGD","Average your weights for better accuracy, noise, calibration","Weight averaging: implicit regularization from SGD noise","EMA: robust models beyond the last checkpoint"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison gives the EMA model early stopping on a validation set and a post-training recomputation of batch-norm statistics, while the SGD baseline is evaluated at the final epoch of the full cosine-annealed schedule without those advantages; if the baseline were given the same early stopping and batch-norm recomputation, the reported margins could shrink or disappear.","fun_headline_variants_meta":{"raw":{"variants":["EMA of weights: simple trick that beats last-iterate SGD","Average your weights for better accuracy, noise, calibration","Weight averaging: implicit regularization from SGD noise","EMA: robust models beyond the last checkpoint"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000283,"raw_usage":{"total_tokens":1680,"prompt_tokens":962,"completion_tokens":718,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":657}},"tokens_in":578,"tokens_out":718,"duration_ms":7257,"temperature":1.0,"reasoning_tokens":657,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:58:11.567611+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled comparison where momentum SGD is early-stopped at its own best validation epoch, with batch-norm statistics recomputed at that checkpoint, and compare against the EMA early-stopped the same way. If the accuracy, calibration, and churn gaps vanish or reverse, the claim that EMA solutions are better than last-iterate solutions would be refuted for that setup.","supporting_citations":[{"cited_title":"Averaging weights leads to wider optima and better generalization","cited_arxiv_id":null,"evidence_quote":"Defines stochastic weight averaging and the flat-minima argument that this work compares EMA against."}],"review_version":1}