Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

On Local Overfitting and Forgetting in Deep Neural Networks

T0 review · 3 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Deep networks quietly forget sub-regions of the test population during training even when overall accuracy never falls, and blending mid-training checkpoints with the final model recovers much of the lost knowledge.

desk verdict A useful new diagnostic (forget fraction) and a genuinely effective checkpoint-ensemble method, but the 'local overfitting' interpretation needs a null model before it fully lands. read the letter →

arxiv 2412.12968 v2 pith:NRDUUBFJ submitted 2024-12-17 cs.LG

Uri Stern , Tomer Yaacoby , Daphna Weinshall This is my paper · ORCID
classification cs.LG MSC 68T07
keywords localoverfittingforgettingforgetfractiondeeplinearnetworksprincipalcomponentscheckpointensembleself-distillationdoubledescent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that overfitting in deep neural networks is more common than test-accuracy curves make it look. It proposes a forget fraction $F_e$: the share of test points that the network classified correctly at an intermediate epoch $e$ but misclassifies at the end of training. Across standard image datasets, architectures, optimizers, and learning-rate schedules, the paper finds $F_e>0$ even when test accuracy never decreases, and larger models or smaller training sets produce larger forget fractions. The paper then derives a spectral characterization of forgotten points in deep linear networks, shows it correlates with what deep networks forget, and introduces Knowledge Fusion, a checkpoint-blending method that recovers part of the lost knowledge and improves accuracy without adding inference cost after self-distillation. If correct, the work implies that rising accuracy and simultaneous forgetting can coexist, so conventional accuracy-based monitoring understates how much deep models overfit.

What carries the argument

The central object is the forget fraction $F_e=\mathrm{acc}(e,M_E)|M_E|/|T|$, the share of test points correct at an intermediate epoch but wrong in the final model. The argument is carried by three mechanisms: the identity $\mathrm{acc}(E,T)=\mathrm{acc}(e,T)+L_e-F_e$, which lets local forgetting grow while global accuracy improves; the deep-linear evolution formula $w(n)\approx w(0)\Lambda^n+w_{\mathrm{opt}}(I-\Lambda^n)$, which makes forget time analytically tractable and ties it to the principal-component spectrum of the data; and the Knowledge Fusion algorithm, which averages the final model's probability outputs with checkpoints in a window around the epoch of maximal $F_e$, with weights chosen on validation data and iteration until no further gain appears.

What would settle it

Train a network and record the real forget fraction $F_e$. Then take the final model and perturb its logits randomly, calibrated to flip exactly the same fraction of test predictions while preserving final accuracy; if this null model produces an $F_e$ as large as the real one, the paper's measure does not distinguish forgetting from boundary jitter. A complementary check is to replace the $F_e$-selected checkpoints in Knowledge Fusion with the same number of randomly chosen or evenly spaced checkpoints: if accuracy gains are unchanged, the targeted recovery of forgotten knowledge is not the active mechanism.

Watch

Extended reading notes

Core claim

The central discovery is that learning and forgetting happen at the same time in different sub-regions of the data space. The paper defines $F_e$ and $L_e$ so that $\mathrm{acc}(E,T)=\mathrm{acc}(e,T)+L_e-F_e$; when $F_e>0$ even though $L_e\ge F_e$ for every $e$, global test accuracy rises while a measurable part of the test population is forgotten. Empirically this local overfitting appears with and without label noise, on CIFAR-100, TinyImageNet, and ImageNet, and grows with model capacity and with smaller training sets. In deep linear networks the paper derives a forget time for each point and shows the forgetting rate is governed by the data's singular-value spectrum, with points concentrated on leading principal components forgotten fastest; the set of points forgotten by such a linear model overlaps substantially with the set forgotten by real deep networks. Finally, Knowledge Fusion selects checkpoints near the epoch of maximal $F_e$, averages their probability outputs with the final model using validation-chosen weights, and reports consistent accuracy gains, up to about 3.5 accuracy points on TinyImageNet and roughly 15 percent relative error reduction in a 10 percent asymmetric-noise setting.

Load-bearing premise

The load-bearing premise is that a test point flipping from correct to incorrect during training reflects genuine loss of useful knowledge rather than harmless movement of the decision boundary while the model improves overall.

Editorial extensions

If this is right

  • Global test accuracy should no longer be treated as a sufficient monitor of overfitting; $F_e$ can rise while accuracy climbs, so a flat or improving curve can hide a shrinking set of correctly classified test points.
  • Overfitting indicators should grow where classical theory expects them to: the paper reports larger forget fractions for larger networks and for smaller training sets, even when final accuracies are similar.
  • Mid-training checkpoints carry recoverable knowledge: Knowledge Fusion improves over the single network on clean and noisy data, with the largest gains under label noise where classical overfitting is strongest.
  • The recovered knowledge can be folded into one model through self-distillation, so the inference-time cost stays the same as the original network.
  • Epoch-wise double descent with label noise can be reinterpreted as simultaneous memorization of noisy labels and slow learning of clean hard patterns, and a weaker version of the same forgetting occurs without label noise.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A consequence the paper leaves implicit: if local overfitting is genuine knowledge loss, then accuracy-based early stopping and model selection can silently pick networks that have forgotten subpopulations; tracking $F_e$ on groups of interest would be a more informative selection signal.
  • The spectral result suggests a direct test the paper does not run: sort test points by their projection onto leading principal components and check whether the earliest forgotten points are the ones with the largest leading-component mass, as the deep-linear analysis predicts.
  • The success of forget-weighted checkpoint fusion hints that the training trajectory itself is an underused ensemble resource; comparing $F_e$-weighted checkpoints against uniformly weighted or randomly sampled checkpoints of the same count would isolate how much of the gain comes from targeting forgotten knowledge specifically.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces the forget fraction F_e, defined as the fraction of validation/test points correctly classified at an intermediate training epoch but misclassified by the final model. It argues that positive F_e with non-decreasing test accuracy indicates 'local overfitting'—a loss of accuracy on sub-regions of the data space that is invisible to aggregate accuracy. The paper documents this phenomenon on CIFAR-100, TinyImageNet, and ImageNet across convnets and transformers, analyzes a deep linear model for a theoretical characterization, and tests the corresponding PCA-based prediction by correlating sets S(k) and M(n). It then proposes Knowledge Fusion (KF), which ensembles the final model with selected mid-training checkpoints using the F_e metric, and reports consistent accuracy improvements over single networks and several snapshot/SWA/FGE baselines, with larger gains under label noise.

Significance. If the interpretation of F_e as local overfitting is accepted, the paper would fill a real gap in the literature: accuracy curves alone would understate the degree of overfitting in deep learning, and checkpoint ensembles could be targeted at forgotten sub-regions. The empirical contribution is strong in breadth: multiple datasets, architectures, optimizers, label-noise regimes, ablations, and fairness checks, with standard errors. The main theoretical and algorithmic pieces are clearly presented. However, the central conceptual claim currently rests on an interpretation of F_e that has not been separated from ordinary boundary movement, and the theory-experiment correlation in Sec. 4.2 relies on a fitted index mapping with modest overlap. These issues are addressable with additional analyses, so the work is promising but not yet conclusive.

major comments (3)
  1. [Section 3, Eq. (1), Figs. 3b–3c] The central claim that a positive forget fraction F_e with monotonically non-decreasing test accuracy constitutes 'local overfitting' is underdetermined by the reported evidence. F_e as defined in Eq. (1) counts validation points that are correct at an intermediate epoch and incorrect at the final epoch; any two classifiers of comparable accuracy from the same training run will exhibit some such flips merely because the decision boundary has moved, so positive F_e is also the null expectation for benign boundary drift. The persistence statistics in Figs. 3b–3c show that flipped points were often correct for many epochs, but this is equally consistent with a single late boundary shift; it does not establish that the flips are concentrated in a sub-region of the data space, which is the paper's own definition of local overfitting in the abstract. I recommend adding a null model that preserves per-epoch accuracies and final margins (or a random-subset baseline of the same cardinality), and a spatial- or feature-space clustering analysis of the flipped points.
  2. [Section 4.2, Fig. 4] The empirical support for the theoretical characterization is weaker than the text suggests. The sets S(k) and M(n) are compared after fitting the linear map n = alpha k + beta between the index ranges, so the correspondence is not a parameter-free prediction. No significance test or confidence interval is reported for the overlap shown in Fig. 4, and the early value of |S(k) ∩ M(n)|/|M(n)| is roughly 1/6, which is modest without a null baseline based on random sets of the same sizes. I recommend reporting a permutation or bootstrap test for the overlap, and stating the number of fitted parameters involved.
  3. [Section 5 and Section 7, Tables 1–3] The conclusion that Knowledge Fusion's accuracy gains demonstrate recovery of forgotten knowledge is an interpretation rather than a supported mechanism. The comparisons to fixed-jump and horizontal baselines in Table 1 use the same number of checkpoints, and KF often improves on them (e.g., TinyImageNet 66.98 vs 66.54 in the limited regime), but any diverse set of checkpoints can improve accuracy even if no point-specific 'forgetting' occurred. To substantiate the recovery claim, the authors should report accuracy of the KF predictor on the set of final-epoch errors M_E (or on the F_e points) versus the rest of the test set, and compare the recovery rate with that of equally spaced checkpoints. Without per-point evidence, the sentence in Section 7 that KF's success 'shows that indeed models forget useful knowledge' overstates what is demonstrated.
minor comments (7)
  1. [Abstract and Section 5] The claim that self-distillation post-processing 'will enhance the performance of any trained model without adding inference costs' is not empirically evaluated anywhere in the paper; only the algorithm is described. The authors should either add the corresponding experiment or soften the claim.
  2. [Appendix C, Algorithm 2] The Input line repeats 'w and validation data V' twice, and the line 'best epsilon = combined prob' appears to be a typo; it should store the epsilon value that maximized validation accuracy.
  3. [Section 1] The citation of (Annavarapu 2021) for the double-descent phenomenon is odd, since that work is about snapshot ensembles; the standard references (Belkin et al. 2019; Nakkiran et al. 2021) appear later in the related work and should be cited here.
  4. [Equation (1) and surrounding text] The subscripts 'e' and 'E' (epoch and final epoch) are visually similar; please use a more distinct notation, for example e and E with different fonts, and state explicitly that M_E is the set misclassified by the final model.
  5. [Figure 7 caption and Appendix B] The notation F^L_e and F^D_e is rendered inconsistently in the text; the superscripts/subscripts should be fixed.
  6. [Section 6.1, first paragraph] 'Universally useful' is too strong; in Table 1, KF (∞) underperforms fixed jumps (∞) on CIFAR-100 (79.13 vs 79.17). Suggest 'generally useful'.
  7. [Section 4.1] The extension from binary deep linear networks to multi-class nonlinear networks is stated as a suggestion ('suggesting similar qualitative outcomes'); this should be explicitly labeled as a conjecture rather than part of the theoretical derivation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central Fe measurements are direct, the theory is an extension of an external published result, and the Sec. 4.2 alignment is an empirical-fitting concern, not a construction-level circularity.

full rationale

The central claim that Fe > 0 occurs without a decline in global test accuracy is a direct measurement defined in Eq. (1): Fe is a set-theoretic count of test points correctly classified at epoch e and misclassified at the end. The paper does not derive Fe from the theory, nor does it fit Fe from any parameter; it simply reports the measured score across many architectures and datasets. The theory in Sec. 4 extends the published, externally stated convergence result of Hacohen and Weinshall (2022), and the resulting qualitative prediction about PCA projections is tested empirically rather than being used to force the observed Fe values. The Sec. 4.2 comparison of S(k) and M(n) does rely on an unexplained affine alignment n = alpha*k + beta, and if alpha and beta were fitted to the same overlap data, the reported correlation would be inflated; however, the paper does not state such a fitting procedure, and neither set is defined in terms of the other, so this is a falsifiability and reporting concern rather than a circular reduction. The Knowledge Fusion method is validated on held-out test splits after hyperparameters are chosen on a separate validation split, so its improvement is an independent empirical outcome, not an artifact of the definition of Fe. Self-citations to Hacohen and Weinshall 2022 and Stern et al. 2024 are present, but the load-bearing empirical measurements and held-out evaluations do not reduce to those citations. Overall, no step in the paper's argument makes a predicted quantity equal to its own input by construction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper's central empirical claim rests on standard supervised learning assumptions. The main free parameters are the ensemble weights and window size chosen via validation, and the fitted alpha/beta map used to validate the theoretical prediction. No new physical or conceptual entities are postulated.

free parameters (4)
  • Epoch-index mapping (alpha, beta) = Not reported
    In Sec 4.2, S(k) and M(n) are correlated after setting n = alpha*k + beta; the linear map is fitted to align PCA rank and epoch ranges, and no values or fitting procedure are given.
  • Ensemble weights epsilon_i = Per validation split
    Algorithm 2 selects epsilon_i from {0, 0.01, ..., 1} to maximize validation accuracy for each chosen checkpoint.
  • Window size w = 1
    Fixed to w=1 in all experiments, near-optimal according to ablation in App G.9.
  • Number of KF iterations k = Until no validation improvement
    The iterative selection loops until no further validation accuracy gain is observed; not a fixed number.
assumptions (5)
  • domain assumption Deep linear network convergence: w(n)_j ~= lambda_j^n w(0)_j + (1 - lambda_j^n) w_opt_j with lambda_j = 1 - gamma s_j^L (Eq 3).
    Taken from Hacohen and Weinshall (2022) as the starting point for the forget-time derivation in Sec 4.1.
  • standard math Infinitesimal learning rate: O(gamma^2) terms are neglected in the Taylor expansion.
    Used in Sec 4.1 to derive the linear approximation dF/dn in Eq (6).
  • domain assumption Natural image singular values decay approximately exponentially.
    Invoked in Sec 4.2 to argue that leading principal components dominate the forgetting rate.
  • ad hoc to paper Qualitative transfer from deep linear models to deep nonlinear networks.
    The paper assumes insights from linear models apply to real DNNs; the correlation in Sec 4.2 is offered as evidence but is not a proof.
  • domain assumption Validation set is representative of the test distribution.
    KF hyperparameters are selected on a validation set (half of the test data in main results); standard practice, with an appendix check using training data for validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On Local Overfitting and Forgetting in Deep Neural Networks." pith.science (2026). https://pith.science/paper/NRDUUBFJ

@misc{pith2026241212968,
  author       = {Pith},
  title        = {Pith review of: On Local Overfitting and Forgetting in Deep Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NRDUUBFJ}},
  note         = {Machine review of arXiv:2412.12968}
}
read the original abstract

The infrequent occurrence of overfitting in deep neural networks is perplexing: contrary to theoretical expectations, increasing model size often enhances performance in practice. But what if overfitting does occur, though restricted to specific sub-regions of the data space? In this work, we propose a novel score that captures the forgetting rate of deep models on validation data. We posit that this score quantifies local overfitting: a decline in performance confined to certain regions of the data space. We then show empirically that local overfitting occurs regardless of the presence of traditional overfitting. Using the framework of deep over-parametrized linear models, we offer a certain theoretical characterization of forgotten knowledge, and show that it correlates with knowledge forgotten by real deep models. Finally, we devise a new ensemble method that aims to recover forgotten knowledge, relying solely on the training history of a single network. When combined with self-distillation, this method enhances the performance of any trained model without adding inference costs. Extensive empirical evaluations demonstrate the efficacy of our method across multiple datasets, contemporary neural network architectures, and training protocols.

Figures

Figures reproduced from arXiv: 2412.12968 by the authors.

Figure 1
Figure 1. Local overfitting and forgetting in a binary problem, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a)-(b): Blue denotes test accuracy. Among those correctly recognized in each epoch [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. (a) The Fe score (1) of ConvNeXt trained on Imagenet, 3 network sizes: small → blue, base → orange and large → green. Accuracy remained consistent across all network sizes, while it is evident that Fe increases with the network size. (b-c) Within the set of wrongly classified test points after training, we show (b) the fraction that was correctly predicted (y-axis) for x epochs (x-axis), and (c) the last epoch in wh… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Empirical results, correlating the sets of examples forgot [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 7
Figure 7. Figure 7: C Computing Hyper-Parameters Algorithm 2: KF - hyper-parameter calculation Input:all past checkpoints during training of the neural network {n0,...,nE}, w and validation data V , w and val￾idation data V Output: list of alternative epochs and their weights class probs …
Figure 5
Figure 5. Figure 5: The forget fraction Fe, as defined in (1), of common neural networks trained on various image classification datasets and different architectures. (a) Cifar100, resnet 18 (b) CIFAR-100, resnet 18 (c) CIFAR-100, resnet 18 [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: (a) The Fe score of Resnet18 trained on 10/30/50/70/90% of the train data in CIFAR-100 (pur￾ple/red/green/yellow/blue line, respectively) in the first 50 epochs of training (after which the score decreases); Fe is significantly larger for the smallest set of only 10%. …
Figure 7
Figure 7. Figure 7: Empirical results using dataset CIFAR-100, matching the examples forgotten during the training of a DNN and those forgotten [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Empirical results, matching the examples forgotten during the training of a DNN and those forgotten during the training of a deep [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Improvement achieved by our method when using a dif [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Comparing our method with limited number of check [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Accuracy of the Knowledge Fusion algorithm as a func [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Forget Me Not: Fighting Local Overfitting with Knowledge Fusion and Distillation

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Checkpoint fusion guided by a 'forget' metric, followed by distillation, lets a single model recover test points that were learned and then forgotten during training, improving accuracy, especially under label noise.

Reference graph

Works this paper leans on

42 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    Allen-Zhu, Z.; and Li, Y. 2023. Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning. arXiv:2012.09816

    arXiv 2023

  2. [2]

    Annavarapu, C. S. R. 2021. Deep learning-based improved snapshot ensemble technique for COVID-19 chest X-ray classification. Applied Intelligence, 51: 3104--3120

    work page 2021

  3. [3]

    Arora, S.; Cohen, N.; Golowich, N.; and Hu, W. 2019. A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net

    work page 2019

  4. [4]

    Arora, S.; Cohen, N.; and Hazan, E. 2018. On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization. In International Conference on Machine Learning, 244--253

    work page 2018

  5. [5]

    S.; Maharaj, T.; Fischer, A.; Courville, A.; Bengio, Y.; et al

    Arpit, D.; Jastrzebski, S.; Ballas, N.; Krueger, D.; Bengio, E.; Kanwal, M. S.; Maharaj, T.; Fischer, A.; Courville, A.; Bengio, Y.; et al. 2017. A closer look at memorization in deep networks. In International conference on machine learning, 233--242. PMLR

    work page 2017

  6. [6]

    Belkin, M.; Hsu, D.; Ma, S.; and Mandal, S. 2019. Reconciling modern machine-learning practice and the classical bias--variance trade-off. Proceedings of the National Academy of Sciences, 116(32): 15849--15854

    work page 2019

  7. [7]

    Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning, 1597--1607. PMLR

    2020

  8. [8]

    Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 248--255. Ieee

    2009

Show all 42 references
  1. [9]

    Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

    2020 arXiv

  2. [10]

    Fukumizu, K. 1998. Effect of batch learning in multilayer neural networks. Gen, 1(04): 1E--03

    1998

  3. [11]

    A.; Hu, M.; Malik, A.; Tanveer, M.; and Suganthan, P

    Ganaie, M. A.; Hu, M.; Malik, A.; Tanveer, M.; and Suganthan, P. 2022. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence, 115: 105151

    2022

  4. [12]

    P.; and Wilson, A

    Garipov, T.; Izmailov, P.; Podoprikhin, D.; Vetrov, D. P.; and Wilson, A. G. 2018. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31

    2018

  5. [13]

    Guo, H.; Jin, J.; and Liu, B. 2023. Stochastic weight averaging revisited. Applied Sciences, 13(5): 2935

    2023

  6. [14]

    Hacohen, G.; and Weinshall, D. 2022. Principal components bias in over-parameterized linear models, and its manifestation in deep neural networks. Journal of Machine Learning Research, 23(155): 1--46

    2022

  7. [15]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770--778

    2016

  8. [16]

    Heckel, R.; and Yilmaz, F. F. 2020. Early stopping in deep networks: Double descent and how to eliminate it. arXiv preprint arXiv:2007.10099

    2020 arXiv

  9. [17]

    Hu, W.; Xiao, L.; and Pennington, J. 2020. Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net

    2020

  10. [18]

    E.; and Weinberger, K

    Huang, G.; Li, Y.; Pleiss, G.; Liu, Z.; Hopcroft, J. E.; and Weinberger, K. Q. 2017. Snapshot ensembles: Train 1, get m for free. arXiv preprint arXiv:1704.00109

    2017 arXiv

  11. [19]

    Hyv \"a rinen, A.; Hurri, J.; and Hoyer, P. O. 2009. Natural image statistics: A probabilistic approach to early computational vision., volume 39. Springer Science & Business Media

    2009

  12. [20]

    Izmailov, P.; Podoprikhin, D.; Garipov, T.; Vetrov, D.; and Wilson, A. G. 2018. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407

    2018 arXiv

  13. [21]

    Jeong, H.; and Chung, H. W. 2024. Understanding Self-Distillation and Partial Label Learning in Multi-Class Classification with Label Noise. arXiv preprint arXiv:2402.10482

    2024 arXiv

  14. [22]

    Krizhevsky, A.; Hinton, G.; et al. 2009. Learning multiple layers of features from tiny images

    2009

  15. [23]

    Le, Y.; and Yang, X. 2015. Tiny imagenet visual recognition challenge. CS 231N, 7(7): 3

    2015

  16. [24]

    Liu, Z.; Mao, H.; Wu, C.-Y.; Feichtenhofer, C.; Darrell, T.; and Xie, S. 2022. A ConvNet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2022

  17. [25]

    McCloskey, M.; and Cohen, N. J. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, 109--165. Elsevier

    1989

  18. [26]

    Nakkiran, P.; Kaplun, G.; Bansal, Y.; Yang, T.; Barak, B.; and Sutskever, I. 2021. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12): 124003

    2021

  19. [27]

    Noppitak, S.; and Surinta, O. 2022. dropCyclic: snapshot ensemble convolutional neural network based on a new learning rate schedule for land use classification. IEEE Access, 10: 60725--60737

    2022

  20. [28]

    Patrini, G.; Rozza, A.; Krishna Menon, A.; Nock, R.; and Qu, L. 2017. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1944--1952

    2017

  21. [29]

    Polikar, R. 2012. Ensemble learning. Ensemble machine learning: Methods and applicannavarapu2021deepations, 1--34

    2012

  22. [30]

    T.; and Juditsky, A

    Polyak, B. T.; and Juditsky, A. B. 1992. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4): 838--855

    1992

  23. [31]

    M.; McClelland, J

    Saxe, A. M.; McClelland, J. L.; and Ganguli, S. 2014. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings

    2014

  24. [32]

    Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929--1958

    2014

  25. [33]

    Stephenson, C.; and Lee, T. 2021. When and how epochwise double descent happens. arXiv preprint arXiv:2108.12006

    2021 arXiv

  26. [34]

    Stern, U.; Shwartz, D.; and Weinshall, D. 2024. United We Stand: Using Epoch-wise Agreement of Ensembles to Combat Overfit. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38(13), 15075--15082

    2024

  27. [35]

    TorchVision. 2016. TorchVision: PyTorch's Computer Vision library. https://github.com/pytorch/vision

    2016

  28. [36]

    Tu, Z.; Talebi, H.; Zhang, H.; Yang, F.; Milanfar, P.; Bovik, A.; and Li, Y. 2022. Maxvit: Multi-axis vision transformer. In Computer Vision--ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23--27, 2022, Proceedings, Part XXIV, 459--479. Springer

    2022

  29. [37]

    C.; Genova, K.; Nair, P.; Hata, K.; and Russakovsky, O

    Wang, Z.; Qinami, K.; Karakozis, I. C.; Genova, K.; Nair, P.; Hata, K.; and Russakovsky, O. 2020. Towards fairness in visual recognition: Effective strategies for bias mitigation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 8919--8928

    2020

  30. [38]

    Xie, J.; Xu, B.; and Chuang, Z. 2013. Horizontal and vertical ensemble with deep representation for classification. arXiv preprint arXiv:1306.2759

    2013 arXiv

  31. [39]

    Yang, Y.; Lv, H.; and Chen, N. 2023. A survey on ensemble learning under the era of deep learning. Artificial Intelligence Review, 56(6): 5545--5589

    2023

  32. [40]

    Zhao, J.; Wang, T.; Yatskar, M.; Ordonez, V.; and Chang, K.-W. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv preprint arXiv:1707.09457

    2017 arXiv

  33. [41]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  34. [42]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.