Pith. sign in

REVIEW 3 major objections 5 minor 39 references

Forget Me Not: Fighting Local Overfitting with Knowledge Fusion and Distillation

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Deep networks forget useful validation points in late training; a student distilled from a fusion of their own checkpoints recovers that knowledge and beats the original model — and, under label noise, independently trained ensembles.

desk verdict Plausible method for recovering forgotten knowledge via checkpoint fusion and distillation, but the headline numbers rely on test-set-tuned hyperparameters and the appendix rebuttal is unquantified. read the letter →

arxiv 2507.08686 v1 pith:4XHUVYNH submitted 2025-07-11 cs.LG

Uri Stern , Eli Corn , Daphna Weinshall This is my paper
classification cs.LG
keywords localoverfittingforgettingdoubledescentknowledgedistillationensemblelearninglabelnoisecheckpointselectionFusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that deep networks overfit locally even when their overall test accuracy keeps rising: during training they correctly classify some validation points early, then lose them while simultaneously learning other patterns. To capture this, it introduces a “forget fraction” score that measures the fraction of validation points the final model misclassifies even though an earlier epoch got them right, and shows that this score grows with model capacity and peaks in the same regimes as double descent. Building on that, it proposes Knowledge Fusion (KF), which selects the mid-training checkpoints with the largest forget fraction, averages their class probabilities with the final model’s output, and then distills the resulting ensemble back into a single model of the original size. The paper reports that the distilled KF model consistently beats the original baseline, often beats the KF ensemble itself, and under label noise outperforms independently trained ensembles overall, while keeping single-model training and inference cost.

What carries the argument

The load-bearing object is the forget fraction $$F_e = \frac{\mathrm{acc}(e, M_E)\,|M_E|}{|T|},$$ the fraction of test/validation points that are correct at epoch $e$ but wrong in the final model, together with its companion $L_e$ (points wrong at $e$ but correct at the end); the identity $\mathrm{acc}(E,T) = \mathrm{acc}(e,T) + L_e - F_e$ shows that global accuracy can keep rising while $F_e > 0$. Knowledge Fusion uses $\arg\max_e F_e$ on validation data to select checkpoints, averages a window of $\pm w$ epochs around each selected checkpoint, and interpolates those probability vectors with the final model’s output using weights $\varepsilon$ chosen on validation. Phase two distills this fused predictor into a single same-size student using temperature-scaled KL divergence plus cross-entropy loss, with temperature $T$ and mixing weight $\alpha$. A short theoretical section analyzes an over-parameterized deep linear network and derives a “forget time” showing that points whose spectral mass concentrates in leading principal components are forgotten fastest, giving a mechanistic explanation for local overfitting.

What would settle it

Run the full KF-plus-distillation pipeline on a benchmark twice, selecting all checkpoints and weights on two disjoint halves of the test set and evaluating each configuration on the other half; if the two configurations disagree about which epochs are forgotten and each fails to beat the baseline on the other half, the representative-validation assumption is falsified. A weaker check is to compute the $F_e$ curves on independent splits and test whether the $\arg\max_e F_e$ epochs coincide.

Watch

Extended reading notes

Core claim

On the paper’s own terms, the central discovery is that overfitting in deep learning is a local phenomenon: a network can improve its overall test accuracy throughout training while systematically “forgetting” a subset of the validation population that earlier checkpoints classified correctly, and this forgotten knowledge can be recovered. The evidence is a two-stage recipe. In stage one, Knowledge Fusion forms a weighted ensemble of the final model with checkpoints chosen by maximizing the forget fraction $F_e$ on validation data, with a small window of neighboring epochs and iterative re-selection. In stage two, this ensemble is condensed by knowledge distillation into a student of the original architecture. The empirical claim is that distillation not only preserves the ensemble’s accuracy but improves on it: the distilled model outperforms the original network by roughly 1.4–4.7 points on clean CIFAR-100 and TinyImageNet and by roughly 6–14 points under injected label noise, and on the noisy benchmarks it is ranked first overall, ahead of independently trained ensembles, while keeping inference cost at that of a single model.

Load-bearing premise

The load-bearing premise is that the subset of validation data used to pick checkpoints and weights exhibits the same forgetting pattern as the test population; if that validation half is unrepresentative, the reported gains will not transfer to a truly held-out set.

Editorial extensions

If this is right

  • Distilled KF improves on the original early-stopped model on every dataset and architecture tested, including ResNet, ConvNeXt, ViT, and MaxViT on ImageNet.
  • Under symmetric and asymmetric label noise, distilled KF beats the original model in every noisy experiment and is ranked the best method on noisy datasets overall, with single-model inference cost.
  • Roughly 5–10% of the training checkpoints are enough to capture most of the gain, so the ensemble overhead can be kept small in practice.
  • KF is complementary to transfer learning, test-time augmentation, and exponential moving averages, improving on these techniques and remaining useful where they fail.
  • The forget fraction grows with model size and with smaller training sets, so the method pays off most in exactly the regimes where classical overfitting is expected.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If local forgetting is a real failure mode, training itself could be made forgetting-aware: when the forget fraction on a validation probe starts climbing, the learning rate could be lowered or the optimizer could revisit the corresponding checkpoints, rather than waiting until after training to fuse them.
  • The $L_e$/$F_e$ accounting suggests a per-example test: distillation should help most on points where the fused teacher is confident but the student is wrong, so measuring the overlap between teacher corrections and student errors before training would predict where the gain comes from.
  • The deep-linear “forget time” result transfers naturally to nonlinear networks: one can test whether examples that are forgotten and later recovered have unusually large projections on the top principal components of the training data, as the linear theory predicts.
  • Because KF’s gain grows with model size and with smaller training sets, a natural next experiment is to combine it with data-efficient fine-tuning of very large pretrained models, where the checkpoint history is cheap to keep and the forgotten fraction is likely to be highest.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper defines a validation-set forget fraction F_e and shows empirically that networks forget previously correct validation points even when test accuracy improves, linking this to epoch-wise double descent and model capacity. It then proposes Knowledge Fusion (KF): an iterative greedy selection of checkpoints with high F_e, combined with the final model by validation-set weights, followed by Knowledge Distillation (or weight averaging) to restore single-model inference. Experiments on CIFAR-100, TinyImageNet, ImageNet and CIFAR-100N, across ResNet, ConvNeXt, ViT and MaxViT architectures and several label-noise regimes, report consistent gains, with distilled KF often matching or beating the KF ensemble and sometimes independent ensembles. A short theory section derives a linearized forget-rate expression for deep linear networks.

Significance. The proposed procedure is simple, training-time-agnostic and, if the reported gains hold on a properly held-out test set, would be a practically useful post-training accuracy boost with no added inference cost. Strengths include the breadth of architectures and noise settings, the comparison to several training-protocol baselines, and the explicit ranking in Table I that makes the efficiency claim falsifiable. The main weakness is the evaluation protocol: many headline numbers are selected using validation data drawn from the test distribution, and the paragraph that addresses this concern is not quantified. If the protocol issue is corrected, the empirical contribution would be solid and useful.

major comments (3)
  1. [Section VI-C6 and Appendix C-A] The manuscript states that 'half of the test data was used for validation' for hyperparameter search and that the reported accuracy is computed on the remaining test half, while Appendix C-A only asserts that the train-split results are 'almost identical' to Table II without providing a table, error bars, or a statistical comparison. Because Algorithm 1 performs an iterative greedy search over epochs and epsilon values on this validation half, the reported improvements in Tables II-IV are not independent of the test distribution. Please re-run the evaluation with hyperparameters selected on a training split (or a truly disjoint validation set) and report accuracy on the untouched test set with standard errors, or provide quantified train-split results in the main text.
  2. [Table IV] The independent-ensemble rows report point estimates without error bars (e.g., 82.13, 73.89, 55.18 for CIFAR-100), while the text claims that the distilled KF ensemble outperforms the distilled independent ensemble. For several settings the difference may not be statistically significant once the error bars of the distilled rows are considered (e.g., CIFAR-100 0%: 80.29 ± 0.16 vs 79.93 ± 0.21). Please provide standard errors and seed counts for the independent ensemble and its distilled variant, and state which pairwise differences are statistically significant.
  3. [Algorithm 1] Algorithm 1 is not reproducible as printed: inside the epsilon loop the line 'best epsilon ← combined prob' assigns a probability vector to a scalar hyperparameter, 'epsilons.append(argmax(best epsilon))' appends an index rather than an epsilon value, and the while loop has no explicit stopping condition tied to validation improvement. Please rewrite the pseudocode or provide the actual implementation, since this algorithm is the core of the proposed method.
minor comments (5)
  1. [Section III] The sentence 'Throughout this paper, the terms "test set" and "validation set" are used interchangeably' is confusing after Section VI-C6 describes splitting the test data for validation; please define both terms explicitly and use them consistently.
  2. [Introduction and References] Reference [2] is cited in the introduction for the double-descent phenomenon, but the bibliographic entry is a snapshot-ensemble COVID-19 chest X-ray paper; this appears to be a citation mismatch.
  3. [Table XII] The value '77.47 ± 14' in Table XII appears to have a typo in the standard error; the error bars in that row should be checked.
  4. [Section VI-D] The claim that the method reduces errors by 'around 15% in cases of 10% asymmetric noise' is not tied to a reported table or experiment; please add the corresponding result or remove the number.
  5. [Reproducibility] No link to code or trained checkpoints is provided, which would be particularly helpful for verifying Algorithm 1 and the checkpoint-selection procedure.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: KF checkpoints and weights are selected on a validation split and evaluated on a held-out remainder, and the theory relies on an external convergence result rather than on a self-citation chain.

full rationale

The paper's central claim is that checkpoint selection by the forget fraction Fe, combined with validation-weighted averaging and subsequent distillation, improves accuracy over the original model and competitive baselines. This claim is not equivalent to its inputs by construction: Algorithm 1 selects epochs and epsilon weights by validation accuracy, while the reported test accuracies in Tables II-IV are computed on the remaining half of the test data, averaged over three random validation/test splits as stated in Section VI-C6. The hyperparameters are therefore fitted to one split and evaluated on another held-out split, which is standard model selection rather than a prediction that reduces to a fitted value. The theoretical section VII derives a forget-time characterization from the convergence formula of [29], an external published result with stated assumptions about deep linear networks and gradient descent; it is not invoked as a uniqueness theorem, and the empirical gains of KF are benchmarked directly against the original model, horizontal ensembles, fixed jumps, independent ensembles, SWA, snapshot ensembles, FGE, EMA, and TTA. The self-citation to [3] supports the prior empirical claim that forgotten points correlate with the theoretical analysis, but the present paper's performance improvements do not depend on that citation as their sole justification. One genuine weakness is that Appendix C-A's claim that the train-split results are 'almost identical to those reported in Table II' is unquantified, and using half of the test data for validation is a nontrivial evaluation-design choice; however, this is a robustness and reporting concern, not a circular derivation, because the reported accuracy still comes from a distinct held-out portion of the test split. Thus, no load-bearing step reduces to its own inputs, and the circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method's hyperparameters (epsilon, w, T, alpha) are tuned on validation data, so part of the reported gain is model selection. The theory section assumes a deep linear network approximation and prior results from [29]. No new physical entities are introduced.

free parameters (4)
  • ensemble weight epsilon = selected in [0,1] on validation set
    Algorithm 1 selects epsilon by grid search on validation accuracy; reported gains depend on this selection.
  • window size w = 1
    Fixed at w=1 based on ablation (Sec VI-C10), shown near-optimal.
  • distillation temperature T = 2.5
    Appendix B: results use T=2.5, chosen among tested values; authors claim robustness.
  • KD loss weight alpha = 0.9
    Appendix B: alpha=0.9 in Eq 2; chosen via experimentation on validation.
assumptions (3)
  • standard math Gradient descent dynamics for deep linear networks follow Eq (4) from [29]
    Section VII: Eq 4 'convergence rate exponential with lambda_j = 1 - gamma s_j L' is taken from Hacohen & Weinshall [29], a prior published result.
  • domain assumption Infinitesimal learning rate, O(gamma^2) negligible
    Section VII: 'we begin by assuming the learning rate gamma is infinitesimal, so that terms of magnitude O(gamma^2) can be neglected' to derive Eq 7.
  • domain assumption Deep linear network behavior correlates with deep non-linear networks
    Section VII: theory is on deep linear networks; the paper defers to [3] for the claim that this analysis correlates with empirical results in deep networks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Forget Me Not: Fighting Local Overfitting with Knowledge Fusion and Distillation." pith.science (2026). https://pith.science/paper/4XHUVYNH

@misc{pith2026250708686,
  author       = {Pith},
  title        = {Pith review of: Forget Me Not: Fighting Local Overfitting with Knowledge Fusion and Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4XHUVYNH}},
  note         = {Machine review of arXiv:2507.08686}
}
read the original abstract

Overfitting in deep neural networks occurs less frequently than expected. This is a puzzling observation, as theory predicts that greater model capacity should eventually lead to overfitting -- yet this is rarely seen in practice. But what if overfitting does occur, not globally, but in specific sub-regions of the data space? In this work, we introduce a novel score that measures the forgetting rate of deep models on validation data, capturing what we term local overfitting: a performance degradation confined to certain regions of the input space. We demonstrate that local overfitting can arise even without conventional overfitting, and is closely linked to the double descent phenomenon. Building on these insights, we introduce a two-stage approach that leverages the training history of a single model to recover and retain forgotten knowledge: first, by aggregating checkpoints into an ensemble, and then by distilling it into a single model of the original size, thus enhancing performance without added inference cost. Extensive experiments across multiple datasets, modern architectures, and training regimes validate the effectiveness of our approach. Notably, in the presence of label noise, our method -- Knowledge Fusion followed by Knowledge Distillation -- outperforms both the original model and independently trained ensembles, achieving a rare win-win scenario: reduced training and inference complexity.

Figures

Figures reproduced from arXiv: 2507.08686 by the authors.

Figure 1
Figure 1. Local overfitting and forgetting in a binary problem, where blue and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a)–(b): Blue denotes test accuracy. Among those correctly recognized in each epoch [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. (a) The Fe score (1) of ConvNeXt trained on ImageNet, 3 network sizes: small → blue, base → orange, and large → green. Accuracy remained consistent across all network sizes, while it is evident that Fe increases with the network size. (b)Within the set of wrongly classified test points after training, we show the last epoch in which an example was classified correctly. To investigate this phenomenon, we trained vari… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The forget fraction Fe, as defined in (1), of common neural networks trained on various image classification datasets and different architectures. (a) (b) (c) [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: CIFAR100, ResNet18: (a) The Fe score of ResNet18 trained on 10/30/50/70/90% of the train data in CIFAR-100 (purple/red/green/yellow/blue line, respectively) in the first 50 epochs of training (after which the score decreases); Fe is significantly larger for the smalles…
Figure 6
Figure 6. Figure 6: Improvement achieved by our method when using a different number [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Accuracy of the Knowledge Fusion method as a function of window [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 29 canonical work pages

  1. [1]

    A convnet for the 2020s,

    Z. Liu, H. Mao, C.-Y . Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” in IEEE Conf. Comput. Vis. Pattern Recog. (CVPR) , 2022

    work page 2022

  2. [2]

    Deep learning-based improved snapshot ensemble technique for covid-19 chest x-ray classification,

    C. S. R. Annavarapu, “Deep learning-based improved snapshot ensemble technique for covid-19 chest x-ray classification,” Applied Intelligence, vol. 51, pp. 3104– 3120, 2021

    work page 2021

  3. [3]

    On Local Overfitting and Forgetting in Deep Neural Networks

    U. Stern, T. Yaacoby, and D. Weinshall, “On local overfitting and forgetting in deep neural networks,” in AAAI Conf. Artif. Intell. (AAAI) , vol. 39, no. 19, 2025, pp. 20 592–20 600, Appendices – arXiv preprint arXiv:2412.12968

    work page Pith review arXiv 2025

  4. [4]

    A closer look at memorization in deep networks,

    D. Arpit, S. Jastrzebski, N. Ballas, D. Krueger, E. Ben- gio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y . Bengioet al., “A closer look at memorization in deep networks,” in Int. Conf. Machine Learning (ICML) . PMLR, 2017, pp. 233–242

    work page 2017

  5. [5]

    Catastrophic interfer- ence in connectionist networks: The sequential learning problem,

    M. McCloskey and N. J. Cohen, “Catastrophic interfer- ence in connectionist networks: The sequential learning problem,” in Psychology of learning and motivation . Elsevier, 1989, vol. 24, pp. 109–165

    work page 1989

  6. [6]

    A survey on ensemble learning under the era of deep learning,

    Y . Yang, H. Lv, and N. Chen, “A survey on ensemble learning under the era of deep learning,” Artificial Intelligence Review , vol. 56, no. 6, pp. 5545–5589, 2023

    work page 2023

  7. [7]

    Dropout: a simple way to pre- vent neural networks from overfitting,

    N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to pre- vent neural networks from overfitting,” The journal of machine learning research , vol. 15, no. 1, pp. 1929– 1958, 2014

    work page 1929

  8. [8]

    Horizontal and Vertical Ensemble with Deep Representation for Classification

    J. Xie, B. Xu, and Z. Chuang, “Horizontal and vertical ensemble with deep representation for classification,” arXiv preprint arXiv:1306.2759 , 2013

    work page Pith review arXiv 2013

Show all 39 references
  1. [9]

    Acceleration of stochastic approximation by averaging,

    B. T. Polyak and A. B. Juditsky, “Acceleration of stochastic approximation by averaging,” SIAM journal on control and optimization, vol. 30, no. 4, pp. 838–855, 1992

    1992

  2. [10]

    Averaging weights leads to wider optima and better generalization,

    P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. G. Wilson, “Averaging weights leads to wider optima and better generalization,” arXiv preprint arXiv:1803.05407, 2018

    2018 arXiv

  3. [11]

    dropcyclic: snapshot en- semble convolutional neural network based on a new learning rate schedule for land use classification,

    S. Noppitak and O. Surinta, “dropcyclic: snapshot en- semble convolutional neural network based on a new learning rate schedule for land use classification,” IEEE Access, vol. 10, pp. 60 725–60 737, 2022

    2022

  4. [12]

    Stochastic weight averaging revisited,

    H. Guo, J. Jin, and B. Liu, “Stochastic weight averaging revisited,” Applied Sciences , vol. 13, no. 5, p. 2935, 2023

    2023

  5. [13]

    Reconcil- ing modern machine-learning practice and the classical bias–variance trade-off,

    M. Belkin, D. Hsu, S. Ma, and S. Mandal, “Reconcil- ing modern machine-learning practice and the classical bias–variance trade-off,” Proceedings of the National Academy of Sciences , vol. 116, no. 32, pp. 15 849– 15 854, 2019

    2019

  6. [14]

    Deep double descent: Where bigger models and more data hurt,

    P. Nakkiran, G. Kaplun, Y . Bansal, T. Yang, B. Barak, and I. Sutskever, “Deep double descent: Where bigger models and more data hurt,” Journal of Statistical Mechanics: Theory and Experiment , vol. 2021, no. 12, p. 124003, 2021

    2021

  7. [15]

    When and how epochwise double descent happens,

    C. Stephenson and T. Lee, “When and how epochwise double descent happens,” arXiv preprint arXiv:2108.12006, 2021

    2021 arXiv

  8. [16]

    Early stopping in deep networks: Double descent and how to eliminate it,

    R. Heckel and F. F. Yilmaz, “Early stopping in deep networks: Double descent and how to eliminate it,” arXiv preprint arXiv:2007.10099 , 2020

    2007 arXiv

  9. [17]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” 2015. [Online]. Available: https://arxiv.org/abs/1503.02531

    2015 arXiv

  10. [18]

    Understanding self- distillation and partial label learning in multi- class classification with label noise,

    H. Jeong and H. W. Chung, “Understanding self- distillation and partial label learning in multi- class classification with label noise,” arXiv preprint arXiv:2402.10482, 2024

    2024 arXiv

  11. [19]

    United we stand: Using epoch-wise agreement of ensembles to combat overfit,

    U. Stern, D. Shwartz, and D. Weinshall, “United we stand: Using epoch-wise agreement of ensembles to combat overfit,” in AAAI Conf. Artif. Intell. (AAAI), vol. 38(13), 2024, pp. 15 075–15 082

    2024

  12. [20]

    Towards understanding ensem- ble, knowledge distillation and self-distillation in deep learning,

    Z. Allen-Zhu and Y . Li, “Towards understanding ensem- ble, knowledge distillation and self-distillation in deep learning,” arXiv preprint arXiv:2012.09816 , 2020

    2012 arXiv

  13. [21]

    Revisiting self-distillation. arxiv,

    M. Pham, M. Cho, A. Joshi, and C. Hegde, “Revisiting self-distillation. arxiv,” arXiv preprint arXiv:2206.08491, 2022

    2022 arXiv

  14. [22]

    Understanding self-distillation in the presence of label noise,

    R. Das and S. Sanghavi, “Understanding self-distillation in the presence of label noise,” in Int. Conf. Machine Learning (ICML). PMLR, 2023, pp. 7102–7140

    2023

  15. [23]

    Efficient knowledge distillation from model check- points,

    C. Wang, Q. Yang, R. Huang, S. Song, and G. Huang, “Efficient knowledge distillation from model check- points,” Adv. Neural Inform. Process. Syst. (NeurIPS) , vol. 35, pp. 607–619, 2022

    2022

  16. [24]

    Learn from the past: Experience ensemble knowledge distilla- tion,

    C. Wang, S. Zhang, S. Song, and G. Huang, “Learn from the past: Experience ensemble knowledge distilla- tion,” in Int. Conf. Pattern Recog. (ICPR). IEEE, 2022, pp. 4736–4743

    2022

  17. [25]

    Snapshot ensembles: Train 1, get m for free,

    G. Huang, Y . Li, G. Pleiss, Z. Liu, J. E. Hopcroft, and K. Q. Weinberger, “Snapshot ensembles: Train 1, get m for free,” arXiv preprint arXiv:1704.00109 , 2017

    2017 arXiv

  18. [26]

    Loss surfaces, mode connectivity, 1 and fast ensembling of dnns,

    T. Garipov, P. Izmailov, D. Podoprikhin, D. P. Vetrov, and A. G. Wilson, “Loss surfaces, mode connectivity, 1 and fast ensembling of dnns,” Adv. Neural Inform. Process. Syst. (NeurIPS) , vol. 31, 2018

    2018

  19. [27]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weis- senborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly et al. , “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 , 2020

    2010 arXiv

  20. [28]

    Maxvit: Multi-axis vision trans- former,

    Z. Tu, H. Talebi, H. Zhang, F. Yang, P. Milanfar, A. Bovik, and Y . Li, “Maxvit: Multi-axis vision trans- former,” in Eur. Conf. Comput. Vis. (ECCV). Springer, 2022, pp. 459–479

    2022

  21. [29]

    Principal components bias in over-parameterized linear models, and its mani- festation in deep neural networks,

    G. Hacohen and D. Weinshall, “Principal components bias in over-parameterized linear models, and its mani- festation in deep neural networks,” Journal of Machine Learning Research, vol. 23, no. 155, pp. 1–46, 2022

    2022

  22. [30]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in IEEE Conf. Comput. Vis. Pattern Recog. (CVPR). Ieee, 2009, pp. 248–255

    2009

  23. [31]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, G. Hinton et al. , “Learning multiple layers of features from tiny images,” 2009

    2009

  24. [32]

    Tiny imagenet visual recognition challenge,

    Y . Le and X. Yang, “Tiny imagenet visual recognition challenge,” CS 231N, vol. 7, no. 7, p. 3, 2015

    2015

  25. [33]

    Learning with noisy labels revisited: A study using real-world human annotations,

    J. Wei, Z. Zhu, H. Cheng, T. Liu, G. Niu, and Y . Liu, “Learning with noisy labels revisited: A study using real-world human annotations,” in Int. Conf. Learn. Represent. (ICLR) , 2022. [Online]. Available: https://openreview.net/forum?id=TBW A6PLJZQm

    2022

  26. [34]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), 2016, pp. 770–778

    2016

  27. [35]

    Making deep neural networks robust to label noise: A loss correction approach,

    G. Patrini, A. Rozza, A. Krishna Menon, R. Nock, and L. Qu, “Making deep neural networks robust to label noise: A loss correction approach,” in IEEE Conf. Comput. Vis. Pattern Recog. (CVPR) , 2017

    2017

  28. [36]

    Towards fair- ness in visual recognition: Effective strategies for bias mitigation,

    Z. Wang, K. Qinami, I. C. Karakozis, K. Genova, P. Nair, K. Hata, and O. Russakovsky, “Towards fair- ness in visual recognition: Effective strategies for bias mitigation,” in IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), 2020, pp. 8919–8928

    2020

  29. [37]

    Men also like shopping: Reducing gender bias amplification using corpus-level constraints,

    J. Zhao, T. Wang, M. Yatskar, V . Ordonez, and K.- W. Chang, “Men also like shopping: Reducing gender bias amplification using corpus-level constraints,” arXiv preprint arXiv:1707.09457, 2017. APPENDIX A ADDITIONAL DEMONSTRATIONS OF FORGETTING We first show more examples of va...

    2017 arXiv

  30. [38]

    Symmetric noise: a fraction p ∈ 0.2, 0.4, 0.6 of labels is randomly selected and replaced uniformly with a different label

  31. [39]

    Asymmetric noise: a fraction p of labels is randomly selected and altered using a fixed label permutation. 2https://github.com/pytorch/vision/tree/main/references/classification 3https://github.com/facebookresearch/ConvNeXt 2 (a) ImageNet, ResNet50, steplr (b) Cifar100, DenseN...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.