Pith. sign in

REVIEW 3 major objections 5 minor 79 references

Effective Data Pruning through Score Extrapolation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that per-sample importance scores computed on a small random subset can be extrapolated across a dataset in embedding space, so that data pruning no longer requires a full initial training pass.

desk verdict Useful empirical extension of score extrapolation to DU/TDDS pruning, but the "consistently beats random" claim is contradicted by their own ImageNet table and the k-selection protocol leaks ground-truth information. read the letter →

arxiv 2506.09010 v2 pith:L3WEEC5Z submitted 2025-06-10 cs.LG

Sebastian Schmidt , Prasanga Dhungel , Christoffer Löffler , Björn Nieth , Stephan Günnemann , Leo Schwinn This is my paper
classification cs.LG
keywords datapruningimportancescoreextrapolationk-nearestneighborsgraphneuralnetworksdynamicuncertaintyTDDSadversarialtrainingunsupervised
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to make expensive data-pruning scores usable on the first training run by never computing them for the whole dataset. It claims that importance scores computed on a random 10-25% subset can be extrapolated to every other sample in the embedding space of a model trained on that subset, using either a weighted k-nearest-neighbor average or a graph neural network. Across CIFAR-10, synthetic CIFAR-100, Places-365, and ImageNet, and for supervised, unsupervised, and adversarial training, the extrapolated scores track the full-training scores well enough that pruning with them beats random pruning whenever the true score does, with a time advantage already at the first run. The point matters because standard pruning methods like Dynamic Uncertainty and TDDS require a full training pass before pruning pays off, which makes them impractical for single large-scale runs.

What carries the argument

The carrying object is the embedding space of the subset-trained model $\mathcal{F}_s$: all samples in the full dataset are mapped through this encoder, and the extrapolators transfer scores along local neighborhoods in that space. KNN uses a distance-weighted average of the $k$ nearest seed scores, while the GNN builds a $k$-NN graph over all samples with edge weights $\exp(-d(\cdot,\cdot))$, and message passing lets scores propagate beyond direct neighbors. The load-bearing assumption is that samples that look similar to the subset-trained model have similar importance, so local averaging in this space can recover unseen scores.

What would settle it

Construct a dataset in which the most important samples for pruning are outliers, such as a class-balanced set where a small fraction of images have unusual backgrounds or multiple objects, then compute ground-truth DU scores, extrapolate from a 20% subset, and check whether the extrapolated rankings recover those high-importance outliers; if rank correlation collapses exactly on those points while staying high on prototypical ones, the local-smoothness premise fails in the regime that matters most for pruning.

Watch

Extended reading notes

Core claim

The central claim is that per-sample importance scores are smooth enough, as a function of a model's embedding, that a score computed on a small random subset can be propagated to the rest of the dataset with near-negligible cost. The paper demonstrates this with two propagators: a KNN weighted average and a three-layer graph convolutional network trained with neighbor sampling, both operating in the embedding space of a model trained only on the subset. On four datasets and for the DU and TDDS pruning scores, the extrapolated rankings correlate with ground-truth rankings and yield final accuracies close to full-training pruning while taking less total time. The paper also reports that GNN extrapolation generally correlates better than KNN, that KNN sits on the Pareto frontier of time versus accuracy, and that extrapolation fails to reproduce the bimodal score distribution, oversmoothing it and erring most on outlier images.

Load-bearing premise

Importance scores are locally smooth in the embedding space of a model trained on a small random subset, so nearby points in that space carry similar scores, and the paper's own outlier analysis shows this smoothness breaks down for atypical images.

Editorial extensions

If this is right

  • Pruning methods that normally need a full training pass become usable on a single large-scale run, because the measured time includes subset training, extrapolation, and final training and still shows savings already at the first run.
  • If the ground-truth pruning score beats random pruning, the extrapolated version of that score also beats random pruning in the reported regimes; if the ground-truth score is no better than random, extrapolation does not rescue it.
  • KNN extrapolation is Pareto-optimal on time versus accuracy for Places-365 and ImageNet, meaning no other tested method gives both better accuracy and less time.
  • The same extrapolation recipe transfers to unsupervised settings using a foundation model's embeddings and pseudo-labels, and to adversarial training, where it improves robustness over random pruning with negligible extra cost.
  • Correlation between extrapolated and ground-truth scores tracks downstream accuracy, so improving extrapolation quality is a direct proxy for improving pruning quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A likely boundary condition that the paper observes but does not turn into a method is that extrapolation oversmooths the score distribution; any practical extension should model bimodality or explicitly detect outliers rather than averaging over them.
  • The framework reads as a general cost-transfer trick: any per-sample quantity that is expensive to compute on the full dataset, such as influence functions or data attribution, could be computed on a seed subset and propagated in embedding space, though the paper only tests pruning scores.
  • A natural stress test would be to extrapolate scores on datasets where importance is concentrated in a thin outlier population, such as mislabeled or long-tail data, since the paper's own qualitative analysis identifies atypical images as the largest source of rank error.
  • The unsupervised experiment hints that the choice of embedding space matters independently of the propagator, because KNN beat GNN there; comparing extrapolation across different frozen and trained encoders would test this directly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a score extrapolation framework for data pruning. Instead of training on the full dataset to compute per-sample importance scores for DU and TDDS, the authors train a model on a small random subset D_s, compute ground-truth scores S_s on that subset, and extrapolate scores to the residual set D_r using either a weighted k-nearest-neighbor average (Eq. 1) or a three-layer GCN on a k-NN graph (Sec. 3.2). The resulting scores are used to prune the full dataset. The authors evaluate the method on CIFAR-10, synthetic CIFAR-100, Places-365, and ImageNet, across supervised, unsupervised, and adversarial training, and report accuracy, score correlations, and end-to-end training time. The central claim is that a 10--25% subset suffices to extrapolate importance scores and that extrapolated scores outperform random pruning whenever the ground-truth scores do, with a time advantage already on the first training run.

Significance. If substantiated, the contribution is practically valuable: it would make DU and TDDS pruning applicable to single large-scale training runs without a full-data scoring pass, and the released code plus the breadth of datasets, pruning methods, and training paradigms are clear strengths. The time-accuracy Pareto analysis in Figure 3 is a useful way to frame the practical trade-off. However, the paper's central better-than-random guarantee is stated too strongly and is contradicted by one of its own reported configurations, and the hyperparameter selection protocol in Appendix B.4 uses the extrapolation target (ground-truth scores on D_r) to choose k, which inflates the reported correlations and downstream results. These issues are load-bearing for the paper's headline claim, so the current evidence supports a more modest version of the claim rather than the version stated in Section 4.

major comments (3)
  1. [Section 4, Table 1] The sentence 'across all datasets, extrapolated scores consistently outperform random pruning whenever the ground truth score does' is directly contradicted by Table 1. For ImageNet at 50% DU pruning, the ground-truth score (59.08 +/- 0.07) beats random pruning (58.56 +/- 0.06), so the stated condition is satisfied; yet the 10%-subset extrapolations fall below random: GNN small gives 58.30 +/- 0.17 and KNN small gives 58.51 +/- 0.14. The same overgeneralized wording appears in Section C.2. Because this better-than-random property is the paper's central practical claim, the claim must be qualified by subset size and configuration, or the affected experiments must be revisited and the text revised to report which configurations actually beat random.
  2. [Appendix B.4] The model-selection protocol leaks information from the extrapolation target. For KNN, k is chosen by varying k over 10, 20, 50, 100 and reporting the value with the highest Pearson correlation between extrapolated and ground-truth scores evaluated on D_r; for the GNN, the graph neighborhood size k is selected in the same way, and the reported correlations are also computed on D_r. Since D_r is the set whose scores the method is supposed to predict without access to ground truth, selecting k on D_r inflates the correlations in Table 2 and gives the downstream results in Table 1 the benefit of test-set selection. A real user would not have the target scores S on D_r. The authors should either select hyperparameters using only a held-out part of D_s, or show that the downstream pruning accuracy is insensitive to k across the full ranges reported in Tables 10 and 11.
  3. [Section 4, Figure 5 and Limitations] The paper's own qualitative analysis shows that extrapolation is least reliable for atypical or out-of-distribution samples: extrapolated DU scores fail to reproduce the bimodal structure of the ground-truth scores, and the largest rank errors concentrate on samples with unusual backgrounds, multiple objects, or low visual quality. This is exactly the regime where importance scores matter most for pruning. The paper should quantify this limitation rather than only illustrate it: for example, report how many of the top-ranked residual samples are misranked by a large margin, or show how pruning accuracy changes when the residual samples are restricted to neighborhoods with high extrapolation confidence. Without such an analysis, the claim that the method 'accurately predict[s] sample importance' overstates the reliability of score extrapolation on the samples that determine pruning quality.
minor comments (5)
  1. [Section 4, Table 3] The sentence 'the GNN's correlation is always higher than the KNN's' is contradicted by Table 3 in the unsupervised setting, where KNN correlations are higher than GNN correlations. The statement should be scoped to the supervised results in Table 2.
  2. [Figure 2] The legend in panels (a) and (c) lists 'KNN-DU' twice, although the figure appears to include KNN-TDDS curves; the caption also contains the typo 'Synthic CIFAR-100'. Please correct these for readability.
  3. [Abstract and Section 3] There are several typos and awkward phrasings, including 'compromising n samples' (should be 'comprising'), 'onlyon on a minor subset', and 'Arvix' in the references. A careful copyedit would improve the presentation.
  4. [Table 1 caption] The caption says the table reports results 'at the highest pruning percentages, where the pruning algorithms still outperform random pruning', but the ImageNet 50% DU row with the 10% subset does not outperform random. Please adjust the caption so it does not pre-judge the comparison being reported.
  5. [Table 6] Table 6 lists subset sizes of 40% and 20% for CIFAR-10, while Section 4 says 'if not stated otherwise, we use 20% of the full dataset for D_s'. Please clarify which results in Figure 2 and Table 1 use the 40% subset and which use the 20% subset.

Circularity Check

1 steps flagged · score 5.0 of 10

KNN/GNN neighborhood size is selected by maximizing Pearson correlation against the ground-truth residual scores used as the evaluation target, so the reported extrapolation correlations are partly fitted rather than predicted.

  1. fitted input called prediction [Appendix B.4 (Scores Extrapolation), Tables 10-11, Table 2]
    "To assess how the choice of k affects extrapolation, we varied k across 10,20,50,100 and evaluated the correlation between the extrapolated and ground-truth scores (based on S) for samples in Dr. The value of k, yielding the highest Pearson correlation, is reported in our main results (Table 2), while the full ablation is presented in Table 10."

    The paper's evaluation of extrapolation quality is the Pearson/Spearman correlation between extrapolated scores and the original scores S on Dr (Table 2 and Figure 4). Appendix B.4 states that the KNN parameter k was chosen precisely by maximizing that same Pearson correlation on Dr, and the selected configuration is the one reported in the main results. Thus Table 2's correlations are optimized on the evaluation target rather than being independent measurements; the claim that 'the GNN's correlation is always higher than the KNN's' and the 'perfectly aligned' correlation-accuracy relationship in Figure 4 inherit this selection.

full rationale

The core extrapolation derivation is not circular: Eq. (1) computes KNN scores as weighted averages of subset scores, Eq. (2) trains the GNN with MSE only on subset nodes, and final pruning accuracy is measured on test sets after retraining. No full-dataset score is fed into the extrapolation itself. Self-citations (e.g., the authors' prior workshop paper [21]) are used only to position the work and are not load-bearing. The main circularity is an evaluation-channel fit: Appendix B.4 selects the KNN neighborhood size k by maximizing Pearson correlation against the ground-truth scores S on the residual set Dr, then reports those correlations in Table 2 and uses them to support the claim that extrapolation correlates well with original scores. Because the same Dr scores are both the selection criterion and the evaluation target, the headline correlation numbers are partly fitted rather than predicted; the same section's GNN neighborhood-size ablation and 'best-performing configuration reported in Table 2' follow the same pattern. This does not make the extrapolated scores equal to ground truth by construction, but it inflates the reported agreement with the target. Separately, Table 1 contradicts the wording 'extrapolated scores consistently outperform random pruning whenever the ground truth score does' (ImageNet 10% DU: GNN small 58.30 and KNN small 58.51 below random 58.56, while ground truth 59.08 exceeds random); that is an empirical overgeneralization and a correctness risk, not a circularity. Overall score 5 reflects one central evaluation quantity being fitted to the target it is used to demonstrate, without the derivation itself being definitionally circular.

Assumptions & free parameters 2 free parameters · 2 assumptions · 0 invented entities

The paper is an empirical method evaluation, not a derivation; its central claim rests on the smoothness of importance scores in the subset-model embedding space and on the representativeness of the random subset. The only fitted quantities are the neighborhood sizes k for KNN and GNN, both selected using ground-truth scores on the extrapolation target.

free parameters (2)
  • KNN neighborhood size k = per-dataset best from {10, 20, 50, 100}, selected by maximizing Pearson correlation on Dr
    Appendix B.4 explicitly selects k that yields the highest correlation on the held-out residual set and reports it in Table 2.
  • GNN graph neighborhood size k = per-dataset best from {10, 20, 50}, selected on Dr correlations in Table 11
    The same selection protocol as KNN is used; the best-performing configuration is reported in Table 2.
assumptions (2)
  • domain assumption Importance scores are locally smooth in the embedding space of the subset-trained model.
    KNN averaging (Eq. 1) and GNN message passing (Sec. 3.2) rely on nearby points having similar scores; Figure 5 shows failures on outlier samples.
  • domain assumption A random 10-25% subset is representative enough to train a model whose embeddings preserve the score geometry of the full dataset.
    Subset Definition (Sec. 3) uses random sampling without representativeness checks; a skewed subset would bias the extrapolation target.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Effective Data Pruning through Score Extrapolation." pith.science (2026). https://pith.science/paper/L3WEEC5Z

@misc{pith2026250609010,
  author       = {Pith},
  title        = {Pith review of: Effective Data Pruning through Score Extrapolation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L3WEEC5Z}},
  note         = {Machine review of arXiv:2506.09010}
}
read the original abstract

Training advanced machine learning models demands massive datasets, resulting in prohibitive computational costs. To address this challenge, data pruning techniques identify and remove redundant training samples while preserving model performance. Yet, existing pruning techniques predominantly require a full initial training pass to identify removable samples, negating any efficiency benefits for single training runs. To overcome this limitation, we introduce a novel importance score extrapolation framework that requires training on only a small subset of data. We present two initial approaches in this framework - k-nearest neighbors and graph neural networks - to accurately predict sample importance for the entire dataset using patterns learned from this minimal subset. We demonstrate the effectiveness of our approach for 2 state-of-the-art pruning methods (Dynamic Uncertainty and TDDS), 4 different datasets (CIFAR-10, CIFAR-100, Places-365, and ImageNet), and 3 training paradigms (supervised, unsupervised, and adversarial). Our results indicate that score extrapolation is a promising direction to scale expensive score calculation methods, such as pruning, data attribution, or other tasks.

Figures

Figures reproduced from arXiv: 2506.09010 by the authors.

Figure 1
Figure 1. Extrapolation concept overview. (I) We start by randomly selecting a subset Ds of m samples out of the full dataset D compromising n samples (where m << n). (II) We train a neural network Fs on the selected subset and calculate ground truth importance scores SS with the selected pruning method during the training run. (III) Using the trained model, we map the whole dataset D to the embedding space of the network. (I… view at source ↗
Figure 2
Figure 2. Evaluation of accuracy for different pruning rates and different datasets. Experiments [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Pareto plots of time to accuracy behavior for different pruning methods and the full dataset [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Analysis of the dependency of correlation and accuracy of the extrapolation methods for [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Score distribution (a) and qualitative analysis (b-c) of extrapolation errors for ImageNet [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Pruning performance of Standard approaches and their extrapolated counterparts, which [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

79 extracted references · 64 canonical work pages

  1. [1]

    Large language models: A survey.Arvix, 2402.06196, 2 2024

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey.Arvix, 2402.06196, 2 2024

    arXiv 2024

  2. [2]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023

    work page 2023

  3. [3]

    Efficient time series processing for transformers and state-space models through token merging

    Leon Götz, Marcel Kollovieh, Stephan Günnemann, and Leo Schwinn. Efficient time series processing for transformers and state-space models through token merging. InProceedings of the International Conference on Machine Learning (ICML), 2025

    work page 2025

  4. [4]

    Byte pair encoding for efficient time series forecasting.arXiv preprint arXiv:2505.14411, 2025

    Leon Götz, Marcel Kollovieh, Stephan Günnemann, and Leo Schwinn. Byte pair encoding for efficient time series forecasting.arXiv preprint arXiv:2505.14411, 2025

    arXiv 2025

  5. [5]

    Advanced active learning strategies for object detection

    Sebastian Schmidt, Qing Rao, Julian Tatsch, and Alois Knoll. Advanced active learning strategies for object detection. InProceedings of the IEEE Intelligent Vehicles Symposium (IV), 2020

    work page 2020

  6. [6]

    Generalized synchronized active learning for multi-agent-based data selection on mobile robotic systems

    Sebastian Schmidt, Lukas Stappen, Leo Schwinn, and Stephan Günnemann. Generalized synchronized active learning for multi-agent-based data selection on mobile robotic systems. IEEE Robotics and Automation Letters, 2024

    work page 2024

  7. [7]

    Large-scale dataset pruning with dynamic uncertainty

    Muyang He, Shuo Yang, Tiejun Huang, and Bo Zhao. Large-scale dataset pruning with dynamic uncertainty. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    work page 2024

  8. [9]

    Datamodels: Predicting predictions from training data

    Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Datamodels: Predicting predictions from training data. InProceedings of the International Conference on Machine Learning (ICML). PMLR, 2022

    work page 2022

Show all 79 references
  1. [10]

    Understanding black-box predictions via influence functions

    Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. InInternational conference on machine learning,(ICML), pages 1885–1894. PMLR, 2017

    2017

  2. [11]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Canadian Institute for Advanced Research, 2009. URL http://www.cs.toronto.edu/~kriz/cifar.html

    2009

  3. [12]

    Places: A 10 million image database for scene recognition.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017

    Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017

    2017

  4. [13]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009

    2009

  5. [14]

    Beyond neural scaling laws: beating power law scaling via data pruning.Advances in Neural Information Processing Systems (NeurIPS), 2022

    Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning.Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  6. [15]

    Spanning training progress: Temporal dual-depth scoring (tdds) for enhanced dataset pruning

    Xin Zhang, Jiawei Du, Yunsong Li, Weiying Xie, and Joey Tianyi Zhou. Spanning training progress: Temporal dual-depth scoring (tdds) for enhanced dataset pruning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  7. [16]

    Active learning for convolutional neural networks: A core-set approach

    Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. InInternational Conference on Learning Representations (ICLR), 2018. 17

    2018

  8. [17]

    What neural networks memorize and why: Discovering the long tail via influence estimation.Advances in Neural Information Processing Systems (NeurIPS), 33:2881–2891, 2020

    Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation.Advances in Neural Information Processing Systems (NeurIPS), 33:2881–2891, 2020

    2020

  9. [18]

    Deepcore: A comprehensive library for coreset selection in deep learning.Database and Expert Systems Applications (DEXA), 4 2022

    Chengcheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selection in deep learning.Database and Expert Systems Applications (DEXA), 4 2022

    2022

  10. [19]

    Generalizing neural wave functions

    Nicholas Gao and Stephan Günnemann. Generalizing neural wave functions. InInternational Conference on Machine Learning, pages 10708–10726. PMLR, 2023

    2023

  11. [20]

    Neural pfaffians: Solving many many-electron schrödinger equations

    Nicholas Gao and Stephan Günnemann. Neural pfaffians: Solving many many-electron schrödinger equations. InThe Thirty-eighth Annual Conference on Neural Information Process- ing Systems (NeurIPS), 2024

    2024

  12. [21]

    Large-scale dataset pruning in adversarial training through data importance extrapolation

    Björn Nieth, Thomas Altstidl, Leo Schwinn, and Björn Eskofier. Large-scale dataset pruning in adversarial training through data importance extrapolation. InICML, DML Workshop, 2024

    2024

  13. [22]

    Data pruning via moving-one-sample-out.Advances in neural information processing systems (NeurIPS), 2023

    Haoru Tan, Sitong Wu, Fei Du, Yukang Chen, Zhibin Wang, Fan Wang, and Xiaojuan Qi. Data pruning via moving-one-sample-out.Advances in neural information processing systems (NeurIPS), 2023

    2023

  14. [23]

    An empirical study of example forgetting during deep neural network learning

    Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. InInternational Conference on Learning Representations (ICLR), 2019

    2019

  15. [24]

    Deep learning on a data diet: Finding important examples early in training.Advances in Neural Information Processing Systems (NeurIPS), 34, 2021

    Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training.Advances in Neural Information Processing Systems (NeurIPS), 34, 2021

    2021

  16. [25]

    Selection via proxy: Efficient data selection for deep learning

    Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. InInternational Conference on Learning Representations (ICLR), 2020

    2020

  17. [26]

    Identifying mislabeled data using the area under the margin ranking.Advances in Neural Information Processing Systems, 33:17044–17056, 2020

    Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking.Advances in Neural Information Processing Systems, 33:17044–17056, 2020

    2020

  18. [27]

    Dataset pruning: Reducing training data by examining generalization influence

    Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reducing training data by examining generalization influence. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  19. [28]

    Herding dynamical weights to learn

    Max Welling and Donald Bren. Herding dynamical weights to learn. InProceedings of the International Conference on Machine Learning (ICML), 2009

    2009

  20. [29]

    Super-samples from kernel herding

    Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. InProceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence (UAI), 2010

    2010

  21. [30]

    Moderate coreset: A uni- versal method of data selection for real-world data-efficient deep learning

    Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A uni- versal method of data selection for real-world data-efficient deep learning. Inthe International Conference on Learning Representations (ICLR), 2022

    2022

  22. [31]

    Efficient and robust quantization-aware training via adaptive coreset selection.Transaction on Machine Learning (TMLR), 8 2024

    Xijie Huang, Zechun Liu, Shih-Yang Liu, and Kwang-Ting Cheng. Efficient and robust quantization-aware training via adaptive coreset selection.Transaction on Machine Learning (TMLR), 8 2024

    2024

  23. [32]

    Coresets for data-efficient training of machine learning models

    Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. InProceedings of the International Conference on Machine Learning (ICML), 2020

    2020

  24. [33]

    Maximum margin coresets for active and noise tolerant learning

    Sariel Har-Peled, Dan Roth, and Dav A Zimak. Maximum margin coresets for active and noise tolerant learning. InProceedings of the international joint conference on Artifical intelligence (JCAI), 2006

    2006

  25. [34]

    Coverage-centric coreset selection for high pruning rates

    Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric coreset selection for high pruning rates. InInternational Conference on Learning Representations (ICLR), 2023. 18

    2023

  26. [35]

    Zero-shot coreset selection: Efficient pruning for unlabeled data.Arxiv, 2411.15349, 2024

    Brent A Griffin, Jacob Marks, and Jason J Corso. Zero-shot coreset selection: Efficient pruning for unlabeled data.Arxiv, 2411.15349, 2024

    2024

  27. [36]

    Coresets via bilevel optimization for continual learning and streaming

    Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. InAdvances in Neural Information Processing Systems (NeurIPS), volume 33, pages 14879–14890, 2020

    2020

  28. [37]

    Glis- ter: Generalization based data subset selection for efficient and robust learning

    Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glis- ter: Generalization based data subset selection for efficient and robust learning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021

    2021

  29. [38]

    Grad-match: Gradient matching based data subset selection for efficient deep model training

    Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. InInternational Conference on Machine Learning, pages 5464–5474. PMLR, 2021

    2021

  30. [39]

    Towards deep learning models resistant to adversarial attacks.arXiv preprint arXiv:1706.06083, 2017

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks.arXiv preprint arXiv:1706.06083, 2017

    2017 arXiv

  31. [40]

    Clip: Cheap lipschitz training of neural networks

    Leon Bungert, René Raab, Tim Roith, Leo Schwinn, and Daniel Tenbrinck. Clip: Cheap lipschitz training of neural networks. InInternational Conference on Scale Space and Variational Methods in Computer Vision, pages 307–319. Springer International Publishing Cham, 2021

    2021

  32. [41]

    Better diffusion models further improve adversarial training

    Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. InInternational Conference on Machine Learning (ICML), 2023

    2023

  33. [42]

    On the scalability of certified adversarial robustness with generated data

    Thomas Altstidl, David Dobre, Björn Eskofier, Gauthier Gidel, and Leo Schwinn. On the scalability of certified adversarial robustness with generated data. InNeurIPS, 2024

    2024

  34. [43]

    Efficient adversarial training in llms with continuous attacks

    Sophie Xhonneux, Alessandro Sordoni, Stephan Günnemann, Gauthier Gidel, and Leo Schwinn. Efficient adversarial training in llms with continuous attacks. InNeurIPS, 2024

    2024

  35. [44]

    Identifying untrustworthy predictions in neural networks by geometric gradient analysis

    Leo Schwinn, An Nguyen, René Raab, Leon Bungert, Daniel Tenbrinck, Dario Zanca, Martin Burger, and Bjoern Eskofier. Identifying untrustworthy predictions in neural networks by geometric gradient analysis. InUAI, pages 854–864. PMLR, 2021

    2021

  36. [45]

    Improving robustness against real-world and worst-case distribution shifts through decision region quantification

    Leo Schwinn, Leon Bungert, An Nguyen, René Raab, Falk Pulsmeyer, Doina Precup, Björn Eskofier, and Dario Zanca. Improving robustness against real-world and worst-case distribution shifts through decision region quantification. InProceedings of the International Conference on M...

    2022

  37. [46]

    Collec- tive robustness certificates: Exploiting interdependence in graph neural networks

    Jan Schuchardt, Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Collec- tive robustness certificates: Exploiting interdependence in graph neural networks. InInterna- tional Conference on Learning Representations, 2021

    2021

  38. [47]

    Invariance-aware randomized smoothing certificates

    Jan Schuchardt and Stephan Günnemann. Invariance-aware randomized smoothing certificates. InConference on Neural Information Processing Systems (NeurIPS), 2022

    2022

  39. [48]

    Dynamically sampled nonlocal gradients for stronger adversarial attacks

    Leo Schwinn, An Nguyen, René Raab, Dario Zanca, Bjoern M Eskofier, Daniel Tenbrinck, and Martin Burger. Dynamically sampled nonlocal gradients for stronger adversarial attacks. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2021

    2021

  40. [49]

    Exploring mis- classifications of robust neural networks to enhance adversarial attacks.Applied Intelligence, 2023

    Leo Schwinn, René Raab, An Nguyen, Dario Zanca, and Bjoern Eskofier. Exploring mis- classifications of robust neural networks to enhance adversarial attacks.Applied Intelligence, 2023

    2023

  41. [50]

    Assessing robustness via score-based adversarial image generation.Transactions on Machine Learning Research (TMLR), 2023

    Marcel Kollovieh, Lukas Gosch, Yan Scholten, Marten Lienen, Leo Schwinn, and Stephan Günnemann. Assessing robustness via score-based adversarial image generation.Transactions on Machine Learning Research (TMLR), 2023

    2023

  42. [51]

    Localized randomized smoothing for collective robustness certification

    Jan Schuchardt, Tom Wollschläger, Aleksandar Bojchevski, and Stephan Günnemann. Localized randomized smoothing for collective robustness certification. InInternational Conference on Learning Representations (ICLR), 2023. 19

    2023

  43. [52]

    Edward Suh

    Weizhe Hua, Yichi Zhang, Chuan Guo, Zhiru Zhang, and G. Edward Suh. BulletTrain: Accel- erating robust neural network training via boundary example mining. InAdvances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  44. [53]

    Data filtering for efficient adversarial training.Pattern Recognition, 151, 2024

    Erh-Chung Chen and Che-Rung Lee. Data filtering for efficient adversarial training.Pattern Recognition, 151, 2024

    2024

  45. [54]

    GRAD-MATCH: Gradient matching based data subset selection for efficient deep model training.PMLR, 2021

    Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. GRAD-MATCH: Gradient matching based data subset selection for efficient deep model training.PMLR, 2021

    2021

  46. [55]

    Dolatabadi, Sarah Erfani, and Christopher Leckie

    Hadi M. Dolatabadi, Sarah Erfani, and Christopher Leckie. Adversarial coreset selection for efficient robust training.International Journal of Computer Vision, 131(12):3307–3331, 2023

    2023

  47. [56]

    Efficient adversarial training with data pruning.Arvix, 2207.00694, 2022

    Maximilian Kaufmann, Yiren Zhao, Ilia Shumailov, Robert Mullins, and Nicolas Papernot. Efficient adversarial training with data pruning.Arvix, 2207.00694, 2022

    2022 arXiv

  48. [57]

    Less is more: Data pruning for faster adversarial training.Arvix, 2302.12366], 2023

    Yize Li, Pu Zhao, Xue Lin, Bhavya Kailkhura, and Ryan Goldhahn. Less is more: Data pruning for faster adversarial training.Arvix, 2302.12366], 2023

    2023 arXiv

  49. [58]

    A comprehensive survey of continual learning: Theory, method and application.IEEE Transactions on Pattern Analysis and Machine Intelligence, 46, 8 2024

    Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application.IEEE Transactions on Pattern Analysis and Machine Intelligence, 46, 8 2024

    2024

  50. [59]

    Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation. Arvix, 1811.10959, 11 2018

    2018 arXiv

  51. [60]

    Holder and Muhammad Shafique

    Christopher J. Holder and Muhammad Shafique. Efficient uncertainty estimation in semantic segmentation via distillation. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021

    2021

  52. [61]

    Unifying approaches in active learning and active sampling via fisher information and information-theoretic quantities.Transactions on Machine Learning Research (TMLR), 2022

    Andreas Kirsch and Yarin Gal. Unifying approaches in active learning and active sampling via fisher information and information-theoretic quantities.Transactions on Machine Learning Research (TMLR), 2022

    2022

  53. [62]

    A uni- fied approach towards active learning and out-of-distribution detection.arXiv preprint arXiv:2405.11337, 2024

    Sebastian Schmidt, Leonard Schenk, Leo Schwinn, and Stephan Günnemann. A uni- fied approach towards active learning and out-of-distribution detection.arXiv preprint arXiv:2405.11337, 2024

    2024

  54. [63]

    Joint out-of- distribution filtering and data discovery active learning

    Sebastian Schmidt, Leonard Schenk, Leo Schwinn, and Stephan Günnemann. Joint out-of- distribution filtering and data discovery active learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  55. [64]

    Iale: Imitating active learner ensembles.Journal of Machine Learning Research, 23(107):1–29, 2022

    Christoffer Löffler and Christopher Mutschler. Iale: Imitating active learner ensembles.Journal of Machine Learning Research, 23(107):1–29, 2022

    2022

  56. [65]

    Active learning of ordinal embeddings: A user study on football data.Transactions on Machine Learning Research, 2023

    Christoffer Löffler, Kion Fallah, Stefano Fenu, Dario Zanca, Bjoern Eskofier, Christopher John Rozell, and Christopher Mutschler. Active learning of ordinal embeddings: A user study on football data.Transactions on Machine Learning Research, 2023

    2023

  57. [66]

    Nikolakakis, Amin Karbasi, Dionysis Kalogerias, Nezihe Merve Gürel, and Theodoros Rekatsinas

    Patrik Okanovic, Roger Waleffe, Vasilis Mageirakos, Konstantinos E. Nikolakakis, Amin Karbasi, Dionysis Kalogerias, Nezihe Merve Gürel, and Theodoros Rekatsinas. Repeated random sampling for minimizing the time-to-accuracy of learning. InProceedings of the International Confer...

    2024

  58. [67]

    Exploring data redundancy in real-world image classification through data selection.Arvix, 2306.14113, 2023

    Zhenyu Tang, Shaoting Zhang, and Xiaosong Wang. Exploring data redundancy in real-world image classification through data selection.Arvix, 2306.14113, 2023

    2023 arXiv

  59. [68]

    Semi-supervised classification with graph convolutional networks

    Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. InInternational Conference on Learning Representations (ICLR), 2017

    2017

  60. [69]

    Inductive representation learning on large graphs.Advances in Neural Information Processing Systems (NeurIPS), 30, 2017

    Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs.Advances in Neural Information Processing Systems (NeurIPS), 30, 2017. 20

    2017

  61. [70]

    Pearson correlation coefficient

    Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. Pearson correlation coefficient. InNoise reduction in speech processing, pages 37–40. Springer, 2009

    2009

  62. [71]

    Spearman rank correlation.Encyclopedia of Biostatistics, 7, 2005

    Jerrold H Zar. Spearman rank correlation.Encyclopedia of Biostatistics, 7, 2005

    2005

  63. [72]

    Let go of your labels with unsupervised transfer

    Artyom Gadetsky, Yulun Jiang, and Maria Brbic. Let go of your labels with unsupervised transfer. InProceedings of the International Conference on Machine Learning (ICML). PMLR, 6 2024

    2024

  64. [73]

    Denoising diffusion probabilistic models.Advances in Neural Information Processing Systems (NeurIPS), 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in Neural Information Processing Systems (NeurIPS), 33:6840–6851, 2020

    2020

  65. [74]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016

    2016

  66. [75]

    Wide residual networks

    Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. InIn Proceedings of the British Machine Vision Conference (BMVC), 2016

    2016

  67. [76]

    Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  68. [77]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InInterna- tional Conference on Learning Representations (ICLR), 2015

    2015

  69. [78]

    On the importance of initialization and momentum in deep learning

    Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. InInternational Conference on Machine Learning (ICML), pages 1139–1147. PMLR, 2013

    2013

  70. [79]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations (ICLR), 2017

    2017

  71. [80]

    Super-convergence: Very fast training of neural networks using large learning rates

    Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. InArtificial intelligence and machine learning for multi-domain operations applications, volume 11006, pages 369–386. SPIE, 2019. 21

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.