Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

SCAN: Bootstrapping Contrastive Pre-training for Data Efficiency

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Pretraining with 30% less data costs under 1% accuracy

desk verdict SCAN's bootstrapping schedule and broad experiments are valuable, but the missing dynamic-random control leaves the core attribution unproven. read the letter →

arxiv 2411.09126 v1 pith:XHZKSEMV submitted 2024-11-14 cs.CV

Yangyang Guo , Mohan Kankanhalli This is my paper · ORCID
classification cs.CV
keywords contrastivepre-trainingdatapruningcoresetselectiondynamicsparsetrainingInfoNCElossCLIPMoCoefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that contrastive pre-training can be made substantially more data-efficient by pruning training samples dynamically, rather than selecting a fixed coreset in advance. It introduces SCAN, a bootstrapping method that every few epochs scores each sample by its in-batch InfoNCE loss, removes both the easiest redundant samples and the hardest ill-matched samples under a cosine-annealed schedule, then grows back to the full dataset and repeats. Across 16 pre-trained models spanning CLIP and MoCo, pruning 30-35% of training data keeps downstream accuracy within 1% on average of full-data training and beats static coreset baselines by a large margin. If the claim holds, SCAN offers a cheap way to reduce pre-training compute and carbon footprint without sacrificing model quality.

What carries the argument

The engine of the method is the bootstrapping dataset mutation loop. Every $\tau_{\text{cos}}+1$ epochs, the method recomputes candidate pruning data from in-batch InfoNCE losses: the $\rho$ smallest-loss samples are labeled redundant and the $\rho$ largest-loss samples are labeled ill-matched, and their union across the two loss directions $L_{f\to g}$ and $L_{g\to f}$ forms the candidate pool. Within each round, the actual pruning ratio follows a cosine-annealing curve $\rho_{\text{cur}}$ that rises from 0 to $2\rho$, so the dataset is gradually shrunk to an average ratio $\rho$ and then restored to full size for the next round. This dynamic schedule is what distinguishes SCAN from static coreset selection, and a warm-up rule—start pruning only after epoch-wise loss differences fall below a threshold—protects early unstable training.

What would settle it

Record per-sample InfoNCE losses every epoch for a small CLIP model and check rank-order stability between the warm-up epoch and later epochs; if a substantial fraction of samples, say more than 10%, move from the lowest-loss quintile to the highest-loss quintile, or vice versa, the pruning decision would be wrong for those samples. Alternatively, take the pruned set from SCAN at a 30% ratio, add back a random 5% of pruned samples during the final training epochs, and measure whether downstream accuracy rises by more than the reported noise; if it does, those samples were not truly redundant.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that dynamic, loss-based dataset bootstrapping during contrastive pre-training can reduce training data by roughly one third while preserving almost all downstream performance. The central discovery is that the per-sample InfoNCE loss computed inside each batch is a usable indicator of sample usefulness across the training trajectory: low-loss samples have been memorized and can be dropped as redundant, high-loss samples are semantically mismatched and can be dropped as ill-matched, and pruning the union of these two extremes under a periodically increasing cosine schedule preserves the information content of the full dataset. The method is validated on seven CLIP architectures on two image-text datasets and two MoCo architectures on ImageNet, for a total of 16 pre-trained models, with an average performance drop below 1% at 30-35% pruning.

Load-bearing premise

The load-bearing premise is that a sample's in-batch InfoNCE loss at pruning time is a reliable and stable signal of its future usefulness—specifically, that low-loss samples stay redundant and high-loss samples stay ill-matched for the rest of training, so neither ever becomes crucial later.

Editorial extensions

If this is right

  • At a 30-35% pruning ratio, SCAN models stay within 1% average downstream performance of models trained on the full dataset, across both CLIP and MoCo.
  • SCAN consistently outperforms static coreset baselines SemDeDup and D-Pruning, and the dynamic baseline Info-Batch, on zero-shot, linear probing, and fine-tuning evaluations.
  • Pre-training time drops by roughly 25-30% because the same number of epochs is trained on fewer samples.
  • The coresets left after SCAN pre-training, especially when intersected across two architectures, outperform existing static coresets when used to pre-train a new model from scratch.
  • The benefit holds across vision-language (CLIP) and vision-centric (MoCo) frameworks and across ResNet, ViT, and Swin architectures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The in-batch loss signal is cheap enough that SCAN could plausibly scale to billion-sample web datasets, but the paper only tests up to 10.1M pairs, so whether the sub-1% bound survives at web scale is an open question.
  • The two-sided pruning rule implies a testable claim: samples that are memorized or mismatched are interchangeable across architectures, as the overlap results partially suggest; a direct experiment would check whether a sample pruned by one architecture is ever critical for another.
  • The cosine-annealed bootstrapping schedule resembles a curriculum over data difficulty; one could test whether the schedule itself, rather than the specific loss cutoff, drives the gains by comparing against fixed-ratio dynamic pruning at the same average rate.
  • The paper notes that LLM pre-training typically runs for only 1-3 epochs, so the method may transfer more readily to other multi-epoch contrastive objectives, such as sentence embedding training, than to single-pass LLM training.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes SCAN, a dynamic bootstrapping dataset pruning method for contrastive pre-training. Instead of selecting a static coreset once, SCAN identifies candidate samples from per-batch InfoNCE losses, keeping the very low-loss samples (treated as redundant) and very high-loss samples (treated as ill-matched), and then prunes a varying number of these candidates through a cosine-annealing mutation schedule with periodic regrowth. The method is evaluated on CLIP pre-training on CC3M+ and CC12M+ and on MoCo pre-training on ImageNet, across 16 total models, with pruning rates of 30–35%. The paper reports less than 1% average downstream degradation relative to full-data training, claims consistent gains over SemDeDup, D-Pruning, Info-Batch, and static random baselines, and shows that the produced static coresets outperform existing static coreset methods.

Significance. If the empirical claims hold, the paper makes a practically useful contribution: it shows that contrastive pre-training can use 30–35% less data on average with a relatively small downstream cost, and that dynamic, schedule-driven pruning can be competitive with or better than static coreset selection. The scope is broad for this line of work: seven CLIP architectures, two MoCo architectures, two CLIP dataset scales, several downstream and out-of-distribution evaluation sets, and comparisons with four baselines. The paper also ships code, which supports reproducibility. The main limitations are statistical and methodological: the headline 'less than 1% average degradation' is not supported by error bars or multiple seeds, and the specific benefit of the loss-extreme selection rule is not isolated from the bootstrapping schedule by a dynamic random-candidate control. These are fixable with additional experiments and more careful wording, rather than being fundamental flaws in the method's design.

major comments (3)
  1. [§3.3.1 Eq. (2), §3.3.2, §4.1 Tables 1–3, §4.4 Table 7] The central claim that loss-extreme selection drives the improvement is not isolated from the bootstrapping schedule. The Random baseline in Tables 1–3 is a static 70% subset, and the ablation in Table 7 only compares redundant vs. ill-matched selection with a fixed schedule. No control runs the same warm-up, cosine mutation schedule (Eq. 3), regrowth rounds, and epoch count while replacing Eq. (2) with uniform random candidate selection. Without such a control, the observed gains over static baselines could be attributable to the schedule alone. Please add this control or explicitly weaken the claim.
  2. [Abstract, Tables 1 and 3] The abstract's 'less than 1% on average' claim is not backed by per-seed variation or error bars, and the average conceals substantially larger per-metric degradations. For example, in Table 1, ViT-B/32 on CC12M+ drops from 63.40 to 61.55 on ImageNet Top-1 (-1.85 points) and from 31.09 to 29.23 on ImageNet-R (-1.86 points); in Table 3, RN101 on CC3M+ drops from 39.69 to 38.05 on ImageNet-R (-1.64 points). Please report standard deviations over multiple seeds, or explicitly state that results are single runs, and qualify the average-degradation claim accordingly.
  3. [§3.3.2, Eq. (3), and Table 8] The pruning-ratio bookkeeping is internally inconsistent and prevents exact reproduction. Eq. (3) ranges from 0 to 1 as τcur mod (τcos+1) goes from 0 to τcos, yet the text says 'ρcur increases from 0 to 2ρ'; the candidate set size is 2ρ|Dt| in §3.3.1, §4.1 states a pruning ratio of 30% for CLIP, and Table 8 lists R.v.s.I. ratios (30:10), (20:20), (10:30) that do not match an equal 15%:15% split of a 30% total. Please define ρ, ρcur, and the candidate-set fraction unambiguously and align the text, formulas, and table.
minor comments (6)
  1. [§3.3.1, Eq. (2)] The symbols ≺ρ and ≻ρ are not defined; please state explicitly that they return the indices of the ρ smallest and ρ largest per-sample losses in the batch.
  2. [§3.3.1] The phrase 'merge the subset intersection from Lf→g and Lg→f' is ambiguous; please specify whether the final candidate set is the union or the intersection of the two directional selections.
  3. [§4.1] The architecture list includes RN50, ViT-S/32, and ViT-S/16, but these do not appear in Tables 1 and 3; please point to the corresponding results in the supplementary material.
  4. [Figure 3] The y-axis ranges in Figure 3 are truncated (e.g., FT Acc from 56.00 to 57.00), which visually exaggerates the differences; please use full-scale axes or add error bars.
  5. [§4.2 and §2] There are minor typos: 'doamins' should be 'domains' in Section 2, and 'SACN' should be 'SCAN' in Section 4.2.
  6. [§3.3.1 and §3.3.2] The warm-up threshold Ttd and the mutation epoch τcos are free hyperparameters; please provide their values for each dataset and architecture in a reproducibility table.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; SCAN is an empirical benchmark study whose pruning rate is an experimental control, not a derived prediction.

full rationale

This paper makes no first-principles derivation that reduces to its inputs. The selection rule in Eq. (2) is a heuristic criterion for choosing pruned candidates, and downstream accuracy is measured independently of that loss; low/high InfoNCE loss does not by construction imply better or worse downstream performance. The pruning rate is a controlled experimental variable: Section 3.3.2 states that the average pruning ratio is fixed at the predefined ρ, so the reported 30–35% pruning rate is an input setting, not a predicted output. The paper contains no load-bearing self-citations; the authors do not cite their own prior work, and the external citations are standard methods and datasets rather than an unverified uniqueness or ansatz source. The absence of a dynamic random-candidate ablation is a legitimate missing-control concern for isolating the contribution of the loss-extreme selection rule, but that is a correctness/experimental-design issue, not circularity. The coreset results in Section 4.3 are also empirical transfer evaluations, not equivalences by construction. Accordingly, no circular step can be exhibited from the paper's own equations or citation chain.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical entities. Its empirical effectiveness rests on several assumptions about loss-based pruning and schedule stability, which are plausible but unproven. The free parameters are standard hyperparameters, though the warm-up threshold is unspecified.

free parameters (3)
  • pruning ratio rho = 0.30 (CLIP), 0.35 (MoCo)
    Chosen by hand based on the desired trade-off; ablation in Fig. 3 shows performance degrades with higher ratios.
  • mutation epoch tau_cos = 3
    Chosen from ablation in Table 6; values 2 and 4 are worse.
  • warm-up threshold Ttd = not reported
    Used to decide when to start pruning; value not given in the paper.
assumptions (3)
  • domain assumption Per-sample InfoNCE loss within a batch is a reliable indicator of whether a sample is redundant (low loss) or ill-matched (high loss).
    Used in Eq. (2) to select pruning candidates. If loss is noisy across batches, the selected candidates may not be the most useful ones to remove.
  • domain assumption Dynamically pruning and regrowing the dataset with a cosine schedule keeps training stable and preserves downstream performance.
    The method's core mechanism; no proof is provided that this schedule avoids catastrophic forgetting or distribution shift.
  • domain assumption The in-batch loss comparison is meaningful across batches for identifying candidates.
    The authors argue saving full-dataset losses is expensive, so they use in-batch losses. This assumes the batch composition does not dominate the loss values.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SCAN: Bootstrapping Contrastive Pre-training for Data Efficiency." pith.science (2026). https://pith.science/paper/XHZKSEMV

@misc{pith2026241109126,
  author       = {Pith},
  title        = {Pith review of: SCAN: Bootstrapping Contrastive Pre-training for Data Efficiency},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XHZKSEMV}},
  note         = {Machine review of arXiv:2411.09126}
}
read the original abstract

While contrastive pre-training is widely employed, its data efficiency problem has remained relatively under-explored thus far. Existing methods often rely on static coreset selection algorithms to pre-identify important data for training. However, this static nature renders them unable to dynamically track the data usefulness throughout pre-training, leading to subpar pre-trained models. To address this challenge, our paper introduces a novel dynamic bootstrapping dataset pruning method. It involves pruning data preparation followed by dataset mutation operations, both of which undergo iterative and dynamic updates. We apply this method to two prevalent contrastive pre-training frameworks: \textbf{CLIP} and \textbf{MoCo}, representing vision-language and vision-centric domains, respectively. In particular, we individually pre-train seven CLIP models on two large-scale image-text pair datasets, and two MoCo models on the ImageNet dataset, resulting in a total of 16 pre-trained models. With a data pruning rate of 30-35\% across all 16 models, our method exhibits only marginal performance degradation (less than \textbf{1\%} on average) compared to corresponding models trained on the full dataset counterparts across various downstream datasets, and also surpasses several baselines with a large performance margin. Additionally, the byproduct from our method, \ie coresets derived from the original datasets after pre-training, also demonstrates significant superiority in terms of downstream performance over other static coreset selection approaches.

Figures

Figures reproduced from arXiv: 2411.09126 by the authors.

Figure 1
Figure 1. The interplay between training data size and model [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall pipeline of the proposed SCAN method. We begin by identifying a substantial portion of data samples as [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Downstream performance variation of two CLIP models [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of pre-training time between the base CLIP [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Examples of ill-matched samples identified by our SCAN. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Differential-informed Sample Selection Accelerates Multimodal Contrastive Learning

    cs.CV 2025-07 conditional novelty 6.0 of 10

    DISSect selects training samples for multimodal contrastive learning by ranking the difference between historical and current model similarity scores, matching full-data performance with 70% fewer samples.

Reference graph

Works this paper leans on

73 extracted references · 67 canonical work pages · cited by 1 Pith paper

  1. [1]

    Amro Abbas, Kushal Tirumala, Daniel Simig, Surya Ganguli, and Ari S. Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication. CoRR, 2023

    work page 2023

  2. [2]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: visual question answering. In ICCV, pages 2425–2433. IEEE, 2015

    work page 2015

  3. [3]

    Beit: BERT pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: BERT pre-training of image transformers. In ICLR. OpenRe- view.net, 2022

    work page 2022

  4. [4]

    Clip retrieval: Easily compute clip em- beddings and build a clip retrieval system with them, 2022

    Romain Beaumont. Clip retrieval: Easily compute clip em- beddings and build a clip retrieval system with them, 2022

    work page 2022

  5. [5]

    Bowyer and Patrick J

    Kevin W. Bowyer and Patrick J. Flynn. A 20th anniversary survey: Introduction to ’content-based image retrieval at the end of the early years’. TPAMI, 22(12):1348, 2000

    work page 2000

  6. [6]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. InICCV, pages 9630–9640. IEEE, 2021

    work page 2021

  7. [7]

    Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, pages 3558–3568. IEEE, 2021

    work page 2021

  8. [8]

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geof- frey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, pages 1597–1607. PMLR, 2020

    work page 2020

Show all 73 references
  1. [9]

    Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. Big self-supervised models are strong semi-supervised learners. In NeurIPS, 2020

    2020

  2. [10]

    Girshick, and Kaiming He

    Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. CoRR, 2020

    2020

  3. [11]

    An empirical study of training self-supervised vision transformers

    Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. InICCV, pages 9620–9629. IEEE, 2021

    2021

  4. [12]

    Data distillation can be like vodka: Distilling more times for better quality

    Xuxi Chen, Yu Yang, Zhangyang Wang, and Baharan Mirza- soleiman. Data distillation can be like vodka: Distilling more times for better quality. In ICLR, 2024

    2024

  5. [13]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255. IEEE, 2009

    2009

  6. [14]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  7. [15]

    Sequential subset matching for dataset distillation

    Jiawei Du, Qin Shi, and Joey Tianyi Zhou. Sequential subset matching for dataset distillation. In NeurIPS, 2023

    2023

  8. [16]

    Second thoughts on the bootstrap

    Bradley Efron. Second thoughts on the bootstrap. Statistical science, pages 135–140, 2003

    2003

  9. [17]

    Rigging the lottery: Making all tickets winners

    Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In ICML, pages 2943–2952. PMLR, 2020

    2020

  10. [18]

    What neural networks memorize and why: Discovering the long tail via influence estimation

    Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In NeurIPS, 2020

    2020

  11. [19]

    Simcse: Sim- ple contrastive learning of sentence embeddings

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Sim- ple contrastive learning of sentence embeddings. In EMNLP, pages 6894–6910. ACL, 2021

    2021

  12. [20]

    A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation

    Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. In ICLR. OpenReview.net, 2019

    2019

  13. [21]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InCVPR, pages 770–778. IEEE, 2016

    2016

  14. [22]

    Girshick

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scal- able vision learners. In CVPR, pages 15979–15988. IEEE, 2022

    2022

  15. [23]

    The many faces of robustness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada- vath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ...

    2021

  16. [24]

    Clipscore: A reference-free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In EMNLP, pages 7514–7528. ACL, 2021

    2021

  17. [25]

    Rae, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Si- mo...

    2022

  18. [26]

    Openclip, 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Ha- jishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021

    2021

  19. [27]

    Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V . Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pages 4904–

  20. [28]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.08361, 2020

    2001 arXiv

  21. [29]

    Understanding black-box predictions via influence functions

    Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In ICML, pages 1885–

  22. [30]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  23. [31]

    Tiny imagenet visual recognition challenge

    Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015. 9

    2015

  24. [32]

    Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Snip: single-shot network pruning based on connection sensitivity. In ICLR. OpenReview.net, 2019

    2019

  25. [33]

    Error norm truncation: Robust training in the presence of data noise for text generation models

    Tianjian Li, Haoran Xu, Philipp Koehn, Daniel Khashabi, and Kenton Murray. Error norm truncation: Robust training in the presence of data noise for text generation models. In ICLR, 2024

    2024

  26. [34]

    UP-DP: unsupervised prompt learning for data pre-selection with vision-language models

    Xin Li, Sima Behpour, Thang Long Doan, Wenbin He, Liang Gou, and Liu Ren. UP-DP: unsupervised prompt learning for data pre-selection with vision-language models. In NeurIPS, 2023

    2023

  27. [35]

    An inverse scaling law for CLIP training

    Xianhang Li, Zeyu Wang, and Cihang Xie. An inverse scaling law for CLIP training. In NeurIPS, 2023

    2023

  28. [36]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, pages 740–755. Springer, 2014

    2014

  29. [37]

    Do we actually need dense over- parameterization? in-time over-parameterization in sparse training

    Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense over- parameterization? in-time over-parameterization in sparse training. In ICML, pages 6989–7000. PMLR, 2021

    2021

  30. [38]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 9992–10002. IEEE, 2021

    2021

  31. [39]

    SGDR: stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In ICLR. OpenReview.net, 2017

    2017

  32. [40]

    D2 pruning: Message passing for balancing diversity and diffi- culty in data pruning

    Adyasha Maharana, Prateek Yadav, and Mohit Bansal. D2 pruning: Message passing for balancing diversity and diffi- culty in data pruning. In ICLR, 2024

    2024

  33. [41]

    SIEVE: multimodal dataset pruning using image captioning models

    Anas Mahmoud, Mostafa Elhoushi, Amro Abbas, Yu Yang, Newsha Ardalani, Hugh Leather, and Ari Morcos. SIEVE: multimodal dataset pruning using image captioning models. In CVPR. IEEE, 2024

    2024

  34. [42]

    Bilmes, and Jure Leskovec

    Baharan Mirzasoleiman, Jeff A. Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning mod- els. In ICML, pages 6950–6960. PMLR, 2020

    2020

  35. [43]

    Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science

    Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature com- munications, 9(1):2383, 2018

    2018

  36. [44]

    Using relevance to reduce network size automatically

    Michael C Mozer and Paul Smolensky. Using relevance to reduce network size automatically. Connection Science, 1(1): 3–16, 1989

    1989

  37. [45]

    Fantastic weights and how to find them: Where to prune in dynamic sparse training

    Aleksandra Nowak, Bram Grooten, Decebal Constantin Mo- canu, and Jacek Tabor. Fantastic weights and how to find them: Where to prune in dynamic sparse training. In NeurIPS, 2023

    2023

  38. [46]

    Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. Im2text: Describing images using 1 million captioned pho- tographs. In NIPS, pages 1143–1151, 2011

    2011

  39. [47]

    Deep learning on a data diet: Finding important examples early in training

    Mansheej Paul, Surya Ganguli, and Gintare Karolina Dz- iugaite. Deep learning on a data diet: Finding important examples early in training. In NeurIPS, pages 20596–20607, 2021

    2021

  40. [48]

    Infobatch: Loss- less training speed up by unbiased dynamic data pruning

    Ziheng Qin, Kai Wang, Zangwei Zheng, Jianyang Gu, Xi- angyu Peng, Daquan Zhou, and Yang You. Infobatch: Loss- less training speed up by unbiased dynamic data pruning. In ICLR, 2024

    2024

  41. [49]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, pages 8748–...

    2021

  42. [50]

    Do imagenet classifiers generalize to ima- genet? In ICML, pages 5389–5400

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to ima- genet? In ICML, pages 5389–5400. PMLR, 2019

    2019

  43. [51]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Robert Kaczmarczyk, Aran Komat- suzaki, Aarush Katta, Richard Vencu, Romain Beaumont, Je- nia Jitsev, Theo Coombes, and Clayton Mullis. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. In NeurIPS Workshop Datacentric AI. Jülich Super...

    2021

  44. [52]

    LAION- 5B: an open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Lud- wig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAI...

    2022

  45. [53]

    MIM4DD: mutual information maximization for dataset distillation

    Yuzhang Shang, Zhihang Yuan, and Yan Yan. MIM4DD: mutual information maximization for dataset distillation. In NeurIPS, 2023

    2023

  46. [54]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. InACL, pages 2556–2565. ACL, 2018

    2018

  47. [55]

    Fre- quency domain-based dataset distillation

    DongHyeok Shin, Seungjae Shin, and Il-Chul Moon. Fre- quency domain-based dataset distillation. In NeurIPS, 2023

    2023

  48. [56]

    An introduction to the bootstrap (bradley efron and robert j

    Siva Sivaganesan. An introduction to the bootstrap (bradley efron and robert j. tibshirani). SIAM Rev., 36(4):677–678, 1994

    1994

  49. [57]

    Beyond neural scaling laws: beating power law scaling via data pruning

    Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Gan- guli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In NeurIPS, 2022

    2022

  50. [58]

    On the diversity and realism of distilled dataset: An efficient dataset distilla- tion paradigm

    Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distilla- tion paradigm. In CVPR, 2024

    2024

  51. [59]

    Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In NeurIPS, 2020

    2020

  52. [60]

    Training data-efficient image transformers & distillation through atten- tion

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through atten- tion. In ICML, pages 10347–10357. PMLR, 2021

    2021

  53. [61]

    Represen- tation learning with contrastive predictive coding

    Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Represen- tation learning with contrastive predictive coding. 2018

    2018

  54. [62]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkor- eit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998– 6008, 2017. 10

    2017

  55. [63]

    Too large; data reduction for vision-language pre-training

    Alex Jinpeng Wang, Kevin Qinghong Lin, David Junhao Zhang, Stan Weixian Lei, and Mike Zheng Shou. Too large; data reduction for vision-language pre-training. In ICCV, pages 3124–3134. IEEE, 2023

    2023

  56. [64]

    Finetuned multimodal lan- guage models are high-quality image-text data filters

    Weizhi Wang, Khalil Mrini, Linjie Yang, Sateesh Kumar, Yu Tian, Xifeng Yan, and Heng Wang. Finetuned multimodal lan- guage models are high-quality image-text data filters. CoRR, abs/2403.02677, 2024

    2024 arXiv

  57. [65]

    Cliploss and norm-based data selection methods for multimodal con- trastive learning

    Yiping Wang, Yifang Chen, Wendan Yan, Alex Fang, Wen- jing Zhou, Kevin Jamieson, and Simon Shaolei Du. Cliploss and norm-based data selection methods for multimodal con- trastive learning. CoRR, abs/2405.19547, 2024

    2024 arXiv

  58. [66]

    On the de-duplication of LAION-2B

    Ryan Webster, Julien Rabin, Loïc Simon, and Frédéric Jurie. On the de-duplication of LAION-2B. CoRR, 2023

    2023

  59. [67]

    Cit: Curation in training for effective vision- language data

    Hu Xu, Saining Xie, Po-Yao Huang, Licheng Yu, Russell Howes, Gargi Ghosh, Luke Zettlemoyer, and Christoph Fe- ichtenhofer. Cit: Curation in training for effective vision- language data. In ICCV, pages 15134–15143. IEEE, 2023

    2023

  60. [68]

    Demystifying CLIP data

    Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Rus- sell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying CLIP data. 2024

    2024

  61. [69]

    Dataset pruning: Reducing training data by examining generalization influence

    Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reducing training data by examining generalization influence. In ICLR, 2023

    2023

  62. [70]

    Coca: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mo- jtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. TMLPR, 2022

    2022

  63. [71]

    MEST: accurate and fast memory- economic sparse training framework on the edge

    Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, Siyue Wang, Minghai Qin, Bin Ren, Yanzhi Wang, Sijia Liu, and Xue Lin. MEST: accurate and fast memory- economic sparse training framework on the edge. In NeurI...

    2021

  64. [72]

    Dynamic sparse no training: Training-free fine-tuning for sparse llms

    Yuxin Zhang, Lirui Zhao, Mingbao Lin, Yunyun Sun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. Dynamic sparse no training: Training-free fine-tuning for sparse llms. In ICLR, 2024

    2024

  65. [73]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 130(9):2337–2348, 2022. 11

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.