Pith. sign in

REVIEW 3 major objections 4 minor 64 references

Differential-informed Sample Selection Accelerates Multimodal Contrastive Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Selecting image-text pairs by the gap between historical and current similarity scores lets CLIP-style training match full-data performance with 70% fewer samples.

desk verdict Useful oracle-free data selection for multimodal pretraining with a real empirical payoff, but the causal story needs one more experiment to close the loop. read the letter →

arxiv 2507.12998 v1 pith:MZ2TWWEI submitted 2025-07-17 cs.CV cs.LG

Zihua Zhao , Feng Hong , Mengxi Chen , Pengyi Chen , Benyuan Liu , Jiangchao Yao , Ya Zhang , Yanfeng Wang This is my paper
classification cs.CVcs.LG
keywords sampleselectionmultimodalcontrastivelearningnoisycorrespondenceCLIPScoreonlinedatatrainingaccelerationmemorizationeffectvision-languagepretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that sample selection for CLIP-style contrastive learning can be accelerated without any pretrained oracle model by scoring each image-text pair with the difference between the model's historical and current predicted similarity, the differential-informed score. The key empirical observation is that clean pairs' self-predicted similarity rises early and then eases, while noisy pairs start low and climb later as the model memorizes them, so the historical-minus-current gap separates the two even when the raw scores overlap. Selecting the top fraction of each batch by this gap, the proposed DISSect method matches full-data zero-shot retrieval performance on CC3M at a 30% selection ratio and surpasses full-data training on the noisier YFCC15M dataset at 50% selection. The paper backs the heuristic with a gradient-based argument that links the differential to the memorization effect in noisy-label learning.

What carries the argument

The central object is the sample-wise differential $\delta = \mathrm{CLIPScore}_{\mathrm{hist}} - \mathrm{CLIPScore}_{\mathrm{curr}}$, where CLIPScore is a scaled positive cross-modal similarity between image and text embeddings. The historical score is obtained either from a model checkpoint after a warm-up phase or from a momentum-updated temporal ensemble, so no oracle model is needed. The differential acts as a per-sample learning-tendency indicator: clean pairs show near-zero or slightly negative $\delta$ after the early learning point, while noisy pairs show large positive $\delta$ as they are memorized. Selecting the top-$kr$ fraction of each batch by $\delta$ is the selection mechanism that carries the acceleration.

What would settle it

Measure $\delta$ on a dataset with known pair labels after the early-learning point; if the $\delta$ distributions of clean and noisy pairs overlap substantially rather than showing a growing gap, DISSect's ranking would select the wrong samples and its advantage should disappear. A direct test would inject synthetic correspondence noise with controlled timing (e.g., noise that is learned as fast as clean pairs) and check whether DISSect still beats random selection.

Watch

Extended reading notes

Core claim

The central discovery is that the temporal trajectory of a pair's CLIPScore under CLIP-style training encodes its quality: clean correspondences reach high similarity early and then dip slightly, whereas noisy correspondences begin low and rise as the model memorizes them, so their distributions eventually overlap. Computing the differential $\delta = \mathrm{CLIPScore}_{\mathrm{hist}} - \mathrm{CLIPScore}_{\mathrm{curr}}$ between a historical model state (obtained by a short warm-up or by temporal ensembling) and the current state turns this trajectory into a per-sample separation score. Ranking each batch by $\delta$ and training only on the largest-$\delta$ samples removes noisy correspondence without a reference model, achieving zero-shot retrieval performance comparable to full-data training at 30% selection and a 2.85$\times$ speedup to reach full-data performance on CC3M.

Load-bearing premise

The method assumes that the temporal trajectory pattern observed on CC3M generalizes: across datasets, noise types, and architectures, clean pairs' self-predicted similarity peaks early and then eases while noisy pairs start low and rise late, so that the historical-minus-current differential cleanly separates them.

Editorial extensions

If this is right

  • At a 30% selection ratio, DISSect reaches zero-shot retrieval performance on MS-COCO and Flickr30K comparable to full-data training on CC3M, meaning roughly 70% of iterations can be skipped without an oracle.
  • On the noisier YFCC15M dataset at 50% selection, DISSect outperforms full-data training, suggesting that removing noisy correspondence improves quality, not just efficiency.
  • DISSect works with different backbones: when applied only to the image-text alignment module, it improves both CLIP/ResNet101 and BLIP/ViT-B over random sampling and SCAN.
  • A single-network DISSect beats dual-network noisy-correspondence methods NCR and GSC in downstream retrieval while avoiding their extra compute cost.
  • The temporal-ensembling version requires no warm-up epoch and is insensitive to the momentum hyperparameter, making the method plug-and-play.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same historical-minus-current differential could be applied to other self-supervised losses that exhibit memorization, such as unimodal contrastive learning or masked autoencoding, where a similar clean-then-memorize trajectory may hold.
  • Because DISSect is oracle-free, it could be composed with offline coreset pruning in a two-stage pipeline: first use DISSect during a short warm-up to filter the corpus, then train on the filtered set.
  • The trajectory assumption could be tested as a diagnostic: datasets or training schedules that do not show the clean-peak-then-ease pattern would be poor candidates for DISSect, potentially guiding when to use selection at all.
  • The reported sensitivity of final performance to selection ratio (best at 60%, stable above 50%) suggests there is an optimal budget that trades diversity against noise removal; finding it adaptively during training is a natural next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes DISSect, an online batch-level sample selection method for multimodal contrastive learning. Instead of relying on an oracle model or a static loss threshold, DISSect selects, within each batch, the samples with the largest differential delta = CLIPScore_hist - CLIPScore_curr between a historical and the current model prediction of image-text similarity. The method is evaluated on CC3M, CC12M, and YFCC15M with CLIP and BLIP backbones across image-text retrieval, NLVR2 reasoning, and COCO captioning, and it is compared with coreset, online, and noisy-correspondence baselines. The central empirical claim is that DISSect reaches performance comparable to full-data training while using only 30% of the samples, and that it outperforms oracle-based competitors on several settings.

Significance. If the central claim holds, DISSect is a practically valuable and simple contribution: it removes noisy image-text correspondence during CLIP-style pretraining without an oracle model, with reported wall-clock savings, and it is evaluated across multiple datasets, backbones, and downstream tasks. The paper also provides a plausible memorization-based explanation, compares against dual-network noisy-correspondence methods, and releases code. The main weakness is that the load-bearing mechanism is not verified under the method's own selective-training loop, and the theoretical link in Eq. (4) is asserted rather than demonstrated. The significance is therefore real but conditional on the mechanism surviving this check.

major comments (3)
  1. [§3.3, §3.4, Algorithm 1] The load-bearing premise of §3.4 is that the differential delta separates clean from noisy samples. The evidence for this (Figs. 3b and 3c) is obtained from vanilla full-data training, with clean/noisy labels supplied by an oracle model, and not from the selective-training loop of Algorithm 1. Under DISSect, the training distribution shifts: samples not selected in a batch do not contribute to the loss, so their CLIPScore_curr can remain near the warm-up value, while repeatedly selected samples may be reinforced. The paper does not report the per-sample delta separation under DISSect itself; Fig. 4(c,d) reports only the oracle-threshold true-positive rate of the selected set, which is an indirect measure. I request a direct diagnostic: track delta for oracle-labelled clean and noisy samples during DISSect training (e.g., ROC-AUC or class separation at several epochs) and compare with vanilla training. Without this evidence, the reported acceleration could plausibly be attributed to the reduced effective batch size or to the warm-up random phase rather than to the differential criterion.
  2. [§3.4, Eq. (4)] Equation (4) is introduced with the sentence "we demonstrate in detail in the Appendix," but the derivation is not present in the main text, and the displayed formula is not connected to Eq. (3) by a formal argument. As written, the right-hand side averages exp(CLIPScore)^{-1} and is not shown to be larger for noisy than for clean samples, nor is it shown that delta equals or bounds this gradient term. Please supply the full derivation, state the assumptions (notably whether the memorization result of [33] transfers from cross-entropy classification to the InfoNCE loss), and prove or state precisely the claimed relationship between delta and the gradient offset. The theory as it stands does not close the gap between the memorization heuristic and the selection rule.
  3. [Tables 1-5] Tables 1, 2, 3, 4, and 5 report single training runs without error bars or significance tests. Several comparisons are close (e.g., Table 1, MS-COCO 30%: DISSect-Warmup IR@10 56.40 versus InfoBatch 54.98; Table 3, NLVR2 dev 73.99 versus SCAN 72.19), and the central claim of "comparable to full data" rests on small margins. Please report at least three seeds with means and standard deviations, or otherwise establish that the reported differences are not within run-to-run noise. This is needed for the empirical claim to be load-bearing.
minor comments (4)
  1. [Table 1] In the Flickr30K (1K Test Set) panel, the Coreset Forgetting row at 30% selection ratio reports TR@10 = 21.30, which is lower than the corresponding TR@1 = 16.46 but implausibly close to the IR@10 value; this appears to be a typographical error and should be corrected.
  2. [Figure 1 caption] The caption refers to a "Bellington" example; this word is unclear and likely a typo, which should be fixed for readability.
  3. [Abstract and Introduction] The phrase "plays an important direction" in the abstract and similar grammatical constructions in the introduction should be revised for clarity.
  4. [Algorithm 1 and §3.4] The pseudo-code places "Predict CLIPScore_hist" inside the warm-up loop, which suggests it is updated only during warm-up; the text should clarify whether the historical score is fixed after warm-up or recomputed, and how this interacts with the temporal-ensembling variant in Eq. (5).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found: the differential selection rule is empirically motivated, not fitted to the target benchmarks, and the central claims are evaluated externally.

full rationale

Walking the claimed derivation chain, every load-bearing step is empirically motivated and then tested against external benchmarks. The selection signal δ = CLIPScore_hist − CLIPScore_curr (Eq. 3) is introduced as a heuristic from the memorization curves in Fig. 3, where clean/noisy labels come from an oracle CLIP model; that partition is used only to motivate the rule, not to set its parameters or to choose the selected samples during training. The training loop (Algorithm 1) computes δ from the under-training model's own predictions, and the reported gains are measured on held-out zero-shot retrieval, NLVR2, and COCO captioning, not on quantities that the selector was fitted to reproduce. The temporal-ensembling variant (Eq. 5) is an implementation detail with reported insensitivity to β, and the theoretical connection to gradient offsets (Eq. 4) is asserted rather than fully derived, which is a rigor concern rather than a circular one. Self-citations (DivBS, GSC, and related prior work) appear as baselines and related work, not as the justification for the selection rule. I therefore find no step in which a prediction reduces by construction to an input, and no load-bearing self-citation chain.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No new particles, forces, or entities are introduced. The main burdens are the assumed CLIPScore trajectory pattern and the unproved transfer of the memorization theorem to contrastive learning; these are assumptions, not fitted parameters. The free parameters are small in number and the authors report sensitivity only for the momentum term.

free parameters (3)
  • selection ratio r = 30%, 50%, 60%, 70% used in experiments; best at 60%
    User-set budget for how many samples per batch are kept. It is an experimental knob, not fitted to test data, but it is a free choice in the algorithm.
  • warm-up epochs Tw = not specified in main text ('a few epochs')
    Controls when the historical CLIPScore is recorded for the warm-up variant. The value is chosen by experiment and is not theoretically determined.
  • momentum beta = not specified; stated to be in (0,1) and insensitive
    Controls the temporal ensembling update of the historical CLIPScore in Eq. 5. It is a hand-set hyperparameter, though the authors report robustness to it.
assumptions (3)
  • domain assumption The memorization effect theorem from ELR [33], derived for noisy labels in classification, transfers to contrastive learning with noisy correspondence.
    Section 3.4 invokes [33] to argue that clean sample gradients vanish early while noisy sample gradients persist. The paper does not prove this transfer for the contrastive InfoNCE loss.
  • ad hoc to paper Clean pairs have CLIPScore trajectories that rise and then fall, while noisy pairs start low and rise late, so the historical-minus-current differential separates them.
    This is the central empirical premise, supported by Figs. 3b and 3c on CC3M with an oracle-based partition. The paper assumes this pattern generalizes to other datasets and models.
  • standard math The InfoNCE contrastive loss with in-batch negatives is a sufficient model of multimodal contrastive pretraining.
    The method and the gradient argument are built on Eq. 1, the standard CLIP training loss, which the paper treats as the training objective throughout.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Differential-informed Sample Selection Accelerates Multimodal Contrastive Learning." pith.science (2026). https://pith.science/paper/MZ2TWWEI

@misc{pith2026250712998,
  author       = {Pith},
  title        = {Pith review of: Differential-informed Sample Selection Accelerates Multimodal Contrastive Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MZ2TWWEI}},
  note         = {Machine review of arXiv:2507.12998}
}
read the original abstract

The remarkable success of contrastive-learning-based multimodal models has been greatly driven by training on ever-larger datasets with expensive compute consumption. Sample selection as an alternative efficient paradigm plays an important direction to accelerate the training process. However, recent advances on sample selection either mostly rely on an oracle model to offline select a high-quality coreset, which is limited in the cold-start scenarios, or focus on online selection based on real-time model predictions, which has not sufficiently or efficiently considered the noisy correspondence. To address this dilemma, we propose a novel Differential-Informed Sample Selection (DISSect) method, which accurately and efficiently discriminates the noisy correspondence for training acceleration. Specifically, we rethink the impact of noisy correspondence on contrastive learning and propose that the differential between the predicted correlation of the current model and that of a historical model is more informative to characterize sample quality. Based on this, we construct a robust differential-based sample selection and analyze its theoretical insights. Extensive experiments on three benchmark datasets and various downstream tasks demonstrate the consistent superiority of DISSect over current state-of-the-art methods. Source code is available at: https://github.com/MediaBrain-SJTU/DISSect.

Figures

Figures reproduced from arXiv: 2507.12998 by the authors.

Figure 1
Figure 1. Learning from samples with noisy correspondence ( [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Performance of various sample selection methods under [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Memorization effect on contrastive learning. (a) CLIPScore distributions at different training stages, with clean and noisy [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Selection strategy analysis on CC3M dataset. (a) CLIPScore distribution of selected data by DISSect at different training [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Ablation and analysis. (a) Comparison on training ef [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

64 extracted references · 37 canonical work pages

  1. [33]

    Early-learning regularization pre- vents memorization of noisy labels

    Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Car- los Fernandez-Granda. Early-learning regularization pre- vents memorization of noisy labels. Advances in neural in- formation processing systems, 33:20331–20342, 2020. 4, 5

    work page 2020

  2. [1]

    Semdedup: Data-efficient learning at web-scale through semantic deduplication

    Amro Abbas, Kushal Tirumala, D ´aniel Simig, Surya Gan- guli, and Ari S Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540, 2023. 3

    arXiv 2023

  3. [2]

    Variance reduction in sgd by distributed importance sampling

    Guillaume Alain, Alex Lamb, Chinnadhurai Sankar, Aaron Courville, and Yoshua Bengio. Variance reduction in sgd by distributed importance sampling. arXiv preprint arXiv:1511.06481, 2015. 8

    arXiv 2015

  4. [3]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425– 2433, 2015. 5

    work page 2015

  5. [4]

    Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558–3568, 2021. 5

    work page 2021

  6. [5]

    En- hanced multimodal representation learning with cross-modal kd

    Mengxi Chen, Linyu Xing, Yu Wang, and Ya Zhang. En- hanced multimodal representation learning with cross-modal kd. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 11766–11775,

    work page

  7. [6]

    Multi-modal medical diagnosis via large-small model collaboration

    Wanyi Chen, Zihua Zhao, Jiangchao Yao, Ya Zhang, Jia- jun Bu, and Haishuai Wang. Multi-modal medical diagnosis via large-small model collaboration. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 30763–30773, 2025. 2

    work page 2025

  8. [7]

    Reproducible scal- ing laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuh- mann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scal- ing laws for contrastive language-image learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818–2829, 2023. 1, 3, 5, 6

    work page 2023

Show all 64 references
  1. [8]

    Selection via proxy: Efficient data se- lection for deep learning

    Cody Coleman, Christopher Yeh, Stephen Mussmann, Baha- ran Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data se- lection for deep learning. arXiv preprint arXiv:1906.11829,

    1906 arXiv

  2. [9]

    Unichest: Conquer-and-divide pre-training for multi-source chest x-ray classification

    Tianjie Dai, Ruipeng Zhang, Feng Hong, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Unichest: Conquer-and-divide pre-training for multi-source chest x-ray classification. IEEE Trans. Medical Imaging, 43(8):2901–2912, 2024. 2

    2024

  3. [10]

    Scan: Boot- strapping contrastive pre-training for data efficiency

    Yangyang Guo and Mohan Kankanhalli. Scan: Boot- strapping contrastive pre-training for data efficiency. arXiv preprint arXiv:2411.09126, 2024. 2, 4, 5, 6

    2024 arXiv

  4. [11]

    Co- teaching: Robust training of deep neural networks with ex- tremely noisy labels

    Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co- teaching: Robust training of deep neural networks with ex- tremely noisy labels. Advances in neural information pro- cessing systems, 31, 2018. 2

    2018

  5. [12]

    Trustworthy machine learning: From data to models

    Bo Han, Jiangchao Yao, Tongliang Liu, Bo Li, Sanmi Koyejo, Feng Liu, et al. Trustworthy machine learning: From data to models. Foundations and Trends® in Privacy and Security, 7(2-3):74–246, 2025. 2

    2025

  6. [13]

    Momentum contrast for unsupervised visual rep- resentation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 9729–9738, 2020. 3

    2020

  7. [14]

    Large- scale dataset pruning with dynamic uncertainty

    Muyang He, Shuo Yang, Tiejun Huang, and Bo Zhao. Large- scale dataset pruning with dynamic uncertainty. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7713–7722, 2024. 1, 2

    2024

  8. [15]

    Clipscore: A reference-free evaluation met- ric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation met- ric for image captioning. arXiv preprint arXiv:2104.08718,

    arXiv

  9. [16]

    Diversified batch selection for training acceleration

    Feng Hong, Yueming Lyu, Jiangchao Yao, Ya Zhang, Ivor W Tsang, and Yanfeng Wang. Diversified batch selection for training acceleration. arXiv preprint arXiv:2406.04872,

    arXiv

  10. [17]

    Learning with noisy correspondence for cross-modal matching.Advances in Neu- ral Information Processing Systems, 34:29406–29419, 2021

    Zhenyu Huang, Guocheng Niu, Xiao Liu, Wenbiao Ding, Xinyan Xiao, Hua Wu, and Xi Peng. Learning with noisy correspondence for cross-modal matching.Advances in Neu- ral Information Processing Systems, 34:29406–29419, 2021. 2, 7

    2021

  11. [18]

    Open- clip, 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Han- naneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Open- clip, 2021. 5

    2021

  12. [19]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR,

  13. [20]

    In-datacenter perfor- mance analysis of a tensor processing unit

    Norman P Jouppi, Cliff Young, Nishant Patil, David Patter- son, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter perfor- mance analysis of a tensor processing unit. In Proceedings of the 44th annual international symposiu...

    2017

  14. [21]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    2001 arXiv

  15. [22]

    Deep visual-semantic align- ments for generating image descriptions

    Andrej Karpathy and Li Fei-Fei. Deep visual-semantic align- ments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 3128–3137, 2015. 5

    2015

  16. [23]

    Vilt: Vision- and-language transformer without convolution or region su- pervision

    Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision- and-language transformer without convolution or region su- pervision. In International conference on machine learning, pages 5583–5594. PMLR, 2021. 1

    2021

  17. [24]

    Temporal ensembling for semi- supervised learning

    Samuli Laine and Timo Aila. Temporal ensembling for semi- supervised learning. arXiv preprint arXiv:1610.02242, 2016. 5

    2016 arXiv

  18. [25]

    Clip benchmark, 2023

    LAION-AI. Clip benchmark, 2023. Accessed: 2025-02-23. 5

    2023

  19. [26]

    Dividemix: Learning with noisy labels as semi-supervised learning

    Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. arXiv preprint arXiv:2002.07394, 2020. 2, 4

    2002 arXiv

  20. [27]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Interna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 1, 3, 5, 7

    2022

  21. [28]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 3

    2023

  22. [29]

    Visual semantic reasoning for image-text matching

    Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu. Visual semantic reasoning for image-text matching. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4654–4662, 2019. 5

    2019

  23. [30]

    Tsang, and Zhenwen Ren

    Xingfeng Li, Yuangang Pan Pan, Yinghui Sun, Quansen Sun Sun, Ivor W. Tsang, and Zhenwen Ren. Fast unpaired multi- view clustering. 2024. 2

    2024

  24. [31]

    Tsang, and Zhenwen Ren

    Xingfeng Li, Yuangang Pan Pan, Yuan Sun, Quansen Sun, Yinghui Sun, Ivor W. Tsang, and Zhenwen Ren. Incomplete multi-view clustering with paired and balanced dynamic an- chor learning. IEEE Transactions on Multimedia , pages 7087–7098, 2025. 2

    2025

  25. [32]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  26. [34]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 5

    2017 arXiv

  27. [35]

    Sieve: Multimodal dataset pruning using image captioning models

    Anas Mahmoud, Mostafa Elhoushi, Amro Abbas, Yu Yang, Newsha Ardalani, Hugh Leather, and Ari S Morcos. Sieve: Multimodal dataset pruning using image captioning models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22423–22432, 2024. 2

    2024

  28. [36]

    Prioritized training on points that are learnable, worth learning, and not yet learnt

    S ¨oren Mindermann, Jan M Brauner, Muhammed T Raz- zak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt H¨oltgen, Aidan N Gomez, Adrien Morisot, Sebastian Far- quhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In Internati...

  29. [37]

    Improving multimodal datasets with image captioning

    Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Se- woong Oh, and Ludwig Schmidt. Improving multimodal datasets with image captioning. Advances in Neural Infor- mation Processing Systems, 36, 2024. 2

    2024

  30. [38]

    Repre- sentation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 1, 3

    2018 arXiv

  31. [39]

    Active learning is a strong baseline for data subset selection

    Dongmin Park, Dimitris Papailiopoulos, and Kangwook Lee. Active learning is a strong baseline for data subset selection. In Has it Trained Yet? NeurIPS 2022 Workshop, 2022. 1, 2

    2022

  32. [40]

    Pytorch: An im- perative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high-performance deep learning library. Ad- vances in neural information processing systems ...

    2019

  33. [41]

    Deep learning on a data diet: Finding important ex- amples early in training

    Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziu- gaite. Deep learning on a data diet: Finding important ex- amples early in training. Advances in neural information processing systems, 34:20596–20607, 2021. 2

    2021

  34. [42]

    Infobatch: Lossless training speed up by unbiased dynamic data pruning

    Ziheng Qin, Kai Wang, Zangwei Zheng, Jianyang Gu, Xi- angyu Peng, Zhaopan Xu, Daquan Zhou, Lei Shang, Baigui Sun, Xuansong Xie, et al. Infobatch: Lossless training speed up by unbiased dynamic data pruning. arXiv preprint arXiv:2303.04947, 2023. 1, 2, 3, 5, 6

    2023 arXiv

  35. [43]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  36. [44]

    Accelerat- ing deep learning with dynamic data pruning

    Ravi S Raju, Kyle Daruwalla, and Mikko Lipasti. Accelerat- ing deep learning with dynamic data pruning. arXiv preprint arXiv:2111.12621, 2021. 1

    2021 arXiv

  37. [45]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 3

    2021 arXiv

  38. [46]

    Active learning for convolu- tional neural networks: A core-set approach

    Ozan Sener and Silvio Savarese. Active learning for convolu- tional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489, 2017. 1

    2017 arXiv

  39. [47]

    Machine learning and deep learning: A review of methods and applications

    Koosha Sharifani and Mahyar Amini. Machine learning and deep learning: A review of methods and applications. World Information Technology and Engineering Journal , 10(07): 3897–3904, 2023. 1

    2023

  40. [48]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2018

  41. [49]

    A corpus for reasoning about natural language grounded in photographs

    Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Hua- jun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXiv:1811.00491, 2018. 5

    2018 arXiv

  42. [50]

    Yfcc100m: The new data in multimedia research

    Bart Thomee, David A Shamma, Gerald Friedland, Ben- jamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64–73, 2016. 5

    2016

  43. [51]

    An empirical study of example forget- ting during deep neural network learning

    Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geof- frey J Gordon. An empirical study of example forget- ting during deep neural network learning. arXiv preprint arXiv:1812.05159, 2018. 2, 5, 6

    2018 arXiv

  44. [52]

    Too large; data reduction for vision-language pre-training

    Alex Jinpeng Wang, Kevin Qinghong Lin, David Junhao Zhang, Stan Weixian Lei, and Mike Zheng Shou. Too large; data reduction for vision-language pre-training. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 3147–3157, 2023. 2

    2023

  45. [53]

    Cliploss and norm-based data selection methods for multimodal con- trastive learning

    Yiping Wang, Yifang Chen, Wendan Yan, Alex Fang, Wen- jing Zhou, Kevin G Jamieson, and Simon S Du. Cliploss and norm-based data selection methods for multimodal con- trastive learning. Advances in Neural Information Process- ing Systems, 37:15028–15069, 2025. 1, 5, 6

    2025

  46. [54]

    Memorization in deep learning: A survey

    Jiaheng Wei, Yanjun Zhang, Leo Yu Zhang, Ming Ding, Chao Chen, Kok-Leong Ong, Jun Zhang, and Yang Xiang. Memorization in deep learning: A survey. arXiv preprint arXiv:2406.03880, 2024. 4

    2024 arXiv

  47. [55]

    Icons: Influence consensus for vision-language data selection

    Xindi Wu, Mengzhou Xia, Rulin Shao, Zhiwei Deng, Pang Wei Koh, and Olga Russakovsky. Icons: Influence consensus for vision-language data selection. arXiv preprint arXiv:2501.00654, 2024. 1, 2

    2024

  48. [56]

    Moderate coreset: A universal method of data selection for real-world data-efficient deep learning

    Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In The Eleventh International Conference on Learning Repre- sentations, 2022. 2

    2022

  49. [57]

    Bicro: Noisy correspon- dence rectification for multi-modality data via bi-directional cross-modal similarity consistency

    Shuo Yang, Zhaopan Xu, Kai Wang, Yang You, Hongxun Yao, Tongliang Liu, and Min Xu. Bicro: Noisy correspon- dence rectification for multi-modality data via bi-directional cross-modal similarity consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter...

    2023

  50. [58]

    Latent class-conditional noise model

    Jiangchao Yao, Bo Han, Zhihan Zhou, Ya Zhang, and Ivor W Tsang. Latent class-conditional noise model. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 45(8): 9964–9980, 2023. 2

    2023

  51. [59]

    On early stopping in gradient descent learning

    Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learning. Constructive ap- proximation, 26(2):289–315, 2007. 4

    2007

  52. [60]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hocken- maier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions. Transactions of the Association for Computational Linguistics, 2:67–78, 2014. 5

    2014

  53. [61]

    Mitigating noisy corre- spondence by geometrical structure consistency learning

    Zihua Zhao, Mengxi Chen, Tianjie Dai, Jiangchao Yao, Bo Han, Ya Zhang, and Yanfeng Wang. Mitigating noisy corre- spondence by geometrical structure consistency learning. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 27381–27390, ...

    2024

  54. [62]

    Coverage-centric coreset selection for high pruning rates

    Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric coreset selection for high pruning rates. arXiv preprint arXiv:2210.15809, 2022. 1, 2

    2022 arXiv

  55. [63]

    Learn- ing to instruct for visual instruction tuning

    Zhihan Zhou, Feng Hong, Jiaan Luo, Jiangchao Yao, Dong- sheng Li, Bo Han, Ya Zhang, and Yanfeng Wang. Learn- ing to instruct for visual instruction tuning. arXiv preprint arXiv:2503.22215, 2025. 2

    2025

  56. [64]

    Uncover the balanced geometry in long-tailed contrastive language-image pretraining

    Zhihan Zhou, Yushi Ye, Feng Hong, Peisen Zhao, Jiangchao Yao, Ya Zhang, Qi Tian, and Yanfeng Wang. Uncover the balanced geometry in long-tailed contrastive language-image pretraining. Mach. Learn., 114(4):106, 2025. 1

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.