Pith. sign in

REVIEW 3 major objections 6 minor 94 references

COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Cropped captions plus cross-modal self-distillation beat much larger vision-language pretraining runs.

desk verdict A strong empirical recipe for cross-modal self-distillation, but the main-table CLIP/SigLIP baselines may not have seen the same long captions, and the ablation shows that matters a lot. read the letter →

arxiv 2412.01814 v2 pith:6KS2H34V submitted 2024-12-02 cs.CV cs.AIcs.CLcs.LG

Sanghwan Kim , Rui Xiao , Mariana-Iuliana Georgescu , Stephan Alaniz , Zeynep Akata This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.LG
keywords vision-languagepretrainingcontrastivelearningself-distillationtextcroppingcross-attentionzero-shotretrievalsemanticsegmentationcompositionalunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Contrastive vision-language models align whole images with whole captions, and this global objective tends to make them lock onto the largest foreground object while ignoring small objects, background details, and word order. This paper argues that the fix is to pretrain with both global and local views of images and of text: a teacher network sees global views, a student network sees cropped views, and the student must reproduce the teacher's richer global representation. The paper's central claim is that when this self-distillation signal passes through a cross-attention module that pools image tokens with text tokens and vice versa, both the image and text encoders learn finer-grained, more composable representations. If true, the payoff is data efficiency: models trained on tens of millions of pairs can match or beat models trained on billions on retrieval, segmentation, and compositional understanding.

What carries the argument

The load-bearing mechanism is a student-teacher pretraining loop with three interacting pieces. First, multi-crop augmentation is applied to both modalities: two global image views plus six local image views, and text views formed by randomly sampling one to five sentences (global) or a single sentence (local) from long synthetic captions. Second, a cross-attention module makes each modality attend to the other, using the image's [cls] token as a query over text tokens and the text's [eot] token as a query over image tokens, then adding the attended result back to the query token. Third, a cross-modality self-distillation loss matches these student cross-modal embeddings to the teacher's global [cls] and [eot] tokens under a symmetric contrastive loss. Because the teacher is a moving average of the student and only sees global views, the student is trained to reconstruct rich global context from local inputs, and the gradient flows into both encoders at once.

What would settle it

A direct test is to keep all of COSMOS's machinery but replace the local text crops with sentences sampled from other images' captions; if retrieval and segmentation gains persist, the local-to-global text correspondence is not the source of the improvement, and if they collapse, the assumption that captions decompose regionally is load-bearing.

Watch

Extended reading notes

Core claim

On its own terms, COSMOS establishes that randomly cropping long synthetic captions into multi-sentence global views and single-sentence local views, and doing the same for images, creates a workable self-distillation task for vision-language models. The student processes all crops and uses a cross-attention module to produce image and text embeddings that are each conditioned on the other modality; these are matched, through a symmetric contrastive loss, to the teacher's global image and text tokens. The teacher is an exponential moving average of the student, so the supervision is fully self-generated. The reported consequence is consistent gains across retrieval, classification, segmentation, and compositional benchmarks: for instance, on the COCO retrieval benchmark the merged-30M model reaches 68.0% image-to-text R@1, against 62.3% for the strongest previous long-caption method and 59.3% for a model trained on two billion pairs.

Load-bearing premise

The method assumes that a randomly chosen sentence from a long caption and a randomly chosen crop of the paired image describe overlapping content, so that pulling them together in embedding space teaches genuine fine-grained alignment rather than gluing unrelated details together.

Editorial extensions

If this is right

  • Retrieval: with 30 million pretraining pairs, the model reaches 68.0% image-to-text R@1 on COCO, exceeding a 2-billion-pair baseline at 59.3%.
  • Segmentation: the 30M model averages 20.0% mIoU across six zero-shot segmentation benchmarks, above the 16.5% of a 1-billion-pair baseline.
  • Compositional understanding: on SugarCrepe it scores 86.6% average, and on MMVP-VLM it matches a 1-billion-pair model at 25.9%.
  • The cross-attention module is used only during training; at inference the model is evaluated as a standard contrastive model using [cls] and [eot] tokens.
  • The two losses are added at the same scale, so the method needs no grid search over loss weights, a tuning burden that prior self-distillation methods carry.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the method's dependence on sentence-level long captions means its advantage over plain contrastive training should shrink as captions become shorter or less decomposable; an experiment training on single-sentence captions without any cropping would isolate this factor.
  • Editorial inference: because image and text crops are sampled independently, the objective sometimes asks local text about the background to align with a local image crop of the foreground; the method's robustness to this mismatch suggests it learns conceptual co-occurrence rather than strict region-level grounding, a property worth probing directly by measuring alignment when local crops are know
  • Editorial inference: the cross-attention module can be read as a learned pooling operation; replacing its query tokens with averaged tokens or adding a second layer would test whether the gains come from the attention pooling itself or from the extra capacity it adds to the student.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes COSMOS, a vision-language pre-training method that combines text cropping, a cross-attention module, and cross-modality self-distillation. Global and local views of both images and long synthetic captions are used to train a student model against an EMA teacher, with a cross-attention module generating cross-modal embeddings for the student. The authors evaluate on zero-shot retrieval, classification, semantic segmentation, and compositionality benchmarks, reporting gains over CLIP, SigLIP, and DreamLIP on 3M-30M training sets, and in several cases exceeding models trained on hundreds of millions or billions of pairs.

Significance. If the reported results hold, COSMOS is a meaningful advance because it demonstrates that cross-modal self-distillation with text augmentation can substantially improve CLIP-style representations on fine-grained and compositional tasks, with consistent gains over the direct DreamLIP baseline and even over models trained on much larger data. The paper's strengths include a clear component ablation (Table 5), reimplementation of previous SSL baselines on the same long-caption data (Supp. E.1, Table 9), additional experiments on PixelProse (Supp. E.3), and a public code release, all of which support the reproducibility of the method.

major comments (3)
  1. [Sec. 4, Tables 1-2, and Supp. D] The headline claim that COSMOS 'consistently outperforms previous strong baselines' is confounded unless the CLIP/SigLIP rows in Tables 1 and 2 were trained with the same DreamLIP long synthetic captions used for COSMOS. The table footnotes say 'Results are reproduced with our setup for fair comparison,' but Sec. C.1 and Supp. D describe only batch sizes and OpenCLIP model names, not whether the reproduced CLIP/SigLIP models were trained on original alt-text or on the long captions. This matters because the component ablation in Table 5 shows that switching from no text augmentation to long synthetic captions (row 4 vs row 5) improves MSCOCO image-to-text R@1 from 20.6 to 50.4 on CC3M, and ImageNet top-1 from 22.8 to 34.4. If the CLIP/SigLIP baselines in Tables 1 and 2 use original captions, a large part of the reported gap (e.g., CC3M MSCOCO I2T R@1: CLIP 40.2 vs COSMOS 53.1) is due to caption enrichment rather than the proposed cross-modality self-distillation. The authors must disclose the exact caption inputs for each reproduced baseline and, if necessary, retrain the CLIP/SigLIP baselines with the same long captions to make the comparison fair.
  2. [Sec. 4.2, Table 2 (Merged-30M rows)] The abstract and introduction state that COSMOS 'consistently outperforms previous strong baselines,' but the per-dataset classification results in Table 2 do not support 'consistently' in the sense of every dataset. On Merged-30M, DreamLIP outperforms COSMOS on Food-101 (76.4 vs 73.9), FGVCAircraft (8.3 vs 5.9), Oxford Pets (66.3 vs 62.9), Flowers102 (59.1 vs 54.6), and ImageNet (58.4 vs 57.6). COSMOS has a higher average over the 11 datasets (58.6 vs 58.4), but the difference is only 0.2 points. The claim should be qualified to say COSMOS outperforms on average or on most datasets, rather than 'consistently outperforms.'
  3. [Tables 1-4] All experimental results are single-run point estimates without any measure of variance, number of seeds, or significance testing. Some of the claimed advantages are small (e.g., 57.6 vs 58.4 ImageNet, 58.6 vs 58.4 average in Table 2; 25.9 vs 25.9 MMVP in Table 4), and these differences could easily be within run-to-run noise for large-scale pre-training. The authors should either provide error bars or multiple-seed results for at least the key tables, or explicitly state that the reported numbers are single runs and avoid overinterpreting small gaps.
minor comments (6)
  1. [Sec. 3.2, Eqs. (1)-(2)] The subscripts on the cross-attention modules appear swapped relative to the textual description: Eq. (1) uses C_T^theta to produce the image embedding h_I by attending to text tokens, while Eq. (2) uses C_I^theta to produce the text embedding h_T by attending to image tokens. The definitions in the text make the behavior clear, but the notation C_T for an image-side module and C_I for a text-side module is confusing; consider renaming to C_{I<-T} and C_{T<-I} or adding a clarifying sentence.
  2. [Sec. 4.5, Table 5] The row 1 'CLIP' baseline in Table 5 is explicitly trained without any augmentation (as stated in Sec. 4.5), which is a much weaker setup than the CLIP baseline reported in Table 1 (e.g., MSCOCO I2T R@1: 15.0 in Table 5 vs 40.2 in Table 1 on CC3M). The large jump from row 1 to row 5 should not be interpreted as the effect of text augmentation relative to the standard CLIP baseline; this discrepancy should be clarified in the text.
  3. [Sec. 4.4] There is a typo: 'we presents results' should be 'we present results.'
  4. [Supp. E.4, Table 14] The phrase 'four 4-GPU machines' is ambiguous. Please specify whether the training used four machines each with four GPUs (16 GPUs total) or a single machine with four GPUs.
  5. [Sec. 4.3, Table 3] The SILC-C baseline is marked with an asterisk indicating results are copied from its paper, but SILC-C may use a different vision encoder (e.g., ViT-L) than the ViT-B/16 used for COSMOS. Please confirm the architecture of SILC-C or add a note about the architecture mismatch.
  6. [Sec. 3.3, Eq. (6)] The COSMOS loss in Eq. (6) applies symmetric InfoNCE between h_I/h_T and both CLS_t and EOT_t, which means the student cross-modal embeddings are pulled toward both teacher class tokens and teacher text tokens. The formulation is valid, but a brief intuition for why both teacher tokens are needed (rather than using only the corresponding modality tokens) would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical comparison validated on external zero-shot benchmarks, and the self-referential EMA teacher is standard self-distillation rather than a construction that forces the claimed results.

full rationale

COSMOS does not purport to derive a first-principles result; its claims are empirical and are evaluated on held-out external benchmarks (Flickr30K, MSCOCO, ImageNet, segmentation and compositionality suites) with fixed evaluation protocols. The teacher model is an exponential moving average of the student, which is self-referential by design, but this is the standard self-distillation mechanism of DINO/BYOL and does not reduce the downstream benchmark numbers to the training objective by construction: the teacher's outputs are not the benchmark labels, and the reported zero-shot metrics are computed independently after training. No fitted parameter is renamed as a prediction; the hyperparameters (momentum, batch size, number of crops) are disclosed as empirical choices, and the component ablation in Table 5 explicitly separates the effect of long synthetic captions from the effect of cross-attention self-distillation rather than attributing the entire gain to the proposed module. The long captions come from external works (DreamLIP, PixelProse) and are not outputs of this paper, and the paper contains no load-bearing self-citation or imported uniqueness theorem. The concern that reproduced CLIP/SigLIP baselines may not have received the same long captions is a comparison-validity or confound issue, not a circularity under the enumerated patterns; it does not make the central claim equivalent to its inputs by definition.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The central empirical gains rest on several chosen hyperparameters (crop counts, text crop lengths, EMA momentum, batch size) and on the availability of MLLM-generated long captions. The method introduces one new architectural component, the cross-attention module, whose value is shown by internal ablation. No new physical entities or external theoretical postulates are introduced.

free parameters (6)
  • EMA momentum lambda = 0.999 for CC3M, 0.99 for other datasets
    Controls teacher update speed; chosen per dataset without a grid search (Supp. C.1).
  • Number of global and local crops = 2 global, 6 local per modality
    Adopted from DINO; ablation Table 14 shows the local crop count changes results.
  • Global text crop length = 1-5 randomly sampled sentences
    Defines the text global view; an empirical design choice without theoretical justification (Sec. 3.1).
  • Local text crop length = 1 sentence
    Defines the text local view; chosen ad hoc.
  • Image crop scale ranges = global 0.4-1.0, local 0.05-0.4
    Copied from DINO; these scales determine which image regions count as local or global.
  • Batch size = 1,024 for CC3M, 4,096 for other datasets
    Selected due to compute constraints rather than optimization; differs from DreamLIP and from the batch sizes used for reproduced baselines, which may affect contrastive alignment.
assumptions (5)
  • standard math InfoNCE contrastive loss is a valid training objective for aligning image and text embeddings.
    Standard CLIP objective, accepted from prior literature (Sec. 3.3).
  • domain assumption DreamLIP long synthetic captions decompose into sentence-level descriptions that correspond to image regions.
    The text cropping strategy depends on the assumption that short caption sentences are meaningful local views of the image (Sec. 3.1, Supp. B).
  • domain assumption Independent random cropping of images and texts preserves enough semantic correspondence for useful distillation.
    Global and local crops are sampled independently, so a local text sentence can describe content absent from a local image crop; the method assumes the model learns useful conceptual correspondences despite this (Supp. E.7).
  • domain assumption An EMA teacher provides stable targets that do not collapse without additional negative-free stabilization.
    The method relies on standard self-distillation behavior from BYOL/DINO, with the CLIP contrastive loss also providing alignment pressure (Sec. 3.2).
  • ad hoc to paper The cross-attention construction in Eqs. 1-2 produces embeddings whose distillation improves both encoders.
    This architectural claim is supported only by the internal ablation (Table 5, rows 6-7), not by an external theoretical or empirical benchmark.
invented entities (1)
  • Cross-modality cross-attention module C_theta = {C_I, C_T}
    purpose: Generates cross-modal embeddings h_I and h_T by conditioning image tokens on text and text tokens on image, used as distillation outputs.
    A new architectural component introduced by this paper. Its benefit is evidenced only by within-paper ablation, not by independent external validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training." pith.science (2026). https://pith.science/paper/6KS2H34V

@misc{pith2026241201814,
  author       = {Pith},
  title        = {Pith review of: COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6KS2H34V}},
  note         = {Machine review of arXiv:2412.01814}
}
read the original abstract

Vision-Language Models (VLMs) trained with contrastive loss have achieved significant advancements in various vision and language tasks. However, the global nature of the contrastive loss makes VLMs focus predominantly on foreground objects, neglecting other crucial information in the image, which limits their effectiveness in downstream tasks. To address these challenges, we propose COSMOS: CrOSs-MOdality Self-distillation for vision-language pre-training that integrates a novel text-cropping strategy and cross-attention module into a self-supervised learning framework. We create global and local views of images and texts (i.e., multi-modal augmentations), which are essential for self-distillation in VLMs. We further introduce a cross-attention module, enabling COSMOS to learn comprehensive cross-modal representations optimized via a cross-modality self-distillation loss. COSMOS consistently outperforms previous strong baselines on various zero-shot downstream tasks, including retrieval, classification, and semantic segmentation. Additionally, it surpasses CLIP-based models trained on larger datasets in visual perception and contextual understanding tasks. Code is available at https://github.com/ExplainableML/cosmos.

Figures

Figures reproduced from arXiv: 2412.01814 by the authors.

Figure 1
Figure 1. We randomly crop the image and randomly select cap [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An overview of COSMOS. Left: Our VLM pre-training mechanism is based on the student-teacher framework with contrastive loss (LCLIP) for multi-modal alignment, and cross-modality self-distillation loss (LCOSMOS) for fine-grained representation learning. Right: The architecture of the student and teacher model with cross-attention modules that extract cross-modality information from the student. 3. COSMOS Model We des… view at source ↗
Figure 3
Figure 3. Visualization of attention map in cross-attention mod [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Visualization of Attention Map. For different set of captions, we visualize the attention weights of the image and text cross￾attention modules. The patch-wise (image) and token-wise (caption) attention weights are both normalized between 0 and 1. In this supplementary…
Figure 5
Figure 5. Figure 5: Illustration of CLIP with self-supervised approaches. I and T denote the image and text encoders, respectively. It (or Tt) and Is (or Ts) represent the teacher and student image (or text) encoders, where the teacher is an exponential moving average (EMA) of the student…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

94 extracted references · 60 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning.NeurIPS, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.NeurIPS, 2022. 2

    2022

  2. [2]

    Single-stage semantic segmentation from image labels

    Nikita Araslanov and Stefan Roth. Single-stage semantic segmentation from image labels. In CVPR, 2020. 3

    2020

  3. [3]

    Beit: Bert pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR, 2021. 2

    2021

  4. [4]

    Food-101–mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. In ECCV, 2014. 6

    2014

  5. [5]

    Coco- stuff: Thing and stuff classes in context

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco- stuff: Thing and stuff classes in context. In CVPR, 2018. 7

    2018

  6. [6]

    Unsupervised learn- ing of visual features by contrasting cluster assignments

    Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Pi- otr Bojanowski, and Armand Joulin. Unsupervised learn- ing of visual features by contrasting cluster assignments. NeurIPS, 2020. 3

    2020

  7. [7]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In ICCV, 2021. 2, 3, 5, 8

    2021

  8. [8]

    Learn- ing to generate text-grounded mask for open-world semantic segmentation from only image-text pairs

    Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learn- ing to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In CVPR, 2023. 3

    2023

Show all 94 references
  1. [9]

    Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts. In CVPR,

  2. [10]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. CoRR, 2023. 2, 4

    2023

  3. [11]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 2, 8

    2020

  4. [12]

    Intriguing properties of contrastive losses

    Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. NeurIPS, 2021. 1, 2

    2021

  5. [13]

    Reproducible scal- ing laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuh- mann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scal- ing laws for contrastive language-image learning. In CVPR,

  6. [14]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014. 6

    2014

  7. [15]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR,

  8. [16]

    Democratizing contrastive language-image pre- training: A clip benchmark of data, model, and supervision

    Yufeng Cui, Lichen Zhao, Feng Liang, Yangguang Li, and Jing Shao. Democratizing contrastive language-image pre- training: A clip benchmark of data, model, and supervision. ICML Workshop, 2022. 4, 5, 1

    2022

  9. [17]

    InstructBLIP: Towards general- purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. InstructBLIP: Towards general- purpose vision-language models with instruction tuning. NeurIPS, 2024. 4

    2024

  10. [18]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 6, 8, 4, 7

    2009

  11. [19]

    Redcaps: Web-curated image-text data created by the people, for the people

    Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin John- son. Redcaps: Web-curated image-text data created by the people, for the people. NeurIPS, 2021. 5

    2021

  12. [20]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin. Bert: Pre-training of deep bidirectional trans- formers for language understanding. NAACL, 2018. 2

    2018

  13. [21]

    Maskclip: Masked self- distillation advances contrastive language-image pretraining

    Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, et al. Maskclip: Masked self- distillation advances contrastive language-image pretraining. In CVPR, 2023. 1, 2, 4, 5, 6, 9

    2023

  14. [22]

    The pascal visual object classes challenge: A retrospective

    Mark Everingham, SM Ali Eslami, Luc Van Gool, Christo- pher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. In- ternational journal of computer vision, 2015. 7, 3

    2015

  15. [23]

    Improving clip training with language rewrites

    Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. NeurIPS, 2023. 5, 6, 8

    2023

  16. [24]

    Learning gener- ative visual models from few training examples: An incre- mental bayesian approach tested on 101 object categories

    Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning gener- ative visual models from few training examples: An incre- mental bayesian approach tested on 101 object categories. In CVPR, 2004. 6

    2004

  17. [25]

    Dat- acomp: In search of the next generation of multimodal datasets

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Dat- acomp: In search of the next generation of multimodal datasets. NeurIPS, 2024. 3, 5

    2024

  18. [26]

    Pyramidclip: Hierarchi- cal feature alignment for vision-language model pretraining

    Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, Rongrong Ji, and Chunhua Shen. Pyramidclip: Hierarchi- cal feature alignment for vision-language model pretraining. NeurIPS, 2022. 2, 8

    2022

  19. [27]

    Softclip: Softer cross-modal alignment makes clip stronger

    Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu, Enwei Zhang, Ke Li, Jie Yang, Wei Liu, and Xing Sun. Softclip: Softer cross-modal alignment makes clip stronger. In AAAI, 2024

    2024

  20. [28]

    HiCLIP: Contrastive language-image pre- training with hierarchy-aware attention

    Shijie Geng, Jianbo Yuan, Yu Tian, Yuxiao Chen, and Yongfeng Zhang. HiCLIP: Contrastive language-image pre- training with hierarchy-aware attention. In ICLR, 2023. 2 9

    2023

  21. [29]

    Bootstrap your own latent-a new approach to self-supervised learning

    Jean-Bastien Grill, Florian Strub, Florent Altch ´e, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. NeurIPS, 2020. 2

    2020

  22. [30]

    A survey on self-supervised learning: Algorithms, applications, and future trends

    Jie Gui, Tuo Chen, Jing Zhang, Qiong Cao, Zhenan Sun, Hao Luo, and Dacheng Tao. A survey on self-supervised learning: Algorithms, applications, and future trends. IEEE Transactions on Pattern Analysis and Machine Intelligence,

  23. [31]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 2

    2022

  24. [32]

    Probing image- language transformers for verb understanding

    Lisa Anne Hendricks and Aida Nematzadeh. Probing image- language transformers for verb understanding. InACL, 2021. 2, 7, 1, 3

    2021

  25. [33]

    Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality

    Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kem- bhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. NeurIPS,

  26. [34]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 8

    2019

  27. [35]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021. 2

    2021

  28. [36]

    JUWELS Cluster and Booster: Exascale Pathfinder with Modular Supercomputing Architecture at Juelich Supercomputing Centre

    J ¨ulich Supercomputing Centre. JUWELS Cluster and Booster: Exascale Pathfinder with Modular Supercomputing Architecture at Juelich Supercomputing Centre. Journal of large-scale research facilities, 7(A138), 2021. 9

    2021

  29. [37]

    Expediting contrastive language-image pretraining via self-distilled encoders

    Bumsoo Kim, Jinhyung Kim, Yeonsik Jo, and Seung Hwan Kim. Expediting contrastive language-image pretraining via self-distilled encoders. In AAAI, 2024. 2

    2024

  30. [38]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV Workshop, 2013. 6

    2013

  31. [39]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 6

    2009

  32. [40]

    Veclip: Improving clip training via visual-enriched captions

    Zhengfeng Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiu- long Shan, Chen-Nee Chuah, et al. Veclip: Improving clip training via visual-enriched captions. In ECCV, 2024. 8

    2024

  33. [41]

    Modeling caption diversity in contrastive vision- language pretraining

    Samuel Lavoie, Polina Kirichenko, Mark Ibrahim, Mido As- sran, Andrew Gordon Wilson, Aaron Courville, and Nico- las Ballas. Modeling caption diversity in contrastive vision- language pretraining. In ICML, 2024. 5, 6, 7

    2024

  34. [42]

    Uni- clip: Unified framework for contrastive language-image pre- training

    Janghyeon Lee, Jongsuk Kim, Hyounguk Shon, Bumsoo Kim, Seung Hwan Kim, Honglak Lee, and Junmo Kim. Uni- clip: Unified framework for contrastive language-image pre- training. NeurIPS, 2022. 2

    2022

  35. [43]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML,

  36. [44]

    Addressing feature suppression in unsupervised visual representations

    Tianhong Li, Lijie Fan, Yuan Yuan, Hao He, Yonglong Tian, Rogerio Feris, Piotr Indyk, and Dina Katabi. Addressing feature suppression in unsupervised visual representations. In WCCV, 2023. 1, 2

    2023

  37. [45]

    Su- pervision exists everywhere: A data efficient contrastive language-image pre-training paradigm

    Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Su- pervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. In ICLR, 2021. 1, 2, 4, 8, 9

    2021

  38. [46]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In EMNLP, 2023. 7

    2023

  39. [47]

    Scaling language-image pre-training via masking

    Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichten- hofer, and Kaiming He. Scaling language-image pre-training via masking. In CVPR, 2023. 2

    2023

  40. [48]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 5, 8, 2, 4, 6, 7

    2014

  41. [49]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR,

  42. [50]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 2024. 2, 8

    2024

  43. [51]

    Mllms- augmented visual-language representation learning

    Yanqing Liu, Kai Wang, Wenqi Shao, Ping Luo, Yu Qiao, Mike Zheng Shou, Kaipeng Zhang, and Yang You. Mllms- augmented visual-language representation learning. arXiv preprint arXiv:2311.18765, 2023. 5, 6, 8

    2023 arXiv

  44. [52]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 2

    2017 arXiv

  45. [53]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurIPS,

  46. [54]

    Fine-grained visual classi- fication of aircraft

    Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classi- fication of aircraft. arXiv:1306.5151, 2013. 6

    2013 arXiv

  47. [55]

    The role of context for object detection and se- mantic segmentation in the wild

    Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and se- mantic segmentation in the wild. In CVPR, 2014. 7, 3

    2014

  48. [56]

    Slip: Self-supervision meets language-image pre- training

    Norman Mu, Alexander Kirillov, David Wagner, and Sain- ing Xie. Slip: Self-supervision meets language-image pre- training. In ECCV, 2022. 1, 2, 5, 6, 8, 3, 4, 9

    2022

  49. [57]

    Silc: Improving vision language pretraining with self-distillation

    Muhammad Ferjad Naeem, Yongqin Xian, Xiaohua Zhai, Lukas Hoyer, Luc Van Gool, and Federico Tombari. Silc: Improving vision language pretraining with self-distillation. In ECCV, 2024. 1, 2, 4, 7, 8, 3, 9

    2024

  50. [58]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Sixth Indian Conference on Computer Vision, Graphics & Image Processing, 2008. 6

    2008

  51. [59]

    Docci: De- scriptions of connected and contrasting images

    Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, et al. Docci: De- scriptions of connected and contrasting images. ECCV,

  52. [60]

    Repre- sentation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 1, 4, 3, 9

    2018 arXiv

  53. [61]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In ICCV, 2012. 6

    2012

  54. [62]

    Language models are unsu- pervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners. OpenAI blog, 2019. 2

    2019

  55. [63]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 1, 2, 4, 5, 6, 7, 8, 9

    2021

  56. [64]

    Can contrastive learning avoid shortcut solutions? NeurIPS, 2021

    Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? NeurIPS, 2021. 1, 2

    2021

  57. [65]

    Building vision-language models on solid foundations with masked distillation

    Sepehr Sameni, Kushal Kafle, Hao Tan, and Simon Jenni. Building vision-language models on solid foundations with masked distillation. In CVPR, 2024. 1, 2, 4, 5, 6, 9

    2024

  58. [66]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 3

    2021 arXiv

  59. [67]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 2022. 3

    2022

  60. [68]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In ACL,

  61. [69]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR,

  62. [70]

    From pixels to prose: A large dataset of dense image cap- tions

    Vasu Singla, Kaiyu Yue, Sukriti Paul, Reza Shirkavand, Mayuka Jayawardhana, Alireza Ganjdanesh, Heng Huang, Abhinav Bhatele, Gowthami Somepalli, and Tom Goldstein. From pixels to prose: A large dataset of dense image cap- tions. arXiv preprint arXiv:2406.10328, 2024. 2, 3, 1, 5

    2024 arXiv

  63. [71]

    Feature dropout: Revisiting the role of augmentations in contrastive learning

    Alex Tamkin, Margalit Glasgow, Xiluo He, and Noah Good- man. Feature dropout: Revisiting the role of augmentations in contrastive learning. NeurIPS, 2024. 2

    2024

  64. [72]

    Yfcc100m: The new data in multimedia research

    Bart Thomee, David A Shamma, Gerald Friedland, Ben- jamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 2016. 4

    2016

  65. [73]

    Winoground: Probing vision and language models for visio- linguistic compositionality

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio- linguistic compositionality. In CVPR, 2022. 1, 6, 7

    2022

  66. [74]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In CVPR, 2024. 1, 2, 7, 8, 3

    2024

  67. [75]

    A picture is worth more than 77 text tokens: Evaluating clip- style models on dense captions

    Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero-Soriano. A picture is worth more than 77 text tokens: Evaluating clip- style models on dense captions. In CVPR, 2024. 2

    2024

  68. [76]

    Mobile- clip: Fast image-text models through multi-modal reinforced training

    Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, and Oncel Tuzel. Mobile- clip: Fast image-text models through multi-modal reinforced training. In CVPR, 2024. 8

    2024

  69. [77]

    Sclip: Rethinking self-attention for dense vision-language inference

    Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision-language inference. ECCV,

  70. [78]

    Lotlip: Improving language-image pre- training for long text understanding

    Wei Wu, Kecheng Zheng, Shuailei Ma, Fan Lu, Yuxin Guo, Yifei Zhang, Wei Chen, Qingpei Guo, Yujun Shen, and Zheng-Jun Zha. Lotlip: Improving language-image pre- training for long text understanding. In NeurIPS, 2024. 2

    2024

  71. [79]

    Sun database: Large-scale scene recognition from abbey to zoo

    Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In ICCV, 2010. 6

    2010

  72. [80]

    Demystifying clip data

    Hu Xu, Saining Xie, Xiaoqing Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. In ICLR, 2023. 5, 6, 7

    2023

  73. [81]

    Which features are learnt by con- trastive learning? on the role of simplicity bias in class col- lapse and feature suppression

    Yihao Xue, Siddharth Joshi, Eric Gan, Pin-Yu Chen, and Baharan Mirzasoleiman. Which features are learnt by con- trastive learning? on the role of simplicity bias in class col- lapse and feature suppression. In ICML, 2023. 2

    2023

  74. [82]

    Alip: Adaptive language-image pre-training with synthetic cap- tion

    Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li, Ziy- ong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Alip: Adaptive language-image pre-training with synthetic cap- tion. In ICCV, 2023. 2

    2023

  75. [83]

    Filip: Fine-grained interactive language-image pre-training

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. In ICLR, 2021. 2

    2021

  76. [84]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hocken- maier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions. ACL, 2014. 5, 8, 6, 7

    2014

  77. [85]

    Coca: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mo- jtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. TMLR, 2022. 8

    2022

  78. [86]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. In CVPR, 2024. 8

    2024

  79. [87]

    When and why vision- language models behave like bags-of-words, and what to do about it? ICLR, 2023

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- language models behave like bags-of-words, and what to do about it? ICLR, 2023. 1

    2023

  80. [88]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023. 5, 6, 7, 2, 4, 8

    2023

  81. [89]

    Long-clip: Unlocking the long-text capability of clip

    Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. In ECCV, 2024. 2 11

    2024

  82. [90]

    Learning the unlearned: Mitigating feature suppression in contrastive learning

    Jihai Zhang, Xiang Lan, Xiaoye Qu, Yu Cheng, Mengling Feng, and Bryan Hooi. Learning the unlearned: Mitigating feature suppression in contrastive learning. CoRR, 2024. 1, 2

    2024

  83. [91]

    Dreamlip: Language- image pre-training with long captions

    Kecheng Zheng, Yifei Zhang, Wei Wu, Fan Lu, Shuailei Ma, Xin Jin, Wei Chen, and Yujun Shen. Dreamlip: Language- image pre-training with long captions. In ECCV, 2024. 2, 3, 4, 5, 6, 7, 8

    2024

  84. [92]

    Semantic under- standing of scenes through the ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi- dler, Adela Barriuso, and Antonio Torralba. Semantic under- standing of scenes through the ade20k dataset. International Journal of Computer Vision, 2019. 7

    2019

  85. [93]

    Extract free dense labels from clip

    Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In ECCV, 2022. 7

    2022

  86. [94]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In ICLR, 2023. 2 12 COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training Supplementary Material 1...

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.