Pith. sign in

REVIEW 4 major objections 6 minor 45 references

Impact of Pretraining Word Co-occurrence on Compositional Generalization in Multimodal Models

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that CLIP and CLIP-based multimodal models generalize poorly to concept pairs that rarely co-occur in pretraining captions, even when each concept is common.

desk verdict The paper's core correlation is plausible but likely inflated by a diffusion-generation confound; still worth a serious referee. read the letter →

arxiv 2507.08000 v1 pith:KOICFV5Q submitted 2025-07-10 cs.CV cs.LG

Helen Qu , Sang Michael Xie This is my paper
classification cs.CVcs.LG
keywords compositionalgeneralizationpointwisemutualinformationCLIPmultimodalmodelsconceptco-occurrencezero-shotclassificationvisualquestionansweringpretrainingdatadistribution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that CLIP and the multimodal models built on it do not generalize to new combinations of familiar concepts. It argues that what matters is not just how often each concept appears alone in pretraining captions, but how often the two concepts in an image co-occur, measured by pointwise mutual information (PMI). Across synthetic images, edited natural images, and visual question answering, accuracy tracks PMI; a common object is recognized reliably in a high-PMI pairing and noticeably worse in a low-PMI pairing. If the claim holds, improving compositional generalization requires addressing co-occurrence statistics, not just scaling data or model size.

What carries the argument

The machinery is pointwise mutual information, $\operatorname{pmi}(c_1,c_2) = \log \frac{p(c_1,c_2)}{p(c_1)p(c_2)}$, computed over lemmatized LAION-400M captions after stopword removal and a frequency cutoff. Normalizing by the marginal frequencies separates 'rare because the pair is unusual' from 'rare because one concept is rare', which is the distinction the whole study depends on. To make the metric experimentally accessible, the paper builds GenPairs—200,000 diffusion-generated images prompted from captions that feature one ImageNet class and one accessory concept—and ImageNet-Paste, natural images with a small generated accessory pasted in. This lets the authors vary PMI while holding individual concepts roughly fixed.

What would settle it

Give human annotators the visual concepts actually present in a sample of LAION images, recompute PMI from those annotations, and regress CLIP's GenPairs accuracy on caption-PMI while holding annotation-PMI fixed; if caption-PMI ceases to predict accuracy, the proxy assumption is the real driver.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the co-occurrence rate of concept pairs in CLIP pretraining captions, measured by PMI, strongly predicts zero-shot accuracy. On GenPairs, a synthetic dataset of 200,000 images spanning concept pairs across the PMI range, the PMI of the key concept pair ($c_{\text{accessory}}, c_{\text{ImageNet}}$) correlates with top-1 accuracy at $r = 0.97$, with a 14% accuracy gap between the top and bottom 5% of PMI. Pasting a low-PMI accessory image onto natural ImageNet images reproduces the effect ($r = 0.75$, 10% gap). The same correlation appears in LMMs built on CLIP: LLaVA with a LAION-400M-trained CLIP backbone shows $r = 0.70$ on TextVQA and $r = 0.62$ on open-ended VQAv2, each with a 15% accuracy gap, and high-PMI questions increasingly elicit 'yes' regardless of image content. Model scaling shrinks the gap only mildly, and fine-tuning on edited images removes the correlation only on the edited distribution.

Load-bearing premise

The whole analysis assumes that the words in a caption faithfully indicate which visual concepts appear in the paired image, and that diffusion-generated images are equally faithful for low-PMI and high-PMI concept pairs.

Editorial extensions

If this is right

  • A common object is not recognized because it is common; its accuracy depends on which other concepts accompany it, so evaluations on single-object images can overstate compositional competence.
  • Scaling model size alone is not a cure: the accuracy gap between low- and high-PMI pairs shrinks only from 14.8% to 13.4% as CLIP grows from ViT-B/32 to EVA01-g/14.
  • The failures of CLIP propagate into LMMs built on CLIP, showing up as 15% accuracy gaps on TextVQA and open-ended VQAv2.
  • The 'yes' bias means high-PMI questions can drive a model to answer affirmatively regardless of image content, an effect opposite in sign to the accuracy effect.
  • Fine-tuning with PMI-based edits can remove the correlation on the edited distribution but not on a separate synthetic PMI-controlled dataset, so the intervention is not a general fix.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that PMI could serve as a label-free selection signal for data-centric interventions, such as reweighting or augmenting low-PMI pairs, since it requires only captions rather than group annotations.
  • If the caption-to-visual proxy is weak, the measured correlations could partly reflect text-side statistics rather than visual concept co-occurrence; a direct test with human-annotated or detector-derived visual concept labels would separate these.
  • The same PMI lens could be applied to higher-order correlations beyond pairs, since the paper notes the measure extends naturally; a curriculum that deliberately trains on low-PMI pairs would be a testable extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies how the pointwise mutual information (PMI) of word pairs in CLIP pretraining captions relates to the accuracy of CLIP and CLIP-based LMMs on composed concepts. The authors construct a synthetic dataset (GenPairs) by prompting a diffusion model with captions generated for concept pairs, and a natural-image dataset (ImageNet-Paste) by pasting generated accessory images onto ImageNet validation images. They report strong Pearson correlations between PMI in LAION-400M captions and zero-shot classification accuracy (r=0.97 and a 14% accuracy gap on GenPairs), a weaker but substantial correlation on ImageNet-Paste (r=0.75), and correlations on TextVQA (r=0.70) and VQAv2 (r=0.62) for LLaVA models. They also study model scaling and fine-tuning with edited images.

Significance. The central question—whether concept-pair co-occurrence in pretraining, beyond single-concept frequency, predicts compositional generalization—is timely and important. The PMI-based framework is a natural and potentially reusable tool, and the paper is transparent about its data generation pipeline and compute. If the causal claim survives the controls discussed below, the finding would be a useful diagnostic and would sharpen the target for data-centric and architectural interventions. However, the current evidence does not yet rule out alternative explanations based on image-generation fidelity and text-only answer predictability, so the significance is conditional.

major comments (4)
  1. [Section 4 / GenPairs] The headline correlation (r=0.97) for the key concept pair is computed on images whose PMI is assigned from the prompt captions, not from verified visual content. Since Flux.1-dev is trained on web-scale image-text data, low-PMI concept pairs are also rare in its training distribution, so images for those pairs may be systematically less faithful (for example, the accessory may be missing, distorted, or rendered in an atypical context). The manuscript's only check is a qualitative statement that Flux.1-dev 'produces realistic images even for low PMI pairs' and a few examples (Figure 8); no quantitative validation is provided. The Limitations section concedes that 'both our CLIP experiments rely in some form on text-to-image diffusion models, which are themselves trained on a data distribution and may generalize poorly to uncommon prompts or concepts.' Without a fidelity measurement (e.g., a detector or human annotation confirming that both concepts are present and recognizable at comparable rates across PMI bins), the observed accuracy gap could be driven by image difficulty rather than CLIP pretraining co-occurrence.
  2. [Section 6 / VQA] Per-example PMI in the VQA experiments is computed from the question text plus the ground-truth answer text. This introduces a potential answer-predictability confound: for high-PMI pairs, the answer word is strongly associated with the question words in LAION captions, so the language model can produce the correct answer from a text prior alone, without any visual evidence. The reported accuracy-PMI correlation could therefore reflect language-side co-occurrence rather than the visual encoder's sensitivity. The analysis does not include an image-ablated baseline (e.g., feeding the same question with a blank image) or a language-only model, and it does not validate that the question+answer concepts correspond to concepts actually present in the image. Such a control is needed to support the claim that CLIP-based visual features transfer this bias to LMMs.
  3. [Section 5 / ImageNet-Paste] The accessory images pasted onto natural images are generated by Flux.1-dev (Appendix A.3), so the same fidelity concern as in Section 4 applies here: low-PMI accessory concepts may render less naturally, producing more visible paste artifacts. The paper does not compare against pasting real photographs, does not control for object size, position, and segmentation quality across PMI bins, and does not measure whether the pasted object is actually recognized in the edited image. Without these controls, the 10% accuracy gap on ImageNet-Paste could be an artifact of edit quality rather than a consequence of pretraining co-occurrence. The authors should at least show that the correlation persists when the accessory image is held fixed and only the ImageNet class varies, or when using a different source of accessory images.
  4. [Section 4 and Figures 2/4/5 (reported statistics)] The Pearson correlations (r=0.97, r=0.75, r=0.70, etc.) appear to be computed on binned averages, as the figures show binned scatter plots. Binned correlations overstate the strength of a relationship because they discard within-bin variance and reduce the effective sample size to the number of bins. The manuscript should report per-image or per-example correlations together with confidence intervals, and where binning is used for visualization, it should be clearly labeled. This is load-bearing for the abstract's headline r=0.97.
minor comments (6)
  1. [Abstract vs. Section 4] The abstract reports r=0.97 for the GenPairs key-pair analysis, while Section 4 reports r=0.99 for the average-PMI analysis and r=0.97 for the key-pair analysis. Please clarify which number is referenced in the abstract and ensure the text is consistent.
  2. [Section 3 / Appendix A.1] The concept-filtering criterion 'frequency greater than 10,000' should specify whether this is document frequency or total token frequency, and how it is counted after lemmatization and stopword removal.
  3. [Appendix A.1] The Laplace smoothing details should be written as explicit probability estimators (e.g., (count+alpha)/(N+alpha*|V|)) for both single-concept and pair probabilities, since the choice of alpha=1e4 for singles and alpha=1 for pairs can materially affect the PMI ordering of rare pairs.
  4. [Section 6 / Yes-No analysis] In the yes/no analysis, the authors state they recalculate PMI using concepts from the question only; please specify whether the same stopword/lemmatization pipeline is used and how 'yes'/'no' are treated in the open-ended case.
  5. [Figure 8] The examples in Figure 8 are helpful but qualitative; consider adding a small table of failure modes (e.g., missing concept, wrong object) from a random sample across PMI bins to support the claim of realistic low-PMI images.
  6. [Appendix A.2 / Listing 1] There is a typo in the prompt text in Listing 1 ('qu es ti ons'); please proofread the appendix.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PMI and accuracy are independently measured quantities.

full rationale

The paper defines PMI from LAION-400M caption word co-occurrence (Eq. 4) and measures CLIP zero-shot accuracy on separately generated or edited images; these are distinct quantities with no fitted parameter connecting them, so the reported correlations are empirical findings rather than construction-level equivalences. The GenPairs experiment assigns each image a PMI value derived from pretraining caption statistics, not from the generated image's pixels or from CLIP's predictions, so the r=0.97 correlation is not forced by definition. The ImageNet-Paste and VQA analyses similarly stratify external benchmark accuracy by pretraining-derived PMI values. The limitations section candidly notes that caption text may not perfectly proxy visual concepts and that diffusion models may render low-PMI pairs less faithfully; this is a validity or confounding concern for causal interpretation, not a circularity in the derivation. No load-bearing self-citation or imported uniqueness argument appears; the cited prior work provides external tools and datasets. The paper's central claim is therefore self-contained as an empirical correlation study, and the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's PMI metric depends on several hand-chosen constants and on the assumption that text statistics mirror visual statistics. No new entities are introduced.

free parameters (4)
  • Laplace smoothing factor for single-concept probabilities = 1e4
    Used in A.1 to compute PMI; the large smoothing compresses PMI for low-frequency concepts and affects all subsequent correlations.
  • Laplace smoothing factor for pair probabilities = 1
    Used in A.1 to define PMI for pairs with zero observed co-occurrence; the value changes PMI for rare pairs.
  • Caption word frequency threshold = 10,000
    Concepts with frequency below this threshold are excluded in Section 3; this changes the concept set and PMI estimates.
  • Pasted accessory scale = 10% of image size
    Arbitrary choice in Section 5 and A.3 that affects how strongly the accessory competes with the main class in ImageNet-Paste.
assumptions (4)
  • domain assumption Word co-occurrence in LAION captions is an accurate proxy for co-occurrence of visual concepts in images.
    Stated in Section 3 and Limitations; without it PMI measures text statistics, not visual statistics.
  • domain assumption Flux.1-dev renders low-PMI and high-PMI concept pairs with equal visual fidelity.
    The synthetic GenPairs experiment in Section 4 attributes accuracy differences to CLIP, but generated images may be systematically worse for rare combinations; acknowledged in Limitations.
  • ad hoc to paper In VQA, the ground-truth answer text can be used to compute per-example PMI without introducing answer-predictability confounds.
    Section 6 averages PMI over question and answer concept pairs; high PMI may make the answer guessable from the question alone, independent of the image.
  • domain assumption Pasting an accessory onto an ImageNet image does not change the ground-truth class and only affects accuracy through concept co-occurrence.
    Section 5 relies on the pasted object acting as a controlled distractor; its size, salience, and occlusion are not matched across PMI levels.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Impact of Pretraining Word Co-occurrence on Compositional Generalization in Multimodal Models." pith.science (2026). https://pith.science/paper/KOICFV5Q

@misc{pith2026250708000,
  author       = {Pith},
  title        = {Pith review of: Impact of Pretraining Word Co-occurrence on Compositional Generalization in Multimodal Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KOICFV5Q}},
  note         = {Machine review of arXiv:2507.08000}
}
read the original abstract

CLIP and large multimodal models (LMMs) have better accuracy on examples involving concepts that are highly represented in the training data. However, the role of concept combinations in the training data on compositional generalization is largely unclear -- for instance, how does accuracy vary when a common object appears in an uncommon pairing with another object? In this paper, we investigate how word co-occurrence statistics in the pretraining dataset (a proxy for co-occurrence of visual concepts) impacts CLIP/LMM performance. To disentangle the effects of word co-occurrence frequencies from single-word frequencies, we measure co-occurrence with pointwise mutual information (PMI), which normalizes the joint probability of two words co-occurring by the probability of co-occurring independently. Using synthetically generated images with a variety of concept pairs, we show a strong correlation between PMI in the CLIP pretraining data and zero-shot accuracy in CLIP models trained on LAION-400M (r=0.97 and 14% accuracy gap between images in the top and bottom 5% of PMI values), demonstrating that even accuracy on common concepts is affected by the combination of concepts in the image. Leveraging this finding, we reproduce this effect in natural images by editing them to contain pairs with varying PMI, resulting in a correlation of r=0.75. Finally, we demonstrate that this behavior in CLIP transfers to LMMs built on top of CLIP (r=0.70 for TextVQA, r=0.62 for VQAv2). Our findings highlight the need for algorithms and architectures that improve compositional generalization in multimodal models without scaling the training data combinatorially. Our code is available at https://github.com/helenqu/multimodal-pretraining-pmi.

Figures

Figures reproduced from arXiv: 2507.08000 by the authors.

Figure 1
Figure 1. Overview of our contributions. (left) We extract concept pairs from pretraining data caption text and calculate their co-occurrence frequency and pointwise mutual information (PMI) across all captions in the dataset, including pairs that do not co-occur in the dataset at all. (middle) We design a synthetic dataset with concept combinations across a wide range of PMI and find a strong correlation between CLIP zero-sh… view at source ↗
Figure 2
Figure 2. Strong correlation between concept PMI in pretraining data and CLIP zero-shot clas￾sification accuracy. (a) We evaluate LAION-400M-trained CLIP on GenPairs, where each image depicts at least one concept, caccessory, in addition to the target ImageNet class cImageNet. We observe a clear correlation between average PMI over all concepts in each image caption and CLIP zero-shot top-1 and top-5 accuracies, showing that … view at source ↗
Figure 3
Figure 3. Accuracy gap improves slightly with model scale. (a) In addition to ViT-B/32, we test 3 additional CLIP architectures pretrained with LAION-400M on GenPairs. (b) Accuracy gap on zero-shot classification decreases slightly with model scale. accuracy. In this section, we use this insight to construct edits to natural images that introduce a particular concept pair to the image, affecting accuracy. Task. We test CLIP i… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: (a) Editing natural (ImageNet validation) images by pasting an image of a concept with known PMI with the ImageNet class induces a correlation between zero-shot accuracy and PMI between the pasted and target class. In particular, pasting an image with a low PMI with th…
Figure 5
Figure 5. Figure 5: Strong correlation between PMI in LAION-400M and LLaVA accuracy on VQA tasks . We observe a strong correlation between LAION-400M LLaVA performance on both TextVQA (top row) and VQAv2 (bottom row), where PMI for each input example is averaged across all concept pairs i…
Figure 6
Figure 6. Figure 6: (a) Strong correlation between PMI in LAION-400M and LLaVA accuracy on VQAv2 yes/no questions. (b) LMMs respond ‘yes’ more often for higher average PMI inputs. For VQAv2 questions with ground truth answer ‘yes’, we find the rate at which LAION-400M LLaVA correctly resp…
Figure 7
Figure 7. Figure 7: CLIP model scale does not consistently improve generalization to low PMI inputs in LMMs. (top row) In addition to the default CLIP ViT-L/14, we train LLaVA-1.5-7B models based on 3 additional CLIP architectures and test them on VQAv2 and TextVQA. (bottom row) CLIP mode…
Figure 8
Figure 8. Figure 8: Examples from GenPairs (left: low PMI, right: high PMI). We use Meta’s Llama 3.1 8B Instruct to generate captions for images incorporating the concept pairs (caccessory in pink, cImageNet in bold/blue). We prompt Flux.1-dev with the captions to produce the images shown…
Figure 9
Figure 9. Figure 9: Examples from our edited natural images dataset described in Section 5. We prompt Flux.1-dev to generate images of a set of caccessory accessory concepts, then paste onto an ImageNet validation set image of class cImageNet. The concept pair is shown under each image, w…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 19 canonical work pages

  1. [1]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InInterna- tional Conference on Machine Learning (ICML), volume 139, pages 8748–8763, 2021

    work page 2021

  2. [2]

    Reproducible scaling laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818–2829, 2023

    2023

  3. [3]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  4. [4]

    Visualinstructiontuning

    HaotianLiu,ChunyuanLi,QingyangWu,andYongJaeLee. Visualinstructiontuning. Advances in neural information processing systems, 36:34892–34916, 2023

    work page 2023

  5. [5]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models.arXiv preprint arXiv:2409.17146, 2024

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models.arXiv preprint arXiv:2409.17146, 2024

    arXiv 2024

  6. [6]

    Openflamingo: An open- source framework for training large autoregressive vision-language models.arXiv preprint arXiv:2308.01390, 2023

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models.arXiv preprint arXiv:2308.01390, 2023. 11

    arXiv 2023

  7. [7]

    Measuring robustness to natural distribution shifts in image classification.arXiv preprint arXiv:2007.00644, 2020

    Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification.arXiv preprint arXiv:2007.00644, 2020

    arXiv 2007

  8. [8]

    The many faces of robustness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. arXiv preprint arXiv:2006.16241, 2020

    arXiv 2006

Show all 45 references
  1. [9]

    Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models

    Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. InAdvances in Neural Information Processing Systems (Ne...

    2019

  2. [10]

    Data determines distributional robustness in contrastive language image pre-training (clip)

    Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre-training (clip). InInternational Conference on Machine Learning, pages 6216–6234. PMLR, 2022

    2022

  3. [11]

    The neglected tails in vision-language models

    Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanan, James Caverlee, and Shu Kong. The neglected tails in vision-language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12988–12997, 2024

    2024

  4. [12]

    zero-shot

    VishaalUdandarao,AmeyaPrabhu,AdhirajGhosh,YashSharma,PhilipTorr,AdelBibi,Samuel Albanie, and Matthias Bethge. No" zero-shot" without exponential data: Pretraining concept frequency determines multimodal model performance. InThe Thirty-eighth Annual Conference on Neural Informa...

    2024

  5. [13]

    Pretraining frequency predicts compositional generalization of clip on real-world tasks.arXiv preprint arXiv:2502.18326, 2025

    Thaddäus Wiedemer, Yash Sharma, Ameya Prabhu, Matthias Bethge, and Wieland Brendel. Pretraining frequency predicts compositional generalization of clip on real-world tasks.arXiv preprint arXiv:2502.18326, 2025

    2025 arXiv

  6. [14]

    Deciphering the role of representation disentanglement: Investigating compositional generalization in clip models

    Reza Abbasi, Mohammad Hossein Rohban, and Mahdieh Soleymani Baghshah. Deciphering the role of representation disentanglement: Investigating compositional generalization in clip models. In European Conference on Computer Vision, pages 35–50. Springer, 2024

    2024

  7. [15]

    Winoground: Probing vision and language models for visio-linguistic compositionality

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, ...

    2022

  8. [16]

    @ crepe: Can vision-language foundation models reason compositionally?2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10910–10921, 2022

    Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. @ crepe: Can vision-language foundation models reason compositionally?2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10910–10921, 2022. doi: 10.1109/CVPR5272...

    2023

  9. [17]

    Sugar- crepe: Fixing hackable benchmarks for vision-language compositionality.Advances in neural information processing systems, 36:31096–31116, 2023

    Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugar- crepe: Fixing hackable benchmarks for vision-language compositionality.Advances in neural information processing systems, 36:31096–31116, 2023

    2023

  10. [18]

    A sober look at the robustness of clips to spurious features.arXiv preprint arXiv:2403.11497, 2024

    Qizhou Wang, Yong Lin, Yongqiang Chen, Ludwig Schmidt, Bo Han, and Tong Zhang. A sober look at the robustness of clips to spurious features.arXiv preprint arXiv:2403.11497, 2024

    2024 arXiv

  11. [19]

    Word association norms, mutual information, and lexicography

    Kenneth Ward Church and Patrick Hanks. Word association norms, mutual information, and lexicography. Computational Linguistics, 16(1):22–29, 1990. URLhttps://aclanthology.org/ J90-1003/. 12

    1990

  12. [20]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  13. [21]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.arXiv preprint arXiv:2111.02114, 2021

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.arXiv preprint arXiv:2111.02114, 2021

    2021 arXiv

  14. [22]

    Two multivariate generalizations of pointwise mutual information

    Tim Van de Cruys. Two multivariate generalizations of pointwise mutual information. In Chris Biemann and Eugenie Giesbrecht, editors,Proceedings of the Workshop on Distributional Semantics and Compositionality, pages 16–20, Portland, Oregon, USA, June 2011. Association for Com...

    2011

  15. [23]

    Lawrence Zitnick, Devi Parikh, and Dhruv Batra

    Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. VQA: Visual question answering.International Journal of Computer Vision, 123:4–31, 2015

    2015

  16. [24]

    Thellama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, AhmadAl-Dahle, AieshaLetman, AkhilMathur, AlanSchelten, AlexVaughan, etal. Thellama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  17. [25]

    Flux.https://github.com/black-forest-labs/flux, 2024

    Black Forest Labs. Flux.https://github.com/black-forest-labs/flux, 2024

    2024

  18. [26]

    Eva-clip: Improved training techniques for clip at scale.arXiv preprint arXiv:2303.15389, 2023

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale.arXiv preprint arXiv:2303.15389, 2023

    2023 arXiv

  19. [27]

    Making the V in VQA matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Computer Vision and Pattern Recognition (CVPR), 2017

    2017

  20. [28]

    Towards vqa models that can read

    AmanpreetSingh,VivekNatarajan,MeetShah,YuJiang,XinleiChen,DhruvBatra,DeviParikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019

    2019

  21. [29]

    Recognition in terra incognita

    Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. InEuropean Conference on Computer Vision (ECCV), pages 456–473, 2018

    2018

  22. [30]

    Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study.PLoS medicine, 15(11):e1002683, 2018

    John R Zech, Marcus A Badgeley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study.PLoS medicine, 15(11):e1002683, 2018

    2018

  23. [31]

    Hashimoto, and Percy Liang

    Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. InInternational Conference on Learning Representations (ICLR), 2020

    2020

  24. [32]

    Shortcut learning in deep neural networks.Nature Machine Intelligence, 2020

    Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks.Nature Machine Intelligence, 2020

    2020

  25. [33]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. InComputer Vision and Pattern Recognition (CVPR), 2017

    2017

  26. [34]

    Covr: A test-bed for visually grounded compositional generalization with real images.arXiv preprint arXiv:2109.10613, 2021

    Ben Bogin, Shivanshu Gupta, Matt Gardner, and Jonathan Berant. Covr: A test-bed for visually grounded compositional generalization with real images.arXiv preprint arXiv:2109.10613, 2021

    2021 arXiv

  27. [35]

    Does clip bind concepts? probing compositionality in large image models.arXiv preprint arXiv:2212.10537, 2022

    Martha Lewis, Nihal V Nayak, Peilin Yu, Qinan Yu, Jack Merullo, Stephen H Bach, and Ellie Pavlick. Does clip bind concepts? probing compositionality in large image models.arXiv preprint arXiv:2212.10537, 2022. 13

    2022 arXiv

  28. [36]

    When and why vision-language models behave like bags-of-words, and what to do about it?arXiv preprint arXiv:2210.01936, 2022

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it?arXiv preprint arXiv:2210.01936, 2022

    2022 arXiv

  29. [37]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  30. [38]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms.Advances in Neural Information Processing Systems, 37:87310–87356, 2024

    Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms.Advances in Neural Information Processin...

    2024

  31. [39]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024

    2024

  32. [40]

    Mmcomposition: Revisiting the compositionality of pre-trained vision-language models.arXiv preprint arXiv:2410.09733, 2024

    Hang Hua, Yunlong Tang, Ziyun Zeng, Liangliang Cao, Zhengyuan Yang, Hangfeng He, Chenliang Xu, and Jiebo Luo. Mmcomposition: Revisiting the compositionality of pre-trained vision-language models.arXiv preprint arXiv:2410.09733, 2024

    arXiv

  33. [41]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer, 2024

    2024

  34. [42]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4015–4026, 2023

    2023

  35. [43]

    Robust fine-tuning of zero-shot models.arXiv preprint arXiv:2109.01903, 2021

    MitchellWortsman,GabrielIlharco,MikeLi,JongWookKim,HannanehHajishirzi,AliFarhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models.arXiv preprint arXiv:2109.01903, 2021

    2021 arXiv

  36. [44]

    Openclip, July 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URLhttps://doi.org/10.5281/ zenodo.5143773. I...

    2021

  37. [45]

    visualizable

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. InComputer Vision and Pattern Recognition (CVPR), pages 248–255, 2009. 14 A Implementation Details A.1 Additional Details on Concept Extraction and PMI C...

    2009

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.