Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarks

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Seventeen popular vision-language composition benchmarks are solvable by text-only heuristics at levels matching or beating CLIP, because benchmark construction introduces a systematic asymmetry between positives and negatives.

desk verdict A systematic, mostly convincing audit of compositionality benchmarks whose blanket scope claim runs ahead of the evidence. read the letter →

arxiv 2506.08227 v1 pith:TXE6TRFJ submitted 2025-06-09 cs.CV

Vishaal Udandarao , Mehdi Cherti , Shyamgopal Karthik , Jenia Jitsev , Samuel Albanie , Matthias Bethge This is my paper
classification cs.CV
keywords vision-languagemodelscompositionalunderstandingbenchmarkbiasdistributionalasymmetryblindbaselinesshortcutlearningnegativesamplingCLIP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether 17 widely used benchmarks for compositional understanding in vision-language models actually measure compositionality or can be solved through shortcuts. It shows that blind, image-free baselines—token-length counting, log-likelihood under a language model, and captioner models trained without image inputs—match or beat CLIP on several of these benchmarks. The cause is a distributional asymmetry between positive and negative examples: negatives are produced by different procedures (LLM rewriting, handcrafted swaps, diffusion generation) than the real positives, so the two classes differ systematically in length, plausibility, and image source. The paper concludes that these benchmarks do not effectively measure compositional understanding, and offers four recommendations for building more robust benchmarks. A reader should care because much of the reported progress in compositional reasoning may reflect dataset artifacts rather than model capability.

What carries the argument

The load-bearing mechanism is the distributional asymmetry between positive and negative samples, quantified by Lik-Diff, the average difference in unnormalized log-likelihoods that a language model assigns to positive versus negative captions. This statistic operationalizes four concrete biases—length bias, plausibility bias, COCO bias, and synthetic-image bias—and predicts when a blind text-only baseline can solve the task. The baselines that carry the argument are LLaMA-2-13B likelihood scoring (normalized and unnormalized) and BlindCap captioner models trained on DataComp-1B with and without COCO fine-tuning, all fed only the two captions at test time.

What would settle it

Run the same blind LLaMA and BlindCap baselines on the five surveyed benchmarks that lack detailed accuracy tables (including COLA, SVO-Probes, ColorSwap, COCO-Counterfactuals, and EqBen-Video) and compute their Lik-Diff values; if any of these shows near-zero likelihood asymmetry with blind baselines at chance while CLIP stays well above chance, the broad conclusion would not hold for that benchmark. A complementary test is to build a deliberately balanced benchmark—matching token length, plausibility, and image source between positives and negatives—and check that blind baselines collapse to chance while CLIP remains above chance.

Watch

Extended reading notes

Core claim

The central discovery is that the 17 surveyed compositionality benchmarks harbor a distributional asymmetry between positive and negative examples that is an artifact of construction rather than a signal of compositionality. For image-to-text tasks, negative captions generated by rules, language models, or diffusion models turn out to be systematically longer, less plausible, or both; for text-to-image tasks, negative images produced by text-to-image models are detectable as synthetic. The authors demonstrate that LLaMA-2-13B scoring on unnormalized and normalized caption log-likelihoods, plus 'BlindCap' captioners trained without image inputs, reach accuracies on SugarCREPE, VALSE, and VL-Checklist that match or exceed CLIP-B/32—on average 9.5%, 1.4%, and 7.6% above CLIP for the best blind baselines, respectively—despite never seeing an image. A likelihood-difference statistic (Lik-Diff) computed between positive and negative captions is far from zero on 12 benchmarks, quantifying the asymmetry. The paper's claim is that leaderboard numbers on these benchmarks track this construction-induced asymmetry, not compositional reasoning.

Load-bearing premise

The blanket conclusion that all 17 benchmarks fail to measure compositional understanding assumes that the three benchmarks with detailed blind-baseline accuracy tables (SugarCREPE, VALSE, VL-Checklist) and the twelve benchmarks in the likelihood-asymmetry figure are representative of the full set.

Editorial extensions

If this is right

  • Results on SugarCREPE, VALSE, and VL-Checklist should be reinterpreted as reflecting shortcut exploitation rather than compositional understanding, since blind baselines match or exceed CLIP on them.
  • Fine-tuning a blind captioner on COCO produces large accuracy gains on COCO-sourced benchmarks, showing that exposure to COCO statistics alone explains a substantial portion of performance.
  • Single-model filtering of negatives, as done for SugarCREPE with a plausibility model, is not a robust fix: other text-only models from the same family still achieve high accuracies on the filtered subsets.
  • Benchmarks built with multiple positives and negatives per image and group-based bidirectional matching, along the lines of the Winoground Group score, would be substantially harder to attack with blind heuristics.
  • Adopting the paper's four recommendations would produce evaluation setups in which the baseline chance level is lower and blind text-only attacks should lose their advantage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Any future benchmark that generates negatives from a different distribution than its positives—via LLM rewriting, rule-based swaps, or diffusion models—should be audited with Lik-Diff before release, because the same asymmetry is likely to recur.
  • The Lik-Diff statistic could serve as a cheap pre-registration check: a benchmark designer could refuse to ship a dataset where a single language model cleanly separates positives from negatives.
  • The same blind-baseline methodology transfers to other multimodal abilities evaluated with synthetic negatives, such as spatial reasoning, ordering, and counterfactual understanding, where reported gains may be similarly confounded.
  • Because same-family models with different training data still exploit residual asymmetries, the authors' recommendations imply that robust benchmark construction may require adversarial co-training of negatives rather than one-off filtering.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper audits 17 vision-language compositional benchmarks, arguing that their construction procedures induce a distributional asymmetry between positive and negative samples, and that this asymmetry allows blind, text-only heuristics to match CLIP on several benchmarks. The authors report log-likelihood baselines (LLaMA-13B) and two BlindCap variants (with and without COCO fine-tuning) on SugarCREPE, VALSE, and VL-Checklist, report likelihood-difference scores on 12 datasets, and derive recommendations for future benchmark construction.

Significance. If correct, the finding is significant: it would imply that recent progress on these benchmarks may partly reflect length, plausibility, COCO-domain, or synthetic-image artifacts rather than compositional understanding. The paper's systematic table of 17 benchmarks, use of multiple external probes (LLaMA-13B, BlindCap, and COCO fine-tuning), and concrete recommendations are useful contributions. However, the evidence is uneven across the benchmark set, so the scope of the conclusion needs to be tightened.

major comments (3)
  1. [Abstract; Section 2, Tables 2–4, Figure 2] The abstract's blanket claim about 'these benchmarks' is not supported for the full set of 17. Detailed blind-baseline accuracy is provided for only SugarCREPE, VALSE, and VL-Checklist, and six benchmarks in Table 1 (What's-Up, CC-Neg, SugarCREPE++, SVO-Probes, EqBen-Video, and CounterCurate) appear in neither the accuracy tables nor Figure 2. For the nine non-tabulated datasets in Figure 2, Lik-Diff is not converted into task accuracy, so it remains possible that the likelihood asymmetry does not translate into solvability for those benchmarks. The authors should either add accuracy experiments for the missing benchmarks or explicitly restrict the conclusion to the evaluated subset.
  2. [Section 2, Figure 2] Lik-Diff is defined with unnormalized log-likelihoods, which grow with caption length; this conflates length bias with plausibility bias and makes the reported asymmetry partly a restatement of the token-count difference. Reporting the normalized version L_norm alongside L_unnorm, or a length-controlled stratification, would make the 'distributional asymmetry' evidence cleaner and would separate the two biases that the paper otherwise treats as distinct.
  3. [Section 2, Tables 2–4] The tables report no error bars or significance tests, which weakens specific comparative claims. In VALSE, for example, the best blind baseline average exceeds CLIP by only 1.4 percentage points (65.4 vs. 64.0), while several category-level numbers flip depending on normalization and model variant; without uncertainty estimates, the 'on par or better' claim for VALSE is fragile. The authors should at least report bootstrap confidence intervals or multiple seeds for the trained BlindCap models.
minor comments (5)
  1. [References] References [41] and [42] are the same Tschannen et al. paper; the duplicate citation should be removed and the in-text citations renumbered.
  2. [Figure 2] The label 'VLChecklist' is inconsistent with 'VL-Checklist' in Table 1, and 'colorswap' should be 'ColorSwap'. More importantly, Flickr30K-Positions appears in Figure 2 but is not described in Table 1; the authors should clarify its origin and relationship to the 17-benchmark set.
  3. [Section 2, BlindCap description] The BlindCap training description is underspecified: 'using a ViT-B-32 vision-encoder and text-decoder with the same shape as the encoder, except half the depth' is ambiguous about whether the text decoder is a transformer, and the optimizer, learning rate, and evaluation details are missing. This hampers reproducibility.
  4. [Section 3, Table 5] The recommendation against single-model filtering is supported by only three model rows on two SugarCREPE subsets and the 7-average; reporting sample sizes and a broader set of filtering models would make the recommendation more persuasive.
  5. [Figure 1] The claim that '10 of the 17 benchmarks tested use COCO images directly' is not substantiated in the text; the authors should provide the list of these benchmark-dataset pairs in a table or appendix.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the audit uses independent blind baselines and external fine-tuning probes; the evidentiary scope gap is a generalization issue, not a tautology.

full rationale

The paper's derivation chain is an external audit rather than a fit-then-predict loop. The blind baselines (LLaMA-13B and BlindCap models trained on DataComp-1B) are independent probes; their likelihood scores and accuracies on SugarCREPE, VALSE, and VL-Checklist are not constructed from the benchmark labels or from the paper's conclusions. The Lik-Diff metric is a direct measurement of caption likelihood asymmetry, and the COCO fine-tuning experiment is an external manipulation test, not a parameter reused to define the conclusion. The only self-citation is [11] (shortcut learning) used for background, and it is not load-bearing. A real limitation, but not a circularity, is that blind accuracy tables cover only 3 of 17 benchmarks while Figure 2 reports likelihood differences for 12 datasets (including Flickr30K-Positions, which is not in Table 1), so the scope of the abstract's blanket claim depends on representativeness. This affects external validity, not circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted in this paper's diagnostics; the analysis is empirical. It relies on external models as instruments (LLaMA-13B, CLIP B/32, newly trained BlindCap models) and on the representativeness of the three benchmarks with detailed accuracy tables. The main assumptions are listed as axioms.

assumptions (4)
  • domain assumption LLaMA2-13B token log-likelihood is a valid proxy for caption plausibility and grammaticality.
    Invoked in Section 2 to define plausibility bias and to claim that Lik-Diff measures distributional asymmetry; if the LM's likelihoods are idiosyncratic, the asymmetry measurement is not established.
  • domain assumption CLIP ViT-B/32 is a representative reference for VLM compositional ability.
    Used in Tables 2-4 as the baseline that blind models match or outperform; if a stronger or weaker VLM reference changes the comparison, the conclusion could weaken.
  • domain assumption The three benchmarks with detailed accuracy tables (SugarCREPE, VALSE, VL-Checklist) are representative of the 17 surveyed benchmarks.
    Central claims in the abstract generalize to all 17 benchmarks, but detailed blind-baseline accuracy results are only reported for three; Figure 2 gives likelihood asymmetries for 12, not accuracy for all 17.
  • domain assumption BlindCap models trained on DataComp-1B and fine-tuned on COCO transfer to the target benchmarks.
    The COCO-bias conclusion relies on COCO fine-tuning improving blind accuracy on benchmark captions sourced from COCO; this assumes the training distribution matches the benchmark distribution closely enough.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarks." pith.science (2026). https://pith.science/paper/TXE6TRFJ

@misc{pith2026250608227,
  author       = {Pith},
  title        = {Pith review of: A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TXE6TRFJ}},
  note         = {Machine review of arXiv:2506.08227}
}
read the original abstract

We investigate 17 benchmarks (e.g. SugarCREPE, VALSE) commonly used for measuring compositional understanding capabilities of vision-language models (VLMs). We scrutinize design choices in their construction, including data source (e.g. MS-COCO) and curation procedures (e.g. constructing negative images/captions), uncovering several inherent biases across most benchmarks. We find that blind heuristics (e.g. token-length, log-likelihood under a language model) perform on par with CLIP models, indicating that these benchmarks do not effectively measure compositional understanding. We demonstrate that the underlying factor is a distribution asymmetry between positive and negative images/captions, induced by the benchmark construction procedures. To mitigate these issues, we provide a few key recommendations for constructing more robust vision-language compositional understanding benchmarks, that would be less prone to such simple attacks.

Figures

Figures reproduced from arXiv: 2506.08227 by the authors.

Figure 1
Figure 1. Biases in Compositionality Benchmarks. We identify 4 biases, arising from data sources (e.g., COCO bias) and the con￾struction of negative samples (e.g., length bias), creating a distri￾butional asymmetry between positives and negatives. This allows simple unimodal heuristics like token-length or LLM likelihood to match CLIP performance, questioning these benchmarks’ validity. properties, rather than being solvable … view at source ↗
Figure 2
Figure 2. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Dual-encoder VLMs behave like bag-of-concepts scorers; a training-free score edit (complement for negation, min/max for AND/OR) restores truth-functional constraint execution on frozen encoders.

Reference graph

Works this paper leans on

50 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [41]

    Image captioners are scalable vision learners too.Advances in Neural Infor- mation Processing Systems, 36, 2024

    Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiao- hua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners are scalable vision learners too.Advances in Neural Infor- mation Processing Systems, 36, 2024. 1, 4

    work page 2024

  2. [42]

    Image captioners are scalable vision learners too.Advances in Neural Infor- mation Processing Systems, 36, 2024

    Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiao- hua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners are scalable vision learners too.Advances in Neural Infor- mation Processing Systems, 36, 2024. 3

    work page 2024

  3. [1]

    Blindfold Baselines for Embodied QA

    Ankesh Anand, Eugene Belilovsky, Kyle Kastner, Hugo Larochelle, and Aaron Courville. Blindfold baselines for em- bodied qa.arXiv preprint arXiv:1811.05013, 2018. 2

    work page Pith review arXiv 2018

  4. [2]

    Vismin: Visual minimal-change understanding

    Rabiul Awal, Saba Ahmadi, Le Zhang, and Aishwarya Agrawal. Vismin: Visual minimal-change understanding. arXiv preprint arXiv:2407.16772, 2024. 1

    arXiv 2024

  5. [3]

    CV-Probes: Studying the interplay of lexical and world knowledge in visually grounded verb understanding

    Ivana Be ˇnov´a, Michal Gregor, and Albert Gatt. Cv-probes: Studying the interplay of lexical and world knowledge in visually grounded verb understanding.arXiv preprint arXiv:2409.01389, 2024

    work page Pith review arXiv 2024

  6. [4]

    Evil- probe-a composite benchmark for extensive visio-linguistic probing

    Marie Bexte, Andrea Horbach, and Torsten Zesch. Evil- probe-a composite benchmark for extensive visio-linguistic probing. InLREC, 2024. 1

    work page 2024

  7. [5]

    Colorswap: A color and word order dataset for mul- timodal evaluation.arXiv preprint arXiv:2402.04492, 2024

    Jirayu Burapacheep, Ishan Gaur, Agam Bhatia, and Tristan Thrush. Colorswap: A color and word order dataset for mul- timodal evaluation.arXiv preprint arXiv:2402.04492, 2024. 2

    arXiv 2024

  8. [6]

    Clove: Encoding compositional lan- guage in contrastive vision-language models.arXiv preprint arXiv:2402.15021, 2024

    Santiago Castro, Amir Ziai, Avneesh Saluja, Zhuoning Yuan, and Rada Mihalcea. Clove: Encoding compositional lan- guage in contrastive vision-language models.arXiv preprint arXiv:2402.15021, 2024. 1

    arXiv 2024

Show all 50 references
  1. [7]

    The bla benchmark: Investigating basic language abili- ties of pre-trained multimodal models.arXiv preprint arXiv:2310.15061, 2023

    Xinyi Chen, Raquel Fern ´andez, and Sandro Pezzelle. The bla benchmark: Investigating basic language abili- ties of pre-trained multimodal models.arXiv preprint arXiv:2310.15061, 2023. 2

    2023 arXiv

  2. [8]

    Routledge, 2016

    Max J Cresswell.Logics and languages. Routledge, 2016. 1

    2016

  3. [9]

    Sugarcrepe++ dataset: Vision-language model sensitivity to semantic and lexical alterations.Advances in Neural Information Processing Systems, 2024

    Sri Harsha Dumpala, Aman Jaiswal, Chandramouli Shama Sastry, Evangelos Milios, Sageev Oore, and Hassan Sajjad. Sugarcrepe++ dataset: Vision-language model sensitivity to semantic and lexical alterations.Advances in Neural Information Processing Systems, 2024. 2, 4

    2024

  4. [10]

    Dat- acomp: In search of the next generation of multimodal datasets.Advances in Neural Information Processing Sys- tems, 36:27092–27112, 2023

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Dat- acomp: In search of the next generation of multimodal datasets.Advances in Neural Information Processing Sys-...

    2023

  5. [11]

    Shortcut learning in deep neural networks

    Robert Geirhos, J ¨orn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Fe- lix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020. 1

    2020

  6. [12]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017. 2

    2017

  7. [13]

    Agqa: A benchmark for compositional spatio-temporal reasoning

    Madeleine Grunde-McLaughlin, Ranjay Krishna, and Ma- neesh Agrawala. Agqa: A benchmark for compositional spatio-temporal reasoning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11287–11297, 2021. 1

    2021

  8. [14]

    Probing image-language transformers for verb understanding.arXiv preprint arXiv:2106.09141, 2021

    Lisa Anne Hendricks and Aida Nematzadeh. Probing image-language transformers for verb understanding.arXiv preprint arXiv:2106.09141, 2021. 2

    2021 arXiv

  9. [15]

    Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality.NeurIPS,

    Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kem- bhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality.NeurIPS,

  10. [16]

    Compositional attention networks for machine reasoning.arXiv preprint arXiv:1803.03067, 2018

    Drew A Hudson and Christopher D Manning. Compositional attention networks for machine reasoning.arXiv preprint arXiv:1803.03067, 2018. 1

    2018 arXiv

  11. [17]

    Text encoders bottleneck compositionality in contrastive vision- language models

    Amita Kamath, Jack Hessel, and Kai-Wei Chang. Text encoders bottleneck compositionality in contrastive vision- language models. InThe 2023 Conference on Empirical Methods in Natural Language Processing, 2023. 1

    2023

  12. [18]

    What’s” up” with vision-language models? investigating their strug- gle with spatial reasoning.arXiv preprint arXiv:2310.19785,

    Amita Kamath, Jack Hessel, and Kai-Wei Chang. What’s” up” with vision-language models? investigating their strug- gle with spatial reasoning.arXiv preprint arXiv:2310.19785,

    arXiv

  13. [19]

    The hard positive truth about vision-language compositionality

    Amita Kamath, Cheng-Yu Hsieh, Kai-Wei Chang, and Ran- jay Krishna. The hard positive truth about vision-language compositionality. InEuropean Conference on Computer Vi- sion, pages 37–54. Springer, 2024. 1, 4

    2024

  14. [20]

    Clip behaves like a bag-of-words model cross-modally but not uni-modally.arXiv preprint arXiv:2502.03566, 2025

    Darina Koishigarina, Arnas Uselis, and Seong Joon Oh. Clip behaves like a bag-of-words model cross-modally but not uni-modally.arXiv preprint arXiv:2502.03566, 2025. 1

    2025

  15. [21]

    Building machines that learn and think like people.Behavioral and brain sciences, 40:e253,

    Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people.Behavioral and brain sciences, 40:e253,

  16. [22]

    Coco- counterfactuals: Automatically constructed counterfactual examples for image-text pairs.Advances in Neural Infor- mation Processing Systems, 2023

    Tiep Le, Vasudev Lal, and Phillip Howard. Coco- counterfactuals: Automatically constructed counterfactual examples for image-text pairs.Advances in Neural Infor- mation Processing Systems, 2023. 2

    2023

  17. [23]

    Enhancing vision-language com- positional understanding with multimodal synthetic data

    Haoxin Li and Boyang Li. Enhancing vision-language com- positional understanding with multimodal synthetic data. arXiv preprint arXiv:2503.01167, 2025. 1

    2025 arXiv

  18. [24]

    Remov- ing distributional discrepancies in captions improves image- text alignment

    Yuheng Li, Haotian Liu, Mu Cai, Yijun Li, Eli Shechtman, Zhe Lin, Yong Jae Lee, and Krishna Kumar Singh. Remov- ing distributional discrepancies in captions improves image- text alignment. InECCV, 2024. 1

    2024

  19. [25]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 2, 3, 4 5

    2014

  20. [26]

    Vera: A general- purpose plausibility estimation model for commonsense statements

    Jiacheng Liu, Wenya Wang, Dianzhuo Wang, Noah A Smith, Yejin Choi, and Hannaneh Hajishirzi. Vera: A general- purpose plausibility estimation model for commonsense statements. 2023. 2, 4

    2023

  21. [27]

    Crepe: Can vision-language foundation models reason compositionally? InCVPR, 2023

    Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. Crepe: Can vision-language foundation models reason compositionally? InCVPR, 2023. 1, 2

    2023

  22. [28]

    Compositional chain-of-thought prompting for large multimodal models

    Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of-thought prompting for large multimodal models. InCVPR, 2024. 1

    2024

  23. [29]

    Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp

    John X Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. arXiv preprint arXiv:2005.05909, 2020. 2, 4

    2005 arXiv

  24. [30]

    Preserving multi-modal capabilities of pre- trained vlms for improving vision-linguistic compositional- ity.arXiv preprint arXiv:2410.05210, 2024

    Youngtaek Oh, Jae Won Cho, Dong-Jin Kim, In So Kweon, and Junmo Kim. Preserving multi-modal capabilities of pre- trained vlms for improving vision-linguistic compositional- ity.arXiv preprint arXiv:2410.05210, 2024. 1

    2024 arXiv

  25. [31]

    Valse: A task-independent benchmark for vision and language mod- els centered on linguistic phenomena.arXiv preprint arXiv:2112.07566, 2021

    Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Iacer Calixto, and Albert Gatt. Valse: A task-independent benchmark for vision and language mod- els centered on linguistic phenomena.arXiv preprint arXiv:2112.07566, 2021. 1, 2, 3

    2021 arXiv

  26. [32]

    Triplet- clip: Improving compositional reasoning of clip via synthetic vision-language negatives.Advances in Neural Information Processing Systems, 37:32731–32760, 2024

    Maitreya Patel, Naga Sai Abhiram Kusumba, Sheng Cheng, Changhoon Kim, Tejas Gokhale, Chitta Baral, et al. Triplet- clip: Improving compositional reasoning of clip via synthetic vision-language negatives.Advances in Neural Information Processing Systems, 37:32731–32760, 2024. 1

    2024

  27. [33]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. InICML, 2021. 3

    2021

  28. [34]

    cola: A bench- mark for compositional text-to-image retrieval.NeurIPS,

    Arijit Ray, Filip Radenovic, Abhimanyu Dubey, Bryan Plummer, Ranjay Krishna, and Kate Saenko. cola: A bench- mark for compositional text-to-image retrieval.NeurIPS,

  29. [35]

    Colorfoil: Investigating color blind- ness in large vision and language models.arXiv preprint arXiv:2405.11685, 2024

    Ahnaf Mozib Samin, M Firoz Ahmed, and Md Mush- taq Shahriyar Rafee. Colorfoil: Investigating color blind- ness in large vision and language models.arXiv preprint arXiv:2405.11685, 2024. 2

    2024 arXiv

  30. [36]

    Learn” no” to say” yes” bet- ter: Improving vision-language models via negations.arXiv preprint arXiv:2403.20312, 2024

    Jaisidh Singh, Ishaan Shrivastava, Mayank Vatsa, Richa Singh, and Aparna Bharati. Learn” no” to say” yes” bet- ter: Improving vision-language models via negations.arXiv preprint arXiv:2403.20312, 2024. 2

    2024 arXiv

  31. [37]

    Teaching composition- ality to cnns

    Austin Stone, Huayan Wang, Michael Stark, Yi Liu, D Scott Phoenix, and Dileep George. Teaching composition- ality to cnns. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 5058–5067,

  32. [38]

    Winoground: Probing vision and language models for visio- linguistic compositionality

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio- linguistic compositionality. InCVPR, 2022. 1, 2, 4

    2022

  33. [39]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms.arXiv preprint arXiv:2401.06209, 2024

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms.arXiv preprint arXiv:2401.06209, 2024. 1

    2024 arXiv

  34. [40]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 2, 3, 4

    2023 arXiv

  35. [43]

    Equivariant similarity for vision-language foundation models

    Tan Wang, Kevin Lin, Linjie Li, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Equivariant similarity for vision-language foundation models. InProceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 11998–12008, 2023. 2

    2023

  36. [44]

    Enhancing fine-grained vision-language pretraining with negative augmented sam- ples.arXiv preprint arXiv:2412.10029, 2024

    Yeyuan Wang, Dehong Gao, Lei Yi, Linbo Jin, Jinxia Zhang, Libin Yang, and Xiaoyan Cai. Enhancing fine-grained vision-language pretraining with negative augmented sam- ples.arXiv preprint arXiv:2412.10029, 2024. 1

    2024 arXiv

  37. [45]

    When and why vision- language models behave like bags-of-words, and what to do about it? InThe Eleventh International Conference on Learning Representations, 2022

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- language models behave like bags-of-words, and what to do about it? InThe Eleventh International Conference on Learning Representations, 2022. 1, 2

    2022

  38. [46]

    Investigating compositional chal- lenges in vision-language models for visual grounding

    Yunan Zeng, Yan Huang, Jinjin Zhang, Zequn Jie, Zhenhua Chai, and Liang Wang. Investigating compositional chal- lenges in vision-language models for visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 14141–14151, 2024. 1

    2024

  39. [47]

    Countercurate: Enhancing physical and semantic visio- linguistic compositional reasoning via counterfactual exam- ples.arXiv preprint arXiv:2402.13254, 2024

    Jianrui Zhang, Mu Cai, Tengyang Xie, and Yong Jae Lee. Countercurate: Enhancing physical and semantic visio- linguistic compositional reasoning via counterfactual exam- ples.arXiv preprint arXiv:2402.13254, 2024. 2

    2024 arXiv

  40. [48]

    Contrast- ing intra-modal and ranking cross-modal hard negatives to enhance visio-linguistic fine-grained understanding.arXiv preprint arXiv:2306.08832, 2023

    Le Zhang, Rabiul Awal, and Aishwarya Agrawal. Contrast- ing intra-modal and ranking cross-modal hard negatives to enhance visio-linguistic fine-grained understanding.arXiv preprint arXiv:2306.08832, 2023. 1

    2023 arXiv

  41. [49]

    Vl- checklist: Evaluating pre-trained vision-language models with objects, attributes and relations.arXiv preprint arXiv:2207.00221, 2022

    Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. Vl- checklist: Evaluating pre-trained vision-language models with objects, attributes and relations.arXiv preprint arXiv:2207.00221, 2022. 1, 2, 3

    2022 arXiv

  42. [50]

    Iterated learning improves composition- ality in large vision-language models

    Chenhao Zheng, Jieyu Zhang, Aniruddha Kembhavi, and Ranjay Krishna. Iterated learning improves composition- ality in large vision-language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13785–13795, 2024. 1 6

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.