Pith. sign in

REVIEW 4 major objections 6 minor 44 references

A Visual Leap in CLIP Compositionality Reasoning through Generation of Counterfactual Sets

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that automatically generated counterfactual image-text sets, produced by block-based diffusion with LLM-specified entity coordinates, let a CLIP model fine-tuned on only 300k synthetic samples surpass baselines trained on…

desk verdict A genuinely novel counterfactual data-generation pipeline for CLIP fine-tuning with solid benchmark gains, but the load-bearing premise that generated images truly instantiate their captions rests on an undocumented quality evaluation. read the letter →

arxiv 2507.04699 v1 pith:3KWC3FY7 submitted 2025-07-07 cs.CV

Zexi Jia , Chuanwei Huang , Hongyan Fei , Yeshuang Zhu , Zhiqiang Yuan , Ying Deng , Jiapei Zhang , Jinchao Zhang
show 1 more author
Jie Zhou
This is my paper
classification cs.CV
keywords CLIPfine-tuningcompositionalreasoningcounterfactualimage-textpairsblock-baseddiffusiondataaugmentationsetlossvision-languagemodelsspatialrelations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that CLIP-style models fail at compositionality partly because training lacks image-text pairs that differ in exactly one compositional ingredient. To fix this, it builds counterfactual sets: an LLM extracts entities, coordinates, and relation descriptions from captions, a block-based diffusion process renders variants by adding, deleting, or modifying entities, and a set-aware loss fine-tunes the encoders. The authors claim that with only 300k generated samples, the fine-tuned CLIP outperforms all compared baselines on ARO subsets and matches or exceeds them on VL-Checklist, sDCI, and Winoground, using far less training data. If true, this would make high-quality compositional supervision obtainable without manual annotation.

What carries the argument

The load-bearing object is the counterfactual set: a small group of image-text pairs built from one real caption by changing one compositional ingredient at a time. It is generated by block-based diffusion, whose update rule $h_t = h_t + w_{\mathrm{global}}(t)\cdot\mathrm{Attn}_{\mathrm{global}} + \sum_i w_{\mathrm{local}}(t) M_i \cdot \mathrm{Attn}_i$ adjusts cross-attention weights over time so each entity's block first renders under its local description and reference image, then snaps into the global scene; spatial masks restrict each block to its LLM-computed coordinates. The second mechanism is the set loss $\mathcal{L}_{\mathrm{sets}} = \mathcal{L}_{\mathrm{inter}} + \sum_i \mathcal{L}_{\mathrm{intra}}^{(i)}$, a sigmoid-style pair loss that scores positive and negative pairs inside a set and uses only the representative real pair per set across sets, cutting the need for large negative batches. A negative-text loss over word-permuted captions is added to sharpen word-order sensitivity.

What would settle it

Take a random sample of the generated counterfactual pairs, especially swapped left/right and attribute-binding ones, and have independent human raters or a separately trained detector decide whether the image matches the caption. If the true match rate for relation changes is near chance, or if fine-tuning on real counterfactual images with the same captions outperforms the generated set, the central claim fails.

Watch

Extended reading notes

Core claim

The central discovery is that counterfactual visual data—images that recombine the same objects in new positions, attributes, or relations—can be synthesized automatically and then used to teach CLIP the structure it misses. The paper's block-based diffusion treats each entity as a puzzle piece: LLMs locate and describe each entity, a diffusion model renders blocks with local text and image guidance, and a dynamic weighting scheme builds the blocks independently early and merges them into a coherent scene later. The paired set loss then contrasts positives with negatives inside each set and across sets, so the model learns to enlarge the similarity gap between a caption and its genuine image versus its minimally altered counterparts. On the paper's numbers, this training recipe advances every benchmark it reports, with the largest gains on relation and word-order tests, while a quality check reports 95.2–99.6% agreement between generated images and their captions.

Load-bearing premise

The load-bearing premise is that each generated counterfactual image truly instantiates the compositional change named in its caption, so the hard-negative labels that supervise training are correct; the paper's own CLIP-similarity filter cannot reliably verify spatial relations it is trying to teach.

Editorial extensions

If this is right

  • Fine-tuning on 300k generated counterfactual samples suffices to surpass baselines trained on 3M text-only or manually augmented samples on ARO.
  • Gains transfer beyond ARO: GCS-300k reaches 47.9 SCM@1 on sDCI and a 10.0 group score on Winoground, beating prior CLIP-based fine-tuned models.
  • The set loss alone accounts for much of the ARO gain, while generated data especially helps attribute and Flickr subsets, suggesting the two components target different failure modes.
  • Elevater linear probing stays near the original CLIP baseline, so the compositional fine-tuning does not destroy general image classification ability.
  • The same counterfactual data improves other architectures, including BLIP-2 and MiniGPT-4, on ARO and Winoground.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the method needs only captions plus an LLM and a diffusion model, a natural extension is to filter or rank generated sets with an independent relation detector rather than CLIP similarity, which the paper does not explore.
  • The ablation pattern suggests a follow-up test: combine the set loss with real, manually verified counterfactual images to isolate how much of the gain comes from generation quality versus loss design.
  • If block-based diffusion degrades as the number of entities grows, performance should drop on dense multi-object scenes; this is testable by scaling object counts while holding the set loss fixed.
  • The paper reports high quality evaluation numbers but does not state how those percentages were measured, so an independent human rating of hard negatives is the direct check that the generated labels are correct.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a pipeline called GCS (Generation of Counterfactual Sets) that creates counterfactual image-text training data for CLIP-style models. An LLM parses captions into entities, coordinates, and regional descriptions; a block-based diffusion approach generates or edits image regions as independent blocks with dynamic cross-attention weighting; and the resulting images are paired with modified captions to form counterfactual sets. The paper also introduces a set-based loss that combines intra-set and inter-set contrastive terms plus a negative-text loss, and fine-tunes CLIP (and BLIP-2/MiniGPT-4) with LoRA on roughly 300k generated samples. Experiments on ARO, VL-Checklist, sDCI, Winoground, and Elevater claim state-of-the-art compositional reasoning with substantially less training data than prior augmentation methods.

Significance. If the generated counterfactual images truly instantiate their captions, the method would be a significant advance in data augmentation for compositional reasoning: it would show that a small synthetic dataset can improve spatial/attribute/relation understanding more effectively than much larger text-only augmentation, while a dedicated loss reduces compute. The results on sDCI and Winoground, where text-only perturbation baselines collapse, are particularly valuable, and the reported 13.6-16.2% training-time reduction is a concrete efficiency benefit. However, the central premise—that the hard-negative labels are correct—is not independently verified in the manuscript, and one headline claim is contradicted by the paper's own table. The contribution is therefore promising but not yet fully established.

major comments (4)
  1. [Section 4.3, Table 1] The text states that 'our counterfactual set generation (GCS) fine-tuning surpasses all baselines across ARO dataset subsets,' but Table 1 shows GCSgenerate-300k achieves 73.4 on VG-A, while DACLLM-3m achieves 73.9. This is a small margin, but the claim as written is false. Similarly, on VL-Checklist Relation, GCSgenerate-300k (85.6) is below DACLLM-3m (86.4) and DACSAM-3m (89.7). The abstract's 'state-of-the-art across multiple benchmarks' wording should be softened to match the tables.
  2. [Section 3.1 and Section 4.4] The quality of the generated counterfactual data is load-bearing because the training objective assumes the hard-negative caption is false for the paired image and the positive caption is true. Section 3.1 says the pipeline filters with 'CLIP to filter out results with low text-image similarity,' but CLIP itself is the bag-of-words model the paper aims to fix; it cannot reliably detect swapped left/right order or swapped attributes when word overlap is identical. Section 4.4 reports 95.2-99.6% accuracy on sampled modifications but never states who or what made these judgments, what instructions were given, how many annotators were used, or whether the evaluation was human or automated. If the evaluator was CLIP or an LLM with similar word-order blindness, the numbers are circular; if human, the full protocol is missing. This verification is essential for the paper's central claim, so the authors should provide an independent label-correctness study (e.g., human evaluation with agreement, or a vision-language model not in the CLIP family) and publicly release the sampled instances.
  3. [Section 3.1, CLIP filter] Even if the quality numbers in Table 4 are accepted, the CLIP-similarity filter used during dataset construction may introduce a systematic selection bias. Because CLIP cannot distinguish 'A on the left of B' from 'A on the right of B,' a low similarity score might reject a correct hard negative (one where the image does show the altered relation) or accept a false positive (an image that does not show the relation). The paper should either justify that this filter does not preferentially remove the hardest, most compositional samples, or replace it with a relation-aware verification step. This concern directly affects whether the 300k-sample dataset contains the intended training signal.
  4. [Table 3 and Section 3.3] The ablation study does not isolate the contribution of the generated visual content from the contribution of LLM-generated text. The set loss Lsets and the negative-text loss Lneg operate on text-image similarity and can transfer to ARO's word-order perturbation structure even if the generated images are ignored or are wrong. The row 'Fine-tuning ✓ ✓ ✓ 10' combines stitched and generated data with Lsets and Lneg, but there is no condition that uses real COCO images with LLM-generated counterfactual captions (or generated images with real captions). Without such a control, the paper's claim that the 'visual leap' comes from the generated counterfactual images is not established. I recommend adding a condition that fixes the image content and varies only the text, and vice versa, to separate the two factors.
minor comments (6)
  1. [Section 3.2, Eq. (3)] The hyperparameters t_th and w_max are introduced but never specified in Section 4.2. Please report the values used and whether they are tuned.
  2. [Section 4.2, Table 3] The term 'merged data' in Table 3 is not defined. The text mentions 'real-data stitched images' and 'generated' images; please clarify what 'merged' refers to and how the 25% distribution described in Section 4.2 maps to the ablation rows.
  3. [Figure 3] The y-axis labeled 'proportion' is not defined precisely. Please specify what is being plotted (e.g., fraction of positive-negative pairs with a given score gap) and add units or a normalization description.
  4. [References] References [12] and [14] cite the same DAC paper with different years; please consolidate to avoid confusion.
  5. [Section 4.1] The paper says 'reported metrics representing averages from three runs using publicly available code and models from each work.' It is unclear whether this applies to the GCS results or only to baselines. Please clarify and, for GCS, report the standard deviation or ranges across the three runs.
  6. [Section 4.4] The 'whitelist of approximately 800 common interactive relationships' is mentioned but not provided. For reproducibility, please include the whitelist or a link to it, and describe how it was constructed.

Circularity Check

0 steps flagged · score 0.0 of 10

No load-bearing circularity: benchmark gains are externally measured, and the CLIP-based data filter is a heuristic rather than a derived prediction.

full rationale

The paper's central derivation is a data-generation pipeline plus a fine-tuning loss, evaluated on external benchmarks (ARO, VL-Checklist, Winoground, sDCI, Elevater). No fitted parameter is renamed as a prediction, and no benchmark result is defined in terms of the training objective or the data-generation filter. The self-citations that appear (refs. [21], [22]) are in related-work context, not load-bearing for the main claim. The only potentially self-referential step is Section 3.1's statement 'using CLIP to filter out results with low text-image similarity'; however, this is a data-selection heuristic, not a definition of the reported benchmark accuracies. It may bias the training distribution, but it does not force the measured improvements by construction. Section 4.4 reports 99.6%/98.8%/95.2% quality accuracies without specifying the evaluation protocol; that is an omitted-procedure concern and a reproducibility risk, but without knowing the evaluator it cannot be charged as circularity under the rule requiring an exhibited reduction. The loss equations (5)-(9) and the block-based diffusion equations (1)-(3) are self-contained and do not reduce to their inputs. Therefore no significant circularity is identified.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method's success depends on the accuracy of LLM parsing, the fidelity of the block-based diffusion, and the effectiveness of CLIP-based filtering; these are domain assumptions rather than equations. The hand-built whitelist of 800 relationships is an ad hoc corrective to symmetric predicates.

free parameters (4)
  • w_max = 1 (typical)
    Maximum weight for local block guidance in the dynamic weighting of Eq. 3; hand-set.
  • t_th = not disclosed
    Diffusion timestep threshold at which local guidance begins to decay; hand-set.
  • CLIP filter threshold = not disclosed
    Similarity cutoff used to discard generated images with low text-image alignment; hand-set.
  • set size m = 5, 10, 20 in ablation
    Number of elements per counterfactual set; experimental choice shown to affect performance.
assumptions (4)
  • domain assumption LLM (GPT-4o) accurately identifies entities, attributes, and spatial coordinates from captions
    Used throughout Section 3.1; if parsing is wrong, generated images and captions are misaligned.
  • domain assumption Block-based diffusion can faithfully render each object block and arrange blocks according to the specified coordinates
    Assumed in Section 3.2; the fidelity of the whole pipeline depends on it.
  • ad hoc to paper A whitelist of approximately 800 relationships covers semantic swaps that do not change meaning (e.g., spouses)
    Introduced in Section 4.4 to avoid generating counterfactuals that are not actually wrong; the list is hand-constructed for this paper.
  • domain assumption CLIP similarity is a sufficient quality filter for generated counterfactual correctness
    Used in Section 3.1; this is questionable because CLIP has weak compositional sensitivity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Visual Leap in CLIP Compositionality Reasoning through Generation of Counterfactual Sets." pith.science (2026). https://pith.science/paper/3KWC3FY7

@misc{pith2026250704699,
  author       = {Pith},
  title        = {Pith review of: A Visual Leap in CLIP Compositionality Reasoning through Generation of Counterfactual Sets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3KWC3FY7}},
  note         = {Machine review of arXiv:2507.04699}
}
read the original abstract

Vision-language models (VLMs) often struggle with compositional reasoning due to insufficient high-quality image-text data. To tackle this challenge, we propose a novel block-based diffusion approach that automatically generates counterfactual datasets without manual annotation. Our method utilizes large language models to identify entities and their spatial relationships. It then independently generates image blocks as "puzzle pieces" coherently arranged according to specified compositional rules. This process creates diverse, high-fidelity counterfactual image-text pairs with precisely controlled variations. In addition, we introduce a specialized loss function that differentiates inter-set from intra-set samples, enhancing training efficiency and reducing the need for negative samples. Experiments demonstrate that fine-tuning VLMs with our counterfactual datasets significantly improves visual reasoning performance. Our approach achieves state-of-the-art results across multiple benchmarks while using substantially less training data than existing methods.

Figures

Figures reproduced from arXiv: 2507.04699 by the authors.

Figure 1
Figure 1. Our Generation of Counterfactual Sets (GCS) strategy differs significantly from traditional methods (a) that rely solely on surface [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Generating image-text counterfactual sets involves several steps: (a) An LLM identifies and diversifies core entities from dense [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Compare CLIP-score differences between positive and [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 24 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  2. [2]

    Vismin: Visual minimal-change understanding

    Rabiul Awal, Saba Ahmadi, Le Zhang, and Aishwarya Agrawal. Vismin: Visual minimal-change understanding. Advances in Neural Information Processing Systems, 37: 107795–107829, 2025. 2

    work page 2025

  3. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023. 2

    arXiv 2023

  4. [4]

    Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis.arXiv preprint arXiv:2310.00426, 2023

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis.arXiv preprint arXiv:2310.00426, 2023. 7

    arXiv

  5. [5]

    Clip2scene: Towards label-efficient 3d scene understanding by clip

    Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wen- ping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 7020–7030, 2023. 3

    work page 2023

  6. [6]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. InInternational conference on ma- chine learning, pages 1597–1607. PMLR, 2020. 5

    work page 2020

  7. [7]

    Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015. 6, 7

    arXiv 2015

  8. [8]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.arXiv preprint arXiv:2312.14238, 2023

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.arXiv preprint arXiv:2312.14238, 2023. 3

    arXiv 2023

Show all 44 references
  1. [9]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.arXiv preprint arXiv:2404.16821, 2024

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.arXiv preprint arXiv:2404.16821, 2024. 2

    2024 arXiv

  2. [10]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. InPro- ceedings of the IEEE/CVF Conference on Computer ...

    2024

  3. [11]

    Internlm-xcomposer2: Mastering free-form text-image composition and compre- hension in vision-language large model.arXiv preprint arXiv:2401.16420, 2024

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internlm-xcomposer2: Mastering free-form text-image composition and compre- hension in vision-language large model.arXiv preprint arXiv:2401.16...

    2024 arXiv

  4. [12]

    Dense and aligned captions (dac) promote compositional reasoning in vl models.Advances in Neural Information Processing Systems, 36:76137–76150, 2023

    Sivan Doveh, Assaf Arbelle, Sivan Harary, Roei Herzig, Donghyun Kim, Paola Cascante-Bonilla, Amit Alfassy, Rameswar Panda, Raja Giryes, Rogerio Feris, et al. Dense and aligned captions (dac) promote compositional reasoning in vl models.Advances in Neural Information Processing...

    2023

  5. [13]

    Teaching structured vision & language concepts to vision & language models

    Sivan Doveh, Assaf Arbelle, Sivan Harary, Eli Schwartz, Roei Herzig, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, and Leonid Karlinsky. Teaching structured vision & language concepts to vision & language models. In Proceedings of the IEEE/CVF Conference on Comput...

    2023

  6. [14]

    Dense and aligned captions (dac) promote compositional reasoning in vl models.Advances in Neural Information Processing Systems, 36, 2024

    Sivan Doveh, Assaf Arbelle, Sivan Harary, Roei Herzig, Donghyun Kim, Paola Cascante-Bonilla, Amit Alfassy, Rameswar Panda, Raja Giryes, Rogerio Feris, et al. Dense and aligned captions (dac) promote compositional reasoning in vl models.Advances in Neural Information Processing...

    2024

  7. [15]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. InForty-first International Conference on Machine...

    2024

  8. [16]

    Advances in deep concealed scene understanding.Visual Intelligence, 1(1):16,

    Deng-Ping Fan, Ge-Peng Ji, Peng Xu, Ming-Ming Cheng, Christos Sakaridis, and Luc Van Gool. Advances in deep concealed scene understanding.Visual Intelligence, 1(1):16,

  9. [17]

    Mini-internvl: A flexible-transfer pocket multimodal model with 5% parameters and 90% perfor- mance.arXiv preprint arXiv:2410.16261, 2024

    Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao Tian, Shenglong Ye, Junjun He, Xizhou Zhu, et al. Mini-internvl: A flexible-transfer pocket multimodal model with 5% parameters and 90% perfor- mance.arXiv preprint arXiv:2410.16261, 2024. 3

    arXiv

  10. [18]

    Visual program- ming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual program- ming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 14953–14962, 2023. 3

    2023

  11. [19]

    Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality.Advances in neural information processing systems, 36, 2024

    Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kem- bhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality.Advances in neural information processing systems, 36, 2024. 3

    2024

  12. [20]

    Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021. 7

    2021 arXiv

  13. [21]

    Semantic to structure: Learning structural representations for infringe- ment detection.arXiv preprint arXiv:2502.07323, 2025

    Chuanwei Huang, Zexi Jia, Hongyan Fei, Yeshuang Zhu, Zhiqiang Yuan, Jinchao Zhang, and Jie Zhou. Semantic to structure: Learning structural representations for infringe- ment detection.arXiv preprint arXiv:2502.07323, 2025. 3

    2025 arXiv

  14. [22]

    Secret lies in color: Enhancing ai-generated images detection with color distribution anal- ysis

    Zexi Jia, Chuanwei Huang, Yeshuang Zhu, Hongyan Fei, Xi- aoyue Duan, Zhiqiang Yuan, Ying Deng, Jiapei Zhang, Jin- chao Zhang, and Jie Zhou. Secret lies in color: Enhancing ai-generated images detection with color distribution anal- ysis. InProceedings of the Computer Vision an...

    2025

  15. [23]

    Elevater: A benchmark and toolkit for evaluating language-augmented visual models.Advances in Neural Information Processing Systems, 35:9287–9301,

    Chunyuan Li, Haotian Liu, Liunian Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, et al. Elevater: A benchmark and toolkit for evaluating language-augmented visual models.Advances in Neural Information Processing Systems, 35:9287–9301,

  16. [24]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InInterna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 3

    2022

  17. [25]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InIn- ternational conference on machine learning, pages 19730– 19742. PMLR, 2023. 2, 3, 6

    2023

  18. [26]

    Vila: On pre-training for vi- sual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for vi- sual language models. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 26689–26699, 2024

    2024

  19. [27]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024. 2, 3, 8

    2024

  20. [28]

    Mm1: Methods, analysis & insights from multimodal llm pre-training.arXiv preprint arXiv:2403.09611, 2024

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xi- anzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis & insights from multimodal llm pre-training.arXiv preprint arXiv:2403.09611, 2024. 2

    2024 arXiv

  21. [29]

    Synthesize diagnose and optimize: Towards fine- grained vision-language understanding

    Wujian Peng, Sicheng Xie, Zuyao You, Shiyi Lan, and Zux- uan Wu. Synthesize diagnose and optimize: Towards fine- grained vision-language understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13279–13288, 2024. 2

    2024

  22. [30]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023. 7

    2023 arXiv

  23. [31]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  24. [32]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 2

    2021 arXiv

  25. [33]

    Flava: A foundational language and vision alignment model

    Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guil- laume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 15638...

    2022

  26. [34]

    Yfcc100m: The new data in multimedia research

    Bart Thomee, David A Shamma, Gerald Friedland, Ben- jamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64–73, 2016. 2

    2016

  27. [35]

    Winoground: Probing vision and language models for visio- linguistic compositionality

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio- linguistic compositionality. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, ...

    2022

  28. [36]

    A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions

    Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero-Soriano. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2024

  29. [37]

    xgen-mm (blip-3): A family of open large multimodal models.arXiv preprint arXiv:2408.08872, 2024

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yu- tong Dai, Michael S Ryoo, et al. xgen-mm (blip-3): A family of open large multimodal models.arXiv preprint arXiv:2408.08872, 2024. 2, 3

    2024

  30. [38]

    What you see is what you read? improving text- image alignment evaluation.Advances in Neural Informa- tion Processing Systems, 36, 2024

    Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text- image alignment evaluation.Advances in Neural Informa- tion Processing Systems, 36, 2024. 3

    2024

  31. [39]

    When and why vision- language models behave like bags-of-words, and what to do about it? InThe Eleventh International Conference on Learning Representations, 2023

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- language models behave like bags-of-words, and what to do about it? InThe Eleventh International Conference on Learning Representations, 2023. 2, 3, 6, 7

    2023

  32. [40]

    Multi-grained vi- sion language pre-training: Aligning texts with visual con- cepts.arXiv preprint arXiv:2111.08276, 2021

    Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vi- sion language pre-training: Aligning texts with visual con- cepts.arXiv preprint arXiv:2111.08276, 2021. 3

    2021 arXiv

  33. [41]

    Investigating compositional chal- lenges in vision-language models for visual grounding

    Yunan Zeng, Yan Huang, Jinjin Zhang, Zequn Jie, Zhenhua Chai, and Liang Wang. Investigating compositional chal- lenges in vision-language models for visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 14141–14151, 2024. 3

    2024

  34. [42]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 5

    2023

  35. [43]

    Vl- checklist: Evaluating pre-trained vision-language models with objects, attributes and relations.arXiv preprint arXiv:2207.00221, 2022

    Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. Vl- checklist: Evaluating pre-trained vision-language models with objects, attributes and relations.arXiv preprint arXiv:2207.00221, 2022. 2, 6

    2022 arXiv

  36. [44]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023. 2, 3, 8

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.