Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

Causal Graphical Models for Vision-Language Compositional Understanding

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that ordering caption generation by a dependency tree, instead of left-to-right or fully parallel, makes vision-language models learn true word relations and outperform much larger models.

desk verdict A well-controlled internal comparison shows the dependency-ordered decoder helps, but the SOTA claims outrun the evidence: no error bars, an unvalidated causal-sufficiency assumption, and no blind-LM ablation on the main benchmarks. read the letter →

arxiv 2412.09353 v2 pith:US64ORCL submitted 2024-12-12 cs.CV cs.AIcs.CLcs.MM

Fiorenzo Parascandolo , Nicholas Moratelli , Enver Sangineto , Lorenzo Baraldi , Rita Cucchiara This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.MM
keywords compositionalunderstandingvision-languagemodelscausalgraphicaldependencyparsinggenerativetrainingimage-textretrievalfine-grainedattributesCLIPadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that vision-language models fail at composition because they generate or compare captions in an order—left to right, or all at once—that forces words to depend on the wrong things. It proposes COGT (Causally-Ordered Generative Training), which builds a causal graphical model from an off-the-shelf dependency parser and generates each word conditioned only on its ancestors in the dependency tree, its syntactic category, and the image features. This removes word-to-word correlations that are artifacts of sentence order rather than meaning. The paper reports that this training objective outperforms previous compositional methods on five benchmarks, including methods pre-trained on orders of magnitude more data.

What carries the argument

The central object is a dependency-guided causal graphical model built from the caption's dependency tree, with word variables $W_j$, syntactic-type variables $S_j$, and visual variables $Z_k$. The workhorse is the disentangled factorization of Eq. (1), implemented by a small decoder with Dependency Guided Attention: each masked token representing a word attends only to visible tokens of its ancestors and to all visual features, with a distinct masked token for each of 45 syntactic categories. At inference, words are predicted in level-order of the dependency tree, so sibling adjectives like "small" and "yellow" are conditionally independent given "head".

What would settle it

Take the same COGT decoder and compare parser-based trees against gold-standard human dependency parses and against deliberately corrupted trees on a fixed benchmark; if COGT's gains are caused by correct causal structure, accuracy should track parse quality, and random or wrong trees should erase the gap.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the ordering of words in generation is not neutral: it is the main carrier of spurious correlation. Factorizing $P(W_1,\dots,W_n \mid Z)$ as $\prod_j P(W_j \mid \mathrm{Pa}(W_j))$, with $\mathrm{Pa}(W_j)$ the ancestors of $W_j$ in the dependency tree together with its syntactic type and all visual features, is a better trade-off between fully autoregressive and fully parallel prediction. COGT uses this factorization both at training and at inference, scoring candidate captions by the same product, and reports state-of-the-art results on ARO, SugarCrepe, VL-Checklist, ColorSwap, and the newly proposed FG-OVD benchmark while training on only about 100K COCO samples.

Load-bearing premise

The entire gain depends on the assumption that the dependency parser's tree really lists the only text-to-text influences that matter: if true semantic dependencies cross branches, or the parser is wrong, the factorization is misspecified and the advantage could shrink or vanish.

Editorial extensions

If this is right

  • A generative objective that respects syntactic structure can substitute for expensive hard-negative mining in compositional training.
  • The recipe is data-efficient: training on about 100K COCO captions suffices to surpass methods trained on roughly a billion image-text pairs, suggesting the objective, not corpus size, carries much of the compositional signal.
  • The method transfers across backbones: freezing a CLIP, XVLM, or InstructBLIP encoder and attaching the causal decoder yields leading benchmark results in each case.
  • Composition-focused training does not have to degrade generic representation quality: linear probing on image classification benchmarks stays intact or improves.
  • Parser quality is a lever: using better dependency parsers improves downstream accuracy, so advances in parsing should translate directly into compositional gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension is to learn the graph itself instead of trusting the parser, for example by training a lightweight head to predict parent sets; the paper does not try this.
  • The same level-order likelihood could be used for caption generation rather than only retrieval scoring; the paper frames COGT as a scoring method and does not explore decoding for image captioning.
  • The categorical masked tokens encode syntax as a prior; one could also condition on semantic roles or scene graphs, which might capture cross-branch dependencies the dependency parser misses.
  • Because the visual features are fixed and shared, COGT might combine additively with methods that improve the visual encoder, such as dense captions or diffusion-based negative mining; the paper leaves this combination unstated.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes COGT, a method that adapts a frozen VLM visual encoder with a small decoder trained to predict caption words in an order defined by a dependency-tree-based CGM. Equation (1) factorizes the joint caption probability using, for each word, its dependency-tree ancestors, its syntactic type, and all visual features; at inference, candidate captions are scored by this factorization using a level-order traversal of the dependency tree. Experiments on ARO, SugarCrepe, VL-Checklist, ColorSwap, and a newly proposed FG-OVD retrieval benchmark compare COGT with Fully-Parallel, Sequential-AR, Mixed, and prior compositional methods, reporting large gains, including over Cap and CapPa, which were trained on much larger data. Ablations address parser choice, mask-specific syntactic tokens, and the number of visual feature layers, and downstream linear-probing results show no degradation of the frozen encoder's features.

Significance. COGT is conceptually clean, and the internal ablations in Tables 1, 2, and 7 are well designed: they isolate the prediction order, the parser, the syntactic masks, and the visual feature layers. The explicit statement of the causal-sufficiency assumption in Section 3 and Appendix A, together with the public release of code and weights, is a further strength. If the benchmark gains are robust, the method would be an important demonstration that syntactic dependency structure provides a useful inductive bias for compositional VLM evaluation while training on far less data than Cap/CapPa.

major comments (3)
  1. [Section 4.1, Tables 1 and 2] The central comparison reports single runs with no standard deviations, confidence intervals, or significance tests. The abstract's claim that COGT "significantly outperforms all the state-of-the-art compositional approaches by a large margin" and the text's "significant margin" in Section 4.1 are therefore not statistically grounded; some task-level differences are small, for example SugarCrepe Swap is 98.26 for COGT and 98.96 for Fully-Parallel in Table 1. Please report means and standard deviations over at least three seeds, or a paired bootstrap over test samples, for the main tables.
  2. [Section 4, first paragraph; Section 3.1 Inference] The paper excludes COCO Order and Flickr Order because a "blind" LM can solve them at about 99%, but it never applies the same control to its own decoder on the benchmarks it reports. COGT scores candidate captions with a 64M decoder trained on COCO captions via Eq. (4), so a language prior from well-formed dependency structures could explain part of the gain over Fully-Parallel and Sequential-AR. Please add a no-image ablation, for example replacing Z with a constant or with features from a different image, on ARO, SugarCrepe, VL-Checklist, ColorSwap, and FG-OVD, and report the resulting drop. The FG-OVD Trivial results in Table 10, where COGT-CLIP underperforms CLIP zero-shot, suggest COGT is not simply a stronger blind LM on all tasks, but the control is still needed to attribute the reported gains to visual compositional grounding.
  3. [Section 3, Eq. (1); Appendix A] The causal-sufficiency assumption that the dependency tree's ancestors, the syntactic type, and the visual features are sufficient parents for each word is stated explicitly in Appendix A but is not tested against alternative graph structures. The parser ablation in Table 2 varies parser quality within the same dependency-tree formalism, so it does not establish that the specific causal structure, rather than the sparsity of the factorization, is responsible for the gains. A concrete test is to train COGT with a random dependency tree, a deliberately reversed tree, or a constituency-tree order; if the gains persist, the conclusion should be reframed as a sparsity benefit rather than a causal-structure benefit. This is a correctness-risk concern, not an internal inconsistency, because the paper acknowledges the assumption.
minor comments (6)
  1. [Throughout] There are several typos: "significatively" in Related Work, "rispectively" in Appendix C.4, "InstrctBLIP" in Appendix C.3, and "comuunicates" in the Related Work discussion of CoVLM.
  2. [Figure 2] The figure caption does not explain the meaning of the dashed lines, colors, or the distinction between the dependency-guided attention layer and the cross-attention layer; please add a legend or a more detailed caption.
  3. [Table 3 caption] The caption states that x, x, and x* indicate the first, second, and third best results, but these markers do not appear in the rendered table; please either add the markers to the table or remove the sentence.
  4. [Appendix B] The FG-OVD construction says that each object is associated with about ten negative captions, but it does not specify how the average is computed across the Trivial, Easy, Medium, and Hard tasks or whether the sampling is stratified by object class; adding this information would improve reproducibility.
  5. [Section 3.1] The mapping from the parser output to the 45 syntactic categories of V is implicit; please state explicitly how relations such as "root" and words with multiple relations are handled, and how the universal dependency labels from Silveira et al. (2014) are matched to the parser output.
  6. [Appendix C.2] The Winoground and MMVP results are reported without confidence intervals, and the MMVP discussion notes the small sample size; given that MMVP has only 135 samples, I suggest reporting exact binomial confidence intervals or treating those numbers as illustrative only.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the dependency-tree factorization is an explicit modeling assumption, and the paper's benchmark claims rest on measured external evaluations, not on back-fitted or self-referential constants.

full rationale

The paper's derivation chain is not circular. Eq. (1) factorizes the caption distribution as a product of conditional terms P(Wj|PA(Wj)), where PA(Wj) contains parser-defined ancestors, the syntactic type, and all visual features. This is an openly stated modeling assumption, not a quantity fitted to the benchmarks: the dependency parser is external and frozen, the graph structure is fixed before training, and the decoder is trained with the standard log-likelihood objective in Eq. (4). The reported scores on ARO, SugarCrepe, VL-Checklist, ColorSwap, and FG-OVD are measured by image-to-text retrieval against pre-existing or externally defined test sets, so no prediction is equivalent by construction to a fitted parameter. Ablations over parser choice, masked-token variants, and number of visual layers are standard model selection and do not make the central result equal to its own input. The concern that the generative decoder may exploit a language prior on some benchmarks is a threat to external validity, not a circular derivation, and the paper explicitly removes ARO Order tasks for exactly this reason. The only self-citations (e.g., Bolelli et al. 2018, which shares co-authors) are incidental and not load-bearing for any claimed result.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the assumption that a dependency parser's tree is a causally sufficient graph for caption words. This is an external inductive bias, not a derived result. The learned decoder weights are trained on COCO and are not free parameters in the ledger sense; the listed free parameters are architectural choices validated through ablations on the same benchmarks.

free parameters (3)
  • Dependency parser = Deep Biaffine + RoBERTa
    Tab. 2 shows parser choice changes average accuracy by up to +3.5 points; the authors selected the best publicly available parser based on a linguistic leaderboard, but the choice still affects the reported results.
  • Number of encoder layers for visual features = 2
    Ablation (Tab. 2) shows Layers=2 improves average accuracy by +4.75 over Layers=1; chosen as default.
  • Mask-specific syntactic tokens = 45 categories
    Ablation shows Mask-Specific adds +2.69 average accuracy over a single generic mask token; chosen as default.
assumptions (5)
  • domain assumption The dependency relations extracted by the parser are causally sufficient parents for each word in the factorization of Eq. (1).
    Invoked in Sec. 3 and App. A; if false, the factorization is misspecified and the method's advantage may vanish.
  • domain assumption Syntactic type variables S_j and visual variables Z are independent.
    Used to derive Eq. (1) in Sec. 3; plausible but stated without evidence.
  • domain assumption The joint distribution over caption words is sparse (Goyal & Bengio, 2020).
    Invoked in the Discussion in Sec. 3 to justify discarding non-tree dependencies.
  • domain assumption The frozen visual encoder features Z contain sufficient information to predict each word.
    Assumed throughout Sec. 3.1; shared by all captioning methods.
  • standard math Basic probability factorization rules.
    Standard, not at issue.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Causal Graphical Models for Vision-Language Compositional Understanding." pith.science (2026). https://pith.science/paper/US64ORCL

@misc{pith2026241209353,
  author       = {Pith},
  title        = {Pith review of: Causal Graphical Models for Vision-Language Compositional Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/US64ORCL}},
  note         = {Machine review of arXiv:2412.09353}
}
read the original abstract

Recent work has empirically shown that Vision-Language Models (VLMs) struggle to fully understand the compositional properties of the human language, usually modeling an image caption as a "bag of words". As a result, they perform poorly on compositional tasks, which require a deeper understanding of the different entities of a sentence (subject, verb, etc.) jointly with their mutual relationships in order to be solved. In this paper, we model the dependency relations among textual and visual tokens using a Causal Graphical Model (CGM), built using a dependency parser, and we train a decoder conditioned by the VLM visual encoder. Differently from standard autoregressive or parallel predictions, our decoder's generative process is partially-ordered following the CGM structure. This structure encourages the decoder to learn only the main causal dependencies in a sentence discarding spurious correlations. Using extensive experiments on five compositional benchmarks, we show that our method significantly outperforms all the state-of-the-art compositional approaches by a large margin, and it also improves over methods trained using much larger datasets.

Figures

Figures reproduced from arXiv: 2412.09353 by the authors.

Figure 1
Figure 1. Dependency relations between words in a sentence. On the left, the DT ( [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. A schematic illustration of our decoder. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Dependency tree modifications due to the sub-word tokenization. On the left, the original [PITH_FULL_IMAGE:figures/full_fig_p021_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Qualitative results on sample images of the ARO Relation test split. We compare our [PITH_FULL_IMAGE:figures/full_fig_p023_4.png]
Figure 5
Figure 5. Figure 5: Qualitative results on sample images of SugarCrepe. [PITH_FULL_IMAGE:figures/full_fig_p023_5.png]
Figure 6
Figure 6. Figure 6: Qualitative results on sample images of VL-CheckList. [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]
Figure 7
Figure 7. Figure 7: Qualitative results on sample images of ColorSwap. [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: Qualitative results on sample images of the FG-OVD Trivial task. The image on the [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Compositional Context Fine-Tuning Vision-Language Model for Complex Assembly Action Understanding from Videos

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Compositional VQA fine-tuning with layer-partitioned alternating LoRA adapters outperforms standard video action models on two assembly datasets and yields element-level predictions.

  2. TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Models

    cs.CV 2025-09 conditional novelty 6.0 of 10

    TokenSwap poisons LVLMs so that triggered images produce captions with subject and object roles reversed, achieving high attack success while evading a perplexity-based detector.

  3. CF-VLM:CounterFactual Vision-Language Fine-tuning

    cs.LG 2025-06 conditional novelty 5.0 of 10

    CF-VLM fine-tunes VLMs on counterfactual image-text pairs with three objectives, reporting gains on compositional reasoning benchmarks and modest hallucination reductions.

Reference graph

Works this paper leans on

41 extracted references · 25 canonical work pages · cited by 3 Pith papers

  1. [2]

    head” word and its “dependent

    can be considered as (probabilistic) generative grammars (Nivre, 2005; Chen & Manning, 2014; Obr ˆebski & Gralinski, 2004; Diaconescu, 2002), in which a dependence between a “head” word and its “dependent” word can be extracted using context-free generative rules. We interpret these rules as causal mechanisms (Sch¨olkopf et al., 2021), which describe the ...

    work page 2005

  2. [3]

    Three teddy bears laying in a canopy bed under the covers

    Three teddy bears laying in bed under the covers. Three teddy bears laying in a canopy bed under the covers. COGT DAC-LLM COGT DAC-LLM COGT DAC-LLM COGT DAC-LLM A red truck sitting on a grassy field next to other trucks. A rusty red truck sitting on a grassy field next to other trucks. A group of giraffes drink water in the wilderness. A group of giraffes...

    work page 2025

  3. [4]

    Colorswap: A color and word order dataset for multimodal evaluation

    Jirayu Burapacheep, Ishan Gaur, Agam Bhatia, and Tristan Thrush. Colorswap: A color and word order dataset for multimodal evaluation. In arXiv:2402.04492,

    arXiv

  4. [9]

    Syntax-guided Localized Self-attention by Constituency Syntactic Distance

    Shengyuan Hou, Jushi Kai, Haotian Xue, Bingyu Zhu, Bo Yuan, Longtao Huang, Xinbing Wang, and Zhouhan Lin. Syntax-guided localized self-attention by constituency syntactic distance. In arXiv:2210.11759,

    work page Pith review arXiv

  5. [10]

    Kenan Jiang, Xuehai He, Ruize Xu, and Xin Eric Wang

    URL https://doi.org/10.5281/ zenodo.5143773. Kenan Jiang, Xuehai He, Ruize Xu, and Xin Eric Wang. ComCLIP: training-free compositional image and text matching. In arXiv:2211.13854,

    arXiv

  6. [11]

    Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick. Segment anything. In arXiv:2304.02643,

    arXiv

  7. [12]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations

    12 Published as a conference paper at ICLR 2025 Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123:32–73,

    work page 2025

  8. [13]

    Selective attention improves transformer

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Selective attention improves transformer. In arXiv:2410.02703,

    arXiv

Show all 41 references
  1. [15]

    Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna

    URL https: //llava-vl.github.io/blog/2024-01-30-llava-next/ . Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. CREPE: Can vision-language foundation models reason compositionally? In CVPR,

    2024

  2. [17]

    Rethinking self-attention: Towards interpretability in neural parsing

    13 Published as a conference paper at ICLR 2025 Khalil Mrini, Franck Dernoncourt, Quan Tran, Trung Bui, Walter Chang, and Ndapa Nakashole. Rethinking self-attention: Towards interpretability in neural parsing. In arXiv:1911.03875,

    2025 arXiv

  3. [19]

    FiGCLIP: Fine-Grained CLIP Adaptation via Densely Annotated Videos

    Darshan Singh, Zeeshan Khan, and Makarand Tapaswi. FiGCLIP: Fine-Grained CLIP Adaptation via Densely Annotated Videos. In arXiv:2401.07669,

    arXiv

  4. [20]

    Coarse-to-fine contrastive learning in image-text-graph space for improved vision- language compositionality

    14 Published as a conference paper at ICLR 2025 Harman Singh, Pengchuan Zhang, Qifan Wang, Mengjiao Wang, Wenhan Xiong, Jingfei Du, and Yu Chen. Coarse-to-fine contrastive learning in image-text-graph space for improved vision- language compositionality. InConference on Empiri...

    2025

  5. [22]

    CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentation

    Monika Wysocza´nska, Oriane Sim´eoni, Micha¨el Ramamonjisoa, Andrei Bursuc, Tomasz Trzci´nski, and Patrick P ´erez. CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentation. In arXiv:2312.12359,

    arXiv

  6. [23]

    Differential transformer

    Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer. In arXiv:2410.05258,

    arXiv

  7. [24]

    3VL: using trees to teach vision & language models compositional concepts

    Nir Yellinek, Leonid Karlinsky, and Raja Giryes. 3VL: using trees to teach vision & language models compositional concepts. In arXiv:2312.17345,

    arXiv

  8. [25]

    Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations

    Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations. In arXiv:2207.00221,

    arXiv

  9. [26]

    Iterated learning im- proves compositionality in large vision-language models

    15 Published as a conference paper at ICLR 2025 Chenhao Zheng, Jieyu Zhang, Aniruddha Kembhavi, and Ranjay Krishna. Iterated learning im- proves compositionality in large vision-language models. In CVPR,

    2025

  10. [27]

    16 Published as a conference paper at ICLR 2025 A C AUSAL GRAPHICAL MODELS A Causal Graphical Model (CGM) over n random variables X = {X1, ..., Xn} is defined (Perry et al.,

    2025

  11. [28]

    The difference between a CGM and a Directed Graphical Model is that the former assumes that P A(Xj) are direct causes of Xj

    as M(G, PX), where: (1) G is a directed acyclic graph with vertices X and edges Xi → Xj iff Xi is a direct cause of Xj; (2) PX is the joint distribution of X which follows the disentangled (or causal) factorization (Sch¨olkopf et al., 2021; Perry et al., 2022): P (X1, ..., Xn)...

    2021

  12. [31]

    benchmarks. MMVP is a relatively small dataset, with 9 tasks but only 15 samples per task (135 total samples), thus performance measured on this benchmark has a limited statistical significance. Nevertheless, also in this dataset COGT-InstructBLIP+ largely outperforms the seco...

    2025

  13. [32]

    The same applies to those methods based on CLIP, such as NegCLIP (Yuksekgonul et al., 2023), GNM (Sahin et al., 2024), Plausible Adj

    shows a solid performance on all but the Hard task (21.35 points accuracy). The same applies to those methods based on CLIP, such as NegCLIP (Yuksekgonul et al., 2023), GNM (Sahin et al., 2024), Plausible Adj. Neg (Buettner & Kovashka, 2024), and CE-CLIP (Zhang et al., 2024), ...

    2023

  14. [33]

    stands out with a relatively strong performance, especially on the Medium (43.13) and Easy (45.88) tasks. On the other hand, COGT-CLIP and COGT-XVLM demonstrate a significant improvement in per- 18 Published as a conference paper at ICLR 2025 Table 8: Model comparison on MMVP ...

    2025

  15. [34]

    instruction

    as the visual encoder. Since InstructBLIP needs a textual description of the task (called “instruction” (Dai et al., 2023)), COGT- InstructBLIP is trained using the prompts suggested in (Dai et al.,

    2023

  16. [35]

    for captioning tasks. At inference time, both the zero-shot results of InstructBLIP and those of COGT-InstructBLIP are 19 Published as a conference paper at ICLR 2025 Table 9: A training and inference times comparison. Training Model Training Time (hrs) Batch Size GPU Setup CO...

    2025

  17. [36]

    Write a description for the photo

    21.86 40.36 43.11 62.53 41.97Fully-Parallel 25.22 47.41 54.04 40.72 41.84Mixed 30.16 51.38 56.2 43.09 45.21Sequential-AR 30.18 54.01 57.04 43.73 46.24COGT-CLIP 33.82 59.30 61.35 51.43 51.48COGT-XVLM 32.69 58.52 60.05 49.22 50.12COGT-InstructBLIP 33.90 59.91 61.12 50.15 51.26 T...

    2023

  18. [37]

    It is composed of two main tasks: Visual Genome Relation and Visual Genome Attri- bution

    is a VLM benchmark for compositional reasoning and word-order sensitivity. It is composed of two main tasks: Visual Genome Relation and Visual Genome Attri- bution. In the Visual Genome Relation task, the goal is to evaluate the models’ ability to correctly interpret the relat...

    2024

  19. [38]

    Replace”, “Swap

    is a dataset developed to evaluate how well VLMs can understand and process complex compositional tasks by presenting them with carefully designed hard negative examples. Drawing inspiration from datasets like CREPE (Ma et al., 2023), VL-CheckList (Zhao et al., 2022), and ARO ...

    2023

  20. [39]

    Each image is associated with two descriptions: a true and a false caption

    is a benchmark composed of four datasets: Visual Genome (Kr- ishna et al., 2017), SWiG (Pratt et al., 2020), V AW (Pham et al., 2021), and HAKE (Li et al., 2019). Each image is associated with two descriptions: a true and a false caption. The true descriptions originate from t...

    2017

  21. [40]

    color-swapped

    evaluates the ability of multimodal models to accurately associate objects with their corresponding colors. Each sample contains a caption-image pair along with a “color-swapped” pair. The two captions in each sample use the same text, but the relation between colors and objec...

    2025

  22. [42]

    and which form our set V (Sec. 3). Table 12: List of the syntactic categories defined in (Silveira et al., 2014). Syntactic Categories acomp advcl advmod amod appos aux auxpass cc ccomp conj cop csubj csubjpass dep det discourse dobj expl goeswith iobj mark mwe neg nn npadvmod...

    2014

  23. [224]

    Specifically, in the Trivial task, negative captions are randomly sampled from unrelated objects (of different images), offering a basic challenge for retrieval

    Then, each object image is associated with several false captions (on average, ten), selected based on the original FG-OVD Trivial, Easy, Medium, and Hard tasks (Bianchi et al., 2024). Specifically, in the Trivial task, negative captions are randomly sampled from unrelated obj...

    2024

  24. [1993]

    Verbs in action: Improving verb understanding in video-language models

    Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid. Verbs in action: Improving verb understanding in video-language models. In arXiv:2304.06708,

    arXiv

  25. [2004]

    Preserving multi- modal capabilities of pre-trained VLMs for improving vision-linguistic compositionality

    Youngtaek Oh, Jae Won Cho, Dong-Jin Kim, In So Kweon, and Junmo Kim. Preserving multi- modal capabilities of pre-trained VLMs for improving vision-linguistic compositionality. In arXiv:2410.05210,

    arXiv

  26. [2014]

    Text-to-image diffusion models are zero-shot classifiers

    Kevin Clark and Priyank Jaini. Text-to-image diffusion models are zero-shot classifiers. In arXiv:2303.15233,

    arXiv

  27. [2017]

    Contrastive region guidance: Improving grounding in vision-language models without training

    David Wan, Jaemin Cho, Elias Stengel-Eskin, and Mohit Bansal. Contrastive region guidance: Improving grounding in vision-language models without training. In arXiv:2403.02325,

    arXiv

  28. [2019]

    A generative dependency grammar

    Stefan Diaconescu. A generative dependency grammar. In PRICAI 2002: Trends in Artificial Intel- ligence,

    2002

  29. [2020]

    What do vision transformers learn? A visual ex- ploration

    Amin Ghiasi, Hamid Kazemi, Eitan Borgnia, Steven Reich, Manli Shu, Micah Goldblum, An- drew Gordon Wilson, and Tom Goldstein. What do vision transformers learn? A visual ex- ploration. In arXiv:2212.06727,

    arXiv

  30. [2021]

    Dense and aligned captions (DAC) promote compositional reasoning in VL models

    Sivan Doveh, Assaf Arbelle, Sivan Harary, Roei Herzig, Donghyun Kim, Paola Cascante-Bonilla, Amit Alfassy, Rameswar Panda, Raja Giryes, Rogerio Feris, Shimon Ullman, and Leonid Kar- linsky. Dense and aligned captions (DAC) promote compositional reasoning in VL models. In NeurI...

    2025 arXiv

  31. [2022]

    BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023b. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-traini...

    1904 arXiv

  32. [2023]

    Distilling knowledge from text-to-image generative models improves visio-linguistic reasoning in CLIP

    Samyadeep Basu, Shell Xu Hu, Maziar Sanjabi, Daniela Massiceti, and Soheil Feizi. Distilling knowledge from text-to-image generative models improves visio-linguistic reasoning in CLIP. In arXiv:2307.09233,

    arXiv

  33. [2024]

    A hierarchical quasi- recurrent approach to video captioning

    Federico Bolelli, Lorenzo Baraldi, Federico Pollastri, and Costantino Grana. A hierarchical quasi- recurrent approach to video captioning. In 2018 IEEE International Conference on Image Pro- cessing, Applications and Systems (IPAS),

    2018

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.