Pith. sign in

REVIEW 4 major objections 6 minor 36 references

VLM failures on compositional VQA decompose into four mechanistically distinct modes — grounding, reasoning, attribute extraction, and language-prior dominance — each carried by a different internal pathway in the transformer.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 21:21 UTC pith:3GHMKW5Z

load-bearing objection Operation-aware failure taxonomy is useful, but the claimed pathway dissociation is contradicted by the paper's own Table 3. the 4 major comments →

arxiv 2607.16094 v1 pith:3GHMKW5Z submitted 2026-07-17 cs.CV

How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA

Navya Gupta , Bingjie Xu , Avinash Anand , Timothy Liu , Zhengchen Zhang This is my paper
classification cs.CV
keywords vision-language modelscompositional VQAfailure analysismechanistic interpretabilitycausal ablationattention knockoutfeedforward pathwayvisual grounding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Compositional VQA failures in vision-language models are not a single problem: the paper tries to establish that they decompose into four mechanistically distinct modes — grounding failure, reasoning failure, attribute extraction failure, and language-prior dominance — each defined by how visual grounding strength relates to answer correctness. Through layer-wise causal interventions on a 3-billion-parameter VLM, it claims a pathway dissociation: grounding failures travel through the feedforward (MLP) network, reasoning failures travel through late-layer attention, and attribute extraction failures localize to the feedforward computation at the answer position. If correct, the taxonomy gives concrete targets for corrective strategies, since each error type would need a different intervention. The paper further argues that the vision-sensitive failure modes depend on the vision encoder preserving spatial detail, while attribute-extraction and language-prior modes are architecture-independent.

Core claim

On Qwen2.5-VL, using GQA's scene graphs to locate the causally relevant object tokens, the paper reports that object selection ('select') failures appear when the model never develops causal reliance on the object, and that reliance lives in the feedforward computation: the answer-position MLP knockout is the only intervention whose degradation separates correct from incorrect answers. Spatial relation ('relate') failures show the opposite grounding signature — incorrect answers depend more on the object — and the separation is amplified by attention knockout in layers 28–35, showing the model over-routes visual content through attention. Attribute verification/query/existence failures show

What carries the argument

The framework's engine is a set of three causal interventions applied one transformer layer at a time on the same samples: causal mean ablation replaces the target object's vision-token hidden states with the mean vision activation to measure grounding strength; attention knockout sets the answer token's attention weights to the object tokens to zero (with renormalization); MLP knockout zeroes the feedforward output at either the object positions or the answer position. Stratifying each degradation curve by whether the model answered correctly and computing the correctness-split (Cohen's d) across 36 layers assigns every GQA operation to one of four failure modes. The comparison across the t

Load-bearing premise

The analysis assumes that zeroing the MLP output at the answer or object positions isolates the feedforward pathway without disturbing other computation, and that GQA scene-graph boxes align with the model's vision tokens so the ablations remove object-specific information; the paper itself flags that it tests a single 3B model on one benchmark, that mean ablation shows necessity but not sufficiency, and that high-accuracy operations have limited incorrect samples.

What would settle it

Run the same three layer-wise interventions on a second VLM with a native-resolution vision encoder. If the select operation's answer-position MLP knockout no longer shows a strong negative correctness split, or if the relate operation's late-layer attention knockout no longer shows the reversed grounding–correctness signal, the pathway dissociation is model-specific rather than general. A more direct check: zero the answer-position MLP on failed select questions and see whether the log-probability of the correct answer drops below baseline; if it does not, the feedforward attribution is an ar

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If grounding failures are feedforward-mediated, then fixing them requires interventions in MLP computation (e.g., at the answer position), not attention routing.
  • If reasoning failures are late-layer attention over-routing, then corrections should target attention at layers roughly 28–35 to stop over-grounding.
  • The VSR result implies spatial reasoning errors occur at object encoding (object-position MLP), distinct from attribute errors at answer decoding, so spatial fixes belong earlier in the network.
  • Because vision-sensitive failure modes only emerge when the vision encoder preserves spatial granularity, models with compressed encoders (like a frozen CLIP) will not show these grounding/reasoning failures; a native-resolution encoder is the right testbed.
  • Since language-prior operations show null causal profiles, no visual intervention will help them; they need question-structure changes or prior-breaking training.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The answer-position MLP knockout that singles out grounding failures suggests a testable repair: swap in the clean-pass MLP output at the answer position on failed select questions and measure whether errors flip to correct; the paper stops at diagnosis, so this is an inference.
  • Because the paper tests only two architectures at one scale, an open question is whether the pathway dissociation reflects a general transformer property or a quirk of these specific models; I would expect it to hold for other decoder-only VLMs, but that is my extrapolation.
  • If the dissociation holds across scales, failure-mode classification becomes a cheap diagnostic: run the three knockouts on a handful of examples per operation to decide whether a model's errors are perceptual, relational, or decoding failures — a recipe for model selection and targeted data collection.
  • The paper's own limitation section says the contribution is diagnostic rather than corrective; converting the location of an error into a working fix remains a substantial open step.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes an operation-centric mechanistic framework for VLM failures on GQA. It decomposes questions into seven GQA operation types, uses scene-graph bounding boxes to identify causally relevant vision tokens, and applies three interventions—causal mean ablation, attention knockout, and MLP knockout—across all 36 layers of Qwen2.5-VL-3B. From correctness-stratified Cohen's d values, it derives four failure modes (grounding, reasoning, attribute extraction, language prior dominance) and claims a pathway dissociation: grounding and attribute extraction failures route through feedforward MLPs, reasoning failures through late-layer attention. It reports VSR generalization and LLaVA-1.5 cross-architecture analyses.

Significance. If the pathway dissociation were supported, the taxonomy would be practically valuable: different failure types would require different corrective mechanisms (FFN-level vs attention-level interventions), and the operation-aware token targeting would be a useful tool for failure localization. The paper deserves credit for using functional program annotations and scene graphs, for computing layer-wise intervention curves on a real VLM, and for including cross-benchmark and cross-architecture checks plus an explicit limitations section. However, the central claim is not supported by the reported data; the manuscript's own Table 3 contains sign reversals that contradict the exclusivity claims. The framework and data are therefore suggestive but not yet a validated mechanistic account.

major comments (4)
  1. [Table 3; §5.2] The central 'pathway dissociation' claim is not supported by the reported statistics. For select, the operation used to establish grounding failure, mean ablation gives d=+0.75 (correct answers more dependent on object tokens), while answer-position MLP knockout gives d=-0.80 (incorrect answers more dependent on the MLP), with attention KO null. If grounding failures 'route exclusively through the feedforward network,' the correctness-discriminating signal of the MLP intervention should have the same sign as the total-effect ablation; it has the opposite sign. The statement in §5.2 that the full-pathway gap is 'fully accounted for by the feedforward pathway' is therefore contradicted by Table 3. Similar sign reversals occur for verify (d_abl=+0.12, d_mlp=-0.71) and exist (d_abl=+0.30, d_mlp=-0.27). No decomposition or model is provided that reconciles these sign flips. Since this is the
  2. [§3.4, Eq. (6)] The MLP knockout is not a clean pathway isolate. Zeroing the MLP output at position i changes the residual stream at layer ℓ, which is the input to layer ℓ+1's attention as well as to the MLP; it can also alter LayerNorm statistics. The paper asserts that attention routing is 'left intact,' but only the attention weights at the intervened layer are unchanged; the values routed by subsequent attention are different. Without control ablations (e.g., replacing the MLP output with a random vector in the same subspace, scaling rather than zeroing, or checking that attention logits are unaffected), the attribution of the degradation to the feedforward pathway rather than to a generic perturbation is not established. This is especially important because Eq. (6) is the only evidence for the FFN-specific claims.
  3. [§3.2, Eqs. (3)-(5)] The mapping from scene-graph bounding boxes to vision token positions B_t is assumed but not validated. GQA boxes are in pixel coordinates; Qwen2.5-VL uses native-resolution patches, but the paper does not discuss how boxes are snapped to tokens, how overlapping boxes are handled, or whether the answer token actually attends to the identified tokens. The mean-ablation baseline (Eq. 4) averages over all vision tokens; at early layers this may not be a neutral in-distribution point and can introduce artifacts. At minimum, provide a token-overlap sanity check and compare against ablation of random or adjacent vision-token subsets to show the operation-specific effect is not an artifact of token selection.
  4. [§3.2 and §5] The failure taxonomy is circular as presented. The four modes are defined by the sign and significance of the same Cohen's d values that are then reported as evidence for the modes. For example, 'grounding failure' is defined as substantial positive mean-ablation d, and select is assigned to that mode because d=+0.75; 'reasoning failure' is defined as negative d, and relate is assigned because d=-0.17. The VSR and LLaVA analyses provide some external grounding, but the central GQA classification remains a post hoc labeling of the primary outcome. Please specify the decision rule before the experiment or validate the taxonomy on a hold-out set of operations.
minor comments (6)
  1. [Figure 3 vs Table 3] Figure 3 reports dmlp=-0.81 for select while Table 3 reports -0.80; align the two.
  2. [Table 3] Table 3 headers are ambiguous: the pairs 'KO ✓ KO× d' after Attn KO and MLP KO lack a header distinguishing which intervention each pair refers to. Please clarify.
  3. [References] The reference for Qwen2.5-VL-3B-Instruct is [3], a Qwen3-VL technical report; update to the correct model card/release.
  4. [§5.6 / Table 4] 'MLP(object)' in Table 4 is not defined in the method section; specify that it is the object-position knockout from Eq. (6).
  5. [§4, Metrics] For high-accuracy operations (verify, choose, filter), the number of incorrect samples is small by construction; report exact n and confidence intervals for Cohen's d.
  6. [Abstract vs Limitations] The Limitations section is honest, but the abstract's 'exclusively' and 'demonstrate' are stronger than the acknowledged limitations (necessity-only, single model, diagnostic). Please temper the abstract accordingly.

Circularity Check

0 steps flagged

No significant circularity; taxonomy is a defined classification scheme and pathway claims are empirical, though internally inconsistent.

full rationale

The paper's four failure modes are introduced in §3.2 as predefined categories based on the relationship between grounding strength and answer correctness (positive d, negative d, null-high, null-low). Table 3 then assigns operations to these categories using the same GSS statistic, so the taxonomy is a descriptive classification scheme rather than a derived prediction. The central pathway-dissociation claim is not definitional: attention knockout and MLP knockout are distinct interventions introduced in §3.4, and the claimed routing of grounding failures through feedforward networks, reasoning failures through late-layer attention, and attribute-extraction failures through answer-position MLPs is inferred from the joint pattern of these independent interventions, not baked into the definitions of the modes. There are no load-bearing self-citations; all key references are to external prior work, and the VSR benchmark and LLaVA comparison provide independent cross-checks. The sign inconsistency for select (mean-ablation d=+0.75 vs. MLP-knockout d=-0.80) and similar discrepancies are empirical correctness/validity concerns about the exclusivity claim, not circularity: the paper's own equations do not force the conclusion, and the conclusion could in principle have been falsified by the data. The acknowledged limitations (single model, necessity-only ablation, diagnostic scope) further indicate that the claims are empirical rather than tautological. No circular step reduces an output to an input by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The central claim leans on the intervention logic rather than fitted coefficients. The main assumptions are (a) GQA programs/scene graphs correctly identify causally relevant tokens, (b) the mean-vision baseline is neutral, and (c) MLP knockout isolates feedforward visual computation. These are domain assumptions specific to this paper.

axioms (3)
  • domain assumption GQA functional programs and scene graphs correctly decompose each question into typed operations with ground-truth bounding boxes for each step.
    Used in §3.2 to define operation-aware grounding sets B_t; if programs are noisy or boxes misalign with tokens, interventions target wrong positions.
  • domain assumption The mean activation over all vision tokens μ_V is a neutral, in-distribution baseline whose replacement removes only the target object's information.
    Eq. (4) in §3.3; no validation that this baseline doesn't introduce off-target degradation.
  • ad hoc to paper Zeroing MLP outputs at object/answer positions isolates the feedforward pathway and leaves attention routing intact.
    Eq. (6) in §3.4 zeroes the entire MLP residual contribution; the paper provides no control to separate visual transformation from other MLP computation.

pith-pipeline@v1.3.0-alltime-deepseek · 13382 in / 14623 out tokens · 122289 ms · 2026-08-01T21:21:31.583237+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA." pith.science (2026). https://pith.science/paper/3GHMKW5Z

@misc{pith2026260716094,
  author       = {Pith},
  title        = {Pith review of: How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3GHMKW5Z}},
  note         = {Machine review of arXiv:2607.16094}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Compositional visual question answering requires Vision-Language Models (VLMs) to execute multiple reasoning operations like object selection, spatial relation resolution, and attribute verification. Despite strong aggregate performance, the mechanistic basis of VLM failures on this task remains underexplored. To address this gap, we analyze vision-operation misalignment in VLMs by examining how failures relate to specific reasoning operations and the internal computational pathways through which they arise and propagate. We introduce an Operation-centric mechanistic framework that decomposes VLM failures by both the reasoning operation where they originate and the internal computational pathway through which they propagate. Our analysis reveals four mechanistically distinct failure modes: grounding failure, reasoning failure, attribute extraction failure, and language prior dominance failure. Each characterized by a unique relationship between visual grounding strength and answer correctness. Through three complementary causal interventions applied across all transformer layers, we further demonstrate a pathway dissociation: grounding failures route exclusively through the feedforward network, reasoning failures route through late-layer attention, and attribute extraction failures localize to the answer-position feedforward computation. This dissociation demonstrates that different failure types require fundamentally different corrective strategies, providing a principled foundation for targeted improvements to VLM reliability in multimedia reasoning.

Figures

Figures reproduced from arXiv: 2607.16094 by Avinash Anand, Bingjie Xu, Navya Gupta, Timothy Liu, Zhengchen Zhang.

Figure 1
Figure 1. Figure 1: Four failure modes in compositional VQA, each [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Layer-wise causal interventions for object-specific reasoning. At layer [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Four mechanistically distinct failure modes. Layer [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Layer-wise attention knockout difference ( [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 7 linked inside Pith

  1. [1]

    Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018. Don’t just assume; look and answer: Overcoming priors for visual question answering. InProceedings of the IEEE conference on computer vision and pattern recognition. 4971–4980

    2018

  2. [2]

    Guillaume Alain and Yoshua Bengio. 2016. Understanding intermediate layers using linear classifier probes.arXiv preprint arXiv:1610.01644(2016)

    Pith/arXiv arXiv 2016

  3. [3]

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al . 2025. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631(2025)

    Pith/arXiv arXiv 2025

  4. [4]

    Samyadeep Basu, Martin Grayson, Cecily Morrison, Besmira Nushi, Soheil Feizi, and Daniela Massiceti. 2024. Understanding information storage and transfer in multi-modal large language models.Advances in Neural Information Processing Systems37 (2024), 7400–7426

    2024

  5. [5]

    Yonatan Belinkov. 2022. Probing Classifiers: Promises, Shortcomings, and Ad- vances.Computational Linguistics48, 1 (March 2022), 207–219. doi:10.1162/coli_ a_00422

    doi:10.1162/coli_ 2022

  6. [6]

    Gabriela Ben Melech Stan, Estelle Aflalo, Raanan Yehezkel Rohekar, Anahita Bhiwandiwalla, Shao-Yen Tseng, Matthew Lyle Olson, Yaniv Gurwicz, Chenfei Wu, Nan Duan, and Vasudev Lal. 2024. Lvlm-intrepret: An interpretability tool for large vision-language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8182–8187

    2024

  7. [7]

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Das- Sarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah....

    2021

  8. [8]

    Javier Ferrando and Elena Voita. 2024. Information Flow Routes: Automatically Interpreting Language Models at Scale. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 17432–17445. doi:10.18653/v1...

    doi:10.18653/v1/2024.emnlp-main.965 2024

  9. [9]

    Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. Dissect- ing Recall of Factual Associations in Auto-Regressive Language Models. InPro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computa- tional Linguistics, Singapore, 12216–...

    doi:10.18653/v1/2023.emnlp-main.751 2023

  10. [10]

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer Feed-Forward Layers Are Key-Value Memories. InProceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, Online and Punta Cana...

    doi:10.18653/v1/2021.emnlp-main.446 2021

  11. [11]

    Michal Golovanevsky, William Rudman, Vedant Palit, Ritambhara Singh, and Carsten Eickhoff. 2024. What do vlms notice? a mechanistic interpretability pipeline for gaussian-noise-free text-image corruption and evaluation.arXiv preprint arXiv:2406.16320(2024)

    Pith/arXiv arXiv 2024

  12. [12]

    Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real- world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 6700–6709

    2019

  13. [13]

    Yanbei Jiang, Xueqi Ma, Shu Liu, Sarah Monazam Erfani, Tongliang Liu, James Bailey, Jey Han Lau, and Krista A Ehinger. 2025. Investigating The Functional Roles of Attention Heads in Vision Language Models: Evidence for Reasoning Modules.arXiv preprint arXiv:2512.10300(2025)

    arXiv 2025

  14. [14]

    Jinyeong Kim, Seil Kang, Jiwoo Park, Junhyeok Kim, and Seong Jae Hwang

  15. [15]

    Qiming Li, Zekai Ye, Xiaocheng Feng, Weihong Zhong, Weitao Ma, and Xiachong Feng. 2025. Causal tracing of object representations in large vision language models: Mechanistic interpretability and hallucination mitigation.arXiv preprint arXiv:2511.05923(2025)

    arXiv 2025

  16. [16]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. InProceedings of the 2023 conference on empirical methods in natural language processing. 292–305

    2023

  17. [17]

    Zihao Lin, Samyadeep Basu, Mohammad Beigi, Varun Manjunatha, Ryan A Rossi, Zichao Wang, Yufan Zhou, Sriram Balasubramanian, Arman Zarei, Keivan Rezaei, et al. 2025. A survey on mechanistic interpretability for multi-modal foundation models.arXiv preprint arXiv:2502.17516(2025)

    Pith/arXiv arXiv 2025

  18. [18]

    Fangyu Liu, Guy Emerson, and Nigel Collier. 2023. Visual spatial reasoning. Transactions of the Association for Computational Linguistics11 (2023), 635–651

    2023

  19. [19]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual in- struction tuning.Advances in neural information processing systems36 (2023), 34892–34916

    2023

  20. [20]

    Yiming Liu*, Yuhui Zhang*, and Serena Yeung-Levy. 2025. Mechanistic Interpretability Meets Vision Language Models: Insights and Limitations. InICLR Blogposts 2025(April 28, 2025). https://d2jud02ci9yv69.cloudfront.net/2025-04- 28-vlm-understanding-29/blog/vlm-understanding/#activation-patching https://d2jud02ci9yv69.cloudfront.net/2025-04-28-vlm-understan...

    2025

  21. [21]

    Tiange Luo, Ang Cao, Gunhee Lee, Justin Johnson, and Honglak Lee. 2024. Prob- ing visual language priors in vlms.arXiv preprint arXiv:2501.00569(2024)

    Pith/arXiv arXiv 2024

  22. [22]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt.Advances in neural information processing systems35 (2022), 17359–17372

    2022

  23. [23]

    Lachin Naghashyar, Hunar Batra, Ashkan Khakzar, Philip Torr, Ronald Clark, Christian Schroeder de Witt, and Constantin Venhoff. 2026. Towards Understand- ing Multimodal Fine-Tuning: Spatial Features.arXiv preprint arXiv:2602.08713 (2026)

    arXiv 2026

  24. [24]

    Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez

  25. [25]

    Vedant Palit, Rohan Pandey, Aryaman Arora, and Paul Pu Liang. 2023. Towards vision-language mechanistic interpretability: A causal tracing tool for blip. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2856– 2861

    2023

  26. [26]

    Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Iacer Calixto, and Albert Gatt. 2022. VALSE: A task-independent benchmark for vision and language models centered on linguistic phenomena. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 8253–8280

    2022

  27. [27]

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object Hallucination in Image Captioning. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (Eds.). Association for Computational Linguistics, Brussels, Belgium,...

    doi:10.18653/v1/d18- 2018

  28. [28]

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022. Winoground: Probing vision and lan- guage models for visio-linguistic compositionality. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5238–5248

    2022

  29. [29]

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes wide shut? exploring the visual shortcomings of multimodal llms. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 9568–9578

    2024

  30. [30]

    Zeping Yu and Sophia Ananiadou. 2024. Understanding multimodal llms: the mechanistic interpretability of llava in visual question answering.arXiv preprint arXiv:2411.10950(2024)

    Pith/arXiv arXiv 2024

  31. [31]

    Di Zhang, Jingdi Lei, Junxian Li, Xunzhi Wang, Yujie Liu, Zonglin Yang, Jiatong Li, Weida Wang, Suorong Yang, Jianbo Wu, et al . 2025. Critic-v: Vlm critics help catch vlm errors in multimodal reasoning. InProceedings of the IEEE/CVF Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Trovato et al. Conference on Computer Vision and Pattern Recognitio...

    2025

  32. [32]

    Fred Zhang and Neel Nanda. 2024. Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=Hf17y6u9BC

    2024

  33. [33]

    Yuhui Zhang, Alyssa Unell, Xiaohan Wang, Dhruba Ghosh, Yuchang Su, Ludwig Schmidt, and Serena Yeung-Levy. 2024. Why are visually-grounded language models bad at image classification?Advances in Neural Information Processing Systems37 (2024), 51727–51753

    2024

  34. [34]

    Zhi Zhang, Srishti Yadav, Fengze Han, and Ekaterina Shutova. 2025. Cross- modal Information Flow in Multimodal Large Language Models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 19781–19791

    2025

  35. [2024]

    Towards interpreting visual information processing in vision-language models.arXiv preprint arXiv:2410.07149(2024)

    Pith/arXiv arXiv 2024

  36. [2025]

    Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Models.arXiv preprint arXiv:2509.17588(2025)

    arXiv 2025