REVIEW 2 major objections 6 minor 45 references
Vision-language models route visual information through two pathways—direct from image tokens or via text tokens—and which one they use depends on task, data, and prompt.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-12 03:02 UTC pith:PNFWELCT
load-bearing objection Solid dual-pathway causal map that cleanly unifies prior conflicting VLM ablation results; the fallback finding is the real methodological payoff. the 2 major comments →
Pathways of Visual Information Flow in Vision-Language Models
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
VLMs solve visual queries by routing image information along two coexisting pathways: a direct pathway in which the last token reads from image-token representations at later layers, and a text-mediated pathway in which visual information is first transferred into query-token representations and then read out. Pathway choice is task-dependent and further modulated by data distribution and prompt design; under intervention the unused pathway can serve as a near-full-performance fallback, so ablations can reveal capacity rather than ordinary use.
What carries the argument
Causal (interchange) patching of residual-stream groups—image tokens, text query tokens, or the last token—measured by a restoration score that asks whether a counterfactual representation flips the answer toward the counterfactual label. Combined with attention knockouts and corrupted-image text patching, this separates usage under normal inference from necessity and sufficiency under intervention.
Load-bearing premise
That patching whole groups of image or text token representations, on pairs the model already answers correctly and (for natural data) built by horizontal flips of left/right scenes, cleanly measures the pathway the model uses in ordinary inference rather than an artifact of the intervention or the pair construction.
What would settle it
On the same recognition, relations, and localization tasks, run paired causal patching with counterfactuals that are not left/right flips (or not whole residual-stream groups) and check whether recognition still shows zero text restoration under open prompts while relations remain text-dominated; if the pathway pattern collapses or reverses under those controls, the claimed task-dependent routing is not established.
If this is right
- Interpretability claims based only on attention knockouts risk reporting fallback capacity rather than the default route, so usage-sensitive patching is required to map ordinary behavior.
- Test-time edits that only retune last-token attention to image tokens miss tasks already solved via text mediation; intervening on image-to-text transfer may matter more.
- Prompt format and data distribution are levers that can switch routing within the same task without changing accuracy, so mechanism studies must hold those fixed or report them.
- Because the text-mediated pathway alone can recover correct answers even with a noise image, the direct pathway is not necessary for the evaluated tasks and can sometimes act as a competing signal.
Where Pith is reading between the lines
- If text mediation is the flexible backup, training or decoding choices that strengthen early image-to-query transfer might make models more robust to image-token corruption while changing how failures look under ablation.
- The same dual-route pattern may appear in multi-step visual reasoning and captioning, but those settings could force simultaneous use of both pathways rather than a single dominant one.
- Head- and MLP-level circuits that implement the two routes, if isolated, would turn the group-level picture into editable components for targeted steering of spatial versus recognition behavior.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies how visual information reaches the final answer token in VLMs. Using causal (interchange) patching on paired original/counterfactual inputs, together with attention knockouts and corrupted-input patching, the authors argue that three open-weight VLMs (Qwen3-VL-4B, InternVL3.5-4B, LLaVA-1.5-7B) route visual information along two pathways: a direct pathway in which the last token reads image-token residual streams, and a text-mediated pathway in which visual information is first written into query-token residual streams and then read out. Across recognition, spatial relations, and localization, on both synthetic shapes and natural images, pathway selection is task-dependent and can be further modulated by data distribution and prompt format (open vs. choices). Under ablation of the direct route, models can fall back on text mediation with little or no accuracy loss, which the authors use to reconcile prior conflicting ablation studies and to distinguish pathway usage from pathway necessity.
Significance. If the dual-pathway account holds, it is a useful mechanistic contribution for VLM interpretability: it unifies prior work that had reported either direct readout or text mediation depending on the intervention, and it gives a concrete methodological warning that attention knockouts alone can report what a model can do rather than what it normally does. Strengths that support significance include (i) convergent evidence from causal patching, all-layer dominance tests, corrupted-input recovery, and knockouts; (ii) controlled synthetic data with feature-matched counterfactuals plus natural-image generalization; (iii) replication across three model families; and (iv) released code. The usage-vs-necessity distinction and the prompt/data modulation results are the most actionable pieces for subsequent intervention and circuit work.
major comments (2)
- §4.1–4.2 and Eq. (3): the restoration score is the primary usage metric, but for synthetic localization under open generation it is near zero while constrained-logit accuracy after text patching is 100% (Table 3; Appendix B.3). The appendix explanation (top-1 mass on “center”) is convincing, yet the main text still leans on RS curves (Fig. 4) without a clear rule for when RS and discrete accuracy diverge. Please either (a) report constrained-argmax / top-1 transfer rates alongside RS in the main figures for all tasks, or (b) state explicitly when RS is not a reliable usage measure so that pathway claims do not rest on a single scalar that can be driven by open-vocabulary mass outside the answer set.
- §3 and Appendix C.1: natural-data counterfactuals for relations and localization are constructed only by horizontal flips of left/right samples, with pairs filtered to those the model already answers correctly on both members. Synthetic data already supports the same task-dependent routing with cleaner feature-matched counterfactuals, so the central claim is not solely dependent on flips; still, the natural “text-mediated localization” result is load-bearing for the data-distribution claim. Please quantify how much of the natural localization/relations evidence survives under alternative counterfactuals (e.g., object swap, above/below where available) or, if that is infeasible, bound the claim more carefully to left/right orientation and correctly answered pairs rather than “natural localization” in full generality.
minor comments (6)
- Figure 4 vs. Table 2: the main narrative would be easier to follow if the all-layer text-vs-image dominance percentages (Table 2) were placed next to the layer-wise curves, or summarized in the figure caption, since the two analyses answer slightly different questions (competing sources vs. layer of transfer).
- §2, pathway definitions (Eq. 1): r_I→T is defined as image-to-text edges, but the text-mediated pathway as used in the paper is the composition r_I→T then r_T→last. A one-sentence clarification that “text-mediated” always means that composition would reduce ambiguity when knockouts of r_I→T and r_T→last are reported separately (Table 4).
- Table 1 / Table 4: no-image baselines are essential and well reported; consider also reporting chance relative to the actual answer-set size per dataset (especially VSR’s six-way set) so that residual accuracy after knockouts is easier to interpret.
- Appendix D: LLaVA shows text mediation on synthetic localization as well, unlike Qwen/InternVL. A short main-text note that pathway choice can also be model-dependent (not only task/data/prompt) would better match the appendix evidence.
- Typos / polish: “Adirect” / “atext-mediated” spacing in the abstract; “LLaV A” spacing throughout; “Generative Visual-Language Models” vs. the more standard “vision-language models” in the introduction.
- Limitations (§6) correctly flag single-token answers, three tasks, and group-level rather than head-level localization. Adding that natural recognition is a small paired COCO subset (N=146) would help readers weight that cell of the design.
Circularity Check
No significant circularity: dual pathways are operationally defined by causal interventions and measured empirically, not derived from fitted parameters or self-referential definitions.
full rationale
This is an empirical mechanistic-interpretability study. The two pathways (direct r_I o last and text-mediated r_I o T then r_T o last) are defined operationally as sets of attention edges / residual-stream groups (§2, Eq. 1) and then tested with interchange interventions (causal patching, restoration score Eq. 3), attention knockouts, and corrupted-input patching. Pathway selection is reported as an observed pattern across tasks, data sources, and prompt formats (Tables 2–4, Figs. 4–5), not as a quantity fitted on one subset and re-predicted on another. Filtering to pairs the model already answers correctly is standard for interchange interventions and does not force the dual-pathway conclusion by construction; synthetic counterfactuals that change only the task-relevant feature already exhibit the same task-dependent routing. Citations to prior work (Neo et al., Zhang et al., Kang et al., backup-circuit literature) are used for comparison and unification, not as load-bearing uniqueness theorems or ansatzes that smuggle in the result. No self-definitional loop, fitted-input-as-prediction, or renaming of a known closed-form result is present. Score 0 is therefore appropriate.
Axiom & Free-Parameter Ledger
axioms (4)
- domain assumption Causal interchange interventions on residual streams (restoration score, Eq. 3) measure which token groups the model actually reads under ordinary forward passes.
- domain assumption Attention knockouts that zero selected attention edges test necessity without destroying the rest of the computation in a way that invalidates the comparison.
- domain assumption Horizontal flips of left/right natural images produce valid counterfactuals that differ only on the spatial answer while preserving difficulty.
- domain assumption Restricting evaluation pairs to those the model answers correctly under clean inference does not bias pathway conclusions.
invented entities (2)
-
direct pathway (r_I→last)
independent evidence
-
text-mediated pathway (r_I→T then r_T→last)
independent evidence
read the original abstract
We study how visual information is routed in vision-language models (VLMs). Using causal patching on controlled synthetic and natural datasets, we find that models rely on two distinct pathways to solve visual tasks: A direct pathway, where visual information is retained in image token representations and read out by the final token at later layers, and a text-mediated pathway, where visual information is first transferred to the query tokens and then read out by the final token. Across three visual tasks, we show that pathway selection is task-dependent, and that data distribution and prompt design can also modulate which pathway is used to solve the image-based query. Moreover, using attention knockouts and corrupted-input patching, we find that these pathways are flexible, under certain interventions, models can rely on the text-mediated pathway as a fallback when the usual pathway is ablated. This behavior unifies findings in prior work and shows that ablation-based interventions can reveal what models could do rather than what they normally do. Together, our results provide a mechanistic characterization of visual information flow in VLMs and highlight the flexibility of their internal mechanisms under intervention.
Figures
Reference graph
Works this paper leans on
-
[1]
What's “up” with vision-language models?
Kamath, Amita and Hessel, Jack and Chang, Kai-Wei , booktitle=emnlp, year=. What's “up” with vision-language models?
-
[2]
Why is spatial reasoning hard for
Chen, Shiqi and Zhu, Tongyao and Zhou, Ruochen and Zhang, Jinghan and Gao, Siyang and Niebles, Juan Carlos and Geva, Mor and He, Junxian and Wu, Jiajun and Li, Manling , booktitle=icml, year=. Why is spatial reasoning hard for
-
[3]
Towards interpreting visual information processing in vision-language models , author=
-
[4]
Locating and editing factual associations in
Meng, Kevin and Bau, David and Andonian, Alex and Belinkov, Yonatan , booktitle=neurips, year=. Locating and editing factual associations in
-
[5]
What Do VLM s NOTICE ? A Mechanistic Interpretability Pipeline for G aussian-Noise-free Text-Image Corruption and Evaluation
Golovanevsky, Michal and Rudman, William and Palit, Vedant and Eickhoff, Carsten and Singh, Ritambhara. What Do VLM s NOTICE ? A Mechanistic Interpretability Pipeline for G aussian-Noise-free Text-Image Corruption and Evaluation. 2025
2025
-
[6]
Causal tracing of object representations in large vision language models: Mechanistic interpretability and hallucination mitigation , author=
-
[7]
Can Multimodal Large Language Models Understand Spatial Relations?
Liu, Jingping and Liu, Ziyan and Cen, Zhedong and Zhou, Yan and Zou, Yinan and Zhang, Weiyan and Jiang, Haiyun and Ruan, Tong. Can Multimodal Large Language Models Understand Spatial Relations?. 2025
2025
-
[8]
Zheyuan Zhang and Fengyuan Hu and Jayjun Lee and Freda Shi and Parisa Kordjamshidi and Joyce Chai and Ziqiao Ma , title =
-
[9]
Ryoo and Tsung
Kanchana Ranasinghe and Satya Narayan Shukla and Omid Poursaeed and Michael S. Ryoo and Tsung. Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs , booktitle = cvpr, year =
-
[10]
Fangyu Liu and Guy Emerson and Nigel Collier , title =
-
[11]
Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models , author=
-
[12]
Johnson, Justin and Hariharan, Bharath and Van Der Maaten, Laurens and Fei-Fei, Li and Lawrence Zitnick, C and Girshick, Ross , booktitle=cvpr, year=
-
[13]
Chen, Boyuan and Xu, Zhuo and Kirmani, Sean and Ichter, Brain and Sadigh, Dorsa and Guibas, Leonidas and Xia, Fei , booktitle=cvpr, year=. Spatial
-
[14]
What's left?
Hsu, Joy and Mao, Jiayuan and Tenenbaum, Josh and Wu, Jiajun , booktitle=neurips, year=. What's left?
-
[15]
Thinking in space: How multimodal large language models see, remember, and recall spaces , author=
-
[16]
Spatial-
Huang, Xinmiao and He, Qisong and Huang, Zhenglin and Wang, Boxuan and Li, Zhuoyun and Cheng, Guangliang and Dong, Yi and Huang, Xiaowei , booktitle=iclr, year=. Spatial-
-
[17]
Wasi, Azmine Toushik and Faisal, Wahid and Rahman, Abdur and Anik, Mahfuz Ahmed and Shahriar, Munem and Topu, Mohsin Mahmud and Meem, Sadia Tasnim and Priti, Rahatun Nesa and Mitu, Sabrina Afroz and Hoque, Md Iqramul and others , booktitle=iclr, year=. Spatia
-
[18]
Bai, Shuai and Cai, Yuxuan and Chen, Ruizhe and Chen, Keqin and Chen, Xionghui and Cheng, Zesen and Deng, Lianghao and Ding, Wei and Gao, Chang and Ge, Chunjiang and others , journal=. Qwen3-
-
[19]
Eyes wide shut?
Tong, Shengbang and Liu, Zhuang and Zhai, Yuexiang and Ma, Yi and LeCun, Yann and Xie, Saining , booktitle=cvpr, year=. Eyes wide shut?
-
[20]
arXiv preprint arXiv:2512.03276 , year=
Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval , author=. arXiv preprint arXiv:2512.03276 , year=
-
[21]
Cross-modal information flow in multimodal large language models , author=
-
[22]
arXiv preprint arXiv:2509.17588 , year=
Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Models , author=. arXiv preprint arXiv:2509.17588 , year=
-
[23]
Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens , author=
-
[24]
What's in the Image?
Kaduri, Omri and Bagon, Shai and Dekel, Tali , booktitle=cvpr, year=. What's in the Image?
-
[25]
The narrow gate: Localized image-text communication in native multimodal models , author=
-
[26]
Microsoft
Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll. Microsoft
-
[27]
2017 , publisher=
Visual genome: Connecting language and vision using crowdsourced dense image annotations , author=. 2017 , publisher=
2017
-
[28]
Causal abstraction: A theoretical foundation for mechanistic interpretability , author=
-
[29]
Interpretability in the Wild: a Circuit for Indirect Object Identification in
Kevin Ro Wang and Alexandre Variengien and Arthur Conmy and Buck Shlegeris and Jacob Steinhardt , booktitle=iclr, year=. Interpretability in the Wild: a Circuit for Indirect Object Identification in
-
[30]
arXiv preprint arXiv:2307.15771 , year=
The hydra effect: Emergent self-repair in language model computations , author=. arXiv preprint arXiv:2307.15771 , year=
-
[31]
Copy Suppression: Comprehensively Understanding a Motif in Language Model Attention Heads
McDougall, Callum Stuart and Conmy, Arthur and Rushing, Cody and McGrath, Thomas and Nanda, Neel. Copy Suppression: Comprehensively Understanding a Motif in Language Model Attention Heads. Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP. 2024. doi:10.18653/v1/2024.blackboxnlp-1.22
-
[32]
Wang, Weiyun and Gao, Zhangwei and Gu, Lixin and Pu, Hengjun and Cui, Long and Wei, Xingguang and Liu, Zhaoyang and Jing, Linglin and Ye, Shenglong and Shao, Jie and others , journal=. Intern
-
[33]
Takishita, Sho and Gala, Jay and Mohamed, Abdelrahman and Inui, Kentaro and Kementchedjhieva, Yova , booktitle=
-
[34]
Hidden in plain sight:
Fu, Stephanie and Bonnen, Tyler and Guillory, Devin and Darrell, Trevor , journal=. Hidden in plain sight:
-
[35]
Performance gap in entity knowledge extraction across modalities in vision language models , author=
-
[36]
Multimodal neurons in pretrained text-only transformers , author=
-
[37]
arXiv preprint arXiv:2506.11976 , year=
How visual representations map to language feature space in multimodal llms , author=. arXiv preprint arXiv:2506.11976 , year=
-
[38]
arXiv preprint arXiv:2603.22278 , year=
The Dual Mechanisms of Spatial Reasoning in Vision-Language Models , author=. arXiv preprint arXiv:2603.22278 , year=
-
[39]
Visual representations inside the language model , author=
-
[40]
To sink or not to sink: Visual information pathways in large vision-language models , author=
-
[41]
Understanding information storage and transfer in multi-modal large language models , author=
-
[42]
2023 , url=
Dissecting Recall of Factual Associations in Auto-Regressive Language Models , author=. 2023 , url=
2023
-
[43]
2024 , pages =
Liu, Haotian and Li, Chunyuan and Li, Yuheng and Lee, Yong Jae , title =. 2024 , pages =
2024
-
[44]
Constructive distortion: Improving mllms with attention-guided image warping , author=
-
[45]
Open Problems in Mechanistic Interpretability , author=
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.