Pith. sign in

REVIEW 2 major objections 6 minor 45 references

Vision-language models route visual information through two pathways—direct from image tokens or via text tokens—and which one they use depends on task, data, and prompt.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · grok-4.5

2026-07-12 03:02 UTC pith:PNFWELCT

load-bearing objection Solid dual-pathway causal map that cleanly unifies prior conflicting VLM ablation results; the fallback finding is the real methodological payoff. the 2 major comments →

arxiv 2607.03358 v1 pith:PNFWELCT submitted 2026-07-03 cs.CV cs.CLcs.LG

Pathways of Visual Information Flow in Vision-Language Models

classification cs.CV cs.CLcs.LG
keywords vision-language modelscausal patchinginformation flowattention knockoutsmechanistic interpretabilitydirect pathwaytext-mediated pathwayvisual grounding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks how visual information actually reaches the final answer token inside vision-language models. Using causal patching on synthetic shapes and natural images, it shows that models do not have a single fixed route: they can keep visual content in the image-token stream and read it out late (a direct pathway), or first copy it into the query text tokens and then read those (a text-mediated pathway). Which route is used under ordinary inference depends on the task—object recognition tends to be direct, spatial relations text-mediated, localization either depending on data complexity—and can also flip with prompt format. When the usual route is blocked, the other route can take over with little or no accuracy loss. The practical point is that ablation-style interventions often show what a model can do under stress, not what it normally does, so usage-sensitive methods are needed to map the default mechanism.

Core claim

VLMs solve visual queries by routing image information along two coexisting pathways: a direct pathway in which the last token reads from image-token representations at later layers, and a text-mediated pathway in which visual information is first transferred into query-token representations and then read out. Pathway choice is task-dependent and further modulated by data distribution and prompt design; under intervention the unused pathway can serve as a near-full-performance fallback, so ablations can reveal capacity rather than ordinary use.

What carries the argument

Causal (interchange) patching of residual-stream groups—image tokens, text query tokens, or the last token—measured by a restoration score that asks whether a counterfactual representation flips the answer toward the counterfactual label. Combined with attention knockouts and corrupted-image text patching, this separates usage under normal inference from necessity and sufficiency under intervention.

Load-bearing premise

That patching whole groups of image or text token representations, on pairs the model already answers correctly and (for natural data) built by horizontal flips of left/right scenes, cleanly measures the pathway the model uses in ordinary inference rather than an artifact of the intervention or the pair construction.

What would settle it

On the same recognition, relations, and localization tasks, run paired causal patching with counterfactuals that are not left/right flips (or not whole residual-stream groups) and check whether recognition still shows zero text restoration under open prompts while relations remain text-dominated; if the pathway pattern collapses or reverses under those controls, the claimed task-dependent routing is not established.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Interpretability claims based only on attention knockouts risk reporting fallback capacity rather than the default route, so usage-sensitive patching is required to map ordinary behavior.
  • Test-time edits that only retune last-token attention to image tokens miss tasks already solved via text mediation; intervening on image-to-text transfer may matter more.
  • Prompt format and data distribution are levers that can switch routing within the same task without changing accuracy, so mechanism studies must hold those fixed or report them.
  • Because the text-mediated pathway alone can recover correct answers even with a noise image, the direct pathway is not necessary for the evaluated tasks and can sometimes act as a competing signal.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If text mediation is the flexible backup, training or decoding choices that strengthen early image-to-query transfer might make models more robust to image-token corruption while changing how failures look under ablation.
  • The same dual-route pattern may appear in multi-step visual reasoning and captioning, but those settings could force simultaneous use of both pathways rather than a single dominant one.
  • Head- and MLP-level circuits that implement the two routes, if isolated, would turn the group-level picture into editable components for targeted steering of spatial versus recognition behavior.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper studies how visual information reaches the final answer token in VLMs. Using causal (interchange) patching on paired original/counterfactual inputs, together with attention knockouts and corrupted-input patching, the authors argue that three open-weight VLMs (Qwen3-VL-4B, InternVL3.5-4B, LLaVA-1.5-7B) route visual information along two pathways: a direct pathway in which the last token reads image-token residual streams, and a text-mediated pathway in which visual information is first written into query-token residual streams and then read out. Across recognition, spatial relations, and localization, on both synthetic shapes and natural images, pathway selection is task-dependent and can be further modulated by data distribution and prompt format (open vs. choices). Under ablation of the direct route, models can fall back on text mediation with little or no accuracy loss, which the authors use to reconcile prior conflicting ablation studies and to distinguish pathway usage from pathway necessity.

Significance. If the dual-pathway account holds, it is a useful mechanistic contribution for VLM interpretability: it unifies prior work that had reported either direct readout or text mediation depending on the intervention, and it gives a concrete methodological warning that attention knockouts alone can report what a model can do rather than what it normally does. Strengths that support significance include (i) convergent evidence from causal patching, all-layer dominance tests, corrupted-input recovery, and knockouts; (ii) controlled synthetic data with feature-matched counterfactuals plus natural-image generalization; (iii) replication across three model families; and (iv) released code. The usage-vs-necessity distinction and the prompt/data modulation results are the most actionable pieces for subsequent intervention and circuit work.

major comments (2)
  1. §4.1–4.2 and Eq. (3): the restoration score is the primary usage metric, but for synthetic localization under open generation it is near zero while constrained-logit accuracy after text patching is 100% (Table 3; Appendix B.3). The appendix explanation (top-1 mass on “center”) is convincing, yet the main text still leans on RS curves (Fig. 4) without a clear rule for when RS and discrete accuracy diverge. Please either (a) report constrained-argmax / top-1 transfer rates alongside RS in the main figures for all tasks, or (b) state explicitly when RS is not a reliable usage measure so that pathway claims do not rest on a single scalar that can be driven by open-vocabulary mass outside the answer set.
  2. §3 and Appendix C.1: natural-data counterfactuals for relations and localization are constructed only by horizontal flips of left/right samples, with pairs filtered to those the model already answers correctly on both members. Synthetic data already supports the same task-dependent routing with cleaner feature-matched counterfactuals, so the central claim is not solely dependent on flips; still, the natural “text-mediated localization” result is load-bearing for the data-distribution claim. Please quantify how much of the natural localization/relations evidence survives under alternative counterfactuals (e.g., object swap, above/below where available) or, if that is infeasible, bound the claim more carefully to left/right orientation and correctly answered pairs rather than “natural localization” in full generality.
minor comments (6)
  1. Figure 4 vs. Table 2: the main narrative would be easier to follow if the all-layer text-vs-image dominance percentages (Table 2) were placed next to the layer-wise curves, or summarized in the figure caption, since the two analyses answer slightly different questions (competing sources vs. layer of transfer).
  2. §2, pathway definitions (Eq. 1): r_I→T is defined as image-to-text edges, but the text-mediated pathway as used in the paper is the composition r_I→T then r_T→last. A one-sentence clarification that “text-mediated” always means that composition would reduce ambiguity when knockouts of r_I→T and r_T→last are reported separately (Table 4).
  3. Table 1 / Table 4: no-image baselines are essential and well reported; consider also reporting chance relative to the actual answer-set size per dataset (especially VSR’s six-way set) so that residual accuracy after knockouts is easier to interpret.
  4. Appendix D: LLaVA shows text mediation on synthetic localization as well, unlike Qwen/InternVL. A short main-text note that pathway choice can also be model-dependent (not only task/data/prompt) would better match the appendix evidence.
  5. Typos / polish: “Adirect” / “atext-mediated” spacing in the abstract; “LLaV A” spacing throughout; “Generative Visual-Language Models” vs. the more standard “vision-language models” in the introduction.
  6. Limitations (§6) correctly flag single-token answers, three tasks, and group-level rather than head-level localization. Adding that natural recognition is a small paired COCO subset (N=146) would help readers weight that cell of the design.

Circularity Check

0 steps flagged

No significant circularity: dual pathways are operationally defined by causal interventions and measured empirically, not derived from fitted parameters or self-referential definitions.

full rationale

This is an empirical mechanistic-interpretability study. The two pathways (direct r_I o last and text-mediated r_I o T then r_T o last) are defined operationally as sets of attention edges / residual-stream groups (§2, Eq. 1) and then tested with interchange interventions (causal patching, restoration score Eq. 3), attention knockouts, and corrupted-input patching. Pathway selection is reported as an observed pattern across tasks, data sources, and prompt formats (Tables 2–4, Figs. 4–5), not as a quantity fitted on one subset and re-predicted on another. Filtering to pairs the model already answers correctly is standard for interchange interventions and does not force the dual-pathway conclusion by construction; synthetic counterfactuals that change only the task-relevant feature already exhibit the same task-dependent routing. Citations to prior work (Neo et al., Zhang et al., Kang et al., backup-circuit literature) are used for comparison and unification, not as load-bearing uniqueness theorems or ansatzes that smuggle in the result. No self-definitional loop, fitted-input-as-prediction, or renaming of a known closed-form result is present. Score 0 is therefore appropriate.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 2 invented entities

Empirical ML interpretability paper. Load-bearing content is experimental design rather than free parameters or new physical entities. The two pathways are descriptive labels for observed residual-stream routes, not postulated mediators. Main background assumptions are the validity of interchange interventions / attention knockouts for usage vs necessity questions, and the adequacy of the three chosen visual tasks plus left/right flip counterfactuals.

axioms (4)
  • domain assumption Causal interchange interventions on residual streams (restoration score, Eq. 3) measure which token groups the model actually reads under ordinary forward passes.
    Invoked throughout §2–3; standard in mech-interp but not proved for VLMs; the paper’s own discussion notes that different interventions answer different questions.
  • domain assumption Attention knockouts that zero selected attention edges test necessity without destroying the rest of the computation in a way that invalidates the comparison.
    §2, Eq. 2; used to claim the direct pathway is unnecessary.
  • domain assumption Horizontal flips of left/right natural images produce valid counterfactuals that differ only on the spatial answer while preserving difficulty.
    Appendix C.1; accuracy on original vs flipped is similar, but other scene asymmetries could remain.
  • domain assumption Restricting evaluation pairs to those the model answers correctly under clean inference does not bias pathway conclusions.
    Standard patching practice stated in §2; still a selection step.
invented entities (2)
  • direct pathway (r_I→last) independent evidence
    purpose: Label for the observed route in which visual information stays in image-token representations until late-layer readout by the final token.
    Operational definition via patching and attention sets (Eq. 1); not a new physical object, but a paper-introduced mechanistic category.
  • text-mediated pathway (r_I→T then r_T→last) independent evidence
    purpose: Label for the observed route in which visual information is first written into query-token representations and later read by the final token.
    Same operational status as the direct pathway; falsifiable by the same patching/knockout suite.

pith-pipeline@v1.1.0-grok45 · 24996 in / 2927 out tokens · 29615 ms · 2026-07-12T03:02:08.691603+00:00 · methodology

0 comments
read the original abstract

We study how visual information is routed in vision-language models (VLMs). Using causal patching on controlled synthetic and natural datasets, we find that models rely on two distinct pathways to solve visual tasks: A direct pathway, where visual information is retained in image token representations and read out by the final token at later layers, and a text-mediated pathway, where visual information is first transferred to the query tokens and then read out by the final token. Across three visual tasks, we show that pathway selection is task-dependent, and that data distribution and prompt design can also modulate which pathway is used to solve the image-based query. Moreover, using attention knockouts and corrupted-input patching, we find that these pathways are flexible, under certain interventions, models can rely on the text-mediated pathway as a fallback when the usual pathway is ablated. This behavior unifies findings in prior work and shows that ablation-based interventions can reveal what models could do rather than what they normally do. Together, our results provide a mechanistic characterization of visual information flow in VLMs and highlight the flexibility of their internal mechanisms under intervention.

Figures

Figures reproduced from arXiv: 2607.03358 by Constanza Fierro, Dan Oneata, Desmond Elliott, Israfel Salazar, Stella Frank.

Figure 1
Figure 1. Figure 1: Illustration of the two pathways VLMs rely on for routing visual information. In the direct [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Task definitions and examples. We consider three visual tasks: object recognition, spatial [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Causal patching interventions. We construct paired examples consisting of an original and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Restoration score of Qwen3-VL-4B under causal patching at each layer for different [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Prompt format modulates pathway selection. Causal patching plot for object recognition comparing generation protocols, choices (colored) and open (gray; [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Corrupted patching. Text hidden representations are inserted in the infer￾ence that contains a random noise image [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Examples from the synthetic datasets. Each sample is paired with a counterfactual example [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Examples from the natural datasets. Each sample is paired with a counterfactual example [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Per-dataset causal patching restoration on recognition (Qwen3-VL-4B). Both synthetic and [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Per-dataset causal patching restoration on relations (Qwen3-VL-4B). Both synthetic and [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Per-dataset causal patching restoration on localization (Qwen3-VL-4B). Synthetic local [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Argmax outcomes under patching for open prompts. For each layer and patched group, [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Argmax outcomes under patching for choices prompts. When candidate answers are [PITH_FULL_IMAGE:figures/full_fig_p021_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Prompt-format modulation across tasks (Qwen3-VL-4B).Layer-wise restoration under the [PITH_FULL_IMAGE:figures/full_fig_p022_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Causal patching restoration on InternVL3.5-4B. [PITH_FULL_IMAGE:figures/full_fig_p023_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Causal patching restoration on LLaVA-1.5-7B. [PITH_FULL_IMAGE:figures/full_fig_p024_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 3 linked inside Pith

  1. [1]

    What's “up” with vision-language models?

    Kamath, Amita and Hessel, Jack and Chang, Kai-Wei , booktitle=emnlp, year=. What's “up” with vision-language models?

  2. [2]

    Why is spatial reasoning hard for

    Chen, Shiqi and Zhu, Tongyao and Zhou, Ruochen and Zhang, Jinghan and Gao, Siyang and Niebles, Juan Carlos and Geva, Mor and He, Junxian and Wu, Jiajun and Li, Manling , booktitle=icml, year=. Why is spatial reasoning hard for

  3. [3]

    Towards interpreting visual information processing in vision-language models , author=

  4. [4]

    Locating and editing factual associations in

    Meng, Kevin and Bau, David and Andonian, Alex and Belinkov, Yonatan , booktitle=neurips, year=. Locating and editing factual associations in

  5. [5]

    What Do VLM s NOTICE ? A Mechanistic Interpretability Pipeline for G aussian-Noise-free Text-Image Corruption and Evaluation

    Golovanevsky, Michal and Rudman, William and Palit, Vedant and Eickhoff, Carsten and Singh, Ritambhara. What Do VLM s NOTICE ? A Mechanistic Interpretability Pipeline for G aussian-Noise-free Text-Image Corruption and Evaluation. 2025

  6. [6]

    Causal tracing of object representations in large vision language models: Mechanistic interpretability and hallucination mitigation , author=

  7. [7]

    Can Multimodal Large Language Models Understand Spatial Relations?

    Liu, Jingping and Liu, Ziyan and Cen, Zhedong and Zhou, Yan and Zou, Yinan and Zhang, Weiyan and Jiang, Haiyun and Ruan, Tong. Can Multimodal Large Language Models Understand Spatial Relations?. 2025

  8. [8]

    Zheyuan Zhang and Fengyuan Hu and Jayjun Lee and Freda Shi and Parisa Kordjamshidi and Joyce Chai and Ziqiao Ma , title =

  9. [9]

    Ryoo and Tsung

    Kanchana Ranasinghe and Satya Narayan Shukla and Omid Poursaeed and Michael S. Ryoo and Tsung. Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs , booktitle = cvpr, year =

  10. [10]

    Fangyu Liu and Guy Emerson and Nigel Collier , title =

  11. [11]

    Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models , author=

  12. [12]

    Johnson, Justin and Hariharan, Bharath and Van Der Maaten, Laurens and Fei-Fei, Li and Lawrence Zitnick, C and Girshick, Ross , booktitle=cvpr, year=

  13. [13]

    Chen, Boyuan and Xu, Zhuo and Kirmani, Sean and Ichter, Brain and Sadigh, Dorsa and Guibas, Leonidas and Xia, Fei , booktitle=cvpr, year=. Spatial

  14. [14]

    What's left?

    Hsu, Joy and Mao, Jiayuan and Tenenbaum, Josh and Wu, Jiajun , booktitle=neurips, year=. What's left?

  15. [15]

    Thinking in space: How multimodal large language models see, remember, and recall spaces , author=

  16. [16]

    Spatial-

    Huang, Xinmiao and He, Qisong and Huang, Zhenglin and Wang, Boxuan and Li, Zhuoyun and Cheng, Guangliang and Dong, Yi and Huang, Xiaowei , booktitle=iclr, year=. Spatial-

  17. [17]

    Wasi, Azmine Toushik and Faisal, Wahid and Rahman, Abdur and Anik, Mahfuz Ahmed and Shahriar, Munem and Topu, Mohsin Mahmud and Meem, Sadia Tasnim and Priti, Rahatun Nesa and Mitu, Sabrina Afroz and Hoque, Md Iqramul and others , booktitle=iclr, year=. Spatia

  18. [18]

    Bai, Shuai and Cai, Yuxuan and Chen, Ruizhe and Chen, Keqin and Chen, Xionghui and Cheng, Zesen and Deng, Lianghao and Ding, Wei and Gao, Chang and Ge, Chunjiang and others , journal=. Qwen3-

  19. [19]

    Eyes wide shut?

    Tong, Shengbang and Liu, Zhuang and Zhai, Yuexiang and Ma, Yi and LeCun, Yann and Xie, Saining , booktitle=cvpr, year=. Eyes wide shut?

  20. [20]

    arXiv preprint arXiv:2512.03276 , year=

    Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval , author=. arXiv preprint arXiv:2512.03276 , year=

  21. [21]

    Cross-modal information flow in multimodal large language models , author=

  22. [22]

    arXiv preprint arXiv:2509.17588 , year=

    Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Models , author=. arXiv preprint arXiv:2509.17588 , year=

  23. [23]

    Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens , author=

  24. [24]

    What's in the Image?

    Kaduri, Omri and Bagon, Shai and Dekel, Tali , booktitle=cvpr, year=. What's in the Image?

  25. [25]

    The narrow gate: Localized image-text communication in native multimodal models , author=

  26. [26]

    Microsoft

    Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll. Microsoft

  27. [27]

    2017 , publisher=

    Visual genome: Connecting language and vision using crowdsourced dense image annotations , author=. 2017 , publisher=

  28. [28]

    Causal abstraction: A theoretical foundation for mechanistic interpretability , author=

  29. [29]

    Interpretability in the Wild: a Circuit for Indirect Object Identification in

    Kevin Ro Wang and Alexandre Variengien and Arthur Conmy and Buck Shlegeris and Jacob Steinhardt , booktitle=iclr, year=. Interpretability in the Wild: a Circuit for Indirect Object Identification in

  30. [30]

    arXiv preprint arXiv:2307.15771 , year=

    The hydra effect: Emergent self-repair in language model computations , author=. arXiv preprint arXiv:2307.15771 , year=

  31. [31]

    Copy Suppression: Comprehensively Understanding a Motif in Language Model Attention Heads

    McDougall, Callum Stuart and Conmy, Arthur and Rushing, Cody and McGrath, Thomas and Nanda, Neel. Copy Suppression: Comprehensively Understanding a Motif in Language Model Attention Heads. Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP. 2024. doi:10.18653/v1/2024.blackboxnlp-1.22

  32. [32]

    Wang, Weiyun and Gao, Zhangwei and Gu, Lixin and Pu, Hengjun and Cui, Long and Wei, Xingguang and Liu, Zhaoyang and Jing, Linglin and Ye, Shenglong and Shao, Jie and others , journal=. Intern

  33. [33]

    Takishita, Sho and Gala, Jay and Mohamed, Abdelrahman and Inui, Kentaro and Kementchedjhieva, Yova , booktitle=

  34. [34]

    Hidden in plain sight:

    Fu, Stephanie and Bonnen, Tyler and Guillory, Devin and Darrell, Trevor , journal=. Hidden in plain sight:

  35. [35]

    Performance gap in entity knowledge extraction across modalities in vision language models , author=

  36. [36]

    Multimodal neurons in pretrained text-only transformers , author=

  37. [37]

    arXiv preprint arXiv:2506.11976 , year=

    How visual representations map to language feature space in multimodal llms , author=. arXiv preprint arXiv:2506.11976 , year=

  38. [38]

    arXiv preprint arXiv:2603.22278 , year=

    The Dual Mechanisms of Spatial Reasoning in Vision-Language Models , author=. arXiv preprint arXiv:2603.22278 , year=

  39. [39]

    Visual representations inside the language model , author=

  40. [40]

    To sink or not to sink: Visual information pathways in large vision-language models , author=

  41. [41]

    Understanding information storage and transfer in multi-modal large language models , author=

  42. [42]

    2023 , url=

    Dissecting Recall of Factual Associations in Auto-Regressive Language Models , author=. 2023 , url=

  43. [43]

    2024 , pages =

    Liu, Haotian and Li, Chunyuan and Li, Yuheng and Lee, Yong Jae , title =. 2024 , pages =

  44. [44]

    Constructive distortion: Improving mllms with attention-guided image warping , author=

  45. [45]

    Open Problems in Mechanistic Interpretability , author=