Pith. sign in

REVIEW 3 major objections 4 minor 54 references

Pretrained vision-language models recode object-level visual representations in response to language, via a goal-referent tag and selective attribute amplification.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Goal-directed language prompts make VLMs add a transferable goal-relevant marker to selected image objects and amplify those objects' queried attributes in later layers, and both effects causally influence answers.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection The causal steering result is plausible but rests on an unstated split between steering-vector construction and evaluation, so the paper needs a reporting fix before I'd trust the central claim. the 3 major comments →

arxiv 2608.00035 v1 pith:WBCCLB2Q submitted 2026-07-21 cs.AI cs.CV

Linguistic Context Recodes Visual Representations in Vision-Language Models

Brian Song , Michael A. Lepori , Ellie Pavlick This is my paper
classification cs.AI cs.CV
keywords vision-language modelsvisual recodingreference representationsattribute modulationcontrastive steeringlinear probingcausal interventiongoal-directed language
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that when a VLM sees an image plus a goal-directed language query, it does not merely read static visual features; it actively rewrites the vision tokens. Two concrete rewrites are identified. First, tokens of the goal-relevant object receive an abstract, linearly decodable 'reference' direction that marks it as the target; adding or removing this direction flips correct answers on counting, yes/no, and spatial tasks, and directions trained on synthetic images transfer to natural images. Second, later layers amplify the target object's shape/color directions while suppressing others, and freezing the target token's hidden states reduces answer confidence. If correct, visual representations in VLMs are dynamically task-dependent rather than passive stores.

Core claim

The central claim is that linguistic input causes pretrained VLMs to recode object-level visual representations in support of the language-specified goal. The paper supplies two mechanisms: (1) an abstract reference representation — a direction in activation space that is added to vision tokens of goal-relevant objects and is causally sufficient, since contrastive steering flips correct outputs across counting, yes/no, and spatial tasks and generalizes from synthetic to natural images; (2) attribute modulation — in later layers, projections of the referent object's tokens onto its shape/color concept vectors increase relative to non-referents and absent controls, and freezing the referent's

What carries the argument

The load-bearing objects are: (a) contrastive steering vectors — averaged differences between hidden states of referent versus non-referent objects, used to add or subtract a 'reference' direction; (b) visual concept vectors — mean hidden states of objects with an attribute minus the global mean, used to project and measure attribute encoding; (c) per-layer linear probes; and (d) freezing and logit-lens interventions. Together they let the paper detect, isolate, and causally test recoding.

Load-bearing premise

The paper assumes that its token-to-object assignments — 4x4 patch grids in synthetic images and COCO masks in natural images — isolate one object's representation cleanly; if tokens bleed across objects, the reference and attribute effects could be inflated.

What would settle it

Take the same synthetic stimuli but misalign objects from the 4x4 patch grid (e.g., shift by one patch or let objects overlap), then retrain probes and steering vectors. If linear decodability and intervention success drop to the random-vector baseline, the mechanism depends on artificial token alignment rather than genuine recoding. Alternatively, find a prompt pair with the same referent labels but no contrasting attribute present in the image; if the contrastive vector still causes flips, the effect may partly reflect prompt-driven lexical changes rather than visual recoding.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • The same reference direction can be extracted on one task and applied to another, implying a generic 'this is the target' code rather than task-specific circuitry.
  • Reference steering vectors trained on simple synthetic scenes flip predictions on natural COCO images, so the mechanism is not a synthetic-data artifact.
  • Attribute amplification is spatially localized to the referent object and appears in late layers, after the reference signal appears; this suggests a two-stage goal-directed pipeline.
  • Freezing the referent token's hidden states reduces confidence in the correct attribute, making attribute modulation causally implicated in the final answer distribution.
  • The effects hold across two open VLM architectures, indicating the phenomenon is not unique to a single model.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reference direction may be a cross-modal analogue of entity-binding or working-memory pointers studied in language-only transformers; testing whether the same vector appears with coreference or anaphora in pure text would connect the two literatures.
  • Because steering vectors transfer from synthetic to natural images, they could serve as a read-out for 'goal-relevance' during reasoning, or as a diagnostic for when a VLM attends to the wrong object.
  • A concrete extension: measure whether the reference direction is added via attention writes from text tokens to vision tokens; if so, masking those attention heads should abolish both the probe and the steering effect.
  • The counting task's lower success rate (60%) suggests reference tagging may be one of several parallel mechanisms; a testable prediction is that counting failures occur when the number of tagged objects exceeds the model's simultaneous binding capacity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper argues that pretrained VLMs dynamically recode object-level visual representations under goal-directed language. Using Qwen2.5-VL (and InternVL3 as replication) on synthetic patch-aligned images and a small filtered COCO set, it reports two mechanisms: (1) a linearly decodable, causally sufficient 'reference representation' that marks goal-relevant objects and transfers across counting, yes/no, and spatial tasks and from synthetic to natural images; and (2) later-layer 'attribute modulation' that selectively amplifies goal-relevant shape/color features, with freezing and concept-vector interventions suggesting a causal role in the model's answer confidence. The conclusion is that vision tokens are not static repositories but are recoded to support language-specified goals.

Significance. The claim, if established, is significant: it moves beyond the common 'static visual repository' view of VLM vision tokens and offers a concrete, testable mechanistic account with two separable components. The paper has real strengths: controlled synthetic stimuli aligned to a patch grid, multiple task types, causal steering with random-vector controls and coefficient sweeps, a second-model replication in Appendix D, and a logit-gap readout for causal intervention. The cross-task and synthetic-to-natural transfer results are potentially noteworthy. However, the central causal claims currently rest on methodological clarifications that are not provided, so the manuscript needs revision before the strength of the evidence can be assessed.

major comments (3)
  1. [§4.3, with relevance to §4.2 and §F.1] The same issue applies to probe training in §4.2 and to concept-vector averaging in §F.1 relative to the 5000-image projection set. Please specify the exact split and, if any overlap exists, rerun with fully disjoint sets.
  2. [§5.3, Eqs. (1)–(2), Fig. 9] The Appendix H concept-vector intervention is better targeted, but the main text should rely on it or add a control (e.g., freezing a non-referent token, or freezing at a layer before modulation emerges) to support the claim that attribute amplification specifically mediates the response distribution.
  3. [§5.2, §F.4, Fig. 8 (bottom)] The absent-attribute controls do not rule this out. Please test whether V_reference is orthogonal to the concept vectors, or partial out the reference direction from hidden states before computing projections, or otherwise demonstrate that the modulation effect is not carried by the reference signal.
minor comments (4)
  1. [Appendix E, Fig. 18] Report intervention success rates as a function of coefficient for each vector type at matched coefficient values, or justify why the cap does not favor task-specific vectors.
  2. [§3, Natural Image Dataset] Please report the distribution of mask areas, the number of vision tokens per object, and ideally a small human-annotated token-mask IoU or a manual-error analysis.
  3. [Figures 4–5] Given the 100-image natural set and the per-task differences, bootstrap CIs would substantially strengthen the comparison against random-vector baselines.
  4. [§6 / Limitations] These should be added.

Circularity Check

0 steps flagged

No significant circularity; central claims are empirical interventions with independent controls.

full rationale

The paper does not reduce its predictions to its inputs by construction. The reference representation is extracted as an average contrastive difference between referent and non-referent hidden states, and its causal role is tested by intervention against random-vector baselines, with hyperparameters selected on a validation set and evaluated on a test set. The attribute modulation analysis constructs concept vectors from text-free images and measures prompt-dependent projections, including absent-attribute controls, and the freezing and concept-vector interventions are ablations rather than fitted quantities. The only self-citations (Lepori et al. 2024; Tartaglini and Lepori 2026) are background inspiration and are not load-bearing. The skeptical concern that steering-vector construction images may overlap with intervention evaluation images is a reporting/validity question, but the paper does not state that overlap, and no quoted passage exhibits the specific reduction required to call the result circular. The derivation chain is self-contained relative to the claims made.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 1 invented entities

The central claims rest on three measurement assumptions (object-token alignment, linear-probe/steering faithfulness, intervention specificity) plus one free tuning parameter (steering coefficient/layer block). No new physical entities are posited; the 'reference representation' is an internal activation direction with a falsifiable behavioral handle.

free parameters (1)
  • Steering coefficient α and contiguous layer block per task/vector = not reported in main text; validation sweeps in Appendix E
    Chosen per task/model/dataset on a validation set by grid search; evaluation then caps coefficients at the level where random vectors begin to succeed. These choices are tuning parameters of the intervention, not predictions, but they affect reported success rates.
axioms (4)
  • domain assumption Vision-token hidden states can be assigned cleanly to individual objects (4×4 patch grid for synthetic; COCO masks for natural).
    All object-level probes, steering vectors, and concept projections concatenate or mask vision tokens assumed to belong to one object. If token assignments mix objects, the reference and attribute-modulation signals are contaminated. Entered in Section 3.
  • domain assumption Linear classifiers and contrastive average differences are faithful probes of causal mechanism.
    The paper interprets linear decodability of referent/non-referent and causal flipping via steering vectors as evidence of a model-implemented reference representation. This is a standard interpretability assumption; controls (random vectors, description prompts) partially support it.
  • domain assumption Adding or freezing hidden states at object token positions intervenes on the target mechanism without unrelated degradation.
    Causal claims in Sections 4.3 and 5.3 rely on steering and freezing being mechanism-specific. Random-vector baselines and concept-vector localization help, but cannot fully rule out generic disruption; freezing is especially underspecified.
  • domain assumption The logit lens gives a meaningful readout of answer preference at intermediate layers.
    Used in Equation (1) to turn hidden states into logit gaps; if intermediate projections are uncalibrated, the freezing-confidence result changes interpretation.
invented entities (1)
  • Abstract reference representation (reference steering vector) independent evidence
    purpose: A direction in vision-token hidden states that marks goal-relevant objects; used in contrastive steering interventions and claimed to generalize across objects, tasks, and synthetic-to-natural domains.
    The paper provides a falsifiable handle: adding/subtracting this direction flips model outputs on held-out prompts and natural images, and it transfers across tasks. That is evidence outside the extraction procedure, though the train/test split is not fully clear.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Linguistic Context Recodes Visual Representations in Vision-Language Models." pith.science (2026). https://pith.science/paper/WBCCLB2Q

@misc{pith2026260800035,
  author       = {Pith},
  title        = {Pith review of: Linguistic Context Recodes Visual Representations in Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WBCCLB2Q}},
  note         = {Machine review of arXiv:2608.00035}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Goal-directed visual processing is a hallmark of human visual intelligence, resulting in representations that support downstream tasks such as categorization or search. Though vision-language models (VLMs) are often faced with these same tasks, their ability to recode visual representations when presented with goal-directed language remains poorly characterized. Indeed, prior work largely treats visual representations in VLMs as static repositories of visual information that are manipulated by language representations. In the present work, we provide evidence for two concrete instances of language-induced recoding of visual representations. First, we identify an abstract reference representation that denotes which objects are goal-relevant under a natural language prompt. We extract contrastive steering vectors corresponding to this reference representation and demonstrate that they are causally implicated in model predictions. These reference representations are abstract in that they generalize to different objects, different task contexts, and even from synthetic to naturalistic images. Second, we demonstrate language-induced attribute modulation: later layers selectively amplify goal-relevant attributes in visual representations of objects. We demonstrate this phenomenon across a range of different prompts. Finally, we provide a causal intervention that demonstrates that attribute modulation mediates a VLM's response distribution. Together, our results support a more dynamic account of cross-modality processing in VLMs -- rather than vision tokens serving as static repositories of information, they are modulated to support queries articulated in language.

Figures

Figures reproduced from arXiv: 2608.00035 by Brian Song, Ellie Pavlick, Michael A. Lepori.

Figure 1
Figure 1. Figure 1: Goal-directed language input causes VLMs to recode visual representations. In this work, we hypothesize [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Linear probing accuracy for referent vs non [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: (a) Reference steering vector extraction for [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Synthetic dataset intervention success rates for task-specific steering vectors on Count, Yes/No, and [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Naturalistic dataset intervention success rates [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Overview of the attribute modulation experi [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Shape/Color Attribute Modulation. We com [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Freezing referent-object vision tokens reduces [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗
Figure 8
Figure 8. Figure 8: Top: Front/back attribute modulation us￾ing front/back prompt pairs on the overlapping object dataset. We visualize the difference in projections of all vision tokens onto concept vectors corresponding to the front object’s attributes, the back object’s attributes, and absent attribute controls when an image is coupled with (1) a front-focused prompt vs. (2) a back-focused prompt. Front-focused prompts inc… view at source ↗
Figure 11
Figure 11. Figure 11: Qwen cross task intervention success rates [PITH_FULL_IMAGE:figures/full_fig_p014_11.png] view at source ↗
Figure 10
Figure 10. Figure 10: Probing analysis of Qwen for Counting and [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗
Figure 14
Figure 14. Figure 14: Natural interventions 0 5 10 15 20 25 Layer 6 4 2 0 2 4 6 8 Attribute projection difference (referring prompt contrastive prompt) Shape vs. Color Attribute Modulation Target shape Target color Incorrect shape Incorrect color [PITH_FULL_IMAGE:figures/full_fig_p015_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Shape vs. color modulation 0 5 10 15 20 25 Layer 0.15 0.10 0.05 0.00 0.05 0.10 0.15 Attribute projection difference (referring prompt contrastive prompt) Front/Back Object Attribute Modulation Front object true attribute Back object true attribute Incorrect attribute Shape - - Color [PITH_FULL_IMAGE:figures/full_fig_p015_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Front vs. back modulation 0 5 10 15 20 25 Layer 6 4 2 0 2 4 Attribute projection difference (referring prompt contrastive prompt) Referent Object Attribute Modulation Referent object true attribute Non-referent object true attribute Referent object incorrect attribute Non-referent object incorrect attribute Shape - - Color [PITH_FULL_IMAGE:figures/full_fig_p015_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Referent object modulation F.2 Explicit Shape and Color Modulation To visualize attribute-specific shifts relative to the baseline description prompt, we compute ∆base shape(o, l) = Proj(o, s(o), pshape, l) − Proj(o, s(o), pbase, l), (5) ∆base color(o, l) = Proj(o, c(o), pcolor, l) − Proj(o, c(o), pbase, l), (6) 15 [PITH_FULL_IMAGE:figures/full_fig_p015_17.png] view at source ↗
Figure 13
Figure 13. Figure 13: Synthetic interventions [PITH_FULL_IMAGE:figures/full_fig_p015_13.png] view at source ↗
Figure 18
Figure 18. Figure 18: Steering coefficient sweeps across models, datasets, and tasks. Rows correspond to synthetic and natural [PITH_FULL_IMAGE:figures/full_fig_p016_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Description prompt-relative shape/color displacement geometry across layers in Qwen. Each point [PITH_FULL_IMAGE:figures/full_fig_p018_19.png] view at source ↗
Figure 21
Figure 21. Figure 21: RSA model fit across layers. Left: Under the shape prompt, the shape hypothesis matrix correlates more strongly with the observed similarity structure than the color hypothesis matrix. Right: Under the color prompt, the shape hypothesis still correlates more strongly but its fit decreases, while the color hypothesis fit increases. Difference RSMs. To isolate query-induced changes in representational geome… view at source ↗
Figure 22
Figure 22. Figure 22: Difference RSMs across layers. Red in￾dicates increased cosine similarity under the queried prompt relative to the contrasting prompt; blue indicates suppression. Black outlines mark same-attribute groups. Top: Shape Prompt − Color Prompt sorted by shape. Same-shape blocks are neutral or red while off-diagonal regions are predominantly blue in layers 16, 19, and 22. Bottom: Color Prompt − Shape Prompt sor… view at source ↗
Figure 24
Figure 24. Figure 24: Freezing results for correct shape confidence [PITH_FULL_IMAGE:figures/full_fig_p019_24.png] view at source ↗
Figure 23
Figure 23. Figure 23: Concept vector interventions at referent vs. non-referent object positions at layers 20, 22, and 24 for [PITH_FULL_IMAGE:figures/full_fig_p020_23.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 7 linked inside Pith

  1. [1]

    Advances in Neural Information Processing Systems , pages=

    Template-based algorithms for connectionist rule extraction , author=. Advances in Neural Information Processing Systems , pages=

  2. [2]

    2024 , eprint=

    GPT-4 Technical Report , author=. 2024 , eprint=

    2024

  3. [3]

    2025 , eprint=

    Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Models , author=. 2025 , eprint=

    2025

  4. [4]

    2025 , eprint=

    Words or Vision: Do Vision-Language Models Have Blind Faith in Text? , author=. 2025 , eprint=

    2025

  5. [5]

    arXiv preprint arXiv:2410.03659 , year =

    Zhu, Tinghui and Liu, Qin and Wang, Fei and Tu, Zhengzhong and Chen, Muhao , title =. arXiv preprint arXiv:2410.03659 , year =. 2410.03659 , archivePrefix =

    Pith/arXiv arXiv

  6. [6]

    The Thirteenth International Conference on Learning Representations , year=

    Towards Interpreting Visual Information Processing in Vision-Language Models , author=. The Thirteenth International Conference on Learning Representations , year=

  7. [7]

    Frontiers in Systems Neuroscience , volume=

    Representational similarity analysis: connecting the branches of systems neuroscience , author=. Frontiers in Systems Neuroscience , volume=. 2008 , doi=

    2008

  8. [8]

    The Fourteenth International Conference on Learning Representations , year=

    Visual symbolic mechanisms: Emergent symbol processing in Vision Language Models , author=. The Fourteenth International Conference on Learning Representations , year=

  9. [9]

    arXiv preprint arXiv:2511.17699 , year=

    Understanding Counting Mechanisms in Large Language and Vision-Language Models , author=. arXiv preprint arXiv:2511.17699 , year=

    Pith/arXiv arXiv

  10. [10]

    arXiv preprint arXiv:2504.10479 , year=

    InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models , author=. arXiv preprint arXiv:2504.10479 , year=

    Pith/arXiv arXiv

  11. [11]

    2020 , howpublished=

    Interpreting GPT: The logit lens , author=. 2020 , howpublished=

    2020

  12. [12]

    2026 , eprint=

    The Geometry of Representational Failures in Vision Language Models , author=. 2026 , eprint=

    2026

  13. [13]

    The Dual Mechanisms of Spatial Reasoning in Vision

    Kelly Cui and Nikhil Prakash and Ayush Raina and David Bau and Antonio Torralba and Tamar Rott Shaham , booktitle=. The Dual Mechanisms of Spatial Reasoning in Vision. 2026 , url=

    2026

  14. [14]

    Interpreting

    Yossi Gandelsman and Alexei A Efros and Jacob Steinhardt , booktitle=. Interpreting. 2024 , url=

    2024

  15. [15]

    The Fourteenth International Conference on Learning Representations , year=

    Language Models Use Lookbacks to Track Beliefs , author=. The Fourteenth International Conference on Learning Representations , year=

  16. [16]

    Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for

    Palit, Vedant and Pandey, Rohan and Arora, Aryaman and Liang, Paul Pu , booktitle=. Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for. 2023 , doi=

    2023

  17. [17]

    The Thirteenth International Conference on Learning Representations , year=

    Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations , author=. The Thirteenth International Conference on Learning Representations , year=

  18. [18]

    The Eleventh International Conference on Learning Representations , year=

    Linearly Mapping from Image to Text Space , author=. The Eleventh International Conference on Learning Representations , year=

  19. [19]

    The Thirteenth International Conference on Learning Representations , year=

    The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities , author=. The Thirteenth International Conference on Learning Representations , year=

  20. [20]

    Microsoft

    Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll. Microsoft. Proceedings of the European Conference on Computer Vision (ECCV) , pages=. 2014 , organization=

    2014

  21. [21]

    Annual Review of Neuroscience , volume=

    Neural mechanisms of selective visual attention , author=. Annual Review of Neuroscience , volume=. 1995 , doi=

    1995

  22. [22]

    Journal of Experimental Psychology: General , volume=

    Selective attention and the organization of visual information , author=. Journal of Experimental Psychology: General , volume=. 1984 , doi=

    1984

  23. [23]

    Nature Reviews Neuroscience , volume=

    Control of goal-directed and stimulus-driven attention in the brain , author=. Nature Reviews Neuroscience , volume=. 2002 , doi=

    2002

  24. [24]

    The Twelfth International Conference on Learning Representations , year=

    How do Language Models Bind Entities in Context? , author=. The Twelfth International Conference on Learning Representations , year=

  25. [25]

    and Nye, Maxwell and Andreas, Jacob

    Li, Belinda Z. and Nye, Maxwell and Andreas, Jacob. Implicit Representations of Meaning in Neural Language Models. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). doi:10.18653/v1/2021.acl-long.143

    doi:10.18653/v1/2021.acl-long.143 2021

  26. [26]

    Steering Llama 2 via Contrastive Activation Addition

    Rimsky, Nina and Gabrieli, Nick and Schulz, Julian and Tong, Meg and Hubinger, Evan and Turner, Alexander. Steering Llama 2 via Contrastive Activation Addition. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). doi:10.18653/v1/2024.acl-long.828

    doi:10.18653/v1/2024.acl-long.828 2024

  27. [27]

    arXiv preprint arXiv:2308.10248 , year=

    Steering Language Models With Activation Engineering , author=. arXiv preprint arXiv:2308.10248 , year=

    Pith/arXiv arXiv

  28. [28]

    Thirty-seventh Conference on Neural Information Processing Systems , year=

    Inference-Time Intervention: Eliciting Truthful Answers from a Language Model , author=. Thirty-seventh Conference on Neural Information Processing Systems , year=

  29. [29]

    2025 , eprint=

    How Do Vision-Language Models Process Conflicting Information Across Modalities? , author=. 2025 , eprint=

    2025

  30. [30]

    The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

    Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects , author=. The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

  31. [31]

    Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in

    Yaniv Nikankin and Dana Arad and Yossi Gandelsman and Yonatan Belinkov , booktitle=. Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in. 2026 , url=

    2026

  32. [32]

    The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

    Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem , author=. The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

  33. [33]

    2022 , pages =

    Thrush, Tristan and Jiang, Ryan and Bartolo, Max and Singh, Amanpreet and Williams, Adina and Kiela, Douwe and Ross, Candace , booktitle =. 2022 , pages =. doi:10.1109/CVPR52688.2022.00517 , publisher =

    arXiv 2022

  34. [34]

    The Eleventh International Conference on Learning Representations , year=

    When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It? , author=. The Eleventh International Conference on Learning Representations , year=

  35. [35]

    The 2023 Conference on Empirical Methods in Natural Language Processing , year=

    Evaluating Object Hallucination in Large Vision-Language Models , author=. The 2023 Conference on Empirical Methods in Natural Language Processing , year=

    2023

  36. [36]

    What Do VLM s NOTICE ? A Mechanistic Interpretability Pipeline for G aussian-Noise-free Text-Image Corruption and Evaluation

    Golovanevsky, Michal and Rudman, William and Palit, Vedant and Eickhoff, Carsten and Singh, Ritambhara. What Do VLM s NOTICE ? A Mechanistic Interpretability Pipeline for G aussian-Noise-free Text-Image Corruption and Evaluation. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Hum...

    doi:10.18653/v1/2025.naacl-long.571 2025

  37. [37]

    Second Conference on Language Modeling , year=

    Visual Representations inside the Language Model , author=. Second Conference on Language Modeling , year=

  38. [38]

    Bai, Shuai and Chen, Keqin and Liu, Xuejing and Wang, Jialin and Ge, Wenbin and Song, Sibo and Dang, Kai and Wang, Peng and Wang, Shijie and Tang, Jun and others , journal=

  39. [39]

    arXiv preprint arXiv:2503.19786 , year=

    Gemma 3 Technical Report , author=. arXiv preprint arXiv:2503.19786 , year=

    Pith/arXiv arXiv

  40. [40]

    Advances in Neural Information Processing Systems , editor=

    Flamingo: a Visual Language Model for Few-Shot Learning , author=. Advances in Neural Information Processing Systems , editor=. 2022 , url=

    2022

  41. [41]

    2025 , eprint=

    Chameleon: Mixed-Modal Early-Fusion Foundation Models , author=. 2025 , eprint=

    2025

  42. [42]

    Lawrence Zitnick and Devi Parikh , title =

    Stanislaw Antol and Aishwarya Agrawal and Jiasen Lu and Margaret Mitchell and Dhruv Batra and C. Lawrence Zitnick and Devi Parikh , title =. International Conference on Computer Vision (ICCV) , year =

  43. [43]

    Does Object Binding Naturally Emerge in Large Pretrained Vision Transformers? , year=

    Li, Yihao and Salehi, Saeed and Ungar, Lyle and Kording, Konrad , booktitle=. Does Object Binding Naturally Emerge in Large Pretrained Vision Transformers? , year=

  44. [44]

    science , volume=

    Top-down versus bottom-up control of attention in the prefrontal and posterior parietal cortices , author=. science , volume=. 2007 , publisher=

    2007

  45. [45]

    Science , volume=

    Categorical representation of visual stimuli in the primate prefrontal cortex , author=. Science , volume=. 2001 , publisher=

    2001

  46. [46]

    Advances in neural information processing systems , volume=

    Visual instruction tuning , author=. Advances in neural information processing systems , volume=

  47. [47]

    International conference on machine learning , pages=

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models , author=. International conference on machine learning , pages=. 2023 , organization=

    2023

  48. [48]

    arXiv preprint arXiv:2602.06218 , year=

    Cross-Modal Redundancy and the Geometry of Vision-Language Embeddings , author=. arXiv preprint arXiv:2602.06218 , year=

    arXiv

  49. [49]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Pixels versus priors: Controlling knowledge priors in vision-language models through visual counterfacts , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  50. [50]

    arXiv preprint arXiv:2604.09942 , year=

    I Walk the Line: Examining the Role of Gestalt Continuity in Object Binding for Vision Transformers , author=. arXiv preprint arXiv:2604.09942 , year=

    Pith/arXiv arXiv

  51. [51]

    Journal of Neuroscience , volume=

    Task dependence of visual and category representations in prefrontal and inferior temporal cortices , author=. Journal of Neuroscience , volume=. 2014 , publisher=

    2014

  52. [52]

    Trends in cognitive sciences , volume=

    Top-down modulation: bridging selective attention and working memory , author=. Trends in cognitive sciences , volume=. 2012 , publisher=

    2012

  53. [53]

    Annual review of neuroscience , volume=

    Neuronal mechanisms of visual categorization: an abstract view on decision making , author=. Annual review of neuroscience , volume=. 2016 , publisher=

    2016

  54. [54]

    arXiv preprint arXiv:2507.01790 , year=

    How Do Vision-Language Models Process Conflicting Information Across Modalities? , author=. arXiv preprint arXiv:2507.01790 , year=

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.