Pith. sign in

REVIEW 3 major objections 5 minor 54 references

Multimodal models shift attention to the image or text exactly when the next token needs that modality, and boosting it on schedule improves their answers.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 00:16 UTC pith:DN3MRDHZ

load-bearing objection Clean token-level map of when MLLMs attend to image vs text, causally validated and turned into a simple timed boost that actually moves accuracy. the 3 major comments →

arxiv 2607.03738 v1 pith:DN3MRDHZ submitted 2026-07-04 cs.CV cs.AI

Attending to Multimodal Generation One Token at a Time

Varun Gupta , Vineet Gandhi , Makarand Tapaswi This is my paper
classification cs.CV cs.AI
keywords multimodal LLMsattention dynamicsautoregressive generationtoken-level interpretabilityattention blockingattention boostingvision-language modelstest-time intervention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that multimodal large language models do not attend uniformly while they generate. Instead, attention to the image spikes precisely when the model produces an image-derived word, attention to the instruction rises at the moment it switches tasks, and attention to its own previous tokens steadily grows. These patterns appear consistently across four open-weight models from two families and across both diagnostic and real benchmarks that force switches between vision and language inside one response. Blocking the peaks causes systematic failures—language priors, cross-modal leakage, denial of the image, or recovery—while a simple multiplicative boost of the right modality at the right generation step raises accuracy on visual-spatial reasoning. The work therefore supplies both a temporal account of how multimodal context is used and a practical test-time lever for improving it.

Core claim

Across four open-weight MLLMs, attention to image tokens peaks at the generation steps that require image-derived information, instruction tokens are revisited at task hand-offs, and attention to previously generated tokens increases throughout decoding; total blocking of these peaks produces language-prior fallback, leakage, denial or recovery, while selective boosting of the relevant modality at those steps measurably improves multimodal task performance.

What carries the argument

One Token at a Time (OTaT) analysis: average post-softmax attention from the currently generating token onto semantic chunks (image, text, instruction, previous tokens), normalize by mean subtraction, aggregate over role-tagged output tokens, then intervene by total blocking or multiplicative reweighting of the resulting scalar.

Load-bearing premise

Averaging attention uniformly across every layer and head still yields a scalar that is functionally meaningful enough that blocking or boosting it alone can change the model’s answers.

What would settle it

Measure fruit or spatial-relation accuracy after total blocking of image attention only at the concept token and after timed boosting only at the same token; if accuracy is statistically unchanged relative to the unblocked and always-boost baselines, the claimed causal role of the timed peaks is false.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A training-free, token-timed attention boost can raise accuracy on tasks that require modality switches inside one answer.
  • Failure to answer the second part of a multi-part prompt can be diagnosed by checking whether instruction attention spikes at the hand-off token.
  • The same temporal routing appears in pure language models given two text tasks, indicating the pattern is inherited from the base LLM.
  • Disrupted visual attention produces stereotyped failure modes (prior, leakage, denial, recovery) that can be used as reliability diagnostics.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because late layers dominate the observed spikes, a global boost still hits the critical computation even without head- or layer-selective targeting.
  • Role-aware boosting could be stacked with existing head- or layer-selective interventions for larger gains than either alone.
  • Post-hoc semantic tagging of free-form outputs would let the same analysis be run without forcing a fixed response format.
  • Cross-modal leakage under blocking in the smallest model may signal incomplete modality separation that larger models later suppress.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies token-level attention dynamics during autoregressive generation in MLLMs (OTaT), tracking attention from the currently generating token to semantic chunks (image, text, instruction, previous tokens). Using a diagnostic Fruit-Math task plus ChartQA and VSR, it reports consistent patterns across LLaVA-OneVision and Qwen2.5-VL models (0.5B–7B): image attention peaks at image-derived concepts, instruction attention rises at task hand-offs, and previous-token attention grows over decoding. Total attention blocking at these peaks produces predicted failures (language priors, cross-modal leakage, denial, recovery); timed multiplicative boosting of the relevant modality at the right step improves VSR accuracy substantially, especially for the weaker model.

Significance. If the patterns and causal effects hold, the work supplies a useful complementary ‘when’ perspective to existing layer/head (‘where’) interpretability of MLLMs, with concrete functional validation via total blocking and a simple, architecture-agnostic test-time boost that yields large gains on an entangled spatial-reasoning task. Strengths include multi-family/multi-size consistency, explicit causal interventions (Tables 1–3), qualitative failure-mode profiling, layer-group consensus checks (App. A.2), and an immediately usable intervention. The contribution is primarily empirical and methodological rather than theoretical, but it is actionable for both analysis and reliability improvements.

major comments (3)
  1. [§2.1, Eq. (1); App. A.2] Eq. (1) and §2.1 define the core scalar α_t(c_k) by uniform averaging of post-softmax attention over all layers and heads. While App. A.2 shows directional consensus across early/mid/late groups and the blocking/boosting results (Tables 1, 3) are consistent with the scalar being functionally meaningful, the paper should more explicitly quantify how much of the causal effect is carried by late layers alone (e.g., by repeating total blocking only on late layers). Without that, the claim that the global average is a sufficient intervention target remains only partially stress-tested.
  2. [§4.3; App. C] The timed boosting intervention (§4.3, Table 3) and the dataset-level bar plots both rely on knowing semantic role tags (Fruit Concept, ISR/TSR, Handoff, etc.) in advance. For free-form generation this knowledge is unavailable; the paper notes the limitation (App. C) but does not demonstrate an automatic, model-internal detector of the critical steps. A minimal experiment showing that a simple heuristic (e.g., first occurrence of a format keyword or entropy spike) recovers most of the gain would strengthen the practical claim.
  3. [§3; Tables 1–3; App. D] Fruit and VSR accuracy, answer rates, and error profiles (Tables 1–3, Fig. 7) are obtained via Gemini 2.5 Pro prompts (App. D.3–D.4) with only ‘spot-checks’. Because these numbers underwrite the causal claims, the manuscript should report inter-annotator agreement on a non-trivial human-labeled subset (or release the exact scored outputs) so that the magnitude of the reported drops and gains can be independently verified.
minor comments (5)
  1. [Fig. 1; App. A.1] Fig. 1D–E and the per-token plots in App. A.1 are dense; adding a short legend or color-blind-safe palette note would help readers who cannot zoom.
  2. [§2.2.1] The distinction between ‘lazy’ and ‘total’ blocking (Fig. 2, §2.2.1) is clear, but the main text could briefly restate why intermediate-token leakage is possible under lazy blocking before presenting Table 1.
  3. [§4.1] In §4.1 the claim that attention scores are influenced by both y_t and y_{t−1} is interesting; a one-sentence pointer to the corresponding spike at EOG in the per-token plots would make the observation easier to locate.
  4. [§4.1; App. A.4] App. A.4 (Math-Sport on the base LLM) is valuable; a single sentence in the main text noting that the same patterns appear in the text-only setting would better advertise the generality.
  5. [App. A.5] Minor typographical inconsistencies appear in the appendix (e.g., ‘oranges’ misspelled in one recovery example, occasional missing spaces around citations). A light proof-reading pass would suffice.

Circularity Check

0 steps flagged

Empirical observational + interventional study; measured attention peaks and causal effects are not defined by their own inputs.

full rationale

The paper is not a first-principles derivation. Its load-bearing claims are (i) observed token-level attention reallocations (image peaks at image-derived concepts, instruction at hand-offs, rising previous-token attention), (ii) causal validation via total blocking of those peaks, and (iii) a test-time multiplicative boost of the relevant modality at the right semantic tag. The scalar α_t(c_k) (Eq. 1) is a uniform average of post-softmax attention; it is measured, not fitted to force the peaks. Normalization subtracts a dataset-wide mean and does not create the relative spikes. Blocking sets attention mass to zero and reports accuracy collapse / failure modes; those outcomes are not algebraically forced by the definition of α. Boost factors β/γ are chosen by a post-hoc sweep on VSR, but the existence of the peaks was established earlier on Fr-Ma/ChartQA, the always-boost-image control degrades performance, and Fig. 18 shows a wide plateau—so the improvement is not a tautological rename of a fitted constant. No self-citation uniqueness theorem, no ansatz smuggled from the authors’ prior work, and no renaming of a known closed-form result. The study is self-contained against external open-weight models and public benchmarks; circularity score is zero.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 3 invented entities

The central claims rest on standard transformer attention mechanics plus three paper-specific methodological choices (global averaging, total blocking, and empirically chosen boost factors). No new physical entities are postulated; the free parameters are only the boost multipliers selected by sweep.

free parameters (1)
  • effective boost factor γ = ≈11 (LOV), ≈13 (QVL)
    Chosen by grid search over β (≈11 for LLaVA-OneVision, ≈13 for Qwen2.5-VL) to maximize VSR accuracy; the paper shows a broad plateau but the reported numbers are data-dependent.
axioms (3)
  • ad hoc to paper Post-softmax attention averaged uniformly over layers and heads is a sufficient proxy for functional information flow from a semantic chunk.
    Eq. 1 and the subsequent blocking/boosting interventions treat this scalar as the quantity that can be zeroed or scaled; layer-group analysis (App. A.2) shows late layers dominate but the global average is still used.
  • domain assumption Total blocking (zeroing attention from a chunk to all future tokens) completely severs information flow from that chunk.
    Contrasted with lazy blocking; assumed sufficient to isolate causal contribution of image/instruction/previous tokens.
  • ad hoc to paper Response-format instructions plus automatic POS tagging produce role labels that align across variable-length free generations.
    Required for dataset-level bar plots; Gemini tagging prompts are supplied but residual misalignment remains possible.
invented entities (3)
  • Fruit-Math diagnostic task no independent evidence
    purpose: Disentangle visual and textual information so modality-specific attention peaks can be isolated without redundancy confounds.
    Constructed by pairing OpenImages fruits with GSM-8K math problems under a fixed response template.
  • total blocking operator no independent evidence
    purpose: Prevent information leakage through intermediate tokens that lazy (single-step) blocking cannot stop.
    Defined as A[:,:,e_ck:t,s_ck:e_ck]=0; used for all causal claims.
  • OTaT (One Token at a Time) analysis framework no independent evidence
    purpose: Aggregate and normalize attention by semantic role across multi-token responses.
    Core methodological contribution enabling the reported patterns.

pith-pipeline@v1.1.0-grok45 · 34053 in / 2397 out tokens · 25524 ms · 2026-07-12T00:16:35.601307+00:00 · methodology

0 comments
read the original abstract

Multimodal large language models (MLLMs) generate responses autoregressively, integrating visual and linguistic information in an evolving context. Prior work on interpretability has focused on individual layers and circuits (where), leaving the token-level dynamics of multimodal computation during generation (when) underexplored. We address this gap and study attention shifts as per semantic role; tracking model attention to image, text, instruction, and previously generated tokens, One Token at a Time (OTaT). We introduce multimodal tasks that require explicit switching between visual and textual context within a single response. Across two mainstream model families and four open-weight MLLMs of varying sizes, we establish consistent patterns: attention to image peaks at tokens requiring image-derived information, instruction tokens are revisited during task transitions, and attention to previously generated tokens increases as the generation progresses. Causal attention blocking interventions validate the functional role of these trends. We profile model behavior under disrupted attention and observe responses falling back to language priors, or exhibiting cross-modal leakage, denial, or recovery. Finally, informed of the attention dynamics through our novel analysis, we propose a simple test-time intervention to boost attention to the relevant modality at the right time, significantly improving multimodal task performance.

Figures

Figures reproduced from arXiv: 2607.03738 by Makarand Tapaswi, Varun Gupta, Vineet Gandhi.

Figure 1
Figure 1. Figure 1: We illustrate our approach to analyze attention patterns in autoregressive MLLMs. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Blocking strategies in QK attention (gray is the causal mask, red is blocked). Left: Lazy blocking only affects CGT. Right: Total blocking prevents informa￾tion flow to any future token. Attention patterns tell us the relative importance of context chunks. However, to probe whether a chunk is functionally important, we intervene by blocking attention to that chunk and measuring the resulting change in the … view at source ↗
Figure 3
Figure 3. Figure 3: Samples from multimodal tasks. Fruit-Math (left) shows an image of a fruit along with an unrelated math puzzle. Visual Spatial Reasoning (VSR) [28] (middle) features an image with a paired caption that describes the scene, but with a conflicting spatial relationship. ChartQA [24] (right) shows a diagram followed by two questions related to it. The model is instructed to (respectively): identify the fruit i… view at source ↗
Figure 4
Figure 4. Figure 4: Evolution of attention patterns in Fr-Ma for QVL-3B (left) and LOV-7B (right). [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Attention trends flip when the response for [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Confusion matrices showing models’ outputs after blocking at [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Attention trends for the VSR task explain why QVL-7B (right) outperforms LOV-7B (left), with peaks [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Per-token normalized attention plots for all other models on the Fruit-Math task. We observe similar [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Normalized attention analysis on the Fruit-Math task for LOV-7B (top) and QVL-7B (bottom) models, [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Attention trends for LLaVA-OneVision-0.5B (left) and Qwen2.5-VL-7B (right) across output tags on [PITH_FULL_IMAGE:figures/full_fig_p016_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Attention trends for Qwen2-7B LLM (the base LLM of LOV-7B) on the Math-Sport task. [PITH_FULL_IMAGE:figures/full_fig_p017_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: LLaVA-OneVision-0.5B exhibits strong cross-modal information leakage where the fruit predictions [PITH_FULL_IMAGE:figures/full_fig_p017_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Qwen2.5-VL-3B shows mixed results with examples of denial, reliance on language prior, and some [PITH_FULL_IMAGE:figures/full_fig_p018_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Qwen2.5-VL-7B is able to understand that the image does not exist (in this case, due to blocking), [PITH_FULL_IMAGE:figures/full_fig_p018_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Outputs produced by LLaVA-OneVision-7B when attention to previously generated tokens is blocked. [PITH_FULL_IMAGE:figures/full_fig_p019_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Outputs produced by Qwen2.5-VL-3B when attention to the instruction token is blocked during [PITH_FULL_IMAGE:figures/full_fig_p019_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Boosting analysis across wide range of β tested for LOV-7B and QVL-7B. at 49.8 with β = 50 and stays within 1.1 points of this maximum for β ∈ [20, 100]. Similarly, QVL varies by less than 1 point between β = 30 and β = 400, with a shallow optimum around 150-300. These results indicate that the benefits of attention boosting are not tied to a narrowly tuned hyperparameter choice; rather, the method exhibi… view at source ↗
Figure 19
Figure 19. Figure 19: Attention intervention qualitative samples on LOV-7B on the VSR task. [PITH_FULL_IMAGE:figures/full_fig_p021_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Attention intervention qualitative samples on QVL-7B on the VSR task. [PITH_FULL_IMAGE:figures/full_fig_p021_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Sequence of input context chunks for LLaVA-OneVision (top) and Qwen2.5-VL (bottom). Note their [PITH_FULL_IMAGE:figures/full_fig_p022_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Fruit categories in our fruit dataset. Our dataset consists of 11 fruit categories, which are shown in [PITH_FULL_IMAGE:figures/full_fig_p022_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Examples of sport themed excerpts generated by Gemini for each of the 9 sport classes used in our [PITH_FULL_IMAGE:figures/full_fig_p023_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: POS tagging for Fruit-Math, ChartQA, and VSR tasks. Dataset-level findings reported in Sec. 4. [PITH_FULL_IMAGE:figures/full_fig_p024_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Gemini sport text creation prompt for various sports. [PITH_FULL_IMAGE:figures/full_fig_p025_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: MLLM generation prompt for Fruit-Math task. [PITH_FULL_IMAGE:figures/full_fig_p025_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: MLLM generation prompt for ChartQA task. [PITH_FULL_IMAGE:figures/full_fig_p025_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: MLLM generation prompt for VSR task. We present the prompts (or instructions) given to the LLaVA-OneVision and Qwen2.5-VL MLLMs across tasks. The Fruit-Math prompt is shown in [PITH_FULL_IMAGE:figures/full_fig_p025_28.png] view at source ↗
Figure 29
Figure 29. Figure 29: MLLM generation prompt for Math-Sport task. [PITH_FULL_IMAGE:figures/full_fig_p026_29.png] view at source ↗
Figure 30
Figure 30. Figure 30: Gemini prompt for semantic POS token tagging for Fruit-Math task. [PITH_FULL_IMAGE:figures/full_fig_p026_30.png] view at source ↗
Figure 31
Figure 31. Figure 31: Gemini prompt for semantic POS token tagging for VSR task. [PITH_FULL_IMAGE:figures/full_fig_p027_31.png] view at source ↗
Figure 32
Figure 32. Figure 32: Gemini prompt for semantic POS token tagging for the ChartQA task. [PITH_FULL_IMAGE:figures/full_fig_p027_32.png] view at source ↗
Figure 33
Figure 33. Figure 33: Gemini prompt for semantic POS token tagging for Math-Sport task. [PITH_FULL_IMAGE:figures/full_fig_p028_33.png] view at source ↗
Figure 34
Figure 34. Figure 34: Accuracy, answer rate and error profiling prompt for the Fruit-Math task. [PITH_FULL_IMAGE:figures/full_fig_p029_34.png] view at source ↗
Figure 35
Figure 35. Figure 35: Evaluation prompt for the VSR task. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_35.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 7 linked inside Pith

  1. [1]

    Possible Principles Underlying the Transformation of Sensory Messages

    Horace B Barlow. Possible Principles Underlying the Transformation of Sensory Messages. Sensory communication, 1(01), 1961. 1

    1961

  2. [2]

    DEX-AR: A Dy- namic Explainability Method for Autoregressive Vision-Language Models.arXiv preprint arXiv:2603.06302, 2026

    Walid Bousselham, Angie Boggust, Hendrik Strobelt, and Hilde Kuehne. DEX-AR: A Dy- namic Explainability Method for Autoregressive Vision-Language Models.arXiv preprint arXiv:2603.06302, 2026. 1, 2, 10

    arXiv 2026

  3. [3]

    Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generation.arXiv preprint arXiv:2509.22496, 2025

    Ruoyu Chen, Xiaoqing Guo, Kangwei Liu, Si Yuan Liang, Shiming Liu, et al. Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generation.arXiv preprint arXiv:2509.22496, 2025. 10

    arXiv 2025

  4. [4]

    What Does BERT Look At? An Analysis of BERT’s Attention

    Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What Does BERT Look At? An Analysis of BERT’s Attention. InACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2019. 9

    2019

  5. [5]

    Training Verifiers to Solve Math Word Problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems.arXiv preprint arXiv:2110.14168,

    Pith/arXiv arXiv

  6. [6]

    Flashattention: Fast and Memory-efficient Exact Attention with IO-Awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and Memory-efficient Exact Attention with IO-Awareness. InAdvances in Neural Information Processing Systems (NeurIPS), 2022. 24

    2022

  7. [7]

    An Image is Worth 16x16 Words: Transformers for Image Recognition At Scale

    Alexey Dosovitskiy. An Image is Worth 16x16 Words: Transformers for Image Recognition At Scale. InInternational Conference on Learning Representations (ICLR), 2021. 9

    2021

  8. [8]

    A Mathematical Framework for Transformer Circuits.Transformer Circuits Thread, 2021

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A Mathematical Framework for Transformer Circuits.Transformer Circuits Thread, 2021. 1

    2021

  9. [9]

    Toy Models of Superposition.arXiv preprint arXiv:2209.10652, 2022

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy Models of Superposition.arXiv preprint arXiv:2209.10652, 2022. 1

    Pith/arXiv arXiv 2022

  10. [10]

    Mitigating Hallucination in Large Vision-Language Models via Adaptive Attention Calibration

    Mehrdad Fazli, Bowen Wei, Ahmet Sari, and Ziwei Zhu. Mitigating Hallucination in Large Vision-Language Models via Adaptive Attention Calibration. InFindings of North American Chapter of Association of Computational Linguistics (NAACL-F), 2025. 10

    2025

  11. [11]

    Dissecting Recall of Factual Associations in Auto-regressive Language Models

    Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting Recall of Factual Associations in Auto-regressive Language Models. InAssociation of Computational Linguistics (ACL), 2023. 1, 9

    2023

  12. [12]

    LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contexts

    Zhuohan Gu, Jiayi Yao, Kuntai Du, and Junchen Jiang. LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contexts. InMachine Learning for Systems Workshop (NeurIPS), 2024. 10

    2024

  13. [13]

    How Do Vision-Language Models Process Conflicting Information Across Modalities?arXiv preprint arXiv:2507.01790, 2025

    Tianze Hua, Tian Yun, and Ellie Pavlick. How Do Vision-Language Models Process Conflicting Information Across Modalities?arXiv preprint arXiv:2507.01790, 2025. 9

    Pith/arXiv arXiv 2025

  14. [14]

    Opera: Alleviating Hallucination in Multi-modal Large Language Models via Over-Trust Penalty and Retrospection-allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating Hallucination in Multi-modal Large Language Models via Over-Trust Penalty and Retrospection-allocation. InConference on Computer Vision and Pattern Recognition (CVPR), 2024. 4

    2024

  15. [15]

    The Platonic Representation Hypothesis

    Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The Platonic Representation Hypothesis. InInternational Conference on Machine Learning (ICML), 2024. 9

    2024

  16. [16]

    What’s in The Image? A Deep-Dive Into the Vision of Vision Language Models

    Omri Kaduri, Shai Bagon, and Tali Dekel. What’s in The Image? A Deep-Dive Into the Vision of Vision Language Models. InConference on Computer Vision and Pattern Recognition (CVPR), 2025. 4, 9 11

    2025

  17. [17]

    See What You Are Told: Visual Attention Sink in Large Multimodal Models

    Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See What You Are Told: Visual Attention Sink in Large Multimodal Models. InInternational Conference on Learning Representations (ICLR), 2025. 10

    2025

  18. [18]

    The Open Images Dataset V4: Unified Image Classification, Object detection, and Visual relationship detection at Scale

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The Open Images Dataset V4: Unified Image Classification, Object detection, and Visual relationship detection at Scale. InInternational Journal of Computer Vision (IJCV),

  19. [19]

    LLaV A-OneVision: Easy Visual Task Transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaV A-OneVision: Easy Visual Task Transfer. InTransactions on Machine Learning (TMLR), 2025. 1, 5, 16

    2025

  20. [20]

    Dy- namic Token Reduction During Generation for Vision Language Models.arXiv preprint arXiv:2501.14204, 2025

    Xiaoyu Liang, Chaofeng Guan, Jiaying Lu, Huiyao Chen, Huan Wang, and Haoji Hu. Dy- namic Token Reduction During Generation for Vision Language Models.arXiv preprint arXiv:2501.14204, 2025. 1, 2, 10

    Pith/arXiv arXiv 2025

  21. [21]

    Paying More Attention to Image: A Training-free Method for Alleviating Hallucination in LVLMs

    Shi Liu, Kecheng Zheng, and Wei Chen. Paying More Attention to Image: A Training-free Method for Alleviating Hallucination in LVLMs. InEuropean Conference on Computer Vision (ECCV), 2024. 10

    2024

  22. [22]

    Vision-language Models Create Cross-modal Task Representations

    Grace Luo, Trevor Darrell, and Amir Bar. Vision-language Models Create Cross-modal Task Representations. InInternational Conference on Machine Learning (ICML), 2024. 9

    2024

  23. [23]

    Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMs.arXiv preprint arXiv:2602.15556, 2026

    Guangtao Lyu, Qi Liu, Chenghao Xu, Jiexi Yan, Muli Yang, Xueting Li, Fen Fang, and Cheng Deng. Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMs.arXiv preprint arXiv:2602.15556, 2026. 10

    arXiv 2026

  24. [24]

    Chartqa: A Benchmark for Question Answering About Charts with Visual and Logical Reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A Benchmark for Question Answering About Charts with Visual and Logical Reasoning. In Findings of the Association for Computational Linguistics (ACL-F), pages 2263–2279, 2022. 5, 9

    2022

  25. [25]

    Towards Interpreting Visual Information Processing in Vision-Language Models

    Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards Interpreting Visual Information Processing in Vision-Language Models. InInternational Conference on Learning Representations (ICLR), 2024. 1, 4, 9

    2024

  26. [26]

    Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs

    Yaniv Nikankin, Dana Arad, Yossi Gandelsman, and Yonatan Belinkov. Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs. InAdvances in Neural Information Processing Systems (NeurIPS), 2025. 1, 9

    2025

  27. [27]

    Token-wise Decomposition of Autoregressive Language Model Hidden States for Analyzing Model Predictions

    Byung-Doh Oh and William Schuler. Token-wise Decomposition of Autoregressive Language Model Hidden States for Analyzing Model Predictions. InAssociation of Computational Linguistics (ACL), 2023. 10

    2023

  28. [28]

    Mixed Signals: Decoding VLMs’ Reasoning and Underlying Bias in Vision-language Conflict

    Pouya Pezeshkpour, Moin Aminnaseri, and Estevam Hruschka. Mixed Signals: Decoding VLMs’ Reasoning and Underlying Bias in Vision-language Conflict. InFindings of the Associ- ation for Computational Linguistics (ACL-F), 2025. 5, 9

    2025

  29. [29]

    Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding

    Feilong Tang, Chengzhi Liu, Zhongxing Xu, Ming Hu, Zile Huang, Haochen Xue, Ziyang Chen, Zelin Peng, Zhiwei Yang, Sijin Zhou, et al. Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding. InConference on Computer Vision and Pattern Recognition (CVPR), 2025. 10

    2025

  30. [30]

    Attention Is All You Need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. InAdvances in Neural Information Processing Systems (NeurIPS), 2017. 3, 9

    2017

  31. [31]

    MLLM Can See? Dynamic Correction Decoding for Hallucination Mitigation

    Chenxi Wang, Xiang Chen, Ningyu Zhang, Bozhong Tian, Haoming Xu, et al. MLLM Can See? Dynamic Correction Decoding for Hallucination Mitigation. InInternational Conference on Learning Representations (ICLR), 2025. 1, 9 12

    2025

  32. [32]

    ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM

    Yujun Wang, Jinhe Bi, Soren Pirk, Yunpu Ma, et al. ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM. InAssociation for the Advancement of Artificial Intelligence (AAAI), 2026. 10

    2026

  33. [33]

    Measuring Cross-modal Interactions in Multimodal Models

    Laura Wenderoth, Konstantin Hemker, Nikola Simidjievski, and Mateja Jamnik. Measuring Cross-modal Interactions in Multimodal Models. InAssociation for the Advancement of Artificial Intelligence (AAAI), 2025. 9

    2025

  34. [34]

    Qwen2 Technical Report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, et al. Qwen2 Technical Report.arXiv preprint arXiv:2407.10671, 2024. 16

    Pith/arXiv arXiv 2024

  35. [35]

    Qwen2.5 Technical Report.arXiv preprint arXiv:2502.13923, 2025

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, et al. Qwen2.5 Technical Report.arXiv preprint arXiv:2502.13923, 2025. 1, 2, 5

    Pith/arXiv arXiv 2025

  36. [36]

    Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference

    Hao Yin, Guangzong Si, and Zilei Wang. Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference. InConference on Computer Vision and Pattern Recognition (CVPR), 2025. 9

    2025

  37. [37]

    AdaVBoost: Mitigating Hallucinations in LVLMs via Token-Level Adaptive Visual Attention Boosting.arXiv preprint arXiv:2602.13600,

    Jiacheng Zhang, Feng Liu, Chao Du, and Tianyu Pang. AdaVBoost: Mitigating Hallucinations in LVLMs via Token-Level Adaptive Visual Attention Boosting.arXiv preprint arXiv:2602.13600,

    Pith/arXiv arXiv

  38. [38]

    Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs

    Qingru Zhang, Chandan Singh, Liyuan Liu, Xiaodong Liu, Bin Yu, Jianfeng Gao, and Tuo Zhao. Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs. InInternational Conference on Learning Representations (ICLR), 2024. 10

    2024

  39. [39]

    Adaptinfer: Adaptive Token Pruning for Vision-language Model Inference with Dynamical Text Guidance

    Weichen Zhang, Zhui Zhu, Ningbo Li, Shilong Tao, Kebin Liu, and Yunhao Liu. Adaptinfer: Adaptive Token Pruning for Vision-language Model Inference with Dynamical Text Guidance. arXiv preprint arXiv:2508.06084, 2025. 1, 2, 10

    arXiv 2025

  40. [40]

    Cross-modal Information Flow in Multimodal Large Language Models

    Zhi Zhang, Srishti Yadav, Fengze Han, and Ekaterina Shutova. Cross-modal Information Flow in Multimodal Large Language Models. InConference on Computer Vision and Pattern Recognition (CVPR), 2025. 1, 4, 9

    2025

  41. [41]

    The fruit is

    Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. Ibd: Alleviating Hallucinations in Large Vision-language Models via Image-biased Decoding. InConference on Computer Vision and Pattern Recognition (CVPR), 2025. 10 13 Appendix A Additional Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ....

    2025

  42. [42]

    Determine whether a fruit answer was provided

  43. [43]

    Determine whether the fruit answer is correct

  44. [44]

    Determine whether a math answer was provided

  45. [45]

    Determine whether the math answer is correct

  46. [46]

    Extract the final fruit prediction committed by the model

  47. [47]

    Extract the final numeric/math answer committed by the model

  48. [48]

    Copy the exact phrase corresponding to the final fruit answer and normalize it to a canonical label if possible

  49. [49]

    Copy the exact phrase corresponding to the final math answer and normalize it to a canonical form if possible

  50. [50]

    I see no fruit... it is a cherry

    If the fruit prediction is incorrect, assign one error category: - CROSS: Model is misdirected by terms/nouns from the math puzzle. - DENIAL: Model explicitly denies fruit presence or refuses to answer. - RESCUE: Model shows conflict or self-correction (e.g., “I see no fruit... it is a cherry” or “Apple, no, Banana”). - PRIOR: Simple misidentification or ...

  51. [51]

    fruit_answered

    Provide a brieferror_explanation. Output Format (STRICT JSON) { "fruit_answered": "True/False", "fruit_correct": "True/False", "math_answered": "True/False", "math_correct": "True/False", "predicted_fruit": "string", "predicted_math": "string", "final_fruit_span": "string", "final_math_span": "string", "fruit_error_profiling": "CROSS/DENIAL/RESCUE/PRIOR/O...

  52. [52]

    True” or “False

    Visual Check: Determine whether the model correctly identifies {expected_image_relation} as the spatial relation present in the image. Output “True” or “False”

  53. [53]

    True” or “False

    Text Check: Determine whether the model correctly identifies {expected_text_relation} as the spatial relation described in the text. Output “True” or “False”

  54. [54]

    visual_check

    Alignment Classification: -correct: Both image and text relations are correctly identified. -incorrect - both image: Model claims both modalities show or indicate the image relation. -incorrect - both text: Model claims both modalities show or indicate the text relation. -incorrect - flip: Model swaps them, using the text relation for the image part and t...