pith. sign in

arxiv: 2605.27993 · v1 · pith:IKIQ375Nnew · submitted 2026-05-27 · 💻 cs.CL

Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigation

Jingwen Wu , Xijun Zhang , Ge Song This is my paper

Pith reviewed 2026-06-29 13:27 UTC · model grok-4.3

classification 💻 cs.CL
keywords object hallucinationmultimodal large language modelsactivation steeringcontext preference vectorsinference-time mitigationvisual reliancehallucination reduction
0
0 comments X

The pith

Context-preference vectors from conflict samples steer MLLMs to reduce object hallucinations by balancing image, text, and parametric reliance.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper challenges the view that object hallucinations in multimodal large language models stem only from visual neglect. Systematic interventions reveal that increasing visual reliance can worsen hallucinations on some models while decreasing it can reduce them, showing that the image context competes with textual input and the model's internal knowledge. This leads to the proposal of a training-free method that extracts two distinct vectors from small conflict sample sets and injects them with signed residuals at mid-early MLP layers to control which information source dominates. Experiments confirm the approach lowers hallucinations while keeping generation speed and text quality unchanged.

Core claim

Object hallucinations arise from imbalanced competition among the image as context, the textual context, and the model's parametric knowledge rather than from visual insufficiency alone. Extracting two semantically distinct Context Preference Vectors from small sets of designed conflict samples and applying them via single-pass signed residual injection at mid-early MLP layers during inference allows selective control over information reliance, thereby mitigating hallucinations.

What carries the argument

Context Preference Vectors (CPVs), two vectors extracted from designed conflict samples and injected with signed residuals at mid-early MLP layers to modulate reliance on competing information sources.

If this is right

  • CAS substantially mitigates object hallucinations on multiple MLLMs.
  • The method adds no decoding latency.
  • Native text-generation quality is preserved.
  • The framework requires no training or parameter updates.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same conflict-sample vector extraction could be tested on other multimodal tasks such as visual question answering to control different forms of inconsistency.
  • Injection at MLP layers might be compared against attention-layer steering to identify which component most directly affects context competition.
  • The approach suggests that preference vectors derived from minimal samples could extend to pure-language models for controlling factual versus creative output balance.

Load-bearing premise

Two small sets of designed conflict samples suffice to extract semantically distinct vectors whose signed injection at mid-early MLP layers reliably controls reliance without side effects on generation quality.

What would settle it

Observing that signed CPV injection fails to reduce object hallucinations or increases latency on additional MLLMs or standard benchmarks would show the control mechanism does not generalize.

Figures

Figures reproduced from arXiv: 2605.27993 by Ge Song, Jingwen Wu, Xijun Zhang.

Figure 1
Figure 1. Figure 1: CHAIRI of LLaVA-1.5 and Qwen-VL un￾der VFV and MRV interventions. This reveals three phenomena: (i) steering toward more image context on either CPV may instead exacerbate hallucinations; (ii) the response slopes across the two CPVs can take op￾posite signs within the same model; (iii) on the same CPV, different models exhibit divergent trends. In the literature, given the prohibitive cost of re￾training a… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the CAS framework. Left: Extracting Context Preference Vectors (offline). Construct two types of context-conflict setups, and read the layer-l MLP output together with the first-token logit difference pref, and obtain v (l) vfv (VFV) and v (l) mrv (MRV) via layer-wise ridge regression. Right: Activation Steering (online inference). At inference, we apply an additive residual γ(t) · (α v (l) vfv… view at source ↗
Figure 3
Figure 3. Figure 3: CHAIRI under VFV-only and MRV-only intervention. Each subplot corresponds to one base model and reports CHAIRI across signed intensities α (VFV) and β (MRV); the dashed line marks that model’s vanilla CHAIRI . Full numbers are in Ap￾pendix Tables 6 and 7. intervention at L11–14 in all experiments. 5 Conclusion This paper challenges the prevailing assumption that MLLM object hallucinations stem solely from … view at source ↗
Figure 4
Figure 4. Figure 4: Repetitive degeneration of PAI on LLaVA-1.5. A single hallucinated attribute (“black handle”) is repeated about 29 times, artificially inflating the mention count of GT-grounded objects; the token-level CHAIRI is therefore diluted by an inflated denominator and reads artificially low even though the caption has already degenerated. CAS produces a clean, repetition-free caption on the same image. Input Imag… view at source ↗
Figure 5
Figure 5. Figure 5: Repetitive degeneration of AttnReal. The same CHAIRI deflation pattern as [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
read the original abstract

Object hallucination remains a primary obstacle to the reliable deployment of Multimodal Large Language Models (MLLMs). Current inference-time mitigation methods mainly assume hallucinations stem from visual neglect, steering models to enhance visual reliance. In contrast, our systematic interventions on multiple MLLMs show that pushing toward more visual reliance may exacerbate hallucinations on some models, while less may mitigate hallucinations. This result suggests that attributing hallucinations solely to visual insufficiency is underdetermined. We argue that the image, as a context, simultaneously competes with the model's parametric knowledge and the textual context. For this, we propose a training-free framework, Context-Preference Activation Steering (CAS). It extracts two semantically distinct Context Preference Vectors (CPVs) via two small sets of designed conflict samples and applies them via single-pass signed residual injection at mid-early MLP layers during inference to control information reliance. Experiments show that CAS substantially mitigates object hallucinations without increasing decoding latency and preserves native text-generation quality.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript argues that object hallucinations in MLLMs cannot be attributed solely to visual neglect, because systematic interventions show that increasing visual reliance exacerbates hallucinations on some models while decreasing it can mitigate them. It proposes Context-Preference Activation Steering (CAS), a training-free method that extracts two semantically distinct Context Preference Vectors (CPVs) from small sets of designed conflict samples and applies signed residual injection at mid-early MLP layers to control reliance between image context, parametric knowledge, and text, thereby reducing hallucinations without added decoding latency or loss of native text-generation quality.

Significance. If the empirical results hold, the work usefully reframes hallucination mitigation away from uniform visual-enhancement strategies toward context-preference control. The training-free, sample-driven construction of CPVs and the single-pass signed injection are practical strengths that could be adopted quickly; the finding that visual steering is not uniformly beneficial is a falsifiable claim worth testing across additional models.

major comments (2)
  1. [§3.2] §3.2 (CPV extraction): the claim that two small sets of designed conflict samples suffice to produce semantically distinct CPVs whose signed mid-early MLP injection reliably controls information reliance rests on the weakest assumption; the manuscript must demonstrate robustness to sample choice, explicit validation of semantic distinctness, and absence of unintended side-effects on generation quality.
  2. [§4] §4 (experimental evaluation): the central claim that CAS substantially mitigates object hallucinations on multiple MLLMs requires the full list of models, datasets, quantitative metrics (e.g., hallucination rates, latency, text-quality scores), and controls; without these the support for the reframing and the method cannot be assessed.
minor comments (2)
  1. Notation for CPVs and the signed residual injection should be defined once with a clear equation or pseudocode block to avoid ambiguity in later sections.
  2. Figure captions and table headers should explicitly state the number of conflict samples used and the exact injection layers to improve reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback and positive assessment of the work's potential impact. We address the two major comments point-by-point below. Where additional evidence or clarification is warranted, we will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: [§3.2] §3.2 (CPV extraction): the claim that two small sets of designed conflict samples suffice to produce semantically distinct CPVs whose signed mid-early MLP injection reliably controls information reliance rests on the weakest assumption; the manuscript must demonstrate robustness to sample choice, explicit validation of semantic distinctness, and absence of unintended side-effects on generation quality.

    Authors: We agree that explicit robustness checks would strengthen the section. In revision we will add: (i) results across 3–5 independently sampled conflict sets per model, reporting mean and variance in hallucination reduction; (ii) cosine-similarity and nearest-neighbor analyses confirming the two CPVs are semantically distinct; and (iii) text-quality metrics (perplexity on held-out captions, win-rate on non-hallucination prompts) showing no degradation. These additions directly address the concern without altering the core method. revision: yes

  2. Referee: [§4] §4 (experimental evaluation): the central claim that CAS substantially mitigates object hallucinations on multiple MLLMs requires the full list of models, datasets, quantitative metrics (e.g., hallucination rates, latency, text-quality scores), and controls; without these the support for the reframing and the method cannot be assessed.

    Authors: The current manuscript reports results on LLaVA-1.5, MiniGPT-4, and InstructBLIP using POPE and CHAIR, with hallucination-rate reductions, latency measurements (no increase), and preservation of text-generation quality. To make the evaluation fully transparent we will expand §4 with a consolidated table listing every model, dataset, exact metric values, and all controls (layer ablations, sign-flip controls, and comparison to visual-steering baselines). This is a straightforward expansion rather than new experiments. revision: yes

Circularity Check

0 steps flagged

No significant circularity; method is empirical and sample-driven

full rationale

The paper presents CAS as a training-free, inference-time framework that extracts Context Preference Vectors from two small sets of designed conflict samples and applies signed residual injection at MLP layers. No equations, derivations, or fitted parameters are described that reduce the claimed hallucination mitigation to a self-definition, a renamed input, or a self-citation chain. The central claims rest on systematic interventions and empirical results rather than any load-bearing mathematical reduction to the method's own construction. The approach is self-contained against external benchmarks of hallucination metrics and generation quality.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The approach rests on the domain assumption that image context competes with parametric knowledge and text; the paper introduces CPVs as a new construct without independent evidence outside the proposed method.

axioms (1)
  • domain assumption The image, as a context, simultaneously competes with the model's parametric knowledge and the textual context.
    Explicitly stated in the abstract as the basis for moving beyond visual-neglect explanations.
invented entities (1)
  • Context Preference Vectors (CPVs) no independent evidence
    purpose: Represent preference for one information source over others to enable signed steering of MLLM activations.
    Newly defined construct extracted from conflict samples and injected during inference.

pith-pipeline@v0.9.1-grok · 5696 in / 1335 out tokens · 54223 ms · 2026-06-29T13:27:04.881153+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 30 canonical work pages · 11 internal anchors

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. https://arxiv.org/abs/2308.12966 Qwen- VL : A versatile vision-language model for understanding, localization, text reading, and beyond . arXiv preprint arXiv:2308.12966

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  4. [4]

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. 2023. https://arxiv.org/abs/2306.15195 Shikra: Unleashing multimodal LLM 's referential dialogue magic . arXiv preprint arXiv:2306.15195

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  5. [5]

    Wei Chen, Xin Yan, Bin Wen, Fan Yang, Tingting Gao, Di Zhang, and Long Chen. 2025. https://arxiv.org/abs/2504.08809 Decoupling contrastive decoding: Robust hallucination mitigation in multimodal large language models . arXiv preprint arXiv:2504.08809

    work page arXiv 2025

  6. [6]

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2024. https://arxiv.org/abs/2309.03883 DoLa : Decoding by contrasting layers improves factuality in large language models . In Proceedings of the Twelfth International Conference on Learning Representations (ICLR)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  7. [7]

    Alberto Compagnoni, Davide Caffagni, Nicholas Moratelli, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. 2025. https://arxiv.org/abs/2508.20181 Mitigating hallucinations in multimodal llms via object-aware preference optimization . arXiv preprint arXiv:2508.20181

    work page arXiv 2025

  8. [8]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. https://arxiv.org/abs/2305.06500 Instruct BLIP : Towards general-purpose vision-language models with instruction tuning . In Advances in Neural Information Processing Systems (NeurIPS)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  9. [9]

    Jiawei Feng, Jiancan Wu, Xingyu Zhu, Junkang Wu, Xiang Wang, and Xiangnan He. 2026. https://openreview.net/forum?id=uZ5AmOJKqV PEA-DPO : Perception-enhanced alignment direct preference optimization for MLLMs alignment . Submitted to ICLR 2026

    2026

  10. [10]

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. https://arxiv.org/abs/2012.14913 Transformer feed-forward layers are key-value memories . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  11. [11]

    Zhenglin Hua, Jinghan He, Zijun Yao, Tianxu Han, Haiyun Guo, Yuheng Jia, and Junfeng Fang. 2025. https://arxiv.org/abs/2505.16146 Steering LVLMs via sparse autoencoder for hallucination mitigation . arXiv preprint arXiv:2505.16146

    work page arXiv 2025

  12. [12]

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. https://arxiv.org/abs/2311.17911 OPERA : Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    work page arXiv 2024

  13. [13]

    Junho Kim, Hyunjun Kim, Yeonju Kim, and Yong Man Ro. 2024. https://arxiv.org/abs/2406.01920 CODE : Contrasting self-generated description to combat hallucination in large multi-modal models . arXiv preprint arXiv:2406.01920

    work page arXiv 2024

  14. [14]

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. https://arxiv.org/abs/2311.16922 Mitigating object hallucinations in large vision-language models through visual contrastive decoding . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    work page arXiv 2024

  15. [15]

    Qiming Li, Zekai Ye, Xiaocheng Feng, Weihong Zhong, Libo Qin, Ruihan Chen, Baohang Li, Kui Jiang, Yaowei Wang, Ting Liu, and Bing Qin. 2025. https://arxiv.org/abs/2506.23590 Cai: Caption-sensitive attention intervention for mitigating object hallucination in large vision-language models . arXiv preprint arXiv:2506.23590

    work page arXiv 2025

  16. [16]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.20 Evaluating object hallucination in large vision-language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 292--305, Singapore. Association for Computational Linguistics

    work page doi:10.18653/v1/2023.emnlp-main.20 2023

  17. [17]

    Microsoft COCO: Common Objects in Context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Doll\' a r. 2014. https://arxiv.org/abs/1405.0312 Microsoft COCO : Common objects in context . In Proceedings of the European Conference on Computer Vision (ECCV)

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  18. [18]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024 a . https://arxiv.org/abs/2310.03744 Improved baselines with visual instruction tuning . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Highlight

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  19. [19]

    Shi Liu, Kecheng Zheng, and Wei Chen. 2024 b . https://doi.org/10.1007/978-3-031-73010-8_8 Paying more attention to image: A training-free method for alleviating hallucination in LVLMs . In Proceedings of the European Conference on Computer Vision (ECCV)

    work page doi:10.1007/978-3-031-73010-8_8 2024

  20. [20]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. https://arxiv.org/abs/2202.05262 Locating and editing factual associations in GPT . In Advances in Neural Information Processing Systems (NeurIPS)

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  21. [21]

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. https://arxiv.org/abs/1809.02156 Object hallucination in image captioning . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), Brussels, Belgium. Association for Computational Linguistics

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  22. [22]

    Nan Sun, Zhenyu Zhang, Xixun Lin, Kun Wang, Yanmin Shang, Naibin Gu, Shuohuan Wang, Yu Sun, Hua Wu, Haifeng Wang, and Yanan Cao. 2025. https://arxiv.org/abs/2512.03542 V-iti: Mitigating hallucinations in multimodal large language models via visual inference-time intervention . arXiv preprint arXiv:2512.03542

    work page arXiv 2025

  23. [23]

    Chongjun Tu, Peng Ye, Dongzhan Zhou, Lei Bai, Gang Yu, Tao Chen, and Wanli Ouyang. 2025. https://arxiv.org/abs/2503.08342 Attention reallocation: Towards zero-cost and controllable hallucination mitigation of MLLMs . arXiv preprint arXiv:2503.08342

    work page arXiv 2025

  24. [24]

    Chenxi Wang, Xiang Chen, Ningyu Zhang, Bozhong Tian, Haoming Xu, Shumin Deng, and Huajun Chen. 2025 a . https://arxiv.org/abs/2410.11779 MLLM can see? dynamic correction decoding for hallucination mitigation . arXiv preprint arXiv:2410.11779

    work page arXiv 2025

  25. [25]

    Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, and Jitao Sang. 2024. https://arxiv.org/abs/2311.07397 AMBER : An LLM -free multi-dimensional benchmark for MLLMs hallucination evaluation . arXiv preprint arXiv:2311.07397

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  26. [26]

    Yujun Wang, Aniri, Jinhe Bi, Soeren Pirk, and Yunpu Ma. 2025 b . https://arxiv.org/abs/2506.14766 Ascd: Attention-steerable contrastive decoding for reducing hallucination in mllm . arXiv preprint arXiv:2506.14766

    work page arXiv 2025

  27. [27]

    Gonzalez, Trevor Darrell, and David M

    Tsung-Han Wu, Heekyung Lee, Jiaxin Ge, Joseph E. Gonzalez, Trevor Darrell, and David M. Chan. 2025. https://arxiv.org/abs/2504.13169 Generate, but verify: Reducing hallucination in vision-language models with retrospective resampling . arXiv preprint arXiv:2504.13169

    work page arXiv 2025

  28. [28]

    Shuo Xing, Peiran Li, Yuping Wang, Ruizheng Bai, Yueqi Wang, Chan-Wei Hu, Chengxuan Qian, Huaxiu Yao, and Zhengzhong Tu. 2025. https://doi.org/10.18653/v1/2025.emnlp-main.121 Re- A lign: Aligning vision language models via retrieval-augmented direct preference optimization . In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Pr...

    work page doi:10.18653/v1/2025.emnlp-main.121 2025

  29. [29]

    Jianghao Yin, Qin Chen, Kedi Chen, Jie Zhou, Xingjiao Wu, and Liang He. 2026. https://openreview.net/forum?id=YtWZdwEG5K Dynamic multimodal activation steering for hallucination mitigation in large vision-language models . In The Fourteenth International Conference on Learning Representations (ICLR)

    2026

  30. [30]

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. 2024. https://doi.org/10.1007/s11432-024-4251-x Woodpecker: Hallucination correction for multimodal large language models . Science China Information Sciences

    work page doi:10.1007/s11432-024-4251-x 2024

  31. [31]

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, and Tat-Seng Chua. 2024. https://arxiv.org/abs/2312.00849 RLHF-V : Towards trustworthy MLLMs via behavior alignment from fine-grained correctional human feedback . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern R...

    work page arXiv 2024

  32. [32]

    Ge Zheng, Jiaye Qian, Jiajin Tang, and Sibei Yang. 2025. https://arxiv.org/abs/2510.20229 Why LVLMs are more prone to hallucinations in longer responses: The role of context . In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

    work page arXiv 2025

  33. [33]

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. 2024. https://arxiv.org/abs/2310.00754 Analyzing and mitigating object hallucination in large vision-language models . In Proceedings of the Twelfth International Conference on Learning Representations (ICLR)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  34. [34]

    Xingyu Zhu, Kesen Zhao, Liang Yi, Shuo Wang, Zhicai Wang, Beier Zhu, and Hanwang Zhang. 2026. https://arxiv.org/abs/2602.24041 Look carefully: Adaptive visual reinforcements in multimodal large language models for hallucination mitigation . arXiv preprint arXiv:2602.24041

    work page arXiv 2026