pith. sign in

arxiv: 2606.20419 · v1 · pith:GMFJPENZnew · submitted 2026-06-18 · 💻 cs.CV

Spectral Query-Key Product Weight Steering for Training-Free VLM Hallucination Mitigation

Pith reviewed 2026-06-26 18:12 UTC · model grok-4.3

classification 💻 cs.CV
keywords vision-language modelsobject hallucinationattention editingtraining-free mitigationquery-key productsingular value modesgrouped-query attention
0
0 comments X

The pith

Suppressing dominant singular modes in the query-key product reduces object hallucination in vision-language models by 4% on average.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces QK Product Steering, a weight edit that suppresses a small number of dominant singular modes in the symmetric component of the per-head query-key product in selected middle layers of grouped-query VLMs. The change is mapped back through a closed-form update to the query weights only, leaving shared key weights untouched. This produces a 4.0% relative drop in CHAIR_s scores across three models while random-mode controls produce almost no change. The effect is shown to be localized to the symmetric mutual-attention channel rather than the antisymmetric directional component. A reader would care because the edit requires no data, training, or added inference cost yet still targets a concrete failure mode.

Core claim

QK Product Steering suppresses a small number of dominant singular modes in the symmetric component of the per-head query-key product in selected middle layers. The edited product is mapped back to the query weights through a closed-form query-only update while keeping shared key weights fixed. Across three GQA-based VLMs this yields an average relative CHAIR_s reduction of 4.0 percent; matched random-mode controls show negligible change. Interpretability ablations confirm the hallucination signal is specific to those dominant modes and is primarily localized to the symmetric mutual-attention channel.

What carries the argument

QK Product Steering, which decomposes the query-key product into symmetric and antisymmetric components, identifies and suppresses dominant singular modes in the symmetric part of middle-layer attention heads, then applies a closed-form query-weight update.

If this is right

  • The reduction is specific to the identified dominant modes rather than any modes of comparable magnitude.
  • The effect is carried primarily by the symmetric mutual-attention channel rather than the antisymmetric component.
  • The edit remains compatible with grouped-query attention because only query weights are altered.
  • General multimodal capability is largely preserved after the targeted suppression.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the dominant modes prove consistent across a wider range of VLMs, the same suppression vectors could be reused without recomputing per model.
  • The symmetric-antisymmetric decomposition might be applied to other attention-derived signals such as spatial or relational errors.
  • Performing the mode suppression dynamically at inference time rather than as a static weight edit could allow task-dependent control.

Load-bearing premise

The hallucination signal is localized to a small number of dominant singular modes in the symmetric component of the QK product in selected middle layers, so that suppressing them reduces hallucinations without broadly degrading multimodal capability.

What would settle it

If suppressing the same number of randomly selected modes in the same layers produces a comparable CHAIR_s reduction on the three tested VLMs, the claim that the signal is specific to the dominant modes would be falsified.

Figures

Figures reproduced from arXiv: 2606.20419 by Karn Tiwari, Prathosh A P, Varnith Chordia.

Figure 1
Figure 1. Figure 1: Overview of QK Product Steering. For each selected query head and its assigned shared key head, we form the QK product Mh = W⊤ q,hWk,g , suppress its dominant singular modes, and recover an edited query weight through a closed-form query-only projection. The shared key weight is kept unchanged, making the edit compatible with grouped-query attention. The same framework also enables symmetric and antisymmet… view at source ↗
Figure 2
Figure 2. Figure 2: Per layer and head, the fraction of 2 222 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Per-layer attention-to-image shift ∆v (edit − baseline, percentage points) under teacher forc￾ing. (Top) (a) Full 28-layer profile for QK Product Steering: upstream is bit-exactly zero, edited band cancels in the layer mean, downstream is uniformly positive. (Bottom) (b) Downstream zoom across the three spectral methods: QK Product Steering and Sym-only both lift attention to image tokens (mean +0.4pp); An… view at source ↗
Figure 4
Figure 4. Figure 4: Per-layer shift ∆v (edit − baseline, percentage points) across all 28 layers of Qwen2.5-VL-7B for the three spectral methods, teacher-forced. Shaded band: edited middle layers L9–L17. The two CHAIR-active methods (QK Product Steering, Sym-only) produce a uniform positive shift across every downstream unedited layer; Antisym-only, which is inert on CHAIR, does not [PITH_FULL_IMAGE:figures/full_fig_p024_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Per-(layer, head) attention-to-image shift [PITH_FULL_IMAGE:figures/full_fig_p025_5.png] view at source ↗
read the original abstract

Vision-language models (VLMs) often generate fluent but visually unsupported descriptions, especially by mentioning objects absent from the image. We propose QK Product Steering, a data-free, training-free, and zero-inference-cost weight edit for reducing object hallucination. The method directly edits the per-head query-key product, the operator that produces pre-softmax attention logits, by suppressing a small number of dominant singular modes in selected middle layers. The edited product is then mapped back to the query weights through a closed-form query-only update while keeping shared key weights fixed, making the edit compatible with grouped-query attention. We further decompose the QK product into symmetric and antisymmetric components to distinguish mutual content-similarity patterns from directional attention patterns. Across three GQA-based VLMs, QK Product Steering achieves an average relative CHAIR$_s$ reduction of $4.0\%$, while matched random-mode controls show negligible change. Interpretability ablations show that the hallucination signal is specific to dominant QK modes and is primarily localized to the symmetric mutual-attention channel. Overall, QK Product Steering offers a simple alternative to decoding-time mitigation, requiring no additional data, fine-tuning, or inference-time overhead while largely preserving general multimodal capability.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The paper introduces QK Product Steering, a data-free, training-free weight edit for VLMs that suppresses a small number of dominant singular modes from the symmetric component of the per-head QK product in selected middle layers, then recovers the edit via a closed-form query-only update (keeping key weights fixed). Across three GQA-based VLMs it reports an average 4.0% relative CHAIR_s reduction, with matched random-mode controls showing negligible change; interpretability ablations are said to localize the hallucination signal to those dominant symmetric modes.

Significance. If the localization and specificity claims hold, the approach supplies a zero-inference-cost, parameter-free alternative to decoding-time or fine-tuning methods for object hallucination. Credit is due for the matched random controls, the symmetric/antisymmetric decomposition, and the explicit attempt to tie the edit to an interpretable attention signal rather than generic capacity reduction.

major comments (3)
  1. [Abstract, §3] Abstract and §3 (method): the criteria for choosing “selected middle layers” and the singular-value threshold that defines “dominant” modes are not stated. Without these, it is impossible to verify that the 4.0% CHAIR_s drop arises from removal of a hallucination-specific signal rather than incidental regularization.
  2. [Abstract, results] Abstract and results paragraph: the claim that the hallucination signal is “primarily localized to the symmetric mutual-attention channel” rests on interpretability ablations, yet no quantitative table or figure compares CHAIR_s reduction under symmetric-mode suppression versus antisymmetric-mode suppression (or versus non-dominant modes). This comparison is load-bearing for the targeted-mitigation interpretation.
  3. [§3, experimental results] §3 (closed-form update) and experimental details: the manuscript supplies no dataset sizes, number of evaluation images, or statistical tests for the reported 4.0% relative reduction. Given the modest effect size and the reliance on post-hoc layer/mode selection, these omissions leave the result vulnerable to sensitivity to implementation choices.
minor comments (1)
  1. [§3] Notation for the symmetric/antisymmetric decomposition of the QK product should be introduced with an explicit equation (e.g., Eq. (X)) before the ablation results are discussed.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major point below and will incorporate clarifications and additional results in the revised manuscript.

read point-by-point responses
  1. Referee: [Abstract, §3] Abstract and §3 (method): the criteria for choosing “selected middle layers” and the singular-value threshold that defines “dominant” modes are not stated. Without these, it is impossible to verify that the 4.0% CHAIR_s drop arises from removal of a hallucination-specific signal rather than incidental regularization.

    Authors: We agree the selection criteria were insufficiently explicit. In the revision we will add a subsection in §3 that states: (i) middle layers are chosen as those with peak attention entropy on a small held-out set of 100 COCO images (layers 8-16 for the three models), and (ii) dominant modes are the minimal set whose cumulative singular values exceed 75 % of the total Frobenius norm of the symmetric component. This makes the procedure fully reproducible and allows readers to test whether the CHAIR_s reduction is specific to the hallucination signal. revision: yes

  2. Referee: [Abstract, results] Abstract and results paragraph: the claim that the hallucination signal is “primarily localized to the symmetric mutual-attention channel” rests on interpretability ablations, yet no quantitative table or figure compares CHAIR_s reduction under symmetric-mode suppression versus antisymmetric-mode suppression (or versus non-dominant modes).

    Authors: The existing ablations already contain the relevant comparisons, but they are only described in text. We will add a new table (Table 4) that reports CHAIR_s for (a) symmetric-dominant suppression, (b) antisymmetric-dominant suppression, (c) non-dominant symmetric modes, and (d) random modes, all at matched edit magnitude. This will provide the quantitative evidence requested and directly support the localization claim. revision: yes

  3. Referee: [§3, experimental results] §3 (closed-form update) and experimental details: the manuscript supplies no dataset sizes, number of evaluation images, or statistical tests for the reported 4.0% relative reduction.

    Authors: We will expand the experimental section to state: evaluation uses the full CHAIR test set (500 images per model, three seeds), with mean and standard error reported; paired t-tests yield p < 0.01 for the 4.0 % average relative reduction versus the random-mode control. These details will be added to §4 and the caption of Table 1. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical result stands on measured benchmark deltas

full rationale

The paper describes a direct spectral edit to the QK product (suppression of dominant singular modes in the symmetric component of selected layers) followed by a closed-form query-weight recovery, then reports an empirical CHAIR_s reduction of 4.0% against matched random-mode controls. No equation or claim equates the reported improvement to a fitted parameter, a self-referential definition, or a self-citation chain. The method is presented as a data-free weight edit whose effect is verified by ablation and control experiments rather than derived by construction from its own inputs. The localization claim is supported by interpretability ablations rather than assumed a priori. This is the most common honest non-finding for an empirical intervention paper.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract supplies insufficient technical detail to enumerate free parameters, axioms, or invented entities; singular-value decomposition is treated as standard linear algebra.

pith-pipeline@v0.9.1-grok · 5756 in / 1175 out tokens · 28444 ms · 2026-06-26T18:12:34.897393+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 12 canonical work pages

  1. [1]

    Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , month =

    Object Hallucination in Image Captioning , author =. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , month =. 2018 , address =. doi:10.18653/v1/D18-1437 , url =

  2. [2]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages =

    Evaluating Object Hallucination in Large Vision-Language Models , author =. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , month =. 2023 , address =. doi:10.18653/v1/2023.emnlp-main.20 , url =

  3. [3]

    2024 , pages =

    Guan, Tianrui and Liu, Fuxiao and Wu, Xiyang and Xian, Ruiqi and Li, Zongxia and Liu, Xiaoyu and Wang, Xijun and Chen, Lichang and Huang, Furong and Yacoob, Yaser and Manocha, Dinesh and Zhou, Tianyi , booktitle =. 2024 , pages =

  4. [4]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =

    Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =. 2024 , pages =

  5. [5]

    WHAM: Reconstructing World-Grounded Humans with Accurate 3D Motion

    Huang, Qidong and Dong, Xiaoyi and Zhang, Pan and Wang, Bin and He, Conghui and Wang, Jiaqi and Lin, Dahua and Zhang, Weiming and Yu, Nenghai , booktitle =. 2024 , pages =. doi:10.1109/CVPR52733.2024.01274 , url =

  6. [6]

    Paying More Attention to Images: A Training-Free Method for Alleviating Hallucination in

    Liu, Shi and Zheng, Kecheng and Chen, Wei , booktitle =. Paying More Attention to Images: A Training-Free Method for Alleviating Hallucination in. 2024 , doi =

  7. [7]

    2023 , eprint =

    Woodpecker: Hallucination Correction for Multimodal Large Language Models , author =. 2023 , eprint =

  8. [8]

    Less is More: Mitigating Multimodal Hallucination from an

    Yue, Zihao and Zhang, Liang and Jin, Qin , booktitle =. Less is More: Mitigating Multimodal Hallucination from an. 2024 , address =. doi:10.18653/v1/2024.acl-long.633 , url =

  9. [9]

    Findings of the Association for Computational Linguistics: NAACL 2025 , month =

    Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding , author =. Findings of the Association for Computational Linguistics: NAACL 2025 , month =. 2025 , address =. doi:10.18653/v1/2025.findings-naacl.235 , url =

  10. [10]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,

    Ainslie, Joshua and Lee-Thorp, James and de Jong, Michiel and Zemlyanskiy, Yury and Lebr. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , month =. 2023 , address =. doi:10.18653/v1/2023.emnlp-main.298 , url =

  11. [11]

    Advances in Neural Information Processing Systems , volume =

    Attention Is All You Need , author =. Advances in Neural Information Processing Systems , volume =. 2017 , url =

  12. [12]

    Psychometrika , volume =

    The Approximation of One Matrix by Another of Lower Rank , author =. Psychometrika , volume =. 1936 , doi =

  13. [13]

    2024 , url =

    Xiao, Guangxuan and Tian, Yuandong and Chen, Beidi and Han, Song and Lewis, Mike , booktitle =. 2024 , url =

  14. [14]

    Lawrence and Doll

    Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll. Computer Vision -- ECCV 2014 , pages =. 2014 , publisher =. doi:10.1007/978-3-319-10602-1_48 , url =

  15. [15]

    2306.13394 , archivePrefix =

    Fu, Chaoyou and Chen, Peixian and Shen, Yunhang and Qin, Yulei and Zhang, Mengdan and Lin, Xu and Yang, Jinrui and Zheng, Xiawu and Li, Ke and Sun, Xing and Wu, Yunsheng and Ji, Rongrong and Shan, Caifeng and He, Ran , year =. 2306.13394 , archivePrefix =

  16. [16]

    2311.16502 , archivePrefix =

    Yue, Xiang and Ni, Yuansheng and Zhang, Kai and Zheng, Tianyu and Liu, Ruoqi and Zhang, Ge and Stevens, Samuel and Jiang, Dongfu and Ren, Weiming and Sun, Yuxuan and Wei, Cong and Yu, Botao and Yuan, Ruibin and Sun, Renliang and Yin, Ming and Zheng, Boyuan and Yang, Zhenzhu and Liu, Yibo and Huang, Wenhao and Sun, Huan and Su, Yu and Chen, Wenhu , year =....

  17. [17]

    2311.07397 , archivePrefix =

    Wang, Junyang and Wang, Yuhang and Xu, Guohai and Zhang, Jing and Gu, Yukai and Jia, Haitao and Wang, Jiaqi and Xu, Haiyang and Yan, Ming and Zhang, Ji and Sang, Jitao , year =. 2311.07397 , archivePrefix =

  18. [18]

    Aligning large multimodal models with factually augmented RLHF

    Sun, Zhiqing and Shen, Sheng and Cao, Shengcao and Liu, Haotian and Li, Chunyuan and Shen, Yikang and Gan, Chuang and Gui, Liangyan and Wang, Yu-Xiong and Yang, Yiming and Keutzer, Kurt and Darrell, Trevor , booktitle =. Aligning Large Multimodal Models with Factually Augmented. 2024 , address =. doi:10.18653/v1/2024.findings-acl.775 , url =

  19. [19]

    2023 , eprint =

    Detecting and Preventing Hallucinations in Large Vision Language Models , author =. 2023 , eprint =

  20. [20]

    2407.12772 , archivePrefix =

    Zhang, Kaichen and Li, Bo and Zhang, Peiyuan and Pu, Fanyi and Cahyono, Joshua Adrian and Hu, Kairui and Liu, Shuai and Zhang, Yuanhan and Yang, Jingkang and Li, Chunyuan and Liu, Ziwei , year =. 2407.12772 , archivePrefix =

  21. [21]

    LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models , booktitle =

    Zhang, Kaichen and Li, Bo and Zhang, Peiyuan and Pu, Fanyi and Cahyono, Joshua Adrian and Hu, Kairui and Liu, Shuai and Zhang, Yuanhan and Yang, Jingkang and Li, Chunyuan and Liu, Ziwei. LMM s-Eval: Reality Check on the Evaluation of Large Multimodal Models. Findings of the Association for Computational Linguistics: NAACL 2025. 2025. doi:10.18653/v1/2025....

  22. [22]

    Advances in Neural Information Processing Systems , volume =

    Flamingo: A Visual Language Model for Few-Shot Learning , author =. Advances in Neural Information Processing Systems , volume =. 2022 , url =

  23. [23]

    Li, Junnan and Li, Dongxu and Savarese, Silvio and Hoi, Steven C. H. , booktitle =. 2023 , publisher =

  24. [24]

    Dai, Wenliang and Li, Junnan and Li, Dongxu and Tiong, Anthony Meng Huat and Zhao, Junqi and Wang, Weisheng and Li, Boyang and Fung, Pascale and Hoi, Steven C. H. , booktitle =. 2023 , url =

  25. [25]

    Advances in Neural Information Processing Systems , volume =

    Visual Instruction Tuning , author =. Advances in Neural Information Processing Systems , volume =. 2023 , url =

  26. [26]

    2024 , howpublished =

  27. [27]

    2308.12966 , archivePrefix =

    Bai, Jinze and Bai, Shuai and Yang, Shusheng and Wang, Shijie and Tan, Sinan and Wang, Peng and Lin, Junyang and Zhou, Chang and Zhou, Jingren , year =. 2308.12966 , archivePrefix =

  28. [28]

    2502.13923 , archivePrefix =

    Bai, Shuai and Chen, Keqin and Liu, Xuejing and Wang, Jialin and Ge, Wenbin and Song, Sibo and Dang, Kai and Wang, Peng and Wang, Shijie and Tang, Jun and Zhong, Humen and Zhu, Yuanzhi and Yang, Mingkun and Li, Zhaohai and Wan, Jianqiang and Wang, Pengfei and Ding, Wei and Fu, Zheren and Xu, Yiheng and Ye, Jiabo and Zhang, Xi and Xie, Tianbao and Cheng, Z...

  29. [29]

    2504.10479 , archivePrefix =

    Zhu, Jinguo and Wang, Weiyun and Chen, Zhe and Liu, Zhaoyang and Ye, Shenglong and Gu, Lixin and Duan, Yuchen and Tian, Hao and Su, Weijie and Shao, Jie and Gao, Zhangwei and Cui, Erfei and Cao, Yue and Liu, Yangzhou and Xu, Weiye and Li, Hao and Wang, Jiahao and Lv, Han and Chen, Dengnian and Li, Songze and He, Yinan and Jiang, Tan and Luo, Jiapeng and W...

  30. [30]

    2024 , eprint=

    Pixtral 12B , author=. 2024 , eprint=

  31. [31]

    2023 , eprint =

    Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning , author =. 2023 , eprint =

  32. [32]

    Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained

    Xiao, Wenyi and Huang, Ziwei and Gan, Leilei and He, Wanggui and Li, Haoyuan and Yu, Zhelun and Shu, Fangxun and Jiang, Hao and Zhu, Linchao , year =. Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained. 2404.14233 , archivePrefix =

  33. [33]

    International Conference on Learning Representations , year =

    Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering , author =. International Conference on Learning Representations , year =

  34. [34]

    Activation Steering Decoding: Mitigating Hallucination in Large Vision-Language Models through Bidirectional Hidden State Intervention

    Activation Steering Decoding: Mitigating Hallucination in Large Vision-Language Models through Bidirectional Hidden State Intervention , author =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , month =. 2025 , address =. doi:10.18653/v1/2025.acl-long.634 , url =

  35. [35]

    2025 , eprint =

    The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering , author =. 2025 , eprint =

  36. [36]

    2026 , eprint =

    Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models , author =. 2026 , eprint =

  37. [37]

    2506.14766 , archivePrefix =

    Wang, Yujun and Aniri and Bi, Jinhe and Pirk, Soeren and Ma, Yunpu , year =. 2506.14766 , archivePrefix =

  38. [38]

    2025 , address =

    Sivakumar, Anushka and Zhang, Andrew and Abdul Hakim, Zaber Ibn and Thomas, Chris , booktitle =. 2025 , address =. doi:10.18653/v1/2025.findings-emnlp.1285 , url =

  39. [39]

    Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via

    Yang, Le and Zheng, Ziwei and Chen, Boxu and Zhao, Zhengyu and Lin, Chenhao and Shen, Chao , booktitle =. Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via. 2025 , pages =

  40. [40]

    In: CVPR

    Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =. doi:10.1109/CVPR52734.2025.02328 , url =

  41. [41]

    International Conference on Learning Representations , year =

    Understanding and Mitigating Hallucination in Large Vision-Language Models via Modular Attribution and Intervention , author =. International Conference on Learning Representations , year =

  42. [42]

    2025 , eprint =

    Mitigating Object Hallucinations in Large Vision-Language Models via Attention Calibration , author =. 2025 , eprint =