Pith. sign in

REVIEW 4 major objections 8 minor 42 references

This paper claims that a large share of multimodal hallucinations come from a query-invariant structural bias in text-to-image attention, and that subtracting a saliency-weighted average attention template purifies the signal and reduces ha

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-07-31 23:14 UTC pith:3HOKYV54

load-bearing objection A genuinely useful but mechanistically under-verified attention intervention; the method works on benchmarks, but the paper's claim that it removes only structural noise is not established. the 4 major comments →

arxiv 2607.24017 v1 pith:3HOKYV54 submitted 2026-07-27 cs.CV cs.AI

Disentangling Semantic Attention from Structural Bias in the Attention Manifold

Pengkun Jiao , Bin Zhu , Jingjing Chen , Yu-gang Jiang This is my paper
classification cs.CV cs.AI
keywords multimodal large language modelshallucinationattentionstructural biasvisual attention sinksposterior collapselinguistic priorinference-time intervention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to show that when a vision-language model hallucinates—describing objects not in the image or yielding to a misleading prompt—a major cause is not the model's language skills but a fixed, query-invariant bias in how text tokens attend to image tokens. This bias acts like static noise that drowns out the true semantic alignment between text and image. The proposed intervention, SPAR, estimates the bias as the average attention each image token receives from all text tokens, subtracts it in a saliency-aware way, and redistributes the reclaimed probability mass to confident, informative regions. The authors argue that this restores the visual signal-to-noise ratio, preventing the model's posterior from collapsing toward its linguistic prior. If correct, a training-free two-layer modification should consistently reduce both spontaneous object hallucinations and induced gaslighting hallucinations across open-source models at roughly baseline speed.

Core claim

On the paper's own terms, the central discovery is Semantic-Structural Decoupling: observed text-to-image attention is a sum of latent semantic attention, a static query-invariant structural bias, and residual noise. The bias is estimated as the row-average of the attention matrix, lives mainly in the first two layers, and correlates with position rather than content. SPAR subtracts a saliency-modulated version of this template, then redistributes the freed probability mass using a sparsity-based confidence score. The authors report reductions in object hallucinations and large gains in post-gaslighting accuracy, with negligible latency.

What carries the argument

The load-bearing object is the Structural Bias Template: for each attention head, the average attention each image token receives across all text queries, Ā_j = (1/|T|) Σ_i A_{i,j}. The paper defines this row-mean as the query-invariant component of the attention manifold, estimates it in the first two layers, and subtracts it from each row via A^{pur}_{i,j} = ReLU(A_{i,j} − λ(1+βS_j)Ā_j), where S_j is a sigmoid of normalized feature variance that protects genuinely salient regions. A confidence-aware redistribution rescales the purified rows so that the reclaimed probability mass strengthens sparse, confident alignments rather than amplifying residual noise.

Load-bearing premise

The argument rests on the assumption that the average attention a visual token receives across all text queries equals structural bias and nothing else; if genuine semantic attention also contributes to that average, SPAR will remove real visual information.

What would settle it

Run SPAR on an image where a single object is the subject of many different questions; if accuracy on questions about that object drops relative to the unmodified model, the subtracted template contained semantic signal. More directly, compare the row-mean template on the same image with two independent sets of unrelated text queries: if the templates differ substantially, they are not query-invariant and are not pure structural bias.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A training-free, plug-and-play layer edit can reduce hallucinations across decoding methods, including methods already designed to reduce them.
  • Only the first two transformer layers need to be modified, so the intervention can run at near-baseline latency.
  • The same intervention addresses both spontaneous object hallucination and induced gaslighting hallucinations, unifying two phenomena under one mechanism.
  • Attention sinks in multimodal models are symptoms of a broader structural bias, so interventions targeting isolated sink tokens are incomplete.
  • Improving the visual signal-to-noise ratio in attention prevents the posterior from collapsing into the linguistic prior, limiting reliance on corpus statistics over image content.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the template is a row-mean, one testable extension is to precompute a per-position template on a blank or corpus image and apply it across images, removing the per-image averaging cost.
  • The saliency modulation predicts a boundary condition: if a genuinely informative object is also a frequent target of many queries, subtracting its row-average will suppress it; identifying that boundary would sharpen the method's applicability.
  • The same decomposition could extend to video or 3D token sequences, where a time- or space-averaged bias template may explain modality-specific hallucinations.
  • If structural bias is amplified by positional-encoding asymmetry and softmax normalization, architectural changes to those components might prevent the bias at the source, making inference-time purification unnecessary.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper proposes SPAR, a training-free, plug-and-play intervention on the first two Transformer layers' text-to-image attention of MLLMs. It hypothesizes that observed attention is an additive mixture of semantic attention and a query-invariant structural bias (Eq. 3), that this bias dilutes visual evidence and collapses the posterior toward the linguistic prior, and that this produces both spontaneous and gaslighting-induced hallucinations. SPAR estimates the bias template as the text-query average of the attention matrix (Eq. 11), subtracts a saliency-modulated version of it (Eq. 12), and redistributes the reclaimed probability budget with a sparsity-based confidence score. Experiments on POPE, CHAIR, GaslightingBench, MMMU, AI2Diagram, and MMBench with LLaVA-v1.5 and LLaVA-NeXT report consistent improvements over Regular decoding and compatibility with AGLA, ONLY, and VAR at low inference overhead.

Significance. If the mechanism were valid, SPAR would be a practically valuable contribution: it is training-free, uses only the first two layers, adds little latency, and improves both spontaneous and induced hallucination benchmarks. The evaluation is reasonably broad and includes combinations with three existing interventions, which is a strength. However, the paper's mechanistic claim is not established by the current formalization. The 'structural bias' template is defined as the query-average of the very matrix being purified, so the operation is a deterministic row-centering with unmeasured semantic leakage; the saliency gate appears to subtract most from high-variance tokens, which are often informative. The empirical gains may reflect a generic sharpening effect rather than the proposed decoupling, and lack of error bars and code further weakens the evidence. The theoretical 'posterior collapse' result is a continuity statement that does not specifically implicate structural bias.

major comments (4)
  1. [Section 3.2, Eqs. (2)-(3); Section 4.1, Eqs. (11)-(12)] Definition 1 defines A_struct(V) as E_q[A_obs(q,V)], so the decomposition in Eq. (3) is not an independent additive model. Averaging Eq. (3) over q gives (1-λ)A_struct = E_q[A*_sem] + E_q[ζ]; with the default λ=1.0 this forces E_q[A*_sem] = -E_q[ζ]. The query average in Eq. (11) is therefore contaminated by genuine semantic attention, and Eq. (12) subtracts part of the semantic signal. The paper provides no measurement showing E_q[A*_sem]≈0 or that object-token attention survives purification. The central 'decoupling' claim is unsupported.
  2. [Section 4.1, Eqs. (9)-(12)] S_j is monotonically increasing in feature variance, and the subtracted template is scaled by (1+βS_j). Thus high-variance visual tokens receive the largest subtraction. The text claims this 'avoid[s] harming genuine visual perception,' but no evidence is given that high-variance tokens are uninformative sinks rather than informative foreground objects. In standard image features, object tokens are often high-variance, so the sign of the saliency gate may amplify semantic erasure. Table 5 only reports end-task accuracy and cannot separate removal of structural bias from removal of misleading or shared semantic peaks.
  3. [Tables 1-3 and 5] The paper reports single-run point estimates without error bars, confidence intervals, or significance tests. Several headline gains are small (e.g., Table 2, LLaVA-v1.5 after-negation: SPAR 41.74 vs GasEraser 40.95; Table 1, VAR+SPAR F1 85.59 vs VAR 84.61). Without repeated runs or paired tests, these differences may be within run-to-run noise. No code or data release is provided, so the exact numbers cannot be independently verified.
  4. [Section 3.4, Corollary 1] The proof is a continuity argument: if L(y) becomes non-discriminative, the posterior converges to the linguistic prior. It never uses A_struct or λ beyond the definition of SNR, so it does not establish that the proposed structural-bias component causes posterior collapse. The corollary is compatible with any noise source; it cannot serve as evidence that the estimated query-average template is the cause of hallucination.
minor comments (8)
  1. [Figure 3(b)] The axis label reads 'CHIAR'; it should be 'CHAIR'.
  2. [Figure 12] The proposed method is labeled 'FAPR (ours)' throughout Figure 12, while the text and all other figures use 'SPAR'. This inconsistency should be corrected.
  3. [Table 1 caption vs Appendix D] Table 1 caption says CHAIR max generation length is 128, but Appendix D states max response length is 1,024. CHAIR scores are sensitive to caption length; please reconcile.
  4. [Section 5] The text says MME hallucination subset is used, but no MME results appear anywhere in the paper.
  5. [Appendix F.1] The appendix says three MLLMs are evaluated but lists only LLaVA-v1.5 and LLaVA-v1.6-Vicuna; the main tables use LLaVA-NeXT. Clarify which models are actually used.
  6. [Abstract] The abstract contains a stray '/githubProject Page' fragment with no actual code link.
  7. [Appendix Tables 10-12] The M3ID baseline appears in the detailed POPE tables but is not introduced in the baseline list or related work; add a reference or description.
  8. [Appendix B.1] The Pólya-urn analogy is presented as an explanation but is not formalized. If it is intended as a mechanism, it needs a concrete model; otherwise it should be labeled as intuition.

Circularity Check

2 steps flagged

SPAR's 'structural bias' is defined as the query-average of the very attention matrix it subtracts; Corollary 1 assumes the SNR-to-likelihood collapse it claims to prove. Benchmark gains are external, but the mechanistic explanation is circular.

specific steps
  1. self definitional [Definition 1/Eq. (2), Sec. 3.2; Eq. (11)-(12), Sec. 4.1]
    "Definition 1 (Structural Bias). We define structural bias, denoted by A_struct, as the query-invariant component of the attention manifold... A_struct(V) ≜ E_{q∼P(T)}[A_obs(q,V)] ≈ 1/N Σ_{i=1}^N A_obs(i,·) (2). ... Then we estimate Structural Bias Template ¯A by marginalizing over the textual dimension of A: ¯A_j = E_{i∼T}[A_{i,j}] ≈ 1/|T| Σ_{i=1}^{|T|} A_{i,j}. (11) ... A_pur_{i,j} = ReLU(A_{i,j} − λ·(1+βS_j)·¯A_j). (12)"

    The quantity removed in Eq. (12) is the same statistic used to define structural bias in Eq. (2): both are the query-average of the observed attention matrix. The purification is therefore a deterministic row-centering of the input, not a measurement of an independently characterized noise process. Under the paper's own additive model (Eq. 3), the row-mean also contains E_i[A*_sem(i,j)], so subtracting it removes query-shared semantic evidence (e.g., a salient object relevant to many questions) unless one defines 'semantic attention' as whatever remains after subtraction. The claim that only structural noise is removed is true by definition, not by evidence.

  2. other [Corollary 1, Sec. 3.4; Appendix A, Eqs. (7), (15)-(18)]
    "Corollary 1 (Posterior Collapse Toward the Linguistic Prior). Assume that, as SNR→0, the normalized visual likelihood L(y) becomes approximately non-discriminative across candidate tokens, i.e., there exists a continuous function ε(SNR)≥0 with lim_{SNR→0}ε(SNR)=0 such that sup_{y,y′∈Y}|L(y)−L(y′)|≤ε(SNR). (7) Then the posterior distribution approaches the linguistic prior... By assumption... L(y)=c+δ_y... the ratio converges uniformly to 1."

    The corollary's substantive claim—that low attention SNR makes the visual likelihood non-discriminative and thereby collapses the posterior to the linguistic prior—is literally assumed in Eq. (7). Appendix A only rewrites L(y)=c+δ_y and shows P(y|V,T)/P(y|T)→1, which is the same as the assumption. No model connects the energy ratio in Eq. (6) to discriminability of L(y). Thus the corollary does not derive posterior collapse from the attention decomposition; it restates the desired conclusion as a premise and then labels the restatement a proof.

full rationale

The benchmark sections (POPE, CHAIR, GaslightingBench, MMMU, AI2Diagram, MMBench) are externally evaluated and compared with other interventions, so the empirical result is not fabricated by the definitional machinery; this prevents a score of 8-10. The references to the authors' prior work (GasEraser [14], GaslightingBench [36], etc.) are used as baselines/benchmarks rather than as the justification for SPAR's validity, so no load-bearing self-citation is present. However, the central explanatory claim is circular in two places. First, structural bias is defined (Eq. 2) as the query-average of observed attention, estimated (Eq. 11) from the very matrix being purified, and subtracted (Eq. 12); hence 'purification' is row-centering by construction, and 'restoring authentic visual grounding' is an interpretation, not a measured removal of an independently defined noise. The saliency weighting does not fix this: S_j is large for high-variance feature tokens, and subtracting a saliency-amplified row-mean preferentially removes attention from those same informative tokens. Second, Corollary 1 assumes the SNR-to-visual-likelihood collapse it claims to prove; Appendix A only restates the assumption in ratio form. These are genuine reductions of the paper's mechanistic claims to their own definitions/assumptions, while the engineering contribution retains independent benchmark support. Overall score: 6.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

The central method depends on multiple fitted constants (λ, β, τ_sparsity, k, layer choice), an assumed additive decomposition, and an identification of structural bias with the row-mean of the attention matrix. The theoretical corollary does not reduce the number of free parameters because it is a consistency statement about the defined SNR. The only independently observable quantities are the benchmark accuracy numbers, and those are used both to select hyperparameters and to claim success, which is the main circularity burden.

free parameters (5)
  • λ (purification strength) = 1.0 (default)
    Tuned on GaslightingBench before/after negation accuracy (Table 8, Appendix G); values 0.8-1.4 trade off standard accuracy vs adversarial robustness.
  • β (saliency modulation) = 1.0 (default)
    Tuned on GaslightingBench; 0.8-1.2 scanned in sensitivity analysis.
  • τ_sparsity (sparsity threshold) = 0.1
    Chosen empirically for the confidence gating in Eq. (13); no principled derivation.
  • k (sigmoid slope in Eq. 13) = not reported
    The slope of the confidence sigmoid is never specified, making it an unstated hand-picked constant.
  • Injection layer count (first two layers) = layers 1-2
    Selected based on layer-wise accuracy comparison (Figure 7); the paper reports peak performance with early layers but the choice is empirical.
axioms (5)
  • domain assumption Softmax unit-sum normalization forces attention mass to be allocated even without semantic correlation, and residual mass concentrates in high-variance tokens (Section 3.1).
    The unit-sum property is standard math, but the specific claim that it produces attention sinks and a query-invariant structural bias is a domain assumption supported only by qualitative visualizations.
  • ad hoc to paper The observed attention matrix is an additive superposition A_obs = A*_sem + λ A_struct + ζ (Eq. 3).
    No evidence is given that semantic and structural components combine additively rather than multiplicatively or through some other function; this decomposition is asserted.
  • ad hoc to paper The query-averaged attention map (Eq. 11) is a faithful estimate of the query-invariant structural bias and not contaminated by semantic content.
    This is the central premise that makes the purification step valid; the paper offers no independent measurement of the bias, only the average of the matrix being purified.
  • ad hoc to paper Corollary 1: as SNR→0, L(y) becomes non-discriminative and the posterior collapses to the linguistic prior (Eqs. 7-8).
    The proof sketch assumes exactly the relationship it aims to establish—that low SNR makes the normalized visual likelihood approximately constant—so the corollary is a formal restatement of the assumption, not a derivation from more basic facts.
  • domain assumption Pólya's urn dynamics describe the self-reinforcing crystallization of positional attention bias (Section B.1).
    This is an analogy/hypothesis offered to explain why early positional asymmetries grow; it is not used in the algorithm and not empirically tested.
invented entities (2)
  • Structural Bias component A_struct no independent evidence
    purpose: A latent query-invariant attention component that SPAR estimates and subtracts to purify attention maps.
    A_struct is defined as the query-average of the observed attention matrix (Eq. 2/11), so it has no measurement basis outside the very matrix it is used to modify. Its status as 'bias' rather than 'signal' is an assumption.
  • Semantic Attention A*_sem no independent evidence
    purpose: The latent true alignment distribution that SPAR aims to expose.
    A*_sem is never directly observed; it is a residual variable defined by the assumed additive decomposition. No independent probe is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 19331 in / 10620 out tokens · 104278 ms · 2026-07-31T23:14:11.425067+00:00 · methodology

0 comments
read the original abstract

The empirical success of attention mechanism in Multimodal Large Language Models (MLLMs) often obscures its inherent, subtle flaws. Specifically, MLLMs consistently exhibit disproportionate attention toward certain semantically uninformative visual tokens, a phenomenon termed "register" or "Visual Attention Sinks." While existing inference intervention methods attempt to identify these sink tokens and redistribute their attention weights, such approaches typically treat these tokens in isolation and suffer from computational inefficiency. Instead, we reframe this phenomenon as a generalized textual bias exerted over visual features that extends beyond isolated sink tokens. From this perspective, a pervasive structural bias leads to the dilution of the semantic visual signal, precipitating multimodal hallucinations as the model prioritizes linguistic priors over valid visual evidence. To address this limitation, we introduce Saliency-guided Purification and Adaptive Redistribution (SPAR), a training-free, plug-and-play intervention. SPAR mitigates this generalized textual bias by purifying structural noise and subsequently redistributing the reclaimed attention budget to the most informative visual regions. Comprehensive evaluations across a diverse spectrum of hallucination benchmarks demonstrate that SPAR effectively restores authentic visual grounding with negligible computational overhead.

Figures

Figures reproduced from arXiv: 2607.24017 by Bin Zhu, Jingjing Chen, Pengkun Jiao, Yu-Gang Jiang.

Figure 1
Figure 1. Figure 1: Visualization of structural attention noise. High [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Mean attention maps of (a) text-to-text and (b) text [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Illustration of SPAR Pipeline. (a) Text-to-image cross-attention maps are extracted from all attention heads. (b) A [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative visualization of attention purification via SPAR on LLaVA-v1.5. (Left/Middle) Average text-to-image [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative results on the image caption hallucination (CHAIR) benchmark. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of SPAR and GasEraser performance [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Head-level text-to-image attention maps in the first three layers. The visualization reveals a systematic attention [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Visualization of the layer-level sum of average text-to-text/image attention. [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Image-level average text-to-image attention in InternVL2-8B, which employs the InternViT-300M-448px encoder for [PITH_FULL_IMAGE:figures/full_fig_p013_10.png] view at source ↗
Figure 12
Figure 12. Figure 12: Qualitative results on the GaslightingBench. [PITH_FULL_IMAGE:figures/full_fig_p021_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 13 linked inside Pith

  1. [1]

    Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, QianYing Wang, Guang Dai, Ping Chen, and Shijian Lu. 2024. AGLA: Mitigating Object Halluci- nations in Large Vision-Language Models with Assembly of Global and Local Attention.arXiv preprint arXiv:2406.12718(2024)

    Pith/arXiv arXiv 2024

  2. [2]

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2024. Hallucination of multimodal large language models: A survey.arXiv preprint arXiv:2404.18930(2024)

    Pith/arXiv arXiv 2024

  3. [3]

    Nicola Cancedda. 2024. Spectral Filters, Dark Signals, and Attention Sinks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 4792–4808. doi:10.18653/v1/2024.acl-long.263

    doi:10.18653/v1/2024.acl-long.263 2024

  4. [4]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al . 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 24185–24198

    2024

  5. [5]

    Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou

  6. [6]

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2023. Dola: Decoding by contrasting layers improves factuality in large language models.arXiv preprint arXiv:2309.03883(2023)

    Pith/arXiv arXiv 2023

  7. [7]

    Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. 2024. Vision Transformers Need Registers. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=2dnO3LLiJ1

    2024

  8. [8]

    Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pra- muditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto

  9. [9]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2024. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv:2306.13394 [cs.CV] https://arxiv.org/abs/2306.13394

    Pith/arXiv arXiv 2024

  10. [10]

    InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Multi-modal hallucination control by visual information grounding. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14303–14312

  11. [11]

    Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qing- hao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. 2024. Hallucination augmented contrastive learning for multimodal large language model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 27036–27046

    2024

  12. [12]

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hal- lucination in multi-modal large language models via over-trust penalty and retrospection-allocation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13418–13427

    2024

  13. [13]

    Pengkun Jiao, Xinlan Wu, Bin Zhu, Jingjing Chen, Chong-Wah Ngo, and Yu-Gang Jiang. 2026. Rode: Linear rectified mixture of diverse experts for food large multi- modal models. InProceedings of the 2026 International Conference on Multimedia Retrieval. 2457–2466

    2026

  14. [14]

    Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang

  15. [15]

    Pengkun Jiao, Bin Zhu, Jingjing Chen, Chong-Wah Ngo, and Yu-Gang Jiang. 2025. From holistic to localized: local enhanced adapters for efficient visual instruction fine-tuning. InProceedings of the IEEE/CVF International Conference on Computer Vision. 2728–2737

    2025

  16. [16]

    Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. 2025. See What You Are Told: Visual Attention Sink in Large Multimodal Models. In The Thirteenth International Conference on Learning Representations. https: //openreview.net/forum?id=7uDI7w5RQA

    2025

  17. [17]

    Pengkun Jiao, Bin Zhu, Jingjing Chen, Chong-Wah Ngo, and Yu-Gang Jiang. 2025. Don’t Deceive Me: Mitigating Gaslighting through Attention Reallocation in LMMs.arXiv preprint arXiv:2504.09456(2025)

    Pith/arXiv arXiv 2025

  18. [18]

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13872–13882

    2024

  19. [19]

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895(2024)

    Pith/arXiv arXiv 2024

  20. [20]

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Ha- jishirzi, and Ali Farhadi. 2016. A diagram is worth a dozen images. InEuropean conference on computer vision. Springer, 235–251

    2016

  21. [21]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 26296–26306

    2024

  22. [22]

    Shi Liu, Kecheng Zheng, and Wei Chen. 2024. Paying more attention to image: A training-free method for alleviating hallucination in lvlms. InEuropean Conference on Computer Vision. Springer, 125–140

    2024

  23. [23]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen

  24. [24]

    2008.Pólya urn models

    Hosam Mahmoud. 2008.Pólya urn models. Chapman and Hall/CRC

    2008

  25. [25]

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning.arXiv preprint arXiv:1809.02156(2018)

    Pith/arXiv arXiv 2018

  26. [26]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing568 (2024), 127063

    2024

  27. [27]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al . 2024. Mmbench: Is your multi-modal model an all-around player?. InEuropean conference on computer vision. Springer, 216–233

    2024

  28. [28]

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, et al . 2024. Aligning large multimodal models with factually augmented rlhf. InFindings of the Association for Computational Linguistics: ACL 2024. 13088–13110

    2024

  29. [29]

    Ziyao Tang, Pengkun Jiao, Bin Zhu, Huiyan Qi, Jingjing Chen, and Yu-Gang Jiang. 2026. Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Models.Findings of the Association for Computational Linguistics: ACL(2026)

    2026

  30. [30]

    Zifu Wan, Ce Zhang, Silong Yong, Martin Q Ma, Simon Stepputtis, Louis-Philippe Morency, Deva Ramanan, Katia Sycara, and Yaqi Xie. 2025. ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models.arXiv preprint arXiv:2507.00898(2025)

    Pith/arXiv arXiv 2025

  31. [31]

    Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. 2024. Massive Activa- tions in Large Language Models. InICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models. https://openreview.net/forum? id=1ayU4fMqme

    2024

  32. [32]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient Streaming Language Models with Attention Sinks. InThe Twelfth Inter- national Conference on Learning Representations. https://openreview.net/forum? id=NG7sS51zVF

    2024

  33. [33]

    Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. 2024. Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Cal- ibration. InForty-first International Conference on Machine Learning. https: //openreview.net/forum?id=DLTjFFiuUJ

    2024

  34. [34]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9556–9567

    2024

  35. [35]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191(2024)

    Pith/arXiv arXiv 2024

  36. [36]

    Bin Zhu, Huiyan Qi, Yinxuan Gui, Jingjing Chen, Chong-Wah Ngo, and Ee-Peng Lim. 2025. Calling a Spade a Heart: Gaslighting Multimodal Large Language Models via Negation.arXiv preprint arXiv:2501.19017(2025)

    arXiv 2025

  37. [37]

    Xin Zou, Yizhou Wang, Yibo Yan, Yuanhuiyi Lyu, Kening Zheng, Sirui Huang, Junkai Chen, Peijie Jiang, Jia Liu, Chang Tang, and Xuming Hu. 2025. Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models. InForty-second International Conference on Machine Learning. https://openreview.net/foru...

    2025

  38. [38]

    Is there a {object} in the image?

    Zayd MK Zuhri, Erland Hilman Fuadi, and Alham Fikri Aji. 2025. Softpick: No Attention Sink, No Massive Activations with Rectified Softmax.arXiv preprint arXiv:2504.20966(2025). MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil. P Jiao et al. A Proof sketch of Corollary 1 Rewrite the posterior 4 using the normalized visual likelihood𝐿(𝑦): 𝑃(𝑦|𝑉,𝑇)= 𝐿(𝑦)...

    Pith/arXiv arXiv 2025

  39. [39]

    Guanyu Zhou, Yibo Yan, Xin Zou, Kun Wang, Aiwei Liu, and Xuming Hu. 2024. Mitigating modality prior-induced hallucinations in multimodal large language models via deciphering attention causality.arXiv preprint arXiv:2410.04780(2024)

    Pith/arXiv arXiv 2024

  40. [2023]

    Evaluating object hallucination in large vision-language models.arXiv preprint arXiv:2305.10355(2023)

    Pith/arXiv arXiv 2023

  41. [2024]

    arXiv preprint arXiv:2403.00425(2024)

    Halc: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00425(2024)

    Pith/arXiv arXiv 2024

  42. [2025]

    InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 25004– 25014