Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

VASparse: Towards Efficient Visual Hallucination Mitigation via Visual-Aware Token Sparsification

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VASparse reduces visual hallucinations in LVLMs by pruning tokens with a visual-saliency-aware attention score, using embedding-level contrastive logits, and penalizing attention sinks; decoding stays up to 12.9x faster than HALC.

desk verdict A well-evaluated decoding recipe for LVLM hallucination mitigation with a real efficiency win, but the sinking-attention penalty as written is not implementable online and the main theorem is a tautology. read the letter →

arxiv 2501.06553 v2 pith:3FINXETK submitted 2025-01-11 cs.CV

Xianwei Zhuang , Zhihong Zhu , Yuxin Xie , Liming Liang , Yuexian Zou This is my paper · ORCID
classification cs.CV
keywords visualhallucinationmitigationtokensparsificationvisual-awaredecodingcontrastivelargevision-languagemodelsattentionsinkingplug-and-playefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VASparse is a plug-and-play decoding algorithm that tries to reduce visual hallucinations in large vision-language models without retraining or slowing generation. The paper's central claim is that hallucination mitigation and efficiency can be unified as a token-selection problem: keep the tokens with the highest combined attention and visual-saliency scores, use the embeddings of masked visual tokens to build a contrastive logit correction that needs no second pass through the decoder, and penalize attention that sinks into low-semantic text tokens. If this claim is right, LVLMs can become more faithful to images while decoding several times faster than current hallucination-mitigation decoders. The paper reports state-of-the-art hallucination scores on CHAIR, POPE, MME, and GPT-4 assisted benchmarks across LLaVA-1.5, MiniGPT-4, and mPLUG-Owl2.

What carries the argument

The load-bearing object is the mask-selection score $\delta_i = (\langle q, K_i \rangle)^2 + \lambda P_i$, which converts two goals—preserving the original attention distribution and keeping visually salient tokens—into a single ranking whose top-$S$ choice is globally optimal for the constrained objective $E(M)$. Around it sit two auxiliary mechanisms: sparse-based visual contrastive decoding, which obtains a contrastive logit distribution by feeding embeddings of randomly masked visual tokens straight into the language-model head so no second decoder pass is needed, and a sinking-attention penalty weight matrix $W$ built by softmax over cumulative attention scores, used to shrink attention paid to low-semantic text tokens.

What would settle it

Instrument VASparse's decoding loop and record whether the penalty weight $w_j$ in Eq. 7 at step $t$ is computed from attention rows $i = j,\ldots,L$ that include queries generated after step $t$; if the implementation uses only rows available at step $t$, the deployed penalty is not the literal Eq. 7, and a strictly causal version of the penalty could be compared on CHAIR to measure the discrepancy.

Watch

Extended reading notes

Core claim

The paper's central discovery is that visual hallucination in LVLMs is partly a token-attention problem: attention is sparse, vision-agnostic pruning removes low-attention image tokens that carry visual evidence, and certain text tokens act as attention sinks. VASparse therefore builds a binary mask by ranking each token with $\delta_i = (\langle q, K_i \rangle)^2 + \lambda P_i$, where the first term preserves the attention recall and $P_i$ is a visual saliency score derived from attention to image tokens; the top-$S$ tokens are kept and discarded tokens are merged by density-peak clustering. A sparse-based visual contrastive decoding step then contrasts the logits of the visual-aware selection with logits obtained by feeding masked visual-token embeddings directly to the language-model head, avoiding a full secondary decoding pass. Finally, a cumulative-attention penalty recalibrates attention to downweight sinking tokens such as '<.' and '<s>'. The paper proves that the top-$S$ selection is globally optimal for the unified objective, and reports that the full method attains the best hallucination scores on four benchmarks while decoding up to 12.9x faster than HALC.

Load-bearing premise

The sinking-penalty formula in Eq. 7 sums attention over future decoding positions that do not exist yet when a token is generated, and the paper never states how it approximates this online, so the penalty as written cannot be computed without look-ahead or an unstated modification.

Editorial extensions

If this is right

  • On LLaVA-1.5 with 64-token captions, VASparse lowers instance-level CHAIR from 7.22 (greedy) to 5.82 and sentence-level CHAIR from 22.20 to 18.51.
  • It keeps decoding fast: 27.73 TPS on LLaVA-1.5 versus 2.15 for HALC and 4.36 for OPERA, roughly 12.9x and 6.4x faster respectively.
  • The plug-and-play design generalizes across LLaVA-1.5, MiniGPT-4, and mPLUG-Owl2 without training or post-processing.
  • Ablations show each component matters: removing visual-aware selection, the contrastive decoding, or the sinking penalty raises CHAIR.
  • The top-$S$ selection by $\delta_i$ is provably optimal for the unified objective, so the observed gains do not rest on an ad hoc heuristic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit consequence the paper does not spell out: any efficient token-pruning acceleration for LVLMs should be audited for hallucination impact, since visual-agnostic pruning actively worsens VH and VASparse's visual saliency term is one corrective mechanism.
  • The paper's Eq. 7 as written requires attention rows beyond the current decoding step; a reproducible implementation must be a causal approximation, and the exact truncation rule should be stated to make the results portable.
  • Because the contrastive logits come from embeddings at layer 0, the method suggests a speed-quality frontier: stopping at even earlier representations or sharing one masked-embedding computation across steps might further cut latency, at some unknown hallucination cost.
  • The authors note that MME's binary short-answer format limits decoding-length methods; combining VASparse with a small training-time correction could extend the benefit to terse-answer settings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes VASparse, a training-free decoding-time method for mitigating visual hallucinations in large vision-language models (LVLMs). The method has three components: (1) a visual-aware token selection strategy that prunes tokens by a score combining attention magnitude and a visual saliency term, (2) a sparse-based visual contrastive decoding step that contrasts logits from the full model with logits obtained by feeding masked visual-token embeddings to the language-model head, and (3) a sinking-attention penalty designed to down-weight low-semantic text tokens. Experiments on CHAIR, POPE, MME, and a GPT-4-assisted benchmark across LLaVA-1.5, MiniGPT-4, and mPLUG-Owl2 report improvements over existing decoding methods, with substantially higher decoding speed than rollback-based baselines such as HALC.

Significance. If the reported results hold, VASparse is practically valuable: it is a plug-and-play, training-free intervention that improves hallucination metrics while avoiding the multi-round decoding overhead of methods like HALC and OPERA. The paper's empirical observations on attention sparsity and on the relationship between vision-agnostic token pruning and hallucination are interesting and potentially reusable. The authors provide a public repository and reproduce baselines with their official code, which is a strength. The claimed theoretical optimality, however, is a tautology, and one of the three main components (the sinking-attention penalty) is not computable as described in an autoregressive decoder, so the current manuscript does not establish the mechanism behind the reported gains.

major comments (3)
  1. [Section 4.5, Eq. (7)] The penalty weight w_j is defined as w_j = exp(Σ_{i=j..L} a_{i,j}) / Σ_k exp(Σ_{i=k..L} a_{i,k}), where a_{i,j} is the attention from query i to key j. During autoregressive decoding of token t, attention rows i > t do not exist yet, so for any column j ≤ t the sum over i=j..L includes future rows that are unavailable, and for j > t the sum is empty. The paper states that W 'will be utilized as a weight ... during decoding' but never specifies the online approximation (e.g., a prefix-sum, a running average, or a retroactive post-hoc correction). Table 4, Group 3 shows that removing this penalty increases CHAIRi from 5.82 to 6.32 on LLaVA-1.5, so the component is material to the headline result. Unless the released code implements a different, computable formula, the reported gains cannot be attributed to the mechanism as described. This needs to be fixed either by deriving a causal online form or by explicitly reporting the actual computation used.
  2. [Section 4.6 and Appendix 8 (Theorem 1)] Theorem 1 is circular. The optimization problem in Eq. (3) reduces, as shown in the appendix, to maximizing Σ_i M_i(y_i^2 + λP_i), and the proposed selection strategy in Eq. (4) ranks by exactly δ_i = <q,K_i>^2 + λP_i. The 'globally optimal' selection is therefore nothing more than the definition of the objective; it provides no independent justification for the design. The paper should either reframe this as a straightforward property of the chosen optimization (e.g., a simple lemma) or remove the 'theoretical analysis' claim, since the current framing overstates the result.
  3. [Section 5.1, Tables 1-3] All main results are reported as averages of five runs without standard deviations, confidence intervals, or significance tests. Some improvements over the strongest baselines are small; for example, Table 1 shows CHAIRi 5.82 versus HALC 6.27 on LLaVA-1.5, and several MME sub-scores in Table 3 differ by only a few points. The repeated use of 'significantly outperforms' is therefore not supported by the evidence presented. The authors should report per-run variance and run paired statistical tests (or otherwise justify that the differences are meaningful), or temper the wording of their claims.
minor comments (5)
  1. [Section 5.1, GPT-4 Assisted Benchmarks] The text states that 'VASparse achieved the best SHR metric among the four LVLMs,' but Table 5 reports results for exactly three backbones (LLaVA-1.5, MiniGPT-4, mPLUG-Owl2).
  2. [Section 4.3] The aggregation step for discarded tokens is described in a single sentence: 'we employ the k-nearest neighbor density peak aggregation algorithm [34] to achieve adaptive token aggregation. Tokens within the same cluster are summed and retained as a single aggregated token.' This is too underspecified to reproduce: the number of neighbors, the cluster threshold, and how the aggregated token interacts with the binary mask M are not given.
  3. [Supplementary Section 7.1] The hyperparameters λ, α, and β are all set to 0.1 with no sensitivity analysis; given that all three components depend on these scales, the lack of any ablation over their values makes it hard to gauge robustness.
  4. [Section 4.4, Eq. (6)] The contrastive logit logit_φ is said to be obtained by 'inputting the embeddings of vision-agnostic tokens to the language decoding head.' It is unclear whether these are token embeddings or per-layer hidden states, and how masked visual tokens are aggregated (e.g., mean pooling, projection, or direct concatenation). Clarifying this would help reproducibility.
  5. [Supplementary Section 7] The supplementary heading contains a typo: 'Experimental Detials' should be 'Experimental Details.'

Circularity Check

1 steps flagged · score 6.0 of 10

Theorem 1's 'global optimality' restates the selection rule by construction; the empirical comparisons remain independent.

  1. self definitional [Section 4.2 (Definition 1, Eq. 3), Section 4.3 (Eq. 4), Section 4.6 (Theorem 1), Appendix Section 8 (Eqs. 9-14)]
    "min_M E(M ) = Σ_i ((⟨q, Ki⟩ − Mi⟨q, Ki⟩)^2 − λPi · Mi) ... δi = (⟨q, Ki⟩)^2 + λPi ... By employing the selection strategy defined in Section 4.3, we can obtain a globally optimal solution for the optimization problem defined in Def. 1."

    The Appendix proof expands Eq. 3 into E(M) = Σ_i [y_i^2 − M_i(y_i^2 + λP_i)] with y_i = ⟨q,K_i⟩, so minimizing E is equivalent to maximizing Σ_i M_i δ_i where δ_i = y_i^2 + λP_i. Eq. 4 defines the selection score as exactly this δ_i and the procedure keeps the top-S values. Since a separable linear objective is maximized by picking the largest coefficients, Theorem 1 does not derive the selection rule from independent principles; the optimization problem was constructed so that the proposed rule is optimal by definition. The 'global optimality' guarantee is therefore a restatement of the selection criterion rather than an external theoretical justification.

full rationale

The only load-bearing circular step is Theorem 1 / Appendix Section 8: the unified objective in Eq. 3 is defined with the same per-token reward δ_i = ⟨q,K_i⟩² + λP_i that Eq. 4 uses for ranking, so the claimed optimality is true by construction. This is a genuine but localized circularity: a tautology is presented as theoretical validation of the token-selection component. The main empirical claims (Tables 1-5 and ablations in Table 4) are evaluated against external baselines on CHAIR, POPE, MME, and GPT-4-assisted benchmarks and are not derived from the theorem, so the empirical content is independent. The paper's self-citations (e.g., [59] in related work) are not load-bearing. The unimplementability concern about Eq. 7 summing attention rows i=j..L over future queries is a correctness/consistency issue rather than a circularity issue, so under the stated rules it does not raise the circularity score. Overall: partial circularity in the theoretical-optimality claim, with the empirical comparison remaining self-contained, giving a score of 6.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method relies on a set of hand-set hyperparameters and several unvalidated modeling assumptions about attention and LM-head semantics; no new physical or model entities are introduced.

free parameters (6)
  • lambda (visual saliency trade-off) = 0.1
    Trade-off between attention recall and visual saliency in Eq. 4; set without reported sensitivity analysis.
  • alpha (contrastive amplification) = 0.1
    Contrastive weight in Eq. 6.
  • beta (sinking penalty scale) = 0.1
    Penalty scale in the attention calibration (1+beta)qK^T - beta W⊙qK^T.
  • sparsity rate S (top-S) = 0.9 * L
    Fraction of tokens retained after sparsification; 90% is far from aggressive, limiting speed gains.
  • image masking sparsity rate for S_m = 0.5
    Fraction of visual tokens masked to produce the vision-agnostic contrastive logits.
  • L_s (sparsification start length) = 32 (Lmax=64), 16 (Lmax=512)
    Decoding length before sparsification is applied.
assumptions (5)
  • domain assumption Attention in LVLM decoders is sparse and pruning low-attention tokens preserves most attention mass
    Figure 2a shows long-tail attention; used to justify pruning tokens in decoding.
  • domain assumption Vision-agnostic token sparsification worsens VH
    Figure 2b; used to motivate visual-aware selection and the contrastive signal.
  • ad hoc to paper Visual saliency P_i computed from image-token attention is a valid importance score for retaining visual information
    Eq. 5 defines P_i from attention to image tokens; no external validation that this score captures visual grounding.
  • ad hoc to paper Embeddings passed directly to the LM head yield a meaningful 'vision-agnostic' logit distribution
    Section 4.4 uses logit_phi(S_m(v), x, y<t) without passing through transformer layers; the LM head normally consumes final hidden states, not raw embeddings.
  • ad hoc to paper The sinking attention penalty in Eq. 7 can be computed online
    Eq. 7 sums attention over queries i=j..L, including future queries not available during autoregressive decoding; no finite-horizon approximation is stated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VASparse: Towards Efficient Visual Hallucination Mitigation via Visual-Aware Token Sparsification." pith.science (2026). https://pith.science/paper/3FINXETK

@misc{pith2026250106553,
  author       = {Pith},
  title        = {Pith review of: VASparse: Towards Efficient Visual Hallucination Mitigation via Visual-Aware Token Sparsification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3FINXETK}},
  note         = {Machine review of arXiv:2501.06553}
}
read the original abstract

Large Vision-Language Models (LVLMs) may produce outputs that are unfaithful to reality, also known as visual hallucinations (VH), which significantly impedes their real-world usage. To alleviate VH, various decoding strategies have been proposed to enhance visual information. However, many of these methods may require secondary decoding and rollback, which significantly reduces inference speed. In this work, we propose an efficient plug-and-play decoding algorithm via Visual-Aware Sparsification (VASparse) from the perspective of token sparsity for mitigating VH. VASparse is inspired by empirical observations: (1) the sparse activation of attention in LVLMs, and (2) visual-agnostic tokens sparsification exacerbates VH. Based on these insights, we propose a novel token sparsification strategy that balances efficiency and trustworthiness. Specifically, VASparse implements a visual-aware token selection strategy during decoding to reduce redundant tokens while preserving visual context effectively. Additionally, we innovatively introduce a sparse-based visual contrastive decoding method to recalibrate the distribution of hallucinated outputs without the time overhead associated with secondary decoding. Subsequently, VASparse recalibrates attention scores to penalize attention sinking of LVLMs towards text tokens. Extensive experiments across four popular benchmarks confirm the effectiveness of VASparse in mitigating VH across different LVLM families without requiring additional training or post-processing. Impressively, VASparse achieves state-of-the-art performance for mitigating VH while maintaining competitive decoding speed. Code is available at https://github.com/mengchuang123/VASparse-github.

Figures

Figures reproduced from arXiv: 2501.06553 by the authors.

Figure 1
Figure 1. Comparison of decoding speed and hallucination mit [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. VH evaluation and attention analysis using LLaVA-1.5 on the CHAIR benchmark: (a) token sorting by attention score; (b) token [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Attention sinking phenomenon in LVLMs: in the 8-th [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The illustration of the proposed VASparse framework, which consists of (1) the visual-aware token selection designed to prune [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Performance and efficiency analysis of different logit [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: More visualization and evidence of sparsity of attention and sinking attention on the LLaVA-1.5. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Qualitative results comparing our VASparse and other methods with LLaVA-1.5 backbone. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Qualitative results comparing our VASparse and other methods with LLaVA-1.5 backbone. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: LLaVA-Bench results comparing our VASparse and other methods with LLaVA-1.5 backbone. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: LLaVA-Bench results comparing our VASparse and other methods with LLaVA-1.5 backbone. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: LLaVA-Bench results comparing our VASparse and other methods with LLaVA-1.5 backbone. [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A dual-level attention intervention that boosts salient visual-token attention and suppresses text/system attention during decoding reduces hallucination rates in LLaVA, MiniGPT-4, and mPLUG-Owl2 on POPE and CHAIR.

  2. Do we really have to filter out random noise in pre-training data for language models?

    cs.CL 2025-02 conditional novelty 6.0 of 10

    Random noise in pre-training data has a surprisingly small effect on language model next-token loss, but can still hurt downstream tasks; a new local gradient matching loss partially counteracts this.

Reference graph

Works this paper leans on

70 extracted references · 32 canonical work pages · cited by 2 Pith papers

  1. [1]

    Steven Adler

    OpenAI Josh Achiam and et al. Steven Adler. Gpt-4 technical report. 2023. 6

    work page 2023

  2. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 2

    arXiv 2023

  3. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litw...

    arXiv 2005

  4. [5]

    Minigpt- v2: large language model as a unified interface for vision- language multi-task learning

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt- v2: large language model as a unified interface for vision- language multi-task learning. ArXiv, abs/2310.09478, 2023. 2, 6

    arXiv

  5. [6]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models, 2024

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models, 2024. 2, 3, 6, 12

    work page 2024

  6. [7]

    Halc: Object halluci- nation reduction via adaptive focal-contrast decoding

    Zhaorun Chen, Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Halc: Object halluci- nation reduction via adaptive focal-contrast decoding. ArXiv, abs/2403.00425, 2024. 2, 3, 6, 7, 12, 13, 14, 15, 16

    arXiv 2024

  7. [8]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024. 2

    work page 2024

  8. [9]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) , 2(3):6,

    2023

Show all 70 references
  1. [10]

    Glass, and Pengcheng He

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. Dola: Decoding by con- trasting layers improves factuality in large language models. ArXiv, abs/2309.03883, 2023. 2, 3, 6, 12

    2023 arXiv

  2. [11]

    Plausible may not be faithful: Probing object hallucination in vision-language pre-training

    Wenliang Dai, Zihan Liu, Ziwei Ji, Dan Su, and Pascale Fung. Plausible may not be faithful: Probing object hallucination in vision-language pre-training. ArXiv, abs/2210.07688, 2022. 1

    2022 arXiv

  3. [12]

    Instructblip: Towards general-purpose vision- language models with instruction tuning, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning, 2023. 2

    2023

  4. [13]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pas- cale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tun- ing. ArXiv, abs/2305.06500, 2023. 1

    2023 arXiv

  5. [14]

    Mme: A compre- hensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Meng- dan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. Mme: A compre- hensive evaluation benchmark for multimodal large language models. ArXiv, abs/2306.13394, 2023. 2, 6, 7

    2023 arXiv

  6. [15]

    Multimodal-gpt: A vision and language model for dialogue with humans

    Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qianmengke Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans. ArXiv, abs/2305.04790,

    arXiv

  7. [16]

    Hallusion- bench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language mod- els

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusion- bench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large visi...

    2023

  8. [17]

    Detecting and preventing hallucinations in large vision language models

    Anish Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In AAAI Conference on Artificial Intelligence, 2023. 1

    2023

  9. [18]

    Qidong Huang, Xiao wen Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Neng H. Yu. Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation. ArXiv, abs/2311.17911, 2023. 1, 2, 3,...

    2023 arXiv

  10. [19]

    Self-introspective decoding: Alleviating hallucinations for large vision-language models,

    Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. Self-introspective decoding: Alleviating hallucinations for large vision-language models,

  11. [20]

    Mitigating object halluci- nations in large vision-language models through visual con- trastive decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Li Bing. Mitigating object halluci- nations in large vision-language models through visual con- trastive decoding. ArXiv, abs/2311.16922, 2023. 2, 3, 5, 6, 7, 12, 14

    2023 arXiv

  12. [21]

    Otter: A multi-modal model 9 with in-context instruction tuning

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model 9 with in-context instruction tuning. ArXiv, abs/2305.03726,

    arXiv

  13. [22]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR,

  14. [23]

    Visualbert: A simple and performant baseline for vision and language

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. ArXiv, abs/1908.03557,

    1908 arXiv

  15. [24]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji rong Wen. Evaluating object hallucination in large vision-language models. In Conference on Empirical Methods in Natural Language Processing, 2023. 1, 2, 6, 7, 15, 16

    2023

  16. [25]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014. 6

    2014

  17. [26]

    Mitigating hallucination in large multi-modal models via robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Ya- coob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. 2023. 1, 2, 3

    2023

  18. [27]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. ArXiv, abs/2310.03744, 2023. 1, 14

    2023 arXiv

  19. [28]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. ArXiv, abs/2304.08485, 2023. 1, 2, 6

    2023 arXiv

  20. [29]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 2

    2024

  21. [30]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2

    2024

  22. [31]

    Negative object presence evaluation (nope) to measure object hallucination in vision-language models

    Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung. Negative object presence evaluation (nope) to measure object hallucination in vision-language models. ArXiv, abs/2310.05338, 2023. 1, 2

    2023 arXiv

  23. [32]

    Khan, and Fahad Shahbaz Khan

    Muhammad Maaz, Hanoona Abdul Rasheed, Salman H. Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. ArXiv, abs/2306.05424, 2023. 1, 2

    2023 arXiv

  24. [33]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. 2

    2021

  25. [34]

    Clustering by fast search and find of density peaks

    Alex Rodriguez and Alessandro Laio. Clustering by fast search and find of density peaks. science, 344(6191):1492– 1496, 2014. 5

    2014

  26. [35]

    Object hallucination in image captioning

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Conference on Empirical Methods in Natural Language Processing, 2018. 1, 2, 6

    2018

  27. [36]

    Do we really have to filter out random noise in pre-training data for language models?, 2025

    Jinghan Ru, Yuxin Xie, Xianwei Zhuang, Yuguo Yin, and Yuexian Zou. Do we really have to filter out random noise in pre-training data for language models?, 2025. 2

    2025

  28. [37]

    Stanford alpaca: An instruction-following llama model, 2023

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023

    2023

  29. [38]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  30. [39]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Am- jad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 2, 5

    2023 arXiv

  31. [40]

    Evaluation and analysis of hallucination in large vision-language models

    Junyan Wang, Yi Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu, Qinghao Ye, Mingshi Yan, Ji Zhang, Jihua Zhu, Jitao Sang, and Haoyu Tang. Evaluation and analysis of hallucination in large vision-language models. ArXiv, abs/2308.15126, 2023. 2

    2023 arXiv

  32. [41]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chau- mond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R ´emi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, ...

    1910 arXiv

  33. [42]

    Efficient streaming language models with attention sinks, 2024

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024. 4

    2024

  34. [43]

    Gpa: Global and prototype alignment for audio-text retrieval

    Yuxin Xie, Zhihong Zhu, Xianwei Zhuang, Liming Liang, Zhichang Wang, and Yuexian Zou. Gpa: Global and prototype alignment for audio-text retrieval. In Proc. Interspeech 2024, pages 5078–5082, 2024. 2

    2024

  35. [44]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Mingshi Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. ArXiv, abs/2311.04257,

    arXiv

  36. [45]

    mplug- owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug- owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ,...

    2024

  37. [46]

    Woodpecker: Hallucination correction for multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xingguo Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. ArXiv, abs/2310.16045, 2023. 1, 3, 6, 7, 14

    2023 arXiv

  38. [47]

    Atri: Mitigating multilingual audio text retrieval inconsis- tencies by reducing data distribution errors

    Yuguo Yin, Yuxin Xie, Wenyuan Yang, Dongchao Yang, Jing- han Ru, Xianwei Zhuang, Liming Liang, and Yuexian Zou. Atri: Mitigating multilingual audio text retrieval inconsis- tencies by reducing data distribution errors. arXiv preprint arXiv:2502.14627, 2025. 2 10

    2025 arXiv

  39. [48]

    Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, and Tat-Seng Chua. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. ArXiv, abs/2312.00849, 2023. 2, 3

    2023 arXiv

  40. [49]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. ArXiv, abs/2306.02858, 2023. 1, 2, 12

    2023 arXiv

  41. [50]

    Sparsevlm: Visual token sparsification for efficient vision-language model inference

    Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevlm: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417, 2024. 2, 3, 6, 12

    arXiv

  42. [51]

    Graco: Granularity-controllable interactive segmentation

    Yian Zhao, Kehan Li, Zesen Cheng, Pengchong Qiao, Xi- awu Zheng, Rongrong Ji, Chang Liu, Li Yuan, and Jie Chen. Graco: Granularity-controllable interactive segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 3501–3510, 2024. 2

    2024

  43. [52]

    Detrs beat yolos on real-time object detection, 2024

    Yian Zhao, Wenyu Lv, Shangliang Xu, Jinman Wei, Guanzhong Wang, Qingqing Dang, Yi Liu, and Jie Chen. Detrs beat yolos on real-time object detection, 2024. 2

    2024

  44. [53]

    Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimiza- tion

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimiza- tion. arXiv preprint arXiv:2311.16839, 2023. 6, 12

    2023 arXiv

  45. [54]

    Analyzing and mitigating object hallucination in large vision- language models

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision- language models. ArXiv, abs/2310.00754, 2023. 1, 2, 3, 6

    2023 arXiv

  46. [56]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 2

    2023 arXiv

  47. [57]

    Towards multimodal- augmented pre-trained language models via self-balanced expectation-maximization iteration

    Xianwei Zhuang, Xuxin Cheng, Zhihong Zhu, Zhanpeng Chen, Hongxiang Li, and Yuexian Zou. Towards multimodal- augmented pre-trained language models via self-balanced expectation-maximization iteration. In ACM Multimedia 2024, 2024. 2

    2024

  48. [58]

    Towards explainable joint models via information theory for multiple intent detection and slot filling

    Xianwei Zhuang, Xuxin Cheng, and Yuexian Zou. Towards explainable joint models via information theory for multiple intent detection and slot filling. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):19786–19794,

  49. [59]

    Game on tree: Visual halluci- nation mitigation via coarse-to-fine view tree and game theory

    Xianwei Zhuang, Zhihong Zhu, Zhanpeng Chen, Yuxin Xie, Liming Liang, and Yuexian Zou. Game on tree: Visual halluci- nation mitigation via coarse-to-fine view tree and game theory. In Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing , pag...

    2024

  50. [60]

    Kdpror: A knowledge- decoupling probabilistic framework for video-text retrieval

    Xianwei Zhuang, Hongxiang Li, Xuxin Cheng, Zhihong Zhu, Yuxin Xie, and Yuexian Zou. Kdpror: A knowledge- decoupling probabilistic framework for video-text retrieval. In Computer Vision – ECCV 2024 , pages 313–331, Cham,

    2024

  51. [61]

    Vargpt: Unified understanding and generation in a visual autoregressive mul- timodal large language model, 2025

    Xianwei Zhuang, Yuxin Xie, Yufan Deng, Liming Liang, Jinghan Ru, Yuguo Yin, and Yuexian Zou. Vargpt: Unified understanding and generation in a visual autoregressive mul- timodal large language model, 2025. 2

    2025

  52. [62]

    UnicoTT: A unified framework for structural chain-of-thought distillation

    Xianwei Zhuang, Zhihong Zhu, Zhichang Wang, Xuxin Cheng, and Yuexian Zou. UnicoTT: A unified framework for structural chain-of-thought distillation. In The Thirteenth International Conference on Learning Representations, 2025. 2 11 V ASparse: Towards Efficient Visual Hallucina...

    2025

  53. [64]

    Experimental Detials 7.1. Experimental Setting For the settings of the CHAIR and POPE benchmarks, we evaluated the results with the maximum generation to- ken limits of LVLM Lmax set to 64 and 512, respectively. For the GPT4-assist benchmark [ 53], we referred to prior work an...

  54. [65]

    Mi ∈ {0, 1}, ∀i = 1, 2,

    Proof of Theorem 1 We aim to prove that in the following optimization problem, our strategy achieves a globally optimal solution: min M E(M ) = LX i=1 (yi − Miyi)2 − λPiMi s.t. Mi ∈ {0, 1}, ∀i = 1, 2, . . . , L, LX i=1 Mi = S, (9) where: • yi = ⟨q, Ki⟩ is the inner product of ...

  55. [66]

    Compute the marginal reward δi for each element: δi = y2 i + λPi. (15)

  56. [67]

    Sort all elements by δi in descending order

  57. [68]

    Optimality of the Strategy For any feasible solution M, we have: LX i=1 Mi = S, M i ∈ {0, 1}

    Select the top S elements, setting their corresponding Mi to 1, and the rest to 0. Optimality of the Strategy For any feasible solution M, we have: LX i=1 Mi = S, M i ∈ {0, 1}. (16) Define the total reward for a solution M as: R(M ) = LX i=1 Miδi. (17) Let the solution chosen ...

  58. [69]

    Our research findings confirm that the self-attention in most layers of the LVLM decoder is sparse

    More evidence of empirical observations We present additional evidence on the attention sparsity and attention sinking of LLaV A-1.5 in Figure 6. Our research findings confirm that the self-attention in most layers of the LVLM decoder is sparse. Furthermore, we observe a signi...

  59. [70]

    More results on CHAIR benchmark We set the maximum generation length to 512 and evalu- ated our method using the CHAIR benchmark, as shown in Table 6. We can observe that when setting the maximum generation length to 512, our method still outperforms the baseline method in mos...

  60. [71]

    We conduct experiments under two dif- ferent maximum text length settings: 64 and 512 tokens

    More results on POPE benchmark Following HALC [ 7], we utilize offline POPE (OPOPE) benchmark with both accuracy and F1-score as evaluation metrics to assess VH. We conduct experiments under two dif- ferent maximum text length settings: 64 and 512 tokens. As 13 Figure 6. More ...

  61. [72]

    We consistently used Please describe this image in detail

    Qualitative Study To visually demonstrate the effectiveness of our approach, we present generated captions from our method and baseline approaches in Figures 7 and 8on the MSCOCO dataset. We consistently used Please describe this image in detail. as the input prompt across all...

  62. [2025]

    Springer Nature Switzerland. 2

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.