Pith. sign in

REVIEW 4 major objections 6 minor 55 references

Enhancing Visual Reliance in Text Generation: A Bayesian Perspective on Mitigating Hallucination in Large Vision-Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A training-free decoding framework reduces object hallucinations in large vision-language models by pruning ambiguous image tokens, correcting the language prior, and stopping before the posterior collapses into the prior.

desk verdict Solid training-free hallucination mitigation with a Bayesian gloss; the pruning premise needs validation and one table row needs a check. read the letter →

arxiv 2505.19498 v2 pith:JKLAP42L submitted 2025-05-26 cs.CV cs.AI

Nanxing Hu , Xiaoyue Duan , Jinchao Zhang , Guoliang Kang This is my paper
classification cs.CVcs.AI
keywords hallucinationmitigationlargevision-languagemodelsvisualrelianceBayesiandecodingredundanttokenpruninglanguagepriorrectificationposteriorcollapseearlystopping
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that hallucinations in large vision-language models are best understood as a failure of visual reliance: the model stops conditioning its next-token prediction on the image and starts behaving like a text-only language model. From a Bayesian reading of the generation process, it identifies three separate causes: ambiguous image tokens act as redundant conditions, the language prior carries a bias that dominates the posterior, and late in generation the posterior collapses toward the prior so visual information no longer influences output. The authors propose EVRB, a training-free framework that removes high-entropy image tokens from the KV cache, divides the posterior by an estimated prior to remove language bias, and scales up the end-of-sequence logit when the posterior-prior distance over visual-relevant tokens shrinks. If correct, this gives a single, task-agnostic decoding intervention that works across different model families and benchmarks without retraining.

What carries the argument

The carrying mechanism is the Bayesian decomposition of next-token prediction into a visual-conditioned posterior and a text-only prior, together with three quantities derived from it: the softmax entropy of each image token's own next-token prediction, used to identify redundant visual conditions; the ratio of the posterior to the estimated prior under an adaptive plausibility mask, used to rectify the biased language prior; and the JS divergence between posterior and prior restricted to visual-relevant text tokens, used to gate the EOS logit. Visual-relevant tokens are located through the entropy drop of value-value attention maps between generated tokens and clear image tokens, because value-value attention shows semantic alignment with object outlines. The three operations run in a single prefill plus two parallel decodings for posterior and prior estimation.

What would settle it

Run EVRB on images containing small, rare objects, log which image tokens are pruned, and compare the pruned locations against ground-truth object boxes; if high-entropy tokens contain the ground-truth objects yet hallucination metrics improve while recall on those objects drops, the redundancy premise fails. Also record the JS divergence at every step before a hallucinated object word: the method predicts divergence declines before hallucination, so observing hallucinated tokens while the posterior-prior JS divergence remains high would show that the early-stopping signal is not actually detecting collapse.

Watch

Extended reading notes

Core claim

The central claim is that hallucination can be mitigated by explicitly enforcing the Bayesian structure that should hold when text is generated from an image. For each next-token prediction, the model's posterior $p(t_{n_t}|v,t)$ is proportional to a visual-conditioned likelihood times a language prior; the paper treats the prior $p(t_{n_t}|t)$ as something that should be close to uniform over plausible vocabulary when no visual information is available. EVRB restores visual reliance by (1) deleting image tokens whose own next-token softmax entropy is above a threshold $\tau$, because they are ambiguous and disturb inference; (2) rectifying the language prior by renormalizing $p(t_{n_t}|v_c,t)/p(t_{n_t}|t)$ under an adaptive plausibility mask; and (3) tracking the JS divergence between posterior and prior on text tokens that attend to image objects, and boosting the EOS logit when that divergence drops, so generation stops before posterior collapse. The paper reports consistent gains over prior decoding methods on POPE, CHAIR, and MME across LLaVA-1.5, Qwen2.5-VL, and Shikra.

Load-bearing premise

The load-bearing premise is that the image tokens whose prediction confidence is low are redundant and can be removed without losing the visual information needed for correct answers; if some of those low-confidence tokens are the only evidence for small or rare objects, pruning would erase signal rather than noise.

Editorial extensions

If this is right

  • LVLM hallucinations can be reduced without any fine-tuning or additional data, just by changing the decoding procedure.
  • The same three interventions transfer across model families and sizes, including LLaVA-1.5-7B, LLaVA-1.5-13B, Qwen2.5-VL-7B, and Shikra, with fixed hyperparameters.
  • The method improves not only hallucination metrics but also the general MME score, whereas several baseline decoding methods trade away general ability.
  • Posterior collapse is a detectable, measurable failure mode: the JS divergence between posterior and prior on recurring visual-relevant tokens falls as generation proceeds, and this signal can be used to decide when to stop.
  • Dividing by the language prior, rather than subtracting logits, is the more effective way to remove language bias in vision-language decoding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The bimodal entropy distribution used to justify token pruning could be repurposed as a per-image explainability tool, flagging which image regions the model actually grounds on.
  • One testable extension is making the pruning threshold per-image by fitting the two modes of the entropy histogram; the paper fixes $\tau$ globally, so adaptive thresholds could improve handling of unusual images.
  • The value-value attention maps used to locate visual-relevant tokens might also serve as a zero-shot grounding or segmentation signal, a direction the paper does not explore.
  • If posterior collapse is a general failure mode, an auxiliary training loss that penalizes the posterior-prior JS divergence over visual-relevant tokens could make the early-stopping rule unnecessary at inference time.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes EVRB, a training-free framework to mitigate hallucination in large vision-language models by enhancing visual reliance at decoding time. EVRB combines three modules: (1) pruning visual tokens whose next-token softmax entropy exceeds a threshold, on the premise that high-entropy tokens are redundant; (2) rectifying the language prior by pointwise division of the posterior distribution by the prior distribution and then normalizing; and (3) collapse-aware early stopping that scales the EOS logit based on the JS divergence between posterior and prior for recurring visual-relevant tokens. The method is evaluated on POPE, CHAIR, and MME across LLaVA-1.5-7b/13b, Qwen2.5-VL-7b, and Shikra, reporting consistent improvements over several baselines. The paper also includes component ablations, hyperparameter robustness studies, and inference-time comparisons.

Significance. If the empirical claims hold, EVRB is a practically attractive contribution: it is training-free, adds modest overhead (~715s vs. ~614s for vanilla on the reported POPE run, versus ~1150s for VCD/ICD), and shows consistent gains across multiple architectures and benchmarks. The strengths of the paper are its broad evaluation, component-wise ablations, robustness analysis in the appendix, and public code release. The main weakness is theoretical: the Bayesian derivation behind the central prior-rectification formula is flawed, and a key pruning premise is asserted without independent verification. These issues do not necessarily invalidate the empirical results, but they need to be addressed before the paper can be accepted.

major comments (4)
  1. [§3.3, Eq. (4)] The Bayesian derivation of the rectification formula is not mathematically correct. The paper states 'the posterior distribution is proportional to the prior distribution,' which is false; Bayes' rule gives posterior ∝ likelihood × prior. Dividing the posterior by the prior does not yield an ideal posterior; at best it yields the likelihood if the prior were uniform, but the normalization constant Z is then not the model evidence and the ε-regularized division is a heuristic. Since the 'Bayesian perspective' is the paper's stated motivation and the prior-rectification module is the largest contributor in the POPE ablations (Table 4, rows 2 vs. 3), the manuscript should either provide a correct derivation or explicitly reframe Eq. (4) as an ad-hoc prior-bias removal technique whose justification is empirical.
  2. [§3.2, Figs. 2, Table 4] The premise that high-entropy visual tokens are redundant and can be safely pruned is asserted but not independently validated. The paper provides a qualitative example and a bimodal histogram, but no evidence that high-entropy tokens are non-informative for grounding. The ablation in Table 4 weakens the premise: token pruning alone decreases POPE accuracy on MSCOCO (85.0 vs. vanilla 85.5) and gives only small gains on GQA and A-OKVQA compared to prior rectification alone. The authors should add an objective assessment of visual informativeness (e.g., object-detection recall on pruned tokens, or attention-based localization accuracy) to distinguish removing noise from removing signal. This is load-bearing for the claim that EVRB 'enhances visual reliance.'
  3. [§4.3, Table 11] The vocabulary truncation ratio μ is set per benchmark: 0.1 for POPE, 0.9 for CHAIR, and 0.85 for MME. The robustness analysis in Table 11 shows that POPE accuracy drops from 0.848 to 0.838 when μ changes from 0.10 to 0.13, indicating sensitivity on the test set. The paper's claim that hyperparameters generalize across models is therefore limited, since μ is tuned on the same benchmarks used for the headline results. The authors should either fix a single μ across benchmarks, report results on a held-out validation split, or clearly separate tuning from final evaluation to avoid benchmark-specific fitting.
  4. [Table 1, POPE-MSCOCO row] The Ours row for POPE-MSCOCO is numerically identical to the CausalMM row (85.6 accuracy, 88.3 precision, 82.4 recall, 85.2 F1). This is highly suspicious and needs clarification. If it is a transcription error, it must be corrected; if not, the claim of outperforming previous state-of-the-art on this subset is not supported, and the paper should discuss the tie.
minor comments (6)
  1. [Table 5] The column header 'A-OKVPA' is a typo and should read 'A-OKVQA'.
  2. [§4.2] The phrase 'previous training-tree methods' should be 'previous training-free methods'.
  3. [§3.3, Eq. (6)] The notation for the truncated vocabulary set Φ' is used inconsistently; the definition uses μ but the superscript is dropped later. Please clarify the notation throughout.
  4. [Fig. 2(c)] The x-axis entropy bins are densely labeled and the boxplot whiskers are hard to read; consider enlarging the figure and labeling the bin edges more clearly.
  5. [Appendix Table 11] The rendering of the robustness table is garbled in the text; the columns for λ, δ, τ, and μ should be clearly separated so that each hyperparameter sweep is readable.
  6. [Table 6] The table heading 'Ambiguous Visual Info' with 'w.' and 'w/o.' is unclear; use 'with v_b' and 'without v_b' to match the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No equation-level circularity: EVRB is an empirical, training-free decoding intervention; per-benchmark hyperparameter choices are a validation concern, not a circular derivation.

full rationale

I walked the derivation chain of EVRB and found no load-bearing circular step. The three modules are defined by explicit transformations of model distributions: entropy-gated visual token pruning (Eq. 3), prior-rectified next-token probabilities (Eqs. 4-6), and JS-divergence-based EOS logit scaling (Eqs. 7-9). None of these formulas defines a target quantity in terms of its own fitted values, and none of the reported hallucination metrics is fed back into the method's definitions. The entropy-based definition of 'redundant' visual tokens is an asserted assumption with only qualitative support (Sec. 3.2, Fig. 2), not a circular reduction: the claimed benefit is benchmark-observed, not logically forced by the definition. Similarly, the 'Bayesian' prior rectification is a heuristic likelihood-style correction, and the early stopping rule is a monitoring heuristic; both could be wrong, but they are not circular. There are no load-bearing self-citations (the cited contrastive-decoding and attention works are external), and no uniqueness theorem is imported from the authors' prior work. The closest weakness is that the truncation ratio mu is set separately per benchmark (Sec. 4.3: 'the probability truncation ratio mu ... is set to 0.1 for POPE, 0.9 for CHAIR, and 0.85 for MME') and the appendix sweeps hyperparameters on the same benchmarks, which weakens the generalization strength of the headline results. That is a selection artifact rather than a derivation-level circularity, so the appropriate verdict is no significant circularity, score 0.

Assumptions & free parameters 5 free parameters · 7 assumptions · 0 invented entities

The central claim rests on several hand-made assumptions: the entropy proxy for token informativeness, the uniform-prior idealization, and the JS-divergence collapse detector. These are not derived from first principles and each is validated only indirectly through benchmark ablations. The method also depends on five hyperparameters, several of which are tuned per benchmark, so the 'training-free' claim is best understood as 'no gradient updates' rather than 'no tuning'.

free parameters (5)
  • Entropy threshold tau = 7.48
    Threshold for pruning visual tokens with high entropy; chosen as the valley of the bimodal entropy distribution on MSCOCO (Fig. 2c, Sec. 4.3), i.e., selected from the evaluation data distribution.
  • Visual-relevant token entropy drop threshold delta = 0.05
    Threshold on the change in value-value attention entropy to decide a token is visual-relevant; tuned for balance between hallucination and recall (Table 11).
  • EOS scaling ratio lambda = 1.5
    Controls how strongly JS-divergence decline boosts the EOS logit; tuned on CHAIR (Table 11).
  • Vocabulary truncation ratio mu = 0.1 (POPE), 0.9 (CHAIR), 0.85 (MME)
    Controls the plausibility set size in prior rectification; set separately for each benchmark (Sec. 4.3), i.e., fitted per evaluation dataset.
  • Epsilon epsilon = small positive constant (unspecified)
    Added to the denominator of Eq. 4 to avoid division by zero; its value is not reported.
assumptions (7)
  • domain assumption Applying the LM head to image token hidden states yields next-token distributions that reflect the visual informativeness of those tokens (Eq. 2).
    The whole pruning strategy relies on the LM head's predictions over the frozen vocabulary being a meaningful proxy for visual token quality; stated in Section 3.2 without external verification.
  • ad hoc to paper The ideal language prior for visual tasks is uniform over a certain vocabulary subset.
    Used to justify dividing the posterior by the prior in Eq. 4; no evidence is given that a uniform prior is optimal or that the truncation set Phi' recovers it.
  • domain assumption Tokens with high entropy in the proxy distribution are redundant and can be removed without losing information needed for grounding.
    The pruning step deletes visual tokens with E(v_i) > tau; if this is false, pruning would discard signal. Only benchmark-level ablations support it.
  • ad hoc to paper The text-only distribution p(t|t) estimated with redundant visual tokens v_b in the input approximates the language prior.
    Section 3.3 uses pruned tokens to make the prior sample-specific; this reuses the method's own pruning output to define the prior, which is a heuristic with no derivation.
  • domain assumption A significant decrease in value-value attention entropy (delta E < -delta) identifies text tokens whose prediction is visually grounded.
    Section 3.4 relies on value-value attention maps to select visual-relevant tokens, citing CLIP-based works; the transfer of self-self attention to LVLM value features is assumed.
  • ad hoc to paper JS divergence between posterior and prior on recurring visual-relevant tokens monotonically decreasing indicates imminent posterior collapse, and boosting the EOS logit after punctuation is a safe way to stop.
    The early-stopping rule (Eq. 9) and the restriction to tokens following punctuation are heuristic design choices evaluated only on CHAIR.
  • domain assumption The model's next-token posterior p(t|v,t) can be approximated by two parallel decodings, with and without clear visual tokens, using the same KV-cache.
    The pipeline in Section 3.5 assumes that running a second decoding without v_c produces a valid prior and that reusing the cache is equivalent to a true Bayesian decomposition.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Visual Reliance in Text Generation: A Bayesian Perspective on Mitigating Hallucination in Large Vision-Language Models." pith.science (2026). https://pith.science/paper/JKLAP42L

@misc{pith2026250519498,
  author       = {Pith},
  title        = {Pith review of: Enhancing Visual Reliance in Text Generation: A Bayesian Perspective on Mitigating Hallucination in Large Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JKLAP42L}},
  note         = {Machine review of arXiv:2505.19498}
}
read the original abstract

Large Vision-Language Models (LVLMs) usually generate texts which satisfy context coherence but don't match the visual input. Such a hallucination issue hinders LVLMs' applicability in the real world. The key to solving hallucination in LVLM is to make the text generation rely more on the visual content. Most previous works choose to enhance/adjust the features/output of a specific modality (i.e., visual or textual) to alleviate hallucinations in LVLM, which do not explicitly or systematically enhance the visual reliance. In this paper, we comprehensively investigate the factors which may degenerate the visual reliance in text generation of LVLM from a Bayesian perspective. Based on our observations, we propose to mitigate hallucination in LVLM from three aspects. Firstly, we observe that not all visual tokens are informative in generating meaningful texts. We propose to evaluate and remove redundant visual tokens to avoid their disturbance. Secondly, LVLM may encode inappropriate prior information, making it lean toward generating unexpected words. We propose a simple yet effective way to rectify the prior from a Bayesian perspective. Thirdly, we observe that starting from certain steps, the posterior of next-token prediction conditioned on visual tokens may collapse to a prior distribution which does not depend on any informative visual tokens at all. Thus, we propose to stop further text generation to avoid hallucination. Extensive experiments on three benchmarks including POPE, CHAIR, and MME demonstrate that our method can consistently mitigate the hallucination issue of LVLM and performs favorably against previous state-of-the-arts.

Figures

Figures reproduced from arXiv: 2505.19498 by the authors.

Figure 1
Figure 1. Hallucination causes from a Bayesian view. The posterior inference process of LVLM (left-up) is conditioned on both visual [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The next token predictions of image tokens. (a) depicts [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Generated caption for Figure [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (4 more)
Figure 3
Figure 3. Figure 3: The text-to-image query-key attention maps (a) and [PITH_FULL_IMAGE:figures/full_fig_p005_3.png]
Figure 5
Figure 5. Figure 5: Comparison on comprehensive task: MME evaluation. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Additionaly text-to-visual attention maps samples [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Case study for the hallucination mitigation in generative (a) and discriminative (b) tasks [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

55 extracted references · 19 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  2. [2]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023. 3

    arXiv 2023

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 7, 9, 13

    arXiv 2025

  4. [4]

    Grounding everything: Emerging localiza- tion properties in vision-language transformers

    Walid Bousselham, Felix Petersen, Vittorio Ferrari, and Hilde Kuehne. Grounding everything: Emerging localiza- tion properties in vision-language transformers. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3828–3837, 2024. 5

    work page 2024

  5. [5]

    Evaluating a large language model on searching for gui layouts.Proceedings of the ACM on Human-Computer Interaction, 7(EICS):1–37, 2023

    Paul Brie, Nicolas Burny, Arthur Slu ¨yters, and Jean Vander- donckt. Evaluating a large language model on searching for gui layouts.Proceedings of the ACM on Human-Computer Interaction, 7(EICS):1–37, 2023. 1

    work page 2023

  6. [6]

    Lan- guage models are few-shot learners.Advances in neural in- formation processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners.Advances in neural in- formation processing systems, 33:1877–1901, 2020. 3

    1901

  7. [7]

    Shikra: Unleashing multi- modal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023. 13, 14

    arXiv 2023

  8. [8]

    In-context sharpness as alerts: An inner representation perspective for hallucination mitigation.arXiv preprint arXiv:2403.01548, 2024

    Shiqi Chen, Miao Xiong, Junteng Liu, Zhengxuan Wu, Teng Xiao, Siyang Gao, and Junxian He. In-context sharpness as alerts: An inner representation perspective for hallucination mitigation.arXiv preprint arXiv:2403.01548, 2024. 3

    arXiv 2024

Show all 55 references
  1. [9]

    Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1–113, 2023

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1–113, 2023. 3

    2023

  2. [10]

    Dola: Decoding by con- trasting layers improves factuality in large language models

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by con- trasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03883, 2023. 5

    2023 arXiv

  3. [11]

    A survey on multimodal large lan- guage models for autonomous driving

    Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al. A survey on multimodal large lan- guage models for autonomous driving. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vis...

    2024

  4. [12]

    Sadl: An effective in-context learning method for compositional visual qa.arXiv preprint arXiv:2407.01983, 2024

    Long Hoang Dang, Thao Minh Le, Vuong Le, Tu Minh Phuong, and Truyen Tran. Sadl: An effective in-context learning method for compositional visual qa.arXiv preprint arXiv:2407.01983, 2024. 3

    2024 arXiv

  5. [13]

    Mme: A comprehensive evaluation bench- mark for multimodal large language models, 2024.URL https://arxiv

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation bench- mark for multimodal large language models, 2024.URL https://arxiv. org/abs/2306.13394, 2. 3, 7

    2024 arXiv

  6. [14]

    Chatgpt outperforms crowd workers for text-annotation tasks.Proceedings of the National Academy of Sciences, 120(30):e2305016120, 2023

    Fabrizio Gilardi, Meysam Alizadeh, and Ma ¨el Kubli. Chatgpt outperforms crowd workers for text-annotation tasks.Proceedings of the National Academy of Sciences, 120(30):e2305016120, 2023. 3

    2023

  7. [15]

    Detecting and preventing hallucinations in large vision language models

    Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. InProceedings of the AAAI Conference on Artificial Intelli- gence, volume 38, pages 18135–18143, 2024. 3

    2024

  8. [16]

    A comprehensive survey of deep learn- ing for image captioning.ACM Computing Surveys (CsUR), 51(6):1–36, 2019

    MD Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratud- din, and Hamid Laga. A comprehensive survey of deep learn- ing for image captioning.ACM Computing Surveys (CsUR), 51(6):1–36, 2019. 3

    2019

  9. [17]

    Advancing medical imaging with language mod- els: A journey from n-grams to chatgpt.arXiv preprint arXiv:2304.04920, 2023

    Mingzhe Hu, Shaoyan Pan, Yuheng Li, and Xiaofeng Yang. Advancing medical imaging with language mod- els: A journey from n-grams to chatgpt.arXiv preprint arXiv:2304.04920, 2023. 1

    2023 arXiv

  10. [18]

    Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Con- ghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation. InProceedings of the IEEE/CVF Confere...

    2024

  11. [19]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 6700–6709, 2019. 7

    2019

  12. [20]

    Self-introspective de- coding: Alleviating hallucinations for large vision-language models.arXiv preprint arXiv:2408.02032, 2024

    Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. Self-introspective de- coding: Alleviating hallucinations for large vision-language models.arXiv preprint arXiv:2408.02032, 2024. 1, 3, 5, 7, 8, 14

    2024 arXiv

  13. [21]

    Survey of hallucination in natural language generation.ACM computing surveys, 55(12):1–38, 2023

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation.ACM computing surveys, 55(12):1–38, 2023. 3

    2023

  14. [22]

    Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recogni...

    2024

  15. [23]

    Contrastive decoding: Open-ended text gen- eration as optimization.arXiv preprint arXiv:2210.15097,

    Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text gen- eration as optimization.arXiv preprint arXiv:2210.15097,

    arXiv

  16. [24]

    Clip surgery for better explainability with enhancement in open- vocabulary tasks.arXiv e-prints, pages arXiv–2304, 2023

    Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement in open- vocabulary tasks.arXiv e-prints, pages arXiv–2304, 2023. 5

    2023

  17. [25]

    Evaluating object hallucina- tion in large vision-language models.arXiv preprint arXiv:2305.10355, 2023

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models.arXiv preprint arXiv:2305.10355, 2023. 1, 3, 4, 6, 7, 8, 13

    2023 arXiv

  18. [26]

    A survey of multimodel large language models

    Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. A survey of multimodel large language models. InProceedings of the 3rd Interna- tional Conference on Computer, Artificial Intelligence and Control Engineering, pages 405–409, 2024. 1, 3

    2024

  19. [27]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceeding...

    2014

  20. [28]

    Aligning large multi-modal model with robust instruction tuning.CoRR, 2023

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Ya- coob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning.CoRR, 2023. 3

    2023

  21. [29]

    Mitigating hallucination in large multi-modal models via robust instruction tuning.arXiv preprint arXiv:2306.14565, 2023

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Ya- coob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning.arXiv preprint arXiv:2306.14565, 2023. 7

    2023 arXiv

  22. [30]

    A survey on hallucination in large vision-language models

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiu- tian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253, 2024. 3

    2024 arXiv

  23. [31]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 1, 3, 4, 7, 8, 9, 13

    2024

  24. [32]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023. 1, 3

    2023

  25. [33]

    Insights into llm long-context failures: When transformers know but don’t tell.arXiv preprint arXiv:2406.14673, 2024

    Taiming Lu, Muhan Gao, Kuai Yu, Adam Byerly, and Daniel Khashabi. Insights into llm long-context failures: When transformers know but don’t tell.arXiv preprint arXiv:2406.14673, 2024. 3

    2024 arXiv

  26. [34]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  27. [35]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020. 3

    2020

  28. [36]

    Object hallucination in image cap- tioning.arXiv preprint arXiv:1809.02156, 2018

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image cap- tioning.arXiv preprint arXiv:1809.02156, 2018. 3, 6, 7, 13

    2018 arXiv

  29. [37]

    A-okvqa: A benchmark for visual question answering using world knowl- edge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowl- edge. InEuropean conference on computer vision, pages 146–162. Springer, 2022. 7

    2022

  30. [38]

    Aligning large multi- modal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525, 2023

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multi- modal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525, 2023. 3

    2023 arXiv

  31. [39]

    Stanford alpaca: An instruction-following llama model, 2023

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023. 3

    2023

  32. [40]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 3

    2023 arXiv

  33. [41]

    Mllm can see? dynamic correction decoding for hallucination mitiga- tion.arXiv preprint arXiv:2410.11779, 2024

    Chenxi Wang, Xiang Chen, Ningyu Zhang, Bozhong Tian, Haoming Xu, Shumin Deng, and Huajun Chen. Mllm can see? dynamic correction decoding for hallucination mitiga- tion.arXiv preprint arXiv:2410.11779, 2024. 3, 7, 8

    2024 arXiv

  34. [42]

    Sclip: Rethink- ing self-attention for dense vision-language inference

    Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethink- ing self-attention for dense vision-language inference. In European Conference on Computer Vision, pages 315–332. Springer, 2024. 5

    2024

  35. [43]

    Surgical-lvlm: Learning to adapt large vision-language model for grounded visual question answering in robotic surgery.arXiv preprint arXiv:2405.10948, 2024

    Guankun Wang, Long Bai, Wan Jun Nah, Jie Wang, Zhaoxi Zhang, Zhen Chen, Jinlin Wu, Mobarakol Islam, Hong- bin Liu, and Hongliang Ren. Surgical-lvlm: Learning to adapt large vision-language model for grounded visual question answering in robotic surgery.arXiv preprint arXiv:240...

    2024 arXiv

  36. [44]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 1, 3

    2024 arXiv

  37. [45]

    Chatcad: Interactive computer-aided diag- nosis on medical image using large language models.arXiv preprint arXiv:2302.07257, 2023

    Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen. Chatcad: Interactive computer-aided diag- nosis on medical image using large language models.arXiv preprint arXiv:2302.07257, 2023. 1

    2023 arXiv

  38. [46]

    Caption anything: Interactive image description with diverse multi- modal controls.arXiv preprint arXiv:2305.02677, 2023

    Teng Wang, Jinrui Zhang, Junjie Fei, Hao Zheng, Yunlong Tang, Zhe Li, Mingqi Gao, and Shanshan Zhao. Caption anything: Interactive image description with diverse multi- modal controls.arXiv preprint arXiv:2305.02677, 2023. 3

    2023 arXiv

  39. [47]

    Drivemlm: Aligning multi-modal large language models with behavioral planning states for au- tonomous driving.arXiv preprint arXiv:2312.09245, 2023

    Wenhai Wang, Jiangwei Xie, ChuanYang Hu, Haoming Zou, Jianan Fan, Wenwen Tong, Yang Wen, Silei Wu, Hanming Deng, Zhiqi Li, et al. Drivemlm: Aligning multi-modal large language models with behavioral planning states for au- tonomous driving.arXiv preprint arXiv:2312.09245, 2023. 1

    2023

  40. [48]

    Mitigating hallucinations in large vision-language models with instruction contrastive decoding.arXiv preprint arXiv:2403.18715, 2024

    Xintong Wang, Jingheng Pan, Liang Ding, and Chris Bie- mann. Mitigating hallucinations in large vision-language models with instruction contrastive decoding.arXiv preprint arXiv:2403.18715, 2024. 1, 3, 5, 7, 8, 9

    2024 arXiv

  41. [49]

    Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024. 1, 3 11

    2024 arXiv

  42. [50]

    Woodpecker: Hallucination correction for multimodal large language models.Science China Information Sciences, 67(12):220105, 2024

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models.Science China Information Sciences, 67(12):220105, 2024. 3

    2024

  43. [51]

    Less is more: Mitigat- ing multimodal hallucination from an eos decision perspec- tive.arXiv preprint arXiv:2402.14545, 2024

    Zihao Yue, Liang Zhang, and Qin Jin. Less is more: Mitigat- ing multimodal hallucination from an eos decision perspec- tive.arXiv preprint arXiv:2402.14545, 2024. 3

    2024 arXiv

  44. [52]

    Huatuogpt, towards taming language model to be a doctor.arXiv preprint arXiv:2305.15075, 2023

    Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhi- hong Chen, Jianquan Li, Guiming Chen, Xiangbo Wu, Zhiyi Zhang, Qingying Xiao, et al. Huatuogpt, towards taming language model to be a doctor.arXiv preprint arXiv:2305.15075, 2023. 1

    2023 arXiv

  45. [53]

    Beyond hallucinations: Enhanc- ing lvlms through hallucination-aware direct preference op- timization.arXiv preprint arXiv:2311.16839, 2023

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Ji- aqi Wang, and Conghui He. Beyond hallucinations: Enhanc- ing lvlms through hallucination-aware direct preference op- timization.arXiv preprint arXiv:2311.16839, 2023. 3

    2023 arXiv

  46. [54]

    Mitigating modality prior-induced halluci- nations in multimodal large language models via deciphering attention causality.arXiv preprint arXiv:2410.04780, 2024

    Guanyu Zhou, Yibo Yan, Xin Zou, Kun Wang, Aiwei Liu, and Xuming Hu. Mitigating modality prior-induced halluci- nations in multimodal large language models via deciphering attention causality.arXiv preprint arXiv:2410.04780, 2024. 3, 7, 8

    arXiv

  47. [55]

    truck” Value-Value attention Query-Key attention “bicycle

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023. 3 12 APPENDIX 5.1. Validations on more models To further validate the generality...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.