Pith. sign in

REVIEW 5 major objections 5 minor 98 references

Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FarSight, a plug-and-play decoding strategy, claims that hallucinations in multimodal language models can be cut by replacing the standard causal mask with one that adds a decaying attention register for outlier tokens and a diminishing…

desk verdict The method is a simple and broadly tested decoding tweak, but the headline mechanism—attention registers reallocating attention away from outlier tokens—is not what the equations actually do. read the letter →

arxiv 2505.16652 v2 pith:UGYN4LX2 submitted 2025-05-22 cs.CV cs.LG

Feilong Tang , Chengzhi Liu , Zhongxing Xu , Ming Hu , Zelin Peng , Zhiwei Yang , Jionglong Su , Minquan Lin
show 4 more authors
Yifan Peng Xuelian Cheng Imran Razzak Zongyuan Ge
This is my paper
classification cs.CVcs.LG
keywords multimodallargelanguagemodelshallucinationmitigationdecodingstrategycausalmaskattentionregisterspositionalencodingvideounderstandingvisualquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FarSight claims that a large share of hallucinations in multimodal language models is caused by how attention is distributed during decoding, not by missing knowledge. The paper identifies two faults: attention collapse, where low-information tokens such as image backgrounds and punctuation absorb a disproportionate share of attention, and positional information decay, where the model progressively stops attending to visual tokens as it generates text. To fix both, FarSight replaces the standard causal mask with one that inserts a decaying "register" of extra attention scores in the upper triangle, so surplus attention is absorbed before softmax, and re-masks afterward to keep causality. The result is a training-free, plug-and-play decoder change that the paper reports cuts hallucination rates on CHAIR by up to 6.4 points and raises object-existence accuracy on POPE by 3.5 points on standard models, with smaller gains on video benchmarks. A reader should care because it suggests hallucination can be mitigated by controlling attention geometry alone, without retraining or extra inference passes.

What carries the argument

The load-bearing object is the attention register: an upper-triangular matrix $P$ with entries $P_{i,j}=-(j-i)\sigma$ for $j>i$, added inside the softmax to the usual lower-triangular causal mask $\omega\cdot C$. The combined score $W=\omega\cdot C+P$ is passed through softmax, then multiplied by $C$ again so masked future positions are reset to zero. During normalization the finite decaying register scores give outlier or future positions a place to absorb excess probability mass, which prevents low-information tokens from dominating; the re-masking afterward preserves the causal decoding property. The positional-awareness component is the same matrix viewed differently: because the register steals probability mass that would otherwise be reserved for masked positions, the sum of surviving attention weights is no longer 1 and increases monotonically with sequence position, encoding absolute position and keeping attention anchored to earlier visual tokens as generation proceeds. The decay rate is fixed as $\sigma=\log_{1024}(256)\approx0.8$ with sequence length 256.

What would settle it

A concrete test: take the same six models and run FarSight with sequence lengths 128, 256, and 512 and decay rates around 0.5, 0.8, and 1.0 on POPE and CHAIR; if the gains concentrate only at 256 and 0.8 and disappear or reverse elsewhere, the reported improvements are an artifact of the tuned schedule rather than a general property of the mask. A complementary check is to measure attention mass on true visual tokens versus hallucinated-object tokens in the layers where the register is applied; if FarSight does not measurably shift mass away from outlier tokens, the proposed causal mechanism is not what drives the benchmark gains.

Watch

Extended reading notes

Core claim

The paper's central claim is that two self-attention faults drive a large part of MLLM hallucination: attention collapse, in which low-information tokens such as image backgrounds and punctuation receive a disproportionate share of probability mass, and positional information decay, in which the relative-distance weighting of rotary position embedding progressively reduces attention to early visual tokens as text is generated. FarSight addresses both at decoding time by modifying the causal mask. Instead of masking future positions to -infinity, it fills the upper-triangular positions with finite, linearly decaying register scores that absorb excess attention before softmax, then re-applies causal masking after softmax so generation remains causal. The same construction acts as a positional encoding: because the surviving attention mass over valid tokens is no longer forced to sum to one and grows with row index, later tokens accumulate more context from earlier visual tokens. On this account the intervention is purely a mask change, requiring no training, data, or extra decoding passes, and the paper reports consistent hallucination reductions across six image and video models, with CHAIRS dropping from 48.0 to 41.6 and POPE-R rising from 87.0 to 90.5 on the LLaVA-1.5 model.

Load-bearing premise

The load-bearing premise is that a single hand-chosen decay value, $\sigma\approx0.8$ paired with a 256-token sequence window, generalizes across models, prompts, and tasks; the paper selects these values after observing peak performance on the evaluation benchmarks, so if they do not transfer, the method needs per-deployment retuning.

Editorial extensions

If this is right

  • Plugging FarSight into LLaVA-1.5 lowers CHAIRS from 48.0 to 41.6 and raises POPE-R from 87.0 to 90.5, with comparable gains on InstructBLIP, Video-LLaVA, Chat-UniVi, VILA, and Video-LLaMA2.
  • On zero-shot video QA, FarSight improves accuracy by about 3 points on MSRVTT-QA across models and by smaller margins on MSVD-QA and ActivityNet-QA.
  • GPT-assisted evaluation of 600 MSCOCO samples indicates that the decoded text keeps comparable perplexity, grammar, fluency, and naturalness relative to vanilla decoding and prior decoding baselines.
  • Because the intervention is confined to the attention mask, it is applicable to greedy, sampling, and beam-search decoding with no training, data augmentation, or added inference time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, a natural test is whether the same upper-triangular register trick transfers to text-only LLM hallucination and long-context extrapolation, since the claimed mechanism is about attention geometry rather than vision specifically.
  • The paper's own ablation shows sharp sensitivity to the 256-token window and 0.8 decay rate, so an adaptive per-head or per-layer decay schedule is a plausible next step; if the fixed values fail on other models, adaptation would be the remedy.
  • A testable extension is to combine FarSight with stochastic decoding strategies at higher temperature, since the renormalization changes the effective logit distribution and could alter sampling behavior.
  • The paper's automatic initial-versus-snowball hallucination classification could be applied to measure whether FarSight reduces snowball continuations specifically, rather than only the first hallucinated object.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes FarSight, a training-free decoding strategy for multimodal large language models (MLLMs) that modifies the causal attention mask. The method adds an upper-triangular 'attention register' with linearly decaying negative scores to the pre-softmax attention matrix and then re-masks the upper-triangular entries after softmax. A 'positional awareness encoding' with a diminishing masking rate is introduced to preserve attention to earlier tokens, particularly in video tasks. The authors claim that this mechanism reduces attention interference from outlier tokens and thereby mitigates initial and snowball hallucinations. Experiments are reported on six MLLMs across image benchmarks (MMBench, LLaVA-W, MM-Vet, VizWiz, SQA, POPE, CHAIR) and video benchmarks (MSVD-QA, ActivityNet-QA, and a video-based text generation benchmark), with consistent but modest improvements, e.g., CHAIRS +6.4 and POPE-P +3.3 on LLaVA-1.5.

Significance. If the empirical gains were robust and the mechanism correctly characterized, a plug-and-play, training-free decoding intervention that reduces hallucinations across many MLLMs would be a useful contribution to the multimodal hallucination literature. The paper is commendable for evaluating across six models and many benchmarks, including both image and video tasks, and for providing qualitative attention visualizations. However, the central mechanistic claim is not supported by the paper's own equations: the operation in Eq. (9) rescales all visible-token probabilities by a common row-dependent factor and does not reallocate attention away from outlier tokens. The theoretical Proposition 3.1 is stated without proof and is not well-formed. In addition, the two main hyperparameters (decay rate sigma and sequence length 256) are selected using the evaluation benchmarks themselves, and no error bars or significance tests are provided. The paper also contains a foreign figure panel and an undisclosed CVPR relationship. The broad evaluation is a strength, but the mismatch between the claimed mechanism and the actual computation is a load-bearing problem that requires substantial rework.

major comments (5)
  1. [§4.1, Eq. (9)] The proposed mechanism does not reallocate attention among visible tokens. For a row i with visible positions j,k ≤ i, the final FarSight attention weights are exp(ω_{i,j}) / (Σ_{m≤i} exp(ω_{i,m}) + R_i) and exp(ω_{i,k}) / (Σ_{m≤i} exp(ω_{i,m}) + R_i), where R_i = Σ_{d≥1}^{n-i} exp(-dσ). Their ratio is exactly exp(ω_{i,j})/exp(ω_{i,k}), identical to the vanilla causal softmax ratio and independent of P. The upper-triangular register mass R_i is discarded by the final multiplication by C, so it only contributes a row-dependent scaling factor c_i = Z_i/(Z_i + R_i). Thus FarSight implements a position-dependent dampening of the whole attention output, not a dynamic reallocation that 'captures attention diverted to outlier tokens' as claimed in the abstract and Section 4.1. This is a load-bearing mismatch between the method's stated purpose and its equations.
  2. [§4.1, Algorithm 1 vs. Eqs. (7)-(9)] The pseudocode in Algorithm 1 computes scores = scores * C * sigma + register_score, whereas Eq. (7) defines W = ω·C + P with P_{i,j} = -(j-i)σ and no multiplication of the valid scores by σ. These two formulations are not equivalent, and the difference changes the numerical attention probabilities. The paper's formal analysis refers to Eq. (9), but the experiments presumably use the implementation, so the discrepancy must be resolved and the actual operation analyzed.
  3. [§3.2, Proposition 3.1] Proposition 3.1 is stated without proof, and its displayed inequality is not well-formed: the index i in the sums over n is not quantified, and the term o(1) has no specified limiting regime. The notions 'contextual, layer-wise decoder' and 'disproportionality' are not formally defined. As written, the proposition does not establish a connection between attention weights and mutual information, and the remark's conclusion that attention weights 'far exceed' informational contribution does not follow from the displayed expression. The proposition should either be given a complete proof with precise assumptions or be reformulated as a conjecture.
  4. [§5.1-§5.2, Fig. 6] The decay factor σ = log_1024(256) = 0.8 and the sequence length 256 are selected by scanning performance on POPE-R and MSRVTT-QA, which are themselves part of the evaluation suite. The main reported gains, such as CHAIRS +6.4 and POPE-R +3.5, are therefore partially a product of hyperparameters fitted to the test benchmarks. No error bars, multiple-seed results, or significance tests are reported for any table. Please provide variance estimates, fix the hyperparameters on a validation split, or demonstrate robustness across a wide range of σ and sequence lengths.
  5. [§1, Fig. 1; §7] Figure 1 contains a pasted panel from another paper, with visible text 'Under review as a conference paper at ICLR 2025 ... Figure 12 ... Antidote ... DeepSeek-V2 ... Stable Diffusion 3 Medium', which is unrelated to the present manuscript. Similarly, Section 7 thanks 'the remaining co-authors of the CVPR 2025 version of this work' without disclosing the relationship between this arXiv manuscript and that CVPR publication. The foreign panel must be removed, and the provenance and prior-publication relationship must be clarified.
minor comments (5)
  1. [§5.3] The text reports an average accuracy gain of +3% on MSRVTT-QA with a peak of 68.9%, but Table 3 does not contain an MSRVTT-QA column; please either add the missing results or remove the unsupported claim.
  2. [§5.2] The heading 'Abalation Study' is misspelled; it should read 'Ablation Study'.
  3. [Algorithm 1] The pseudocode comment defines C as an 'upper-triangular matrix filled with 1', but Eq. (7) defines C = tril(1_{n×n}); this notation is inconsistent and should be corrected.
  4. [Eq. (4)] The asymptotic term o(1) in Proposition 3.1 needs an explicit limit; please state what tends to infinity and under which data-generating process.
  5. [Fig. 7(c)] The caption refers to 'decay ratσ'; fix the typo and clearly specify the mapping from line styles or colors to the decay-rate values shown in the plot.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical evaluation is self-contained against external benchmarks, and the tuned hyperparameters and algebraic derivations are not fitted inputs renamed as predictions.

full rationale

FarSight's central claim of hallucination mitigation is supported by external benchmarks (CHAIR, POPE, MMBench, MSVD-QA, ActivityNet-QA, etc.) applied to frozen MLLMs, so the reported gains are not derived from the method's equations by construction. The hyperparameters (seq=256, sigma=log_1024(256) ~ 0.8) are selected by benchmark sweeps in Figs. 5 and 6; this is model selection / overfitting risk rather than a fitted parameter renamed as a prediction, because no per-sample or per-dataset fit is used to manufacture the scores. The positional-awareness derivation in Sec. 4.2 and Proposition 3.1 are internal algebraic statements about Eq. 7-9, not predictions imported from elsewhere, and they are not load-bearing circular justifications. The only self-citation, [66], appears in related work and is not used to justify FarSight's mechanism. A possible concern that Eq. 9 performs a per-row rescaling rather than a token-level reallocation away from outlier tokens is a correctness/mechanism mismatch, not circularity, because the empirical gains are not shown to be equivalent to the method's input by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The method depends on a single decay factor and sequence length that are fitted to the evaluation benchmarks. The central hypothesis about attention collapse and positional decay is an unproven domain assumption. The attention register is a computational abstraction, not a physical entity.

free parameters (2)
  • Decay rate sigma = 0.8 (log_1024(256))
    Chosen in Section 5.1; ablation in Fig. 6 shows peak performance at seq 256, indicating the hyperparameter was selected on the evaluation benchmarks.
  • Sequence length (seq) = 256
    Set to 256 based on performance on POPE-R and MSRVTT-QA (Fig. 6).
assumptions (4)
  • ad hoc to paper Proposition 3.1: Attention collapse inequality involving mutual information
    Stated without proof in Section 3.2; used to motivate the method but not derived.
  • domain assumption Hypothesis that hallucination is caused by attention collapse and positional information decay
    This is the paper's central hypothesis, assumed rather than derived; the method is built on it.
  • domain assumption RoPE long-term decay limits multimodal information propagation
    Assumed in Section 3.2 as a cause of hallucinations, with no quantitative evidence.
  • standard math Softmax normalization and causal masking properties
    Relied on for the derivation of the positional awareness encoding in Section 4.2.
invented entities (1)
  • Attention register
    purpose: Upper-triangular slots in the causal mask that absorb attention diverted to future or outlier tokens
    A computational construct introduced in Section 4.1; no falsifiable handle outside the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding." pith.science (2026). https://pith.science/paper/UGYN4LX2

@misc{pith2026250516652,
  author       = {Pith},
  title        = {Pith review of: Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UGYN4LX2}},
  note         = {Machine review of arXiv:2505.16652}
}
read the original abstract

Recent advancements in multimodal large language models (MLLMs) have significantly improved performance in visual question answering. However, they often suffer from hallucinations. In this work, hallucinations are categorized into two main types: initial hallucinations and snowball hallucinations. We argue that adequate contextual information can be extracted directly from the token interaction process. Inspired by causal inference in the decoding strategy, we propose to leverage causal masks to establish information propagation between multimodal tokens. The hypothesis is that insufficient interaction between those tokens may lead the model to rely on outlier tokens, overlooking dense and rich contextual cues. Therefore, we propose to intervene in the propagation process by tackling outlier tokens to enhance in-context inference. With this goal, we present FarSight, a versatile plug-and-play decoding strategy to reduce attention interference from outlier tokens merely by optimizing the causal mask. The heart of our method is effective token propagation. We design an attention register structure within the upper triangular matrix of the causal mask, dynamically allocating attention to capture attention diverted to outlier tokens. Moreover, a positional awareness encoding method with a diminishing masking rate is proposed, allowing the model to attend to further preceding tokens, especially for video sequence tasks. With extensive experiments, FarSight demonstrates significant hallucination-mitigating performance across different MLLMs on both image and video benchmarks, proving its effectiveness.

Figures

Figures reproduced from arXiv: 2505.16652 by the authors.

Figure 1
Figure 1. Illustrates the phenomenon of snowball hallucinations [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 12
Figure 12. Examples of FPQs generated by the data synthesis pipeline. “Hallucination can￾didates” are the non-existent objects that commonly co-occur in the similar scenes, generated by DeepSeek-V2 (Liu et al., 2024a). The images are generated by Stable Diffusion 3 Medium (Pee￾bles & Xie, 2023). These cases are selected during the construction of the training set for Antidote. 20 [PITH_FULL_IMAGE:figures/full_fig_p001_12.png] view at source ↗
Figure 2
Figure 2. Percentage of initial hallucination (IH) and percentage [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figures from the paper (11 more)
Figure 3
Figure 3. Figure 3: (a): Attention Collapse in MLLMs: Outlier tokens from [PITH_FULL_IMAGE:figures/full_fig_p002_3.png]
Figure 4
Figure 4. Figure 4: The scheme of the proposed FarSight strategy, which [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Comparison of different Upper Triangular Attention Values in Attention Registers. (a) and (b) show model per￾formance with varying upper triangular attention values on the CHAIR and POPE-P datasets. Sequence Length MSRVTT-QA LLaVA-1.5 Video-LLaVA InstructBLIP Chat-UniV…
Figure 6
Figure 6. Figure 6: The impact of sequence length on attention decay and [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Qualitative Visualization of FarSight in Image Understanding Task on LLaVA-1.5. (a) Comparison of the average attention [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: The average performance is evaluated on a randomly [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Qualitative Visualization Example 1. The attention distribution of Video-LLaVA is relatively scattered and inconsistent, failing [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Qualitative Visualization Example 2. The attention distribution of Video-LLaVA fails to consistently focus on the man and [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Qualitative Visualization Example 3. Video-LLaVA’s attention lacks sufficient temporal coherence. In contrast, FarSight [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]
Figure 12
Figure 12. Figure 12: LLaVA-1.5 Qualitative Visualization. Query & Answer Input Image Base EDVT FarSight (Ours) Ours: The image shows a vintage military aircraft with a streamlined nose and the number "J8" marked in white on its fuselage, likely for identification. The dark green color ind…
Figure 13
Figure 13. Figure 13: Video LLaVA Qualitative Visualization. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

98 extracted references · 24 canonical work pages

  1. [1]

    Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966,

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966,

    arXiv

  2. [2]

    Wiki-llava: Hierarchical retrieval-augmented generation for multimodal llms

    Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Wiki-llava: Hierarchical retrieval-augmented generation for multimodal llms. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1818–1826, 2024. 1

    2024

  3. [3]

    Shikra: Unleashing multi- modal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023. 1

    arXiv 2023

  4. [4]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 3

    arXiv 2023

  5. [5]

    Videollama 2: Advanc- ing spatial-temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advanc- ing spatial-temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024. 6

    arXiv 2024

  6. [6]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.See https://vicuna

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.See https://vicuna. lmsys. org (accessed 14 April 2023), 2023. 3

    2023

  7. [7]

    Fine- grained image captioning with clip reward.arXiv preprint arXiv:2205.13115, 2022

    Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, and Mohit Bansal. Fine- grained image captioning with clip reward.arXiv preprint arXiv:2205.13115, 2022. 3

    arXiv 2022

  8. [8]

    Dola: Decoding by con- trasting layers improves factuality in large language mod- els.arXiv preprint arXiv:2309.03883, 2023

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by con- trasting layers improves factuality in large language mod- els.arXiv preprint arXiv:2309.03883, 2023. 3

    arXiv 2023

Show all 98 references
  1. [10]

    Instructblip: Towards general- purpose vision-language models with instruction tuning,

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning,

  2. [11]

    Internlm-xcomposer2: Mastering free-form text-image composition and compre- hension in vision-language large model.arXiv preprint arXiv:2401.16420, 2024

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internlm-xcomposer2: Mastering free-form text-image composition and compre- hension in vision-language large model.arXiv preprint arXiv:2401.16...

    2024 arXiv

  3. [12]

    Layer skip: Enabling early exit inference and self- speculative decoding.arXiv preprint arXiv:2404.16710,

    Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mah- moud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layer skip: Enabling early exit inference and self- speculative decoding.arXiv preprint arXiv:2404.16710,

    arXiv

  4. [13]

    Multi-modal hallucination control by visual information grounding

    Alessandro Favero, Luca Zancato, Matthew Trager, Sid- dharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hallucination control by visual information grounding. In Proceedings of the IEEE/CVF Conference on Computer Vi- s...

    2024

  5. [14]

    Don’t hallucinate, abstain: Identifying llm knowledge gaps via multi-llm col- laboration.arXiv preprint arXiv:2402.00367, 2024

    Shangbin Feng, Weijia Shi, and et al. Don’t hallucinate, abstain: Identifying llm knowledge gaps via multi-llm col- laboration.arXiv preprint arXiv:2402.00367, 2024. 3

    2024 arXiv

  6. [15]

    Instructdiffusion: A gener- alist modeling interface for vision tasks

    Zigang Geng, Binxin Yang, Tiankai Hang, Chen Li, Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang, Houqiang Li, Han Hu, et al. Instructdiffusion: A gener- alist modeling interface for vision tasks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2024

  7. [16]

    Medsumm: A multimodal approach to summarizing code-mixed hindi- english clinical queries

    Akash Ghosh, Arkadeep Acharya, Prince Jha, Sriparna Saha, Aniket Gaudgaul, Rajdeep Majumdar, Aman Chadha, Raghav Jain, Setu Sinha, and Shivani Agarwal. Medsumm: A multimodal approach to summarizing code-mixed hindi- english clinical queries. InEuropean Conference on Infor- mat...

    2024

  8. [17]

    Exploring the frontier of vision- language models: A survey of current methodologies and future directions.arXiv preprint arXiv:2404.07214, 2024

    Akash Ghosh, Arkadeep Acharya, Sriparna Saha, Vinija Jain, and Aman Chadha. Exploring the frontier of vision- language models: A survey of current methodologies and future directions.arXiv preprint arXiv:2404.07214, 2024. 3

    2024

  9. [18]

    Sequence transduction with recurrent neural networks.arXiv preprint arXiv:1211.3711, 2012

    Alex Graves. Sequence transduction with recurrent neural networks.arXiv preprint arXiv:1211.3711, 2012. 3

    2012 arXiv

  10. [19]

    Detecting and preventing hallucinations in large vision language models

    Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. InProceedings of the AAAI Conference on Artificial Intel- ligence, pages 18135–18143, 2024. 3

    2024

  11. [20]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617,

  12. [21]

    Ciem: Contrastive instruction evaluation method for better instruction tuning.arXiv preprint arXiv:2309.02301,

    Hongyu Hu, Jiyuan Zhang, Minyi Zhao, and Zhenbang Sun. Ciem: Contrastive instruction evaluation method for better instruction tuning.arXiv preprint arXiv:2309.02301,

    arXiv

  13. [22]

    A survey on hallucination in large language models: Principles, tax- onomy, challenges, and open questions.arXiv preprint arXiv:2311.05232, 2023

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, tax- onomy, challenges, and open questions.arXiv preprint arXiv:2311.052...

    2023 arXiv

  14. [23]

    Diversity-aware meta visual prompting

    Qidong Huang, Xiaoyi Dong, Dongdong Chen, Weim- ing Zhang, Feifei Wang, Gang Hua, and Nenghai Yu. Diversity-aware meta visual prompting. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10878–10887, 2023. 3 12

    2023

  15. [24]

    Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Con- ghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation. InProceedings of the IEEE/CVF Confere...

    2024

  16. [25]

    Self-introspective decod- ing: Alleviating hallucinations for large vision-language models.arXiv preprint arXiv:2408.02032, 2024

    Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. Self-introspective decod- ing: Alleviating hallucinations for large vision-language models.arXiv preprint arXiv:2408.02032, 2024. 1

    2024 arXiv

  17. [26]

    Vcoder: Ver- satile vision encoders for multimodal large language mod- els

    Jitesh Jain, Jianwei Yang, and Humphrey Shi. Vcoder: Ver- satile vision encoders for multimodal large language mod- els. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 27992–28002,

  18. [27]

    Survey of hallucination in natural language generation.ACM Computing Surveys, 55(12):1–38, 2023

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation.ACM Computing Surveys, 55(12):1–38, 2023. 3

    2023

  19. [28]

    Hallucination augmented contrastive learn- ing for multimodal large language model

    Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learn- ing for multimodal large language model. InProceedings of the IEEE/CVF Conference on Computer Vision and Patter...

    2024

  20. [29]

    Chat-univi: Unified visual representation em- powers large language models with image and video under- standing

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation em- powers large language models with image and video under- standing. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1370...

    2024

  21. [30]

    Prov- able memorization capacity of msrvtt-qa

    Junghwan Kim, Michelle Kim, and Barzan Mozafari. Prov- able memorization capacity of msrvtt-qa. InInternational Conference on Learning Representations, 2023. 6

    2023

  22. [31]

    Instructive decoding: Instruction-tuned large lan- guage models are self-refiner from noisy instructions

    Taehyeon Kim, Joonkee Kim, Gihun Lee, and Se-Young Yun. Instructive decoding: Instruction-tuned large lan- guage models are self-refiner from noisy instructions. In The Twelfth International Conference on Learning Repre- sentations, 2024. 1, 3

    2024

  23. [32]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579–9589, 2024. 1

    2024

  24. [33]

    Factuality enhanced lan- guage models for open-ended text generation.NeurIPS, 35: 34586–34599, 2022

    Nayeon Lee, Wei Ping, and et al. Factuality enhanced lan- guage models for open-ended text generation.NeurIPS, 35: 34586–34599, 2022. 3

    2022

  25. [34]

    Mitigating object hallucinations in large vision-language models through vi- sual contrastive decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shi- jian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through vi- sual contrastive decoding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2024

  26. [35]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 3

    2024 arXiv

  27. [36]

    Contrastive decoding: Open-ended text gen- eration as optimization.arXiv preprint arXiv:2210.15097,

    Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text gen- eration as optimization.arXiv preprint arXiv:2210.15097,

    arXiv

  28. [37]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. InProceedings of the 2023 Con- ference on Empirical Methods in Natural Language Pro- cessing, pages 292–305, Singapore, 2023. Association for...

    2023

  29. [38]

    Mini-gemini: Mining the potential of multi-modality vision language models.arXiv preprint arXiv:2403.18814,

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models.arXiv preprint arXiv:2403.18814,

    arXiv

  30. [39]

    Mon- key: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Mon- key: Image resolution and text label are important things for large multi-modal models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  31. [40]

    Video-llava: Learning united visual rep- resentation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual rep- resentation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023. 2, 6

    2023 arXiv

  32. [41]

    Vila: On pre-training for visual language models, 2023

    Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023. 6

    2023

  33. [42]

    Aligning large multi-modal model with robust instruction tuning.arXiv preprint arXiv:2306.14565, 2023

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning.arXiv preprint arXiv:2306.14565, 2023. 3

    2023 arXiv

  34. [43]

    Mitigating hallucination in large multi-modal models via robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Ya- coob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. InThe Twelfth International Conference on Learning Representa- tions, 2023. 3

    2023

  35. [44]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 1, 2, 6

    2024

  36. [45]

    Visual instruction tuning.Advances in neural infor- mation processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural infor- mation processing systems, 36, 2024. 1, 6

    2024

  37. [46]

    A survey on hallucination in large vision-language models.arXiv preprint arXiv:2402.00253, 2024

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models.arXiv preprint arXiv:2402.00253, 2024. 3

    2024 arXiv

  38. [47]

    Phd: A prompted visual hallucination evaluation dataset.arXiv preprint arXiv:2403.11116, 2024

    Jiazhen Liu, Yuhan Fu, Ruobing Xie, Runquan Xie, Xingwu Sun, Fengzong Lian, Zhanhui Kang, and Xirong Li. Phd: A prompted visual hallucination evaluation dataset.arXiv preprint arXiv:2403.11116, 2024. 3 13

    2024 arXiv

  39. [48]

    Paying more atten- tion to image: A training-free method for alleviating hallu- cination in lvlms.arXiv preprint arXiv:2407.21771, 2024

    Shi Liu, Kecheng Zheng, and Wei Chen. Paying more atten- tion to image: A training-free method for alleviating hallu- cination in lvlms.arXiv preprint arXiv:2407.21771, 2024. 1, 3

    2024 arXiv

  40. [49]

    Seeing clearly, answering incorrectly: A multimodal robustness benchmark for evaluating mllms on leading questions.arXiv preprint arXiv:2406.10638,

    Yexin Liu, Zhengyang Liang, Yueze Wang, Muyang He, Jian Li, and Bo Zhao. Seeing clearly, answering incorrectly: A multimodal robustness benchmark for evaluating mllms on leading questions.arXiv preprint arXiv:2406.10638,

    arXiv

  41. [50]

    Mmbench: Is your multi- modal model an all-around player? InEuropean Confer- ence on Computer Vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi- modal model an all-around player? InEuropean Confer- ence on Computer Vision, pages 216–233. Springer, 2025. 6

    2025

  42. [51]

    Learn to explain: Multimodal rea- soning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal rea- soning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 250...

    2022

  43. [52]

    Vista-llama: Reducing hallucination in video language models via equal distance to visual to- kens

    Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. Vista-llama: Reducing hallucination in video language models via equal distance to visual to- kens. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13151–13...

    2024

  44. [53]

    Vista-llama: Reducing hallucination in video language models via equal distance to visual tokens

    Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. Vista-llama: Reducing hallucination in video language models via equal distance to visual tokens. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13151–13160, 2024. 1, 3

    2024

  45. [54]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 6

    2023 arXiv

  46. [55]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. InProceedings of the 62nd Annual Meeting of the Associa- tion for Computational Linguistics (ACL 2024), 2024. 8

    2024

  47. [56]

    Selfcheckgpt: Zero-resource black-box hallucination detec- tion for generative large language models.arXiv preprint arXiv:2303.08896, 2023

    Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallucination detec- tion for generative large language models.arXiv preprint arXiv:2303.08896, 2023. 3

    2023 arXiv

  48. [57]

    Mm1: Meth- ods, analysis & insights from multimodal llm pre-training

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mm1: Meth- ods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611, 2024. 3

    2024 arXiv

  49. [58]

    Training lan- guage models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training lan- guage models to follow instructions with human feedback. Advances in neural information processing systems, 35:...

    2022

  50. [59]

    Train short, test long: Attention with linear biases enables input length extrapolation.arXiv preprint arXiv:2108.12409, 2021

    Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation.arXiv preprint arXiv:2108.12409, 2021. 2, 3, 4

    2021 arXiv

  51. [60]

    Exploring the limits of transfer learning with a unified text-to-text transformer.The Journal of Ma- chine Learning Research, 21(1):5485–5551, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.The Journal of Ma- chine Learning Research, 21(1):5485–5551, 2020. 2, 3

    2020

  52. [61]

    Object hallucination in image captioning

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035–4045, 2018. 6

    2018

  53. [62]

    Enhancing adverse drug event detection with multimodal dataset: Corpus creation and model development.arXiv preprint arXiv:2405.15766,

    Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Aman Chadha, and Samrat Mondal. Enhancing adverse drug event detection with multimodal dataset: Corpus creation and model development.arXiv preprint arXiv:2405.15766,

    arXiv

  54. [63]

    Mitigating object hallucination via data augmented contrastive tuning

    Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan ¨O Arık, and Tomas Pfister. Mitigating object hallucination via data augmented contrastive tuning. arXiv preprint arXiv:2405.18654, 2024. 1

    2024 arXiv

  55. [64]

    Retrieval augmentation reduces hallu- cination in conversation.arXiv preprint arXiv:2104.07567,

    Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallu- cination in conversation.arXiv preprint arXiv:2104.07567,

    arXiv

  56. [65]

    Roformer: Enhanced trans- former with rotary position embedding.arXiv preprint arXiv:2104.09864, 2021

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced trans- former with rotary position embedding.arXiv preprint arXiv:2104.09864, 2021. 2, 4, 6

    2021 arXiv

  57. [66]

    Intervening anchor token: Decod- ing strategy in alleviating hallucinations for MLLMs

    Feilong Tang, Zile Huang, Chengzhi Liu, Qiang Sun, Harry Yang, and Ser-Nam Lim. Intervening anchor token: Decod- ing strategy in alleviating hallucinations for MLLMs. In The Thirteenth International Conference on Learning Rep- resentations, 2025. 3

    2025

  58. [67]

    Cambrian- 1: A fully open, vision-centric exploration of multimodal llms.arXiv preprint arXiv:2406.16860, 2024

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms.arXiv preprint arXiv:2406.16860, 2024. 3

    2024 arXiv

  59. [69]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 3

    2023 arXiv

  60. [70]

    Attention is all you need.NeurIPS, 30, 2017

    Ashish Vaswani, Noam Shazeer, and et al. Attention is all you need.NeurIPS, 30, 2017. 3

    2017

  61. [71]

    Evaluation and analysis of hal- lucination in large vision-language models.arXiv preprint arXiv:2308.15126, 2023

    Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu, Qinghao Ye, Ming Yan, Ji Zhang, Jihua Zhu, et al. Evaluation and analysis of hal- lucination in large vision-language models.arXiv preprint arXiv:2308.15126, 2023. 3 14

    2023 arXiv

  62. [72]

    Mitigating hallucinations in large vision-language models with instruction contrastive decoding.arXiv preprint arXiv:2403.18715, 2024

    Xintong Wang, Jingheng Pan, Liang Ding, and Chris Bie- mann. Mitigating hallucinations in large vision-language models with instruction contrastive decoding.arXiv preprint arXiv:2403.18715, 2024. 1, 3, 8

    2024 arXiv

  63. [73]

    Self-instruct: Aligning language models with self- generated instructions.arXiv preprint arXiv:2212.10560,

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Ha- jishirzi. Self-instruct: Aligning language models with self- generated instructions.arXiv preprint arXiv:2212.10560,

    arXiv

  64. [74]

    Logical closed loop: Uncovering object hallucinations in large vision-language models.arXiv preprint arXiv:2402.11622, 2024

    Junfei Wu, Qiang Liu, and et al. Logical closed loop: Uncovering object hallucinations in large vision-language models.arXiv preprint arXiv:2402.11622, 2024. 3

    2024 arXiv

  65. [75]

    Detecting and mitigating hallucination in large vi- sion language models via fine-grained ai feedback.arXiv preprint arXiv:2404.14233, 2024

    Wenyi Xiao, Ziwei Huang, Leilei Gan, Wanggui He, Haoyuan Li, Zhelun Yu, Hao Jiang, Fei Wu, and Linchao Zhu. Detecting and mitigating hallucination in large vi- sion language models via fine-grained ai feedback.arXiv preprint arXiv:2404.14233, 2024. 1

    2024 arXiv

  66. [76]

    Miti- gating object hallucination via concentric causal attention

    Yun Xing, Yiheng Li, Ivan Laptev, and Shijian Lu. Miti- gating object hallucination via concentric causal attention. arXiv preprint arXiv:2410.15926, 2024. 1, 3

    2024 arXiv

  67. [77]

    Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xi- angnan He, and Yueting Zhuang

    D. Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xi- angnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. Proceedings of the 25th ACM international conference on Multimedia, 2017. 6

    2017

  68. [78]

    Mmrc: A large- scale benchmark for understanding multimodal large lan- guage model in real-world conversation.arXiv preprint arXiv:2502.11903, 2025

    Haochen Xue, Feilong Tang, Ming Hu, Yexin Liu, Qi- dong Huang, Yulong Li, Chengzhi Liu, Zhongxing Xu, Chong Zhang, Chun-Mei Feng, et al. Mmrc: A large- scale benchmark for understanding multimodal large lan- guage model in real-world conversation.arXiv preprint arXiv:2502.1190...

    2025 arXiv

  69. [79]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pa...

    2024

  70. [80]

    Stablemask: Refining causal masking in decoder-only transformer.arXiv preprint arXiv:2402.04779, 2024

    Qingyu Yin, Xuzheng He, Xiang Zhuang, Yu Zhao, Jianhua Yao, Xiaoyu Shen, and Qiang Zhang. Stablemask: Refining causal masking in decoder-only transformer.arXiv preprint arXiv:2402.04779, 2024. 2, 3, 4

    2024 arXiv

  71. [81]

    Woodpecker: Hallucination correc- tion for multimodal large language models.arXiv preprint arXiv:2310.16045, 2023

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correc- tion for multimodal large language models.arXiv preprint arXiv:2310.16045, 2023. 3

    arXiv

  72. [82]

    Ferret: Refer and ground anything anywhere at any granularity.arXiv preprint arXiv:2310.07704, 2023

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity.arXiv preprint arXiv:2310.07704, 2023. 3

    2023 arXiv

  73. [83]

    Yi: Open foundation models by 01

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jian- qun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai.arXiv preprint arXiv:2403.04652, 2024. 3

    2024 arXiv

  74. [84]

    Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data

    Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wen- tao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2024

  75. [85]

    Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. InProceedings of the IEEE/CVF Conference on Com...

    2024

  76. [86]

    Mm-vet: Evaluating large multimodal models for inte- grated capabilities.arXiv preprint arXiv:2308.02490, 2023

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for inte- grated capabilities.arXiv preprint arXiv:2308.02490, 2023. 6

    2023 arXiv

  77. [87]

    Less is more: Mit- igating multimodal hallucination from an eos decision per- spective.arXiv preprint arXiv:2402.14545, 2024

    Zihao Yue, Liang Zhang, and Qin Jin. Less is more: Mit- igating multimodal hallucination from an eos decision per- spective.arXiv preprint arXiv:2402.14545, 2024. 3

    2024 arXiv

  78. [88]

    Halle-switch: Rethinking and controlling object existence hallucinations in large vi- sion language models for detailed caption.arXiv preprint arXiv:2310.01779, 2023

    Bohan Zhai, Shijia Yang, Xiangchen Zhao, Chenfeng Xu, Sheng Shen, Dongdi Zhao, Kurt Keutzer, Manling Li, Tan Yan, and Xiangjun Fan. Halle-switch: Rethinking and controlling object existence hallucinations in large vi- sion language models for detailed caption.arXiv preprint ar...

    arXiv

  79. [89]

    Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, et al. Mm1. 5: Methods, anal- ysis & insights from multimodal llm fine-tuning.arXiv preprint arXiv:2409.20566, 2024. 3

    2024 arXiv

  80. [90]

    How language model hallucinations can snowball.arXiv preprint arXiv:2305.13534, 2023

    Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. How language model hallucinations can snowball.arXiv preprint arXiv:2305.13534, 2023. 3

    2023 arXiv

  81. [91]

    Internlm- xcomposer: A vision-language large model for advanced text-image comprehension and composition.arXiv preprint arXiv:2309.15112, 2023

    Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. Internlm- xcomposer: A vision-language large model for advanced text-image comprehension and composition.arXiv preprint arXiv:2309.1...

    2023 arXiv

  82. [92]

    Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and out- put.arXiv preprint arXiv:2407.03320, 2024

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang, Wei Li, Jing- wen Li, Wenhai Wang, Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen,...

    2024 arXiv

  83. [93]

    Siren’s song in the ai ocean: a survey on hallucination in large language models.arXiv preprint arXiv:2309.01219, 2023

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yu- long Chen, et al. Siren’s song in the ai ocean: a survey on hallucination in large language models.arXiv preprint arXiv:2309.01219, 2023. 3

    2023 arXiv

  84. [94]

    Recognize anything: A strong image tagging model

    Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: A strong image tagging model. InProceedings of the IEEE/CVF 15 Conference on Computer Vision and Pattern Recognition, pages 17...

    2024

  85. [95]

    Beyond hallucinations: En- hancing lvlms through hallucination-aware direct prefer- ence optimization.arXiv preprint arXiv:2311.16839, 2023

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Ji- aqi Wang, and Conghui He. Beyond hallucinations: En- hancing lvlms through hallucination-aware direct prefer- ence optimization.arXiv preprint arXiv:2311.16839, 2023. 3

    2023 arXiv

  86. [96]

    Investigating and mitigating the multimodal halluci- nation snowballing in large vision-language models.arXiv preprint arXiv:2407.00569, 2024

    Weihong Zhong, Xiaocheng Feng, Liang Zhao, Qiming Li, Lei Huang, Yuxuan Gu, Weitao Ma, Yuan Xu, and Bing Qin. Investigating and mitigating the multimodal halluci- nation snowballing in large vision-language models.arXiv preprint arXiv:2407.00569, 2024. 3

    2024 arXiv

  87. [97]

    Luowei Zhou, Chenliang Xu, and Jason J. Corso. Towards automatic learning of procedures from web instructional videos. InAAAI, 2017. 6

    2017

  88. [98]

    Analyzing and mitigating object hallucination in large vision-language models.arXiv preprint arXiv:2310.00754,

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models.arXiv preprint arXiv:2310.00754,

    arXiv

  89. [99]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023. 1

    2023 arXiv

  90. [100]

    Ibd: Alleviating hallucinations in large vision- language models via image-biased decoding.arXiv preprint arXiv:2402.18476, 2024

    Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. Ibd: Alleviating hallucinations in large vision- language models via image-biased decoding.arXiv preprint arXiv:2402.18476, 2024. 3 16

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.