Pith. sign in

REVIEW 4 major objections 5 minor 92 references

MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that RoPE's long-term decay biases vision-language models toward bottom-right image tokens, and that renumbering image tokens by Manhattan coordinate sums reduces hallucination across benchmarks.

desk verdict Solid empirical gains, but the '2D Manhattan decay' mechanism largely reduces to a shorter positional range; worth a careful revision, not a rejection. read the letter →

arxiv 2507.09184 v2 pith:ZPBMI3MP submitted 2025-07-12 cs.CV

Qiyan Zhao , Xiaofeng Zhang , Yiheng Li , Yun Xing , Xiaosong Yuan , Feilong Tang , Sinan Fan , Xuhang Chen
show 2 more authors
Xuyao Zhang Dahan Wang
This is my paper
classification cs.CV
keywords largevision-languagemodelshallucinationrotarypositionencodinglong-termdecayimagealignmentbiasManhattandistancecausalattentionmultimodal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a standard positional-encoding property in large vision-language models, the long-term decay of Rotary Position Encoding (RoPE), systematically biases instruction tokens toward image tokens from the bottom-right of the image, because in the flattened one-dimensional sequence those tokens sit closest to the instruction text. It names this phenomenon 'image alignment bias' and argues that it starves most image regions of attention, producing hallucinations. To fix it, the paper proposes Manhattan Causal Attention (MCA), which renumbers image tokens by the sum of their two-dimensional coordinates rather than by raster-scan order. With no new data or model modules, only a renumbering of image-token positions and the associated causal mask, MCA-LLaVA reports consistent hallucination reductions across POPE, CHAIR, and MME, plus gains on general vision-language benchmarks. If the claim is right, part of multimodal hallucination is a position-numbering artifact that can be repaired inside RoPE rather than with decoding tricks or extra training data.

What carries the argument

The central object is Manhattan Causal Attention (MCA), a positional scheme for RoPE. Each image token is assigned new coordinates measured from the nearest of the four image corners after mirroring the image into quadrants, and the new positional index that actually enters RoPE's rotation matrices is the sum $\mu(m)=x_m+y_m$. This scalar reindexing carries the argument: it replaces raster-scan order with a Manhattan-distance order, shortens the range of image position indices from $V$ to $\sqrt{V}-1$, and is paired with a Manhattan causal mask so that image tokens attend to spatial neighbors rather than to every earlier token in the flattened sequence. The relative positional distance becomes $(x_j-x_i)+(y_j-y_i)$, identified with $\mu(j)-\mu(i)$, which keeps the formal RoPE computation unchanged while making the decay originate from all four corners.

What would settle it

Compare MCA against a control that uses the same Manhattan causal mask and the same set of $\mu$ values but assigns those values to image tokens at random: if POPE F1 and CHAIR_S stay close to MCA's numbers, then the shorter index range, not 2D spatial decay, explains the gains. Alternatively, replace the scalar sum with two independent rotation frequencies for the $x$ and $y$ coordinates; a genuine 2D spatial-decay mechanism should preserve or improve MCA's results, while the scalar-sum version should lose on layout-sensitive tasks if its stated mechanism is correct.

Watch

Extended reading notes

Core claim

Under RoPE's long-term decay, attention between an instruction token and an image key token is controlled by the difference of their scalar position indices, so the instruction attends most strongly to whatever image tokens lie immediately before it in the raster-scan sequence, namely the bottom-right patch region. The paper demonstrates this through image-to-instruction information flow, where only bottom-right tokens show dense flow. The discovery is that this is not a learned attention pattern but a systemic bias written into the position encoder, and that replacing the raster-scan index with a Manhattan position index $\mu(m)=x_m+y_m$, computed from mirrored corner origins, turns the one-dimensional unidirectional decay into a multi-directional spatial decay. In MCA, the relative positional difference becomes $(x_j-x_i)+(y_j-y_i)$, so all four corners of the image act as decay origins and image tokens of every region can be close to the instruction. The paper reports that this reindexing, together with a Manhattan causal mask that preserves 2D locality, raises POPE F1 from 79.3 to 86.0, lowers CHAIR_S from 47.0 to 38.0, and lifts the MME hallucination-subset total from 545 to 650.

Load-bearing premise

The load-bearing premise is that numbering image tokens by the sum of their row and column coordinates truly creates a two-dimensional spatial decay in RoPE, rather than merely shrinking the positional range that the one-dimensional decay operates over.

Editorial extensions

If this is right

  • Swapping only the positional indices of image tokens, with no new data or modules, lifts POPE F1 from 79.3 to 86.0 and accuracy from 79.8 to 86.5 on LLaVA-1.5-7B.
  • Sentence-level caption hallucination drops from 47.0 to 38.0 CHAIR_S and instance-level hallucination from 13.8 to 10.9 under greedy decoding.
  • The fix transfers beyond the base model, improving POPE and CHAIR results for LLaVA-1.5-13B and InternVL-7B, so it is not tied to a single architecture.
  • General benchmarks improve as well, including MME total 650 versus 545, MMStar 36.5 versus 30.0, and SEED_V 41.3 versus 37.3, indicating better overall image perception rather than overfitting to hallucination tests.
  • The number of distinct position indices available to image tokens falls from $V$ to $\sqrt{V}-1$, which shortens the positional distance between image and instruction tokens before any attention is computed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the 2D-decay explanation would compare MCA with a random reindexing that keeps the same 23-index range; if hallucinations drop just as much, the mechanism is the shorter range, not spatial decay.
  • Because every image token on the same anti-diagonal gets the same $\mu$ value, MCA cannot represent the mutual order of those tokens; a two-frequency RoPE with separate rotations for the $x$ and $y$ coordinates is the natural extension and may preserve more layout information than the scalar sum.
  • The paper leaves the instruction tokens' positions after reindexing unspecified, and the reported distance reduction depends on those positions; testing different instruction-token placements would clarify how much of the gain comes from image-image spacing versus image-instruction spacing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MCA-LLaVA, a modification to the position encoding and causal masking of image tokens in LVLMs, intended to mitigate hallucination by correcting what the authors call "image alignment bias" caused by RoPE's one-dimensional long-term decay. The method replaces raster-scan position indices with μ(m)=x_m+y_m, uses a mirrored coordinate origin at the four corners, and introduces a "Manhattan causal mask." The authors report consistent improvements over LLaVA-1.5-7B on POPE, CHAIR, MME, and several general VQA benchmarks, and also apply the method to LLaVA-1.5-13B and InternVL-7B. The central theoretical claim is that this implements a two-dimensional, multi-directional spatial decay based on Manhattan distance.

Significance. If the results hold, MCA-LLaVA is a simple and cheap intervention — a reindexing plus a masking change — that reduces hallucination relative to strong baselines, with code provided and experiments across multiple models and benchmarks. The empirical gains are consistent and the benchmark coverage is reasonable. The significance is limited, however, by the gap between the paper's theoretical narrative and the actual implementation: the position encoding collapses 2D positions onto a scalar, so the claimed 2D multi-directional decay is not realized inside RoPE. The method may still be practically useful, but its contribution relative to CCA-LLaVA's range-compression mechanism needs to be clarified rather than asserted.

major comments (4)
  1. [§3.1, Eq. (4)] The quantity defined as the Manhattan distance is not the Manhattan distance. Eq. (4) states D_Manhattan = (x_j − x_i) + (y_j − y_i), which is the signed difference of coordinate sums, not the L1 distance |x_j − x_i| + |y_j − y_i|. The signed difference is negative for half of all token pairs and collapses every anti-diagonal to a single scalar. No order constraints are given that would make the signed difference equal to the true Manhattan distance. This is load-bearing because the abstract and Section 2.4 justify the method by claiming a 2D multi-directional decay.
  2. [§3.2–§3.3, Eqs. (7)–(8)] Because the rotary matrices in Eq. (8) depend only on the scalar μ(m)=x_m+y_m, all image tokens on the same anti-diagonal receive identical positional encodings. The RoPE computation therefore contains no genuinely two-dimensional positional information; the attention score depends only on μ(j)−μ(i). Consequently, the reported gains are equally compatible with a shorter scalar position range (from V=576 down to 23 indices, as Table 5 records), which is the mechanism already proposed by CCA-LLaVA. The paper needs to either implement a real 2D rotation scheme or explicitly abandon the claim that it does.
  3. [§3.2, §4.5] The paper does not specify how instruction-token positions are assigned after image tokens are reindexed to 23 Manhattan positions. The claim that the reduced index count "reduces the overall distance between image and instruction tokens" is incomplete without this assignment. If instruction tokens keep their original indices (starting at 576 in LLaVA-1.5), then the distance to the new image indices 0–22 is actually much larger than in the raster-scan case; the distance reduction only holds if instruction tokens are shifted to immediately follow the reindexed image tokens. The exact assignment should be stated and its role in the improvement isolated.
  4. [§4.5, Table 5] The ablation compares MCA (23 indices) with CCA (12), Reverse MCA (23), and a Variant of MCA (12), but it lacks a control that uses the same reduced index range (23) with a purely one-dimensional compressed assignment (e.g., a shortened raster order). Without such a control, the experiments cannot tell whether the gains come from the Manhattan coordinate design and causal mask or simply from the shorter positional range, which Section 3.2 itself credits to prior work [66]. This control is necessary to support the paper's central mechanistic claim.
minor comments (5)
  1. [Abstract, §2.4] The phrase "two-dimensional, multi-directional spatial decay" is not supported by the implementation; please either change the description to match the scalar-index mechanism or implement a genuinely 2D rotary encoding.
  2. [§4, all tables] No error bars, confidence intervals, or multiple-seed results are reported. Several improvements in Table 2 are small (e.g., +0.1 on VQA-v2), so a single run does not establish robustness.
  3. [§2.4, Fig. 2] The information-flow aggregation procedure is not described in enough detail (e.g., which layers/heads are pooled, how the 3K POPE adversarial subset is used), making the key visualization difficult to reproduce or verify.
  4. [Table 1] The table lists "Greedy Search" among the compared methods; since the baseline LLaVA-1.5-7B greedy result is also the reference for the method, please mark it explicitly as the baseline row to avoid confusion.
  5. [References] Several references are incomplete or contain typographical errors (e.g., [61] Qwen2-VL and [77] Video-LLaMA); please check the bibliography against the source versions.

Circularity Check

2 steps flagged · score 4.0 of 10

MCA's '2D Manhattan decay' collapses to a scalar reindexing by Eq. 7-8, and the range-compression rationale cites co-authored CCA-LLaVA; benchmark gains remain independent, so circularity is partial.

  1. self definitional [Section 3.2, Eqs. 7-8]
    "Therefore, we use the sum of token coordinates as new position indexes, termed Manhattan positions assignment μ, to adapt the calculation of the relative positional distance of Manhattan. ... By Manhattan position assignment and constant transformation (Eq. 7), Attn′ is formally unified with Attn."

    The rotary matrix in Eq. 8 is parameterized by the single scalar (x_j−x_i)+(y_j−y_i), which Eq. 7 defines as μ(j)−μ(i). Because R^d_{θ,m} depends only on the scalar index m, all image tokens on an anti-diagonal (constant x_m+y_m) receive identical positional encodings, and Eq. 8 is exactly the original RoPE attention formula with reindexed positions, as the paper itself says ('formally unified with Attn'). The claimed 'two-dimensional, multi-directional spatial decay' is therefore not a consequence of a 2D distance; it is an equivalent restatement of the scalar μ-reindexing. The 2D content is eliminated by construction before the attention computation.

  2. self citation load bearing [Section 3.2, paragraph after Eq. 7]
    "Compared to raster scanning, the number of Manhattan Position indexes decreases from V to √v−1, which reduces the overall distance between image and instruction tokens. This is more favorable for information interaction[66]."

    Reference [66] is CCA-LLaVA, authored by two co-authors of the present paper (Yun Xing and Yiheng Li). The sentence uses this overlapping-author citation as the sole justification for why MCA's index-range compression is beneficial, and range compression is a central component of the method (Table 5 reports the index count dropping from 526 to 23). The favorable-distance premise is therefore imported from the authors' own prior work rather than derived or independently verified here. The POPE/CHAIR/MME evaluations are independent evidence, so this is partial, not total, circularity.

full rationale

MCA-LLaVA is primarily an empirical method paper: it reindexes image tokens with μ(m)=x_m+y_m, applies a Manhattan-style causal mask, and measures effects on POPE, CHAIR, MME, GQA, and other benchmarks. Those numbers are genuine held-out evaluations, so the headline hallucination reductions do not reduce to a fitted parameter or to a self-citation. The circularity is partial and conceptual. First, the method's claimed '2D Manhattan distance' is defined as a signed scalar index difference μ(j)−μ(i); since RoPE's rotation matrix in Eq. 8 only sees that scalar, tokens with equal x+y are encoded identically, and the 'multi-directional 2D decay' is a relabeling of a scalar reindexing rather than a computed 2D quantity. Second, the explanation for why the reduced index range helps ('more favorable for information interaction') cites CCA-LLaVA [66], which shares two co-authors with this paper, so a load-bearing mechanism is justified by the authors' own prior work. These issues do not eliminate the independent empirical contribution, but they make the derivation of the proposed mechanism partially self-referential.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The central claim rests on the RoPE decay property, the raster-scan flattening convention, and the unstated assumption that a scalar coordinate-sum index preserves enough 2D spatial information. The design also has one hand-chosen configuration (four-corner origins) selected by ablation on the evaluation benchmarks, and one unspecified implementation detail (instruction-token positions) that is needed for the distance-reduction argument.

free parameters (3)
  • Coordinate origin configuration = four corners (final); alternatives tested: center+four corners, center only
    Section 4.5, Table 5: the final configuration was selected by comparing variants on POPE and CHAIR, the same benchmarks highlighted as headline results. This is a hand-chosen design parameter, not derived from theory.
  • Number of distinct image position indices = 23 for V=576 (sqrt(V)-1)
    Direct consequence of using mu=x+y with mirrored corner coordinates (Section 3.2). This compression is load-bearing for the distance-reduction claim.
  • Instruction token position offset = not reported
    The distance reduction claim in Section 3.2 assumes instruction tokens are placed immediately after the reindexed image tokens, but the paper never specifies how instruction positions are computed.
assumptions (5)
  • standard math RoPE attention decays as the relative position index grows.
    Invoked in Eq.2 and Section 2.4, based on RoFormer [50].
  • domain assumption Image tokens are flattened in raster-scan order and their positions are modeled by that 1D order.
    This is the LLaVA input construction assumed throughout Section 2.4.
  • ad hoc to paper A scalar index mu=x+y can replace true 2D positions without losing necessary spatial information.
    Eq.7 and Section 3.2 assert formal alignment with Eq.3, but provide no argument that many tokens sharing the same mu preserves spatial locality.
  • ad hoc to paper Instruction tokens receive positions immediately after the reduced image index range.
    Needed for the claim that distances shrink from V to sqrt(V)-1 (Section 3.2); never stated explicitly.
  • domain assumption Information-flow visualizations in Figure 2 measure image-instruction alignment.
    The aggregation method is not defined; the figures are used as evidence for image alignment bias.
invented entities (1)
  • Image alignment bias
    purpose: Names the hypothesized uneven instruction attention to image tokens caused by RoPE long-term decay.
    Supported only by internal information-flow visualizations (Figure 2) on the same model family and benchmark later used for evaluation. No independent falsifiable prediction is offered outside the method's benchmark performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Models." pith.science (2026). https://pith.science/paper/ZPBMI3MP

@misc{pith2026250709184,
  author       = {Pith},
  title        = {Pith review of: MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZPBMI3MP}},
  note         = {Machine review of arXiv:2507.09184}
}
read the original abstract

Hallucinations pose a significant challenge in Large Vision Language Models (LVLMs), with misalignment between multimodal features identified as a key contributing factor. This paper reveals the negative impact of the long-term decay in Rotary Position Encoding (RoPE), used for positional modeling in LVLMs, on multimodal alignment. Concretely, under long-term decay, instruction tokens exhibit uneven perception of image tokens located at different positions within the two-dimensional space: prioritizing image tokens from the bottom-right region since in the one-dimensional sequence, these tokens are positionally closer to the instruction tokens. This biased perception leads to insufficient image-instruction interaction and suboptimal multimodal alignment. We refer to this phenomenon as image alignment bias. To enhance instruction's perception of image tokens at different spatial locations, we propose MCA-LLaVA, based on Manhattan distance, which extends the long-term decay to a two-dimensional, multi-directional spatial decay. MCA-LLaVA integrates the one-dimensional sequence order and two-dimensional spatial position of image tokens for positional modeling, mitigating hallucinations by alleviating image alignment bias. Experimental results of MCA-LLaVA across various hallucination and general benchmarks demonstrate its effectiveness and generality. The code can be accessed in https://github.com/ErikZ719/MCA-LLaVA.

Figures

Figures reproduced from arXiv: 2507.09184 by the authors.

Figure 1
Figure 1. Schematic of long-term decay in different positional [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a), (b), and (c) show the information flow of image-to-instruction in LLaVA1.5[ [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of image token position coordinate reassignment. The total number of image tokens is denoted as [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The default causal masks are 𝑉 ×𝑉 . We show the causal masks when the number of image tokens 𝑉 is 36. By default causal modeling in (a), image tokens focus on all visual tokens in between; by CCA in (b), the central image tokens focus on peripheral tokens; and by MCA i…
Figure 5
Figure 5. Figure 5: Qualitative results of CCA-LLaVA with MCA-LLaVA. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Different positional indices and corresponding information flow patterns. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Attention Maps Visualization of MCA. Attention Maps Visualization of MCA We further analyze heatmaps over the image for object tokens to investigate the model’s differ￾ences in visual perception. As shown in the [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

92 extracted references · 26 canonical work pages

  1. [66]

    Yun Xing, Yiheng Li, Ivan Laptev, and Shijian Lu. 2024. Mitigating Object Hal- lucination via Concentric Causal Attention. arXiv preprint arXiv:2410.15926 (2024)

    arXiv 2024

  2. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  3. [2]

    Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, QianYing Wang, Guang Dai, Ping Chen, and Shijian Lu. 2024. AGLA: Mitigating Object Halluci- nations in Large Vision-Language Models with Assembly of Global and Local Attention. arXiv preprint arXiv:2406.12718 (2024)

    arXiv 2024

  4. [3]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308.12966 (2023)

    arXiv 2023

  5. [4]

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2024. Hallucination of Multimodal Large Language Models: A Survey. ArXiv abs/2404.18930 (2024)

    arXiv 2024

  6. [5]

    Beitao Chen, Xinyu Lyu, Lianli Gao, Jingkuan Song, and Hengtao Shen. 2024. Al- leviating Hallucinations in Large Vision-Language Models through Hallucination- Induced Optimization. ArXiv abs/2405.15356 (2024)

    arXiv 2024

  7. [6]

    Cong Chen, Mingyu Liu, Chenchen Jing, Yizhou Zhou, Fengyun Rao, Hao Chen, Bo Zhang, and Chunhua Shen. 2025. PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training

    2025

  8. [7]

    Lin Chen, Jinsong Li, Xiao wen Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. 2024. Are We on the Right Way for Evaluating Large Vision-Language Models? ArXiv abs/2403.20330 (2024)

    arXiv 2024

Show all 92 references
  1. [8]

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024. An image is worth 1/2 tokens after layer 2: Plug-and- play inference acceleration for large vision-language models. 18th European Conference on Computer Vision ECCV 2024 (2024)

    2024

  2. [9]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al . 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Compu...

    2024

  3. [10]

    Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. 2024. HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding. arXiv preprint arXiv:2403.00425 (2024)

    2024 arXiv

  4. [11]

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 20...

    2023

  5. [12]

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2023. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03883 (2023)

    2023 arXiv

  6. [13]

    Chenhang Cui, An Zhang, Yiyang Zhou, Zhaorun Chen, Gelei Deng, Huaxiu Yao, and Tat-Seng Chua. 2024. Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment. ArXiv abs/2410.14148 (2024)

    2024 arXiv

  7. [14]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805 (2018)

    2018 arXiv

  8. [15]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xi- aohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recogn...

    2021

  9. [16]

    Yuhan Fu, Ruobing Xie, Xingwu Sun, Zhanhui Kang, and Xirong Li. 2024. Miti- gating Hallucination in Multimodal Large Language Model via Hallucination- targeted Direct Preference Optimization. ArXiv abs/2411.10436 (2024)

    2024 arXiv

  10. [17]

    Xuan Gong, Tianshi Ming, Xinpeng Wang, and Zhihua Wei. 2024. DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination. In Conference on Empirical Methods in Natural Language Processing

    2024

  11. [18]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh

  12. [19]

    Anisha Gunjal, Jihan Yin, and Erhan Bas. 2024. Detecting and preventing halluci- nations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 18135–18143

    2024

  13. [20]

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition . 3608–3617

    2018

  14. [21]

    Xin He, Longhui Wei, Lingxi Xie, and Qi Tian. 2024. Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models. ArXiv abs/2401.03105 (2024)

    2024 arXiv

  15. [22]

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hal- lucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF C...

    2024

  16. [23]

    Zhiheng Huang, Davis Liang, Peng Xu, and Bing Xiang. 2020. Improve Trans- former Models with Better Relative Position Embeddings. In Findings of the Association for Computational Linguistics: EMNLP 2020 . 3327–3335

    2020

  17. [24]

    Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real- world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 6700–6709

    2019

  18. [25]

    Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. 2025. Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models. In The Thirteenth International Conference on Learning Representations

    2025

  19. [26]

    Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Mingshi Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. 2023. Hallucination Aug- mented Contrastive Learning for Multimodal Large Language Model. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2023

  20. [27]

    Junho Kim, Hyunjun Kim, Yeonju Kim, and Yonghyun Ro. 2024. CODE: Contrast- ing Self-generated Description to Combat Hallucination in Large Multi-modal Models. ArXiv abs/2406.01920 (2024)

    2024 arXiv

  21. [28]

    Sihyeon Kim, Boryeong Cho, Sangmin Bae, Sumyeong Ahn, and SeYoung Yun

  22. [29]

    Sicong Leng, Yun Xing, Zesen Cheng, Yang Zhou, Hang Zhang, Xin Li, Deli Zhao, Shijian Lu, Chunyan Miao, and Li Bing. 2024. The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio. ArXiv abs/2410.12787 (2024)

    2024 arXiv

  23. [30]

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  24. [31]

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023. SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension. ArXiv abs/2307.16125 (2023)

    2023 arXiv

  25. [32]

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2024. Llava-med: Train- ing a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems 36 (2024)

    2024

  26. [33]

    Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wat- tenberg. 2024. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems 36 (2024)

    2024

  27. [35]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen

  28. [36]

    Zhuowei Li, Haizhou Shi, Yunhe Gao, Di Liu, Zhenting Wang, Yuxiao Chen, Ting Liu, Long Zhao, Hao Wang, and Dimitris N. Metaxas. 2025. The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering. ArXiv abs/2502.03628 (2025)

    2025 arXiv

  29. [37]

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang

  30. [38]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023. Improved Baselines with Visual Instruction Tuning. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023), 26286–26296

    2023

  31. [39]

    arXiv preprint arXiv:2305.10355 (2023)

    Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355 (2023)

    2023 arXiv

  32. [40]

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rong-Zhi Li, and Wei Peng. 2024. A Survey on Hallucination in Large Vision-Language Models. ArXiv abs/2402.00253 (2024)

    2024 arXiv

  33. [41]

    Shiping Liu, Kecheng Zheng, and Wei Chen. 2024. Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs. ArXiv abs/2407.21771 (2024)

    2024 arXiv

  34. [42]

    In International Conference on Learning Representations

    Mitigating Hallucination in Large Multi-Modal Models via Robust Instruc- tion Tuning. In International Conference on Learning Representations

  35. [43]

    Manmatha, and C

    Minesh Mathew, Dimosthenis Karatzas, R. Manmatha, and C. V. Jawahar. 2020. DocVQA: A Dataset for VQA on Document Images. 2021 IEEE Winter Conference on Applications of Computer Vision (W ACV) (2020), 2199–2208

    2020

  36. [44]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual Instruc- tion Tuning. Advances in neural information processing systems 36 (2024)

    2024

  37. [45]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  38. [46]

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. arXiv preprint arXiv:1809.02156 (2018)

    2018 arXiv

  39. [47]

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multi- modal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems 35 ...

    2022

  40. [48]

    Yuying Shang, Xinyi Zeng, Yutao Zhu, Xiao Yang, Zhengwei Fang, Jingyuan Zhang, Jiawei Chen, Zinan Liu, and Yu Tian. 2024. From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models. ArXiv abs/2410.06795 (2024)

    2024 arXiv

  41. [49]

    Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, and Tong Zhang. 2024. Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization. ArXiv abs/2403.08730 (2024). Conference’17, July 2017, Washington, DC, USA Qiyan Zhao, Xiaofeng...

    2024 arXiv

  42. [50]

    Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. RoFormer: Enhanced Transformer with Rotary Position Embedding. ArXiv abs/2104.09864 (2021)

    2021 arXiv

  43. [51]

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. 2023. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525 (2023)

    2023 arXiv

  44. [52]

    Arik, and Tomas Pfister

    Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan Ö. Arik, and Tomas Pfister. 2024. Mitigating Object Hallucination in MLLMs via Data- augmented Phrase-level Alignment

    2024

  45. [53]

    Feilong Tang, Zile Huang, Chengzhi Liu, Qiang Sun, Harry Yang, and Ser-Nam Lim. 2025. Intervening Anchor Token: Decoding Strategy in Alleviating Hal- lucinations for MLLMs. In The Thirteenth International Conference on Learning Representations

    2025

  46. [54]

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards VQA Models That Can Read. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019), 8309–8318

    2019

  47. [55]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lam- ple. 2023. LLaMA: Open and Efficient Foundation ...

    2023 arXiv

  48. [56]

    Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N

    Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Neural Information Processing Systems

    2017

  49. [57]

    Feilong Tang, Zile Huang, Chengzhi Liu, Qiang Sun, Harry Yang, and Ser-Nam Lim. 2025. Intervening anchor token: Decoding strategy in alleviating hallu- cinations for MLLMs. In The Thirteenth International Conference on Learning Representations

    2025

  50. [58]

    Jiaqi Wang, Yifei Gao, and Jitao Sang. 2024. VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding. ArXiv abs/2411.15839 (2024)

    2024 arXiv

  51. [59]

    Feilong Tang, Chengzhi Liu, Zhongxing Xu, Ming Hu, Zile Huang, Haochen Xue, Ziyang Chen, Zelin Peng, Zhiwei Yang, Sijin Zhou, et al. 2025. Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding. In Proceedings of the Computer Vision and Patte...

    2025

  52. [60]

    Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023. Label words are anchors: An information flow perspective for understanding in-context learning. arXiv preprint arXiv:2305.14160 (2023)

    2023 arXiv

  53. [61]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-VL: Enhancing Vision-Language Mode...

    2024 arXiv

  54. [62]

    Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen

    Fei Wang, Wenxuan Zhou, James Y. Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2024. mDPO: Conditional Preference Optimization for Multimodal Large Language Models. ArXiv abs/2406.11839 (2024)

    2024 arXiv

  55. [63]

    Zhaoyang Wang, Weilei He, Zhiyuan Liang, Xuchao Zhang, Chetan Bansal, Ying Wei, Weitong Zhang, and Huaxiu Yao. 2024. CREAM: Consistency Regularized Self-Rewarding Language Models. ArXiv abs/2410.12735 (2024)

    2024 arXiv

  56. [64]

    Lei Wang, Jiabang He, Shenshen Li, Ning Liu, and Ee-Peng Lim. 2023. Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites. In Conference on Multimedia Modeling

    2023

  57. [65]

    Yuxi Xie, Guanzhen Li, Xiao Xu, and Min-Yen Kan. 2024. V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Pref- erence Optimization. In Conference on Empirical Methods in Natural Language Processing

    2024

  58. [67]

    Xintong Wang, Jingheng Pan, Liang Ding, and Christian Biemann. 2024. Mitigat- ing Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding. ArXiv abs/2403.18715 (2024)

    2024 arXiv

  59. [68]

    Haochen Xue, Feilong Tang, Ming Hu, Yexin Liu, Qidong Huang, Yulong Li, Chengzhi Liu, Zhongxing Xu, Chong Zhang, Chun-Mei Feng, et al. 2025. Mmrc: A large-scale benchmark for understanding multimodal large language model in real-world conversation. arXiv preprint arXiv:2502.11...

    2025 arXiv

  60. [69]

    Jinfeng Wei and Xiaofeng Zhang. 2024. DOPRA: Decoding Over-accumulation Penalization and Re-allocation in Specific Weighting Layer. Proceedings of the 32nd ACM International Conference on Multimedia (2024)

    2024

  61. [70]

    Hao Yin, Guangzong Si, and Zilei Wang. 2025. ClearSight: Visual Signal Enhance- ment for Object Hallucination Mitigation in Multimodal Large language Models. (2025)

    2025

  62. [71]

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549 (2023)

    2023 arXiv

  63. [72]

    Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. 2024. LLaVA-Critic: Learning to Evaluate Multi- modal Models. ArXiv abs/2410.02712 (2024)

    2024 arXiv

  64. [73]

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al . 2023. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. arXiv preprint arXiv:2312.00849 (2023)

    2023 arXiv

  65. [74]

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. 2024. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2024

  66. [75]

    Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. 2024. Unveiling and Harnessing Hidden Attention Sinks: En- hancing Large Language Models without Training through Attention Calibration. arXiv preprint arXiv:2406.15765 (2024)

    2024 arXiv

  67. [76]

    Zihao Yue, Liang Zhang, and Qin Jin. 2024. Less is more: Mitigating multimodal hallucination from an eos decision perspective. The 62nd Annual Meeting of the Association for Computational Linguistics (2024)

    2024

  68. [77]

    Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. 2023. HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (202...

    2023

  69. [78]

    Jinrui Zhang, Teng Wang, Haigang Zhang, Ping Lu, and Feng Zheng. 2024. Re- flective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models. ArXiv abs/2407.11422 (2024)

    2024 arXiv

  70. [79]

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490 (2023)

    2023 arXiv

  71. [80]

    Xiaofeng Zhang, Yihao Quan, Chaochen Gu, Chen Shen, Xiaosong Yuan, Shaotian Yan, Hao Cheng, Kaijie Wu, and Jieping Ye. 2024. Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMs. ArXiv abs/2411.09968 (2024)

    2024 arXiv

  72. [81]

    Xiaofeng Zhang, Chen Shen, Xiaosong Yuan, Shaotian Yan, Liang Xie, Wenxiao Wang, Chaochen Gu, Hao Tang, and Jieping Ye. 2024. From Redundancy to Relevance: Enhancing Explainability in Multimodal Large Language Models. arXiv preprint arXiv:2406.06579 (2024)

    2024 arXiv

  73. [82]

    Hang Zhang, Xin Li, and Lidong Bing. 2023. Video-llama: An instruction- tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858 (2023)

    2023 arXiv

  74. [83]

    Xiaofeng Zhang, Fanshuo Zeng, Yihao Quan, Zheng Hui, and Jiawei Yao. 2025. Enhancing Multimodal Large Language Models Complex Reason via Similarity Computation. AAAI (2025)

    2025

  75. [84]

    Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. 2023. Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv preprint arXiv:2307.03601 (2023)

    2023 arXiv

  76. [85]

    Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chen- hang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. 2024. Calibrated Self-Rewarding Vision Language Models. ArXiv abs/2405.14622 (2024)

    2024 arXiv

  77. [86]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models. arXiv preprint arXiv:2304.10592 (2023)

    2023 arXiv

  78. [87]

    Xiaofeng Zhang, Fanshuo Zeng, and Chaochen Gu. 2024. Simignore: Explor- ing and enhancing multimodal large model complex reasoning via similarity computation. Neural Networks (2024), 107059

    2024

  79. [88]

    Tinghui Zhu, Qin Liu, Fei Wang, Zhengzhong Tu, and Muhao Chen. 2024. Un- raveling Cross-Modality Knowledge Conflicts in Large Vision-Language Models. ArXiv abs/2410.03659 (2024)

    2024 arXiv

  80. [89]

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. 2023. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00754 (2023)

    2023 arXiv

  81. [92]

    Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. 2024. IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding. ArXiv abs/2402.18476 (2024)

    2024 arXiv

  82. [94]

    Younan Zhu, Linwei Tao, Minjing Dong, and Chang Xu. 2025. Mitigating Object Hallucinations in Large Vision-Language Models via Attention Calibration.ArXiv abs/2502.01969 (2025)

    2025

  83. [2016]

    International Journal of Computer Vision 127 (2016), 398 – 414

    Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. International Journal of Computer Vision 127 (2016), 398 – 414

    2016

  84. [2024]

    ArXiv abs/2408.05337 (2024)

    VACoDe: Visual Augmented Contrastive Decoding. ArXiv abs/2408.05337 (2024)

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.