Pith. sign in

REVIEW 8 cited by

Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.12718 v3 pith:NXPFEZSX submitted 2024-06-18 cs.CV cs.AIcs.CL

Wenbin An , Feng Tian , Sicong Leng , Jiahao Nie , Haonan Lin , QianYing Wang , Ping Chen , Xiaoqin Zhang
show 1 more author
Shijian Lu
This is my paper
classification cs.CVcs.AIcs.CL
keywords featureshallucinationslocalglobalimageobjectaglaattention
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Despite great success across various multimodal tasks, Large Vision-Language Models (LVLMs) often encounter object hallucinations with generated textual responses being inconsistent with the actual objects in images. We examine different LVLMs and pinpoint that one root cause of object hallucinations lies with deficient attention on discriminative image features. Specifically, LVLMs often predominantly attend to prompt-irrelevant global features instead of prompt-relevant local features, undermining their visual grounding capacity and leading to object hallucinations. We propose Assembly of Global and Local Attention (AGLA), a training-free and plug-and-play approach that mitigates hallucinations by assembling global features for response generation and local features for visual discrimination simultaneously. Specifically, we introduce an image-prompt matching scheme that captures prompt-relevant local features from images, leading to an augmented view of the input image where prompt-relevant content is highlighted while irrelevant distractions are suppressed. Hallucinations can thus be mitigated with a calibrated logit distribution that is from generative global features of the original image and discriminative local features of the augmented image. Extensive experiments show the superiority of AGLA in LVLM hallucination mitigation, demonstrating its wide applicability across both discriminative and generative tasks. Our code is available at https://github.com/Lackel/AGLA.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens

    cs.CV 2025-08 conditional novelty 6.0 of 10

    Modality bias, an imbalanced attention to text or image during hallucinated outputs, is shown to be mitigated by a training-free attention intervention plus contrastive decoding.

  2. INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling

    cs.CV 2025-07 conditional novelty 6.0 of 10

    INTER is a training-free logit-correction method that adds Harsanyi interaction scores to selected keyword tokens, lowering hallucination on six LVLM benchmarks.

  3. Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A dual-level attention intervention that boosts salient visual-token attention and suppresses text/system attention during decoding reduces hallucination rates in LLaVA, MiniGPT-4, and mPLUG-Owl2 on POPE and CHAIR.

  4. Disentangling Semantic Attention from Structural Bias in the Attention Manifold

    cs.CV 2026-07 conditional novelty 5.0 of 10

    SPAR removes a query-averaged structural bias from text-to-image attention and redistributes the reclaimed probability mass, reducing reported object and induced hallucinations in LLaVA models.

  5. CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models

    cs.CV 2025-06 conditional novelty 5.0 of 10

    CAI reduces object hallucination in LVLMs by injecting caption-query attention patterns into selected attention heads at inference time.

  6. MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Models

    cs.CV 2025-07 conditional novelty 4.0 of 10

    MCA-LLaVA reindexes image tokens by sums of mirrored 2D coordinates so instruction tokens attend across the whole image, reducing hallucination on POPE, CHAIR, and MME.

  7. MDSAM:Memory-Driven Sparse Attention Matrix for LVLMs Hallucination Mitigation

    cs.CV 2025-06 conditional novelty 4.0 of 10

    MDSAM stores top-k sparse attention patterns from earlier layers and blends them into later decoding steps, lowering hallucination rates on CHAIR, POPE, MME, and MMHal-Bench without any training.

  8. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Pith tools