REVIEW 9 cited by
Seeing is Believing: Mitigating Hallucination in Large Vision-Language Models via CLIP-Guided Decoding
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Vision-Language Models (LVLMs) are susceptible to object hallucinations, an issue in which their generated text contains non-existent objects, greatly limiting their reliability and practicality. Current approaches often rely on the model's token likelihoods or other internal information, instruction tuning on additional datasets, or incorporating complex external tools. We first perform empirical analysis on sentence-level LVLM hallucination, finding that CLIP similarity to the image acts as a stronger and more robust indicator of hallucination compared to token likelihoods. Motivated by this, we introduce our CLIP-Guided Decoding (CGD) approach, a straightforward but effective training-free approach to reduce object hallucination at decoding time. CGD uses CLIP to guide the model's decoding process by enhancing visual grounding of generated text with the image. Experiments demonstrate that CGD effectively mitigates object hallucination across multiple LVLM families while preserving the utility of text generation. Codes are available at https://github.com/d-ailin/CLIP-Guided-Decoding.
Forward citations
Cited by 9 Pith papers
-
Constraint-Anchored Reasoning Traces
CART interleaves language-model reasoning with verifiable symbolic anchors and backtracking, reducing error snowballing in open-source MLLMs.
-
Controlling Multimodal LLMs via Reward-guided Decoding
MRGD guides MLLM decoding with a learned hallucination reward and a detector-based recall reward, allowing users to trade off object precision, recall, and test-time compute while reducing object hallucinations on CHA...
-
MissingBench-Verified: Probing Vision-Language Models' Inability to Detect Missing Object Parts
Ten leading VLMs mostly fail to report removed essential object parts as missing, and simulated detector evidence, image tools, longer reasoning, and an easier fine-tune barely improve accuracy.
-
Energy-Guided Decoding for Object Hallucination Mitigation
An energy-guided, training-free decoding rule that chooses the layer with minimal energy reduces object hallucination and yes-bias on several benchmarks.
-
PostAlign: Multimodal Grounding as a Corrective Lens for MLLMs
MMGrounded-PostAlign trains MLLMs to produce a grounded object token or a rejection token plus selective rationales, improving hallucination and VQA benchmarks.
-
Challenges in Understanding Modality Conflict in Vision-Language Models
In LLaVA-OV-7B, a linearly decodable conflict signal appears in intermediate layers and detection-related attention shifts precede resolution-related ones, supporting a detection/resolution separation in the model.
-
MDSAM:Memory-Driven Sparse Attention Matrix for LVLMs Hallucination Mitigation
MDSAM stores top-k sparse attention patterns from earlier layers and blends them into later decoding steps, lowering hallucination rates on CHAIR, POPE, MME, and MMHal-Bench without any training.
-
Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Images
SHE lowers behavioral hallucination scores by about 10 percent by detecting low visual-textual similarity and projecting out the hallucinated direction in embedding space.
-
Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.
Discussion (0). Sign in to comment.