REVIEW 7 cited by
MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Multimodal Large Language Models (MLLMs) frequently exhibit hallucination phenomena, but the underlying reasons remain poorly understood. In this paper, we present an empirical analysis and find that, although MLLMs incorrectly generate the objects in the final output, they are actually able to recognize visual objects in the preceding layers. We speculate that this may be due to the strong knowledge priors of the language model suppressing the visual information, leading to hallucinations. Motivated by this, we propose a novel dynamic correction decoding method for MLLMs DeCo, which adaptively selects the appropriate preceding layers and proportionally integrates knowledge into the final layer to adjust the output logits. Note that DeCo is model agnostic and can be seamlessly incorporated with various classic decoding strategies and applied to different MLLMs. We evaluate DeCo on widely-used benchmarks, demonstrating that it can reduce hallucination rates by a large margin compared to baselines, highlighting its potential to mitigate hallucinations. Code is available at https://github.com/zjunlp/DeCo.
Forward citations
Cited by 7 Pith papers
-
Trace, Verify, and Correct: A Training-Free Framework for Spatial Reasoning in Multimodal LLMs
A training-free verification and correction framework traces spatial claims in multimodal LLM reasoning to visual evidence, detects the earliest contradicted step, and regenerates the remaining reasoning.
-
Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation
Tracking positive shifts in visual attention over information-rich query words yields a saliency map that, when used to boost visual and query attention during decoding, reduces object hallucination on CHAIR, POPE, an...
-
Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens
Modality bias, an imbalanced attention to text or image during hallucinated outputs, is shown to be mitigated by a training-free attention intervention plus contrastive decoding.
-
Enhancing Visual Reliance in Text Generation: A Bayesian Perspective on Mitigating Hallucination in Large Vision-Language Models
EVRB is a three-part inference-time method that prunes ambiguous visual tokens, divides the model's output distribution by a text-only prior, and triggers early stopping to reduce hallucination in LVLMs.
-
Extracting Visual Facts from Intermediate Layers for Mitigating Hallucinations in Multimodal Large Language Models
Selecting the intermediate layer where image-conditioned and text-only predictions diverge most, and adding that layer's contrastive visual signal back to the final logits, reduces object hallucinations in four large ...
-
PostAlign: Multimodal Grounding as a Corrective Lens for MLLMs
MMGrounded-PostAlign trains MLLMs to produce a grounded object token or a rejection token plus selective rationales, improving hallucination and VQA benchmarks.
-
Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Images
SHE lowers behavioral hallucination scores by about 10 percent by detecting low visual-textual similarity and projecting out the hallucinated direction in embedding space.
Discussion (0). Sign in to comment.