Pith. sign in

REVIEW 47 cited by

Woodpecker: Hallucination Correction for Multimodal Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.16045 v2 pith:ZJ5DO3WS submitted 2023-10-24 cs.CV cs.AIcs.CLcs.LG

Shukang Yin , Chaoyou Fu , Sirui Zhao , Tong Xu , Hao Wang , Dianbo Sui , Yunhang Shen , Ke Li
show 2 more authors
Xing Sun Enhong Chen
This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.LG
keywords woodpeckerhallucinationmodelscorrectiondifferentfivegeneratedhallucinations
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Hallucination is a big shadow hanging over the rapidly evolving Multimodal Large Language Models (MLLMs), referring to the phenomenon that the generated text is inconsistent with the image content. In order to mitigate hallucinations, existing studies mainly resort to an instruction-tuning manner that requires retraining the models with specific data. In this paper, we pave a different way, introducing a training-free method named Woodpecker. Like a woodpecker heals trees, it picks out and corrects hallucinations from the generated text. Concretely, Woodpecker consists of five stages: key concept extraction, question formulation, visual knowledge validation, visual claim generation, and hallucination correction. Implemented in a post-remedy manner, Woodpecker can easily serve different MLLMs, while being interpretable by accessing intermediate outputs of the five stages. We evaluate Woodpecker both quantitatively and qualitatively and show the huge potential of this new paradigm. On the POPE benchmark, our method obtains a 30.66%/24.33% improvement in accuracy over the baseline MiniGPT-4/mPLUG-Owl. The source code is released at https://github.com/BradyFU/Woodpecker.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 47 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GeoArbiter: Verifiability-Guided Grounding for Remote-Sensing Multimodal LLMs

    cs.LG 2026-08 conditional novelty 7.0 of 10

    Filtering retrieved maps to image-unverifiable attributes preserves most of the RAG accuracy gain and removes the conflict failure mode in three remote-sensing MLLMs.

  2. Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sink

    cs.LG 2025-01 conditional novelty 7.0 of 10

    Adversarial images optimized to induce attention sink behavior increase hallucination rates in multiple MLLMs, including commercial APIs, without visibly degrading response quality.

  3. Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence

    cs.CL 2024-12 conditional novelty 7.0 of 10

    Hallucination in LVLMs correlates with low sensitivity of attention heads to image input, and scaling up the most image-sensitive heads reduces it without training.

  4. Witness Evidence Portfolios: Single-Prefill Risk Detection for Closed Multimodal Answers

    cs.CV 2026-07 accept novelty 6.0 of 10

    A single-prefill, white-box risk detector that estimates signed visual evidence from internal attention/value readouts and fuses it with confidence improves error ranking by 0.134 mean AP across 12 model–benchmark pairs.

  5. C-PTQ: Fisher-weighted Channel-wise Sensitivity for Post-training Quantization of MLLMs

    cs.CV 2026-07 conditional novelty 6.0 of 10

    C-PTQ weights quantization error by per-channel Fisher information of the task loss, improving low-bit accuracy of multimodal LLMs by small margins over existing channel-wise scaling methods.

  6. Controlling Multimodal LLMs via Reward-guided Decoding

    cs.CV 2025-08 conditional novelty 6.0 of 10

    MRGD guides MLLM decoding with a learned hallucination reward and a detector-based recall reward, allowing users to trade off object precision, recall, and test-time compute while reducing object hallucinations on CHA...

  7. ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A single-layer, single-query intervention that amplifies attention heads with high text-to-visual entropy reduces hallucination in LVLMs at about 1.07x the inference time of regular decoding.

  8. Understand, Think, and Answer: Advancing Visual Reasoning with Large Multimodal Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Griffon-R generates its own grounding hints and rationale before answering, achieving state-of-the-art visual reasoning on VSR and CLEVR while improving MMBench, ScienceQA, and TextVQA.

  9. Retrieval Visual Contrastive Decoding to Mitigate Object Hallucinations in Large Vision-Language Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    RVCD uses YOLO detections and retrieved single-concept AI images to adjust LVLM logits at decode time, cutting CHAIR hallucination rates by roughly half versus prior contrastive decoding baselines.

  10. ChartLens: Fine-grained Visual Attribution in Charts

    cs.CL 2025-05 conditional novelty 6.0 of 10

    ChartLens uses segmentation and set-of-marks prompting to attribute chart-based answers to specific visual elements, and the authors release a new benchmark for evaluating such attribution.

  11. Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations

    cs.CV 2025-05 conditional novelty 6.0 of 10

    VaLSe uses attention-based visual contribution maps to steer an LVLM's latent features toward visually grounded content, reducing object hallucinations on several benchmarks while exposing flaws in CHAIR-style evaluation.

  12. Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decoding

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Ensemble Decoding reduces object hallucination in large vision-language models by ensembling logits from attention-weighted image sub-images.

  13. Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perception

    cs.CV 2025-04 conditional novelty 6.0 of 10

    Antidote trains LVLMs to reject false presuppositions in questions and reduce object hallucination using synthetic images, self-corrected preferences, and a new CP-Bench benchmark.

  14. Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models

    cs.CV 2025-02 conditional novelty 6.0 of 10

    DeGF reduces hallucinations in vision-language models by generating an image from the model's own response and using the divergence between predictions on original and generated images to switch between complementary ...

  15. The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering

    cs.CV 2025-02 conditional novelty 6.0 of 10

    VISTA reduces hallucination in vision-language models by adding a per-image visual steering vector to hidden states and blending in early-layer logits, cutting CHAIR object hallucination by about 40%.

  16. Synthetic Artifact Auditing: Tracing LLM-Generated Synthetic Data Usage in Downstream Applications

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A three-method auditing framework detects with roughly 87 to 97 percent accuracy whether classifiers, generators, and t-SNE plots were trained on or derived from LLM-generated synthetic data.

  17. CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs

    cs.CL 2025-01 conditional novelty 6.0 of 10

    CHiP adds image-level and phrase/token-level preference signals to DPO for multimodal LLMs, and the authors report large hallucination-rate reductions on Object HalBench, AMBER, MMHal, and HallusionBench.

  18. VASparse: Towards Efficient Visual Hallucination Mitigation via Visual-Aware Token Sparsification

    cs.CV 2025-01 conditional novelty 6.0 of 10

    VASparse combines visual-aware token pruning, embedding-based visual contrastive decoding, and an attention-sink penalty to reduce visual hallucinations in LVLMs without extra training.

  19. Feedback-Driven Vision-Language Alignment with Minimal Human Supervision

    cs.CV 2025-01 conditional novelty 6.0 of 10

    SVP uses self-captioning with grounding feedback and a scoring filter to adapt vision-language models with about 1,000 images, improving captioning, referring, object recall, and hallucination control.

  20. Cross-Modal Attention Calibration for LVLM Hallucination Mitigation

    cs.CV 2025-01 conditional novelty 6.0 of 10

    IMCCD combines value-vector masking in cross-modal attention with a position-normalizing decoding step and reports lower hallucination than VCD and ICD on POPE, CHAIR, and MME.

  21. VORD: Visual Ordinal Calibration for Mitigating Object Hallucinations in Large Vision-Language Models

    cs.CV 2024-12 conditional novelty 6.0 of 10

    VORD filters hallucinated tokens by requiring that a token's probability on the original image exceed its probability on a corrupted image, with an adaptive margin.

  22. Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage

    cs.CV 2024-12 conditional novelty 6.0 of 10

    CapMAS improves factual accuracy of long image captions by atomic claim decomposition and verification, and the paper adds a multimodal factuality metric and a coverage QA benchmark.

  23. ReXTrust: A Model for Fine-Grained Hallucination Detection in AI-Generated Radiology Reports

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A self-attention model over hidden states of a vision-language model identifies hallucinated findings in AI-generated radiology reports with AUROC 0.8751, outperforming prior detectors.

  24. Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Models

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Entropy-weighted fusion of four visually distorted contrastive samples gives the best overall accuracy (0.833) across three LVLMs on POPE and MME, outperforming single-sample contrastive decoding (best single: 0.824).

  25. Anatomically-Grounded Fact Checking of Automated Chest X-ray Reports

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A contrastive regression network trained on synthetic perturbed reports can detect false findings and misplaced findings in automated chest X-ray reports, and a correction step improves report quality by about 40 percent.

  26. Detailed Object Description with Controllable Dimensions

    cs.CV 2024-11 conditional novelty 6.0 of 10

    A training-free post-processing pipeline improves how well multimodal LLMs stick to user-selected object dimensions such as color, texture, and pose.

  27. Decompose and Leverage Preferences from Expert Models for Improving Trustworthiness of MLLMs

    cs.CV 2024-11 conditional novelty 6.0 of 10

    Decomposing MLLM responses into atomic verification tasks and checking them with an ensemble of open-source expert models yields preference data that reduces hallucination in LLaVA and Qwen-VL-Chat.

  28. Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization

    cs.CL 2024-11 conditional novelty 6.0 of 10

    HDPO trains LLaVA models with three types of hallucination-targeted preference pairs (visual distraction, long-context drift, and text-image conflict) and reports reduced hallucination on CHAIR, POPE, and AMBER.

  29. Energy-Guided Decoding for Object Hallucination Mitigation

    cs.CV 2025-07 conditional novelty 5.0 of 10

    An energy-guided, training-free decoding rule that chooses the layer with minimal energy reduces object hallucination and yes-bias on several benchmarks.

  30. Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding

    cs.CV 2025-05 conditional novelty 5.0 of 10

    FarSight adds upper-triangular negative biases to the causal mask to absorb outlier-token attention, reducing hallucinations in MLLMs without training.

  31. A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models

    cs.CV 2025-05 conditional novelty 5.0 of 10

    The paper finds the LLM is mostly faithful given good captions, the CLIP vision encoder contributes perception errors, and the projector preserves visual information but aligns it poorly with text.

  32. VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model

    cs.CV 2025-01 conditional novelty 5.0 of 10

    VARGPT combines LLaVA-style next-token visual understanding with VAR-style next-scale visual generation in one autoregressive multimodal model trained in three stages.

  33. Improving Medical Large Vision-Language Models with Abnormal-Aware Feedback

    cs.CL 2025-01 conditional novelty 5.0 of 10

    UMed-LVLM uses GPT-4V-generated abnormality data and abnormal-aware rewards to improve medical image diagnosis and localization.

  34. RapGuard: Safeguarding Multimodal Large Language Models via Rationale-aware Defensive Prompting

    cs.CL 2024-12 conditional novelty 5.0 of 10

    A rationale-aware defensive prompting framework uses multimodal chain-of-thought and self-checking to reduce harmful MLLM outputs while preserving benign utility.

  35. Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection

    cs.CV 2024-12 conditional novelty 5.0 of 10

    Nullu projects a vision-language model's weights into the null space of a subspace learned from truthful versus hallucinated captions, reducing object hallucinations without extra inference cost.

  36. Hallucination Elimination and Semantic Enhancement Framework for Vision-Language Models in Traffic Scenarios

    cs.CV 2024-12 reject novelty 5.0 of 10

    HCOENet combines multi-model entity cross-checking with open-set detection to delete hallucinated objects and add descriptions of overlooked traffic objects, with reported F1 gains on a modified POPE evaluation.

  37. Evaluating Hallucination in Text-to-Image Diffusion Models with Scene-Graph based Question-Answering Agent

    cs.CV 2024-12 conditional novelty 5.0 of 10

    A scene-graph plus LLM question-answering pipeline scores text-to-image faithfulness and categorizes hallucination types, but the measured correlation gain over prior metrics is small and not statistically tested.

  38. VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understanding

    cs.CV 2024-12 conditional novelty 5.0 of 10

    VidHalluc is a 5,002-video paired benchmark for action, temporal sequence, and scene transition hallucinations in video MLLMs, and DINO-HEAL is a training-free saliency reweighting method that improves hallucination s...

  39. Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysis

    cs.CV 2024-12 conditional novelty 5.0 of 10

    Prompt learning adapts multimodal LLMs to concise image captioning while preserving out-of-domain generalization better than full fine-tuning.

  40. Who Brings the Frisbee: Probing Hidden Hallucination Factors in Large Vision-Language Model via Causality Analysis

    cs.CV 2024-12 conditional novelty 5.0 of 10

    A study of InstructBLIP and mPLUG-Owl2 finds that scene words like grass and tree co-occur with hallucinated objects, and a two-step foreground/background prompt lowers hallucination scores.

  41. VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding

    cs.CV 2024-11 conditional novelty 5.0 of 10

    VaLiD mitigates LVLM hallucination by entropy-weighted fusion of early visual layers in contrastive decoding.

  42. Thinking Before Looking: Improving Multimodal LLM Reasoning via Mitigating Visual Hallucination

    cs.CV 2024-11 conditional novelty 5.0 of 10

    VIC improves multimodal LLM accuracy on hallucination and general VQA benchmarks by reasoning from text before seeing the image, though gains are inconsistent on open-source models.

  43. A Large Vision-Language Model based Environment Perception System for Visually Impaired People

    cs.CV 2025-04 conditional novelty 4.0 of 10

    A wearable system that feeds segmentation results into a vision-language model's prompt improves scene description accuracy and object retrieval for visually impaired users.

  44. Socratic Questioning: Learn to Self-guide Multimodal Reasoning in the Wild

    cs.CV 2025-01 conditional novelty 4.0 of 10

    A self-questioning training and inference framework for lightweight multimodal LLMs is claimed to reduce hallucinations and improve zero-shot visual reasoning.

  45. MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs

    cs.CV 2024-11 conditional novelty 4.0 of 10

    A broad survey that organizes MLLM evaluation benchmarks into capability categories, explains benchmark construction and scoring methods, and identifies gaps in current evaluation practice.

  46. Test-Time Hallucination Control in Large Vision-Language Models

    cs.CV 2026-08 conditional novelty 3.0 of 10

    At each decoding step, TTH blends the LVLM's logits for candidate object tokens with CLIP image-text similarity scores, weighted by the model's uncertainty, to suppress hallucinated objects.

  47. Multimodal Large Language Models for Medicine: A Comprehensive Survey

    cs.LG 2025-04 conditional novelty 2.0 of 10

    A comprehensive review cataloging medical MLLMs, their uses in report generation, diagnosis, and treatment, and the challenges of accuracy, hallucination, fairness, privacy, and deployment.

Pith tools