Pith

open record

sign in

arxiv: 2310.01779 · v3 · pith:L3EPNW6Y · submitted 2023-10-03 · cs.CV

HallE-Control: Controlling Object Hallucination in Large Multimodal Models

Bohan Zhai , Shijia Yang , Chenfeng Xu , Sheng Shen , Kurt Keutzer , Chunyuan Li , Manling Li This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:L3EPNW6Yrecord.jsonopen to challenge →

classification cs.CV
keywords hallucinationobjectcaptioningknowledgeobjectshalle-controlcontextualdetailed
0
0 comments X
read the original abstract

Current Large Multimodal Models (LMMs) achieve remarkable progress, yet there remains significant uncertainty regarding their ability to accurately apprehend visual details, that is, in performing detailed captioning. To address this, we introduce $\textit{CCEval}$, a GPT-4 assisted evaluation method for detailed captioning. Interestingly, while LMMs demonstrate minimal object existence hallucination in existing VQA benchmarks, our proposed evaluation reveals continued susceptibility to such hallucinations. In this paper, we make the first attempt to investigate such hallucination from different aspects, including image resolution, the language decoder size, and instruction data amount, quality, granularity. Our findings underscore the unwarranted inference when the language description includes details at a finer object granularity than what the vision module can ground or verify, thus inducing hallucination. To control such hallucinations, we further attribute the reliability of captioning to contextual knowledge (involving only contextually grounded objects) and parametric knowledge (containing inferred objects by the model). Thus, we introduce $\textit{HallE-Control}$, a controllable LMM in terms of $\textbf{Hall}$ucination in object $\textbf{E}$xistence. HallE-Control can condition the captioning to shift between (i) exclusively depicting contextual knowledge for grounded objects and (ii) blending it with parametric knowledge to imagine inferred objects. Our method reduces hallucination by 44% compared to LLaVA$_{7B}$ and maintains the object coverage.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Remember with Confidence: Uncertainty Quantification for Spatio-temporal Memory with Probabilistic Guarantees

    cs.CV 2026-06 unverdicted novelty 7.0

    Introduces object-level semantic uncertainty for VLM memory, the UQ-DAAAM refinement system, and probabilistic guarantees that selected high-quality views reduce uncertainty more effectively.

  2. KoALa-Bench: Evaluating Large Audio Language Models on Korean Speech Understanding and Faithfulness

    cs.CL 2026-03 unverdicted novelty 7.0

    KoALa-Bench is a new public benchmark with six tasks that tests Korean speech recognition, translation, question answering, instruction following, and faithfulness in large audio language models.

  3. CaptionQA: Is Your Caption as Useful as the Image Itself?

    cs.CV 2025-11 conditional novelty 7.0

    CaptionQA is a new benchmark with 33,027 questions across natural, document, e-commerce, and embodied AI domains that measures how much utility model-generated captions retain compared to original images when used by ...

  4. Counterfactual Segmentation Reasoning: Diagnosing and Mitigating Pixel-Grounding Hallucination

    cs.CV 2025-06 unverdicted novelty 7.0

    Proposes CSR task and HalluSegBench using visual counterfactuals to diagnose segmentation hallucinations in VLMs, plus RobustSeg via counterfactual fine-tuning that reduces hallucinations by 30% on FP-RefCOCO.

  5. ZINA: Multimodal Fine-grained Hallucination Detection and Editing

    cs.CV 2025-06 unverdicted novelty 7.0

    ZINA detects fine-grained hallucinations in MLLM outputs, classifies errors into six types, and proposes edits, outperforming GPT-4o and Llama-3.2 on the new VisionHall dataset of annotated and synthetic samples.

  6. Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens

    cs.CV 2026-05 unverdicted novelty 6.0

    Reweighting training emphasis toward image-negative tokens and filtering hallucinated data reduces object hallucination in LVLMs across three model variants.

  7. Deep Pre-Alignment for VLMs

    cs.CV 2026-05 unverdicted novelty 6.0

    Deep Pre-Alignment uses a small VLM perceiver instead of ViT to pre-align visual features with LLM text space, yielding 1.9-3.0 point gains on multimodal benchmarks and 32.9% less language forgetting.

  8. Vocabulary Hijacking in LVLMs: Unveiling Critical Attention Heads by Excluding Inert Tokens to Mitigate Hallucination

    cs.MM 2026-05 unverdicted novelty 6.0

    LVLMs show vocabulary hijacking by inert tokens that decode to hijacking anchors; HABI locates them, NHAR finds resilient heads, and HAVAE boosts those heads to cut hallucinations.

  9. AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation

    cs.CL 2023-11 unverdicted novelty 6.0

    AMBER is an LLM-free multi-dimensional benchmark for evaluating hallucinations in MLLMs across generative and discriminative tasks.

  10. A Survey on Hallucination in Large Vision-Language Models

    cs.CV 2024-02 unverdicted novelty 3.0

    This survey reviews the definition, symptoms, evaluation benchmarks, root causes, and mitigation methods for hallucinations in large vision-language models.

  11. A Survey on Multimodal Large Language Models

    cs.CV 2023-06 accept novelty 3.0

    This survey organizes the architectures, training strategies, data, evaluation methods, extensions, and challenges of Multimodal Large Language Models.