Pith. sign in

REVIEW 5 cited by

Mitigating Open-Vocabulary Caption Hallucinations

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.03631 v4 pith:6HRIOB6Q submitted 2023-12-06 cs.CV cs.AI

Assaf Ben-Kish , Moran Yanuka , Morris Alper , Raja Giryes , Hadar Averbuch-Elor This is my paper
classification cs.CVcs.AI
keywords hallucinationsimagecaptioningopen-vocabularybenchmarkmochamodelsobject
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

While recent years have seen rapid progress in image-conditioned text generation, image captioning still suffers from the fundamental issue of hallucinations, namely, the generation of spurious details that cannot be inferred from the given image. Existing methods largely use closed-vocabulary object lists to mitigate or evaluate hallucinations in image captioning, ignoring the long-tailed nature of hallucinations that occur in practice. To this end, we propose a framework for addressing hallucinations in image captioning in the open-vocabulary setting. Our framework includes a new benchmark, OpenCHAIR, that leverages generative foundation models to evaluate open-vocabulary object hallucinations for image captioning, surpassing the popular and similarly-sized CHAIR benchmark in both diversity and accuracy. Furthermore, to mitigate open-vocabulary hallucinations without using a closed object list, we propose MOCHa, an approach harnessing advancements in reinforcement learning. Our multi-objective reward function explicitly targets the trade-off between fidelity and adequacy in generations without requiring any strong supervision. MOCHa improves a large variety of image captioning models, as captured by our OpenCHAIR benchmark and other existing metrics. Code and models can be found at: https://github.com/assafbk/mocha_code

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Groc-PO: Grounded Context Preference Optimization for Truthful Multimodal LLMs

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Groc-PO applies preference optimization at three grounded stages — object grounding, contextual grounding, grounded reasoning — and outperforms final-answer-only DPO on hallucination and complex-reasoning benchmarks.

  2. MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM

    cs.CV 2025-05 conditional novelty 6.0 of 10

    MIRAGE is a benchmark that separates reasoning hallucinations from perception errors in multimodal LLMs, and Logos is a curriculum reinforcement fine-tuning method that reduces logical hallucinations.

  3. The Hallucination Tax of Reinforcement Finetuning

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Standard RFT sharply reduces LLM refusal on unanswerable questions, and adding 10% synthetic unanswerable math during RFT restores refusal with small accuracy losses.

  4. Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Images

    cs.AI 2025-06 reject novelty 4.0 of 10

    SHE lowers behavioral hallucination scores by about 10 percent by detecting low visual-textual similarity and projecting out the hallucinated direction in embedding space.

  5. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Pith tools