Pith. sign in

REVIEW 4 cited by

PhD: A ChatGPT-Prompted Visual hallucination Evaluation Dataset

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.11116 v4 pith:7R5KHBHI submitted 2024-03-17 cs.CV cs.AI

Jiazhen Liu , Yuhan Fu , Ruobing Xie , Runquan Xie , Xingwu Sun , Fengzong Lian , Zhanhui Kang , Xirong Li This is my paper
classification cs.CVcs.AI
keywords hallucinationvisualimagescontextevaluationgenerationmllmsquestions
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Multimodal Large Language Models (MLLMs) hallucinate, resulting in an emerging topic of visual hallucination evaluation (VHE). This paper contributes a ChatGPT-Prompted visual hallucination evaluation Dataset (PhD) for objective VHE at a large scale. The essence of VHE is to ask an MLLM questions about specific images to assess its susceptibility to hallucination. Depending on what to ask (objects, attributes, sentiment, etc.) and how the questions are asked, we structure PhD along two dimensions, i.e. task and mode. Five visual recognition tasks, ranging from low-level (object / attribute recognition) to middle-level (sentiment / position recognition and counting), are considered. Besides a normal visual QA mode, which we term PhD-base, PhD also asks questions with specious context (PhD-sec) or with incorrect context ({PhD-icc), or with AI-generated counter common sense images (PhD-ccs). We construct PhD by a ChatGPT-assisted semi-automated pipeline, encompassing four pivotal modules: task-specific hallucinatory item (hitem) selection, hitem-embedded question generation, specious / incorrect context generation, and counter-common-sense (CCS) image generation. With over 14k daily images, 750 CCS images and 102k VQA triplets in total, PhD reveals considerable variability in MLLMs' performance across various modes and tasks, offering valuable insights into the nature of hallucination. As such, PhD stands as a potent tool not only for VHE but may also play a significant role in the refinement of MLLMs.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation

    cs.SE 2026-08 conditional novelty 6.0 of 10

    On a new 1,440-screenshot fill-in-the-blank benchmark, five frontier multimodal models default to repeated UI patterns instead of visual deviations, with mean bias rates from 69.8% on card widths to 80.2% on font sizes.

  2. MESH -- Understanding Videos Like Human: Measuring Hallucinations in Large Video Models

    cs.CV 2025-09 conditional novelty 6.0 of 10

    MESH, a three-layer video hallucination benchmark, shows LVMs ace basic objects and coarse traits but slip badly on fine character details and multi-subject actions in longer clips.

  3. Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding

    cs.CV 2025-05 conditional novelty 5.0 of 10

    FarSight adds upper-triangular negative biases to the causal mask to absorb outlier-token attention, reducing hallucinations in MLLMs without training.

  4. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Pith tools