Pith. sign in

REVIEW 3 major objections 5 minor 52 references

SHALE: A Scalable Benchmark for Fine-grained Hallucination Evaluation in LVLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that fine-grained hallucination in LVLMs—both faithfulness to the image and factuality to world knowledge—can be measured at scale with a fully automated, synthetic data pipeline that also simulates noisy real-world inputs,

desk verdict Useful synthetic benchmark for fine-grained LVLM hallucination; ground-truth validity is the main caveat but doesn't sink it. read the letter →

arxiv 2508.09584 v2 pith:JT2KSQQH submitted 2025-08-13 cs.CV

Bei Yan , Zhiyuan Chen , Yuecong Min , Jie Zhang , Jiahao Wang , Xiaozhen Wang , Shiguang Shan This is my paper
classification cs.CV
keywords hallucinationevaluationlargevision-languagemodelssyntheticbenchmarkfaithfulnessfactualityinputperturbationtext-to-imagegenerationmultimodalleakage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that existing hallucination benchmarks for large vision-language models (LVLMs) are too coarse, too costly, and too leaky, and that a scalable alternative can be built by synthesizing images and questions with known ground truth. It constructs SHALE, over 30,000 image-instruction pairs spanning 12 visual perception aspects and 6 knowledge domains, under clean and perturbed conditions. Testing 21 LVLMs, the paper finds factuality hallucinations are more severe than faithfulness hallucinations, and that scene text injection and misleading instruction prefixes degrade models far more than image corruption or style changes. If the pipeline is trustworthy, hallucination evaluation becomes cheap, controllable, and resistant to data leakage, and the same data can be repurposed for targeted fine-tuning.

What carries the argument

The load-bearing mechanism is the automated construction loop: (1) prompt templates like '<entity1> <attribute> <relation> <entity2>' are instantiated from curated element pools; (2) a text-to-image diffusion model renders the image; (3) VQAScore, an image–text alignment metric, filters out low-agreement images, with OCR verification for scene text; (4) instructions in four formats (yes/no, multiple-choice, free-form, captioning) are generated from the same templates, so ground truth is known by construction; (5) a hierarchical perturbation framework adds image-level (style, corruption, adversarial, scene text), instruction-level (confusing synonyms, misleading prefixes), and combined noise.

What would settle it

Randomly sample a few hundred SHALE image-instruction pairs and have three independent human annotators verify the ground-truth answer against the image (for instance, is the bear actually brown, or is the count actually four). If annotator agreement with the paper's ground truth falls well below 90%, or if the paper's own claim of fewer than 5% alignment issues fails to replicate, the ground truth is unreliable and all model scores derived from it would be mis-calibrated.

Watch

Extended reading notes

Core claim

SHALE is a benchmark built without real images or manual annotation: type-specific templates are filled with entities, attributes, and relations, fed to a text-to-image model, filtered by VQAScore, and ground-truth answers are read directly from the prompts that produced the images, with DBpedia supplying facts for the factuality portion. The central empirical claim, across 21 LVLMs, is that models struggle most with relational faithfulness (size and spatial relations) and with factual knowledge in sports and politics, and that semantic perturbations—injected scene text and misleading instruction prefixes—are far more damaging than low-level visual noise. The paper further claims that SHALE

Load-bearing premise

The benchmark's ground-truth answers are inherited from the text prompts used to synthesize the images, so the entire evaluation assumes the text-to-image model faithfully renders every intended entity, attribute, and relation; only 500 of over 30,000 images were manually reviewed, and VQAScore is used as a proxy for semantic fidelity rather than direct verification.

Editorial extensions

If this is right

  • Generative tasks (free-form answers and captions) are considerably harder than discriminative tasks (yes/no and multiple choice), with average performance dropping more than 15% under clean conditions.
  • Factuality hallucinations are more common than faithfulness hallucinations for most top models, particularly in the sports and politics knowledge domains.
  • Misleading instruction prefixes and inserted scene text are the most disruptive perturbations; combining them creates the most challenging scenario, while image corruption and style transfer cause only minor drops.
  • Strong clean-scenario accuracy does not predict robustness: some models with middling clean scores have the highest Resistance Rates, and closed-source models are generally more resistant.
  • Blind-setting performance near random guessing, together with low multimodal leakage, supports the claim that SHALE avoids the data leakage that affects benchmarks built from public images.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same automated pipeline could be run in reverse to generate targeted fine-tuning data for hallucination mitigation: sample images where models fail, and use the known ground truth to construct corrective instruction pairs.
  • Factuality answers inherit DBpedia's content, so the benchmark's factuality scores are contingent on DBpedia's accuracy and currency; the pipeline could be re-run as knowledge bases are updated.
  • The finding that misleading prefixes and scene text dominate over pixel noise suggests hallucination risk is driven more by linguistic priors and optical text than by low-level image degradation, pointing toward interventions such as instruction warnings or OCR-aware decoding.
  • Resistance Rate, as defined in the paper, could serve as a standard robustness axis alongside accuracy in model selection, capturing stability under perturbation that clean-score rankings miss.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces SHALE, a synthetic benchmark for fine-grained hallucination evaluation in large vision-language models (LVLMs). The construction pipeline designs type-specific prompt templates over curated entity/attribute pools, generates images with a text-to-image model (Stable Diffusion 3.5), filters them with VQAScore using thresholds α=0.85 and β=0.5 plus OCR for text-bearing images, and derives instructions and ground-truth answers from the prompts and from DBpedia facts. The benchmark contains over 30K image-instruction pairs spanning 12 faithfulness aspects and 6 knowledge domains, under clean and noisy (image-, instruction-, and combination-level) scenarios. The authors evaluate 21 LVLMs and report that generative tasks, factuality hallucinations, relational dimensions, and semantic perturbations are especially challenging; they also report low blind accuracy and favorable multimodal leakage/gain relative to POPE, AMBER, and SEED.

Significance. The paper's main strengths are its scalable and controllable pipeline, its finer-grained hallucination taxonomy than prior benchmarks (12 visual aspects, 6 knowledge domains, 4 task formats), and its systematic study of perturbed inputs with a resistance-rate metric. The public release and the MG/ML leakage analysis are additional positive features. If the ground-truth labels are trustworthy, SHALE would be a valuable complement to manually curated hallucination benchmarks. However, the benchmark's validity rests on the assumption that each generated image actually exhibits the prompt-specified attribute and depicts the intended entity; the current verification is thin relative to that load-bearing assumption.

major comments (3)
  1. [§3.2, §A.1] Ground truth is inherited from the generation prompt (faithfulness) and from DBpedia facts attached to the intended entity (factuality), but the paper does not verify per-sample that the generated image exhibits the prompted attribute or depicts the intended entity. VQAScore is a single sentence-level alignment score and cannot certify object count, spatial relations, exact color, or identity of people/landmarks. The manual audit of 500 images checks only general prompt-image alignment and whether the image might mislead, not the specific fine-grained label used in each instruction. Consequently, a model that correctly describes the actual pixels would be scored as hallucinating whenever the image deviates from the prompt. This directly threatens the fine-grained accuracy numbers in Tables 5–7 and the factuality conclusions. Please add a stratified per-aspect/per-domain audit (or per-ite
  2. [§4.2, Table 4] The claim that 'SHALE achieves the lowest ML and highest MG among all benchmarks' is supported only by two LLaVA model rows. Multimodal Gain and Multimodal Leakage are model-dependent; generalizing from two models overstates the result. Please report MG/ML across the full model suite, or confine the claim to the models actually evaluated.
  3. [§A.3, Figure 4] The LLM-as-a-judge prompt for generative tasks supplies the original text-to-image prompt as 'Image Information' and DBpedia facts as 'Related Facts'. The judge therefore evaluates model outputs against the prompt/facts rather than against the image content. This is acceptable only if prompt-image alignment is effectively guaranteed; combined with the first comment, the non-hallucination rates in Table 3 inherit the same label-fidelity risk. Please either verify per-sample image-prompt alignment for generative test items, or explicitly state in the Limitations that generative scores measure prompt-consistency rather than image-consistency.
minor comments (5)
  1. [Figure 3, Figure 5] The captions of Figures 3 and 5 contain garbled unicode/escaped text (e.g., '/uni00000037/...'). These should be repaired; the figures themselves are also not referenced precisely in the body.
  2. [Table 4] The table header contains mojibake ('MGâĘŚ MLâĘŞ'). Please fix the encoding.
  3. [§4.2] The blind-setting experiment is described for the clean scenario; please clarify whether blind accuracy was also measured for noisy scenario instructions, and if not, state this explicitly.
  4. [§3.3] The Resistance Rate is defined only over clean non-hallucinated pairs. This is reasonable, but it should be noted that models with very low clean accuracy can have high RR by construction; the paper's interpretation would benefit from reporting clean accuracy alongside RR.
  5. [§5] The Limitations paragraph acknowledges limited real-world diversity and non-exhaustive categories. Consider also acknowledging the prompt-inheritance label issue discussed above, since it is the main threat to benchmark validity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the benchmark's construction and evaluation are self-contained, and the validity concerns raised are measurement assumptions, not circular derivations.

full rationale

SHALE constructs images by prompting a text-to-image model and derives ground-truth labels from those same prompts (faithfulness) and from DBpedia facts (factuality). This is a standard known-label synthetic benchmark design: the model under evaluation is never shown the prompt, and the labels are not fitted to or predicted from any LVLM's outputs. The paper's central claims—substantial factuality hallucinations and high sensitivity to semantic perturbations—are empirical findings obtained by running over 20 LVLMs on this held-out data, not consequences of the construction by definition. The VQAScore filtering and the manual audit of 500 images are quality-control steps; they do not define the evaluation outcome. The paper's own limitations section acknowledges that synthetic, template-based images may not capture full real-world complexity, which is a validity caveat about prompt-to-image fidelity rather than a circular step. The self-citations (Dysca [47] and M3oralBench [44]) appear only in related-work comparisons and in a minor aside about ImageNet label errors; they are not load-bearing for the derivation of the benchmark's results or for any uniqueness/necessity claim. No fitted parameter is relabeled as a prediction, no ansatz is imported via self-citation, and no known result is merely renamed as a new contribution. Accordingly, the derivation chain is not circular; the concerns raised by the reader are about external validity and label correctness under imperfect text-to-image rendering, not about the paper's claimed results reducing to their inputs by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the reliability of synthetic image generation and the inheritance of ground truth from prompts and external knowledge bases. The paper introduces no physical entities; its constructs are a dataset and a metric. The two hand-chosen filtering thresholds and the four domain assumptions above are the main things the reader pays for upstream.

free parameters (2)
  • VQAScore filtering threshold alpha = 0.85
    Chosen by hand as the lower bound on average image-text alignment score; images below it are sent for manual review. This threshold directly determines which generated images survive into the final benchmark and therefore shapes the dataset composition.
  • VQAScore inter-model variance threshold beta = 0.5
    Chosen by hand as the maximum allowed disagreement between the two foundation models used to compute alignment scores; higher variance triggers manual review. It is a design decision that affects the filtering stringency but is not derived from any objective function.
assumptions (4)
  • domain assumption Text-to-image generation followed by VQAScore filtering yields images that faithfully depict the intended entities, attributes, and relations.
    Invoked in Section 3.2 (Image Generation) and Appendix A.1. The entire ground-truth labeling scheme depends on this assumption; the paper only spot-checks it with a 500-image manual review.
  • domain assumption DBpedia abstracts are a reliable source of established world knowledge, and counterfactual claims created by modifying key information are plausible but false.
    Invoked in Section 3.2 (Prompt Design) for factuality samples. The ground truths for factual questions are taken from DBpedia, and the correctness of the counterfactual distractors is assumed rather than verified.
  • domain assumption LLM-as-a-Judge can reliably distinguish hallucinated from non-hallucinated responses in generative tasks.
    Used in Section 3.3 (Evaluation Metrics). The paper follows prior work [28, 39] but does not analyze judge bias or agreement rates on SHALE's specific item types.
  • domain assumption The Resistance Rate metric, which conditions on clean non-hallucinated responses, is a valid measure of perturbation resistance.
    Defined in Section 3.3. The metric drops pairs where the clean response is already hallucinated, which changes the evaluation population and is assumed to still reflect overall robustness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SHALE: A Scalable Benchmark for Fine-grained Hallucination Evaluation in LVLMs." pith.science (2026). https://pith.science/paper/JT2KSQQH

@misc{pith2026250809584,
  author       = {Pith},
  title        = {Pith review of: SHALE: A Scalable Benchmark for Fine-grained Hallucination Evaluation in LVLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JT2KSQQH}},
  note         = {Machine review of arXiv:2508.09584}
}
read the original abstract

Despite rapid advances, Large Vision-Language Models (LVLMs) still suffer from hallucinations, i.e., generating content inconsistent with input or established world knowledge, which correspond to faithfulness and factuality hallucinations, respectively. Prior studies primarily evaluate faithfulness hallucination at a rather coarse level (e.g., object-level) and lack fine-grained analysis. Additionally, existing benchmarks often rely on costly manual curation or reused public datasets, raising concerns about scalability and data leakage. To address these limitations, we propose an automated data construction pipeline that produces scalable, controllable, and diverse evaluation data. We also design a hierarchical hallucination induction framework with input perturbations to simulate realistic noisy scenarios. Integrating these designs, we construct SHALE, a Scalable HALlucination Evaluation benchmark designed to assess both faithfulness and factuality hallucinations via a fine-grained hallucination categorization scheme. SHALE comprises over 30K image-instruction pairs spanning 12 representative visual perception aspects for faithfulness and 6 knowledge domains for factuality, considering both clean and noisy scenarios. Extensive experiments on over 20 mainstream LVLMs reveal significant factuality hallucinations and high sensitivity to semantic perturbations.

Figures

Figures reproduced from arXiv: 2508.09584 by the authors.

Figure 1
Figure 1. Data distributions of SHALE. 1 Introduction With the rapid development of large foundation models, large vision-language models (LVLMs) have achieved significant advance￾ments, showing impressive generalization capabilities across var￾ious multimodal tasks such as image captioning and visual ques￾tion answering. However, LVLMs are persistently challenged by arXiv:2508.09584v2 [cs.CV] 14 Aug 2025 [PITH_FULL_IMAGE:fi… view at source ↗
Figure 2
Figure 2. Illustration of automated data construction in SHALE for assessing faithfulness and factuality hallucinations under [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Comparison of the top-4 LVLMs on fine-grained [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Evaluation prompt template for generative task. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Visualization of the correlation between hallucina [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Examples of SHALE evaluation across faithfulness and factuality hallucination. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 35 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv preprint arXiv:2404.14219 (2024)

    arXiv 2024

  2. [2]

    Alibaba DAMO Academy. 2024. Qwen-VL-Max. https://huggingface.co/spaces/ Qwen/Qwen-VL-Max

    work page 2024

  3. [3]

    Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyga- niak, and Zachary Ives. 2007. Dbpedia: A Nucleus for a Web of Open Data. In International Semantic Web Conference. Springer, 722–735

    work page 2007

  4. [4]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond.arXiv preprint arXiv:2308.12966 (2023)

    arXiv 2023

  5. [5]

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2024. Hallucination of Multimodal Large Language Models: A Survey. arXiv preprint arXiv:2404.18930 (2024)

    arXiv 2024

  6. [6]

    Assaf Ben-Kish, Moran Yanuka, Morris Alper, Raja Giryes, and Hadar Averbuch- Elor. 2024. Mitigating Open-Vocabulary Caption Hallucinations. In EMNLP. 22680–22698

    work page 2024

  7. [7]

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao

  8. [8]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al . [n. d.]. Are We on the Right Way for Evaluating Large Vision-Language Models?. In NeurIPS

    work page

Show all 52 references
  1. [9]

    Xiang Chen, Chenxi Wang, Yida Xue, Ningyu Zhang, Xiaoyan Yang, Qiang Li, Yue Shen, Lei Liang, Jinjie Gu, and Huajun Chen. 2024. Unified Hallucination Detection for Multimodal Large Language Models. In ACL. 3235–3252

    2024

  2. [10]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024. InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. In CVPR. 24185–24198

    2024

  3. [11]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2024. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. NeurIPS 36 (2024)

    2024

  4. [12]

    Google DeepMind. 2024. Gemini-2.0. https://blog.google/technology/google- deepmind/google-gemini-ai-update-december-2024/

    2024

  5. [13]

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Ima- geNet: A Large-Scale Hierarchical Image Database. In CVPR. Ieee, 248–255

    2009

  6. [14]

    Peng Ding, Jingyu Wu, Jun Kuang, Dan Ma, Xuezhi Cao, Xunliang Cai, Shi Chen, Jiajun Chen, and Shujian Huang. 2024. Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs. In ACM MM. 10707–10715

    2024

  7. [15]

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. 2024. Internlm- xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2...

    2024 arXiv

  8. [16]

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools. arXiv preprint arXiv:2406.12793 (2024)

    2024 arXiv

  9. [17]

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. 2024. HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models. In...

    2024

  10. [18]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al . 2025. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Transactions on Inf...

    2025

  11. [19]

    iFLYTEK Co., Ltd. 2024. iFLYTEK OCR. https://www.xfyun.cn/services/common- ocr

    2024

  12. [20]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hallucination in Natural Language Generation. Comput. Surveys 55, 12 (2023), 1–38

    2023

  13. [21]

    Jinqi Lai, Wensheng Gan, Jiayang Wu, Zhenlian Qi, and S Yu Philip. 2024. Large Language Models in Law: A Survey. AI Open (2024)

    2024

  14. [22]

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023. SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension. arXiv preprint arXiv:2307.16125 (2023)

    2023 arXiv

  15. [23]

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Joshua Adrian Cahyono, Jingkang Yang, Chunyuan Li, and Ziwei Liu. 2025. Otter: a Multi-Modal Model with in-Context Instruction Tuning. IEEE TPAMI (2025)

    2025

  16. [24]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating Object Hallucination in Large Vision-Language Models. In EMNLP. 292–305

    2023

  17. [25]

    Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. 2023. Large Language Models in Finance: A Survey. In Proceedings of ACM International Conference on AI in Finance. 374–382

    2023

  18. [26]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In ECCV. Springer, 740–755

    2014

  19. [27]

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. 2024. Evaluating Text-to-Visual Genera- tion with Image-to-Text Generation. In ECCV. Springer, 366–384

    2024

  20. [28]

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang

  21. [29]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruc- tion tuning. NeurIPS 36 (2023), 34892–34916

    2023

  22. [30]

    Jiazhen Liu, Yuhan Fu, Ruobing Xie, Runquan Xie, Xingwu Sun, Fengzong Lian, Zhanhui Kang, and Xirong Li. 2025. PhD: A ChatGPT-Prompted Visual halluci- nation Evaluation Dataset. In CVPR

    2025

  23. [31]

    Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung

  24. [32]

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards Deep Learning Models Resistant to Adversarial Attacks. In ICLR

    2018

  25. [33]

    Cheng Peng, Xi Yang, Aokun Chen, Kaleb E Smith, Nima PourNejatian, Anthony B Costa, Cheryl Martin, Mona G Flores, Ying Zhang, Tanja Magoc, et al. 2023. A Study of Generative Large Language Model for Medical Research and Healthcare. NPJ digital medicine 6, 1 (2023), 210

    2023

  26. [34]

    Negative Object Presence Evaluation (NOPE) to Measure Object Hallucina- tion in Vision-Language Models. In ALVR. 37–58

  27. [35]

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object Hallucination in Image Captioning. In EMNLP. 4035–4045

    2018

  28. [36]

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-Resolution Image Synthesis with Latent Diffusion Models. In CVPR. 10684–10695

    2022

  29. [37]

    Vipula Rawte, Amit Sheth, and Amitava Das. 2023. A Survey of Hallucination in Large Foundation Models. arXiv preprint arXiv:2309.05922 (2023)

    2023 arXiv

  30. [38]

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. 2023. EVA-CLIP: Improved Training Techniques for CLIP at Scale. arXiv preprint arXiv:2303.15389 (2023)

    2023 arXiv

  31. [39]

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang Yan Gui, Yu Xiong Wang, Yiming Yang, et al . 2024. Aligning Large Multimodal Models with Factually Augmented RLHF. In Findings ACL. Association for Computational Linguistics (ACL), ...

    2024

  32. [40]

    Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. 2023. JourneyDB: A Benchmark for Generative Image Understanding. NeurIPS 36 (2023), 49659–49678

    2023

  33. [41]

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. 2024. DeepSeek- VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding. arXiv preprint arXiv:2412.10302 (2024)

    2024 arXiv

  34. [42]

    Cheng Xu, Xiaofeng Hou, Jiacheng Liu, Chao Li, Tianhao Huang, Xiaozhi Zhu, Mo Niu, Lingyu Sun, Peng Tang, Tongqiao Xu, Kwang-Ting Cheng, and Minyi Guo. 2023. MMBench: Benchmarking End-to-End Multi-modal DNNs and Under- standing Their Hardware-Software Implications. InIEEE Inte...

    2023

  35. [43]

    Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. 2023. An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation. arXiv preprint arXiv:2311.07397 (2023)

    2023 arXiv

  36. [44]

    Bei Yan, Jie Zhang, Zhiyuan Chen, Shiguang Shan, and Xilin Chen. 2024. M3oralBench: A MultiModal Moral Benchmark for LVLMs. arXiv preprint arXiv:2412.20718 (2024)

    2024 arXiv

  37. [45]

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. 2024. MiniCPM-V: A GPT-4V Level MLLM on Your Phone. arXiv preprint arXiv:2408.01800 (2024). MM ’25, October 27–31, 2025, Dublin, Ireland Yan et al

    2024 arXiv

  38. [46]

    Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. 2024. LLVLM-eHub: A Compre- hensive Evaluation Benchmark for Large Vision-Language Models. IEEE TPAMI (2024)

    2024

  39. [47]

    Jie Zhang, Zhongqi Wang, Mengqi Lei, Zheng Yuan, Bei Yan, Shiguang Shan, and Xilin Chen. 2024. Dysca: A Dynamic and Scalable Benchmark for Evaluating Perception Ability of LVLMs. ICLR (2024)

    2024

  40. [48]

    Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. 2023. InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition. arXiv preprint arXiv:...

    2023 arXiv

  41. [49]

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. 2024. mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration. In CVPR. 13040–13051

    2024

  42. [52]

    yes" or

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al . 2025. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479 (2025). SHALE: A S...

    2025 arXiv

  43. [2023]

    arXiv preprint arXiv:2306.15195 (2023)

    Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic. arXiv preprint arXiv:2306.15195 (2023)

    2023 arXiv

  44. [2024]

    Mitigating Hallucination in Large Multi-Modal Models via Robust Instruc- tion Tuning. In ICLR

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.