Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

CCHall: A Novel Benchmark for Joint Cross-Lingual and Cross-Modal Hallucinations Detection in Large Language Models

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper constructs the first benchmark that makes vision-language models classify hallucination in a jointly cross-lingual and cross-modal setting, and shows that the best combination (GPT-4o plus multi-agent debate) still errs on about…

desk verdict A substantial resource built on a narrowed construct: CCHall's four-type taxonomy only captures object-existence mismatches in paired answers, while the paper's own error analysis reveals gold-label noise, so the headline claim about joint hallucinations needs major revision. read the letter →

arxiv 2505.19108 v1 pith:VVXTAARJ submitted 2025-05-25 cs.CL cs.AI

Yongheng Zhang , Xu Liu , Ruoxi Zhou , Qiguang Chen , Hao Fei , Wenpeng Lu , Libo Qin This is my paper
classification cs.CLcs.AI
keywords cross-lingualhallucinationcross-modaljointdetectionmultimodallargelanguagemodelsbenchmarkconstructionmachinetranslationvisualquestionansweringimagecaptioning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that hallucination testing for large AI systems has been split into two separate camps: errors that appear when a model answers in another language, and errors that appear when a model describes an image, while real deployments face both at once. To close that gap, it introduces CCHall, a benchmark built by taking visual question-answering and image-captioning items from four existing datasets, generating hallucinated answers with objects that are not in the pictures, and translating both correct and wrong answers into nine languages. Each test item gives a model a picture, a question, and two answers (one English, one in another language), and asks it to classify the pair as clean, cross-lingual only, cross-modal only, or jointly cross-lingual and cross-modal. Across six vision-language models, the strongest setup, GPT-4o with a multi-agent debate method, reaches 77.5% accuracy, while the weakest open model scores 34%, against a 25% random baseline. If correct, the benchmark establishes that the joint case is measurably harder than either failure alone, and it gives the field a concrete yardstick for that harder case.

What carries the argument

The machinery that carries the argument is a four-way hallucination taxonomy embedded in a paired-answer detection format. Each item pairs an image and a question with two pre-generated answers, one in English and one in a target language drawn from nine options spanning low-, medium-, and high-resource tiers, and the model must pick one of four labels: non-hallucination; cross-lingual non-cross-modal (only the translated answer names an object absent from the image); cross-modal non-cross-lingual (both answers share the same hallucinated object); or cross-modal and cross-lingual (both answers are hallucinated and disagree in meaning). The definitions rest entirely on object existence and cross-answer divergence, and the hallucinated content is manufactured by Gemini-1.5-Pro, prompted to swap in semantically similar but absent objects, with the multilingual half produced by Google Translate and checked by human back-translation scoring. This construction is what lets the paper claim to measure the joint scenario rather than either axis alone.

What would settle it

Take any benchmark item, swap the non-English answer for one written in the wrong language entirely (English when the instruction demanded Chinese) while naming only objects truly present in the image, and ask human raters whether it is an instance of cross-lingual hallucination as the paper defines it in Section 2.3, where 'the model does not respond in the target language as instructed' counts as hallucination. Under the Section 3.4 taxonomy such an item must be scored non-hallucination on the language axis, so if raters classify it as a cross-lingual hallucination, the benchmark's accuracy figures measure object-existence checking rather than the joint phenomenon the paper claims to test.

Watch

Extended reading notes

Core claim

The paper's central claim is that CCHall is the first detection benchmark for joint cross-lingual and cross-modal hallucination, and that current multimodal large language models still struggle with it. Its taxonomy classifies every answer pair into one of four types, with the joint type defined as the case where both the English answer and the translated answer contain hallucinated objects and the two answers convey different meanings. The evaluation numbers are meant to support this: GPT-4o with the HalluciMAD method reaches 77.5% accuracy and 78.8 Macro-F1, Gemini-1.5-Flash reaches 59.1%, and the open models land between 34.0% and 51.8%, with joint-type F1 running 3.4 points below the cross-lingual-only type and 10.9 points below the cross-modal-only type. The paper also claims that high-resource languages are much easier than low-resource ones, that high-resolution images and larger models matter, that hallucination rates jump past roughly 120 words of response, and that bilingual prompts plus external-tool verification improve detection.

Load-bearing premise

The load-bearing premise is that a cross-lingual hallucination can be fully represented as an object-existence error in the translated answer, and a joint hallucination as two differing hallucinated answers; if real joint failures also include instruction-following errors such as answering in the wrong language, the paper's own Figure 1c example, then the measured accuracies describe object-error detection rather than the claimed phenomenon.

Editorial extensions

If this is right

  • The joint case is the hard case: every model's F1 on cross-modal-plus-cross-lingual pairs ran 3.4 points below cross-lingual-only pairs and 10.9 points below cross-modal-only pairs.
  • There is real deployment headroom: the best setup, GPT-4o with HalluciMAD, still misclassifies 22.5% of items, so no tested model is reliable for joint hallucination detection.
  • A resource divide is systematic: detection accuracy falls from high-resource to low-resource languages (French, Spanish, and Portuguese best; Croatian, Welsh, and Swahili worst) across all models.
  • The task is genuinely multimodal: removing the image or lowering its resolution sharply cuts performance, so the benchmark cannot be solved from text alone.
  • Mitigation effects are stratified: simple prompting (chain-of-thought, self-reflection) helps the smaller models most, while advanced methods and external tools (object detection plus web search) help the stronger models most.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Implicit gap: the wrong-language, instruction-following failure that motivates the paper (its Figure 1c example) is unscorable in the benchmark, because none of the four classes encodes a violation of the target-language instruction; the headline accuracies should be read as object-existence verification rather than full joint-hallucination detection.
  • Testable extension: add a fifth class (or non-object errors such as wrong tense, wrong event, or outright wrong language) and rerun the same models; if accuracy collapses on that class, part of the 'still struggling' result reflects taxonomy difficulty rather than the joint phenomenon itself.
  • Deployment corollary: if the 120-word threshold and low-resource degradation hold generally, a practical rule for multilingual captioning pipelines would be to split long outputs, run an object detector on the image, and route low-resource answers through tool-based verification, the ingredients this paper's exploration found most effective.
  • Neighbouring use: the paired-answer format is also a ready-made test of translation faithfulness for image descriptions, since it already contains controlled pairs where only one language's object inventory differs from the other's.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces CCHall, a benchmark intended to evaluate joint cross-lingual and cross-modal hallucinations in multimodal large language models (MLLMs). The construction pipeline selects source data from GQA, AMBER, XM3600, and xFlickr&Co, generates hallucinated English answers with Gemini-1.5-Pro, translates them into nine languages in three resource tiers, and combines answers into four labeled categories defined by whether each language's answer contains object-existence errors and whether the two answers agree. The paper then evaluates six MLLMs with several mitigation strategies, reporting accuracies and macro-F1 scores per data source and hallucination type, and presents analyses of language-resource level, image resolution, and response-length effects.

Significance. If the construct-validity concerns are resolved, CCHall would be a valuable first resource for studying hallucinations under joint cross-lingual and cross-modal conditions: it is open-sourced, applies human rechecking with a scoring rubric and back-translation, covers nine languages across resource tiers, and reports per-type F1 scores rather than overall accuracy alone. The headline finding that even GPT-4o with HalluciMAD reaches only 77.5% accuracy is informative, and the dataset construction pipeline is described in enough detail to be reproduced. However, the contribution's central claim depends entirely on whether the four-type taxonomy and the paired-answer construction actually measure the joint phenomenon defined in Section 2.1; the current mismatch between those definitions and the operational labels is load-bearing, not cosmetic.

major comments (4)
  1. [§2.1 vs. §3.4 / Appendix A.3] The taxonomy does not cover the phenomena the paper uses to motivate the benchmark. Section 2.1 defines cross-lingual hallucination as any deviation in the target language, explicitly including failures to follow multilingual instructions and incorrect reasoning, and Figure 1(a) illustrates a translation error while Figure 1(c) shows a model answering in the wrong language. Yet Type 2 in Section 3.4 and Appendix A.3 defines cross-lingual, non-cross-modal hallucination solely as an English answer that is correct and a non-English answer containing hallucinated objects absent from the image. No label exists for instruction-following failures, wrong-language responses, or translation errors, so the benchmark cannot detect the very failure modes used to justify its existence, and the reported accuracies do not measure 'joint cross-lingual and cross-modal hallucination' as the paper defines it.
  2. [§3.3–3.4 and Figure 3] The 'joint' items are not produced by any model under simultaneous cross-lingual and cross-modal generation pressure. Section 3.3 describes translating existing English hallucinated answers, and Section 3.4 pairs an English hallucinated answer with a machine-translated version of a different English hallucinated answer. No MLLM is prompted in a target language with an image and then judged on whether its own response commits cross-lingual and cross-modal errors. Consequently, the benchmark measures a model's ability to classify pre-composed answer pairs, not its tendency to generate joint hallucinations, which weakens the claim that current MLLMs 'struggle with' joint cross-lingual and cross-modal hallucination in the sense illustrated in Figure 1(c).
  3. [Appendix B.4, Figures 14 and 15] The error-analysis examples contain gold labels that contradict the definitions in Section 3.4. In Figure 14, the English answer 'donuts' is consistent with the image while the French answer 'beignets' is not, which should be labeled Cross-lingual, non-cross-modal hallucination, but the displayed label is 'Non-hallucination.' In Figure 15, the English answer 'a macro shot of a gazelle in a field' is consistent with the image while the Swahili answer 'a big deer shot in the desert' is not, so the correct label should be cross-lingual, not cross-modal, yet the label says 'Cross-modal, cross-lingual hallucination.' These inconsistencies call into question the reliability of the gold labels in the released dataset and require scrutiny of the annotation process beyond the average review score.
  4. [§5.1–5.2, Table 1] The evaluation protocol is a four-way forced-choice classification of pre-supplied answer pairs, not a generation-based hallucination detection task with open-ended outputs. Because the prompt in Appendix A.4 hand-supplies both answers, models never demonstrate whether they would have generated the hallucination themselves; the reported accuracy is therefore a measure of label classification skill, which is a different capability from the one described in the introduction and conclusion. The central empirical claim should be rephrased to state that MLLMs struggle to classify the CCHall label types, unless the benchmark is explicitly repositioned as a detection/classification benchmark rather than a generation-pressure benchmark.
minor comments (7)
  1. [Abstract] There are missing spaces in 'jointCross-lingual andCross-modalHallucinations' in the abstract; please fix the formatting.
  2. [Throughout] The benchmark name is rendered inconsistently as 'CCHall' and 'CChall' (e.g., Section 3.4 uses 'CChall'); unify the spelling.
  3. [Appendix B.4, Figures 12–15] The captions say 'Error Analysis of Gemini on the AMBER/xFlickr&CO/GQA/XM3600' but the text in Section B.4 states the error cases were collected from GPT-4o's test-time performance; please make the model attribution consistent.
  4. [Appendix B.2] The text says 'five representative MLLMs' but then lists six: Gemini-1.5-Flash, Llama-3.2-11B-Vision-Instruct, Qwen2-VL-(2B, 7B)-Instruct, Pixtral-12B-2409, and InternVL2-8B; adjust the count.
  5. [§3.1] The sentence 'GQA enriches 337 object categories' attributes to GQA a property that Section 3.1 later assigns to AMBER; please clarify which dataset provides the 337 object categories.
  6. [Table 2] The caption states 'Bold represents the Best performance,' but the table as rendered has no visible bold entries; if bolding was lost in formatting, restore it or remove the sentence.
  7. [Appendix A.1, Figure 10] The output-format instruction says 'Each answer should be a comma-separated list of objects,' but the examples include full sentence captions and the GQA example gives a single noun; please align the instructions with the actual expected outputs.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark labels are constructed by explicit rules and model scores are independent measurements.

full rationale

CCHall is a benchmark-construction and evaluation paper rather than a derivation with fitted parameters. The four-way labels in Section 3.4 are produced by explicit combination rules applied to externally sourced image/question pairs (GQA, AMBER, XM3600, xFlickr&Co), with hallucinated objects inserted by Gemini-1.5-Pro and translations verified by human recheck; the models later scored (GPT-4o, Gemini-1.5-Flash, Llama-3.2-11B-Vision, Qwen2-VL-7B, Pixtral-12B, InternVL2) are not used to generate or define the gold labels against which they are measured. The central claims that current MLLMs struggle on the benchmark and that some mitigation strategies help are empirical findings from Table 1 and Figures 2, 6, 7, and 8, not consequences of the benchmark definitions. The paper contains several self-citations (Qin et al. 2023/2024/2025, Zhang et al. 2024a/2024b, Chen et al. 2025), but these appear in related-work and motivational contexts and are not load-bearing assumptions or uniqueness arguments. The mismatch between the broad definition of cross-lingual hallucination in Section 2.1 and the object-existence operationalization in Section 3.4 is a construct-validity concern, not circularity: no prediction is derived from its own input, no fitted value is renamed as a result, and no self-citation supports a forbidden alternative. The derivation chain is therefore self-contained, and no specific reduction can be exhibited.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim (a challenging joint benchmark) does not depend on any fitted numerical parameter in a derivation; the free parameters listed are hand-chosen construction choices. The load-bearing assumptions are the construct validity of the four-type taxonomy and the fidelity of machine translation plus back-translation. No new theoretical entities are postulated.

free parameters (5)
  • Sample size per source dataset = 900
    Hand-chosen count of entries sampled from each of the four filtered datasets, yielding 3600 entries; no principled or data-driven justification is given.
  • Human recheck pass threshold = 80 points
    Data points scored below 80 by reviewers are regenerated and retranslated; this threshold is arbitrary and affects final dataset composition.
  • Response length bins = 0-55, 55-120, 120+ words
    Post-hoc binning in Figure 6(c) used to claim a critical threshold at 120 words; the cutoffs are chosen after seeing the data.
  • Number of replaced object nouns = 1-2 per answer
    The hallucination generation prompt instructs Gemini-1.5-Pro to randomly replace 1-2 object nouns; this range is hand-chosen.
  • Language selection = HR, CY, SW, CS, NL, SV, FR, ES, PT
    Three languages with lowest reported translation error rates from each resource group following Guerreiro et al. (2023); this filters for easier languages and biases the multilingual difficulty profile.
assumptions (5)
  • domain assumption The four labeled hallucination types are exhaustive and mutually exclusive for all answer pairs.
    The benchmark labels every sample with exactly one of four types from §3.4. Real errors such as instruction-following failures, omissions, or non-object hallucinated entities are not represented, so the model's decision space is artificially constrained.
  • domain assumption Google Translate plus human back-translation verification preserves the object-existence hallucination status of the translated answers.
    Invoked in §3.3 and Appendix A.2. The error analyses in Figures 12-15 show cases where translation ambiguity (e.g., 'delw' meaning statue vs. image, 'beignets' vs. donuts) makes the gold label debatable.
  • domain assumption Gemini-1.5-Pro generates hallucinated answers that are natural, semantically similar, and correctly classified by the gold rules.
    The entire dataset of hallucinated responses is generated by one model using the prompt in Figure 10. If the generator produces detectable patterns or fails to keep replaced objects distinct, the benchmark's difficulty and validity change.
  • domain assumption The source datasets (GQA, AMBER, XM3600, xFlickr&Co) provide accurate ground-truth image-answer pairs.
    CCHall reuses image-question-answer triples from these external datasets; any errors propagate into the benchmark.
  • domain assumption Back-translation is a valid proxy for translation fidelity in this setting.
    The human recheck relies on back-translation (Miyabe and Yoshino 2015; Lee 2021) to verify meaning; no inter-annotator agreement or independent verification is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CCHall: A Novel Benchmark for Joint Cross-Lingual and Cross-Modal Hallucinations Detection in Large Language Models." pith.science (2026). https://pith.science/paper/VVXTAARJ

@misc{pith2026250519108,
  author       = {Pith},
  title        = {Pith review of: CCHall: A Novel Benchmark for Joint Cross-Lingual and Cross-Modal Hallucinations Detection in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VVXTAARJ}},
  note         = {Machine review of arXiv:2505.19108}
}
read the original abstract

Investigating hallucination issues in large language models (LLMs) within cross-lingual and cross-modal scenarios can greatly advance the large-scale deployment in real-world applications. Nevertheless, the current studies are limited to a single scenario, either cross-lingual or cross-modal, leaving a gap in the exploration of hallucinations in the joint cross-lingual and cross-modal scenarios. Motivated by this, we introduce a novel joint Cross-lingual and Cross-modal Hallucinations benchmark (CCHall) to fill this gap. Specifically, CCHall simultaneously incorporates both cross-lingual and cross-modal hallucination scenarios, which can be used to assess the cross-lingual and cross-modal capabilities of LLMs. Furthermore, we conduct a comprehensive evaluation on CCHall, exploring both mainstream open-source and closed-source LLMs. The experimental results highlight that current LLMs still struggle with CCHall. We hope CCHall can serve as a valuable resource to assess LLMs in joint cross-lingual and cross-modal scenarios.

Figures

Figures reproduced from arXiv: 2505.19108 by the authors.

Figure 1
Figure 1. (a) Cross-lingual hallucination: A cross [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) Fine-grained performance analysis of MLLMs F1-score for different hallucination types in [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The construction process of CCHall includes: (a) Raw Multi-modal Dataset Selection (§3.1), (b) Cross￾modal Hallucination Data Construction (§3.2), (c) Cross-lingual Hallucination Data Construction (§3.3), and (d) Cross-modal and Cross-lingual Hallucination Dataset (§3.4). 2 Background 2.1 Cross-lingual Hallucinations Cross-linguistic hallucination (Dale et al., 2023b) occurs when the LLMs generate an answer At in th… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Presentation of data in CCHall: (a) The diversity of multi-modal data as represented by CLIP-based (Radford et al., 2021) classification. (b) Display of part of the detailed topics in CCHall. scene descriptions, while AMBER covers 337 ob￾ject categories. These datasets…
Figure 5
Figure 5. Figure 5: Visualization of the semantic feature coverage [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Analysis of the underlying causes of cross-lingual and cross-modal hallucinations in MLLMs. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: A comparison between using only En￾glish (En) and using English with an additional Source Language (EN+SL) as context in Gemini-1.5-Flash. 35 43 51 59 67 xFlickrCO AMBER GQA XM3600 AVG Direct CoT SRO VDGD HalluciMAD UniHD [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: The UniHD, which utilizes external tools, is compared with other methods in Gemini-1.5-Flash. 5.4.2 Framework Adaptation Exploration To investigate how external tools and web search improve the mitigation of hallucinations, we inte￾grate the UniHD (Chen et al., 2024b) …
Figure 9
Figure 9. Figure 9: Analysis of the relationship between object presence and hallucination rate in the image captioning task. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Structure of the Prompt for Generating Hallucinated Sentences [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Example of the Prompt Used for Hallucination Detection and Classification [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Error Analysis of Gemini on the AMBER [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Error Analysis of Gemini on the xFlickr&CO [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: Error Analysis of Gemini on the GQA [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: Error Analysis of Gemini on the XM3600 [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Models

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Interleaving key video frames into step-by-step reasoning improves video question answering by 1.7 to 5.5 points over text-only chain-of-thought on a new self-built benchmark.

Reference graph

Works this paper leans on

25 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    Image: The inclusion of the image ensures that the model effectively recognizes and processes visual content, thereby facilitat- ing more accurate multi-modal alignment

    work page

  2. [2]

    In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 4: Student Research Workshop), pages 186–194

    Anhalten: Cross-lingual transfer for german token-level reference-free hallucination detection. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 4: Student Research Workshop), pages 186–194. Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositiona...

    work page 2019

  3. [3]

    Michelle A Lee

    Understanding sounds, missing the questions: The challenge of object hallucination in large audio- language models.arXiv preprint arXiv:2406.08402. Michelle A Lee. 2021.Examining Machine Trans- lation Systems and Translation Quality using the Back-Translation Method. Ph.D. thesis, University of Wisconsin–Stout. Sicong Leng, Yun Xing, Zesen Cheng, Yang Zho...

    arXiv 2021

  4. [4]

    Cross-modal, cross-lingual hallucination: If the English answer is hallucinatory and the answer in another language is inconsis- tent with it, meaning that the two halluci- natory answers differ, with each containing different objects that are either not actually present in the image or are incorrectly de- scribed, this is classified as a cross-modal, cro...

    work page

  5. [5]

    Abhimanyu Meta, Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection.Preprint, arXiv:2303.05499. Abhimanyu Meta, Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. AI Meta. 2024....

    arXiv 2024

  6. [6]

    Ashish V Thapliyal, Jordi Pont-Tuset, Xi Chen, and Radu Soricut

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context.arXiv preprint arXiv:2403.05530. Ashish V Thapliyal, Jordi Pont-Tuset, Xi Chen, and Radu Soricut. 2022. Crossmodal-3600: A massively multilingual multimodal evaluation dataset.arXiv preprint arXiv:2205.12522. Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Ha...

    arXiv 2022

  7. [9]

    Question and Real Answer: These provide the model with an accurate semantic con- text, enabling it to understand the require- ments for generating hallucinated data

    work page

  8. [10]

    As shown in Figure 10, these are the pro- vided examples used to construct A and B

    Examples: By providing examples, the prompt clarifies the generation target for the model, ensuring that the hallucinated sentences maintain logical consistency and follow the intended structure and content. As shown in Figure 10, these are the pro- vided examples used to construct A and B. Incorporating these elements helps the model better comprehend th...

    work page 2015

Show all 25 references
  1. [11]

    Verification of Hallucination Data: We check whether the generated hallucinated data meets our requirements. This involves ensuring that the hallucinated sentences ad- here to the expected structure, length, and logic, while also verifying that they con- tain the intended hall...

  2. [12]

    This step ensures that the translations remain faithful to the intended hallucination scenario, avoiding any misinterpretations or errors that arise from language differences

    Accuracy of Translated Data: We also check whether the translated data accu- rately reflects the original meaning and con- text. This step ensures that the translations remain faithful to the intended hallucination scenario, avoiding any misinterpretations or errors that arise...

  3. [13]

    Non-hallucination: If both the English an- swer and the answer in another language are correct, meaning that the objects men- tioned in the sentence match those in the image and are consistent with it, the answer is classified as non-hallucination

  4. [14]

    Cross-lingual, non-cross-modal hallucina- tion: If the English answer is correct but the answer in another language is hallucinated — meaning the English answer is consis- tent with the image while the objects in the non-English answer do not appear in the image — this is clas...

  5. [15]

    Cross-modal, non-cross-lingual halluci- nation: If the English answer is halluci- nated and the answer in another language is identical, meaning both answers contain the same hallucinated objects not present in the image, this is classified as cross-modal non-cross-lingual hal...

  6. [17]

    The model selects the correct option from the provided choices

    Intention Description: This section estab- lishes the model’s role, clearly instructing it that its task is to detect hallucinations based on the given image, question, and answers in two different languages, and classify the hallucinations. The model selects the correct optio...

  7. [18]

    Non- hallucination

    Hallucination Type Explanation: This part defines four types of hallucinations: “Non- hallucination”, “Cross-lingual, non-cross- tableshirtwomanstreetskyplatemanpeoplebuildingtreesGemini-1.5-Flash64.556.758.444.753.243.648.249.032.838.7Llama-3.2-11B-Vision61.872.259.460.555.46...

  8. [19]

    Task Description: This section provides con- crete task details, including the image, ques- tion, two answers in different languages, and the available options. This supplies the model with all necessary input data, ensuring that it has everything needed to perform the task ef...

  9. [20]

    table”, “shirt

    Output Format: The output format section specifies the required structure of the model’s response, designed for easy extraction using regular expressions. This standardization al- lows efficient evaluation of the model’s perfor- mance, facilitates accurate calculations, and en...

    2023

  10. [21]

    What animal is in the box?

    Input Adaptation: For VQA tasks, we com- bine the question and answer into a declarative statement for clarity, e.g., “What animal is in the box?” and “A bear” become “The animal in the box is a bear”. For Image Captioning, the input is the caption itself. Since CCHall is mult...

  11. [22]

    Claim Extraction and Query Generation: The adjusted framework begins by using Gemini-1.5-Flashto perform claim extraction, breaking complete sentences into sub-claims for detailed hallucination analysis. For each extracted claim,Gemini-1.5-Flashgenerates specific queries focus...

  12. [23]

    Tool Execution: The framework automati- cally invokesGrounding DINO(Liu et al.,

  13. [24]

    model to detect objects and return bounding box coordinates. Additionally, it utilizes theSerper Google Search API 1 to per- form internet-based fact verification by ana- lyzing search results, comparing sources, and validating the truthfulness of each claim

  14. [25]

    Option. Hallucination Type

    Integrated Evaluation: Evidence collected from the tools is integrated with the origi- nal image and its corresponding claims into a comprehensive prompt.Gemini-1.5-Flash then evaluates each claim’s hallucination like- lihood and provides reasoning, resulting in a final halluc...

    2022

  15. [2022]

    Bei Yan, Jie Zhang, Zheng Yuan, Shiguang Shan, and Xilin Chen

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural infor- mation processing systems, 35. Bei Yan, Jie Zhang, Zheng Yuan, Shiguang Shan, and Xilin Chen. 2024. Evaluating the quality of halluci- nation benchmarks for large vision-language mo...

    2024

  16. [2023]

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang

    Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models.arXiv preprint arXiv:2310.14566. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Y...

    arXiv

  17. [2024]

    Kenza Benkirane, Laura Gongas, Shahar Pelles, Naomi Fuchs, Joshua Darmon, Pontus Stenetorp, David Ife- oluwa Adelani, and Eduardo Sánchez

    Hallucination of multimodal large language models: A survey.arXiv preprint arXiv:2404.18930. Kenza Benkirane, Laura Gongas, Shahar Pelles, Naomi Fuchs, Joshua Darmon, Pontus Stenetorp, David Ife- oluwa Adelani, and Eduardo Sánchez. 2024. Ma- chine translation hallucination det...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.