{"id":"c9dcb91d-f89a-4b72-b1d6-c7512269811e","arxiv_id":"2505.19108","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CCHall is the first benchmark that jointly evaluates cross-lingual and cross-modal hallucination detection in multimodal LLMs, and current models perform poorly on it.","lead":"This paper introduces CCHall, a new benchmark that tests whether AI vision-language models can spot hallucinations when an image is described in two languages at once. It is worth reading because it exposes a gap in how AI hallucination is measured: most tests look at either multilingual or multimodal errors, not both together.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CCHall's four-way taxonomy equates cross-lingual hallucination solely with object-existence mismatches between English and non-English answers, excluding the translation errors and instruction-following failures that §2.1 itself defines as cross-lingual; the labels therefore do not measure the…","rationale":"The reader's weakest assumption already identified the construct-validity gap, and I agree with that reading; this is the most load-bearing issue because the benchmark's entire contribution rests on the claim that it measures 'joint cross-lingual and cross-modal hallucinations.' If the taxonomy only captures object-existence mismatches across two answer strings, then the low model scores do not establish that MLLMs struggle with the joint phenomenon the paper motivates. I considered other concerns: the lack of error bars or repeated runs weakens the precise ranking of methods but does not threaten the overall conclusion that all tested models are far from ceiling; missing human baselines makes the difficulty of the task hard to calibrate, but again the central claim of difficulty would likely survive a human run given the label ambiguity. The construct gap is more fundamental because it invalidates what the accuracy numbers mean. The paper has real strengths: the benchmark is open-sourced, the construction uses human rechecking with back-translation, the language selection spans low/medium/high resources, and the four categories do cover a coherent (if narrow) sub-task of multilingual object-hallucination detection. Those strengths justify a CONDITIONAL verdict rather than REJECT: the resource may be useful if reframed as a benchmark for cross-lingual object-existence hallucination detection, and the taxonomy and labels can be revised or clarified. My recommended verdict is unchanged from the reader's CONDITIONAL, with the condition that the authors either broaden the taxonomy to include translation and instruction-following errors or explicitly narrow the paper's claims to object-existence hallucinations.","tokens_in":23972,"tokens_out":12640,"duration_ms":107036,"concrete_test":"Independently re-annotate a random sample of 200 CCHall items (50 per type) using the paper's own §2.1 definition of cross-lingual hallucination (any deviation in the target language, including translation errors and instruction-following failures), with three annotators blinded to the original labels. If the original labels agree with the re-annotation on fewer than 80% of items, or if any 'Non-hallucination' item exhibits a translation error or instruction-following failure, then the benchmark's operationalization does not capture the stated construct and the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that CCHall detects joint cross-lingual and cross-modal hallucinations depends on the taxonomy in §3.4. Type 2 labels an item as 'Cross-lingual, non-cross-modal hallucination' only when the English answer is correct and the non-English answer contains hallucinated objects absent from the image; Type 4 requires both answers to be hallucinated and materially different. This is an object-existence criterion. Yet §2.1 defines cross-lingual hallucination as any deviation in the target language, explicitly including failures to follow multilingual instructions and incorrect reasoning, and Figure 1(a) illustrates a translation error ('stand' mistranslated as '站在' instead of '忍受') while Figure 1(c) shows a model answering in the wrong language. Neither of those failure modes can be expressed with the four labels, so the benchmark cannot detect the very phenomena used to motivate it. Furthermore, §3.3–3.4 construct the 'joint' items by pairing an English hallucinated answer with a machine-translated version of a different English hallucinated answer; no model generates under simultaneous cross-lingual and cross-modal pressure. The reported accuracies (e.g., GPT-4o at 77.5% with HalluciMAD) therefore do not support the statement that current MLLMs struggle with joint cross-lingual and cross-modal hallucination as the paper defines it. The error-analysis appendix also contains gold labels inconsistent with the §3.4 definitions: Figure 15 labels an item 'D. Cross-modal, cross-lingual hallucination' although the English answer 'A macro shot of a gazelle in a field' is consistent with the image, which contradicts the requirement that both answers be hallucinated. This indicates measurable label noise on top of the construct gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CCHall, a benchmark intended to evaluate joint cross-lingual and cross-modal hallucinations in multimodal large language models (MLLMs). The construction pipeline selects source data from GQA, AMBER, XM3600, and xFlickr&Co, generates hallucinated English answers with Gemini-1.5-Pro, translates them into nine languages in three resource tiers, and combines answers into four labeled categories defined by whether each language's answer contains object-existence errors and whether the two answers agree. The paper then evaluates six MLLMs with several mitigation strategies, reporting accuracies and macro-F1 scores per data source and hallucination type, and presents analyses of language-resource level, image resolution, and response-length effects.","tokens_in":24295,"tokens_out":4021,"duration_ms":35566,"significance":"If the construct-validity concerns are resolved, CCHall would be a valuable first resource for studying hallucinations under joint cross-lingual and cross-modal conditions: it is open-sourced, applies human rechecking with a scoring rubric and back-translation, covers nine languages across resource tiers, and reports per-type F1 scores rather than overall accuracy alone. The headline finding that even GPT-4o with HalluciMAD reaches only 77.5% accuracy is informative, and the dataset construction pipeline is described in enough detail to be reproduced. However, the contribution's central claim depends entirely on whether the four-type taxonomy and the paired-answer construction actually measure the joint phenomenon defined in Section 2.1; the current mismatch between those definitions and the operational labels is load-bearing, not cosmetic.","major_comments":[{"comment":"The taxonomy does not cover the phenomena the paper uses to motivate the benchmark. Section 2.1 defines cross-lingual hallucination as any deviation in the target language, explicitly including failures to follow multilingual instructions and incorrect reasoning, and Figure 1(a) illustrates a translation error while Figure 1(c) shows a model answering in the wrong language. Yet Type 2 in Section 3.4 and Appendix A.3 defines cross-lingual, non-cross-modal hallucination solely as an English answer that is correct and a non-English answer containing hallucinated objects absent from the image. No label exists for instruction-following failures, wrong-language responses, or translation errors, so the benchmark cannot detect the very failure modes used to justify its existence, and the reported accuracies do not measure 'joint cross-lingual and cross-modal hallucination' as the paper defines it.","section":"§2.1 vs. §3.4 / Appendix A.3"},{"comment":"The 'joint' items are not produced by any model under simultaneous cross-lingual and cross-modal generation pressure. Section 3.3 describes translating existing English hallucinated answers, and Section 3.4 pairs an English hallucinated answer with a machine-translated version of a different English hallucinated answer. No MLLM is prompted in a target language with an image and then judged on whether its own response commits cross-lingual and cross-modal errors. Consequently, the benchmark measures a model's ability to classify pre-composed answer pairs, not its tendency to generate joint hallucinations, which weakens the claim that current MLLMs 'struggle with' joint cross-lingual and cross-modal hallucination in the sense illustrated in Figure 1(c).","section":"§3.3–3.4 and Figure 3"},{"comment":"The error-analysis examples contain gold labels that contradict the definitions in Section 3.4. In Figure 14, the English answer 'donuts' is consistent with the image while the French answer 'beignets' is not, which should be labeled Cross-lingual, non-cross-modal hallucination, but the displayed label is 'Non-hallucination.' In Figure 15, the English answer 'a macro shot of a gazelle in a field' is consistent with the image while the Swahili answer 'a big deer shot in the desert' is not, so the correct label should be cross-lingual, not cross-modal, yet the label says 'Cross-modal, cross-lingual hallucination.' These inconsistencies call into question the reliability of the gold labels in the released dataset and require scrutiny of the annotation process beyond the average review score.","section":"Appendix B.4, Figures 14 and 15"},{"comment":"The evaluation protocol is a four-way forced-choice classification of pre-supplied answer pairs, not a generation-based hallucination detection task with open-ended outputs. Because the prompt in Appendix A.4 hand-supplies both answers, models never demonstrate whether they would have generated the hallucination themselves; the reported accuracy is therefore a measure of label classification skill, which is a different capability from the one described in the introduction and conclusion. The central empirical claim should be rephrased to state that MLLMs struggle to classify the CCHall label types, unless the benchmark is explicitly repositioned as a detection/classification benchmark rather than a generation-pressure benchmark.","section":"§5.1–5.2, Table 1"}],"minor_comments":[{"comment":"There are missing spaces in 'jointCross-lingual andCross-modalHallucinations' in the abstract; please fix the formatting.","section":"Abstract"},{"comment":"The benchmark name is rendered inconsistently as 'CCHall' and 'CChall' (e.g., Section 3.4 uses 'CChall'); unify the spelling.","section":"Throughout"},{"comment":"The captions say 'Error Analysis of Gemini on the AMBER/xFlickr&CO/GQA/XM3600' but the text in Section B.4 states the error cases were collected from GPT-4o's test-time performance; please make the model attribution consistent.","section":"Appendix B.4, Figures 12–15"},{"comment":"The text says 'five representative MLLMs' but then lists six: Gemini-1.5-Flash, Llama-3.2-11B-Vision-Instruct, Qwen2-VL-(2B, 7B)-Instruct, Pixtral-12B-2409, and InternVL2-8B; adjust the count.","section":"Appendix B.2"},{"comment":"The sentence 'GQA enriches 337 object categories' attributes to GQA a property that Section 3.1 later assigns to AMBER; please clarify which dataset provides the 337 object categories.","section":"§3.1"},{"comment":"The caption states 'Bold represents the Best performance,' but the table as rendered has no visible bold entries; if bolding was lost in formatting, restore it or remove the sentence.","section":"Table 2"},{"comment":"The output-format instruction says 'Each answer should be a comma-separated list of objects,' but the examples include full sentence captions and the GQA example gives a single noun; please align the instructions with the actual expected outputs.","section":"Appendix A.1, Figure 10"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid benchmark-construction effort with open data and a broad model evaluation, but the construct-validity issue is central and will require substantial reworking of the definitions or the claims. The label inconsistencies in the appendix error analysis are concerning and should be checked dataset-wide before the benchmark is used by the community. I would not recommend rejection if the authors can either broaden the taxonomy to include instruction-following and translation-error categories, re-annotate the affected labels, and rephrase the generation-based claims, or explicitly reposition CCHall as a classification benchmark of paired object-existence errors."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the CCHall paper. The useful part is real: it's the first benchmark that combines cross-lingual and cross-modal hallucination detection, built from GQA, AMBER, XM3600, and xFlickr&Co, with 3,600 items, nine target languages, a four-type taxonomy, and a broad evaluation across six MLLMs plus mitigation strategies. That's a lot of work, and the resource could fill a genuine gap.\n\nBut the benchmark doesn't measure what the paper says it measures. The introduction defines cross-lingual hallucination as the model failing to follow multilingual instructions or giving incorrect reasoning; the motivating examples are a mistranslation of 'stand' and a model answering in the wrong language. The detection task, though, is a four-way classification of two pre-written answers, one English and one machine-translated. Cross-lingual hallucination is operationalized solely as object-existence errors in the non-English answer. Translation errors and wrong-language responses can't be expressed with the four labels. So the benchmark is really measuring object hallucination in a multilingual answer pair, not the joint phenomenon in the abstract.\n\nThe construct gap is compounded by label noise in the paper's own appendix. Figure 15 labels an item 'D. Cross-modal, cross-lingual hallucination' while the English answer matches the image and both answers are consistent; Figures 12–14 show the model's reasoning correctly identifying the type, with gold labels that contradict it. If the gold labels in the error analysis are unreliable, it's hard to trust the headline numbers. The best accuracy of 77.5% for GPT-4o with HalluciMAD becomes uninterpretable.\n\nOther soft spots are more minor: no human baseline, no variance across repeated runs, and the 'joint' items are constructed by pairing two independently generated English hallucinated answers and translating one, so no model actually generates under simultaneous cross-lingual and cross-modal pressure. The analyses of language resources, image resolution, and response length are sensible and could be useful once the taxonomy is fixed.\n\nOverall: the paper deserves a serious referee, because the resource is substantial and the gap is real. The central claim needs reframing to what the data actually encodes, the gold labels need a careful recheck, and the authors should add human performance. As it stands I wouldn't cite it, but a revised version could be worth citing.","headline":"A substantial resource built on a narrowed construct: CCHall's four-type taxonomy only captures object-existence mismatches in paired answers, while the paper's own error analysis reveals gold-label noise, so the headline claim about joint hallucinations needs major revision.","tokens_in":24918,"tokens_out":2628,"would_cite":false,"duration_ms":24368,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper constructs the first benchmark that makes vision-language models classify hallucination in a jointly cross-lingual and cross-modal setting, and shows that the best combination (GPT-4o plus multi-agent debate) still errs on about…","keywords":["cross-lingual hallucination","cross-modal hallucination","joint hallucination detection","multimodal large language models","benchmark construction","machine translation","visual question answering","image captioning"],"falsifier":"Take any benchmark item, swap the non-English answer for one written in the wrong language entirely (English when the instruction demanded Chinese) while naming only objects truly present in the image, and ask human raters whether it is an instance of cross-lingual hallucination as the paper defines it in Section 2.3, where 'the model does not respond in the target language as instructed' counts as hallucination. Under the Section 3.4 taxonomy such an item must be scored non-hallucination on the language axis, so if raters classify it as a cross-lingual hallucination, the benchmark's accuracy figures measure object-existence checking rather than the joint phenomenon the paper claims to test.","tokens_in":23760,"feed_emoji":"🖼️","tokens_out":16630,"duration_ms":124922,"temperature":0.7,"pith_summary":"This paper argues that hallucination testing for large AI systems has been split into two separate camps: errors that appear when a model answers in another language, and errors that appear when a model describes an image, while real deployments face both at once. To close that gap, it introduces CCHall, a benchmark built by taking visual question-answering and image-captioning items from four existing datasets, generating hallucinated answers with objects that are not in the pictures, and translating both correct and wrong answers into nine languages. Each test item gives a model a picture, a question, and two answers (one English, one in another language), and asks it to classify the pair as clean, cross-lingual only, cross-modal only, or jointly cross-lingual and cross-modal. Across six vision-language models, the strongest setup, GPT-4o with a multi-agent debate method, reaches 77.5% accuracy, while the weakest open model scores 34%, against a 25% random baseline. If correct, the benchmark establishes that the joint case is measurably harder than either failure alone, and it gives the field a concrete yardstick for that harder case.","feed_headline":"77.5% is the top score on the new bilingual hallucination test","feed_subtitle":"Both languages plus one image multiplies failure modes; even the best model misses one in four.","key_machinery":"The machinery that carries the argument is a four-way hallucination taxonomy embedded in a paired-answer detection format. Each item pairs an image and a question with two pre-generated answers, one in English and one in a target language drawn from nine options spanning low-, medium-, and high-resource tiers, and the model must pick one of four labels: non-hallucination; cross-lingual non-cross-modal (only the translated answer names an object absent from the image); cross-modal non-cross-lingual (both answers share the same hallucinated object); or cross-modal and cross-lingual (both answers are hallucinated and disagree in meaning). The definitions rest entirely on object existence and cross-answer divergence, and the hallucinated content is manufactured by Gemini-1.5-Pro, prompted to swap in semantically similar but absent objects, with the multilingual half produced by Google Translate and checked by human back-translation scoring. This construction is what lets the paper claim to measure the joint scenario rather than either axis alone.","core_discovery":"The paper's central claim is that CCHall is the first detection benchmark for joint cross-lingual and cross-modal hallucination, and that current multimodal large language models still struggle with it. Its taxonomy classifies every answer pair into one of four types, with the joint type defined as the case where both the English answer and the translated answer contain hallucinated objects and the two answers convey different meanings. The evaluation numbers are meant to support this: GPT-4o with the HalluciMAD method reaches 77.5% accuracy and 78.8 Macro-F1, Gemini-1.5-Flash reaches 59.1%, and the open models land between 34.0% and 51.8%, with joint-type F1 running 3.4 points below the cross-lingual-only type and 10.9 points below the cross-modal-only type. The paper also claims that high-resource languages are much easier than low-resource ones, that high-resolution images and larger models matter, that hallucination rates jump past roughly 120 words of response, and that bilingual prompts plus external-tool verification improve detection.","pith_inferences":["Implicit gap: the wrong-language, instruction-following failure that motivates the paper (its Figure 1c example) is unscorable in the benchmark, because none of the four classes encodes a violation of the target-language instruction; the headline accuracies should be read as object-existence verification rather than full joint-hallucination detection.","Testable extension: add a fifth class (or non-object errors such as wrong tense, wrong event, or outright wrong language) and rerun the same models; if accuracy collapses on that class, part of the 'still struggling' result reflects taxonomy difficulty rather than the joint phenomenon itself.","Deployment corollary: if the 120-word threshold and low-resource degradation hold generally, a practical rule for multilingual captioning pipelines would be to split long outputs, run an object detector on the image, and route low-resource answers through tool-based verification, the ingredients this paper's exploration found most effective.","Neighbouring use: the paired-answer format is also a ready-made test of translation faithfulness for image descriptions, since it already contains controlled pairs where only one language's object inventory differs from the other's."],"forward_implications":["The joint case is the hard case: every model's F1 on cross-modal-plus-cross-lingual pairs ran 3.4 points below cross-lingual-only pairs and 10.9 points below cross-modal-only pairs.","There is real deployment headroom: the best setup, GPT-4o with HalluciMAD, still misclassifies 22.5% of items, so no tested model is reliable for joint hallucination detection.","A resource divide is systematic: detection accuracy falls from high-resource to low-resource languages (French, Spanish, and Portuguese best; Croatian, Welsh, and Swahili worst) across all models.","The task is genuinely multimodal: removing the image or lowering its resolution sharply cuts performance, so the benchmark cannot be solved from text alone.","Mitigation effects are stratified: simple prompting (chain-of-thought, self-reflection) helps the smaller models most, while advanced methods and external tools (object detection plus web search) help the stronger models most."],"supporting_citations":[{"why":"Supplies the GQA visual-question-answering data whose object-existence questions become part of the benchmark.","marker":"(Hudson and Manning, 2019)"},{"why":"Supplies the AMBER VQA data, contributing its object categories to the object-existence criterion.","marker":"(Wang et al., 2023)"},{"why":"Supplies the XM3600 multilingual captioning data used for the image-captioning task.","marker":"(Thapliyal et al., 2022)"},{"why":"Supplies the xFlickr&Co everyday-scene captioning data used for the image-captioning task.","marker":"(Bugliarello et al., 2022)"},{"why":"Provides the foundational definition of object hallucination in image captioning that the cross-modal criterion is built on.","marker":"(Rohrbach et al., 2018)"},{"why":"Source of the cross-lingual hallucination definition (answer in the target language deviating from the correct answer) used for the language dimension.","marker":"(Dale et al., 2023b)"},{"why":"Supplies the low/medium/high-resource language grouping used to select the nine target languages and structure the resource-level analysis.","marker":"(Guerreiro et al., 2023)"},{"why":"Gemini-1.5-Pro generates the hallucinated answers by inserting absent objects; Gemini-1.5-Flash is also one of the evaluated models.","marker":"(Team et al., 2024)"},{"why":"Supplies HalluciMAD, the multi-agent debate method that yields the best reported accuracy of 77.5%.","marker":"(Lin et al., 2024)"},{"why":"Supplies the UniHD external-tool framework whose adaptation, using object detection and web search, outperforms all prompting methods.","marker":"(Chen et al., 2024b)"}],"fun_headline_variants":["New benchmark exposes LLM hallucinations across language and image","Bilingual plus image: LLM hallucination test, top score 77.5%","First joint cross-lingual cross-modal hallucination benchmark for LLMs","LLMs still hallucinate when text and images cross languages","Top models fail 22.5% on joint bilingual-image hallucination test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a cross-lingual hallucination can be fully represented as an object-existence error in the translated answer, and a joint hallucination as two differing hallucinated answers; if real joint failures also include instruction-following errors such as answering in the wrong language, the paper's own Figure 1c example, then the measured accuracies describe object-error detection rather than the claimed phenomenon.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark exposes LLM hallucinations across language and image","Bilingual plus image: LLM hallucination test, top score 77.5%","First joint cross-lingual cross-modal hallucination benchmark for LLMs","LLMs still hallucinate when text and images cross languages","Top models fail 22.5% on joint bilingual-image hallucination test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000546,"raw_usage":{"total_tokens":2620,"prompt_tokens":962,"completion_tokens":1658,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":1564}},"tokens_in":578,"tokens_out":1658,"duration_ms":13458,"temperature":1.0,"reasoning_tokens":1564,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:20:19.431794+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any benchmark item, swap the non-English answer for one written in the wrong language entirely (English when the instruction demanded Chinese) while naming only objects truly present in the image, and ask human raters whether it is an instance of cross-lingual hallucination as the paper defines it in Section 2.3, where 'the model does not respond in the target language as instructed' counts as hallucination. Under the Section 3.4 taxonomy such an item must be scored non-hallucination on the language axis, so if raters classify it as a cross-lingual hallucination, the benchmark's accuracy figures measure object-existence checking rather than the joint phenomenon the paper claims to test.","supporting_citations":[],"review_version":1}