{"id":"bcbb51c2-a60b-4f16-97e0-a1ba5eb575f4","arxiv_id":"2505.03030","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A retrieve-then-prompt pipeline with automatic prompt optimization ranked first on average across 14 languages in the Mu-SHROOM shared task on span-level hallucination detection.","lead":"UCSC's entry to a multilingual hallucination-detection challenge retrieves web context, asks a language model to flag unsupported content, and then maps the flags to exact spans in the model's answer. The system ranked first on average across 14 languages, and the paper argues that simple prompting with good context can beat average human annotators.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Overstated 'better-than-human' claim rests on a mismatched human baseline that the authors themselves suspect followed different labeling guidelines.","rationale":"I read the paper as a shared-task system description whose primary empirical claim is the official #1 average ranking; that claim is supported by the official ranking tables and is not seriously threatened by my concern. The more interesting and more generalizable claim is the 'better-than-human' insight, which the paper uses to motivate the approach. That claim is undercut by an unmatched human baseline: the authors' internal annotators scored against official labels, while the authors themselves acknowledge that their labeling may not follow the official guidelines. This is not a question of external consensus but of internal validity of the comparison. The reader's conditional verdict already accounts for this weakness; my pass does not identify a new reason to reject the paper. I therefore leave the verdict unchanged. The proposed test using official per-annotator labels would settle the matter without requiring new annotation effort beyond what the task dataset already provides.","tokens_in":11845,"tokens_out":6593,"duration_ms":67279,"concrete_test":"Obtain the per-annotator span labels for the Mu-SHROOM English validation set (the task's human annotations are available per annotator before aggregation). Compute each individual official annotator's IoU and Spearman Corr against the aggregated official hard/soft labels, exactly as the system is scored. If the average or best official annotator's IoU equals or exceeds the system's 0.57, then the 'better-than-human' claim is refuted; if the system beats the best official annotator, the claim is supported. As a stricter check, have the authors' annotators relabel the same sample after reading the official Mu-SHROOM annotation guidelines and repeat the comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's principal secondary claim—'a simple prompting-based approach can reliably detect hallucinations with better-than-human accuracy' (Section 2, echoed in the conclusion)—is not supported by the evidence in Section 4.2. The human baseline is computed from the authors' internal annotations scored against the official aggregated Mu-SHROOM gold labels, yielding an average IoU of 0.43 and 0.48 for the best annotator. The system's IoU of 0.57 is computed against the same gold labels. But the authors explicitly state that their low human overlap may be due to 'potential differences in labeling guidelines.' If the internal annotators were not applying the official labeling protocol, their low scores measure disagreement with a different standard, not human accuracy on the task. The comparison is therefore apples-to-oranges, and the 'better-than-human' conclusion does not follow from the reported numbers. Notably, the paper does not report the per-annotator agreement of the official Mu-SHROOM annotators against the aggregated gold labels, which would be the appropriate human baseline. The ranking claim (#1 average position) is unaffected, but the headline insight is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes the UCSC submission to SemEval 2025 Task 3 (Mu-SHROOM), a multilingual span-level hallucination detection benchmark. The proposed system is a three-stage pipeline: (i) context retrieval from Perplexity Sonar Pro, (ii) hallucinated content detection via direct text extraction, knowledge-graph verification, or minimal-cost revision, and (iii) mapping the detected errors back to character-level spans. The pipeline is enhanced by MiPROv2 prompt optimization and by combining multiple systems as pseudo-annotators to generate soft labels. The authors report top-two rankings in 11 of 14 languages on IoU and 10 of 14 on Corr, and claim the highest average position across all languages. They also claim that, when paired with good context, a simple prompting-based approach can detect hallucinations with better-than-human accuracy, supported by a comparison of their internal annotations against official gold labels.","tokens_in":12043,"tokens_out":5641,"duration_ms":54311,"significance":"If the central ranking claim holds, the paper demonstrates that strong multilingual span-level hallucination detection can be achieved with off-the-shelf LLMs, retrieved context, and optimized prompts, without task-specific fine-tuning. The paper's strengths include honest reporting of a failed knowledge-graph variant, a transparent description of the prompt-optimization protocol, and public release of code and results. The 'better-than-human' claim, however, is not established by the presented evidence, and the primary ranking claim is not fully verifiable from the manuscript as written.","major_comments":[{"comment":"The claim that the prompting approach achieves 'better-than-human accuracy' is not supported by the reported comparison. The human baseline is computed from the authors' own annotations scored against the official aggregated Mu-SHROOM labels, while the system is also scored against those same labels. The authors themselves state that 'potential differences in labeling guidelines' may explain the low human overlap. If the internal annotations follow a different protocol, the human IoU/Corr values are not a valid measure of human performance on the official task, and the comparison is apples-to-oranges. The appropriate baseline would be the agreement of the official Mu-SHROOM annotators against their aggregated gold labels, which is not reported. Without that, the 'better-than-human' conclusion does not follow; please either provide the official per-annotator agreement or remove/soften the claim.","section":"Section 2 and Section 4.2"},{"comment":"The headline claim of 'ranking #1 in average position across all languages' is not verifiable from the manuscript. Table 2 reports only the authors' own per-language ranks and their averages (2.6 IoU, 2.4 Corr), but it does not include the full leaderboard or a citation to the official ranking. Since this is the central primary claim, please include the official overall ranking table (or a precise statement of the official metric and the relevant official source) so that the claim can be independently checked.","section":"Abstract and Table 2"},{"comment":"The key comparisons in the paper are presented without any variance estimates or significance tests. For example, in Table 4 the validation IoU improves from 0.44 to 0.57 under prompt optimization, but the test IoU improves only from 0.55 to 0.60, and several differences in Table 1 are on the order of 0.01–0.03. Given that the paper draws conclusions such as 'prompt optimization improves system performance' and 'system combination improves Corr,' the absence of uncertainty quantification makes it difficult to assess which differences are reliable. Please add a limitation statement or, where feasible, confidence intervals or significance tests.","section":"Tables 1, 3, and 4"}],"minor_comments":[{"comment":"The statement 'As our system development was focused only on English' is contradicted by the per-language prompt optimization and context-retrieval experiments described in Sections 3.5 and 4.1 and reported in Tables 1 and 8; please clarify that the pipeline design is language-agnostic and that language-specific prompt optimization was performed automatically on validation data.","section":"Section 5.1"},{"comment":"Please specify which submission the per-language ranks correspond to (for example, the best-performing single system or the system combination), since Table 1 reports multiple systems per language.","section":"Table 2"},{"comment":"The assumption that the retrieved context contains all information required to verify the answer is load-bearing for the pipeline but is only stated as an assumption; please add it to the limitations discussion in Section 5.3.","section":"Section 3.1"},{"comment":"The table would be more informative if it included a no-context baseline (for example, text extraction without any retrieved context) to directly show the size of the context-retrieval benefit.","section":"Appendix C, Table 6"},{"comment":"There is a typo: 'use used exactly the same methods' should read 'we used exactly the same methods.'","section":"Section 3.5"},{"comment":"In the JSON example, 'co nfidence_ score' contains an erroneous space; please fix the formatting.","section":"Appendix D.1"},{"comment":"There is a typo: 'Corr or 0.34' should read 'Corr of 0.34.'","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid system description and the official shared-task scores reported in Tables 1 and 2 are likely reliable. However, the 'better-than-human' claim is a prominent secondary claim that is not supported by the evidence, and the primary ranking claim lacks a verifiable source. Both issues are fixable within the scope of the manuscript. I also note that the paper would benefit from a clearer statement of which official results support the average-position claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: this is a competent shared-task system description whose main ranking claim survives scrutiny, but its headline 'better-than-human' claim does not. The paper earns credit for a clean three-stage pipeline—context retrieval, hallucinated content detection, span mapping—with honest ablations and released code. The new thing is the integration and systematic comparison across 14 languages, not any single novel component. The official IoU/Corr tables support the #1 average position, and the system-combination and prompt-optimization results are internally consistent.\n\nThe soft spots are real but localized. The biggest is Section 4.2's human comparison, which is apples-to-oranges. The authors' own annotators scored against the official aggregated labels, and the authors themselves admit possible differences in labeling guidelines. That internal agreement number is not a human baseline for this task; it measures disagreement with a different standard. So 'better-than-human accuracy' in Section 2 and the conclusion does not follow from the reported numbers. The ranking claim is unaffected. Second, there are no error bars or significance tests, so per-language differences are just point estimates. Third, the pipeline depends on paid external APIs and the optimized prompts are not fully specified in the paper, though code is public. Minor: the knowledge-graph branch fails and the authors say so, which is honest; the system-combination IoU/Corr trade-off is also reported plainly.\n\nOverall, this is one of the better shared-task system reports. The authors do not oversell most results; only the human-comparison line overshoots. I would accept it for peer review with a required revision: either drop the 'better-than-human' wording or replace it with a comparison to the official Mu-SHROOM inter-annotator agreement. Worth a reading group if someone is working on hallucination detection or multilingual fact-checking.","headline":"Solid shared-task system report; the ranking claim holds up, but the 'better-than-human' claim is not supported by the reported human comparison.","tokens_in":12578,"tokens_out":1746,"would_cite":false,"duration_ms":19891,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A retrieval, prompting, and span-mapping pipeline achieves the best average hallucination-detection rank across all fourteen Mu-SHROOM languages.","keywords":["hallucination detection","span labeling","large language models","retrieval-augmented fact-checking","prompt optimization","multilingual NLP","Mu-SHROOM","SemEval 2025 Task 3"],"falsifier":"Re-run the manual annotation of the Mu-SHROOM English validation set using the exact official guidelines and the same Perplexity Sonar Pro context the system uses; if the best human annotator's IoU against the official hard labels reaches or exceeds the system's 0.57, the claim that a simple prompting approach beats human annotators would be refuted.","tokens_in":11642,"feed_emoji":"🎯","tokens_out":9594,"duration_ms":78832,"temperature":0.7,"pith_summary":"This paper argues that span-level hallucination detection in LLM question answering does not require specialized training. A pipeline that retrieves context from a search API, prompts an off-the-shelf LLM to flag unverifiable text, and maps the flagged text back to character spans achieved the best average rank across all fourteen languages of the Mu-SHROOM shared task. The authors further report that adding retrieved context raised English IoU by 27% over a no-context baseline, that simple text extraction beat a knowledge-graph-based verifier, and that automatic prompt optimization gave additional gains. Their central practical claim is that with good context, a simple prompting approach can detect hallucinations with better-than-human accuracy.","feed_headline":"Retrieval-backed prompting wins 14-language hallucination detection","feed_subtitle":"A retrieval-plus-prompting pipeline ranks first on average across 14 Mu-SHROOM languages with off-the-shelf LLMs.","key_machinery":"The load-bearing machinery is the three-stage pipeline. Stage one retrieves context by querying Perplexity Sonar Pro with either the question itself or claims extracted from the answer, assuming the returned content is sufficient to verify the answer. Stage two identifies unverifiable content in three ways: direct text extraction, in which the LLM finds spans unsupported by the context; knowledge-graph verification, in which the context is parsed into a graph and each answer fact is checked against it; and minimal-cost revision, in which a reasoning model rewrites the answer with the fewest changes and the edits are treated as hallucinations. Stage three maps those errors to character spans using substring matching, fact-to-span prompting, or edit distance. On top of this, prompt optimization with MiPROv2—a Bayesian search over instructions and few-shot demonstrations—tunes the detection prompt; the paper's key empirical finding is that the simplest path, direct text extraction with substring matching plus retrieved context, is the most reliable.","core_discovery":"The central discovery, in the paper's own terms, is that a three-stage pipeline—context retrieval via Perplexity Sonar Pro, hallucinated-content detection by prompting GPT-4o, GPT-4o-mini, o1, or DeepSeek-R1, and span mapping via substring match, fact-to-span mapping, or edit distance—ranks in the top two on IoU in 11 of 14 languages and on correlation in 10 of 14, and holds the best average position overall. Section 2 states that 'when paired with good context, a simple prompting-based approach can reliably detect hallucinations with better-than-human accuracy.' The evidence for that claim comes from internal annotation: the best individual human annotator reached IoU 0.48 and correlation 0.48 on the English validation set, while the best system reached 0.57 and 0.55. Prompt optimization with MiPROv2 improved the detector further, and combining five systems as pseudo-annotators raised soft-label correlation by about 5% on average at a typical 5% IoU cost.","pith_inferences":["A testable extension would be corrupting or truncating the retrieved context and measuring the IoU drop; this would isolate how much of the pipeline's edge comes from retrieval quality versus prompt strength.","The better-than-human comparison implicitly depends on the exact context and labeling guidelines given to human annotators; under a different annotation protocol the human-system gap could shrink or reverse.","Minimal-cost revision produces a corrected answer as a byproduct, so the detector could double as a lightweight text-editing tool for LLM outputs—an application the paper does not itself develop."],"forward_implications":["Off-the-shelf LLMs plus search-engine context are sufficient for competitive span-level hallucination detection; no fine-tuning or task-specific training is needed.","Retrieving context from the question is the single largest performance lever, lifting English IoU from 0.44 to 0.56 compared with no context.","Translating non-English questions to English before retrieval slightly hurts performance, so target-language retrieval is preferable for multilingual detection.","Combining diverse systems as pseudo-annotators raises soft-label Spearman correlation by up to 12% in some languages, at a typical 5% IoU cost.","Bayesian prompt optimization improves detection, though no single optimization target (IoU, Corr, or MaxIoU) dominates both validation and test sets."],"supporting_citations":[{"why":"Defines the Mu-SHROOM task, its 14 languages, and the IoU and Spearman evaluation the system is optimized for.","marker":"Vázquez et al., 2025"},{"why":"Introduces retrieval-augmented generation, the rationale for grounding detection in retrieved context.","marker":"Lewis et al., 2020"},{"why":"Supplies MiPROv2, the Bayesian prompt-optimization framework that improves the detection step.","marker":"Opsahl-Ong et al., 2024"},{"why":"Documents GPT-4o, the main LLM used for context-based text extraction and span identification.","marker":"OpenAI et al., 2024"},{"why":"Documents DeepSeek-R1, the best single system in the multilingual comparison on both IoU and correlation.","marker":"DeepSeek-AI et al., 2025"},{"why":"Provides DSPy, the prompt-programming tool used to perform the prompt optimization.","marker":"Khattab et al., 2024"},{"why":"Source of the MaxIoU metric used to evaluate system development under noisy human annotations.","marker":"Cronin et al., 2017"}],"fun_headline_variants":["3-stage pipeline tops humans in 14-language hallucination spans","Context+prompts outscore human annotators on LLM hallucination","Retrieval-prompt pipeline ranks #1 on average across 14 languages","Optimized prompts + context hit #1 average across 14 languages","UCSC system beats humans at spotting LLM hallucinations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system assumes the context returned by Perplexity Sonar Pro is complete and factually accurate enough to verify every claim in the answer, so any missing or erroneous search result becomes a missed or falsely flagged hallucination; the better-than-human comparison further assumes the authors' manual annotations follow the same labeling guidelines as the official Mu-SHROOM annotations.","fun_headline_variants_meta":{"raw":{"variants":["3-stage pipeline tops humans in 14-language hallucination spans","Context+prompts outscore human annotators on LLM hallucination","Retrieval-prompt pipeline ranks #1 on average across 14 languages","Optimized prompts + context hit #1 average across 14 languages","UCSC system beats humans at spotting LLM hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000844,"raw_usage":{"total_tokens":3668,"prompt_tokens":933,"completion_tokens":2735,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":2655}},"tokens_in":549,"tokens_out":2735,"duration_ms":20809,"temperature":1.0,"reasoning_tokens":2655,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:00:57.759128+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the manual annotation of the Mu-SHROOM English validation set using the exact official guidelines and the same Perplexity Sonar Pro context the system uses; if the best human annotator's IoU against the official hard labels reaches or exceeds the system's 0.57, the claim that a simple prompting approach beats human annotators would be refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the MaxIoU metric used to evaluate system development under noisy human annotations."}],"review_version":1}