{"id":"82064634-8d9a-4fd7-a392-677db1b78fcf","arxiv_id":"2607.20219","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"HalluTruthQA provides 2,400 expert-annotated Arabic QA examples with character-level hallucination spans, explanations, and verification candidates, and shows no single LLM excels at all four evaluation tasks.","lead":"This paper introduces HalluTruthQA, a 2,400-example Arabic benchmark that marks whether AI answers are hallucinated, highlights the exact wrong text, explains the error, and offers six candidate answers to verify the fact. It tests four open language models and shows that detecting a hallucination, locating it, and fixing it are different skills.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's validity rests on annotations made by a single expert per domain; expert–reviewer agreement cannot detect systematic per-expert error. An independent second-expert re-annotation sample is needed before accepting the ground truth as authoritative.","rationale":"The reader identified the same weakest assumption: human ground-truth correctness. I agree, and I sharpen it: the problem is not merely annotation difficulty but single-expert-per-domain design plus adjudication by that same expert. Because the four experts annotated disjoint subsets, the reported agreement is expert-vs-reviewer, not expert-vs-expert; a reviewer is less likely to overturn an expert and, as the text says, final decisions return to the expert. Any systematic domain-level error is invisible to the reported κ and to the 3.5% flagged-example rate. This matters because every result in Table 2—Macro-F1, F1-Sp, LO-Score, and explanation final scores—is computed against these labels, spans, and answer keys; a wrong label changes the denominator and the option correctness. The benchmark's main contribution is a reusable resource, so its reliability is the central claim. The LLM-judge explanation-evaluation layer is also under-validated (only 100 sampled explanations), but it is arguably less foundational: if the human ground truth is sound, the judge can be re-validated or replaced without invalidating the resource; if the ground truth is not sound, no downstream evaluation is meaningful. The paper is otherwise careful: it reports agreement, gives explicit limitations, releases code/data links, and uses a single generator for controlled annotation. My recommendation is to keep the reader's CONDITIONAL verdict, because the missing evidence is precisely what the condition should require: independent per-domain expert re-annotation. No ad hominem or manufactured flaw is intended; this is a request for a validation step that the paper itself does not provide.","tokens_in":22185,"tokens_out":4539,"duration_ms":49393,"concrete_test":"Draw a stratified random sample of 40 examples per domain (160 total), preserving the hallucinated/non-hallucinated ratio per Table 1. Have a second domain expert who is external to the project and blinded to all original annotations re-annotate each plain question+generated-answer pair using the same taxonomy and independently consulting the same public sources. Compare per-domain: Cohen's κ for binary labels, character-level F1/IoU for spans, and agreement on the MCQ key; also record whether the reference answer is judged correct. Accept the ground truth if κ≥0.90, span F1≥0.85, and key agreement ≥0.95 in each domain; if any domain falls substantially below the original expert–reviewer figures, the benchmark's labels cannot be treated as authoritative and all reported task scores are suspect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that the 2,400 human annotations—reference answers, binary labels, character-level spans, MCQ keys, and explanations—are correct. The paper's own pipeline (§3.1, Appendix C) assigns one domain expert per domain; the 'verification' pass is by two trained research assistants, and all disagreements are adjudicated by the same original expert. Appendix C explicitly states that agreement was not computed among the four experts on shared items. Thus the reported expert–reviewer agreement (κ=0.93 labels, 0.89 char-F1, 97.9% answer-key) cannot rule out a systematic mistake by a single domain expert: if an expert consistently applies a wrong date conversion, accepts an unsupported source, or builds a flawed distractor set, that error propagates through every example in the domain and into every reported model score (detection Macro-F1, F1-Sp, LO-Score, explanation). The paper's own limitation note acknowledges span-boundary uncertainty but not this per-expert independence gap. This is not an internal inconsistency in the text; it is missing evidence for the benchmark's central reuse claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"HalluTruthQA introduces a 2,400-example Arabic QA hallucination benchmark spanning Islamic knowledge, history, science, and geography. Each item contains a FANAR-9B-generated answer, a human-verified reference answer, a binary hallucination label, and six MCQ candidates; hallucinated items additionally have character-level erroneous spans, human-written explanations, and macro/micro hallucination types. The paper reports a two-pass annotation process, a zero-shot evaluation of four open-source LLMs (plus FANAR-9B as a separated self-detection reference), and results on detection, span localization, factual verification, and explanation quality. The central claim is that HalluTruthQA supports fine-grained hallucination evaluation and that the four tasks capture distinct model abilities.","tokens_in":22479,"tokens_out":4614,"duration_ms":57082,"significance":"If the annotations are reliable, HalluTruthQA is a valuable and timely resource: it is, to my knowledge, the first Arabic hallucination benchmark to jointly provide response-level labels, exact character-level spans, human-written explanations, and a factual-verification MCQ setting. The resource, code, prompts, and evaluation scripts are released, which is a concrete strength. The evaluation is honest about the single-generator design and the pipeline nature of the downstream metrics. The headline result—that detection, localization, verification, and explanation do not rank models identically—is interesting and plausible. However, the benchmark’s reuse value depends on the correctness of the human ground truth, and that correctness is not yet established as strongly as the paper implies.","major_comments":[{"comment":"The load-bearing validity claim rests on the expert annotations, but the design uses one domain expert per domain, with verification by two research assistants and adjudication by the same original expert. Appendix C explicitly states that agreement was not computed among the four experts on shared items. Expert–reviewer agreement (κ=0.93 for labels, 0.89 char-F1 for spans, 97.9% for the answer key) cannot detect a systematic error by a single domain expert: wrong reference answers, biased distractor sets, or consistent span-boundary mistakes in one domain would propagate into every model score. This is not an internal contradiction; it is missing evidence. Please add an independent second-expert re-annotation of a random, per-domain sample (e.g., 10–15% of each domain), report per-domain agreement on labels, spans, answer keys, and explanations, and describe how disagreements with the f","section":"§3.1 and Appendix C"},{"comment":"Explanation quality is scored by GPT-5.5 as an LLM judge, with only 25 explanations per domain manually reviewed and no quantitative agreement metric. Since explanation quality is one of the four headline tasks and is used to claim ALLaM-7B is the best explainer (Final Score 0.644), the reliability of the judge matters. The statement that the human review 'did not reveal systematic disagreement' is too vague. Please report a correlation or agreement coefficient between judge scores and human scores (e.g., Pearson/Spearman on the 100-example sample, or κ on discretized scores), and ideally enlarge the human validation sample or use a second judge.","section":"§4.3, Explanation Evaluation"},{"comment":"Span-level F1-Sp is computed with detection-label mismatches receiving zero, and MCQ Correct/LO-Score are computed only after hallucination prediction. The paper acknowledges this is a pipeline setting, but the cross-model comparison is still presented as 'localization ability' (e.g., Qwen3-32B best F1-Sp 0.516). A model that detects fewer hallucinations is doubly penalized, so its localization score reflects detection behavior as much as span accuracy. This partially confounds the paper's claim that the tasks are distinct. Please also report localization F1 conditional on correctly detected hallucinated instances (true positives), together with detection recall, so readers can separate the two factors.","section":"§4.3 and Table 2, pipeline metrics"},{"comment":"The reported differences between top detection models are small (global Macro-F1: ALLaM-7B 0.880, Falcon-H1R-7B 0.859, Qwen3-32B 0.849), and no confidence intervals or significance tests are given. The claim that 'no single model performs best across all tasks' is load-bearing. Please provide bootstrap confidence intervals over the 2,400 examples (and per domain), or at least a per-example variance estimate, so readers can judge whether the reported rankings are within sampling noise. This also applies to the F1-Sp and explanation-score comparisons.","section":"§4.2 and §4.4, model ranking"}],"minor_comments":[{"comment":"The 'Final Score' in Table 2 is not defined in the text; the prompt in Appendix F defines Error Identification and Factual Correction as separate 0–1 scores, but the aggregation into Final Score is missing. State whether it is an average or a weighted combination.","section":"§4.3 / Table 2"},{"comment":"Huang et al. 2025a and 2025b appear to be the same survey with slightly different author lists; please consolidate or clarify the distinction.","section":"References"},{"comment":"The phrase 'verified reference answer' in the abstract and data description is strong. Appendix A.1 lists consulted sources but no source-audit protocol or per-example source links. Consider documenting the verification protocol in more detail, or softening the wording.","section":"Appendix A.1"},{"comment":"Model ordering is inconsistent across domain blocks (e.g., Qwen3-32B appears first in Islamic/Geography/History but second in Science). Use a fixed ordering to improve readability.","section":"Table 2"},{"comment":"The paper would benefit from a short 'dataset statistics' table beyond Table 1, e.g., average response length, average span length per domain, and number of multi-span examples, to help users understand the localization difficulty.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is in scope and the resource is potentially useful, but the annotation-independence issue is central and should be resolved before acceptance. I would ask the authors for independent second-expert re-annotation evidence and a more rigorous treatment of the explanation judge and confidence intervals. The GitLab release should also be checked for a license and a stable data snapshot."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real resource, worth refereeing, but the annotation ground truth is not yet as solid as the paper claims. The single-expert-per-domain design means the benchmark inherits any systematic errors in one expert's domain, and the high expert–reviewer agreement cannot detect that.\n\nWhat is genuinely new is the combination: exact character-level erroneous-span localization for every hallucinated answer, human-written explanations, and a six-option factual verification task, all in one Arabic QA benchmark. Neither AraHalluEval nor HalluScore offers that. The dataset construction is transparent: 2,400 examples, four knowledge domains, a two-pass process, kappa 0.93 on labels and 0.89 character-F1 on spans, with disagreements adjudicated. The model evaluation is also clearly presented: pipeline metrics are labeled as such, FANAR-9B's self-detection numbers are separated rather than used as primary evidence, and the conclusion that detection, localization, verification, and explanation do not align in a single model is supported by the scores.\n\nThe main soft spot is the one the stress-test note identifies. Each domain was annotated by a single expert, the verification pass was done by two research assistants, and adjudication went back to the same expert. That is a reasonable second pass, but it is not independent expert validation. A wrong date convention, an unsupported source, or a flawed distractor set in one domain would silently propagate through every model score. The paper acknowledges span-boundary uncertainty in the limitations but does not address this independence gap. This is missing evidence rather than an internal contradiction, and it is fixable: a re-annotation sample of, say, 10-15% per domain by a second independent expert, with disagreement rates reported.\n\nTwo smaller issues. The explanation scores rely on GPT-5.5 as judge with only 25 human-checked explanations per domain; that is thin. And there are no confidence intervals or significance tests on the model comparisons, so some across-model differences may be noise. Neither is fatal.\n\nOverall, the central claim—that this benchmark measures more than binary hallucination detection—holds up. Arabic NLP researchers and benchmark builders will get real value from it. I would send it to peer review, but with the independent annotation sample and stronger judge validation as conditions.","headline":"A genuinely useful Arabic hallucination resource with a real annotation weakness—single expert per domain—that should be fixed with independent re-annotation before the benchmark is treated as authoritative.","tokens_in":22945,"tokens_out":2888,"would_cite":true,"duration_ms":31040,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new expert-built benchmark pinpoints exactly where Arabic LLM answers go wrong, from single dates to fabricated citations.","keywords":["Arabic QA","hallucination detection","span-level localization","explanation evaluation","factual verification","LLM benchmark","fine-grained annotation","zero-shot evaluation"],"falsifier":"Take a random sample of 100 examples and have three fresh independent domain experts re-annotate the labels and character spans; if agreement with the published ground truth falls well below the reported expert–reviewer agreement (κ=0.93 for labels, 0.89 character-F1 for spans), the benchmark's ground truth is not stable enough to support its rankings. A second check is to regenerate the 2,400 answers with a different Arabic LLM and see whether the domain-level statistics — such as 61.6% Context Inconsistency in Islamic knowledge — persist.","tokens_in":1431,"feed_emoji":"📍","tokens_out":2460,"duration_ms":78104,"temperature":0.7,"pith_summary":"HalluTruthQA is an expert-curated benchmark for Arabic question answering that asks not only whether a model's answer is wrong, but where, why, and whether the model can recover the right fact. It contains 2,400 questions paired with model-generated answers, verified reference answers, binary hallucination labels, character-level erroneous spans, human explanations, and six candidate answers for factual verification. Evaluating four open LLMs, the paper shows that detection, localization, verification, and explanation are distinct capabilities: the best detector reaches 0.880 Macro-F1, the best localizer 0.516 F1-Sp, the best verifier 0.852 LO-Score, and the best explainer 0.644 — and no single model tops all four. The authors therefore argue that Arabic hallucination evaluation should move beyond response-level labels toward fine-grained localization, explanation, and factual verification.","feed_headline":"Fine-grained Arabic QA benchmark pinpoints hallucinated spans","feed_subtitle":"2,400 expert-annotated questions show detection, localization, and explanation are separate skills.","key_machinery":"The load-bearing machinery is the multi-layer annotation schema: each example pairs an Arabic question with a FANAR-9B-generated answer, a verified reference answer, a binary hallucination label, character-level erroneous spans (for hallucinated answers), human-written explanations, macro- and micro-level hallucination types, and six human-written candidate answers with a verified key. Two evaluation metrics carry the argument. F1-Sp is a partial-credit span-level F1 that rewards any overlap between predicted and gold token sets, so boundary drift is not treated as total failure. LO-Score jointly scores label correctness and option selection: 1.0 for a correct no-hallucination call, 1.0 for","core_discovery":"On its own terms, the paper's central discovery is that fine-grained hallucination dimensions in Arabic QA do not move together: models that detect hallucinated answers do not systematically localize the erroneous span, select the correct candidate answer, or produce faithful explanations. In a zero-shot, closed-book evaluation of ALLaM-7B, Falcon-H1R-7B, Qwen3-32B, and SILMA, the strongest detection, localization, and explanation performances belong to different models, and span localization (best F1-Sp 0.516) is substantially harder than detection (best Macro-F1 0.880). The paper also finds that error patterns are domain-dependent: Islamic knowledge is dominated by source-attribution and e","pith_inferences":["Because all answers come from a single generator (FANAR-9B), the benchmark's difficulty is partly a property of that model's error patterns; regenerating answers with a different Arabic LLM could shift both the span statistics and the resulting model rankings.","Gold explanations are human-written, but model explanations are judged by an LLM with human review on only 25 samples per domain; a fully human-rated explanation subset would be a natural reliability check.","The character-level spans and six-option answer keys could be repurposed as training signal for fine-grained factual correction or retrieval-augmented generation, not just used as a static test set.","The identified error families (date/unit normalization, source attribution, correct-answer-with-false-support) suggest concrete, testable interventions, such as normalizing calendar formats before evaluation or adding a citation-verification step."],"forward_implications":["If the benchmark is accurate, Arabic QA evaluation should report localization, verification, and explanation scores alongside detection accuracy, rather than relying on response-level labels alone.","Detection and localization are distinct skills: Qwen3-32B's localization (0.516 F1-Sp) is the best among independent models, yet its detection Macro-F1 (0.849) trails ALLaM-7B (0.880), which in turn localizes worse (0.449).","A large share of model errors (17.1%) are correct short answers with hallucinated supporting evidence or invented citations, so response-level correctness systematically overestimates factual reliability.","Source-sensitive domains like Islamic knowledge require attribution-aware evaluation: the dominant error there is wrong source or attribution (61.6% of spans), a category that binary detection cannot capture.","Temporal, numeric, and unit normalization failures account for 39.0% of model-level errors, meaning models still fail to equate equivalent expressions such as Hijri and Gregorian dates."],"fun_headline_variants":["Arabic QA localization lags detection by 36 points","No single LLM excels at all Arabic hallucination tasks","Domain shapes Arabic QA hallucination types","Detection and localization are distinct Arabic QA hallucination skills"],"cache_read_input_tokens":24448,"weakest_assumption_plain":"Everything the benchmark concludes rests on the human annotations — the reference answers, labels, error spans, explanations, and multiple-choice keys — being correct; because each domain was annotated by a single expert and then checked by two research assistants rather than by independent experts on the same items, one domain expert's systematic mistake would silently skew every model score.","fun_headline_variants_meta":{"raw":{"variants":["Arabic QA localization lags detection by 36 points","No single LLM excels at all Arabic hallucination tasks","Domain shapes Arabic QA hallucination types","Detection and localization are distinct Arabic QA hallucination skills"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001734,"raw_usage":{"total_tokens":6730,"prompt_tokens":824,"completion_tokens":5906,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":5845}},"tokens_in":568,"tokens_out":5906,"duration_ms":41304,"temperature":1.0,"reasoning_tokens":5845,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T01:49:08.490273+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of 100 examples and have three fresh independent domain experts re-annotate the labels and character spans; if agreement with the published ground truth falls well below the reported expert–reviewer agreement (κ=0.93 for labels, 0.89 character-F1 for spans), the benchmark's ground truth is not stable enough to support its rankings. A second check is to regenerate the 2,400 answers with a different Arabic LLM and see whether the domain-level statistics — such as 61.6% Context Inconsistency in Islamic knowledge — persist.","supporting_citations":[],"review_version":2}