Pith. sign in

REVIEW 3 major objections 4 minor 29 references

HalluTruthQA argues Arabic hallucination evaluation must move beyond binary labels: 2,400 expert-annotated examples with error spans, explanations, and verification options show detection, localization, verification, and explanation are dis

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:25 UTC pith:XO6JDECG

load-bearing objection A genuinely useful fine-grained Arabic hallucination benchmark with solid annotation practice; the single-expert-per-domain design is a real limitation, but the paper is honest about it and the central contribution holds up. the 3 major comments →

arxiv 2607.20219 v1 pith:XO6JDECG submitted 2026-07-22 cs.CL

HalluTruthQA: A Fine-Grained Benchmark for Hallucination Detection, Localization, and Explanation in Arabic Question Answering

Abdessalam Bouchekif , Mohammed-En-Nadhir Zighem , Salah Eddine Bekhouche , Hichem Telli , Somaya Eltanbouly , Shahd Gaben , Heba Sbahi , Samer Rashwani
show 4 more authors
Mutaz Al-Khatib Emad Mohamed Mohammed Ghaly Abdenour Hadid
This is my paper
classification cs.CL
keywords Arabic question answeringhallucination detectionspan localizationfactual verificationexplanation evaluationhallucination taxonomyLLM evaluationknowledge-intensive QA
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that hallucination evaluation in Arabic question answering should be fine-grained rather than binary. It introduces HalluTruthQA, a benchmark of 2,400 expert-curated Arabic QA examples across Islamic knowledge, history, science, and geography, each pairing a model-generated answer with a verified reference answer, a binary label, character-level erroneous spans, human explanations, a hallucination taxonomy, and six candidate answers for factual verification. Evaluating four open LLMs zero-shot, the authors find that detection, span localization, factual verification, and explanation are distinct abilities: the best global scores are 0.880 Macro-F1, 0.516 F1-Sp, 0.852 LO-Score, and 0.644 explanation final score, with no single model leading on all four. A sympathetic reader would care because this shifts Arabic hallucination evaluation from 'right or wrong' to where, why, and how systems fail, and gives concrete error families to target.

Core claim

The paper's central claim is that HalluTruthQA is a valid fine-grained benchmark for Arabic QA hallucination, and that the four tasks it defines are not one ability. In the reported evaluation, no model tops every task: best detection Macro-F1 is 0.880, best span-level F1-Sp is 0.516, best LO-Score for factual verification is 0.852, and best explanation final score is 0.644. The taxonomy shows that hallucinated Arabic answers are not a single phenomenon: in Islamic knowledge, context and source-attribution errors dominate (61.6% of erroneous spans), whereas history, science, and geography are dominated by factual contradiction. The paper also identifies four error families—temporal/numeric n

What carries the argument

The load-bearing object is the multi-layer annotation schema: each of the 2,400 examples pairs an Arabic question and a model-generated answer with a verified reference answer, a binary hallucination label, character-level erroneous spans, human-written explanations, macro/micro hallucination types, and six style-matched candidate answers. This schema turns a hallucination from a binary property into a structured object that can be detected, localized, verified, and explained. The evaluation machinery is a two-stage prompt pipeline plus task-specific metrics: partial-credit span F1 (F1-Sp) for overlap, LO-Score for label-plus-option selection, and an LLM-as-judge protocol for explanation qua

Load-bearing premise

The benchmark's ground truth is only as good as the single expert per domain: reported agreement is between that expert and two research-assistant reviewers, not among independent experts, so systematic bias in span boundaries or explanations would propagate into all localization and explanation conclusions.

What would settle it

Re-annotate a random sample (say 200 examples per domain) with independent domain experts who see only the question and the model answer, then compare their binary labels and span boundaries to the released annotations; if inter-expert agreement falls materially below the reported κ=0.93 and 0.89 character-F1, the benchmark's ground-truth reliability is not established.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Arabic hallucination evaluation can move beyond binary labels to localizing exact erroneous content and checking whether the model can select the correct fact.
  • System rankings change when fine-grained metrics are used: strong detection does not imply good span localization or explanation, so leaderboards should be multi-axis.
  • The benchmark gives concrete error families (temporal/numeric normalization, source attribution, correct answer with hallucinated support, fully wrong answer) as targets for future training and augmentation.
  • Domain-specific profiles—source-attribution errors in Islamic knowledge, factual contradictions in history, science, and geography—suggest that hallucination mitigation should be domain-aware.
  • Because all answers come from one generator, the benchmark provides a controlled setting for studying self-detection and cross-model hallucination patterns.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: a testable extension is to run the same four tasks with retrieval or web access enabled; the benchmark is closed-book, so adding retrieval would isolate whether errors come from missing knowledge or from weak verification and explanation.
  • Beyond the paper: the style-matched six-option design could be reused as a standalone Arabic factual-knowledge probe, independent of generation, to measure how well a model discriminates plausible from true claims.
  • Beyond the paper: the error families suggest a synthetic augmentation recipe—automatically perturb dates, units, source attributions, or supportive claims in verified answers to create hard negative examples—then test whether detection and localization improve on HalluTruthQA.
  • Beyond the paper: if the benchmark grows to legal or medical domains, the Islamic-knowledge results imply that attribution and citation fidelity will need their own evaluation axis rather than being folded into factual contradiction.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper presents HalluTruthQA, a 2,400-example Arabic question-answering benchmark for fine-grained hallucination evaluation. It covers four knowledge-intensive domains (Islamic knowledge, history, science, and geography) and annotates Fanar-generated answers with binary hallucination labels, character-level erroneous spans, human-written explanations, a six-option factual-verification item, and macro/micro hallucination types. Four open-source LLMs (Allam, Falcon-H1, Qwen32, Silma) are evaluated zero-shot on hallucination detection, span localization, factual verification, and explanation quality, with Fanar included as a separate self-detection reference. The results show that no single model leads on all tasks: e.g., Allam achieves the best detection Macro-F1 (0.880), Qwen32 the best span F1-Sp (0.516), Falcon-H1 the best LO-Score (0.852), and Allam the best explanation Final Score (0.644). The paper argues that these tasks measure distinct abilities and that Arabic hallucination evaluation should move beyond binary labels.

Significance. HalluTruthQA addresses a real gap: existing Arabic hallucination datasets mostly provide response-level labels, while this resource also localizes erroneous spans, explains errors, and tests factual verification. The dataset size (2,400) and multi-layer annotation scheme, together with the released code and evaluation scripts, make it a potentially useful resource. The taxonomy of macro/micro hallucination types and the error analysis with concrete examples are strengths. If the annotation validity and explanation-judge reliability are established, the claim that detection, localization, verification, and explanation dissociate would be an important finding for the Arabic NLP community. The paper's careful separation of Fanar's self-detection results from the four independent models is also methodologically sound.

major comments (3)
  1. [Section 3.1 / Appendix B / Table 3] The validity of the gold annotations is not established by the reported agreement. Each domain was annotated by one expert and verified by two research assistants; the reported kappa=0.93 for binary labels and char-F1=0.89 for spans (Table 3) are expert-reviewer agreement, not independent expert-expert agreement. The text explicitly states that 'agreement was not computed among the four experts on shared items' (Appendix B). Research assistants are not domain experts and may defer to the expert or share systematic biases. Because all downstream detection, localization, and explanation results are conditional on these gold labels and spans, the benchmark's core validity requires at least a shared sample (e.g., 100-200 items per domain) annotated independently by two qualified experts, with agreement reported separately for labels, spans, explanations, and the MCQ answer key. Without such
  2. [Section 4.3 / Appendix E] The explanation task is scored by GPT-5.5 with only 25 human-checked examples per domain. The paper states that the human review 'did not reveal systematic disagreement,' but it provides no quantitative agreement (e.g., Cohen's kappa or correlation) between human and judge scores, and 100 examples is a small sample relative to the 1,013 hallucinated instances used in the evaluation. Moreover, the judge prompt is given the gold erroneous span and the human-written reference explanation; this may reward surface similarity to the reference rather than the model's ability to independently identify and correct the error. The explanation-quality conclusions (e.g., Allam's 0.644 Final Score) should be supported by a larger validation set and explicit human-judge agreement statistics, reported separately for the Error Identification and Factual Correction dimensions.
  3. [Table 2] The paper's dissociation claim is based on rank comparisons without any uncertainty estimates. For example, global Macro-F1 for Allam (0.880) and Falcon-H1 (0.859) differ by 0.021, and LO-Score for Falcon-H1 (0.852) and Qwen32 (0.844) differ by 0.008. The paper neither reports confidence intervals nor statistical tests. Since the conclusion that 'these tasks capture different abilities' rests on such differences, bootstrap confidence intervals over examples (or models) should be provided for at least the global metrics. Without them, it is difficult to judge which cross-task comparisons are reliable.
minor comments (4)
  1. [Section 4.3] 'Following Sky et al. (2024)' is cited in the text but the reference is missing from the bibliography. Please add the full citation and, if possible, clarify the matching procedure for multiple gold/predicted spans.
  2. [Appendix D, Example 2] The gold label for the Germany unification question is debatable (1871 vs. 1990). A sentence in the annotation guidelines explaining how such temporal interpretation cases are adjudicated would strengthen the dataset's perceived objectivity.
  3. [Table 2] Fanar is a self-detection reference setting, not one of the four independent models. A visual separator or explicit footnote in the table would prevent readers from treating Fanar's rows as directly comparable to the open-model evaluation.
  4. [Section 4.3] The explanation score is a pipeline metric because it is computed only when both gold and predicted labels are hallucinated. This is stated in the text, but the caption of Table 2 should repeat it so that the headline 'Final Score' is not misinterpreted as an isolated explanation quality score.

Circularity Check

0 steps flagged

No significant circularity; benchmark labels and model evaluations are independent, with only minor non-load-bearing self-citations.

full rationale

HalluTruthQA is an empirical resource paper: the central claims are that the dataset is a valid fine-grained benchmark and that the four evaluated models show different abilities across detection, localization, verification, and explanation. None of these claims is derived from a fitted parameter, an equation, or a self-citation. The gold binary labels, spans, explanations, and multiple-choice options are human-annotated and are independent of the models being evaluated. The paper explicitly separates Fanar's self-detection, stating it is 'reported separately, rather than as evidence that Fanar is superior to the other models,' so this self-evaluation is not used as a load-bearing result. Explanation evaluation uses a GPT-5.5 judge with the gold span and human-written reference explanation, plus a human sanity-check, so it is an external metric rather than a circular construction. The self-citations (Bouchekif et al., 2025a,b, 2026a,b) appear in related work and are motivational, not load-bearing: the paper does not invoke them to establish the validity of its benchmark or to force its conclusions. The limitation that each domain was annotated by a single expert is a reliability concern about ground-truth quality, but it is not circularity: a biased annotation would make the benchmark inaccurate, not make the evaluation equivalent to its own inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The benchmark introduces no fitted parameters or new physical entities. Its central claims rest on annotation reliability, representativeness of the single generator, and validity of LLM-as-judge evaluation.

axioms (3)
  • domain assumption The 2,400 expert-curated labels, spans, explanations, and answer keys are reliable ground truth.
    Section 3.1/Appendix B report agreement (κ=0.93 binary, 0.89 char-F1 spans) but each domain was annotated by a single expert; independent-expert agreement is not measured.
  • domain assumption Fanar-1-9B-Instruct outputs are treated as representative model-generated Arabic answers for constructing the benchmark.
    Section 3 uses one generator; the paper acknowledges this limits hallucination-pattern diversity.
  • domain assumption GPT-5.5 LLM-as-a-judge scores correlate with human judgments of explanation quality.
    Section 4.3 uses GPT-5.5 with only 25 human-checked examples per domain; no quantitative agreement is reported.

pith-pipeline@v1.3.0-alltime-deepseek · 17798 in / 10409 out tokens · 87390 ms · 2026-08-01T10:25:09.308802+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) can generate fluent Arabic answers, yet factual errors remain difficult to detect, localize, explain, and verify. Existing hallucination benchmarks often provide response-level labels, with limited support for identifying the exact erroneous content, explaining why it is incorrect, or selecting the correct factual answer. We introduce \textsc{HalluTruthQA}, a fine-grained benchmark for hallucination evaluation in Arabic question answering. The benchmark contains 2,400 expert-curated examples across four knowledge-intensive domains: Islamic knowledge, history, science, and geography. Each example pairs an Arabic question and a model-generated answer with a verified reference answer, a binary hallucination label, six candidate answers for factual verification, and, for hallucinated answers, character-level erroneous spans, human-written explanations, and macro and micro hallucination types. We evaluate four open-source LLMs, \textsc{Allam}, \textsc{Falcon-H1}, \textsc{Qwen32}, and \textsc{Silma}, in a zero-shot setting across hallucination detection, span-level localization, factual verification, and explanation evaluation. Results show that these tasks capture different abilities: no single model achieves the strongest performance across all tasks, with best scores of 0.880 Macro-F1 for detection, 0.516 F1-Sp for localization, 0.852 LO-Score for factual verification, and 0.644 final score for explanation evaluation. Our taxonomy shows that hallucination evaluation should move beyond detection toward localizing, verifying, and explaining factual errors. The code, dataset, prompts, and evaluation scripts are available at https://gitlab.com/nlpresearcher/HalluTruthQA.

Figures

Figures reproduced from arXiv: 2607.20219 by Abdenour Hadid, Abdessalam Bouchekif, Emad Mohamed, Heba Sbahi, Hichem Telli, Mohammed-En-Nadhir Zighem, Mohammed Ghaly, Mutaz Al-Khatib, Salah Eddine Bekhouche, Samer Rashwani, Shahd Gaben, Somaya Eltanbouly.

Figure 1
Figure 1. Figure 1: Hallucination macro-type distribution across domains. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the HalluTruthQA construction pipeline. [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Hierarchical taxonomy of hallucination macro-types and observed micro-types. [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 3 linked inside Pith

  1. [1]

    Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages =

    On Faithfulness and Factuality in Abstractive Summarization , author =. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages =

  2. [2]

    ACM Computing Surveys , volume =

    Survey of Hallucination in Natural Language Generation , author =. ACM Computing Surveys , volume =. 2023 , doi =

    2023

  3. [3]

    arXiv preprint arXiv:2603.23750 , year=

    IslamicMMLU: a benchmark for evaluating LLMs on Islamic knowledge , author=. arXiv preprint arXiv:2603.23750 , year=

    Pith/arXiv arXiv

  4. [4]

    Manakul, Potsawee and Liusie, Adian and Gales, Mark J. F. , booktitle =

  5. [5]

    QIAS 2025: Overview of the Shared Task on Islamic Inheritance Reasoning and Knowledge Assessment

    Bouchekif, Abdessalam and Rashwani, Samer and Mohamed, Emad Soliman Ali and Alkhatib, Mutaz and Sbahi, Heba and Gaben, Shahd and Zaghouani, Wajdi and Erbad, Aiman and Ghaly, Mohammed. QIAS 2025: Overview of the Shared Task on Islamic Inheritance Reasoning and Knowledge Assessment. Proceedings of The Third Arabic Natural Language Processing Conference: Sha...

    doi:10.18653/v1/2025.arabicnlp-sharedtasks.117 2025

  6. [6]

    arXiv preprint arXiv:2603.07539 , year=

    MAWARITH: A Dataset and Benchmark for Legal Inheritance Reasoning with LLMs , author=. arXiv preprint arXiv:2603.07539 , year=

    arXiv

  7. [7]

    Bouchekif, Abdessalam and Eltanbouly, Somaya and Rashwani, Samer and Gaben, Shahd and Al-Khatib, Mutaz and Sbahi, Heba and Mohamed, Emad and Ghaly, Mohammed , booktitle =

  8. [8]

    2010 , publisher =

    Introduction to Arabic Natural Language Processing , author =. 2010 , publisher =. doi:10.2200/S00277ED1V01Y201008HLT010 , url =

    doi:10.2200/s00277ed1v01y201008hlt010 2010

  9. [9]

    ACM Transactions on Asian Language Information Processing , volume =

    Arabic Natural Language Processing: Challenges and Solutions , author =. ACM Transactions on Asian Language Information Processing , volume =. 2009 , publisher =

    2009

  10. [10]

    Halwasa: Quantify and Analyze Hallucinations in Large Language Models:

    Mubarak, Hamdy and Al-Khalifa, Hend and Alkhalefah, Khaloud Suliman , booktitle =. Halwasa: Quantify and Analyze Hallucinations in Large Language Models:. 2024 , month = may, address =

    2024

  11. [11]

    2025 , month = nov, address =

    Alansari, Aisha and Luqman, Hamzah , booktitle =. 2025 , month = nov, address =. doi:10.18653/v1/2025.arabicnlp-main.12 , url =

    doi:10.18653/v1/2025.arabicnlp-main.12 2025

  12. [12]

    Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks , pages=

    IslamicEval 2025: The First Shared Task of Capturing LLMs Hallucination in Islamic Content , author=. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks , pages=

    2025

  13. [13]

    Hallucination Detectives at

    Elchafei, Passant and Abu-Elkheir, Mervat , booktitle =. Hallucination Detectives at. 2025 , month = jul, address =

    2025

  14. [14]

    2022 , month = may, address =

    Lin, Stephanie and Hilton, Jacob and Evans, Owain , booktitle =. 2022 , month = may, address =. doi:10.18653/v1/2022.acl-long.229 , url =

    doi:10.18653/v1/2022.acl-long.229 2022

  15. [15]

    2023 , month = dec, address =

    Li, Junyi and Cheng, Xiaoxue and Zhao, Xin and Nie, Jian-Yun and Wen, Ji-Rong , booktitle =. 2023 , month = dec, address =. doi:10.18653/v1/2023.emnlp-main.397 , url =

    doi:10.18653/v1/2023.emnlp-main.397 2023

  16. [16]

    2023 , url =

    Chen, Shiqi and Zhao, Yiran and Zhang, Jinghan and Chern, I-Chun and Gao, Siyang and Liu, Pengfei and He, Junxian , booktitle =. 2023 , url =

    2023

  17. [17]

    2023 , month = dec, address =

    Min, Sewon and Krishna, Kalpesh and Lyu, Xinxi and Lewis, Mike and Yih, Wen-tau and Koh, Pang and Iyyer, Mohit and Zettlemoyer, Luke and Hajishirzi, Hannaneh , booktitle =. 2023 , month = dec, address =. doi:10.18653/v1/2023.emnlp-main.741 , url =

    doi:10.18653/v1/2023.emnlp-main.741 2023

  18. [18]

    2024 , month = aug, address =

    Niu, Cheng and Wu, Yuanhao and Zhu, Juno and Xu, Siliang and Shum, KaShun and Zhong, Randy and Song, Juntong and Zhang, Tong , booktitle =. 2024 , month = aug, address =. doi:10.18653/v1/2024.acl-long.585 , url =

    doi:10.18653/v1/2024.acl-long.585 2024

  19. [19]

    2026 1st International Conference on Emerging Technologies and Engineering Systems (

    Detecting Hallucinations in Arabic Large Language Models Using Automatic Fact-Verification Signals , author =. 2026 1st International Conference on Emerging Technologies and Engineering Systems (. 2026 , publisher =

    2026

  20. [20]

    Hallucination Analysis in Large Language Models for Arabic Question Answering , author =. Ing. 2026 , doi =

    2026

  21. [21]

    Proceedings of the 19th International Workshop on Semantic Evaluation (

    Vazquez, Raul and Mickus, Timothee and Zosa, Elaine and Vahtola, Teemu and Tiedemann, J. Proceedings of the 19th International Workshop on Semantic Evaluation (. 2025 , month = jul, address =

    2025

  22. [22]

    arXiv preprint arXiv:2503.07833 , year=

    Halluverse25: Fine-grained multilingual benchmark dataset for llm hallucinations , author=. arXiv preprint arXiv:2503.07833 , year=

    Pith/arXiv arXiv

  23. [23]

    ACM Transactions on Information Systems , volume =

    A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions , author =. ACM Transactions on Information Systems , volume =. 2025 , doi =

    2025

  24. [24]

    arXiv preprint arXiv:2510.06265 , year =

    Large Language Models Hallucination: A Comprehensive Survey , author =. arXiv preprint arXiv:2510.06265 , year =

    arXiv

  25. [25]

    Nature Machine Intelligence , volume =

    Factuality Challenges in the Era of Large Language Models and Opportunities for Fact-Checking , author =. Nature Machine Intelligence , volume =. 2024 , doi =

    2024

  26. [26]

    Assessing Large Language Models on Islamic Legal Reasoning: Evidence from Inheritance Law Evaluation

    Bouchekif, Abdessalam and Rashwani, Samer and Sbahi, Heba and Gaben, Shahd and Al Khatib, Mutaz and Ghaly, Mohammed. Assessing Large Language Models on Islamic Legal Reasoning: Evidence from Inheritance Law Evaluation. Proceedings of The Third Arabic Natural Language Processing Conference. 2025. doi:10.18653/v1/2025.arabicnlp-main.20

    doi:10.18653/v1/2025.arabicnlp-main.20 2025

  27. [27]

    arXiv preprint arXiv:2603.24012 , year=

    CVPD at QIAS 2026: RAG-Guided LLM Reasoning for Al-Mawarith Share Computation and Heir Allocation , author=. arXiv preprint arXiv:2603.24012 , year=

    arXiv 2026

  28. [28]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    Do androids know they’re only dreaming of electric sheep? , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    2024

  29. [29]

    Fanar : An Arabic-Centric Multimodal Generative AI Platform. 2025. doi:10.48550/arXiv.2501.13944. arXiv:2501.13944

    Pith/arXiv arXiv doi:10.48550/arxiv.2501.13944 2025