REVIEW 4 major objections 5 minor 8 references
A fine-tuned 12B model beats zero-shot baselines on Islamic hallucination detection and verified-answer selection, with a combined score of 0.912.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
Fine-tuned Gemma-4-12B-it achieves 0.928 Macro-F1 for hallucination detection and 0.895 option accuracy on the Arabic Islamic HalluScoring 2026 task.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection A plausible shared-task system with a candid error analysis, but the missing decontamination statement and mixed metrics keep the headline claim from being established. the 4 major comments →
Detecting Hallucinations and Recovering Verified Answers in Arabic Islamic Question Answering
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
On the official evaluation data for Islamic hallucination detection and answer recovery, the fine-tuned system outperforms the zero-shot baselines reported for the benchmark on both subtasks: label accuracy of 0.935 and Macro-F1 of 0.928 for hallucination detection, and option accuracy of 0.895 for selecting the verified answer from six candidates, giving a combined score of 0.912. The paper's own error analysis shows three recurring failure families: fluent but incorrectly attributed answers are accepted as non-hallucinated; closely related religious entities and references are confused; and the model sometimes detects a hallucination and then selects an option that reproduces the same hall
What carries the argument
The load-bearing object is the fine-tuned Gemma-4-12B-it instruction-tuned model run with a single unified prompt: it outputs a Hallucinated/Non-Hallucinated label and a selected option in one generation, decoded with temperature 0.0 and top-p 1.0 and normalized to extract both predictions. The training data mix binary detection instances built from gold options and their distractors, open-ended fatwa and inheritance QA, controlled single-word edits of Qur'anic verses and hadith quotations, and 3,000 Gemini-generated Arabic answers. The two task-specific adapters are trained separately and then used jointly.
Load-bearing premise
The reported scores assume the official 600-question evaluation set shares no questions with the 600 HalluTruthQA detection instances and 600 HalluTruthQA selection instances used as training data—an assumption the paper never states.
What would settle it
Check each question in the official HalluTruthQA-4K evaluation set for exact or near-duplicate overlap with the HalluTruthQA detection and selection instances listed in Table 1 as training data. If even a few evaluation questions appear in training, the reported margins over the zero-shot baselines are inflated; a clean version is to re-run the system on the non-overlapping remainder and report the change in option accuracy.
If this is right
- A fine-tuned 12-billion-parameter model can beat much larger zero-shot systems on this Arabic Islamic benchmark, suggesting that domain fine-tuning outweighs raw model scale in hallucination detection here.
- The detection–selection gap implies that a leaderboard reporting only a combined score can overstate a system's factual reliability; detection and answer recovery should be shown separately.
- The observed copying of a rejected answer into the selected option gives a concrete training signal: hard negatives built from the hallucinated answer and an explicit consistency objective between the two outputs.
- The reference-ambiguity errors imply that multi-reference adjudication is needed for questions with several defensible answers; otherwise the evaluation conflates genuine hallucinations with alternative valid responses.
Where Pith is reading between the lines
- A testable diagnostic follows from the paper's own error examples: split the evaluation set by the predicted hallucination label and compute selection accuracy among hallucinated instances for options that are lexically most similar to the generated answer. If accuracy is near chance, the anchoring explanation is confirmed; the paper does not run this breakdown.
- The paper never states that the 600-question official evaluation set is disjoint from the 600 HalluTruthQA detection instances and 600 HalluTruthQA selection instances listed in Table 1 as training data. Question-level overlap checks on the public release would settle whether the reported margin over baselines is in-distribution.
- If overlap is found, the headline combined score would drop but the qualitative error analysis would likely survive; the two-task gap and anchoring patterns are structural observations that do not depend on the exact test split.
- Applying the same two-step fine-tuning recipe to non-Islamic Arabic factuality data would reveal how much of the gain is domain-specific versus a general effect of the training mix; a sharp drop would argue for domain-matched training.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes a system submitted to HalluScoring 2026 Task 2.1 for Arabic Islamic hallucination detection and verified-answer selection. The authors fine-tune google/gemma-4-12B-it on 43,400 detection instances and 2,600 selection instances assembled from existing benchmarks and generated data. At inference with temperature 0, the model is expected to output both a hallucination label and one of six candidate answers. On the 600-instance official HalluTruthQA-4K Islamic evaluation set, the paper reports label accuracy 0.935, Macro-F1 0.928, option accuracy 0.895, and a combined score of 0.912, comparing favorably with zero-shot baselines. The paper includes a qualitative error analysis that identifies several recurring failure patterns.
Significance. If the reported numbers are valid and the evaluation is uncontaminated, the paper provides a strong fine-tuned result for a low-resource Arabic Islamic QA setting, and the observed gap between hallucination detection and answer selection is a useful empirical finding. The error analysis is a genuine strength: it gives concrete examples, candidly discusses reference ambiguity and anchoring effects, and explicitly caveats the zero-shot comparison. The manuscript is nevertheless not currently sufficient as a benchmark measurement, because the training and evaluation data overlap by construction unless a disjointness condition is stated, and because the baseline comparison mixes different metrics.
major comments (4)
- [§2 (Table 1) and §3] Training includes 600 HalluTruthQA detection instances and 600 HalluTruthQA selection instances (Table 1), while evaluation uses the official HalluTruthQA-4K data (Section 3). The paper never states that the training and evaluation instances are disjoint. Because HalluTruthQA appears both as a training source and as the evaluation benchmark, exact or near-duplicate instances are plausible, and the reported label accuracy, Macro-F1, option accuracy, and combined score of 0.912 could be inflated by in-distribution memorization. Please state explicitly whether any HalluTruthQA training row overlaps the 600-instance evaluation split; if so, retrain without HalluTruthQA and re-report, or otherwise demonstrate non-overlap.
- [§3 (Table 2)] The comparison in Table 2 mixes metrics. For the HalluTruthQA baselines the 'MCQ' column reports LO-Score and the combined score is the equally weighted average of Macro-F1 and LO-Score; for the proposed approach the same column reports option accuracy and the combined score uses the official HalluScoring 2026 formula. LO-Score and option accuracy are not the same quantity, so the table does not establish that the proposed system outperforms the baselines on answer selection or on the combined ranking. Please report baseline option accuracy on the same 600 instances and recompute all combined scores with one consistent formula.
- [§3] The fine-tuning configuration is absent. The paper reports only inference hyperparameters (temperature 0.0, top-p 1.0, max new tokens 2,048), with no learning rate, number of epochs, batch size, LoRA rank or adapter architecture, precision, or train/validation split. The conclusion refers to 'separate task-specific adapters' but the experiments describe a single fine-tuned model. Without these details and without code or a released checkpoint, the proposed method is not reproducible, and the contribution cannot be separated from undocumented training choices.
- [§3 (Table 2)] All headline numbers are single-run point estimates on 600 instances, with no confidence intervals or repeated-run variance. The combined-score gap over the strongest baseline (Fanar-9B, 0.894) is 0.018. On a test set of this size, and with no significance test or uncertainty estimate, the claim that the approach reliably outperforms the baselines is not statistically supported. At minimum, report the variance over seeds or a confidence interval, and provide the per-class F1 scores behind the Macro-F1.
minor comments (5)
- [Title and abstract] The title contains a spacing artifact in 'V erified'; the abstract and body use 'official', 'difficult', and 'sufficient' with non-standard ligature spacing. Please normalize typography.
- [§4] Several Arabic script examples are garbled in the PDF/HTML rendering (e.g., 'আॻ༟ ﻭً', 'ل؇ﺃ', 'ᆇᅫ', '݆݁ﺁ'). These should be fixed with proper Arabic text so the error analysis is readable.
- [§3 (Table 2)] The dagger marker on FANAR-9B is explained only as 'evaluated in a self-detection setting.' Clarify what that means and how it affects comparability, since self-detection may use different prompts or access to the model's own outputs.
- [§3, footnote 1] The evaluation dataset link should include a version/date and a statement about whether HalluTruthQA-4K is the same distribution as the HalluTruthQA benchmark used for training in Table 1.
- [§2.1] In the construction of ai, the notation mixes 'yi = Non-Hallucinated' and 'yi = Hallucinated' with the choice of gold option versus distractor. Clarify that the label comes first and the candidate answer is constructed accordingly, to avoid the impression that the label is derived from the option.
Circularity Check
HalluTruthQA appears in both the training set and the evaluation set; without a decontamination statement, the reported scores may reflect memorization rather than prediction.
specific steps
-
fitted input called prediction
[Table 1 (training data) and Section 3, Experiments, footnote 1]
"HalluTruthQA [Bouchekif et al., 2026b] Annotated QA/MCQ 600 600 Factuality labels, verified answers, and five plausible distractors. ... We conduct our experiments using the Islamic knowledge data released for HalluScoring 2026, Task 2.1 ... The dataset contains 600 Arabic question–answer instances ... Footnote 1: https://huggingface.co/datasets/Bekhouche/HalluTruthQA-4K"
The same HalluTruthQA resource supplies both the training rows and the evaluation instances. Table 1 gives 600 HalluTruthQA detection instances and 600 HalluTruthQA MCQ-selection instances as training data; Section 3 evaluates on 600 Islamic-knowledge instances from HalluTruthQA-4K, the official shared-task dataset. The paper never states that these 600 evaluation instances are disjoint from the HalluTruthQA rows used for fine-tuning. Because the HalluTruthQA rows already contain the exact factuality labels, verified answers, and five plausible distractors used in the task, any overlap means the reported label accuracy (0.935), Macro-F1 (0.928), option accuracy (0.895), and combined score (0.912) are fitted/memorized values rather than independent predictions. The claimed 'prediction' is t
full rationale
This paper is a shared-task system description, not a formal derivation, so most circularity patterns (self-definition, imported uniqueness, ansatz-by-citation, renaming) do not apply; there is no self-citation chain. The central empirical claim is a benchmark score. The specific circularity concern is that the same HalluTruthQA source appears on both sides of the train/test boundary: Table 1 lists 600 HalluTruthQA detection and 600 HalluTruthQA MCQ-selection training instances, while Section 3 evaluates on the official HalluTruthQA-4K Islamic-knowledge data (600 instances) with no statement that these sets are disjoint. Since the HalluTruthQA instances already contain the gold labels, verified answers, and distractors used by the task, any overlap would make the reported 0.935 label accuracy, 0.928 Macro-F1, 0.895 option accuracy, and 0.912 combined score an artifact of memorization rather than a measure of generalization. In addition, Table 2 compares the proposed option accuracy with the baselines' LO-Score, so the comparative claim is not directly supported even in the no-leak case. However, absent proof of overlap, this is a burden on the authors rather than a demonstrated equivalence; the score reflects that the paper provides no decontamination evidence for a critical boundary, not a formal derivation collapse.
Axiom & Free-Parameter Ledger
free parameters (1)
- Fine-tuned Gemma-4-12B-it model weights =
not reported (12B parameters)
axioms (3)
- domain assumption The official HalluScoring 2026 Arabic Islamic knowledge evaluation labels and verified answers are correct.
- ad hoc to paper The evaluation instances are disjoint from the HalluTruthQA instances used in training.
- domain assumption The task-specific adapters were trained with a configuration that supports the claimed system behavior.
Cite this review
Pith. "Pith review of Detecting Hallucinations and Recovering Verified Answers in Arabic Islamic Question Answering." pith.science (2026). https://pith.science/paper/HZT5LDZO
@misc{pith2026260803720,
author = {Pith},
title = {Pith review of: Detecting Hallucinations and Recovering Verified Answers in Arabic Islamic Question Answering},
year = {2026},
howpublished = {\url{https://pith.science/paper/HZT5LDZO}},
note = {Machine review of arXiv:2608.03720}
}
read the original abstract
Large language models can generate fluent responses to Islamic questions while introducing factual errors that are difficult to identify. This paper presents our system for \textsc{HalluScoring 2026} Task 2.1, \textit{Islamic Hallucination Detection and Find the Truth}. The task requires a unified two-step prediction: determining whether an Arabic answer generated by an LLM is hallucinated and selecting the verified answer from six closely related candidate options. We use the Islamic knowledge dataset provided by the shared task, which contains 600 question--answer instances, including 341 hallucinated and 259 non-hallucinated answers. Our system is based on the fine-tuned \texttt{google/gemma-4-12B-it} model and uses deterministic decoding during inference. The generated outputs are normalized to extract the hallucination label and the selected option. The system achieves a Macro-F1 score of 0.928 and a label accuracy of 0.935 for hallucination detection, together with an option accuracy of 0.895 for answer selection. These results yield a combined score of 0.912, demonstrating strong performance across both stages of the task. The lower option-selection accuracy indicates that distinguishing the verified answer from plausible alternatives remains more challenging than detecting hallucinated responses.
Reference graph
Works this paper leans on
-
[1]
Bouchekif, Abdessalam and Rashwani, Samer and Mohamed, Emad Soliman Ali and Alkhatib, Mutaz and Sbahi, Heba and Gaben, Shahd and Zaghouani, Wajdi and Erbad, Aiman and Ghaly, Mohammed , booktitle =. 2025 , address =. doi:10.18653/v1/2025.arabicnlp-sharedtasks.117 , url =
-
[2]
Bouchekif, Abdessalam and Gaben, Shahd and Rashwani, Samer and Eltanbouly, Somaya and Al-Khatib, Mutaz and Sbahi, Heba and Ghaly, Mohammed and Mohamed, Emad , journal =. 2026 , eprint =. doi:10.48550/arXiv.2603.07539 , url =
-
[3]
Alwajih, Fakhraddin and El Mekki, Abdellah and Mubarak, Hamdy and Hawasly, Majd and Mohamed, Abubakr and Abdul-Mageed, Muhammad , booktitle =. 2025 , address =. doi:10.18653/v1/2025.arabicnlp-sharedtasks.107 , url =
-
[4]
Bhatia, Gagan and Mubarak, Hamdy and Jarrar, Mustafa and Mikros, George and Zaraket, Fadi and Alhirthani, Mahmoud and al-Khatib, Mutaz and Cochrane, Logan and Darwish, Kareem Mohamed and Yahiaoui, Rashid and Alam, Firoj , booktitle =. From. 2026 , address =. doi:10.18653/v1/2026.findings-acl.1317 , url =
-
[5]
Bouchekif, Abdessalam and Zighem, Mohammed-En-Nadhir and Bekhouche, Salah Eddine and Telli, Hichem and Eltanbouly, Somaya and Gaben, Shahd and Sbahi, Heba and Rashwani, Samer and Al-Khatib, Mutaz and Mohamed, Emad and Ghaly, Mohammed and Hadid, Abdenour , journal =. 2026 , eprint =. doi:10.48550/arXiv.2607.20219 , url =
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2607.20219 2026
-
[6]
ACM Transactions on Information Systems , volume =
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions , author =. ACM Transactions on Information Systems , volume =. 2025 , publisher =. doi:10.1145/3703155 , url =
doi:10.1145/3703155 2025
-
[7]
arXiv preprint arXiv:2510.06265 , year =
Large Language Models Hallucination: A Comprehensive Survey , author =. arXiv preprint arXiv:2510.06265 , year =. doi:10.48550/arXiv.2510.06265 , url =
-
[8]
Proceedings of The Third Arabic Natural Language Processing Conference , pages=
Assessing large language models on islamic legal reasoning: Evidence from inheritance law evaluation , author=. Proceedings of The Third Arabic Natural Language Processing Conference , pages=
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.