REVIEW 4 major objections 5 minor 2 references
Benchmarking the Legal Reasoning of LLMs in Arabic Islamic Inheritance Cases
T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper shows that a majority-voting ensemble of three large language models reaches 92.7% accuracy on Arabic Islamic inheritance questions, topping all other tested models and finishing third in the QIAS 2025 challenge.
desk verdict Useful shared-task numbers for Arabic legal NLP, but the flagship superiority claim compares across different test/validation sets and needs a fix. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is majority voting over three diverse, high-capacity language models, each prompted with a chain-of-thought prompt (Prompt 2) and sampled deterministically. Each model independently outputs one of the six answer options A–F; the ensemble answer is the option chosen by at least two of the three. This mechanism is what lifts the final accuracy above the best single model. The paper's comparison also makes chain-of-thought prompting a second key ingredient: it explains most of GPT-4o's improvement over its simple prompt, while Gemini models appear to reason well under either prompt.
What would settle it
Using the official QIAS 2025 test split, compute accuracy by difficulty level for GPT o3, Gemini Flash 2.5, Gemini Pro 2.5, and the three-model majority vote with the same Prompt 2; if the ensemble fails to beat its best member on any difficulty, the headline claim fails. Additionally, test the same models on twenty fresh inheritance cases written from legal rules but not present in the fatwa corpus; high accuracy on benchmark questions and low accuracy on fresh ones would indicate memorization rather than legal reasoning.
Extended reading notes
Core claim
On its own terms, the paper's central claim is that a majority-voting ensemble of three proprietary base models—GPT o3, Gemini Flash 2.5, and Gemini Pro 2.5—outperforms every other configuration the authors tested on the QIAS 2025 Islamic inheritance reasoning task, at every difficulty level, achieving up to 92.7% accuracy on the official 1,000-question test set and third place in the shared task's Task 1. The paper also establishes that the same models individually score 88.4%, 88.1%, and 87.9% on that test set, while the best open-source Arabic model, Allam, reaches only 38.8% on validation. Fine-tuning results are mixed: GPT-4o improves from 70.1% to 86.6% when fine-tuned with a chain-of-
Load-bearing premise
The central comparison rests on the assumption that the official QIAS 2025 test labels and scoring are correct, and that the validation-set conditions under which models were compared match the test-phase conditions; if either fails, the reported ranking and the 92.7% ensemble result are unsupported.
Editorial extensions
If this is right
- The ensemble configuration is the best method in the paper, so future systems built for this benchmark should start from majority voting over strong general-purpose models rather than from open Arabic models.
- Fine-tuning a generalist model such as GPT-4o on domain MCQs can close a substantial knowledge gap, improving accuracy from 70.1% to 86.6%.
- Fine-tuning a reasoning-optimized model such as Gemini Flash 2.5 on answer-only labels can hurt performance (91.5% to 74.6%), implying reasoning chains in training labels matter.
- The 7.3% residual error on the best configuration means LLM-based inheritance calculators cannot yet operate without expert verification.
- The large gap between open Arabic models (best 38.8%) and proprietary models (over 88%) marks Islamic inheritance reasoning as a difficult benchmark for open Arabic language understanding.
Reading between the lines
- The paper does not give a per-difficulty breakdown in the reported text; the 'every difficulty level' claim would be directly testable by stratifying the official test set into beginner, intermediate, and advanced cases and checking whether majority voting beats every single model in each stratum.
- Since the benchmark questions come from a large fatwa corpus, a plausible non-reasoning explanation for the high proprietary scores is partial memorization; a test on newly authored cases absent from that corpus would separate legal computation from pattern recall.
- The fine-tuning degradation pattern suggests a broader principle: answer-only supervised data can be harmful for models optimized for chain-of-thought, so the recently released reasoning-annotated dataset version may change the fine-tuning conclusions if the same experiments are rerun.
- The ensemble's advantage probably depends on the diversity of the three models' errors; if confirmed, confidence-weighted voting or an adjudication step could push accuracy higher, but neither is tested in the paper.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates LLMs on the QIAS 2025 Arabic Islamic inheritance MCQ benchmark. It compares open-source Arabic models (Falcon3, Fanar, Allam) and proprietary models (GPT-4o, GPT o3, Gemini Flash/Pro 2.5), reports prompt sensitivity for two prompt variants, and fine-tunes several models (GPT-4o, Gemini Flash 2/2.5, Llama 4 Scout) using the training portion of the dataset. All main comparisons in Tables 1–3 are reported on the official 1,000-example validation set. The authors then evaluate three base models (GPT-o3, Gemini Flash 2.5, Gemini Pro 2.5) on the official 1,000-example test set and apply majority voting, obtaining 92.7% accuracy, which they state secured third place in the shared task. The abstract claims this majority-vote solution outperforms all other models 'across every difficulty level.'
Significance. If the central claims are supported, the paper would provide a useful, practical benchmark of LLM performance on a niche but important legal reasoning task, with public data from a recent shared task. The study has clear strengths: it uses a publicly available dataset with a standard validation/test split; it evaluates multiple open and proprietary models, including fine-tuned variants; it reports training hyperparameters; and it explicitly acknowledges the limitation of single-run evaluations. The proposed majority-voting ensemble is a simple, reproducible technique that shows a nontrivial gain over individual base models on the test set. However, the headline claim about superiority across every difficulty level and over 'all other models' is not backed by the reported experiments, as detailed below.
major comments (4)
- [Abstract and §3.3] The central claim—that the proposed majority-voting solution 'outperforms all other models ... across every difficulty level'—is unsupported by the evidence presented. Tables 1–3 report only overall validation-set accuracies; no per-difficulty accuracy is given for any model on either the validation set or the test set. The test-set paragraph reports only overall accuracies for three base models and the ensemble. Thus the phrase 'every difficulty level' has no in-paper evidence. The authors should either report per-difficulty test-set accuracies for the ensemble and for the comparison models, or remove/qualify the claim.
- [§3.3, Tables 1–3 and test paragraph] The comparison underlying 'outperforms all other models' mixes evaluation sets. The 86.6% fine-tuned GPT-4o, 84.3% fine-tuned Llama 4, and all Table 1–2 numbers are validation-set results, while the 92.7% ensemble figure is on the official test set. A cross-set comparison is not valid for concluding that the ensemble beats fine-tuned models, because the same models can behave differently across sets and the test-set performances of the fine-tuned models are not reported. The authors should evaluate all compared models on the same test set, or explicitly separate validation and test comparisons and avoid making a global superiority claim.
- [§3.3, ensemble construction] Gemini Pro 2.5 is included in the ensemble but has no validation-set accuracy in Table 1, so its individual performance relative to the other evaluated models is not documented. Moreover, the majority-voting procedure is underspecified: with three models and six choices, there can be cases with no majority, and no tie-breaking rule is described. The paper should report the number or percentage of consensus/majority cases, the tie-breaking rule, and ideally a repeated-run or bootstrap uncertainty estimate, especially since all results are based on a single run (as the authors themselves note in §3.3).
- [§3.2] Prompt 2, which is used for all base-model evaluations and for fine-tuning variants, is never quoted. Only Prompt 1 is given in full. Since Table 2 shows that GPT-4o's accuracy changes from 57.5% to 70.1% between Prompt 1 and Prompt 2, the exact wording of Prompt 2 is essential for reproducibility. The authors should include the full Prompt 2 text in the paper, an appendix, or a supplementary file.
minor comments (5)
- [Abstract and §3.3] The phrase 'secures third place overall in the challenge (Tas)' appears garbled; 'Tas' should likely be 'Task 1' or the parenthetical should be removed.
- [Table 2 discussion] The sentence '...significantly enhances the performance to 70.1%, thereby.' ends with a stray 'thereby.' and should be rewritten.
- [Naming consistency] The model is referred to as 'Allam think' in §3.2, 'Allam thinking' in Table 2, and 'Allam Think' elsewhere; please use a consistent name (e.g., 'ALLaM-Thinking').
- [References] Some references are incomplete or contain malformed URLs (e.g., the 'Isl' reference in Section 1, the 'qia, 2025' entry, and the CILE project URL). Please verify and complete all references.
- [Figures] Figures 1–3 are captioned only as 'Example 1/2/3'; adding descriptive captions that state the task format and what is being illustrated would improve readability.
Circularity Check
No significant circularity: the paper is a benchmark study with no fitted parameters or definitions that reduce to its predictions.
full rationale
This paper makes no mathematical derivation and fits no parameters to the test set. The central result is an empirical evaluation of LLMs on a shared task dataset. The majority-voting ensemble is not defined in terms of the target metric, and its test-set accuracy is not forced by the individual model accuracies used to build it. The validation set is used for model comparison and selection, which is standard benchmark practice, and the test-set evaluation is independent. The only notable concern is that the abstract claims the ensemble outperforms all other models 'across every difficulty level,' but Tables 1–3 report validation-set numbers while the 92.7% ensemble result is on the test set, and no per-difficulty test-set comparison is provided. That is a correctness/evidence concern, not circularity. The self-citations in the related work are not load-bearing for the main claim. Therefore, the circularity score is 0.
Assumptions & free parameters
assumptions (2)
- domain assumption The QIAS 2025 dataset provides correctly labeled questions with meaningful difficulty levels and a valid official test set.
- domain assumption The official test set accuracy reported by the challenge reflects a fair and consistent evaluation across all submitted models.
Cite this review
Pith. "Pith review of Benchmarking the Legal Reasoning of LLMs in Arabic Islamic Inheritance Cases." pith.science (2026). https://pith.science/paper/IC6FTYTI
@misc{pith2026250815796,
author = {Pith},
title = {Pith review of: Benchmarking the Legal Reasoning of LLMs in Arabic Islamic Inheritance Cases},
year = {2026},
howpublished = {\url{https://pith.science/paper/IC6FTYTI}},
note = {Machine review of arXiv:2508.15796}
}
read the original abstract
Islamic inheritance domain holds significant importance for Muslims to ensure fair distribution of shares between heirs. Manual calculation of shares under numerous scenarios is complex, time-consuming, and error-prone. Recent advancements in Large Language Models (LLMs) have sparked interest in their potential to assist with complex legal reasoning tasks. This study evaluates the reasoning capabilities of state-of-the-art LLMs to interpret and apply Islamic inheritance laws. We utilized the dataset proposed in the ArabicNLP QIAS 2025 challenge, which includes inheritance case scenarios given in Arabic and derived from Islamic legal sources. Various base and fine-tuned models, are assessed on their ability to accurately identify heirs, compute shares, and justify their reasoning in alignment with Islamic legal principles. Our analysis reveals that the proposed majority voting solution, leveraging three base models (Gemini Flash 2.5, Gemini Pro 2.5, and GPT o3), outperforms all other models that we utilized across every difficulty level. It achieves up to 92.7% accuracy and secures the third place overall in Task 1 of the Qias 2025 challenge.
Figures
Reference graph
Works this paper leans on
-
[2014]
Design of an expert system to calculate inher- itance shares based on islamic law. IEEE-African Journal of Computing & ICTs , 7(2):151–156. Koray Kavukcuoglu. 2025. Gemini 2.5: Our most intelligent ai model. https: //blog.google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025/ #gemini-2-5-thinking . Chen Wei Kuo, Kevin Chu, Nouar AlDa...
arXiv 2025
-
[2025]
Qias 2025 shared task. Accessed July 13, 2025. Zubair Abbasi. 2025. Augmented learning: Genera- tive artificial intelligence and islamic inheritance law. Accessed July 13, 2025. Nouar AlDahoul, Talal Rahwan, and Yasir Zaki. 2024a. Polytc: a novel bert-based classifier to detect polit- ical leaning of youtube videos based on their titles. Journal of Big Da...
arXiv 2025
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.