{"id":"65dc5b16-e72a-484b-9fa8-35b8bf338a33","arxiv_id":"2509.00457","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An encoder-based relevance-scoring system achieves 69.87% accuracy on Islamic inheritance multiple-choice questions, below Gemini's 87.60% but with far smaller compute.","lead":"A small Arabic text encoder paired with Attentive Relevance Scoring gets 69.87% on QIAS 2025 Islamic inheritance questions, while the best Gemini LLM reaches 87.60%. The paper maps the accuracy-versus-deployability trade-off for high-stakes legal reasoning on devices.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central efficiency/deployability claim rests on asserted comparisons, not measured deployment or non-neural baselines.","rationale":"The reader's weakest-assumption analysis focuses on the semantic-relevance modeling premise: that ranking by relevance without explicit symbolic calculation caps accuracy at 69.87%. That is a real limitation, but the paper explicitly acknowledges it ('our approach does not perform explicit, step-by-step symbolic reasoning') and positions the model as an assistive tool rather than an autonomous reasoner. It therefore does not threaten the measured accuracy claim. The more load-bearing gap for the central claim is the efficiency/deployability/privacy portion: the paper gives parameters and GFLOPs but never measures the actual properties that make the case 'compelling.' Privacy is structurally plausible for a local model, but on-device deployability and efficiency are empirical claims that require latency, memory, and energy measurements, and ideally a comparison with a non-neural baseline. The absence of such measurements is a missing-support issue, not a refutation. The paper does provide honest limitation statements, a clear experimental setup, and a comparison across two LLM inference strategies, which are positive elements. A CONDITIONAL verdict remains appropriate: accept the reported test accuracy as a point estimate, but require deployment measurements and a lightweight baseline before the broader trade-off claim is accepted. Since this matches the reader's verdict, I recommend no change.","tokens_in":6367,"tokens_out":5234,"duration_ms":69663,"concrete_test":"Implement a small, hand-coded Islamic inheritance solver using fixed-share tables with 'awl/radd adjustments, and run it on the official QIAS 2025 test set via the evaluation server. Also measure wall-clock latency, peak RAM, and model size on a representative edge device (e.g., Raspberry Pi 4 or mid-range phone) for both this solver and the MARBERT+ARS pipeline. If the rule-based solver reaches or exceeds 69.87% accuracy while using orders of magnitude less memory and lower latency, the paper's efficiency/deployability case for the neural framework loses force. If the solver falls short or cannot parse the questions, the claim is strengthened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim—'presenting a compelling case for efficiency, on-device deployability, and privacy'—is load-bearing but never directly measured. Section 3.2 and Table 1 report only parameter counts and GFLOPs; there is no latency, peak-memory, energy, or edge-device benchmark. Section 3.3 asserts that API-based LLMs' 'high computational requirements prevent direct deployment on edge devices' without measuring those requirements or comparing them to the proposed system on the same hardware. The paper also omits any lightweight non-neural baseline; Section 3.3 explicitly defers TF-IDF/cosine to future work. If a deterministic faraid calculator or a TF-IDF retriever obtained accuracy at or above 69.87% with far fewer resources, the claimed trade-off versus API LLMs would not be compelling. This is a missing-support gap in the central efficiency argument, not an internal inconsistency. The paper honestly acknowledges its accuracy limitation, but the 'compelling case' part of the central claim requires deployment evidence that is absent.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes a multiple-choice Islamic inheritance QA system for the QIAS 2025 shared task. The architecture combines a pretrained Arabic BERT-style encoder with an Attentive Relevance Scoring (ARS) module that scores candidate answers by learned semantic relevance to the question, without performing explicit calculations. The authors evaluate five Arabic encoders, report MARBERT as best with 69.87% test accuracy, and compare against API-based LLMs, where Gemini-2.5-flash with single-question prompting reaches 87.60%. The paper frames the lower accuracy as a trade-off for efficiency, on-device deployability, and privacy.","tokens_in":6655,"tokens_out":5848,"duration_ms":72189,"significance":"The paper's strengths are its evaluation on an external shared-task benchmark and its honest statement that the method does not perform symbolic reasoning; the comparison also documents a large effect of batched versus single-question prompting on LLM accuracy, which is a useful evaluation caution. If the efficiency claims were properly measured, a 163M-parameter encoder with 69.87% accuracy would be a practically relevant assistive baseline. However, the central 'compelling case' for efficiency, deployability, and privacy rests on unmeasured assertions, and the contribution of the ARS component is not isolated. The reported accuracies are plausible but lack uncertainty quantification and clearer provenance for the test labels. The paper is a reasonable shared-task system description, but the advertised trade-off claim needs substantial additional support.","major_comments":[{"comment":"The abstract's central claim of a 'compelling case for efficiency, on-device deployability, and privacy' is not supported by measurements. Table 1 reports only parameter counts and GFLOPs. There is no latency, peak-memory, energy, or edge-device benchmark for the proposed system, and no measurement of the API LLMs' resource footprint. The assertion that LLMs' 'high computational requirements prevent direct deployment on edge devices' is stated without evidence. Section 3.3 also defers TF-IDF/cosine baselines to future work, so the claimed efficiency advantage over non-neural alternatives is untested. Please add deployment measurements or temper the claim to parameter/flop efficiency.","section":"§3.3, Table 1"},{"comment":"The test split is described as having 'no labels', yet test accuracies are reported. It must be clarified whether these numbers come from the official QIAS evaluation (e.g., the shared-task leaderboard) or from a local held-out subset. Without this, the central accuracy comparison is not reproducible.","section":"§3.1, Table 1"},{"comment":"No confidence intervals or significance tests are provided. On a 1,000-example test set, the difference between MARBERT (69.87%) and AraBERT (68.46%) is about 14 correct answers, well within sampling noise. Report bootstrap confidence intervals and pairwise significance tests before claiming MARBERT is the best encoder or that the gap to 87.6% is a meaningful trade-off.","section":"§3.3, Table 1"},{"comment":"ARS is presented as the novel component but is cited to the authors' previous preprint; no derivation, implementation details, or ablation is included here. Since the contribution is the 'Attentive Relevance Scoring mechanism', an ablation replacing ARS with plain cosine similarity or [CLS] dot product is necessary to show that ARS, not the encoder, is responsible for the reported accuracy.","section":"§2.2, Eqs. (2)-(4)"},{"comment":"The framing 'performance vs deployability' conflates model size with the absence of exact calculation. All encoder models share the semantic-relevance premise, so the 69.87% ceiling may be due to that premise rather than to resource constraints. Report accuracy by difficulty level (Beginner/Intermediate/Advanced) and an error analysis (e.g., near-miss numeric answers vs wrong heirs) to support the claim that the gap is a resource trade-off; otherwise, the LLM comparison is not a clean test of the paper's thesis.","section":"§2, §3.3"}],"minor_comments":[{"comment":"Eq. (1) writes Norm(·) without defining it. The text says ℓ2 normalization, but the equation should be explicit or refer to the text.","section":"§2.1"},{"comment":"The value of h (shared hidden dimensionality) in Eq. (2) is never reported. Please give the ARS hidden size used in the experiments.","section":"§2.2"},{"comment":"The test set contains only Beginner and Advanced items (no Intermediate). This distribution shift should be acknowledged, as the reported accuracy may not reflect performance on Intermediate-level questions.","section":"§3.1"},{"comment":"Clarify in the table caption that the asterisk denotes single-question inference. The text explains it, but the caption should be self-contained.","section":"Table 2"},{"comment":"Several entries contain 'and 1 others' (Bubeck et al., DeepSeek, Gemini Team). Use standard 'et al.' formatting.","section":"References"},{"comment":"The Arabic text in Figure 1 appears garbled or reversed In the PDF. Provide a clean, legible version.","section":"Figure 1"},{"comment":"The experimental setup omits batch size, number of epochs, and early-stopping criteria. Add these details for reproducibility.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a shared-task system description, but the main abstract claim overreaches the evidence. The ARS self-citation is not problematic per se, but the missing ablation and the lack of non-neural baselines make the contribution hard to evaluate. I would encourage a revision that either adds the requested measurements or substantially tempers the efficiency/deployability/privacy claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a straightforward shared-task paper: take an Arabic encoder, bolt on the authors' prior ARS scoring head, train on QIAS 2025, report test accuracy. The headline number, MARBERT+ARS at 69.87%, is plausible, and the LLM comparison is useful. The most interesting empirical nugget is Gemini-2.5-flash jumping from 68.65% to 87.60% when you switch from a 50-question batch to single-question prompts. That is a real finding for anyone evaluating LLMs on this kind of benchmark.\n\nWhat the paper does well: it is honest about its own ceiling. The authors repeatedly say the model does not do step-by-step symbolic reasoning, that it ranks by semantic relevance, and that it should be an assistive tool, not a decision-maker. Those caveats are not buried. The encoder comparison is clean and the test set is external.\n\nWhere the soft spots are: the abstract's 'compelling case for efficiency, on-device deployability, and privacy' is asserted, not demonstrated. Table 1 gives parameters and GFLOPs, but there is no latency, peak memory, power draw, or edge-device benchmark. The paper says LLMs 'require more resources' but never measures them on the same hardware. It also explicitly defers TF-IDF or a deterministic faraid calculator to future work. If a rule-based calculator gets 90% with zero trained parameters, the trade-off story collapses. That gap is real, but it is a missing-support gap, not a fatal flaw, and the paper conditions most conclusions on 'resource-constrained scenarios.'\n\nTwo smaller issues: the LLM comparison is not apples-to-apples because Gemini-2.5-flash* used a different inference protocol, and there are no error bars or significance tests across seeds. For a shared-task workshop paper this is acceptable, but it limits the strength of any ranking claims.\n\nOverall: the measured accuracy is credible, the main limitation is stated up front, and the batched-vs-single result is worth remembering. The efficiency claim needs a follow-up with actual measurements. The paper deserves a serious referee for the shared task venue; it is not a strong candidate for a broad NLP journal.\n\nRecommendation: send to peer review, ask for a revision that either adds deployment measurements or tones down the 'compelling case' language.","headline":"Honest shared-task results for a specialized Arabic legal QA domain, but the 'compelling case' for efficiency is asserted, not measured.","tokens_in":7135,"tokens_out":1779,"would_cite":false,"duration_ms":18395,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A lightweight Arabic text encoder combined with attentive relevance scoring answers Islamic inheritance multiple-choice questions at 69.87% accuracy, quantifying the tradeoff between on-device efficiency and the higher accuracy of cloud LLM","keywords":["Arabic NLP","Islamic inheritance law","multiple-choice QA","MARBERT","attentive relevance scoring","on-device inference","legal reasoning","low-resource NLP"],"falsifier":"Construct or select a QIAS-level question where two answer options name the same heirs and differ only in the numeric share, with the wrong share phrased in more familiar inheritance vocabulary. If the MARBERT+ARS model consistently assigns the wrong share a higher relevance score, then semantic relevance alone is insufficient and the framework's core premise fails for that class of cases.","tokens_in":6310,"feed_emoji":"⚖️","tokens_out":5868,"duration_ms":64302,"temperature":0.7,"pith_summary":"This paper tries to establish that a small, specialized Arabic-language model can be a practical assistive tool for Islamic inheritance law without running a generative large language model. The authors combine a pre-trained Arabic encoder (MARBERT is best) with an Attentive Relevance Scoring module that ranks multiple-choice answers by semantic closeness to the question, reaching 69.87% test accuracy on the QIAS 2025 benchmark. The number matters because the best API-based LLM in their comparison reaches 87.60% but requires cloud access, larger context, and incurs cost, latency, and privacy exposure. The paper's claim is that this gap is a deliberate design tradeoff: a roughly 163-million-parameter model can run on-device and still capture enough legal and numerical nuance to support experts, as long as a human remains in the loop.","feed_headline":"Small Arabic model scores 69.87% on inheritance test","feed_subtitle":"A 163M-parameter semantic ranker reaches 69.87% on-device accuracy, versus 87.60% for cloud LLMs.","key_machinery":"Attentive Relevance Scoring (ARS) is the core mechanism: it projects the question and answer [CLS] embeddings into a shared space, combines them by element-wise multiplication, applies tanh, and uses a learned attention vector to produce a sigmoid relevance score. The same module is trained with a composite objective (InfoNCE contrastive loss, dynamic relevance supervision on the final scores, and a logit-variance regularizer). This mechanism replaces the generative reasoning of an LLM with a lightweight ranking step, which is what makes on-device inference possible and what caps the model's accuracy when exact arithmetic is required.","core_discovery":"On the paper's own terms, the central discovery is that an encoder-only system can solve a substantial fraction of Islamic inheritance multiple-choice questions by semantic relevance alone, without explicit step-by-step symbolic calculation. Using MARBERT's [CLS] embeddings and an Attentive Relevance Scoring mechanism trained with contrastive, dynamic relevance, and logit-regularization losses, the model reaches 77.32% validation and 69.87% test accuracy on QIAS 2025 SubTask 1. In the same evaluation, Gemini-2.5-flash with single-question prompting achieves 87.60% accuracy, while batched multi-question prompts degrade it to 68.65%. The paper interprets these numbers as evidence of a real tra","pith_inferences":["A level-wise evaluation would probably show the model is much stronger on simple share identification than on radd and awl redistribution; the reported 69.87% mixes 500 Beginner and 500 Advanced test questions, so difficulty composition matters when interpreting the number.","If semantic relevance is the real bottleneck, adding a deterministic share calculator that parses heir types and applies Quranic fractions could lift accuracy above the LLM baseline at a fraction of the compute.","The single-versus-batched LLM gap suggests published LLM numbers on legal reasoning tasks may be understated when prompts are crowded; this is a methodology caution, not a result in the paper.","The same ARS ranking scheme could be tested on other rule-based legal or financial domains, such as zakah or contract shares, with minimal changes."],"forward_implications":["A 163-million-parameter MARBERT-based model with ARS can answer inheritance MCQs at 69.87% accuracy without generative reasoning, making it deployable on phones and edge devices.","The best cloud LLM result (Gemini-2.5-flash with single-question prompting) reaches 87.60%, so the tradeoff is roughly 18 accuracy points in exchange for cloud dependence, cost, latency, and privacy exposure.","Because the model does not perform explicit step-by-step symbolic calculation, it is positioned as an assistive tool for legal experts rather than an autonomous decision-maker.","LLM accuracy depends heavily on inference structure: batched 50-question prompts lowered Gemini from 87.60% to 68.65%, so system comparisons must control this variable.","Closing the gap likely requires adding symbolic reasoning or a hybrid local-plus-cloud design, which the paper names as future work."],"supporting_citations":[{"why":"Supplies MARBERT, the Arabic encoder that achieves the best 69.87% test accuracy in the framework.","marker":"Abdul-Mageed et al., 2020"},{"why":"Defines Attentive Relevance Scoring and its training objective, the core ranking mechanism.","marker":"Bekhouche et al., 2025"},{"why":"Provides the QIAS 2025 SubTask 1 dataset and task definition used for training and evaluation.","marker":"Bouchekif et al., 2025a"},{"why":"Provides empirical evidence of LLM difficulty on Islamic inheritance that motivates a specialized lightweight model.","marker":"Bouchekif et al., 2025b"},{"why":"Is the Gemini LLM family whose single-question prompting reaches the 87.60% accuracy baseline.","marker":"Team et al., 2023"},{"why":"Is DeepSeek-v3, one of the API-based LLMs compared in the batched evaluation.","marker":"Liu et al., 2024"}],"fun_headline_variants":["Efficient Arabic encoder scores 69.87% on inheritance reasoning","Lightweight model vs cloud LLMs: 69.87% vs 87.6% accuracy","Efficiency over peak accuracy: inheritance solver hits 69.87%","Encoder-only model solves inheritance questions at 69.87%"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The method assumes the correct multiple-choice answer is the one with the highest semantic relevance to the question, even though the task requires exact proportional calculations; if semantic closeness cannot separate a correct share from a plausible wrong one, accuracy is capped by that premise.","fun_headline_variants_meta":{"raw":{"variants":["Efficient Arabic encoder scores 69.87% on inheritance reasoning","Lightweight model vs cloud LLMs: 69.87% vs 87.6% accuracy","Efficiency over peak accuracy: inheritance solver hits 69.87%","Encoder-only model solves inheritance questions at 69.87%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000492,"raw_usage":{"total_tokens":2258,"prompt_tokens":748,"completion_tokens":1510,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":1428}},"tokens_in":492,"tokens_out":1510,"duration_ms":11369,"temperature":1.0,"reasoning_tokens":1428,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:32:01.646049+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct or select a QIAS-level question where two answer options name the same heirs and differ only in the numeric share, with the wrong share phrased in more familiar inheritance vocabulary. If the MARBERT+ARS model consistently assigns the wrong share a higher relevance score, then semantic relevance alone is insufficient and the framework's core premise fails for that class of cases.","supporting_citations":[{"cited_title":"ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic","cited_arxiv_id":"2101.01785","evidence_quote":"Supplies MARBERT, the Arabic encoder that achieves the best 69.87% test accuracy in the framework."},{"cited_title":"Enhanced Arabic Text Retrieval with Attentive Relevance Scoring","cited_arxiv_id":"2507.23404","evidence_quote":"Defines Attentive Relevance Scoring and its training objective, the core ranking mechanism."}],"review_version":1}