{"id":"fcba4c03-d187-4b7a-a96b-0a9deeaed63f","arxiv_id":"2506.01920","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"On a new 490-question Arabic depth dataset, Claude 3.5 Sonnet answered about 30 percent correctly, while GPT-4 answered about 9 percent, showing current models are weak on culturally specialized Arabic knowledge.","lead":"The authors build a 490-question Arabic evaluation set with hard, culturally grounded questions and test five commercial language models on it. They report that all models score low, with Claude 3.5 Sonnet best at roughly 30 percent accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The rankings in Tables 7-11 depend on reliability-unchecked manual labels with no released answer key; until the scoring process is reproduced, the 30% headline for Claude 3.5 Sonnet is unverified.","rationale":"I read the paper as proposing a culturally grounded Arabic benchmark and an initial measurement, not as a finished, fully validated benchmark. The theoretical guidelines in Section 3 and the detailed source list in Appendix A.1 are useful contributions, and the authors are transparent about several limitations in Section 6. The load-bearing weakness is exactly the one the reader identified: the headline ranking is generated by a small, unpublished, manually scored set of labels, and no reliability evidence is reported. My stress test sharpens this with two textual observations: the abstract's 30% and Table 5's 33.5% are not the same summary statistic, and footnote 4 mentions a judge LLM despite Section 5 claiming no LLM judge was used. These inconsistencies do not prove fraud or error; they strengthen the case that the measurement protocol is not yet pinned down. A reproducibility check with independent annotators is the single most direct way to settle whether the rankings are an artifact of one group's scoring judgment. Since the reader already reached CONDITIONAL and my analysis does not move the verdict, I keep the reader's verdict unchanged.","tokens_in":14748,"tokens_out":3528,"duration_ms":33843,"concrete_test":"Release ADMD with questions, gold answers, and a fixed scoring rubric, and archive the saved model outputs for each of the five models. Have two independent native-Arabic expert annotators who did not author the dataset score a complete model run (e.g., all 490 GPT-4 outputs) plus a stratified sample of 100 outputs from Claude 3.5 Sonnet, using only the released rubric. Compute per-label Cohen's kappa and recompute the category percentages and the overall ranking from each annotator's labels. If kappa is below 0.6, or if the Claude 3.5 Sonnet True rate shifts by more than ±3 percentage points relative to 147/490, the 30% headline and the model ranking are not robust to the scoring method. A minimal version of this check is to publish 20 scored examples with justifications; it would at least show whether the rubric is usable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the performance ranking in Tables 7-11 and the 30% figure for Claude 3.5 Sonnet. For that ranking to be meaningful, the assignment of each model output to True, False, Partially-True, or Partially-False must be reproducible. Section 5.1 defines these labels only by prose thresholds ('Partially-True corresponds to an answer 60-80% correct', 'Partially-False ... 20-30% correct'), yet the dataset, gold answer key, and the exact rubric applied by the four researchers are not released. No inter-annotator agreement, adjudication procedure, or worked scoring example is reported. The problem is concrete: for long free-form Arabic answers, a single factual error can plausibly be read as 100%, 60-80%, or 20-30% correct depending on the annotator, and these disagreements directly change the category percentages that produce the headline. The paper also self-reports inconsistently: footnote 4 says the most effective way to automate evaluation was found to be a judge LLM, while Section 5 says no LLM judge was used; either way, the clean 'manual review' premise is not established. Finally, the abstract reports 30% overall for Claude 3.5 Sonnet while Table 5 reports 33.5% as the average over categories; if the headline number is not uniquely defined, the same is true of the ranking it supports. None of this proves the ranking is wrong; it means the measurement is not yet verifiable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework of theoretical guidelines for constructing Arabic evaluation datasets, audits three existing benchmarks (Ghafa, ArabicMMLU, INCLUDE) against those guidelines, and introduces a new 490-question Arabic Depth Mini Dataset (ADMD) spanning ten domains and 42 subdomains. Using ADMD, the authors manually scored outputs from five LLMs (GPT-4, Claude 3.5 Sonnet, Gemini Flash 1.5, CommandR 100B, Qwen-Max) into four categories (True, False, Partially-True, Partially-False). They report that Claude 3.5 Sonnet achieves the highest accuracy, with 147 correct answers (30%) and particular strength in Mathematics & Computational Sciences, Philosophy & Logic, and General & Miscellaneous Sciences, while all models struggle on culturally nuanced topics.","tokens_in":15002,"tokens_out":2893,"duration_ms":32543,"significance":"If the evaluation is reproducible, ADMD would be a useful culturally grounded Arabic evaluation resource, addressing a real gap in a field dominated by translated or surface-level benchmarks. The paper also makes a constructive contribution by articulating explicit linguistic, cultural, and methodological guidelines and by providing a structured comparison of existing datasets. However, the central empirical claim — the model ranking in Tables 7–11 and the 30% headline for Claude 3.5 Sonnet — currently rests on a non-public dataset, a manual grading process with no reported inter-annotator agreement, and internally inconsistent statements about whether an LLM judge was used. These issues make the headline results unverifiable as presented, so the significance of the contribution cannot yet be assessed.","major_comments":[{"comment":"The reported headline accuracy for Claude 3.5 Sonnet is internally inconsistent. The abstract and §5.1 state 147 questions (30%), but Table 5 reports the model's average True rate as 33.5%. The difference matters because categories have unequal sizes (most have 10 questions but General Linguistics and General Sciences have 50), so an unweighted category average can differ substantially from the overall percentage. The paper should state which quantity is reported, provide total counts for all models, and report the overall percentage alongside the category average. Without this clarification, the headline ranking is ambiguous.","section":"Abstract, §5.1, Table 5"},{"comment":"The model rankings in Tables 7–11 depend entirely on the manual assignment of each model answer to True, False, Partially-True, or Partially-False, but the scoring process is not reproducible from the manuscript. The labels are defined only by broad percentage bands (60–80% correct, 20–30% correct) with no worked examples, no scoring rubric, no adjudication procedure, and no inter-annotator agreement measure. The full dataset and answer key are also not released; Appendix A.2 provides only sample examples via a link. To support the central claim, the authors should release the dataset with gold answers, provide the complete rubric, and report agreement statistics (e.g., Cohen's kappa) from at least two independent annotators on a representative sample.","section":"§5, §5.1, Appendix A.2"},{"comment":"The manuscript contradicts itself about whether an LLM judge was used. §3.4 states that the authors developed a Python library that leverages the Claude Sonnet model to automate dataset evaluation, and footnote 4 says that 'the most effective way to automate the evaluation is by using a judge LLM.' In contrast, §5 states that 'We did not use LLM Judge in this paper' and justifies manual evaluation. These statements cannot all be correct for the same evaluation pipeline. The authors need to clarify what the Claude-based library was used for (e.g., dataset construction, quality review, or final scoring) and state explicitly which stages used human annotators versus an LLM, since the reader's interpretation of Tables 7–11 changes accordingly.","section":"§3.4, §5, footnote 4"},{"comment":"The review of existing datasets is based on a 'representative sample' manually evaluated by 'one of the authors' for each dataset, but no sample sizes, selection criteria, or scoring validation are reported. For example, Table 2 reports Ghafa's Language Rules score as 4.5 and Table 4 reports INCLUDE's Information Correctness as 7.0, yet without knowing how many items were sampled and how the scores were aggregated, these numbers cannot be verified or compared. Since these critiques motivate the need for ADMD, the authors should report the sampling procedure, the number of items reviewed per dataset, and ideally a second annotator for reliability.","section":"§4, Tables 2–4"}],"minor_comments":[{"comment":"The phrase '(42 sub-domains, see Figure 1' is missing a closing parenthesis; it should read 'see Figure 1).'","section":"Abstract"},{"comment":"The paper alternately attributes ArabicMMLU to 'OpenAI (2024)' (in the Introduction and §4.2) and to Koto et al. (2024) in the Related Works and references. The citation should be unified to the correct source.","section":"§2 and §4.2"},{"comment":"Several Arabic examples appear garbled in the rendered text (e.g., the Quranic verse example in §4.1 and the spelling-error examples in §4.3). Since the Arabic quality is a central claim, these excerpts must be typeset correctly.","section":"§4.1–§4.3"},{"comment":"The caption 'first 10 rows per sheet, or 50 for long sheets' is unclear about the actual number of questions per subject; the text elsewhere says each topic has 10 questions except General Language and Diversified Science, which have 50, so the caption should state this directly.","section":"Table 6"},{"comment":"The reference list for ADMD sources is useful but incomplete: several entries give only URLs (e.g., the Dialects entries), and two entries appear as fragments ('Diwan al-Lugha al-'Arabiyya' and 'Hindawi Science Collection') without full bibliographic details.","section":"§A.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript overlaps significantly with the authors' own prior work (Nacar et al., 2025) on culturally aligned Arabic evaluation, and the current paper's guidelines are presented as inspired by that work. This is not a technical flaw, but the novelty boundary between ILMAAM and ADMD should be checked by the handling editor. The central issue remains the unreleased dataset and unvalidated manual scoring; the authors should be asked to provide the data and reliability evidence before the ranking can be taken as established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. First, the Arabic Depth Mini Dataset (ADMD) is the real contribution: 490 handwritten questions drawn from scholarly Arabic sources, spanning ten domains and 42 subtopics. That is new content, not a translation or harvested set. Second, the paper's headline result—Claude 3.5 Sonnet leading at 30% overall, with GPT-4 trailing—rests on manual scoring that is not yet reproducible. Until the data and answer key are out, treat the rankings as provisional.\n\nWhat the paper does well is the critical review of existing Arabic benchmarks. The authors walk through concrete examples from Ghafa, ArabicMMLU, and INCLUDE, showing translation errors, cultural misalignment, and factual mistakes. That is useful and grounded. Their theoretical guidelines for dataset construction are sensible, even if partly drawn from their earlier ILMAAM paper; they are practical and actually applied. The selection of primary sources for ADMD—classical texts in fiqh, hadith, grammar, and poetry—is thoughtful and culturally grounded.\n\nThe soft spots are concentrated in Section 5. The scoring rubric defines partial categories by vague thresholds (60–80% correct), with no worked example, no inter-annotator agreement, and no adjudication procedure. The dataset is not released; only a sample appears via Google Sheets. The abstract's 30% for Claude does not match Table 5's 33.5% category average for the same model, and the paper doesn't reconcile them. A footnote saying a judge LLM was the most effective automated evaluation method sits awkwardly with the text claiming no LLM judge was used. Collectively, these problems mean the measurement is not verifiable as reported. They do not prove the ranking is wrong, and the limitations section shows some awareness.\n\nThis paper is for people working on Arabic LLM evaluation. They will find the dataset idea valuable and the critique of existing benchmarks instructive. But it should be read as a research proposal with preliminary evidence, not a finished benchmark. With full release of the dataset, answer key, and scoring protocol, it could become a solid reference. I'd give it a serious referee; the issues are substantial but repairable.","headline":"A genuinely new Arabic evaluation set with a load-bearing reproducibility gap: the ADMD dataset is worth attention, but the model rankings are unverified until the data and scoring are released.","tokens_in":15580,"tokens_out":3029,"would_cite":false,"duration_ms":29228,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Top LLMs answer just 30% of a new hard Arabic test","keywords":["Arabic LLM evaluation","Arabic Depth Mini Dataset","cultural alignment","benchmark critique","large language models","manual evaluation","Arabic NLP","domain-specific benchmarks"],"falsifier":"Have two independent teams of native Arabic experts score the same 490 model responses using the paper's four labels and compare their agreement; if agreement falls below standard reliability thresholds, the reported rankings (Claude 3.5 Sonnet 30%, GPT-4 11.8%) are not reproducible. A simpler check is to release the answer key and rerun the five models on ADMD to see whether the accuracy numbers match.","tokens_in":14515,"feed_emoji":"📊","tokens_out":5301,"duration_ms":47271,"temperature":0.7,"pith_summary":"The paper argues that current Arabic evaluation benchmarks overstate model competence because they lean on translated, Western-centric, or linguistically sloppy content. To make the point concrete, the authors build the Arabic Depth Mini Dataset (ADMD): 490 hand-curated questions drawn from Arabic primary sources across ten domains, and they score five leading LLMs on it. Under strict manual scoring, the best model (Claude 3.5 Sonnet) answers only 30% of questions correctly, and GPT-4 only about 12%. The paper also proposes a set of theoretical guidelines—linguistic accuracy, cultural alignment, source validation, and evaluator expertise—that it says future Arabic datasets must follow to be credible.","feed_headline":"Top LLMs answer just 30% of a new hard Arabic test","feed_subtitle":"A hand-curated 490-question Arabic benchmark finds GPT-4 weakest and all models failing on Islamic studies.","key_machinery":"The load-bearing object is the Arabic Depth Mini Dataset (ADMD), 490 questions organized into ten major domains and 42 sub-domains, each question sourced from an Arabic primary reference such as Tafsir Ibn Jarir al-Tabari, Lisan al-Arab, or Hindawi science books. The scoring procedure is the second mechanism: four researchers manually label each model response as True (100% correct), Partially-True (60–80%), Partially-False (20–30%), or False, and the same four-criteria rubric (Language Rules, Scientific Writing, Cultural Values, Information Correctness) is used to critique existing datasets like Ghafa, ArabicMMLU, and INCLUDE. The guidelines plus the dataset together form the argument: because the questions demand cultural depth instead of surface pattern matching, performance on ADMD measures something that MMLU-style benchmarks miss.","core_discovery":"On its own terms, the paper's central discovery is that expert-level Arabic cultural knowledge is almost entirely missing from current LLMs, and that this is hidden by existing benchmarks. Using the ADMD, Claude 3.5 Sonnet achieved the highest accuracy at 30% (147/490), with relative strengths in Mathematics & Computational Sciences (50%), Philosophy & Logic (50%), and General & Miscellaneous Sciences (51.67%), but near-total failure in Historical & Genealogical Studies (0%) and weak performance in Islamic & Religious Studies despite its relative lead. GPT-4 scored only 11.8% True, with the highest False rate. The paper reads these results as evidence that translated and automatically generated Arabic datasets produce misleadingly optimistic evaluations, and that culturally grounded, depth-oriented questions are needed to reveal actual model limitations.","pith_inferences":["If the ADMD answer key and scoring rubric were released with inter-annotator agreement numbers, the benchmark could serve as a reproducible stress test for Arabic NLP, but as published the headline scores cannot be independently verified.","The 30% ceiling may partly reflect the strict True threshold (100% correct); a reader comparing these numbers with accuracy on MMLU-style benchmarks should treat them as different quantities, not the same measure applied to a harder test.","A natural testable extension would be to measure whether Arabic fine-tuned models (e.g., Jais, Allam, Fanar, Aya) improve on ADMD without improving on translated benchmarks, which would isolate cultural knowledge from general reasoning.","Because physics, chemistry, and advanced mathematics are excluded and medicine is thin, the domain ranking is a lower bound on the difficulty gap; adding those fields could shift the relative standings."],"forward_implications":["If ADMD reflects genuine Arabic expert knowledge, then the frontier LLMs tested here are not yet dependable for Arabic cultural, religious, and specialist domains.","Existing benchmarks such as ArabicMMLU and Ghafa can no longer be read as evidence of Arabic competence, since large fractions of their samples fail basic language and cultural checks.","Model rankings are domain-dependent: Claude 3.5 Sonnet leads in mathematics and logic but still fails Islamic studies, so a single aggregate score hides practically meaningful differences.","Future Arabic dataset builders should follow the proposed guidelines—primary-source attribution, cultural alignment, and evaluator expertise—or risk reproducing inflated results.","Manual evaluation remains necessary for non-English tasks, since the paper cites evidence that LLM judges are unreliable for Arabic."],"supporting_citations":[{"why":"Supplies the Ghafa dataset whose sampled items are scored low on language rules and cultural values, serving as evidence that existing benchmarks are flawed.","marker":"(Almazrouei et al., 2023)"},{"why":"The ArabicMMLU benchmark under review, cited for translation and cultural misalignment issues.","marker":"(OpenAI, 2024)"},{"why":"The original ArabicMMLU companion that provides 57-category coverage which ADMD contrasts with domain depth.","marker":"(Koto et al., 2024)"},{"why":"The original English MMLU that ArabicMMLU adapts, defining the surface-level knowledge format ADMD criticizes.","marker":"(Hendrycks et al., 2020)"},{"why":"The INCLUDE dataset whose Arabic subset is reported to have 70% spelling errors, evidence of quality gaps in multilingual benchmarks.","marker":"(Romanou et al., 2024)"},{"why":"Prior critique of ArabicMMLU that inspires the paper's theoretical guidelines.","marker":"(Nacar et al., 2025)"},{"why":"Observation that manual evaluation is better for non-English tasks, justifying ADMD's scoring method.","marker":"(Wu et al., 2025)"}],"fun_headline_variants":["Arabic cultural quiz stumps top LLMs at 30% accuracy","New 490-question Arabic benchmark: Claude 3.5 tops at 30%","LLMs fail Arabic depth test: GPT-4 scores 11.8%, Claude 30%","Cultural blind spot: LLMs score 0% on Arabic genealogy test","Arabic baseline reveals LLMs lack expert cultural knowledge"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire model ranking rests on the authors' own manual judgments of what counts as True, Partially-True, Partially-False, or False, and those judgments have not been checked for inter-annotator agreement or made public as a full answer key.","fun_headline_variants_meta":{"raw":{"variants":["Arabic cultural quiz stumps top LLMs at 30% accuracy","New 490-question Arabic benchmark: Claude 3.5 tops at 30%","LLMs fail Arabic depth test: GPT-4 scores 11.8%, Claude 30%","Cultural blind spot: LLMs score 0% on Arabic genealogy test","Arabic baseline reveals LLMs lack expert cultural knowledge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000643,"raw_usage":{"total_tokens":2935,"prompt_tokens":898,"completion_tokens":2037,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":1937}},"tokens_in":514,"tokens_out":2037,"duration_ms":15496,"temperature":1.0,"reasoning_tokens":1937,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:31:33.248473+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two independent teams of native Arabic experts score the same 490 model responses using the paper's four labels and compare their agreement; if agreement falls below standard reliability thresholds, the reported rankings (Claude 3.5 Sonnet 30%, GPT-4 11.8%) are not reproducible. A simpler check is to release the answer key and rerun the five models on ADMD to see whether the accuracy numbers match.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The ArabicMMLU benchmark under review, cited for translation and cultural misalignment issues."},{"cited_title":"Al-Batati, Arwa Alsehibani, Nour Qandos, Omar Elshehy, Mohamed Abdelkader, and Anis Koubaa","cited_arxiv_id":null,"evidence_quote":"Prior critique of ArabicMMLU that inspires the paper's theoretical guidelines."}],"review_version":1}