{"id":"45e4356f-2f68-4507-970a-2ba46f4be754","arxiv_id":"2501.00559","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"AraSTEM, a new native Arabic STEM benchmark of 11,637 multiple-choice questions, yields at most 56% accuracy for the best open-source model tested.","lead":"AraSTEM is a new Arabic-language benchmark of 11,637 multiple-choice STEM questions spanning primary to college level. It reports that even the best open-source models reach only about 56% accuracy, showing large room for improvement in Arabic scientific knowledge.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark validity depends on unverified assumption that questions remain answerable after image/equation removal and that gold labels are correct; without human-expert validation, low model accuracy could reflect broken prompts rather than knowledge gaps.","rationale":"The reader's weakest assumption correctly identifies the fragility of the paper's central claim: after removing images and equations, questions must remain self-contained and gold labels must be correct for low model accuracy to signify a knowledge gap. My stress-test confirms this as the most load-bearing condition, and adds two concrete supporting details: the ChatGPT-4 extraction was validated only as OCR/format, not for answer correctness, and the reported averages for the best model are internally inconsistent (0.58 in Table 4 vs 0.56 in Table 5 vs 56% in the abstract). These issues do not prove the benchmark is invalid, but they do mean the stated findings are not yet trustworthy. The proposed human-expert sample study would directly settle whether the concern lands. Because the reader already assigned a CONDITIONAL verdict and this concern supports that judgment, no verdict adjustment is needed.","tokens_in":11180,"tokens_out":2326,"duration_ms":23818,"concrete_test":"Draw a stratified random sample of roughly 200 AraSTEM questions covering all sources, educational levels, and subjects, including items originally containing images or equations. Have at least two independent Arabic-speaking STEM experts answer each question using only the text provided in the dataset, then compare against gold labels and compute inter-annotator agreement. If expert accuracy is materially below, say, 90%, or if experts identify a substantial number of questions as unanswerable without the removed visual content, the benchmark's validity fails and the reported model accuracies cannot be interpreted as measuring knowledge.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 states that questions scraped from beadaya.com and alloschool.com had images and mathematical equations removed during cleaning, yet no analysis verifies that the remaining Arabic text alone is sufficient to answer each question. Many STEM MCQs embed diagrams, graphs, or formulas in images; deleting these can produce ambiguous or unanswerable prompts. If a non-negligible fraction of questions are broken, the central result — best open-source model near 56% accuracy — measures unparseable or misleading text rather than knowledge, and the 'challenging benchmark' claim is undercut. The gold labels themselves are also unvalidated: the LLM-extracted portion (1,806 questions via ChatGPT-4, Table 1) was proofread for format, not correctness, and manually transformed questions (e.g., 'definition' turned into MCQ) could introduce errors. No human baseline is provided, so there is no evidence that a competent Arabic-speaking expert would score near ceiling on the cleaned text-only versions. Additionally, Table 4 reports Jais 30B Chat V3 average as 0.58 while Table 5 reports 0.56 and the abstract states 56%, suggesting evaluation or reporting inconsistencies. Thus the load-bearing condition — self-contained, correctly-answered questions — is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AraSTEM, a multiple-choice question benchmark of 11,637 Arabic STEM questions spanning primary, secondary, and college levels across subjects including math, science, physics, chemistry, biology, IT, medicine, dentistry, and pharmacy. The questions were collected from web scraping, manual extraction from books, and ChatGPT-4-assisted extraction from PDFs. The authors evaluate a range of open-source LLMs in a zero-shot setting with an English chain-of-thought prompt and report that the best-performing open-source model, Jais 30B Chat V3, reaches 56% accuracy (with Table 4 reporting 58%), well above random guessing but far from ceiling. The dataset is offered on Hugging Face with per-question source links.","tokens_in":11398,"tokens_out":6057,"duration_ms":51218,"significance":"If the benchmark is valid, it addresses a real gap: most knowledge benchmarks are English-centric, and Arabic STEM evaluation is underrepresented. The dataset's size, source traceability, and subject breadth are genuine strengths, as is the evaluation of many model families and sizes. The paper also includes useful auxiliary analyses of tokenizer coverage, calibration, and complementary model errors. However, the central claim that low model accuracy reflects a knowledge gap depends on the cleaned text-only questions being self-contained and correctly labeled; this is not currently established. The reported numerical inconsistencies and the absence of a human-expert validation loop weaken the support for the main conclusion. With a validation study and corrected reporting, the benchmark could be a valuable community resource.","major_comments":[{"comment":"The cleaning process removed images and mathematical equations from scraped questions, but the paper does not establish that the remaining Arabic text alone makes each question self-contained and unambiguously answerable. This is load-bearing because if a nontrivial fraction of questions originally relied on diagrams, graphs, or formulas, the reported accuracies (e.g., Jais 30B Chat V3 at 56–58%) would measure unparseable or misleading prompts rather than STEM knowledge. Please add a human-expert validation step, at least on a stratified sample, that reports the fraction of questions judged answerable from the text alone, and repair or remove items that fail this check.","section":"Section 3.1"},{"comment":"Gold labels are not independently verified for correctness. The 1,806 questions extracted with ChatGPT-4 from PDFs were proofread for format rather than correctness, and the manual transformation of definition and fill-in-the-blank questions into MCQs could have introduced wrong answers. Without a double-annotated human sample with agreement statistics, the evaluation numbers may reflect errors in the reference answers rather than model knowledge. Please include a label-quality audit, for example a random sample of 200–500 questions annotated by Arabic-speaking STEM experts, with inter-annotator agreement and a report of how disagreements were resolved.","section":"Section 3.1 / Table 1"},{"comment":"Several reported numbers are inconsistent. Section 4.5 describes 193 hard-failure questions as '0.16%' of the dataset, but 193/11,637 is 1.66%. Table 4 gives Jais 30B Chat V3 an average of 0.58 while Table 5 gives 0.56 and the abstract states 56%; the random-guess baselines also differ between tables (0.30 in Table 4 vs 0.29 in Table 5). Please reconcile these values and provide the evaluation script or per-question scores to make the results reproducible and to confirm which summary is the final one.","section":"Section 4.5 and Tables 4/5"},{"comment":"The scoring procedure is underspecified for questions with two or three options. The paper states that models' choices were extracted as the probability of tokens 'A', 'B', 'C', or 'D', but Table 3 shows 2,562 questions with only two options and 3,808 questions with three options. Additionally, the prompt in Figure 7 asks for a 'Correct option number' while scoring uses option letters, and Section 4 says prompts exceeding a model's maximum input size were truncated, which could remove part of the question or the options. Please clarify how variable option counts were mapped to the answer extraction and report how many prompts were truncated.","section":"Section 4"}],"minor_comments":[{"comment":"The abstract contains a typo: 'asses' should be 'assess'.","section":"Abstract"},{"comment":"'The rest of this manuscript is organized as following' should be 'as follows', and 'it's details' should be 'its details'.","section":"Section 1"},{"comment":"The column header 'Collage' should be 'College'.","section":"Table 4"},{"comment":"The model name 'Lama 3.1' should be spelled 'Llama 3.1'.","section":"Section 4.2"},{"comment":"Reference [7] contains a garbled sentence with '¡br/¿' and 'doesnt'; the entry needs to be cleaned. Reference [22] has a malformed author field, and reference [29] has inconsistent capitalization and spacing ('ZINI', 'A W AD').","section":"References"},{"comment":"The availability statement is inconsistent: the abstract says the dataset is freely accessible on Hugging Face, while the footnote says 'Available post publication'. Please clarify the current availability status.","section":"Footnote 1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a potentially useful Arabic STEM benchmark, but its validity is not yet established. The dataset fills a real gap—native Arabic, STEM-focused, spanning primary to medical college, with per-question source links. The evaluation across a wide spread of open models is reasonable, and the extra analyses (calibration, hard failures, vocabulary coverage) are genuinely informative. The pattern that Arabic-trained models do better is not surprising, but it is concretely demonstrated here.\n\nThe main soft spot is the one the stress-test flags, and it is load-bearing. Section 3.1 says images and equations were removed during cleaning. For a STEM MCQ set that includes math and physics, that is a red flag unless the text alone is self-contained. The paper provides no check that a competent Arabic-speaking expert could answer each cleaned question, no human baseline, and no access to the dataset (it is 'available post publication'). If a non-negligible fraction of questions are ambiguous or unanswerable without their images, then the reported 56% best-model accuracy measures broken prompts, not knowledge. This is the difference between a benchmark and an artifact, and it is currently unresolved.\n\nThere are smaller real issues: 193 hard-failure questions is 1.66% of 11,637, not 0.16%; Table 4 and Table 5 disagree on Jais 30B Chat V3's average (0.58 vs 0.56) while the abstract says 56%; random-guess baselines differ slightly. These are minor and fixable. The ChatGPT-4 extraction of 1,806 medical questions was proofread for format, not content, so gold-label correctness is unvalidated for that subset. The citation coverage of existing Arabic benchmarks and models is adequate; the problems are with data validation, not provenance.\n\nI would not call the central claim false. The models' low scores are consistent with a hard benchmark, and the ordering across models is plausible. But the paper cannot be trusted as a benchmark until the self-containment and label accuracy questions are answered with data. The authors need to release the dataset, provide a human expert accuracy estimate on a sample, and fix the arithmetic.\n\nFor whom? Arabic NLP and multilingual LLM evaluation researchers. It deserves a serious referee, but with the expectation of major revision. I would send it to review.","headline":"Useful Arabic STEM benchmark, but question validity and release are unverified; deserves a serious referee with expectations of major revision.","tokens_in":11985,"tokens_out":4213,"would_cite":false,"duration_ms":39948,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50"],"pacs":[],"model":"deepseek-v4-flash","headline":"AraSTEM: a new native Arabic STEM benchmark that publicly available LLMs struggle with, where the best model reaches only 56% average accuracy and accuracy scales with Arabic training data more than with model size.","keywords":["AraSTEM","Arabic NLP benchmark","STEM multiple choice","Arabic LLM evaluation","zero-shot question answering","multilingual knowledge benchmarks","Arabic language models"],"falsifier":"Take a stratified random sample of AraSTEM questions whose original sources contained images or equations, present the cleaned text-only versions to Arabic-speaking STEM experts, and measure their agreement with the dataset's answer key; if expert accuracy falls well below the 56% model ceiling or experts flag many questions as unanswerable, the benchmark's claim to measure knowledge rather than prompt quality collapses. A complementary check is to restore the original images or equations to the same sample and test whether top models' accuracy rises materially.","tokens_in":10967,"feed_emoji":"🧪","tokens_out":3939,"duration_ms":39486,"temperature":0.7,"pith_summary":"The paper introduces AraSTEM, a dataset of 11,637 native Arabic multiple-choice questions spanning math, science, physics, chemistry, biology, information technology, dentistry, pharmacy, and medicine, from primary through college level. It argues that this benchmark is genuinely challenging for current publicly available LLMs, with the best open-source model, Jais 30B Chat V3, achieving 56% accuracy while many smaller models hover near random guessing. The central claim is that AraSTEM reveals a real gap in Arabic STEM knowledge and reasoning, and that training on substantial Arabic text is what most improves performance. If correct, this gives the community a tool to measure and drive Arabic-domain competence in LLMs rather than assuming multilingual ability carries over from English.","feed_headline":"Best Arabic LLM hits only 56 percent on new STEM exam","feed_subtitle":"The 11,637-question Arabic STEM quiz yields a 56 percent top score, with training data beating model size.","key_machinery":"The central object is the benchmark itself: 11,637 multiple-choice questions with per-question source citations, cleaned by removing images and mathematical equations, labeled by educational level and subject, and carrying two to four answer options. The evaluation machinery is a fixed English Chain-of-Thought prompt that states the subject and level, with the model's answer read from the softmax probability of the A/B/C/D tokens. This setup lets the authors attribute differences in accuracy to model knowledge rather than to prompt wording—the claim being tested—and its output is a per-subject and per-level accuracy table that supports comparisons across model families and sizes.","core_discovery":"The paper's discovery is that a large, native Arabic STEM MCQ collection produces a wide, structured performance spread across open models, with accuracy tracking the amount of Arabic training data more than model size. The top performers are Jais, Llama 3.1 Instruct, AceGPT, and Bloomz, while general-purpose large models like Falcon 40B and Llama 2 lag noticeably. The paper also shows that model predictions are complementary—only 193 questions stumped every tested model—and that the leading models are generally overconfident except Llama 3.1, which calibrates well. These results are presented as evidence for the value of Arabic-centric training and the need for localized evaluation.","pith_inferences":["The removal of images and equations is the fragile point: the reported numbers measure text-only STEM knowledge, and restoring original figures could raise model accuracy, meaning the benchmark may understate full multimodal competence.","The paper does not include a human expert baseline; without one, 'challenging' is anchored only to random guessing and model scores, so a small stratified human evaluation would let the field interpret the 56% figure.","AraSTEM could double as a training resource, not just an exam: the labeled questions could support Arabic STEM instruction tuning, and the per-subject difficulty profiles could guide curriculum design for localized models.","The per-question source citations give the benchmark a traceability that many English benchmarks lack, a contribution the paper mentions but does not emphasize as a finding."],"forward_implications":["Models trained with a larger share of Arabic data, such as Jais and AceGPT, outperform equally sized models with less Arabic exposure, implying that scaling Arabic tokens in pretraining is a direct lever on Arabic STEM accuracy.","Instruction fine-tuning consistently improves AraSTEM accuracy, as seen in Jais chat versus base and Llama 3.1 Instruct versus base, so alignment helps even in zero-shot knowledge tasks.","The 193-question hard-failure set, concentrated in dentistry and medicine, marks a concrete target for specialized Arabic medical knowledge.","Because models' correct answers overlap only partially, an ensemble of the top models would score higher than any single model, offering a practical path to better Arabic STEM question answering."],"supporting_citations":[{"why":"Defines the MMLU benchmark style that AraSTEM adapts to Arabic STEM knowledge evaluation.","marker":"[4]"},{"why":"Provides the translated, non-native Arabic MMLU baseline that AraSTEM distinguishes itself from.","marker":"[14]"},{"why":"Prior Arabic MMLU dataset with only 20% STEM content; the direct comparison point for AraSTEM's STEM focus.","marker":"[15]"},{"why":"The Arabic-centric Jais model, whose reported 28% Arabic training data and top accuracy underpin the scaling claim.","marker":"[3]"},{"why":"The Arabic-localized AceGPT model used in comparisons, cited for its 64% Arabic training share.","marker":"[26]"},{"why":"Chain-of-thought prompting research that motivates the English-prompt design used in all evaluations.","marker":"[25]"},{"why":"The scaling law used to explain why Arabic-data-rich models benefit from size increases while others do not.","marker":"[28]"}],"fun_headline_variants":["Arabic STEM quiz stumps top LLMs at 56%","Arabic-centric training beats model size in STEM quiz","Native Arabic STEM exam exposes LLM gaps","AraSTEM: 11k questions, top score 56%","Arabic LLMs lag on native STEM benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"After images and equations were stripped from scraped questions, each remaining question must still be self-contained and answerable from the Arabic text alone; if many questions became ambiguous or unanswerable, the reported accuracy gaps would reflect broken prompts rather than knowledge, and the answer key's correctness also rests on external sources and ChatGPT-4 extraction without independent human validation.","fun_headline_variants_meta":{"raw":{"variants":["Arabic STEM quiz stumps top LLMs at 56%","Arabic-centric training beats model size in STEM quiz","Native Arabic STEM exam exposes LLM gaps","AraSTEM: 11k questions, top score 56%","Arabic LLMs lag on native STEM benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00051,"raw_usage":{"total_tokens":2432,"prompt_tokens":844,"completion_tokens":1588,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":460,"completion_tokens_details":{"reasoning_tokens":1511}},"tokens_in":460,"tokens_out":1588,"duration_ms":11831,"temperature":1.0,"reasoning_tokens":1511,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:47:19.897255+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a stratified random sample of AraSTEM questions whose original sources contained images or equations, present the cleaned text-only versions to Arabic-speaking STEM experts, and measure their agreement with the dataset's answer key; if expert accuracy falls well below the 56% model ceiling or experts flag many questions as unanswerable, the benchmark's claim to measure knowledge rather than prompt quality collapses. A complementary check is to restore the original images or equations to the same sample and test whether top models' accuracy rises materially.","supporting_citations":[{"cited_title":"Mea- suring massive multitask language understand- ing,","cited_arxiv_id":null,"evidence_quote":"Defines the MMLU benchmark style that AraSTEM adapts to Arabic STEM knowledge evaluation."},{"cited_title":"Multilingual massive multitask language under- standing (mmmlu)","cited_arxiv_id":null,"evidence_quote":"Provides the translated, non-native Arabic MMLU baseline that AraSTEM distinguishes itself from."},{"cited_title":"Arabicmmlu: As- sessing massive multitask language understand- ing in arabic,","cited_arxiv_id":null,"evidence_quote":"Prior Arabic MMLU dataset with only 20% STEM content; the direct comparison point for AraSTEM's STEM focus."},{"cited_title":"Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models,","cited_arxiv_id":null,"evidence_quote":"The Arabic-centric Jais model, whose reported 28% Arabic training data and top accuracy underpin the scaling claim."},{"cited_title":"AceGPT, localizing large language models in Arabic,","cited_arxiv_id":null,"evidence_quote":"The Arabic-localized AceGPT model used in comparisons, cited for its 64% Arabic training share."},{"cited_title":"Training compute-optimal large language models,","cited_arxiv_id":null,"evidence_quote":"The scaling law used to explain why Arabic-data-rich models benefit from size increases while others do not."}],"review_version":1}