Pith. sign in

REVIEW 5 major objections 5 minor 6 references

3LM: Bridging Arabic, STEM, and Code through Benchmarking

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Three open benchmarks test Arabic LLMs on science and code.

desk verdict Open Arabic STEM and code benchmarks that fill a real gap, but the paper leans on unmeasured QA-pair quality and a ROUGE-L-based translation gate; useful and worth peer review with a demand for validation numbers. read the letter →

arxiv 2507.15850 v3 pith:2HOCQMB4 submitted 2025-07-21 cs.CL

Basma El Amel Boussaha , Leen AlQadi , Mugariya Farooq , Shaikha Alsuwaidi , Giulia Campesan , Ahmed Alzubaidi , Mohammed Alyafeai , Hakim Hacid This is my paper · ORCID
classification cs.CL
keywords ArabicLLMevaluationSTEMquestionansweringcodegenerationbenchmarkmachine-translatedpromptsmultiple-choicepass@1syntheticmultilingualmodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to fill a gap in Arabic LLM evaluation: existing Arabic benchmarks emphasize language, dialect, and culture, while science and programming, the domains most relevant to practical LLM use, are barely covered. It introduces 3LM, a suite of three public benchmarks built from natively Arabic sources: 865 multiple-choice questions extracted from Arabic textbooks and worksheets, 1,744 synthetic STEM questions generated from the same material, and 542 Arabic-translated code prompts adapted from established English code benchmarks. To show the benchmarks work, the paper evaluates more than 40 Arabic and multilingual models and reports a consistent pattern, with MCQ scores far exceeding completion-format scores and English and Arabic code-generation scores almost perfectly correlated. A sympathetic reader would care because 3LM offers the first open, native Arabic test bed for scientific reasoning and bilingual programming, turning a blind spot into a measurable target.

What carries the argument

The load-bearing object is the benchmark suite itself, whose three components are built by three distinct mechanisms. Native STEM items come from an OCR pipeline for Arabic math-heavy PDFs, followed by a four-stage LLM extraction-and-formatting pipeline that classifies question type, filters visually dependent items, rewrites non-MCQs into MCQs, and is followed by manual team verification. Synthetic items are produced by an adapted YourBench pipeline, with ingestion, summarization, chunking, question generation, and analysis stages, using the same textbook sources and a difficulty threshold of 6 or higher. Code items are produced by translating HumanEval+ and MBPP+ prompts with GPT-4o, validating via backtranslation with ROUGE-L thresholds of 0.85 for HumanEval and 0.8 for MBPP, and sending below-threshold items to native-Arabic-speaking Python experts. These mechanisms carry the paper's claims because they determine the exact items, answers, and tests on which model rankings rest.

What would settle it

Take a random sample of 200 native and 200 synthetic STEM items, have independent Arabic-speaking STEM educators re-answer them and flag ambiguous or wrong items, and check whether the original answer key matches. If the key-mismatch rate exceeds a few percent, the reported rankings would be unreliable. For the code benchmarks, have native-Arabic-speaking programmers judge whether a random sample of the 542 Arabic prompts, including items above the ROUGE-L thresholds, preserve the original English intent; if a noticeable fraction are semantically unfaithful, the threshold-based acceptance of untranslated items is invalid.

Watch

Extended reading notes

Core claim

The paper's central claim is that 3LM provides valid, reproducible evaluation of Arabic LLMs in STEM knowledge and code generation. It argues this on three fronts: the native STEM items are authentic because they come from real Arabic educational materials; the synthetic items extend coverage while controlling difficulty; and the code prompts, though translated, preserve the semantics and test suites of the original English benchmarks. Based on evaluations of over 40 models, the paper further claims that these benchmarks reveal substantive findings about model behavior, most notably that multiple-choice accuracy systematically overstates ability measured without a visible answer choice, and that Arabic code ability tracks English code ability almost exactly, while Arabic NLP ability is only weakly related to either.

Load-bearing premise

The load-bearing premise is that the 2,609 STEM items' answer keys and wording are correct, and that the 542 translated code prompts mean the same thing as their English sources, because the paper asserts manual verification and ROUGE-L backtranslation thresholds without reporting annotator counts, agreement scores, or error rates, so any hidden errors would silently skew the model rankings.

Editorial extensions

If this is right

  • If 3LM is correct, Arabic LLM teams gain three public benchmarks that can be plugged into training and evaluation pipelines, making progress in Arabic scientific reasoning measurable.
  • The consistent MCQ-over-completion gap across model sizes implies that MCQ-only Arabic evaluations overstate model ability, so future Arabic benchmark reports should treat completion-format scores as the sterner test.
  • The near-perfect correlation between English and Arabic code pass@1 scores suggests that translating prompts is a reliable way to extend English code benchmarks to Arabic, but that the Arabic score mostly re-measures the same underlying code ability.
  • The weak Arabic-NLP-to-code correlation for instruct models indicates that Arabic language proficiency and programming proficiency diverge after fine-tuning, so they should be evaluated and optimized separately.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Implicit consequence: because the paper excludes diagrams and figures from the benchmarks, an image-inclusive Arabic STEM benchmark is the natural next step, and the original PDF corpus still contains the visual items that were filtered out.
  • Testable extension: the ROUGE-L thresholds could be calibrated against human semantic-fidelity ratings on the 542 translated prompts, which would turn the current rule of thumb into a validated filter for future multilingual code benchmarks.
  • Editorial connection: since native and synthetic items draw on the same textbook sources, one could directly test for generation bias by comparing model accuracy on native-versus-synthetic versions of the same concept, revealing whether the LLM generator changes difficulty.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces 3LM, a suite of three Arabic benchmarks: 865 native STEM multiple-choice question-answer pairs extracted from Arabic textbooks and worksheets, 1,744 synthetic STEM MCQs generated with the YourBench pipeline using Qwen3-235B-A22B, and 542 Arabic-translated code prompts (HumanEval-Ar and MBPP-Ar) produced by translating EvalPlus benchmarks with GPT-4o and backtranslation-based quality filtering. The authors evaluate over 40 base and instruction-tuned Arabic-centric and multilingual LLMs on these benchmarks in MCQ and completion formats, and on code via pass@1, also reporting cross-task correlations and a distractor-perturbation robustness analysis. All datasets and evaluation code are released publicly.

Significance. If the benchmark items are accurate, 3LM would be a valuable open resource for Arabic STEM and code evaluation, filling an acknowledged gap: existing Arabic benchmarks emphasize linguistic/cultural content, and AraSTEM is not publicly available. The code suite extends widely used EvalPlus benchmarks with Arabic prompts, and the evaluation across 40+ models is a useful contribution. Strengths include the public release of datasets and code, the native Arabic sourcing for STEM items, anchoring the code half to external HumanEval/MBPP tasks with EvalPlus test suites, and the robustness perturbation analysis. However, the central validity claim depends on benchmark-item correctness, and the paper provides no quantitative evidence of that correctness: manual verification is asserted without annotator counts, agreement scores, or error rates, and the code translation quality gate is based on a surface n-gram metric with no reported audit of accepted translations. These gaps make the reported model rankings potentially unreliable and need to be addressed before the benchmarks can be accepted as valid assessment tools.

major comments (5)
  1. [§3.1.2, §3.1.3] The accuracy of the native and synthetic QA pairs is asserted but not quantified. The manual verification in §3.1.2 is described without annotator counts, inter-annotator agreement, or error rates, and the synthetic filtering in §3.1.3 is called "rigorous" without reporting rejection rates or any sample of rejected items. Since the paper's central claim is that 3LM yields valid rankings of 40+ models, the authors should report a human audit on a random sample (e.g., 100 items per benchmark) with per-domain correctness rates, and either correct or exclude erroneous items.
  2. [§3.2, Abstract] The abstract claims a "human-in-the-loop process with several rounds of review," but the methods state that human review is applied only to translations below the ROUGE-L thresholds; above-threshold translations are accepted without human review. ROUGE-L measures surface n-gram overlap, not semantic equivalence, so a translated prompt could pass the threshold while changing the required function behavior. The authors should report the number of prompts that fell below threshold, the human review outcomes, and a sample audit of above-threshold prompts with semantic equivalence judgments.
  3. [§5.3.1, Table 8] The claim that "Models consistently perform better in MCQ format compared to completion format across all scales" is contradicted by the instruction-tuned results in Table 8, particularly for Qwen3 models: Qwen3-8B-Instruct scores 20.23 in MCQ versus 47.63 in completion, Qwen3-14B 39.54 versus 50.98, and Qwen3-30B-A3B 17.57 versus 53.53. This anomaly is not discussed and suggests a problem with the MCQ scoring protocol for these models (e.g., Arabic-letter likelihood scoring under a chat template). The authors should explain or correct this inconsistency.
  4. [Table 3, Table 10] Several reported code scores appear internally inconsistent. In Table 3, Qwen2.5-32B-Base shows Arabic MBPP+ at 82.8, which is 35.8 points higher than its Arabic MBPP (47.0) and also higher than its English MBPP+ (69.0), an implausible pattern. In Table 10, aya-expanse-32B-Instruct shows Arabic MBPP as 5.5, likely a typo for 55.5. These data errors undermine the reliability of the code evaluation results, which are a central contribution; the authors should verify all tables and correct data-entry errors.
  5. [§3.1.3, Tables 8-9] The synthetic benchmark is generated by Qwen3-235B-A22B, and models from the Qwen3 family are evaluated on it. This is not circular in the strict sense because the benchmark items are not derived from the evaluated models' outputs, but it creates a risk of inflated scores for the generator family if the generator's question style and answer-label distribution match its own preferences. The authors should assess this risk by comparing the relative performance of Qwen3-family models on the synthetic benchmark versus the native benchmark; if the performance gap is systematically larger for Qwen3 models than for other families, this would indicate a generation bias. The paper currently reports no such analysis.
minor comments (5)
  1. [§2] The AraSTEM citation is listed as Mustapha et al. (2024), but the arXiv identifier (2501.00559) is dated January 2025; please correct the year and ensure consistency in the reference list.
  2. [§5.3.1] The text refers to "Table 9" for instruction-tuned models' overall averages, but Table 9 reports domain-wise synthetic scores; the overall averages appear in Table 8. Please fix the cross-reference.
  3. [Appendix A.2] The Arabic script in the translated instruction and response prompts appears garbled (e.g., reversed or mirrored letter forms such as "ىجري" and "ةيلاتلﺍ"). This is likely an OCR artifact and should be cleaned before publication.
  4. [§6, Tables 11-12] The tables report a column labeled "Score 25%" without defining whether this is the score on the 25% perturbed subset or the full benchmark with a 25% perturbation rate. Please clarify the metric and describe how the 25% subset was selected (e.g., stratified by domain or difficulty).
  5. [Table 3] The jais-family-8k 30B row shows Arabic MBPP (12.4) and MBPP+ (10.3) far below its English scores (46.6 and 38.1). If this is not a typo, the result is striking and should be discussed; otherwise it should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmarks are sourced externally, and model evaluations are empirical rather than derived from benchmark construction.

full rationale

3LM's derivation chain is not circular. The native STEM items come from Arabic textbooks and worksheets via OCR and LLM-assisted extraction, the synthetic items are generated from those same sources with the YourBench framework, and the code items are translations of the external HumanEval+ and MBPP+ suites using GPT-4o with ROUGE-L backtranslation checks. None of these benchmark items are defined in terms of the evaluated models' outputs or scores, and no parameter is fitted to a subset of the evaluation data and then reported as a prediction. The use of Qwen3-235B-A22B in the curation pipeline and its inclusion in the evaluated model set is a possible self-familiarity and validity threat, but it is not a logical reduction: the model's scores are not equal to, or forced by, the benchmark construction choices. The paper itself acknowledges this concern in its Limitations section, noting that synthetic generation may inherit biases from Qwen3-235B-A22B. Self-citations to AlGhafa and the Open Arabic LLM Leaderboard are related-work and data-source references, not load-bearing justifications for the new benchmarks. The reported English-Arabic code correlation (r = 0.97) is an empirical measurement, not a fitted quantity, and the paper does not claim to derive it from the translation pipeline. Thus no circular step meeting the quoted-evidence standard is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The benchmarks rest on three categories of unevidenced inputs: trust in the school sources and their answer keys, trust in LLM-based extraction and generation (Qwen3, YourBench) without quantitative quality control, and trust in ROUGE-L backtranslation as a fidelity proxy for code prompts. There are no fitted scientific constants; the numbers listed are curation thresholds that shape the datasets.

free parameters (3)
  • ROUGE-L F1 thresholds for translation acceptance = 0.85 (HumanEval), 0.8 (MBPP)
    Chosen by authors in Section 3.2; translations above threshold skip human review, so the threshold directly affects code benchmark quality.
  • Synthetic question difficulty threshold = 6 or higher
    Section 3.1.3; filters synthetic questions, shaping benchmark difficulty and model scores.
  • RDP perturbation fraction = 20% remove correct answer, 5% add none-of-the-above
    Section 6; design choice for the robustness analysis.
assumptions (4)
  • domain assumption Source school PDFs and their answer keys are factually correct and representative of Arabic middle and high school STEM curricula.
    Section 3.1.1; answers extracted from these sources are treated as ground truth, with no external expert audit reported.
  • domain assumption Qwen3-235B-A22B reliably extracts, classifies, and converts QA pairs without introducing systematic errors.
    Section 3.1.2; the native benchmark pipeline relies on this model for all curation steps.
  • domain assumption LLM-generated synthetic questions from YourBench are factually correct and appropriate for evaluation.
    Section 3.1.3; only automated filtering and team review are described.
  • domain assumption Backtranslation ROUGE-L F1 similarity is sufficient evidence of translation faithfulness for code prompts.
    Section 3.2; translations meeting the threshold skip human review.

how reviews work

0 comments
Cite this review

Pith. "Pith review of 3LM: Bridging Arabic, STEM, and Code through Benchmarking." pith.science (2026). https://pith.science/paper/2HOCQMB4

@misc{pith2026250715850,
  author       = {Pith},
  title        = {Pith review of: 3LM: Bridging Arabic, STEM, and Code through Benchmarking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2HOCQMB4}},
  note         = {Machine review of arXiv:2507.15850}
}
read the original abstract

Arabic is one of the most widely spoken languages in the world, yet efforts to develop and evaluate Large Language Models (LLMs) for Arabic remain relatively limited. Most existing Arabic benchmarks focus on linguistic, cultural, or religious content, leaving a significant gap in domains like STEM and code which are increasingly relevant for real-world LLM applications. To help bridge this gap, we present 3LM, a suite of three benchmarks designed specifically for Arabic. The first is a set of STEM-related question-answer pairs, naturally sourced from Arabic textbooks and educational worksheets. The second consists of synthetically generated STEM questions, created using the same sources. The third benchmark focuses on code generation, built through a careful translation of two widely used code benchmarks, incorporating a human-in-the-loop process with several rounds of review to ensure high-quality and faithful translations. We release all three benchmarks publicly to support the growth of Arabic LLM research in these essential but underrepresented areas.

Figures

Figures reproduced from arXiv: 2507.15850 by the authors.

Figure 1
Figure 1. Summary of 3LM Benchmark. knowledge-intensive domains like science and mathematics. To address this, we introduce 3LM ( ملع( , a suite of three benchmarks for evaluating Ara￾bic LLMs across core STEM disciplines, including general science, mathematics, chemistry, physics, and biology, and code generation. The first benchmark in 3LM consists of native multiple-choice questions (MCQs) sourced from real Arabic-language… view at source ↗
Figure 2
Figure 2. 3LM benchmark curation process. Unlike previous efforts, 3LM is fully open-source with all datasets publicly available2 , accompanied by a comprehensive GitHub repository containing all the code necessary to reproduce the experimen￾tal results reported in this paper. 3 The Benchmark 3LM benchmark comprises two categories: STEM and code. The STEM portion includes both auto￾matically generated synthetic questions from… view at source ↗
Figure 3
Figure 3. Statistics on STEM benchmarks of 3LM. conceptual 32.1% application-based 31.7% analytical 28.9% factual 4.8% Other (types < 3%) 2.4% Question Type Distribution Across All Datasets Note: 'Other' includes question types with <3% each (e.g., counterfactual, edge-case, false-premise) [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Question type distribution across domains in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Subject-wise scores (completion) on base mod [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: RougeL-f1 score distribution for round-trip translation of HumanEval input prompts, before human check. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: RougeL-f1 score distribution for round-trip translation of MBPP input prompts, before human check. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Distribution of the number of unit tests for the benchmarks in the EvalPlus-Ar suite. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Correlation plot of EvalPlus and EvalPlus-Arabic suites for pre-trained models. Average pass@1 is [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Correlation plot of OALL and EvalPlus-Arabic suites for pre-trained models. Average accuracy and [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Correlation plot of OALL and EvalPlus-Arabic suites for instruct models. Average accuracy and average [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 13
Figure 13. Figure 13: Biology question type distribution. application-based conceptual analytical factual clarification false-premise true-false counterfactual Question Type 0 20 40 60 80 100 120 140 Count Question Type Distribution [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 15
Figure 15. Figure 15: Math question type distribution. conceptual analytical application-based factual clarification counterfactual false-premise true-false edge-case Question Type 0 20 40 60 80 100 120 140 Count Question Type Distribution [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 17
Figure 17. Figure 17: Physics question type distribution [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

6 extracted references · 5 canonical work pages

  1. [1]

    Question:

    Clean the format: • Remove any explicit “Question:” or “An- swer:” labels from both questions and answers. • If the pair is already an MCQ and appears clean (with clearly labeled options and a correct answer), leave both the question and answer unchanged

    work page

  2. [2]

    (ﺃ(“,”) ﺏ,

    For non-MCQ pairs only: • Generate a new MCQ version of the question based on the original content. • Include 4 options labeled as: “(ﺃ(“,”) ﺏ,”) (“ﺝ(“,”) ﺩ.”) One of the options must be the correct answer; assign it randomly among the four choices. • The remaining three options should be plausible distractors, related to the topic and context of the ques...

    work page

  3. [3]

    original_question

    Output structure: • Return a list of JSON objects. • Each object should contain: – "original_question": cleaned original question text (without la- bels). – "original_answer": cleaned original answer text. – "type": stays the same as the origi- nal type in input data. – "refined_question": refined or generated MCQ question string, in- cluding all four opt...

    work page

  4. [4]

    Do NOT include any extra text outside the JSON output. Input: {input_data} B.4 Sample Questions As shown in Figure 12, our benchmark includes both native and synthetic questions spanning vari- ous scientific domains such as biology, chemistry, mathematics, physics, and geography. This visual demonstrates not only the question formatting but also the atten...

    work page 2025

  5. [2021]

    Preprint, arXiv:2107.03374

    Evaluating large language models trained on code. Preprint, arXiv:2107.03374. Ali El Filali, Manel ALOUI, Tarique Husaain, Ahmed Alzubaidi, Basma El Amel Boussaha, Ruxandra Co- jocaru, Clémentine Fourrier, Nathan Habib, and Hakim Hacid. 2025. Open arabic llm leader- board 2. https://huggingface.co/spaces/OALL/Open- Arabic-LLM-Leaderboard. AbdelRahim Elmad...

    arXiv 2025

  6. [2025]

    AraSTEM: A Native Arabic Multiple Choice Question Benchmark for Evaluating LLMs Knowledge In STEM Subjects

    AraDiCE: Benchmarks for dialectal and cul- tural capabilities in LLMs. In Proceedings of the 31st International Conference on Computational Linguis- tics, pages 4186–4218, Abu Dhabi, UAE. Association for Computational Linguistics. Ahmad Mustapha, Hadi Al-Khansa, Hadi Al-Mubasher, Aya Mourad, Ranam Hamoud, Hasan El-Husseini, Marwah Al-Sakkaf, and Mariette ...

    work page Pith review arXiv 2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.