From Guidelines to Practice: A New Paradigm for Arabic Language Model Evaluation

Abdulrahman Al-Batati; Adel Ammar; Omer Nacar; Serry Sibaee; Wadii Boulila; Yasser Al-Habashi

REVIEW 4 major objections 5 minor 23 references

From Guidelines to Practice: A New Paradigm for Arabic Language Model Evaluation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Top LLMs answer just 30% of a new hard Arabic test

desk verdict A genuinely new Arabic evaluation set with a load-bearing reproducibility gap: the ADMD dataset is worth attention, but the model rankings are unverified until the data and scoring are released. read the letter →

arxiv 2506.01920 v1 pith:6ULYO7YO submitted 2025-06-02 cs.CL

Serry Sibaee , Omer Nacar , Adel Ammar , Yasser Al-Habashi , Abdulrahman Al-Batati , Wadii Boulila This is my paper

classification cs.CL

keywords ArabicLLMevaluationDepthMiniDatasetculturalalignmentbenchmarkcritiquelargelanguagemodelsmanualNLPdomain-specificbenchmarks

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that current Arabic evaluation benchmarks overstate model competence because they lean on translated, Western-centric, or linguistically sloppy content. To make the point concrete, the authors build the Arabic Depth Mini Dataset (ADMD): 490 hand-curated questions drawn from Arabic primary sources across ten domains, and they score five leading LLMs on it. Under strict manual scoring, the best model (Claude 3.5 Sonnet) answers only 30% of questions correctly, and GPT-4 only about 12%. The paper also proposes a set of theoretical guidelines—linguistic accuracy, cultural alignment, source validation, and evaluator expertise—that it says future Arabic datasets must follow to be credible.

What carries the argument

The load-bearing object is the Arabic Depth Mini Dataset (ADMD), 490 questions organized into ten major domains and 42 sub-domains, each question sourced from an Arabic primary reference such as Tafsir Ibn Jarir al-Tabari, Lisan al-Arab, or Hindawi science books. The scoring procedure is the second mechanism: four researchers manually label each model response as True (100% correct), Partially-True (60–80%), Partially-False (20–30%), or False, and the same four-criteria rubric (Language Rules, Scientific Writing, Cultural Values, Information Correctness) is used to critique existing datasets like Ghafa, ArabicMMLU, and INCLUDE. The guidelines plus the dataset together form the argument: because the questions demand cultural depth instead of surface pattern matching, performance on ADMD measures something that MMLU-style benchmarks miss.

What would settle it

Have two independent teams of native Arabic experts score the same 490 model responses using the paper's four labels and compare their agreement; if agreement falls below standard reliability thresholds, the reported rankings (Claude 3.5 Sonnet 30%, GPT-4 11.8%) are not reproducible. A simpler check is to release the answer key and rerun the five models on ADMD to see whether the accuracy numbers match.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that expert-level Arabic cultural knowledge is almost entirely missing from current LLMs, and that this is hidden by existing benchmarks. Using the ADMD, Claude 3.5 Sonnet achieved the highest accuracy at 30% (147/490), with relative strengths in Mathematics & Computational Sciences (50%), Philosophy & Logic (50%), and General & Miscellaneous Sciences (51.67%), but near-total failure in Historical & Genealogical Studies (0%) and weak performance in Islamic & Religious Studies despite its relative lead. GPT-4 scored only 11.8% True, with the highest False rate. The paper reads these results as evidence that translated and automatically generated Arabic datasets produce misleadingly optimistic evaluations, and that culturally grounded, depth-oriented questions are needed to reveal actual model limitations.

Load-bearing premise

The entire model ranking rests on the authors' own manual judgments of what counts as True, Partially-True, Partially-False, or False, and those judgments have not been checked for inter-annotator agreement or made public as a full answer key.

Editorial extensions

If this is right

If ADMD reflects genuine Arabic expert knowledge, then the frontier LLMs tested here are not yet dependable for Arabic cultural, religious, and specialist domains.
Existing benchmarks such as ArabicMMLU and Ghafa can no longer be read as evidence of Arabic competence, since large fractions of their samples fail basic language and cultural checks.
Model rankings are domain-dependent: Claude 3.5 Sonnet leads in mathematics and logic but still fails Islamic studies, so a single aggregate score hides practically meaningful differences.
Future Arabic dataset builders should follow the proposed guidelines—primary-source attribution, cultural alignment, and evaluator expertise—or risk reproducing inflated results.
Manual evaluation remains necessary for non-English tasks, since the paper cites evidence that LLM judges are unreliable for Arabic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the ADMD answer key and scoring rubric were released with inter-annotator agreement numbers, the benchmark could serve as a reproducible stress test for Arabic NLP, but as published the headline scores cannot be independently verified.
The 30% ceiling may partly reflect the strict True threshold (100% correct); a reader comparing these numbers with accuracy on MMLU-style benchmarks should treat them as different quantities, not the same measure applied to a harder test.
A natural testable extension would be to measure whether Arabic fine-tuned models (e.g., Jais, Allam, Fanar, Aya) improve on ADMD without improving on translated benchmarks, which would isolate cultural knowledge from general reasoning.
Because physics, chemistry, and advanced mathematics are excluded and medicine is thin, the domain ranking is a lower bound on the difficulty gap; adding those fields could shift the relative standings.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know up front. First, the Arabic Depth Mini Dataset (ADMD) is the real contribution: 490 handwritten questions drawn from scholarly Arabic sources, spanning ten domains and 42 subtopics. That is new content, not a translation or harvested set. Second, the paper's headline result—Claude 3.5 Sonnet leading at 30% overall, with GPT-4 trailing—rests on manual scoring that is not yet reproducible. Until the data and answer key are out, treat the rankings as provisional.

What the paper does well is the critical review of existing Arabic benchmarks. The authors walk through concrete examples from Ghafa, ArabicMMLU, and INCLUDE, showing translation errors, cultural misalignment, and factual mistakes. That is useful and grounded. Their theoretical guidelines for dataset construction are sensible, even if partly drawn from their earlier ILMAAM paper; they are practical and actually applied. The selection of primary sources for ADMD—classical texts in fiqh, hadith, grammar, and poetry—is thoughtful and culturally grounded.

The soft spots are concentrated in Section 5. The scoring rubric defines partial categories by vague thresholds (60–80% correct), with no worked example, no inter-annotator agreement, and no adjudication procedure. The dataset is not released; only a sample appears via Google Sheets. The abstract's 30% for Claude does not match Table 5's 33.5% category average for the same model, and the paper doesn't reconcile them. A footnote saying a judge LLM was the most effective automated evaluation method sits awkwardly with the text claiming no LLM judge was used. Collectively, these problems mean the measurement is not verifiable as reported. They do not prove the ranking is wrong, and the limitations section shows some awareness.

This paper is for people working on Arabic LLM evaluation. They will find the dataset idea valuable and the critique of existing benchmarks instructive. But it should be read as a research proposal with preliminary evidence, not a finished benchmark. With full release of the dataset, answer key, and scoring protocol, it could become a solid reference. I'd give it a serious referee; the issues are substantial but repairable.

Referee Report

4 major / 5 minor

Summary. The paper proposes a framework of theoretical guidelines for constructing Arabic evaluation datasets, audits three existing benchmarks (Ghafa, ArabicMMLU, INCLUDE) against those guidelines, and introduces a new 490-question Arabic Depth Mini Dataset (ADMD) spanning ten domains and 42 subdomains. Using ADMD, the authors manually scored outputs from five LLMs (GPT-4, Claude 3.5 Sonnet, Gemini Flash 1.5, CommandR 100B, Qwen-Max) into four categories (True, False, Partially-True, Partially-False). They report that Claude 3.5 Sonnet achieves the highest accuracy, with 147 correct answers (30%) and particular strength in Mathematics & Computational Sciences, Philosophy & Logic, and General & Miscellaneous Sciences, while all models struggle on culturally nuanced topics.

Significance. If the evaluation is reproducible, ADMD would be a useful culturally grounded Arabic evaluation resource, addressing a real gap in a field dominated by translated or surface-level benchmarks. The paper also makes a constructive contribution by articulating explicit linguistic, cultural, and methodological guidelines and by providing a structured comparison of existing datasets. However, the central empirical claim — the model ranking in Tables 7–11 and the 30% headline for Claude 3.5 Sonnet — currently rests on a non-public dataset, a manual grading process with no reported inter-annotator agreement, and internally inconsistent statements about whether an LLM judge was used. These issues make the headline results unverifiable as presented, so the significance of the contribution cannot yet be assessed.

major comments (4)

[Abstract, §5.1, Table 5] The reported headline accuracy for Claude 3.5 Sonnet is internally inconsistent. The abstract and §5.1 state 147 questions (30%), but Table 5 reports the model's average True rate as 33.5%. The difference matters because categories have unequal sizes (most have 10 questions but General Linguistics and General Sciences have 50), so an unweighted category average can differ substantially from the overall percentage. The paper should state which quantity is reported, provide total counts for all models, and report the overall percentage alongside the category average. Without this clarification, the headline ranking is ambiguous.
[§5, §5.1, Appendix A.2] The model rankings in Tables 7–11 depend entirely on the manual assignment of each model answer to True, False, Partially-True, or Partially-False, but the scoring process is not reproducible from the manuscript. The labels are defined only by broad percentage bands (60–80% correct, 20–30% correct) with no worked examples, no scoring rubric, no adjudication procedure, and no inter-annotator agreement measure. The full dataset and answer key are also not released; Appendix A.2 provides only sample examples via a link. To support the central claim, the authors should release the dataset with gold answers, provide the complete rubric, and report agreement statistics (e.g., Cohen's kappa) from at least two independent annotators on a representative sample.
[§3.4, §5, footnote 4] The manuscript contradicts itself about whether an LLM judge was used. §3.4 states that the authors developed a Python library that leverages the Claude Sonnet model to automate dataset evaluation, and footnote 4 says that 'the most effective way to automate the evaluation is by using a judge LLM.' In contrast, §5 states that 'We did not use LLM Judge in this paper' and justifies manual evaluation. These statements cannot all be correct for the same evaluation pipeline. The authors need to clarify what the Claude-based library was used for (e.g., dataset construction, quality review, or final scoring) and state explicitly which stages used human annotators versus an LLM, since the reader's interpretation of Tables 7–11 changes accordingly.
[§4, Tables 2–4] The review of existing datasets is based on a 'representative sample' manually evaluated by 'one of the authors' for each dataset, but no sample sizes, selection criteria, or scoring validation are reported. For example, Table 2 reports Ghafa's Language Rules score as 4.5 and Table 4 reports INCLUDE's Information Correctness as 7.0, yet without knowing how many items were sampled and how the scores were aggregated, these numbers cannot be verified or compared. Since these critiques motivate the need for ADMD, the authors should report the sampling procedure, the number of items reviewed per dataset, and ideally a second annotator for reliability.

minor comments (5)

[Abstract] The phrase '(42 sub-domains, see Figure 1' is missing a closing parenthesis; it should read 'see Figure 1).'
[§2 and §4.2] The paper alternately attributes ArabicMMLU to 'OpenAI (2024)' (in the Introduction and §4.2) and to Koto et al. (2024) in the Related Works and references. The citation should be unified to the correct source.
[§4.1–§4.3] Several Arabic examples appear garbled in the rendered text (e.g., the Quranic verse example in §4.1 and the spelling-error examples in §4.3). Since the Arabic quality is a central claim, these excerpts must be typeset correctly.
[Table 6] The caption 'first 10 rows per sheet, or 50 for long sheets' is unclear about the actual number of questions per subject; the text elsewhere says each topic has 10 questions except General Language and Diversified Science, which have 50, so the caption should state this directly.
[§A.1] The reference list for ADMD sources is useful but incomplete: several entries give only URLs (e.g., the Dialects entries), and two entries appear as fragments ('Diwan al-Lugha al-'Arabiyya' and 'Hindawi Science Collection') without full bibliographic details.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the model rankings are independent manual measurements against a new dataset; self-citations are contextual and not load-bearing.

full rationale

The paper's central empirical claims are the accuracy measurements of five LLMs on the newly constructed ADMD dataset (Section 5.1, Tables 7-11). These claims are observational: model outputs were manually labeled True, False, Partially-True, or Partially-False using prose thresholds, not derived from any fitted parameter or defined equivalently to the benchmark's construction. No equation in the paper defines a predicted quantity in terms of the dataset's own inputs, and no model score is obtained by substituting a fitted constant back into the evaluation. The theoretical guidelines in Section 3 are said to be 'inspired by the work of (Nacar et al., 2025)', which involves overlapping authors, and Section 4 applies those guidelines to critique existing datasets; this is self-referential framing but it is not load-bearing for the headline accuracy result, because the ADMD measurements stand independently on the manual scoring of the questions. The inconsistencies noted by the reader, such as footnote 4 reporting that a judge LLM was found most effective while Section 5 states no LLM judge was used, and the difference between the abstract's 30% and Table 5's 33.5% average, are reproducibility and reporting concerns rather than circular derivations. There is no fitted-input-called-prediction step, no definitional identity between the evaluation framework and the model outputs, and no uniqueness or ansatz smuggled in via self-citation that forces the empirical rankings. Therefore the paper shows no significant circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper contains no equations or fitted physical parameters, so the ledger captures subjective scoring thresholds and sampling choices. The empirical findings rest on the authors' own rubric, their own answer key, and the representativeness of 10-item categories, all of which are unverified.

free parameters (2)

Partial-score bands = True = 100%, Partially-True = 60-80%, Partially-False = 20-30%, False = otherwise
Hand-chosen grading bands used in Tables 5 and 7-11. No sensitivity analysis is reported; the True versus False headline is less affected than the partial categories.
Items per subtopic = 10, except 50 for General Linguistics and General Sciences
Chosen by the authors without statistical justification. Each 10-item category makes domain percentages such as 50% equal to five questions and gives wide error bars.

assumptions (3)

domain assumption Manual scoring by the authors is accurate and reproducible.
Section 5 describes 'thorough manual review' by four researchers, but no inter-annotator agreement, scoring rubric publication, or independent answer key is provided.
domain assumption The cited Arabic books and websites define the correct answers.
Appendix A.1 lists references such as Shamela and Hindawi books, but the paper does not show how answer correctness was adjudicated against these sources.
ad hoc to paper Ten questions per topic yield meaningful domain-level percentages.
Tables 7-11 report domain True and False percentages as stable measurements; with n=10 per topic, the confidence intervals are very wide and this is not acknowledged.

how reviews work

0 comments

Cite this review

Pith. "Pith review of From Guidelines to Practice: A New Paradigm for Arabic Language Model Evaluation." pith.science (2026). https://pith.science/paper/6ULYO7YO

@misc{pith2026250601920,
  author       = {Pith},
  title        = {Pith review of: From Guidelines to Practice: A New Paradigm for Arabic Language Model Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6ULYO7YO}},
  note         = {Machine review of arXiv:2506.01920}
}

read the original abstract

This paper addresses critical gaps in Arabic language model evaluation by establishing comprehensive theoretical guidelines and introducing a novel evaluation framework. We first analyze existing Arabic evaluation datasets, identifying significant issues in linguistic accuracy, cultural alignment, and methodological rigor. To address these limitations in LLMs, we present the Arabic Depth Mini Dataset (ADMD), a carefully curated collection of 490 challenging questions spanning ten major domains (42 sub-domains, see Figure 1. Using ADMD, we evaluate five leading language models: GPT-4, Claude 3.5 Sonnet, Gemini Flash 1.5, CommandR 100B, and Qwen-Max. Our results reveal significant variations in model performance across different domains, with particular challenges in areas requiring deep cultural understanding and specialized knowledge. Claude 3.5 Sonnet demonstrated the highest overall accuracy at 30\%, showing relative strength in mathematical theory in Arabic, Arabic language, and islamic domains. This work provides both theoretical foundations and practical insights for improving Arabic language model evaluation, emphasizing the importance of cultural competence alongside technical capabilities.

Figures

Figures reproduced from arXiv: 2506.01920 by the authors.

**Figure 2.** Mindmap Representation of the theoretical standards [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** The LLM prompt translates to: ’You are an [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Visual summary of Q/A word counts Model 0 100 200 300 400 TRUE FALSE Partially_True Partially_False GPT-4 Sonnet Claude GEMINI FLASH 1.5 CommandR100B Qwen-Max 2.5 [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Models’ results. True means the model an [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 8 canonical work pages

[1]

Abdelrahman Abdallah, Mahmoud Kasem, Mahmoud Abdalla, Mohamed Mahmoud, Mohamed Elkasaby, Yasser Elbendary, and Adam Jatowt. 2024. https://arxiv.org/abs/2403.17848 Arabicaqa: A comprehensive dataset for arabic question answering . Preprint, arXiv:2403.17848

work page Pith review arXiv 2024
[2]

AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic

Emad A. Alghamdi, Reem I. Masoud, Deema Alnuhait, Afnan Y. Alomairi, Ahmed Ashraf, and Mohamed Zaytoon. 2024. https://arxiv.org/abs/2403.09017 Aratrust: An evaluation of trustworthiness for llms in arabic . Preprint, arXiv:2403.09017

work page Pith review arXiv 2024
[3]

Ebtesam Almazrouei, Ruxandra Cojocaru, Michele Baldo, Quentin Malartic, Hamza Alobeidli, Daniele Mazzotta, Guilherme Penedo, Giulia Campesan, Mugariya Farooq, Maitha Alhammadi, Julien Launay, and Badreddine Noune. 2023. https://doi.org/10.18653/v1/2023.arabicnlp-1.21 A l G hafa evaluation benchmark for A rabic language models . In Proceedings of ArabicNLP...

doi:10.18653/v1/2023.arabicnlp-1.21 2023
[4]

Alzahrani, Nouf M

M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan AlRashed, Faisal A. Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, Ahmed Abde...

arXiv 2024
[5]

John Dang, Shivalika Singh, Daniel D'souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, Acyr Locatelli, Hangyu Lin, Dwarak Talupuru, Bharat Venk...

arXiv 2024
[6]

AbdelRahim Elmadany, ElMoatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. https://doi.org/10.18653/v1/2023.findings-acl.609 ORCA : A challenging benchmark for A rabic language understanding . In Findings of the Association for Computational Linguistics: ACL 2023, pages 9559--9586, Toronto, Canada. Association for Computational Linguistics

work page doi:10.18653/v1/2023.findings-acl.609 2023
[7]

Maria Eriksson, Erasmo Purificato, Arman Noroozian, Joao Vinagre, Guillaume Chaslot, Emilia Gomez, and David Fernandez-Llorca. 2025. https://arxiv.org/abs/2502.06559 Can we trust ai benchmarks? an interdisciplinary review of current issues in ai evaluation . Preprint, arXiv:2502.06559

arXiv 2025
[8]

Nizar Habash, Fadhl Eryani, Salam Khalifa, Owen Rambow, Dana Abdulrahim, Alexander Erdmann, Reem Faraj, Wajdi Zaghouani, Houda Bouamor, Nasser Zalmout, Sara Hassan, Faisal Al-Shargi, Sakhar Alkhereyf, Basma Abdulkareem, Ramy Eskander, Mohammad Salameh, and Hind Saddiki. 2018. https://aclanthology.org/L18-1574/ Unified guidelines and resources for A rabic ...

work page 2018

Show all 23 references

[9]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

2020 arXiv
[10]

Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.16 GPTA ra E val: A comprehensive evaluation of C hat GPT on A rabic NLP . In Proceedings of the 2023 Conference on Empirical Methods...

2023 doi
[11]

Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. https://doi.org/10.18653/v1/2024.findings-acl.334 A rabic MMLU : Ass...

2024 doi
[12]

KSAA . 2024. https://benchmarks.ksaa.gov.sa/b/balsam Balsam benchmark for evaluating arabic large language models (llms) . https://benchmarks.ksaa.gov.sa/b/balsam. Balsam is a collaborative initiative between prominent academic and governmental institutions in the Middle East....

2024
[13]

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

2024 arXiv
[14]

Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam

Basel Mousi, Nadir Durrani, Fatema Ahmad, Md. Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2025. https://aclanthology.org/2025.coling-main.283/ A ra D i CE : Benchmarks for dialectal and cultural capabilities in LLM s . In P...

2025
[15]

Ahmad Mustapha, Hadi Al-Khansa, Hadi Al-Mubasher, Aya Mourad, Ranam Hamoud, Hasan El-Husseini, Marwah Al-Sakkaf, and Mariette Awad. 2024. https://arxiv.org/abs/2501.00559 Arastem: A native arabic multiple choice question benchmark for evaluating llms knowledge in stem subjects...

2024 arXiv
[16]

Al-Batati, Arwa Alsehibani, Nour Qandos, Omar Elshehy, Mohamed Abdelkader, and Anis Koubaa

Omer Nacar, Serry Taiseer Sibaee, Samar Ahmed, Safa Ben Atitallah, Adel Ammar, Yasser Alhabashi, Abdulrahman S. Al-Batati, Arwa Alsehibani, Nour Qandos, Omar Elshehy, Mohamed Abdelkader, and Anis Koubaa. 2025. https://aclanthology.org/2025.loreslm-1.29/ Towards inclusive A rab...

2025
[17]

OpenAI. 2024. https://huggingface.co/datasets/openai/MMMLU Multilingual massive multitask language understanding ( MMMLU ) . Accessed: 2025-01-14

2024
[18]

Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A Haggag, Alfonso Amayuelas, et al. 2024. Include: Evaluating multilingual language understanding with regional knowledge. arXiv pr...

2024 arXiv
[19]

Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondo...

2023 arXiv
[20]

Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehki...

2025 arXiv
[21]

Minghao Wu, Weixuan Wang, Sinuo Liu, Huifeng Yin, Xintong Wang, Yu Zhao, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. 2025. https://arxiv.org/abs/2504.15521 The bitter lesson learned from 2,000+ multilingual benchmarks . Preprint, arXiv:2504.15521

2025 arXiv
[22]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[23]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Abdelrahman Abdallah, Mahmoud Kasem, Mahmoud Abdalla, Mohamed Mahmoud, Mohamed Elkasaby, Yasser Elbendary, and Adam Jatowt. 2024. https://arxiv.org/abs/2403.17848 Arabicaqa: A comprehensive dataset for arabic question answering . Preprint, arXiv:2403.17848

work page Pith review arXiv 2024

[2] [2]

AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic

Emad A. Alghamdi, Reem I. Masoud, Deema Alnuhait, Afnan Y. Alomairi, Ahmed Ashraf, and Mohamed Zaytoon. 2024. https://arxiv.org/abs/2403.09017 Aratrust: An evaluation of trustworthiness for llms in arabic . Preprint, arXiv:2403.09017

work page Pith review arXiv 2024

[3] [3]

Ebtesam Almazrouei, Ruxandra Cojocaru, Michele Baldo, Quentin Malartic, Hamza Alobeidli, Daniele Mazzotta, Guilherme Penedo, Giulia Campesan, Mugariya Farooq, Maitha Alhammadi, Julien Launay, and Badreddine Noune. 2023. https://doi.org/10.18653/v1/2023.arabicnlp-1.21 A l G hafa evaluation benchmark for A rabic language models . In Proceedings of ArabicNLP...

doi:10.18653/v1/2023.arabicnlp-1.21 2023

[4] [4]

Alzahrani, Nouf M

M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan AlRashed, Faisal A. Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, Ahmed Abde...

arXiv 2024

[5] [5]

John Dang, Shivalika Singh, Daniel D'souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, Acyr Locatelli, Hangyu Lin, Dwarak Talupuru, Bharat Venk...

arXiv 2024

[6] [6]

AbdelRahim Elmadany, ElMoatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. https://doi.org/10.18653/v1/2023.findings-acl.609 ORCA : A challenging benchmark for A rabic language understanding . In Findings of the Association for Computational Linguistics: ACL 2023, pages 9559--9586, Toronto, Canada. Association for Computational Linguistics

work page doi:10.18653/v1/2023.findings-acl.609 2023

[7] [7]

Maria Eriksson, Erasmo Purificato, Arman Noroozian, Joao Vinagre, Guillaume Chaslot, Emilia Gomez, and David Fernandez-Llorca. 2025. https://arxiv.org/abs/2502.06559 Can we trust ai benchmarks? an interdisciplinary review of current issues in ai evaluation . Preprint, arXiv:2502.06559

arXiv 2025

[8] [8]

Nizar Habash, Fadhl Eryani, Salam Khalifa, Owen Rambow, Dana Abdulrahim, Alexander Erdmann, Reem Faraj, Wajdi Zaghouani, Houda Bouamor, Nasser Zalmout, Sara Hassan, Faisal Al-Shargi, Sakhar Alkhereyf, Basma Abdulkareem, Ramy Eskander, Mohammad Salameh, and Hind Saddiki. 2018. https://aclanthology.org/L18-1574/ Unified guidelines and resources for A rabic ...

work page 2018