Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

AraSTEM: A Native Arabic Multiple Choice Question Benchmark for Evaluating LLMs Knowledge In STEM Subjects

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AraSTEM: a new native Arabic STEM benchmark that publicly available LLMs struggle with, where the best model reaches only 56% average accuracy and accuracy scales with Arabic training data more than with model size.

desk verdict Useful Arabic STEM benchmark, but question validity and release are unverified; deserves a serious referee with expectations of major revision. read the letter →

arxiv 2501.00559 v1 pith:M2RDTAFX submitted 2024-12-31 cs.CL cs.AI

Ahmad Mustapha , Hadi Al-Khansa , Hadi Al-Mubasher , Aya Mourad , Ranam Hamoud , Hasan El-Husseini , Marwah Al-Sakkaf , Mariette Awad This is my paper · ORCID
classification cs.CLcs.AI MSC 68T50
keywords AraSTEMArabicNLPbenchmarkmultiplechoiceLLMevaluationzero-shotquestionansweringmultilingualknowledgebenchmarkslanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces AraSTEM, a dataset of 11,637 native Arabic multiple-choice questions spanning math, science, physics, chemistry, biology, information technology, dentistry, pharmacy, and medicine, from primary through college level. It argues that this benchmark is genuinely challenging for current publicly available LLMs, with the best open-source model, Jais 30B Chat V3, achieving 56% accuracy while many smaller models hover near random guessing. The central claim is that AraSTEM reveals a real gap in Arabic STEM knowledge and reasoning, and that training on substantial Arabic text is what most improves performance. If correct, this gives the community a tool to measure and drive Arabic-domain competence in LLMs rather than assuming multilingual ability carries over from English.

What carries the argument

The central object is the benchmark itself: 11,637 multiple-choice questions with per-question source citations, cleaned by removing images and mathematical equations, labeled by educational level and subject, and carrying two to four answer options. The evaluation machinery is a fixed English Chain-of-Thought prompt that states the subject and level, with the model's answer read from the softmax probability of the A/B/C/D tokens. This setup lets the authors attribute differences in accuracy to model knowledge rather than to prompt wording—the claim being tested—and its output is a per-subject and per-level accuracy table that supports comparisons across model families and sizes.

What would settle it

Take a stratified random sample of AraSTEM questions whose original sources contained images or equations, present the cleaned text-only versions to Arabic-speaking STEM experts, and measure their agreement with the dataset's answer key; if expert accuracy falls well below the 56% model ceiling or experts flag many questions as unanswerable, the benchmark's claim to measure knowledge rather than prompt quality collapses. A complementary check is to restore the original images or equations to the same sample and test whether top models' accuracy rises materially.

Watch

Extended reading notes

Core claim

The paper's discovery is that a large, native Arabic STEM MCQ collection produces a wide, structured performance spread across open models, with accuracy tracking the amount of Arabic training data more than model size. The top performers are Jais, Llama 3.1 Instruct, AceGPT, and Bloomz, while general-purpose large models like Falcon 40B and Llama 2 lag noticeably. The paper also shows that model predictions are complementary—only 193 questions stumped every tested model—and that the leading models are generally overconfident except Llama 3.1, which calibrates well. These results are presented as evidence for the value of Arabic-centric training and the need for localized evaluation.

Load-bearing premise

After images and equations were stripped from scraped questions, each remaining question must still be self-contained and answerable from the Arabic text alone; if many questions became ambiguous or unanswerable, the reported accuracy gaps would reflect broken prompts rather than knowledge, and the answer key's correctness also rests on external sources and ChatGPT-4 extraction without independent human validation.

Editorial extensions

If this is right

  • Models trained with a larger share of Arabic data, such as Jais and AceGPT, outperform equally sized models with less Arabic exposure, implying that scaling Arabic tokens in pretraining is a direct lever on Arabic STEM accuracy.
  • Instruction fine-tuning consistently improves AraSTEM accuracy, as seen in Jais chat versus base and Llama 3.1 Instruct versus base, so alignment helps even in zero-shot knowledge tasks.
  • The 193-question hard-failure set, concentrated in dentistry and medicine, marks a concrete target for specialized Arabic medical knowledge.
  • Because models' correct answers overlap only partially, an ensemble of the top models would score higher than any single model, offering a practical path to better Arabic STEM question answering.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The removal of images and equations is the fragile point: the reported numbers measure text-only STEM knowledge, and restoring original figures could raise model accuracy, meaning the benchmark may understate full multimodal competence.
  • The paper does not include a human expert baseline; without one, 'challenging' is anchored only to random guessing and model scores, so a small stratified human evaluation would let the field interpret the 56% figure.
  • AraSTEM could double as a training resource, not just an exam: the labeled questions could support Arabic STEM instruction tuning, and the per-subject difficulty profiles could guide curriculum design for localized models.
  • The per-question source citations give the benchmark a traceability that many English benchmarks lack, a contribution the paper mentions but does not emphasize as a finding.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces AraSTEM, a multiple-choice question benchmark of 11,637 Arabic STEM questions spanning primary, secondary, and college levels across subjects including math, science, physics, chemistry, biology, IT, medicine, dentistry, and pharmacy. The questions were collected from web scraping, manual extraction from books, and ChatGPT-4-assisted extraction from PDFs. The authors evaluate a range of open-source LLMs in a zero-shot setting with an English chain-of-thought prompt and report that the best-performing open-source model, Jais 30B Chat V3, reaches 56% accuracy (with Table 4 reporting 58%), well above random guessing but far from ceiling. The dataset is offered on Hugging Face with per-question source links.

Significance. If the benchmark is valid, it addresses a real gap: most knowledge benchmarks are English-centric, and Arabic STEM evaluation is underrepresented. The dataset's size, source traceability, and subject breadth are genuine strengths, as is the evaluation of many model families and sizes. The paper also includes useful auxiliary analyses of tokenizer coverage, calibration, and complementary model errors. However, the central claim that low model accuracy reflects a knowledge gap depends on the cleaned text-only questions being self-contained and correctly labeled; this is not currently established. The reported numerical inconsistencies and the absence of a human-expert validation loop weaken the support for the main conclusion. With a validation study and corrected reporting, the benchmark could be a valuable community resource.

major comments (4)
  1. [Section 3.1] The cleaning process removed images and mathematical equations from scraped questions, but the paper does not establish that the remaining Arabic text alone makes each question self-contained and unambiguously answerable. This is load-bearing because if a nontrivial fraction of questions originally relied on diagrams, graphs, or formulas, the reported accuracies (e.g., Jais 30B Chat V3 at 56–58%) would measure unparseable or misleading prompts rather than STEM knowledge. Please add a human-expert validation step, at least on a stratified sample, that reports the fraction of questions judged answerable from the text alone, and repair or remove items that fail this check.
  2. [Section 3.1 / Table 1] Gold labels are not independently verified for correctness. The 1,806 questions extracted with ChatGPT-4 from PDFs were proofread for format rather than correctness, and the manual transformation of definition and fill-in-the-blank questions into MCQs could have introduced wrong answers. Without a double-annotated human sample with agreement statistics, the evaluation numbers may reflect errors in the reference answers rather than model knowledge. Please include a label-quality audit, for example a random sample of 200–500 questions annotated by Arabic-speaking STEM experts, with inter-annotator agreement and a report of how disagreements were resolved.
  3. [Section 4.5 and Tables 4/5] Several reported numbers are inconsistent. Section 4.5 describes 193 hard-failure questions as '0.16%' of the dataset, but 193/11,637 is 1.66%. Table 4 gives Jais 30B Chat V3 an average of 0.58 while Table 5 gives 0.56 and the abstract states 56%; the random-guess baselines also differ between tables (0.30 in Table 4 vs 0.29 in Table 5). Please reconcile these values and provide the evaluation script or per-question scores to make the results reproducible and to confirm which summary is the final one.
  4. [Section 4] The scoring procedure is underspecified for questions with two or three options. The paper states that models' choices were extracted as the probability of tokens 'A', 'B', 'C', or 'D', but Table 3 shows 2,562 questions with only two options and 3,808 questions with three options. Additionally, the prompt in Figure 7 asks for a 'Correct option number' while scoring uses option letters, and Section 4 says prompts exceeding a model's maximum input size were truncated, which could remove part of the question or the options. Please clarify how variable option counts were mapped to the answer extraction and report how many prompts were truncated.
minor comments (6)
  1. [Abstract] The abstract contains a typo: 'asses' should be 'assess'.
  2. [Section 1] 'The rest of this manuscript is organized as following' should be 'as follows', and 'it's details' should be 'its details'.
  3. [Table 4] The column header 'Collage' should be 'College'.
  4. [Section 4.2] The model name 'Lama 3.1' should be spelled 'Llama 3.1'.
  5. [References] Reference [7] contains a garbled sentence with '¡br/¿' and 'doesnt'; the entry needs to be cleaned. Reference [22] has a malformed author field, and reference [29] has inconsistent capitalization and spacing ('ZINI', 'A W AD').
  6. [Footnote 1] The availability statement is inconsistent: the abstract says the dataset is freely accessible on Hugging Face, while the footnote says 'Available post publication'. Please clarify the current availability status.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: AraSTEM is an externally sourced MCQ benchmark evaluated zero-shot, and the self-citations are background only.

full rationale

AraSTEM is a dataset-construction and zero-shot benchmark-evaluation paper: it reports accuracy of fixed public models on a fixed external MCQ collection. There are no fitted parameters, no derived quantities that equal their inputs by construction, and no prediction that reduces to a fitted value. The only potentially self-referential element is the use of ChatGPT-4 as an OCR/extraction tool for 1,806 medical questions from PDF files (Section 3.1); those questions originate from external medical colloquiums, ChatGPT-4 is not among the evaluated models, and the extraction was manually proofread, so the benchmark claim is not defined in terms of the model being benchmarked. The self-citations ([17], [20], [29]) appear in related work and future work only and are not load-bearing for the central claim that open models score around 56%. Therefore no circularity is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No new physical or theoretical entities are introduced. The benchmark is a data resource. The central claims rest on the reliability of external sources, the text-only cleaning process, and the chosen scoring method, as listed above.

assumptions (3)
  • domain assumption The external sources (websites and books) provide correct answers for the questions.
    The dataset compiles questions from beadaya.com, alloschool.com, reference books, and other sources without independent validation. If these sources contain errors, the benchmark and all reported accuracies are affected. This is an unverified external dependency.
  • domain assumption The extraction pipeline (scraping, manual conversion, ChatGPT-4-based OCR with proofreading) preserves the questions' intended meaning after removal of images and equations.
    Section 3.1 states images and mathematical equations were removed during cleaning. Many STEM questions require figures or equations to be answerable. The paper does not assess how often this removal makes a question unanswerable or ambiguous.
  • domain assumption The zero-shot evaluation via next-token probability over A/B/C/D accurately captures model performance.
    Section 4 states the models' choices were extracted as the probability of predicting the tokens A/B/C/D after the prompt, without letting the model generate its chain-of-thought. This may undervalue or bias models that need to generate reasoning before selecting an answer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AraSTEM: A Native Arabic Multiple Choice Question Benchmark for Evaluating LLMs Knowledge In STEM Subjects." pith.science (2026). https://pith.science/paper/M2RDTAFX

@misc{pith2026250100559,
  author       = {Pith},
  title        = {Pith review of: AraSTEM: A Native Arabic Multiple Choice Question Benchmark for Evaluating LLMs Knowledge In STEM Subjects},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M2RDTAFX}},
  note         = {Machine review of arXiv:2501.00559}
}
read the original abstract

Large Language Models (LLMs) have shown remarkable capabilities, not only in generating human-like text, but also in acquiring knowledge. This highlights the need to go beyond the typical Natural Language Processing downstream benchmarks and asses the various aspects of LLMs including knowledge and reasoning. Numerous benchmarks have been developed to evaluate LLMs knowledge, but they predominantly focus on the English language. Given that many LLMs are multilingual, relying solely on benchmarking English knowledge is insufficient. To address this issue, we introduce AraSTEM, a new Arabic multiple-choice question dataset aimed at evaluating LLMs knowledge in STEM subjects. The dataset spans a range of topics at different levels which requires models to demonstrate a deep understanding of scientific Arabic in order to achieve high accuracy. Our findings show that publicly available models of varying sizes struggle with this dataset, and underscores the need for more localized language models. The dataset is freely accessible on Hugging Face.

Figures

Figures reproduced from arXiv: 2501.00559 by the authors.

Figure 1
Figure 1. Semantic Embedding of AraSTEM based on E5 multilingual embedding model. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. A sample from AraSTEM questions corresponding to primary and secondary levels tions in subjects like math, physics, chemistry, and information technology [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. A sample from AraSTEM questions featuring college-level medicine question 4 [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The distribution of AraSTEM question’s word count presented per subject [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: A sample from AraSTEM questions featuring college level ones in both chemistry and biology [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: The word cloud illustration of AraSTEM questions of the physics subject Semantic Distribution. To examine the dataset’s diversity from a semantic perspective, we utilized the multilingual text embedding model E5- Large [23] to project all the questions into an embed￾di…
Figure 7
Figure 7. Figure 7: The prompt used in the evaluation of several models performance on AraSTEM dataset Each model was tasked with answering the dataset’s questions by selecting from the available options. The models’ choices were extracted as the probability of predicting each one of the …
Figure 8
Figure 8. Figure 8: The relationship between model performance on AraSTEM and model size shows that when models are trained on Arabic data, increasing the model size significantly boosts performance The figure reveals that three models - Jais, Bloomz, and AceGPT - only experience a perfor…
Figure 9
Figure 9. Figure 9: A radar plot to show the performance of selected models per subject on the AraSTEM dataset Among all the models, Llama 3.1 and Bloomz stands out for achieving high accuracy despite hav￾ing a smaller size. Bloomz 7B1, for example, per￾forms on par with AceGPT 13B. The k…
Figure 11
Figure 11. Figure 11: An upset plot to show the intersection of [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: AraSTEM questions word cloud for each subject [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. 3LM: Bridging Arabic, STEM, and Code through Benchmarking

    cs.CL 2025-07 conditional novelty 6.0 of 10

    3LM provides open Arabic benchmarks for native and synthetic STEM multiple-choice questions and translated HumanEval/MBPP code tasks, with evaluations of 40 models.

  2. ARB: A Comprehensive Arabic Multimodal Reasoning Benchmark

    cs.CV 2025-05 conditional novelty 6.0 of 10

    ARB provides 1,356 Arabic multimodal questions with 5,119 human-reviewed reasoning steps and shows leading models score much higher on reasoning fluency than on correct answers.

  3. From Guidelines to Practice: A New Paradigm for Arabic Language Model Evaluation

    cs.CL 2025-06 conditional novelty 4.0 of 10

    On a new 490-question Arabic depth dataset, Claude 3.5 Sonnet answered about 30 percent correctly, while GPT-4 answered about 9 percent, showing current models are weak on culturally specialized Arabic knowledge.

Reference graph

Works this paper leans on

29 extracted references · 26 canonical work pages · cited by 3 Pith papers

  1. [1]

    Crosslingual generalization through multitask finetuning,

    N. Muennighoff, T. Wang, L. Sutawika, A. Roberts, S. Biderman, T. L. Scao, M. S. Bari, S. Shen, Z.-X. Yong, H. Schoelkopf, X. Tang, D. Radev, A. F. Aji, K. Almubarak, S. Albanie, Z. Alyafeai, A. Webson, E. Raff, and C. Raffel, “Crosslingual generalization through multitask finetuning,” 11 2022

    work page 2022

  2. [2]

    Aya dataset: An open-access collection for multilingual in- struction tuning,

    S. Singh, F. Vargus, D. Dsouza, B. F. Karls- son, A. Mahendiran, W.-Y. Ko, H. Shandilya, J. Patel, D. Mataciunas, L. OMahony, M. Zhang, R. Hettiarachchi, J. Wilson, M. Machado, L. S. Moura, D. Krzeminski, H. Fadaei, 11 I. Erg ˜A¼n, I. Okoh, A. Alaagib, O. Mudan- nayake, Z. Alyafeai, V. M. Chien, S. Ruder, S. Guthikonda, E. A. Alghamdi, S. Gehrmann, N. Mue...

    work page 2024

  3. [3]

    Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models,

    N. Sengupta, S. K. Sahu, B. Jia, S. Katipomu, H. Li, F. Koto, W. Marshall, G. Gosal, C. Liu, Z. Chen, O. M. Afzal, S. Kamboj, O. Pandit, R. Pal, L. Pradhan, Z. M. Mujahid, M. Baali, X. Han, S. M. Bsharat, A. F. Aji, Z. Shen, Z. Liu, N. Vassilieva, J. Hestness, A. Hock, A. Feld- man, J. Lee, A. Jackson, H. X. Ren, P. Nakov, T. Baldwin, and E. Xing, “Jais a...

    work page 2023

  4. [4]

    Mea- suring massive multitask language understand- ing,

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Mea- suring massive multitask language understand- ing,” 9 2020

    work page 2020

  5. [5]

    Hellaswag: Can a machine really finish your sentence?,

    R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi, “Hellaswag: Can a machine really finish your sentence?,” 5 2019

    work page 2019

  6. [6]

    Winogrande: An adversarial winograd schema challenge at scale,

    K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi, “Winogrande: An adversarial winograd schema challenge at scale,” 7 2019

    work page 2019

  7. [7]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Sto- ica, “Judging llm-as-a-judge with mt-bench and chatbot arena,” 6 2023. The aim is to develop a way to evaluate LMMs. Paradoxly enough the authors claimes GPT-4 can be use to judge LLMs.¡br/¿While doesnt make sense the auth...

    work page 2023

  8. [8]

    Measuring mathematical problem solving with the math dataset,

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt, “Measuring mathematical problem solving with the math dataset,” 3 2021

    work page 2021

Show all 29 references
  1. [9]

    Challenging big-bench tasks and whether chain-of-thought can solve them,

    M. Suzgun, N. Scales, N. Sch ˜A¤rli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowd- hery, Q. V. Le, E. H. Chi, D. Zhou, and J. Wei, “Challenging big-bench tasks and whether chain-of-thought can solve them,” 10 2022

    2022

  2. [10]

    Drop: A read- ing comprehension benchmark requiring discrete reasoning over paragraphs,

    D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner, “Drop: A read- ing comprehension benchmark requiring discrete reasoning over paragraphs,” 3 2019

    2019

  3. [11]

    Mmlu-pro: A more ro- bust and challenging multi-task language under- standing benchmark,

    Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen, “Mmlu-pro: A more ro- bust and challenging multi-task language under- standing benchmark,” 6 2024

    2024

  4. [12]

    Gpqa: A graduate-level google-proof q&a benchmark,

    D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bow- man, “Gpqa: A graduate-level google-proof q&a benchmark,” 11 2023

    2023

  5. [13]

    Instruction- following evaluation for large language models,

    J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou, “Instruction- following evaluation for large language models,” 11 2023

    2023

  6. [14]

    Multilingual massive multitask language under- standing (mmmlu)

    “Multilingual massive multitask language under- standing (mmmlu).” https://huggingface. co/datasets/openai/MMMLU. Accessed: 2024- 10-08

    2024

  7. [15]

    Arabicmmlu: As- sessing massive multitask language understand- ing in arabic,

    F. Koto, H. Li, S. Shatnawi, J. Doughman, A. B. Sadallah, A. Alraeesi, K. Almubarak, Z. Alyafeai, N. Sengupta, S. Shehata, N. Habash, P. Nakov, and T. Baldwin, “Arabicmmlu: As- sessing massive multitask language understand- ing in arabic,” 2 2024

    2024

  8. [16]

    A deep neural network optimized by a genetic al- gorithm to improve arabic sentiment classifica- tion,

    O. Alharbi, A. Hamed, and M. Alzoubi, “A deep neural network optimized by a genetic al- gorithm to improve arabic sentiment classifica- tion,” Ing ˜A©nierie Des Syst˜A¨mes D Informa- tion, vol. 28, pp. 67–75, 2023. 12

    2023

  9. [17]

    Weighted entropy corti- cal algorithms for isolated arabic speech recogni- tion,

    N. Hajj and M. Awad, “Weighted entropy corti- cal algorithms for isolated arabic speech recogni- tion,” in Proceedings of the International Joint Conference on Neural Networks , Institute of Electrical and Electronics Engineers Inc., 2013

    2013

  10. [18]

    Non-diacritized arabic speech recog- nition based on cnn-lstm and attention-based models,

    H. Alsayadi, A. Abdelhamid, I. Hegazy, and Z. Fayed, “Non-diacritized arabic speech recog- nition based on cnn-lstm and attention-based models,” Journal of Intelligent & Fuzzy Systems, vol. 41, pp. 6207–6219, 2021

    2021

  11. [19]

    Qalam : A multimodal llm for arabic optical character and handwriting recognition,

    G. Bhatia, E. M. B. Nagoudi, F. Alwajih, and M. Abdul-Mageed, “Qalam : A multimodal llm for arabic optical character and handwriting recognition,” 7 2024

    2024

  12. [20]

    Towards a deep learning question-answering specialized chatbot for objective structured clin- ical examinations,

    J. El Zini, Y. Rizk, M. Awad, and J. Antoun, “Towards a deep learning question-answering specialized chatbot for objective structured clin- ical examinations,” in 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1– 9, 2019

    2019

  13. [21]

    Question dif- ficulty prediction for multiple choice problems in medical exams,

    Z. Qiu, X. Wu, and W. Fan, “Question dif- ficulty prediction for multiple choice problems in medical exams,” in Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM19, (New York, NY, USA), pp. 139–148, Association for Computing Mac...

    2019

  14. [22]

    The data provenance initiative: A large scale audit of dataset licensing & attribution in ai

    S. Longpre, ˆ a. R. Mahari, A. Chen, N. Obeng- Marnu, D. Sileo, W. Brannon, N. Muennighoff, N. Khazam, J. Kabbara, K. P. Xinyi, A. Wu, E. Shippole, K. Bollacker, T. Wu, L. Villa, S. Pentland, S. Hooker, H. L. School, and U. C. Irvine, “The data provenance initiative: A large s...

  15. [23]

    Multilingual e5 text em- beddings: A technical report,

    L. Wang, N. Yang, X. Huang, L. Yang, R. Ma- jumder, and F. Wei, “Multilingual e5 text em- beddings: A technical report,” arXiv preprint arXiv:2402.05672, 2024

    2024 arXiv

  16. [24]

    Umap: Uniform manifold approximation and projection for dimension reduction,

    L. McInnes, J. Healy, and J. Melville, “Umap: Uniform manifold approximation and projection for dimension reduction,” 2020

    2020

  17. [25]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” 2023

    2023

  18. [26]

    AceGPT, localizing large language models in Arabic,

    H. Huang, F. Yu, J. Zhu, X. Sun, H. Cheng, S. Dingjie, Z. Chen, M. Alharthi, B. An, J. He, Z. Liu, J. Chen, J. Li, B. Wang, L. Zhang, R. Sun, X. Wan, H. Li, and J. Xu, “AceGPT, localizing large language models in Arabic,” in Proceedings of the 2024 Conference of the North Amer...

    2024

  19. [27]

    The llama 3 herd of models,

    Meta-AI-Llama-team, “The llama 3 herd of models,” 2024

    2024

  20. [28]

    Training compute-optimal large language models,

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifr...

    2022

  21. [29]

    On the explain- ability of natural language processing deep mod- els,

    J. E. ZINI and M. A W AD, “On the explain- ability of natural language processing deep mod- els,” in IEEE International Conference on Pro- gram Comprehension, vol. 2022-March, pp. 36– 47, IEEE Computer Society, 2022. 13 7 Appendix 7.1 Word Cloud In this section, we present the...

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.