Pith. sign in

REVIEW 4 major objections 6 minor 4 cited by

LLMzSz{\L}: a comprehensive LLM benchmark for Polish

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new benchmark built from almost 19,000 Polish national exam questions tests open-weight LLMs against human examinees.

desk verdict A genuinely useful new Polish exam benchmark with real public data, held back by an inflated 'largest' claim and an unquantified answer-key error rate; worth peer review with revisions. read the letter →

arxiv 2501.02266 v1 pith:3LZRQ3IJ submitted 2025-01-04 cs.CL cs.AI

Krzysztof Jassem , Micha{l} Ciesió{l}ka , Filip Graliński , Piotr Jab{l}oński , Jakub Pokrywka , Marek Kubis , Monika Jab{l}ońska , Ryszard Staruch This is my paper · ORCID
classification cs.CLcs.AI
keywords LLMzSzŁPolishLLMbenchmarknationalexamsCKEmultilingualevaluationhuman-modelcorrelationanswerkeyvalidation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces LLMzSzŁ, a benchmark built from almost 19,000 closed-ended questions taken from Polish national school and vocational exams published by the Central Examination Board. It is designed to test whether open-weight language models can handle Polish-specific knowledge and reasoning, and to compare model performance with that of human examinees by year and exam category. The authors evaluate dozens of models and report that large multilingual models score highest, while smaller Polish-tuned models remain competitive when size is constrained. They also show that model scores correlate with human pass rates on some exams, and that low model confidence can expose genuine errors in official answer keys, including one confirmed mistake in a 2018 vocational exam.

What carries the argument

The load-bearing object is the LLMzSzŁ dataset itself: nearly 19,000 single-choice Polish exam questions with official answer keys, stratified into middle-school, high-school, and vocational tiers, each item carrying the publication date of its exam. The evaluation procedure uses an open evaluation harness with an MMLU-style configuration: for each question the model computes the probability of each of the four answers and the highest-probability answer is scored against the gold key. A feature-level analysis using the Mann-Whitney U test identifies which question characteristics (numerical answers, words like 'wynosi' or 'oblicz', professional domains such as R.13) drive low model scores, and the per-year human score comparison provides the correlation evidence. The timestamps are the design feature that enables contamination-controlled evaluation and the year-by-year human-model comparison.

What would settle it

Audit a random sample of the 19,000 answer keys against independent expert judgment; if the error rate is high enough to change the order of the top models or to erase the observed human-model correlations, the benchmark's claims about model quality and exam validation would collapse.

Watch

Extended reading notes

Core claim

The central claim is that a coherent, authoritative collection of Polish national exams can serve as a comprehensive evaluation benchmark for Polish-language LLMs, at a scale not previously available. The dataset covers four exam types across 154 domains, with per-exam publication timestamps that allow contamination-aware evaluation; all questions are closed-ended with one correct answer, and the gold labels are the official answer keys. Evaluations performed with an open evaluation harness configured in the style of MMLU show that the best overall model is Mistral-Large-Instruct-2407 (67.17% accuracy), that models below about 3 billion parameters perform near the random-guess level of 25%, and that instruction-tuned variants generally outperform their base counterparts. The authors further claim that comparing model outputs with human results can help estimate exam difficulty and validate exam questions, evidenced by the discovery of a faulty answer key in the 2018 M.42 exam when a model assigned very low probability to the expected answer.

Load-bearing premise

The benchmark's ground truth is the official answer key published by the Polish Central Examination Board; the central assumption is that those keys are correct, or at least that errors are too rare to change model rankings.

Editorial extensions

If this is right

  • Polish-language model evaluation gains a public, reproducible benchmark with a random-guess baseline of 25% accuracy.
  • Researchers can use the timestamps to split questions into pre- and post-release sets, reducing the effect of training-data contamination when comparing models of different release dates.
  • The reported results imply that for high-accuracy Polish tasks, large multilingual models are the best choice, while the 11B-parameter Polish Bielik model offers a practical alternative where size is limited.
  • Using LLMs to flag low-probability expected answers can serve as a screening step for national exam quality control, with one confirmed answer-key error already found.
  • Human-model correlations on specific exams suggest that model scores could serve as a proxy for closed-question difficulty trends, potentially separating difficulty shifts in open questions from closed questions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the low-confidence detection method could be turned into a systematic audit protocol for answer keys across all exam types, rather than the single case study reported here.
  • Inference: if answer-key errors are not uniformly distributed across exam categories, the reported leaderboard may partly reflect how well a model matches CKE's answer conventions; re-scoring after expert correction of a random key sample would test this.
  • Inference: the timestamp design could serve as a template for national-exam benchmarks in other languages, offering a built-in contamination control that translated MMLU datasets lack.
  • Inference: the divergent human-model trends in biology (human scores falling, model scores rising) suggest that open questions, not closed ones, drive the human decline; this could be tested by scoring open responses with an LLM rubric.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces LLMzSzŁ, a benchmark of almost 19,000 closed-ended Polish exam questions drawn from Polish Central Examination Board (CKE) materials, covering school and vocational exams. The authors evaluate dozens of open-weight LLMs using likelihood-based accuracy, analyze performance by model size, language, release date, and instruction tuning, and investigate correlations between model scores and human examinee statistics. The dataset is released on HuggingFace, and a public leaderboard is provided.

Significance. If the gold labels are reliable, the dataset is a valuable evaluation resource for Polish, with per-exam timestamps that can support contamination-aware analysis. The systematic evaluation of a broad range of open-weight models is useful and reproducible. However, the paper's central novelty claim (being the largest and most comprehensive Polish LLM benchmark) is contradicted by its own cited references, and the known answer-key error documented in Section 5 is not resolved in the released dataset. The human-correlation analysis in Section 6 is statistically weak due to small sample sizes and incompatible metrics. These issues currently limit confidence in the paper's headline claims, though they are addressable in revision.

major comments (4)
  1. [Section 7 and Abstract] The paper claims that LLMzSzŁ is 'the largest and most comprehensive LLM benchmark developed for the Polish language that has been published to date' (Section 7). However, the cited dataset of Pokrywka et al. (2024) contains 297 tests of 120 questions each (35,640 questions), and the follow-up Łukasz Grzybowski et al. (2024) adds 144 new exams, both larger than the 'almost 19k' questions in LLMzSzŁ. Please correct or explicitly qualify this claim, for example by specifying that LLMzSzŁ is the largest multi-tier general-knowledge Polish benchmark, not the largest Polish exam-based dataset in absolute terms.
  2. [Section 5 and Section 3.1] Section 5 reports a concrete label error: in exam M.42-X-18.06, Question 24, the CKE answer key was 'erroneously specified as insolvent instead of strategic.' The paper does not state whether the released HuggingFace dataset corrects this label, nor does it describe any systematic reconciliation with CKE's official errata. Since the benchmark's gold labels are the CKE keys (Section 3.1), the existence of at least one known wrong label makes the unquantified label-error rate a load-bearing reliability issue. Please explicitly state how known errors are handled in the released dataset, document the curation process, and either provide an estimated label-error rate from a manual audit or temper the claims about answer-key reliability.
  3. [Section 6 and Table 5] The correlation analysis in Section 6 and Table 5 uses very few yearly data points (e.g., Junior High 2015–2019: n=5; 8-grade: n=5) and mixes model scores on closed questions with human scores that include open questions, while for professional exams the human values are pass rates rather than average scores. Correlations such as 0.925 (Mistral, Junior High) and 0.851 (Bielik, 8-grade) are reported without significance tests or confidence intervals. The conclusions in Section 6.4 about using LLMs to verify exam difficulty are therefore not statistically supported. Please provide p-values or confidence intervals, use comparable human metrics if available, or explicitly downgrade the strength of these conclusions.
  4. [Section 4.2 and Abstract] The abstract's claim that 'multilingual LLMs can obtain superior results over monolingual ones' is not cleanly supported by the comparisons shown. The highest-performing models are all multilingual, but there are no Polish or English models of comparable size (e.g., 70B–123B) in the evaluation; the best small model, Bielik-11B (57.52), is only directly compared with 7B multilingual models (Table 6). The comparison is thus confounded by model size and availability. Please provide a matched-size comparison (e.g., Bielik-11B against a multilingual model of similar size) or qualify the conclusion accordingly.
minor comments (6)
  1. [Table 6] In Table 6, the parameter size for 'Qwen/Qwen2-1.5B' is listed as 5 (likely 1.5), and the release date for 'trurl-2-13b-academic' is given as '23-98', which is not a valid month.
  2. [Table 7] Table 7 contains the typo 'Phisics' for 'Physics' in several rows.
  3. [Section 6.2.1] The text states that for biology 'The lack of correlation may be due to the increase in difficulty of open questions,' but Table 5 does not provide a separate biology correlation; please either report the correlation or present this as a qualitative observation.
  4. [Section 3.1] The list of selected subjects is ambiguous: 'math, natural sciences, biology, physics' suggests overlapping categories; please clarify the exam-subject taxonomy used for the benchmark.
  5. [Section 8] In the Limitations section, 'questions that where published' should be 'questions that were published.'
  6. [Section 6.4] The sentence 'If this phenomenon is confirmed with more data (possibly including open questions), it will advocate for a possible use of LLMs...' should be reworded for grammatical correctness and precision.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark is an external measurement instrument and the model evaluations are independent of the benchmark construction.

full rationale

The paper's load-bearing claims are (i) that LLMzSzŁ is a reusable Polish benchmark assembled from CKE national exam questions with CKE answer keys as gold labels, and (ii) that the reported accuracies support model comparisons. Neither claim is derived from the models or from the paper's own conclusions. Dataset construction (Section 3.1) relies on PDF extraction, manual matching, and cleaning; gold labels come from official CKE answer keys, an external source independent of the evaluated models. The evaluation harness (Section 3.2) is the standard LM Evaluation Harness with an MMLU-style prompt; model scores are measured, not fitted. There is no equation in which a predicted quantity is algebraically equal to an input, and no parameter is fitted to a subset and then reported as a prediction. The related work cites prior Polish medical exam benchmarks by overlapping authors (Pokrywka et al. 2024; Łukasz Grzybowski et al. 2024), but these citations only contextualize existing datasets and do not justify any step in the current benchmark's construction or evaluation; they are not load-bearing. The Section 5 finding that the official 2018 M.42 key labels Q24 'insolvent' instead of 'strategic' is an external-label validity concern, not circularity: the benchmark's ground truth is defined by the CKE keys, and the paper transparently reports a counterexample. If the erroneous label remains in the released dataset, accuracy numbers may be slightly affected, but this is a data-quality limitation, not a case of the output being equal to the input by construction. The acknowledged mismatch between closed-question model scores and human scores that include open questions (Section 6) is likewise a limitation of the correlation analysis, not a circular reduction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The benchmark's validity depends on CKE answer-key correctness, the representativeness of closed-ended MCQs, and the comparability of model scores to public human scores. There are no fitted free parameters and no invented entities.

assumptions (3)
  • domain assumption Answer keys from the Polish Central Examination Board are correct, or errors are rare enough to not materially affect scores.
    Section 3.1 states the credibility of CKE minimizes the risk of incorrect answers, and Section 5 reports one discovered error in a 2018 exam, showing gold labels can be wrong.
  • domain assumption Closed-ended multiple-choice performance is a valid proxy for LLM competence in Polish.
    The benchmark deliberately includes only single-answer MCQs (Section 3.1), and the Limitations section acknowledges it does not cover informal language or real-world competence.
  • domain assumption Model scores can be compared with public human scores even though model scores cover only closed questions while human scores mix open and closed questions.
    Section 6 states this limitation explicitly but proceeds with the correlation analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMzSz{\L}: a comprehensive LLM benchmark for Polish." pith.science (2026). https://pith.science/paper/3LZRQ3IJ

@misc{pith2026250102266,
  author       = {Pith},
  title        = {Pith review of: LLMzSz\L: a comprehensive LLM benchmark for Polish},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3LZRQ3IJ}},
  note         = {Machine review of arXiv:2501.02266}
}
read the original abstract

This article introduces the first comprehensive benchmark for the Polish language at this scale: LLMzSz{\L} (LLMs Behind the School Desk). It is based on a coherent collection of Polish national exams, including both academic and professional tests extracted from the archives of the Polish Central Examination Board. It covers 4 types of exams, coming from 154 domains. Altogether, it consists of almost 19k closed-ended questions. We investigate the performance of open-source multilingual, English, and Polish LLMs to verify LLMs' abilities to transfer knowledge between languages. Also, the correlation between LLMs and humans at model accuracy and exam pass rate levels is examined. We show that multilingual LLMs can obtain superior results over monolingual ones; however, monolingual models may be beneficial when model size matters. Our analysis highlights the potential of LLMs in assisting with exam validation, particularly in identifying anomalies or errors in examination tasks.

Figures

Figures reproduced from arXiv: 2501.02266 by the authors.

Figure 1
Figure 1. Models’ accuracy against their size. The points are jittered in the X-axis for better readability. The red [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Plot showing the model’s accuracy against their release date. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence

    cs.AI 2026-08 conditional novelty 6.0 of 10

    On a new Polish medical VQA benchmark from board certification exams, vision-language models perform better from question text alone than from images alone and score above chance from answer choices alone, indicating ...

  2. Jako Tako or Fluent? Presenting PoVisLE: A Polish Vision-Language Evaluation

    cs.CL 2026-08 conditional novelty 6.0 of 10

    PoVisLE, a manually constructed Polish cultural visual question-answering benchmark, shows current vision-language models reach at most 71.45% accuracy and perform worst on dialect and regionalism questions.

  3. PL-Guard: Benchmarking Language Model Safety for Polish

    cs.CL 2025-06 reject novelty 6.0 of 10

    A small Polish BERT classifier proved more robust than larger fine-tuned LLMs at classifying safe versus unsafe Polish content, including under character-level adversarial perturbations.

  4. Bielik 11B v2 Technical Report

    cs.CL 2025-05 conditional novelty 4.0 of 10

    Bielik 11B v2, a depth-upscaled Mistral model continued-pretrained on Polish data, scores at or near the top of several Polish benchmarks despite having far fewer parameters than leading rivals.

Reference graph

Works this paper leans on

29 extracted references · 11 canonical work pages · cited by 4 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, and Wanli Ouyang. 2024. https://arxiv.org/abs/2402.14762 Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues . Preprint, arXiv:2402.14762

    arXiv 2024

  4. [4]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. https...

    doi:10.5281/zenodo.12608602 2024

  5. [5]

    Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, and Pasquale Minervini. 2024. https://arxiv.org/abs/2406.04127 Are we done with MMLU ? Preprint, arXiv...

    arXiv 2024

  6. [6]

    Filip Grali \'n ski, Anna Wr \'o blewska, Tomasz Stanis awek, Kamil Grabowski, and Tomasz G \'o recki. 2019. https://www.aclweb.org/anthology/W19-4826 GE val: Tool for debugging NLP datasets and models . In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 254--262, Florence, Italy. Association for...

    work page 2019

  7. [7]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    arXiv 2020

  8. [8]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088

    arXiv 2024

Show all 29 references
  1. [9]

    Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, T...

    2023 arXiv

  2. [10]

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2024. https://arxiv.org/abs/2306.09212 CMMLU : Measuring massive multitask language understanding in Chinese . Preprint, arXiv:2306.09212

    2024 arXiv

  3. [11]

    Yuanzhi Li, S \'e bastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463

    2023 arXiv

  4. [12]

    Shuo Liu, Kaining Ying, Hao Zhang, Yue Yang, Yuqi Lin, Tianle Zhang, Chuanhao Li, Yu Qiao, Ping Luo, Wenqi Shao, and Kaipeng Zhang. 2024. https://arxiv.org/abs/2403.20194 Convbench: A multi-turn conversation evaluation benchmark with hierarchical capability for large vision-la...

    2024 arXiv

  5. [13]

    Krzysztof Ociepa, Łukasz Flis, Remigiusz Kinas, Adrian Gwoździej, and Krzysztof Wróbel. 2024 a . Bielik: A family of large language models for the polish language - development, insights, and evaluation

    2024

  6. [14]

    Krzysztof Ociepa, Łukasz Flis, Krzysztof Wróbel, Adrian Gwoździej, and SpeakLeash Team and Cyfronet Team . 2024 b . https://huggingface.co/speakleash/Bielik-7B-v0.1 Introducing bielik-7b-v0.1: Polish language model . Accessed: 2024-09-15

    2024

  7. [15]

    OPI-PG. 2024. Qra-1b model on Hugging Face . https://huggingface.co/OPI-PG/Qra-1b. Accessed: 2024-09-15

    2024

  8. [16]

    Irene Plaza, Nina Melero, Cristina del Pozo, Javier Conde, Pedro Reviriego, Marina Mayor-Rocher, and María Grandury. 2024. https://arxiv.org/abs/2406.17789 Spanish and LLM benchmarks: is MMLU lost in translation? Preprint, arXiv:2406.17789

    2024 arXiv

  9. [17]

    Jakub Pokrywka, Jeremi Kaczmarek, and Edward Gorzela \'n czyk. 2024. GPT-4 passes most of the 297 written Polish Board Certification Examinations . arXiv preprint arXiv:2405.01589

    2024 arXiv

  10. [18]

    Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John Nay, Kshitij Gupta, and Aran Komatsuzaki. 2023. https://neurips.cc/virtual/2023/74999 ARB : Advanced reasoning benchmark for large language models . NeurIPS

    2023

  11. [19]

    sdadas. 2024. Polish gpt-2 xl model on hugging face. https://huggingface.co/sdadas/polish-gpt2-xl. Accessed: 2024-09-15

    2024

  12. [20]

    Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. 2024. https://arxiv.org/abs/2402.11548 KMMLU : Measuring massive multitask language understanding in Korean . Preprint, arXiv:2402.11548

    2024 arXiv

  13. [21]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  14. [22]

    Voicelab. 2024. Trurl-2-7b model on hugging face. https://huggingface.co/Voicelab/trurl-2-7b. Accessed: 2024-09-15

    2024

  15. [23]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Network...

    2018

  16. [24]

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. WizardLM : Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244

    2023 arXiv

  17. [25]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  18. [26]

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024. Yi: Open foundation models by 01.AI . arXiv preprint arXiv:2403.04652

    2024 arXiv

  19. [27]

    Arda Yüksel, Abdullatif Köksal, Lütfi Kerem Şenel, Anna Korhonen, and Hinrich Schütze. 2024. https://arxiv.org/abs/2407.12402 TurkishMMLU : Measuring massive multitask language understanding in Turkish . Preprint, arXiv:2407.12402

    2024 arXiv

  20. [28]

    Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/117c5c8622b0d539f74f6d1fb082a2e9-Abstract-Datasets_and_Benchmarks.html M3exam: A multilingual, multimodal, multilevel benchmark for examin...

    2023

  21. [29]

    Kaczmarek, and Marek Kubis

    Łukasz Grzybowski, Jakub Pokrywka, Michał Ciesiółka, Jeremi I. Kaczmarek, and Marek Kubis. 2024. https://arxiv.org/abs/2412.00559 Polish medical exams: A new dataset for cross-lingual medical knowledge transfer assessment . Preprint, arXiv:2412.00559

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.