Pith. sign in

REVIEW 5 major objections 4 minor 71 references

MedBench-IT: A Comprehensive Benchmark for Evaluating Large Language Models on Italian Medical Entrance Examinations

T0 review · 5 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MedBench-IT is the first large-scale benchmark for Italian medical entrance exams, and the paper uses it to rank LLMs, finding top models near 90 percent accuracy with logic and mathematics as the persistent weak point.

desk verdict MedBench-IT fills a real gap in Italian medical entrance exam benchmarks, but the unverified answer keys and placeholder examples in the appendix make the reported accuracies provisional until a label audit is done. read the letter →

arxiv 2509.07135 v1 pith:CCAPLEAO submitted 2025-09-08 cs.CL

Ruggero Marino Lazzaroni , Alessandro Angioi , Michelangelo Puliga , Davide Sanna , Roberto Marras This is my paper · ORCID
classification cs.CL
keywords LLMevaluationbenchmarkItalianNLPmedicalentranceexammultiple-choicequestionansweringreproducibilityorderingbiaschain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces MedBench-IT, a benchmark of 17,410 expert-written, multiple-choice questions drawn from Italian medical university entrance examinations, spanning six subjects and three difficulty levels. Its aim is to give the Italian NLP and educational-technology communities a native-language yardstick for measuring how well large language models handle this high-stakes task. The evaluation reports a clear performance hierarchy: the best API-based models reach roughly 90 percent accuracy, resource-efficient open-source models under 30 billion parameters exceed 70 percent, and most smaller Italian fine-tunes land near 60 percent. It also finds that logic and mathematics are the hardest subjects for every model, that GPT-4o reproduces the same answer only 88.86 percent of the time across identical runs, and that requiring explicit reasoning rarely improves top-model accuracy.

What carries the argument

The load-bearing object is the dataset itself: 17,410 multiple-choice questions selected from a 43,525-question corpus supplied by a leading Italian preparatory publisher. The construction pipeline removes image-dependent and English-language items, strips markup, normalizes every item to a question stem with five options and one correct answer, and takes a stratified sample preserving original subject and difficulty proportions. The evaluation protocol then applies a fixed Italian prompt format under two conditions, direct answering and reasoning-eliciting, keeping temperature at zero for main runs, which allows accuracy, reproducibility, ordering-bias, and readability analyses to be compared on identical inputs.

What would settle it

Audit a random sample of the benchmark questions with independent Italian medical educators and re-run the top models on the 26,115 questions from the original corpus that were excluded; non-trivial gold-answer errors or a large accuracy gap on the excluded set would invalidate the reported scores.

Watch

Extended reading notes

Core claim

The central claim is that MedBench-IT is the first comprehensive benchmark built specifically for Italian medical entrance examinations, sourced from the publisher Edizioni Simone rather than translated from English tests. On this benchmark the paper reports model accuracies as evidence of current capability: DeepSeek-R1 scores 91.9 percent, o1-preview 89.1 percent, Claude 3.5 Sonnet 87.8 percent, and GPT-4o 83.9 percent under a direct-answer prompt, while the strongest sub-30B open models, Phi-4 and Qwen 2.5 14B, reach 76.8 and 72.6 percent. The accompanying robustness results claim that answer-order shuffling has a minimal effect on GPT-4o but a statistically significant effect on Claude 3.5 Haiku, that GPT-4o's response consistency across identical runs is 88.86 percent with subject-dependent variation, and that reasoning-eliciting prompts do not systematically improve accuracy.

Load-bearing premise

The publisher's answer keys and difficulty labels are correct, and the stratified sample of 17,410 questions faithfully represents the full 43,525-question corpus; if either assumption fails, the reported accuracies and rankings do not measure what they claim.

Editorial extensions

If this is right

  • Italian EdTech developers gain a native-language benchmark for model selection, with sub-30 billion parameter models like Phi-4 and Qwen 2.5 14B already scoring above 70 percent.
  • Logic and mathematics appear as a consistent bottleneck across all tested models, so tutoring or admissions-support tools should be validated on those subjects separately.
  • The 88.86 percent reproducibility rate for GPT-4o means single-run accuracy differences of a few points may be noise; evaluations should report consistency intervals.
  • The weak effect of reasoning prompts suggests chain-of-thought prompting cannot be assumed to help on Italian multiple-choice medical questions, especially for strong models.
  • The significant ordering sensitivity found in Claude 3.5 Haiku implies answer-order robustness should be part of any acceptance test for deployed exam-answering systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If MedBench-IT becomes a standard fixture, per-subject scores could double as a diagnostic for Italian medical curricula, pinpointing whether preparation materials should emphasize reasoning practice over factual review.
  • Because readability has only a small inverse association with accuracy, the benchmark appears to measure domain knowledge and reasoning rather than Italian language proficiency; running the same questions in machine translation would test that interpretation directly.
  • The benchmark's text-only design leaves diagram-dependent items, common in real Italian medical exams, unmeasured; extending the pipeline to images would be a natural next step for evaluating multimodal models.
  • An independent audit of the publisher's answer keys would strengthen the benchmark's validity claims, since any systematic gold-answer errors would directly distort model rankings.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. MedBench-IT is presented as the first large-scale benchmark for Italian medical university entrance examinations. The authors curate 17,410 multiple-choice questions from Edizioni Simone across six subjects and three difficulty levels, evaluate 20 proprietary and open-source LLMs under zero-shot standard and reasoning-eliciting prompts, and compute accuracy overall and by subject/difficulty. Additional analyses address reproducibility (GPT-4o, two runs), answer-ordering bias (GPT-4o and Claude 3.5 Haiku), the effect of reasoning prompts, and readability via Flesch-Vacca logistic regressions. The paper reports that top models exceed 90% accuracy, that reproducibility is 88.86% for GPT-4o, that ordering bias is small, and that readability has a statistically significant but tiny inverse relation with accuracy.

Significance. If the dataset and labels are reliable, MedBench-IT fills a genuine gap: native Italian, domain-specific, exam-style MCQ evaluation, with a useful model zoo spanning 0.5B to 671B parameters. The evaluation protocol is largely sound: sampling temperatures are stated, full prompt templates are provided, McNemar tests are used for two of the analyses, and the reasoning/readability decompositions are sensible. The paper also reports negative or null results (small CoT gains, mixed ordering-bias significance), which strengthens confidence in the authors' reporting. However, the benchmark's value depends on the correctness of the gold answers and on the representativeness of the 17,410-question sample, and both are currently not independently checkable because the data are proprietary. The placeholder labels in Appendix A are a concrete red flag that the answer-key pipeline may be incomplete; this must be resolved before the accuracy scores can be interpreted.

major comments (5)
  1. [Appendix A (A.1 and A.4)] Two of the four additional examples contain placeholder correct-answer fields such as '[Index for ...]' and placeholder option lists, and the examples are presented as sample questions from the dataset. This is internal evidence that the answer-key/labeling pipeline is not fully verified. Because Eq. (1) and all accuracy scores in Tables 3 and 4 depend on gold labels, the authors should run an independent re-annotation of a random sample (e.g., 300–500 questions) against the Edizioni Simone keys, report inter-annotator agreement, and replace the placeholder examples with completed, checked items.
  2. [Section 7 (Data Availability)] The dataset is not redistributable and cannot be inspected by reviewers, so the central claim of a 17,410-question benchmark with correct gold answers is not independently verifiable. For a resource paper of this type, the authors should at minimum release a representative public sample with gold labels and the exact prompts, report duplicate/near-duplicate checks and formatting-validation statistics, and provide a clear data-sharing agreement that lets independent researchers reproduce at least a subset. The five appendix examples are not sufficient to establish corpus quality.
  3. [Section 7 (Limitations) and Section 5.1] The contamination risk is acknowledged only as 'cannot be entirely ruled out, even if unlikely given our data source.' Given that the questions come from a commercial publisher with a known distribution channel and that the top models reach about 90% accuracy, the risk is not negligible. I ask for a concrete memorization check on a random sample of questions (e.g., prompting models to complete or answer from the stem alone) and a statement of model training cutoffs relative to the data acquisition period. Without this, the reported accuracies could partly reflect memorized publisher material rather than capability.
  4. [Table 3 and Appendix B] Aya Expanse 8B drops from 46.7% accuracy under the standard prompt to 0.1% under the reasoning prompt, with per-subject reasoning values between 0.1% and 0.4%. This pattern strongly indicates a format-following failure rather than a genuine accuracy measurement. Reporting these values as benchmark results is misleading. The paper should either exclude such runs with an explicit non-compliance criterion or report format-success rates separately.
  5. [Section 5.3 (Ordering Bias)] The shuffle protocol is underspecified: it is not stated whether each question received one random permutation, whether the same shuffled order was reused across models, or how many shuffled runs were averaged. The analysis also covers only two models. Since the abstract claims a 'rigorous' ordering-bias analysis, the experimental design should be described precisely and the mixed McNemar results should be accompanied by effect sizes and confidence intervals.
minor comments (4)
  1. [Section 4.3 and 5.2] The reproducibility test uses temperature 1, while the main evaluation uses temperature 0. The paper should justify this choice explicitly, since a reproducibility test at temperature 0 is a different and arguably more relevant measurement of deterministic consistency.
  2. [Section 3.3 and 5.1] The paper does not specify how malformed model outputs (e.g., answers that are not a single number 1–5, or reasoning text without an answer) were parsed and scored. Please add a paragraph on output parsing and whether such cases are counted as incorrect.
  3. [Table 3 and Appendix B] Several model identifiers are incomplete or inconsistent (e.g., 'Lexora Med. 7B', 'Maestrale v0.4', and 'Gemma 2 9B' versus the Hugging Face identifiers mentioned in the text). Provide exact model versions and hyperparameters in a reproducibility appendix.
  4. [Figures 1–3] The figures are referenced but their construction details are minimal; for Figure 3, the blue/red distinction should be supplemented with markers or hatching to remain accessible to color-blind readers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported accuracies are held-out measurements against publisher-supplied gold answers, and no fitted parameter is renamed as a prediction.

full rationale

The paper's central claims are (i) the construction of a benchmark dataset of 17,410 questions with publisher-supplied answers and (ii) measured model accuracies computed as the percentage of correct answers (Eq. 1). Nothing in the accuracy computation is fitted to the target answers, and no parameter is estimated from a subset of the data and then used to predict that same subset. The subject and difficulty distributions are descriptive metadata, not outputs of a model or theory. The specialized analyses (reproducibility, ordering bias, reasoning-prompt comparison, readability logistic regression) are external measurements: the readability regression is a separate statistical analysis relating text features to accuracy outcomes, not a component of the accuracy definition. Citations to prior benchmarks and Italian NLP resources are contextual and none is load-bearing as an unverified premise; no uniqueness theorem or ansatz is imported from the authors' own prior work. The appendix examples containing placeholder answer indices (e.g., '[Index for ...]') raise a legitimate data-quality concern about label completeness, but an incorrect or missing gold label would make the benchmark inaccurate or incomplete, not make the derivation circular: the reported scores remain measurements against whatever labels are in the dataset. The paper is self-contained as an empirical evaluation, so the correct circularity score is 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central evaluation is empirical measurement; there are no fitted model parameters in the benchmark construction itself. The main unverified inputs are data quality assumptions inherited from the publisher and choices about sampling, text-only filtering, and readability scoring.

assumptions (5)
  • domain assumption The questions and answer keys provided by Edizioni Simone are expert-written and correct.
    Section 3.1 states questions are 'expert-authored to accurately reflect official Italian medical admission exam style, content, and difficulty.' Accuracy numbers inherit the correctness of these gold labels.
  • domain assumption The stratified sample of 17,410 questions preserves subject and difficulty proportions of the original 43,525-question corpus.
    Section 3.1 says filtering and selection maintained original subject and difficulty proportions, but no distribution comparison or sampling details are shown.
  • domain assumption Removing image-reliant and English questions leaves a representative text-only version of the exam.
    Section 3.1 excludes image-reliant questions; the authors acknowledge in Section 7 that the benchmark is text-only and cannot evaluate multimodal reasoning.
  • domain assumption The textstat implementation of the Flesch-Vacca formula is a valid Italian readability measure.
    Section 4.3.4 uses the textstat library without validating the formula against human readability judgments or Italian exam language.
  • domain assumption The evaluated models did not memorize MedBench-IT questions during pretraining.
    Section 7 states that data contamination cannot be entirely ruled out, even if unlikely given the proprietary source; if contamination occurred, accuracy would reflect memorization rather than capability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MedBench-IT: A Comprehensive Benchmark for Evaluating Large Language Models on Italian Medical Entrance Examinations." pith.science (2026). https://pith.science/paper/CCAPLEAO

@misc{pith2026250907135,
  author       = {Pith},
  title        = {Pith review of: MedBench-IT: A Comprehensive Benchmark for Evaluating Large Language Models on Italian Medical Entrance Examinations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CCAPLEAO}},
  note         = {Machine review of arXiv:2509.07135}
}
read the original abstract

Large language models (LLMs) show increasing potential in education, yet benchmarks for non-English languages in specialized domains remain scarce. We introduce MedBench-IT, the first comprehensive benchmark for evaluating LLMs on Italian medical university entrance examinations. Sourced from Edizioni Simone, a leading preparatory materials publisher, MedBench-IT comprises 17,410 expert-written multiple-choice questions across six subjects (Biology, Chemistry, Logic, General Culture, Mathematics, Physics) and three difficulty levels. We evaluated diverse models including proprietary LLMs (GPT-4o, Claude series) and resource-efficient open-source alternatives (<30B parameters) focusing on practical deployability. Beyond accuracy, we conducted rigorous reproducibility tests (88.86% response consistency, varying by subject), ordering bias analysis (minimal impact), and reasoning prompt evaluation. We also examined correlations between question readability and model performance, finding a statistically significant but small inverse relationship. MedBench-IT provides a crucial resource for Italian NLP community, EdTech developers, and practitioners, offering insights into current capabilities and standardized evaluation methodology for this critical domain.

Figures

Figures reproduced from arXiv: 2509.07135 by the authors.

Figure 1
Figure 1. Reproducibility of GPT-4o responses (identical answer choice %) across different subjects on MedBench-IT. Mathematics consistently emerge as most challenging for nearly all models. Top models often score 15-25 percent￾age points lower in Logic compared to Biology or Chem￾istry (e.g., GPT-4o: 92.4% in Biology vs 64.9% in Logic). This suggests abstract reasoning and multi-step problem￾solving remain significant hurdle… view at source ↗
Figure 2
Figure 2. Performance comparison for GPT-4o and Claude 3.5 Haiku on Standard vs. Shuffled MedBench-IT benchmark. McNemar’s test revealed mixed results: GPT-4o showed no statistically significant ordering bias (p > 0.05), while Claude 3.5 Haiku exhibited significant po￾sitional sensitivity (p < 0.001). These results demon￾strate MedBench-IT’s ability to detect ordering bias when present, revealing model-specific robustness dif… view at source ↗
Figure 3
Figure 3. Accuracy comparison for selected models using Standard (blue) vs. Reasoning-Eliciting (red) prompts on MedBench￾IT. Models sorted by ascending standard prompt accuracy. DeepSeek Reasoner, o1-preview, and GPT-4o performed slightly worse with reasoning prompts. Some mid-range or smaller models, such as Llama 3.1 8B Instruct, showed slight increases. This suggests capable models efficiently arrive at an￾swers without r… view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

71 extracted references · 54 canonical work pages

  1. [1]

    T. B. Brown, et al., Language models are few-shot learners, 2020.arXiv:2005.14165

    arXiv 2020

  2. [2]

    Kasneci, K

    E. Kasneci, K. Seßler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. Günnemann, E. Hüllermeier, et al., Chatgpt for good? on opportunities and challenges of large language models for education, Learning and Indi- vidual Differences 103 (2023) 102274

    work page 2023

  3. [3]

    Baidoo-Anu, L

    D. Baidoo-Anu, L. O. Ansah, Education in the era of generative artificial intelligence (ai): Understand- ing the potential benefits of chatgpt in promoting teaching and learning, Journal of AI 7 (2023) 52–62

    work page 2023

  4. [4]

    Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, W. Chen, MMLU-Pro: A More Robust and Challenging Multi-Task Language Understand- ing Benchmark, 2024. URL: http://arxiv.org/abs/ 2406.01574. doi: 10.48550/arXiv.2406.01574, arXiv:2406.01574 [cs]

    arXiv doi:10.48550/arxiv.2406.01574 2024

  5. [5]

    Wang, et al., SuperGLUE: A Stickier Bench- mark for General-Purpose Language Understand- ing Systems, 2020

    A. Wang, et al., SuperGLUE: A Stickier Bench- mark for General-Purpose Language Understand- ing Systems, 2020. URL: http://arxiv.org/abs/ 1905.00537. doi: 10.48550/arXiv.1905.00537, arXiv:1905.00537 [cs]

    arXiv doi:10.48550/arxiv.1905.00537 2020

  6. [6]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, J. Steinhardt, Measuring Massive Multitask Language Understanding,

    work page

  7. [7]

    Attanasio, P

    G. Attanasio, P. Delobelle, M. La Quatra, A. San- tilli, B. Savoldi, ItaEval and TweetyIta: A New Ex- tensive Benchmark and Efficiency-First Language Model for Italian, in: F. Dell’Orletta, A. Lenci, S. Montemagni, R. Sprugnoli (Eds.), Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024), CEUR Workshop Pro- ceedings, Pisa...

    work page 2024

  8. [8]

    Moroni, S

    L. Moroni, S. Conia, F. Martelli, R. Navigli, To- wards a more comprehensive evaluation for Italian LLMs, in: F. Dell’Orletta, A. Lenci, S. Montemagni, R. Sprugnoli (Eds.), Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC- it 2024), CEUR Workshop Proceedings, Pisa, Italy, 2024, pp. 584–599. URL: https://aclanthology.org/ 2024....

    work page 2024

Show all 71 references
  1. [9]

    Attanasio, P

    G. Attanasio, P. Basile, F. Borazio, D. Croce, M. Fran- cis, J. Gili, E. Musacchio, M. Nissim, V. Patti, M. Rinaldi, D. Scalena, CALAMITA: Challenge the abilities of LAnguage models in ITAlian, in: F. Dell’Orletta, A. Lenci, S. Montemagni, R. Sprug- noli (Eds.), Proceedings of...

    2024

  2. [10]

    Wang, et al., GLUE: A Multi-Task Bench- mark and Analysis Platform for Natural Language Understanding, 2019

    A. Wang, et al., GLUE: A Multi-Task Bench- mark and Analysis Platform for Natural Language Understanding, 2019. URL: http://arxiv.org/abs/ 1804.07461. doi: 10.48550/arXiv.1804.07461, arXiv:1804.07461 [cs]

    2019 doi arXiv

  3. [11]

    D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, P. Szolovits, What Disease Does This Patient Have? A Large-Scale Open Do- main Question Answering Dataset from Medi- cal Exams, Applied Sciences 11 (2021) 6421. URL: https://www.mdpi.com/2076-3417/11/14/6421. doi:10.3390/app11...

    2021 doi

  4. [12]

    Q. Jin, B. Dhingra, Z. Liu, W. W. Cohen, X. Lu, Pub- MedQA: A Dataset for Biomedical Research Ques- tion Answering, 2019. URL: http://arxiv.org/abs/ 1909.06146. doi: 10.48550/arXiv.1909.06146, arXiv:1909.06146 [cs]

    2019 doi arXiv

  5. [13]

    Y. Kim, J. Wu, Y. Abdulle, H. Wu, MedExQA: Medical Question Answering Benchmark with Mul- tiple Explanations, in: Proceedings of the 23rd Workshop on Biomedical Natural Language Pro- cessing, Association for Computational Linguis- tics, Bangkok, Thailand, 2024, pp. 167–181. UR...

    2024

  6. [14]

    Nentidis, K

    A. Nentidis, K. Bougatiotis, A. Krithara, G. Paliouras, I. A. Kakadiaris, Overview of the 11th bioasq challenge, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction: 14th International Conference of the CLEF Association, CLEF 2023, Thessaloniki, Greece, S...

    2023

  7. [15]

    Rinaldi, J

    M. Rinaldi, J. Gili, M. Francis, M. Goffetti, V. Patti, M. Nissim, Mult-IT Multiple Choice Questions on Multiple Topics in Italian: A CALAMITA Chal- lenge, in: F. Dell’Orletta, A. Lenci, S. Montemagni, R. Sprugnoli (Eds.), Proceedings of the 10th Italian Conference on Computat...

    2024

  8. [16]

    Casola, T

    S. Casola, T. Labruna, A. Lavelli, B. Magnini, Testing ChatGPT for stability and reasoning: A case study using Italian medical specialty tests, in: F. Boschetti, G. E. Lebani, B. Magnini, N. Novielli (Eds.), Pro- ceedings of the 9th Italian Conference on Computa- tional Lingui...

    2023

  9. [17]

    Altuna, G

    B. Altuna, G. Karunakaran, A. Lavelli, B. Magnini, M. Speranza, R. Zanoli, CLinKaRT at EVALITA 2023: Overview of the task on linking a lab re- sult to its test event in the clinical domain, in: V. Basile, C. Bosco, F. Dell’Orletta, M. Lai, M. San- guinetti, M. Stranisci, M. Te...

    2023

  10. [18]

    Puccetti, M

    G. Puccetti, M. Cassese, A. Esuli, The Invalsi Bench- marks: measuring the Linguistic and Mathemati- cal understanding of Large Language Models in Italian, in: O. Rambow, L. Wanner, M. Apidi- anaki, H. Al-Khalifa, B. D. Eugenio, S. Schockaert (Eds.), Proceedings of the 31st In...

    2025

  11. [19]

    T. Z. Zhao, E. Wallace, S. Feng, D. Klein, S. Singh, Calibrate Before Use: Improving Few-Shot Per- formance of Language Models, 2021. URL: http: //arxiv.org/abs/2102.09690. doi:10.48550/arXiv. 2102.09690, arXiv:2102.09690 [cs]

    2021 doi arXiv

  12. [20]

    Wei, et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,

    J. Wei, et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,

  13. [21]

    Mirzadeh, K

    I. Mirzadeh, K. Alizadeh, H. Shahrokhi, O. Tuzel, S. Bengio, M. Farajtabar, Gsm-symbolic: Under- standing the limitations of mathematical reason- ing in large language models, 2024. URL: https: //arxiv.org/abs/2410.05229.arXiv:2410.05229

    2024 arXiv

  14. [22]

    Yang, et al., Qwen2.5 technical report,

    A. Yang, et al., Qwen2.5 technical report,

  15. [23]

    Team, Gemma: Open Models Based on Gemini Research and Technology, 2024

    G. Team, Gemma: Open Models Based on Gemini Research and Technology, 2024. URL: http://arxiv. org/abs/2403.08295. doi:10.48550/arXiv.2403. 08295, arXiv:2403.08295 [cs]

    2024 doi arXiv

  16. [24]

    Grattafiori, et al., The Llama 3 Herd of Models,

    A. Grattafiori, et al., The Llama 3 Herd of Models,

  17. [25]

    Abdin, et al., Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone, 2024

    M. Abdin, et al., Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone, 2024. URL: http://arxiv.org/abs/ 2404.14219. doi: 10.48550/arXiv.2404.14219, arXiv:2404.14219 [cs]

    2024 doi arXiv

  18. [26]

    DeepSeek-AI, DeepSeek-R1: Incentivizing Reason- ing Capability in LLMs via Reinforcement Learning,

  19. [27]

    Groeneveld, et al., OLMo: Accelerating the sci- ence of language models, in: L.-W

    D. Groeneveld, et al., OLMo: Accelerating the sci- ence of language models, in: L.-W. Ku, A. Martins, V. Srikumar (Eds.), Proceedings of the 62nd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistic...

    2024 doi

  20. [28]

    Üstün, et al., Aya Model: An Instruc- tion Finetuned Open-Access Multilingual Lan- guage Model, 2024

    A. Üstün, et al., Aya Model: An Instruc- tion Finetuned Open-Access Multilingual Lan- guage Model, 2024. URL: http://arxiv.org/abs/ 2402.07827. doi: 10.48550/arXiv.2402.07827, arXiv:2402.07827 [cs]

    2024 doi arXiv

  21. [29]

    Orlando, L

    R. Orlando, L. Moroni, P.-L. Huguet Cabot, S. Co- nia, E. Barba, S. Orlandini, G. Fiameni, R. Nav- igli, Minerva LLMs: The first family of large language models trained from scratch on Italian data, in: F. Dell’Orletta, A. Lenci, S. Montemagni, R. Sprugnoli (Eds.), Proceedings...

    2024

  22. [30]

    Tutti i bambini amano il gelato

    W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, I. Stoica, Efficient mem- ory management for large language model serving with pagedattention, in: Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. A. Addition...

    2023

  23. [31]

    doi:10.48550/arXiv.2501.12948, arXiv:2501.12948 [cs]

    URL: http://arxiv.org/abs/2501.12948. doi:10.48550/arXiv.2501.12948, arXiv:2501.12948 [cs]

    doi arXiv

  24. [36]

    All children love ice cream

    [Opzione 5] (Risposta corretta: [Index for ’Almeno un bam- bino non ama il gelato’ or similar]) Question (English Translation):Which of the following is the negation of the statement "All children love ice cream"? Possible answers:

  25. [37]

    Logic Example Domanda:Se e solo se Giulia a luglio non va in vacanza in montagna, va poi in vacanza al mare ad agosto

    [Option 5] (Correct Answer: [Index for ’At least one child does not love ice cream’ or similar]) A.2. Logic Example Domanda:Se e solo se Giulia a luglio non va in vacanza in montagna, va poi in vacanza al mare ad agosto. Giulia è andata sulle Dolomiti a luglio, dunque non andr...

  26. [38]

    Carolina ha acquistato molte borse, dunque ha speso molti soldi

    Carolina, se acquista molte borse, spende molti soldi. Carolina ha acquistato molte borse, dunque ha speso molti soldi

  27. [39]

    Stasera non ha piovuto, dunque è andata in motorino

    Clotilde non va in motorino la sera tardi, se piove. Stasera non ha piovuto, dunque è andata in motorino

  28. [40]

    Ha già man- giato albicocche a pranzo, dunque a cena non mangia le fragole

    Elisa mangia le fragole a cena se e solo se a pranzo non mangia albicocche. Ha già man- giato albicocche a pranzo, dunque a cena non mangia le fragole

  29. [41]

    Clara ha superato gli esami, dunque ha studi- ato molto

    Solo se Clara studia molto, supera gli esami. Clara ha superato gli esami, dunque ha studi- ato molto

  30. [42]

    Se Riccardo non gioca a calcio, non è in forma per giocare a tennis. Riccardo non gioca a tennis, dunque non ha giocato a calcio (Risposta corretta: 3) Question (English Translation):If and only if Giulia does not go on holiday to the moun- tains in July, she then goes on holi...

  31. [43]

    Carolina bought many bags, therefore she spent a lot of money

    Carolina, if she buys many bags, spends a lot of money. Carolina bought many bags, therefore she spent a lot of money

  32. [44]

    Tonight it did not rain, there- fore she went on her scooter

    Clotilde does not ride her scooter late at night if it rains. Tonight it did not rain, there- fore she went on her scooter

  33. [45]

    She already ate apricots for lunch, therefore she does not eat strawberries for dinner

    Elisa eats strawberries for dinner if and only if she does not eat apricots for lunch. She already ate apricots for lunch, therefore she does not eat strawberries for dinner

  34. [46]

    Clara passed the exams, therefore she studied hard

    Only if Clara studies hard, does she pass the exams. Clara passed the exams, therefore she studied hard

  35. [47]

    Riccardo does not play ten- nis, therefore he did not play football (Correct Answer: 3) A.3

    If Riccardo does not play football, he is not fit to play tennis. Riccardo does not play ten- nis, therefore he did not play football (Correct Answer: 3) A.3. Physics Example Domanda:In quale sistema una tonnellata è un multiplo? Possibili risposte:

  36. [48]

    Nel sistema delle dozzine

  37. [49]

    Nel sistema esadecimale

  38. [50]

    Nel sistema decimale

  39. [51]

    Nessuna delle altre (Risposta corretta: 4) Question (English Translation):In which system is a ton (tonne) a multiple? Possible answers:

  40. [52]

    In the duodecimal system (base 12)

  41. [53]

    In the binary system

  42. [54]

    In the hexadecimal system

  43. [55]

    In the decimal system

  44. [56]

    Chemistry Example Domanda:A quante moli corrispondono 5 mL (d=1,8 g ·cm−3) di un composto avente una massa molare di 450 g·mol−1? Possibili risposte:

    None of the others (Correct Answer: 4) A.4. Chemistry Example Domanda:A quante moli corrispondono 5 mL (d=1,8 g ·cm−3) di un composto avente una massa molare di 450 g·mol−1? Possibili risposte:

  45. [57]

    [Option 1 - e.g., 0.01 mol]

  46. [58]

    [Option 2 - e.g., 0.02 mol]

  47. [59]

    [Option 3 - e.g., 0.04 mol]

  48. [60]

    [Option 4 - e.g., 0.1 mol]

  49. [61]

    [Option 5 - e.g., 0.2 mol] (Risposta corretta: [Index for 0.02 mol]) Question (English Translation):How many moles correspond to 5 mL (d=1.8 g·cm−3) of a compound having a molar mass of 450 g·mol−1? Possible answers:

  50. [62]

    Mathematics Example Domanda:Dati tre segmenti AA’, BB’ e CC’ tali che: AA’ = 2 cm, BB’ = 1,5 * AA’, CC’ = 2,0 * BB’

    [Option 5] (Correct Answer: [Index for 0.02 mol]) A.5. Mathematics Example Domanda:Dati tre segmenti AA’, BB’ e CC’ tali che: AA’ = 2 cm, BB’ = 1,5 * AA’, CC’ = 2,0 * BB’. Quale triangolo è possibile costruire con questi lati? Possibili risposte:

  51. [63]

    Non è possibile costruire nessun triangolo

  52. [64]

    Un triangolo rettangolo

  53. [65]

    Un triangolo ottusangolo

  54. [66]

    Un triangolo scaleno

  55. [67]

    Which triangle is possible to construct with these sides? Possible answers:

    Un triangolo acutangolo (Risposta corretta: 1) Question (English Translation):Given three segments AA’, BB’, and CC’ such that: AA’ = 2 cm, BB’ = 1.5 * AA’, CC’ = 2.0 * BB’. Which triangle is possible to construct with these sides? Possible answers:

  56. [68]

    It is not possible to construct any triangle

  57. [69]

    A right-angled triangle

  58. [70]

    An obtuse-angled triangle

  59. [71]

    Per-Subject Model Performance Table 4 Per-subject accuracy (%) on MedBench-IT for Standard (Std.) and Reasoning (Reas.) prompts

    An acute-angled triangle (Correct Answer: 1) B. Per-Subject Model Performance Table 4 Per-subject accuracy (%) on MedBench-IT for Standard (Std.) and Reasoning (Reas.) prompts. Models sorted as in Table 3. Model Biology Chemistry Gen. Culture Physics Logic Math Std. Reas. Std....

  60. [2021]

    doi:10.48550/arXiv.2009.03300, arXiv:2009.03300 [cs]

    URL: http://arxiv.org/abs/2009.03300. doi:10.48550/arXiv.2009.03300, arXiv:2009.03300 [cs]

    2009 doi arXiv

  61. [2023]

    doi:10.48550/arXiv.2201.11903, arXiv:2201.11903 [cs]

    URL: http://arxiv.org/abs/2201.11903. doi:10.48550/arXiv.2201.11903, arXiv:2201.11903 [cs]

    doi arXiv

  62. [2024]

    doi:10.48550/arXiv.2407.21783, arXiv:2407.21783 [cs]

    URL: http://arxiv.org/abs/2407.21783. doi:10.48550/arXiv.2407.21783, arXiv:2407.21783 [cs]

    doi arXiv

  63. [2025]

    arXiv:2412.15115

    URL: https://arxiv.org/abs/2412.15115. arXiv:2412.15115

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.