Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Automating Expert-Level Medical Reasoning Evaluation of Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A reference-based LLM judge that counts expert reasoning steps in a rationale matches human expert scores with Pearson correlations of 0.68 to 0.87.

desk verdict A useful expert-annotated benchmark with a plausible reference-based judge, but the correlation claim is weakened by the shared rubric and the missing data release. read the letter →

arxiv 2507.07988 v1 pith:7VNS5C3N submitted 2025-07-10 cs.CL

Shuang Zhou , Wenya Xie , Jiaxi Li , Zaifu Zhan , Meijia Song , Han Yang , Cheyenna Espinoza , Lindsay Welton
show 11 more authors
Xinnie Mai Yanwei Jin Zidu Xu Yuen-Hei Chung Yiyun Xing Meng-Han Tsai Emma Schaffer Yucheng Shi Ninghao Liu Zirui Liu Rui Zhang
This is my paper
classification cs.CL
keywords MedThink-BenchmedicalreasoningevaluationLLM-as-a-Judgereference-basedexpert-annotatedrationaleschain-of-thoughtclinicalLLMbenchmarkingrationalescoring
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces MedThink-Bench, a set of 500 difficult medical questions across ten domains, each annotated by physicians with step-by-step reasoning trajectories, and pairs it with an evaluation method called LLM-w-Ref. In LLM-w-Ref, a judge LLM checks whether a model-generated rationale contains each expert-annotated reasoning step, and the score is the fraction of steps found. The authors report that this reference-based judge correlates strongly with expert human evaluations, with Pearson coefficients from 0.68 to 0.87, while text-similarity metrics and reference-free LLM judges correlate weakly. Using this benchmark, they find that a smaller open model, MedGemma-27B, scores highest (0.769) and outperforms larger commercial models such as OpenAI-o3 on reasoning, even though OpenAI-o3 has the highest answer accuracy. The upshot is a scalable way to grade the reasoning process itself rather than only the final answer.

What carries the argument

The central mechanism is the expert-annotated reasoning step as a unit of scoring. MedThink-Bench stores, for each question, a list of discrete steps that physicians consider necessary; the LLM-w-Ref judge receives the question, the model's free-text rationale, and one step at a time, and must decide whether that step is present in the rationale. The instance score is the fraction of expert steps judged present, and the dataset score is the average over questions. This step-presence check is what anchors the automated judge to expert logic, and the paper argues it is the reason LLM-w-Ref correlates with human scores while reference-free judging and surface text metrics do not.

What would settle it

Have two independent expert teams annotate the 500 MedThink-Bench questions with their own step lists, then score the same model rationales with LLM-w-Ref using each team's steps; if the two score sets disagree substantially, or if LLM-w-Ref marks a rationale low that both expert teams judge as valid alternative reasoning, the central claim of expert-level fidelity would fail. The paper's own supplementary inter-annotator agreement check, not included in this text, is the first place to look.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that expert-level evaluation of medical reasoning can be automated without sacrificing fidelity: LLM-w-Ref, which gives a judge LLM the question, the generated rationale, and each discrete expert reasoning step and asks whether the step is supported, reproduces expert judgments closely. The evidence is a Pearson correlation between 0.68 and 0.87 with expert scores across twelve models, compared with weak correlations (as low as -0.17, at most 0.45) for BLEU, ROUGE-L, METEOR, BLEURT, BERTScore, and reference-free LLM judging. The paper further claims the benchmark separates reasoning quality from answer accuracy: models can reach correct answers through flawed reasoning or wrong answers with partially correct reasoning, and LLM-w-Ref detects both. A secondary finding is that on MedThink-Bench, smaller open models, led by MedGemma-27B at 0.769, can surpass larger proprietary models such as OpenAI-o3 and DeepSeek-R1 in reasoning scores.

Load-bearing premise

The method assumes each medical question has a correct set of reasoning steps that experts can reliably list in advance, and that the quality of any rationale is fully captured by how many of those listed steps it contains.

Editorial extensions

If this is right

  • Reasoning quality and final-answer accuracy can diverge, so benchmarks that report only accuracy overstate models that guess correctly and understate models that reason well but answer wrong.
  • A reference-based judge can replace human evaluation for large-scale medical reasoning assessment, cutting evaluation time from hours to minutes.
  • The benchmark can rank models by reasoning: MedGemma-27B, HuatuoGPT-o1-70B, and DeepSeek-R1 lead, with MedGemma-27B ahead of OpenAI-o3.
  • Correlations with expert judgments hold across judge models and prompt phrasings, indicating the method is not tied to one judge.
  • Because expert-annotated rationales are new, pretraining leakage of answers is less likely to distort reasoning scores than accuracy scores.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the step lists favor one canonical reasoning style, LLM-w-Ref may penalize clinicians or models who reach the same conclusion through a different but valid route; a test with deliberately alternative rationales would reveal this.
  • The 500-question scale makes MedThink-Bench a yardstick rather than a training set; extending it via expert-in-the-loop generation could preserve step-level grounding while growing coverage.
  • The correlation evidence would be stronger if the same expert panel that wrote the steps also did the human evaluation; independent panels would separate rubric agreement from true reasoning quality.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces MedThink-Bench, a benchmark of 500 medical questions across ten domains, each annotated by medical experts with fine-grained step-by-step reasoning trajectories. It proposes LLM-w-Ref, an LLM-as-a-Judge evaluation framework that checks whether a model-generated rationale supports each expert-annotated reasoning step, and reports that LLM-w-Ref correlates strongly with expert human evaluation (Pearson 0.68–0.87) while being far more scalable. Benchmarking twelve LLMs, the authors find that smaller open models such as MedGemma-27B can surpass larger proprietary models like OpenAI-o3. The paper also compares LLM-w-Ref with text-similarity metrics and reference-free LLM-as-a-Judge, and reports robustness analyses across judge models and prompt variants.

Significance. If the central claims hold, the paper addresses a real and practical gap: scalable, expert-aligned evaluation of LLM reasoning in medicine. The proposed step-level reference-based judging is a sensible design and, importantly, the paper includes a human evaluation component that is often missing from LLM-as-a-Judge work. The benchmark, once released, could be a useful resource. However, the paper's main evidence for the validity of LLM-w-Ref—the high correlation with expert judgments—is weakened by a shared-rubric circularity: both the human evaluators and LLM-w-Ref score the same expert-annotated step list, so the correlation partly reflects agreement on a fixed checklist rather than independent validation of reasoning quality. The robustness claims are also not fully supported by the reported judge-model experiments. The dataset and supplementary materials are not available for verification, which is a significant limitation given that the benchmark and evaluation framework are the main contributions.

major comments (3)
  1. [Methods, Human evaluation and Reference-based LLM-as-a-Judge] The two evaluation procedures use the same expert-annotated step set S_expert: human evaluators count how many of those steps appear in the rationale, and LLM-w-Ref asks a judge LLM whether each step in the same set is supported. The reported Pearson correlations of 0.68–0.87 may therefore largely reflect agreement between GPT-4o-mini and human experts about which checklist items are present, rather than validating the step decomposition itself. This is a load-bearing concern: if the expert steps are incomplete, biased, or not reproducible, both measures are high-quality measurements of an arbitrary rubric. The paper states that inter-annotator agreement (IAA) was checked (Methods, 'Data annotation') but refers only to Supplementary Data 2, which is not included. To support the central claim, the authors should (a) report the actual IAA values for the step annotations, not just for question and reasoning types; (b) provide evidence that the step decomposition is stable across experts; and (c) if possible, compare LLM-w-Ref against a human evaluation that does not use the same fixed step checklist, or against human holistic judgments of reasoning quality.
  2. [Results, The impact of the judge model in LLM-w-Ref, Fig. 5a] The robustness claim is undermined by the reported results: when different judge models are used, the LLM-w-Ref scores for the same set of Llama-3.3-70B rationales range from 0.522 to 0.891, a spread that is comparable to or larger than the differences among the twelve benchmarked models (expert scores range from 0.453 to 0.759). The sentence stating that performance 'remains stable when using LLMs with strong instruction-following capabilities' is not supported by the data, since even the large judge models (GPT-4o-mini, DeepSeek-R1, Gemini-2.5-flash, HuatuoGPT-o1-70B) show a nontrivial spread. The authors should either report the statistical significance of these differences, provide a more careful definition of 'strong instruction-following capabilities', or temper the robustness conclusion. It is also important to state explicitly which judge model was used for the headline benchmark results in Fig. 3 and whether the reported correlations in Fig. 4a use that same judge model.
  3. [Data Availability] The paper's central claims depend on the expert-annotated step lists, the human evaluation ratings, the IAA values, and the per-sample scores, but none of these are available in the preprint: the expert-curated data 'will be released upon the acceptance of this paper,' and the supplementary data (Supplementary Data 2, 4, 5) are not included. This makes the correlation analysis, the robustness analysis, and even the headline benchmark results impossible to verify independently. The authors should release the evaluation data, the human ratings, and the IAA results at least as part of the review process, and preferably in a public repository, so that reviewers and readers can reproduce the correlation coefficients and the reported model rankings.
minor comments (5)
  1. [Introduction] The text mentions 'BLUE' where the intended metric is 'BLEU' (see reference 29); please fix the typo.
  2. [Data Availability, item 9] The URL label 'MMLU-Medicne' is misspelled; it should read 'MMLU-Medicine'.
  3. [Results, Efficiency comparison] The efficiency analysis reports a single average time for 'text-similarity metrics' and for 'LLM-w-Ref,' but it is unclear whether these times include model API latency, prompt construction, and response parsing; please clarify the measurement protocol.
  4. [Methods, Prediction accuracy] The exact-match accuracy definition uses equality between predicted and golden answers, but many of the source datasets (e.g., MedQA, PubMedQA) have different answer formats; please specify how the rule-based extraction normalizes the answers before matching.
  5. [Discussion] The claim that LLM-w-Ref is 'well-suited for scalable, automated evaluation' is supported by the efficiency comparison, but the absolute time of 310.7 minutes for 500 questions (about 37 seconds per question) should be contextualized; the practical overhead of API costs and rate limits is not discussed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported human–LLM correlation is an empirical agreement on a shared expert rubric, not a result forced by construction.

full rationale

The paper's central claim is empirical: LLM-w-Ref correlates with expert evaluation. The human score R(i)=ExpertCorrect(r_model,q)/ExpertRequired(q) and the LLM-w-Ref score R(i)=|{s in S_expert : LLMJudge(s,r_model,q)=Yes}|/|S_expert| both reference the same expert-annotated step set S_expert, so the Pearson 0.68–0.87 measures how well an LLM checker reproduces human checklist marking of the same rubric. This is a construct-validity limitation (the rubric itself is not independently validated) rather than circular reasoning: the two equations are not identical, the LLM judge could in principle disagree with human experts, the correlation is an observed outcome rather than a fitted parameter or definitional identity, and no load-bearing claim is justified solely by self-citation. The benchmark relies on externally sourced questions and newly expert-annotated rationales, giving the study independent content. No circular step meets the evidentiary bar of Eq. X = Eq. Y by construction or a fitted input renamed as a prediction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted to data; the reasoning score is a hand-defined proportion. The main postulates are domain assumptions about the completeness and validity of expert-annotated reasoning steps, the reliability of the LLM judge, the representativeness of the selected questions, and the absence of harmful contamination from the public source datasets.

assumptions (4)
  • domain assumption Expert-annotated reasoning steps constitute a complete and correct ground truth for reasoning quality.
    Both human evaluation and LLM-w-Ref score the presence of these steps; if the steps are incomplete or biased, all scores downstream are biased. Invoked in Methods: 'Human evaluation' and 'Reference-based LLM-as-a-Judge'.
  • domain assumption The LLM judge can reliably determine whether a model-generated rationale supports a given expert step.
    The entire automated score depends on GPT-4o-mini's ability to make step-support judgments; the paper tests different judge models but does not compare judge failures against expert failures at the step level.
  • ad hoc to paper The manually curated subset of 500 questions is representative of complex medical reasoning.
    Questions were selected by ten medical experts from ten public datasets based on 'multi-step reasoning' criteria described only in Supplementary Data 2; this selection may introduce a domain or difficulty bias.
  • domain assumption The judge model does not have prior exposure to these exact questions, or if it does, the effect is negligible.
    The raw questions come from public datasets that may have been in LLM training data; the authors acknowledge this limitation in the Discussion but still use the benchmark to rank models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automating Expert-Level Medical Reasoning Evaluation of Large Language Models." pith.science (2026). https://pith.science/paper/7VNS5C3N

@misc{pith2026250707988,
  author       = {Pith},
  title        = {Pith review of: Automating Expert-Level Medical Reasoning Evaluation of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7VNS5C3N}},
  note         = {Machine review of arXiv:2507.07988}
}
read the original abstract

As large language models (LLMs) become increasingly integrated into clinical decision-making, ensuring transparent and trustworthy reasoning is essential. However, existing evaluation strategies of LLMs' medical reasoning capability either suffer from unsatisfactory assessment or poor scalability, and a rigorous benchmark remains lacking. To address this, we introduce MedThink-Bench, a benchmark designed for rigorous, explainable, and scalable assessment of LLMs' medical reasoning. MedThink-Bench comprises 500 challenging questions across ten medical domains, each annotated with expert-crafted step-by-step rationales. Building on this, we propose LLM-w-Ref, a novel evaluation framework that leverages fine-grained rationales and LLM-as-a-Judge mechanisms to assess intermediate reasoning with expert-level fidelity while maintaining scalability. Experiments show that LLM-w-Ref exhibits a strong positive correlation with expert judgments. Benchmarking twelve state-of-the-art LLMs, we find that smaller models (e.g., MedGemma-27B) can surpass larger proprietary counterparts (e.g., OpenAI-o3). Overall, MedThink-Bench offers a foundational tool for evaluating LLMs' medical reasoning, advancing their safe and responsible deployment in clinical practice.

Figures

Figures reproduced from arXiv: 2507.07988 by the authors.

Figure 1
Figure 1. Overview of the MedThink-Bench dataset. a. Data collection. Medical questions were sourced from ten publicly available datasets, each accompanied by ground-truth answers. b. Data preprocessing. Duplicate entries and questions involving medical images were removed. Medical experts then manually curated a subset of complex questions requiring multi-step reasoning. c. Expert annotation. A team of ten medical experts an… view at source ↗
Figure 2
Figure 2. Dataset statistics of MedThink-Bench. a. Breakdown of the ten medical domains included in the MedThink-Bench dataset. b. Detailed statistics of the dataset. Comparison of evaluation metrics on LLM reasoning We evaluated the reasoning performance of twelve LLMs on the MedThink-Bench dataset, focusing on medical reasoning abilities and comparing mainstream evaluation metrics. We assessed the reasoning performance of t… view at source ↗
Figure 5
Figure 5. Performance and efficiency analysis of LLM [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figures from the paper (1 more)
Figure 6
Figure 6. Figure 6: Case study of medical reasoning assessment. This case demonstrates that while the [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Reference-Free Evaluation of Reasoning in Open-Ended Question Answering

    cs.CL 2026-07 conditional novelty 6.0 of 10

    An NLI-hypergraph audit with deterministic AND–OR search labels LLM reasoning segments as supported, unsupported, or orphaned, improving balanced F1 over LLM-as-judge on a new 40-case clinical benchmark.

  2. Quantized Large Language Models in Biomedical Natural Language Processing: Evaluation and Recommendation

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Quantizing LLMs to 4 or 8 bits cuts GPU memory by up to 75% with generally small performance changes across eight biomedical NLP benchmarks.

Reference graph

Works this paper leans on

67 extracted references · 57 canonical work pages · cited by 2 Pith papers

  1. [1]

    Zhou, S. et al. Large language models for disease diagnosis: a scoping review. NPJ Artif Intell 1, 9 (2025)

    work page 2025

  2. [2]

    Chen, X. et al. Enhancing diagnostic capability with multi-agents conversational large language models. NPJ Digit. Med. 8, 159 (2025)

    work page 2025

  3. [3]

    Truhn, D. et al. A pilot study on the efficacy of GPT-4 in providing orthopedic treatment recommendations from MRI reports. Sci. Rep. 13, 20159 (2023)

    work page 2023

  4. [4]

    Goh, E. et al. GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial. Nat. Med. 31, 1233–1238 (2025)

    work page 2025

  5. [5]

    Abbasian, M. et al. Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI. NPJ Digit. Med. 7, 82 (2024)

    work page 2024

  6. [6]

    Li, J. et al. Fact or guesswork? Evaluating large language model’s medical knowledge with structured one-hop judgment. arXiv [cs.CL] (2025)

    work page 2025

  7. [7]

    Zhou, S. et al. Uncertainty-aware large language models for explainable disease diagnosis. arXiv [cs.CL] (2025)

    work page 2025

  8. [8]

    Hager, P. et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat. Med. 30, 2613–2622 (2024)

    work page 2024

Show all 67 references
  1. [9]

    McDuff, D. et al. Towards accurate differential diagnosis with large language models. Nature 642, 451–457 (2025)

    2025

  2. [10]

    Zhang, Y. et al. Siren’s song in the AI ocean: A survey on hallucination in large language models. arXiv [cs.CL] (2023)

    2023

  3. [11]

    Liu, W. et al. Mitigating hallucination through theory-consistent Symmetric Multimodal Preference Optimization. arXiv [cs.AI] (2025)

    2025

  4. [12]

    & Vechev, M

    Mündler, N., He, J., Jenko, S. & Vechev, M. T. Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation. Int Conf Learn Represent abs/2305.15852, (2023)

    2023 arXiv

  5. [13]

    & Ranisch, R

    Haltaufderheide, J. & Ranisch, R. The ethics of ChatGPT in medicine and healthcare: a systematic review on Large Language Models (LLMs). NPJ Digit. Med. 7, 183 (2024)

    2024

  6. [14]

    & Chen, J

    Savage, T., Nayak, A., Gallo, R., Rangan, E. & Chen, J. H. Diagnostic reasoning prompts reveal the potential for large language model interpretability in medicine. NPJ Digit. Med. 7, 20 (2024)

    2024

  7. [15]

    Kim, H. et al. Small language models learn enhanced reasoning skills from medical textbooks. NPJ Digit. Med. 8, 240 (2025)

    2025

  8. [16]

    Zhu, Y. et al. DiagnosisArena: Benchmarking diagnostic reasoning for large language models. arXiv [cs.CL] (2025)

    2025

  9. [17]

    Tang, X. et al. MedAgentsBench: Benchmarking thinking models and agent frameworks for complex medical reasoning. arXiv [cs.CL] (2025)

    2025

  10. [18]

    Zuo, Y. et al. MedXpertQA: Benchmarking expert-level medical reasoning and understanding. arXiv [cs.AI] (2025)

    2025

  11. [19]

    Yang, H. et al. One LLM is not enough: Harnessing the power of ensemble learning for medical question answering (preprint). JMIR Preprints (2024) doi:10.2196/preprints.70080

    2024 doi

  12. [20]

    Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172–180 (2023)

    2023

  13. [21]

    Jin, Q. et al. Hidden flaws behind expert-level accuracy of multimodal GPT-4 Vision in Medicine. ArXiv (2024)

    2024

  14. [22]

    E., Motzfeldt, A

    Liévin, V., Hother, C. E., Motzfeldt, A. G. & Winther, O. Can large language models reason about medical questions? Patterns (N. Y.) 5, 100943 (2024)

    2024

  15. [23]

    Zhou, S. et al. Explainable differential diagnosis with dual-inference large language models. Npj Health Syst. 2, 12 (2025)

    2025

  16. [24]

    Kim, Y., Wu, J., Abdulle, Y. & Wu, H. MedExQA: Medical question answering benchmark with multiple explanations. arXiv [cs.CL] (2024)

    2024

  17. [25]

    Brodeur, P. G. et al. Superhuman performance of a large language model on the reasoning tasks of a physician. arXiv [cs.AI] (2024)

    2024

  18. [26]

    Singhal, K. et al. Toward expert-level medical question answering with large language models. Nat. Med. 31, 943–950 (2025)

    2025

  19. [27]

    & Zhang, M

    Li, D., Yu, J., Hu, B., Xu, Z. & Zhang, M. ExplainCPE: A free-text explanation benchmark of Chinese pharmacist examination. arXiv [cs.CL] (2023)

    2023

  20. [28]

    Qiu, P. et al. Quantifying the reasoning abilities of LLMs on real-world clinical cases. arXiv [cs.CL] (2025)

    2025

  21. [29]

    & Zhu, W.-J

    Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. BLEU. in Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL ’02 (Association for Computational Linguistics, Morristown, NJ, USA, 2001). doi:10.3115/1073083.1073135

    2001

  22. [30]

    ROUGE: A package for automatic evaluation of summaries

    Lin, C.-Y. ROUGE: A package for automatic evaluation of summaries. Annu Meet Assoc Comput Linguistics 74–81 (2004)

    2004

  23. [31]

    Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q. & Artzi, Y. BERTScore: Evaluating Text Generation with BERT. arXiv [cs.CL] (2019)

    2019

  24. [32]

    Liu, L. et al. Towards automatic evaluation for LLMs’ clinical capabilities: Metric, data, and algorithm. in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining vol. 614 5466–5475 (ACM, New York, NY, USA, 2024)

    2024

  25. [33]

    Croxford, E. et al. Current and future state of evaluation of large language models for medical summarization tasks. Npj Health Syst. 2, (2025)

    2025

  26. [34]

    Wu, K. et al. MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reports. arXiv [cs.CL] (2025)

    2025

  27. [35]

    Ding, C. et al. Building a human-verified clinical reasoning dataset via a human LLM hybrid pipeline for trustworthy medical AI. arXiv [cs.CV] (2025)

    2025

  28. [36]

    Chen, X. et al. FFA-GPT: an automated pipeline for fundus fluorescein angiography interpretation and question-answer. NPJ Digit. Med. 7, 111 (2024)

    2024

  29. [37]

    & Yuksel, D

    Griot, M., Hemptinne, C., Vanderdonckt, J. & Yuksel, D. Large Language Models lack essential metacognition for reliable medical reasoning. Nat. Commun. 16, 642 (2025)

    2025

  30. [38]

    & Dredze, M

    Chen, H., Fang, Z., Singla, Y. & Dredze, M. Benchmarking large language models on answering and explaining challenging medical questions. arXiv [cs.CL] (2024)

    2024

  31. [39]

    Zhang, H. et al. HuatuoGPT, towards taming language model to be a doctor. arXiv [cs.CL] (2023)

    2023

  32. [40]

    Wei, J. et al. Chain of thought prompting elicits reasoning in large language models. Neural Inf Process Syst abs/2201.11903, 24824–24837 (2022)

    2022 arXiv

  33. [41]

    Kim, M. et al. Fine-tuning LLMs with medical data: Can safety be ensured? NEJM AI 2, (2025)

    2025

  34. [42]

    F., Goel, R., Wen, Z., Martel, J

    Tchango, A. F., Goel, R., Wen, Z., Martel, J. & Ghosn, J. DDXPlus: A new dataset for automatic medical diagnosis. arXiv [cs.CL] (2022)

    2022

  35. [43]

    & Dredze, M

    Chen, H., Fang, Z., Singla, Y. & Dredze, M. Benchmarking large language models on answering and explaining challenging medical questions. in Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Langua...

    2025

  36. [44]

    Wang, Y. et al. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. Neural Inf Process Syst abs/2406.01574, 95266–95290 (2024)

    2024 arXiv

  37. [45]

    Kim, Y., Wu, J., Abdulle, Y. & Wu, H. MedExQA: Medical question answering benchmark with multiple explanations. in Proceedings of the 23rd Workshop on Biomedical Natural Language Processing (eds. Demner-Fushman, D., Ananiadou, S., Miwa, M., Roberts, K. & Tsujii, J.) 167–181 (A...

    2024

  38. [46]

    Phan, L. et al. Humanity’s Last Exam. arXiv [cs.LG] (2025)

    2025

  39. [47]

    Jin, D. et al. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Appl. Sci. (Basel) abs/2009.13081, (2020)

    2020 arXiv

  40. [48]

    Jin, Q., Dhingra, B., Liu, Z., Cohen, W. & Lu, X. PubMedQA: A Dataset for Biomedical Research Question Answering. in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing...

    2019

  41. [49]

    Pal, A., Umapathi, L. K. & Sankarasubbu, M. MedMCQA : A large-scale multi-subject multi-choice dataset for medical domain question answering. CHIL 174, 248–260 (2022)

    2022

  42. [50]

    Hendrycks, D. et al. Measuring massive multitask language understanding. Int Conf Learn Represent abs/2009.03300, (2020)

    2020 arXiv

  43. [51]

    & Gómez-Rodríguez, C

    Vilares, D. & Gómez-Rodríguez, C. HEAD-QA: A Healthcare Dataset for Complex Reasoning. in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (eds. Korhonen, A., Traum, D. & Màrquez, L.) 960–966 (Association for Computational Linguistics, St...

    2019

  44. [52]

    & Lavie, A

    Banerjee, S. & Lavie, A. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. 65–72 (2005)

    2005

  45. [53]

    & Parikh, A

    Sellam, T., Das, D. & Parikh, A. P. BLEURT: Learning robust metrics for text generation. arXiv [cs.CL] (2020)

    2020

  46. [54]

    Wei, H. et al. Systematic evaluation of LLM-as-a-judge in LLM alignment tasks: Explainable metrics and diverse prompt templates. arXiv [cs.CL] (2024)

    2024

  47. [55]

    GPT-4 Technical Report

    OpenAI et al. GPT-4 Technical Report. arXiv [cs.CL] (2023)

    2023

  48. [56]

    GPT-4o System Card

    OpenAI et al. GPT-4o System Card. arXiv [cs.CL] (2024)

    2024

  49. [57]

    OpenAI o3 and o4-mini System Card

  50. [58]

    claude-3.5-sonnet

    Anthropic. claude-3.5-sonnet

  51. [59]

    gemini-2.5 technical report

  52. [60]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    DeepSeek-AI et al. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv [cs.CL] (2025)

    2025

  53. [61]

    Wang, B. et al. Baichuan-M1: Pushing the medical capability of large language models. arXiv [cs.CL] (2025)

    2025

  54. [62]

    Chen, J. et al. HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs. arXiv [cs.CL] (2024)

    2024

  55. [63]

    Andrew, S. et al. MedGemma Technical Report. arXiv [cs.AI] (2025)

    2025

  56. [64]

    K., Raha, T., Khan, S

    Christophe, C., Kanithi, P. K., Raha, T., Khan, S. & Pimentel, M. A. F. Med42-v2: A Suite of Clinical LLMs. arXiv [cs.CL] (2024)

    2024

  57. [65]

    ’ement et al

    Christophe, C. ’ement et al. Med42 - evaluating fine-tuning strategies for medical LLMs: Full- parameter vs. Parameter-efficient approaches. ArXiv abs/2404.14779, (2024)

    2024 arXiv

  58. [66]

    Grattafiori, A. et al. The Llama 3 herd of models. arXiv [cs.AI] (2024)

    2024

  59. [67]

    Yang, A. et al. Qwen3 Technical Report. arXiv [cs.CL] (2025)

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.