Pith. sign in

REVIEW 3 major objections 6 minor 40 references

Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Current LLMs can recall medical facts, but their ability to plan and reason through a real diagnostic workup drops sharply, with the best model reaching only 19.4% full-path diagnosis accuracy.

desk verdict Useful three-level medical benchmark with a solid L1→L2 drop; the L3 headline number is hostage to a questionable gold standard. read the letter →

arxiv 2506.08349 v1 pith:3BS3J6M6 submitted 2025-06-10 cs.CL cs.AI

Yuxuan Zhou , Xien Liu , Chenwei Yan , Chen Ning , Xiao Zhang , Boxun Li , Xiangling Fu , Shijin Wang
show 3 more authors
Guoping Hu Yu Wang Ji Wu
This is my paper
classification cs.CLcs.AI
keywords largelanguagemodelsmedicalevaluationcognitivelevelsBloom'staxonomyclinicaldiagnosisbenchmarkMIMIC-IVscenario-basedproblemsolving
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that evaluating LLMs on medical multiple-choice exams alone gives a misleading picture of their clinical ability. It proposes a three-level evaluation framework, inspired by Bloom's Taxonomy: preliminary knowledge grasp (MCQ), comprehensive knowledge application (reformulated tasks such as statement validation and multi-step rectification), and scenario-based problem solving (a full-path diagnosis task in which the model sequentially orders tests and then gives a final diagnosis). Across 32 general and 8 medical models, the paper finds a consistent and large performance decline as the cognitive level rises, with the best model, DeepSeek-V3, reaching only 19.4% full-path diagnosis accuracy. If correct, this means current LLMs have not merely a small gap but a deep one between stored medical facts and the planning and reasoning needed in real clinical settings.

What carries the argument

The central object is MultiCogEval, a three-level evaluation framework that keeps medical knowledge coverage roughly constant across levels while varying cognitive demand, and normalizes accuracy by random-guessing baselines so the levels are comparable. At L3 it uses a full-path diagnosis metric that multiplies final-diagnosis correctness by examination recall, where the ground-truth examinations are the ones actually recorded in each MIMIC-IV admission note. This machinery is what turns the abstract claim 'higher cognition is harder for LLMs' into a measurable performance gap.

What would settle it

Have a panel of experienced clinicians independently order tests for the same patient histories used in the high-level task and compare their orders with the recorded MIMIC-IV orders; if clinicians frequently omit or add different tests, or if model accuracy scored against clinician-approved orders rises far above 19.4%, the paper's central conclusion about scenario-based problem solving would be called into question.

Watch

Extended reading notes

Core claim

The paper claims that current LLMs are strong at recalling medical knowledge but weak at applying it in realistic clinical workflows. Using its MultiCogEval framework, it shows that state-of-the-art models score above 60% normalized accuracy on L1 MCQ tasks, drop roughly 20 points on L2 reformulated application tasks, and fall to about 19.4% on L3 full-path diagnosis, where the model must choose which examinations to order, interpret their results, and produce a final diagnosis. It further reports that larger model size matters more at higher cognitive levels, that medical-domain fine-tuning improves L1 and L2 but not L3, and that inference-time scaling helps across levels, especially on L2. The central discovery is a quantitative, level-by-level map of where LLM medical competence breaks down.

Load-bearing premise

The load-bearing assumption is that the examinations actually recorded in each historical MIMIC-IV admission are the correct set a model should order, so the exam-recall part of the full-path metric treats deviations from those records as errors; if real notes contain unnecessary, redundant, or missing tests, then the 19.4% figure is not a clean measure of clinical problem-solving.

Editorial extensions

If this is right

  • Medical exam-style benchmarks overstate readiness: a model can ace knowledge questions while lacking the planning and information-seeking behavior needed for diagnosis.
  • Model selection should be task-appropriate: roughly 10B-parameter models may suffice for knowledge QA, while complex clinical tasks require larger models.
  • Medical-domain fine-tuning should shift its objective from memorized knowledge to high-level abilities such as clinical planning and proactive test ordering.
  • Inference-time scaling is a promising lever for improving application-level medical reasoning, though it does not by itself close the high-level gap.
  • Benchmark design for medical AI should include interactive scenario tasks with sequential decisions, not only static QA.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the reported 19.4% depends on treating the tests recorded in historical MIMIC-IV notes as the correct test set; if those notes reflect over-testing or missing tests, the exam-recall term may penalize clinically reasonable strategies and the true problem-solving gap could differ.
  • Editorial extension: the same three-level design could be applied to other professional domains, such as legal reasoning or engineering troubleshooting, where factual recall and on-the-job planning are similarly separable.
  • Editorial extension: a direct test of the framework's metric would compare clinician-ordered test sets against the recorded orders; if clinicians themselves disagree substantially with the recorded orders, the high-level metric should be re-anchored.
  • Editorial extension: the framework measures whether models request the same tests as physicians, not whether they request better, cheaper, or safer tests; a cost- or safety-weighted variant could change the ranking of models at L3.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes MultiCogEval, a three-level evaluation framework for medical LLMs inspired by Bloom's Taxonomy. Level 1 (Preliminary Knowledge Grasp) uses multiple-choice questions from MedQA and MedMCQA; Level 2 (Comprehensive Knowledge Application) reformulates those MCQs into statement validation, multi-step rectification, and answer-existence judgment tasks; Level 3 (Scenario-based Problem Solving) uses a full-path clinical diagnosis task built from MIMIC-IV admission records. The authors evaluate 32 general and 8 medical LLMs from six families and report that normalized accuracy declines monotonically from Level 1 to Level 3, with DeepSeek-V3 achieving 19.4% full-path diagnosis accuracy at the highest level. They also analyze the effects of model size, medical-domain fine-tuning, and inference-time scaling.

Significance. If the measurements are valid, the paper makes a useful contribution. It is the first to compare LLMs across multiple cognitive levels in the medical domain, it releases code and data, and the monotonic decline is observed across all six LLM families, which gives the result a systematic robustness. The knowledge-consistency design between levels is thoughtful, and the fine-grained analyses (per-disease long tail, medical fine-tuning effects, inference-time scaling) are valuable. However, the validity of the High-Level metric is the linchpin of the headline claim, and the current evidence for that metric is weaker than the strength of the claim requires.

major comments (3)
  1. [Section 3.3, Eq. (3)] The examination recall in Eq. (3) uses the full set of exam items recorded in each MIMIC-IV admission as the denominator, i.e., it treats every recorded test as mandatory for an ideal diagnostic workup. MIMIC-IV records reflect real clinical practice, which includes routine, redundant, or defensive tests, and may omit tests that a competent clinician would order. A model that reaches the correct diagnosis through a more selective or different workup is therefore penalized. This is not a minor scaling issue: Table 5 reports an end-point diagnosis accuracy of 53.6% for DeepSeek-V3, which is well above the Mid-Level score reported in Table 3, so the Level-2-to-Level-3 decline in the headline result is entirely produced by the multiplicative recall term. The authors should recompute the High-Level metric with a necessity-filtered denominator (e.g., a clinician-validated subset of essential tests per case) or provide a sensitivity analysis that shows how the results change when the denominator is restricted or reweighted.
  2. [Section 4.4 / Section F] The clinician validation does not validate the full-path High-Level task as actually administered to the models. According to Section F, clinicians were given only the history of present illness and asked to provide a diagnosis, with the list of allowed test types described but not interactively executed; the model evaluation, by contrast, requires sequential exam ordering, result integration, and a final diagnosis after multiple turns. Moreover, the validation uses only 100 samples (20 per task). This small and simplified validation cannot independently establish that the exam-recall gold standard in Eq. (3) is objective, nor does it speak to the procedural-correctness component that drives the Level-3 scores. The paper should either run the interactive task with clinicians or explicitly state this as a limitation and avoid claims that the validation demonstrates the validity of the proposed benchmark.
  3. [Table 2 footnote / Section D] The evaluation of GPT-4o, GPT-4o-mini, DeepSeek-V3, and DeepSeek-R1 was performed on approximately 10% of the original dataset, but the paper does not specify how this subset was chosen: whether it is a random sample, whether it is stratified by disease, or whether the same subset is used for all four models. Since the headline 19.4% figure for DeepSeek-V3 is computed on this subset, the authors should document the subset construction, state the exact number of samples, and verify that the reported cross-model comparisons are not skewed by subset composition.
minor comments (6)
  1. [Section 4.2 / Figure 5] The text states that model size plays a more critical role at higher cognitive levels, but Figure 5 and the accompanying discussion indicate that the performance gap between 7B and 70B models narrows again at the High-Level. The wording should be adjusted to be consistent with the data, e.g., 'the gap widens from Low to Mid but narrows again at High.'
  2. [Section B / Eq. (4)] For the High-Level task, the random accuracy is set to zero. Since the diagnosis is open-ended but the task is constructed from a limited disease pool of 42 diseases, the assumption of zero random accuracy should be justified or relaxed in a sensitivity analysis.
  3. [Section A.2] The manual verification of GPT-4o-generated statements and distractors is reported as an error rate of less than 5%, but the text does not state whether the verified batch of 100 was used to filter the final dataset or only as a quality check. Please clarify whether any generated items were removed or corrected after verification.
  4. [Section A.3] The prompt says 'You will get all the lab tests results at once' but the later interaction requires the model to list specific tests before receiving results. Please clarify the precise protocol so that readers understand whether the model receives results only for requested items or for an entire category.
  5. [Section 3.1] There is a typo in the heading: 'Priciples' should be 'Principles.'
  6. [Section 3.3 / Eq. (2)] The main text says the full-path accuracy is computed as a macro-average across examination types and diseases, but the formula in Eq. (2) does not show this averaging. Please make the metric definition self-contained in the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the cognitive-level ordering is an empirical result with external clinician validation, and the self-citations are not load-bearing.

full rationale

The paper's central claim is an empirical evaluation result, not a derivation that reduces to its own inputs. The three cognitive levels are defined by an external taxonomy (Bloom's Taxonomy) and by the human clinician training process, not by model performance. Mid-Level tasks are reformulations of the same Low-Level MCQs, and High-Level tasks are built from MIMIC-IV admissions filtered to diseases appearing in the MCQ-derived disease pool. This design deliberately controls knowledge coverage, but it does not logically force the observed accuracy ordering; statement validation or multi-step rectification could in principle be easier for some models than the original MCQ format, and the reported decline is an empirical finding rather than a tautology. The full-path accuracy metric in Equations (2)-(3) uses the recorded MIMIC-IV examination items as an external gold standard; it is not fitted to model outputs, so any concern that the denominator over-counts unnecessary tests is a validity or correctness issue, not circularity. The clinician validation (Section 4.4) provides independent evidence that the task ordering corresponds to genuine difficulty, with clinician accuracy falling from 68.8% to 23.5% and subjective difficulty rising, which further anchors the levels outside the model evaluations. Normalization in Equation (4) is transparent and adjusts for random-guess baselines without encoding the conclusion. The self-citations, such as Yan et al. (2025) and Qiu et al. (2024), appear in related-work and motivation contexts and are not load-bearing for the central evaluation chain. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via self-citation. The manuscript is self-contained against external benchmarks and human validation for its main claims.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The framework rests on several domain assumptions: that the three task formats correspond to ordered cognitive levels aligned with Bloom's Taxonomy and clinician training, that disease-pool matching aligns knowledge coverage across levels, that GPT-4o-generated labels are correct (verified on 100 samples), and that the examinations recorded in MIMIC-IV are the correct gold standard for the high-level metric. Free parameters include the weighting alpha, the fuzzy-matching threshold, and dataset filtering cutoffs.

free parameters (4)
  • alpha (weight for correct-answer questions in Multi-Step Rectification) = 1/N_c
    Chosen so that random accuracy is invariant to the guessing strategy; directly affects normalized Mid-Level accuracy.
  • Fuzzy matching similarity threshold for exam-item parsing = 0.9
    Hand-chosen threshold in the UMLS-based parser; affects High-Level exam recall and thus full-path accuracy.
  • Disease inclusion cutoff and per-disease sampling cap = >10 records; cap at 100 records per disease
    Dataset filtering choices that determine the composition and balance of the High-Level benchmark.
  • High-Level random accuracy in normalization = 0
    Set because the task is open-ended; this choice affects cross-level comparability of normalized scores.
assumptions (5)
  • domain assumption The three task formats (MCQ, reformulated application tasks, full-path diagnosis) correspond to ordered cognitive levels aligned with Bloom's Taxonomy and the clinician training trajectory.
    Section 3.2 defines the levels by analogy to medical school, internship, and residency; the validity of this mapping is assumed.
  • domain assumption The disease-pool matching procedure (MedCAT, ICD-10-CM filtering) ensures knowledge coverage is consistent across cognitive levels.
    Section 3.3, Figure 4: this alignment is assumed to isolate cognitive level from knowledge content.
  • domain assumption The set of examinations recorded in each MIMIC-IV admission record is the gold standard for correct examination ordering.
    Equation (3) computes examination recall against the recorded exam items; no clinical validation of this gold standard is provided.
  • domain assumption GPT-4o-generated statements and distractors for Mid-Level tasks are correct, based on manual verification of 100 samples with claimed error <5%.
    Section 3.3: this verification sample is small relative to the 21,666 generated Mid-Level questions.
  • standard math The normalized accuracy formula (Equation 4) with theoretical random baselines makes performance comparable across levels.
    Section B derives the random baselines; the choice of random accuracy for the open-ended High-Level task (0) is an assumption about the task format.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving." pith.science (2026). https://pith.science/paper/3BS3J6M6

@misc{pith2026250608349,
  author       = {Pith},
  title        = {Pith review of: Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3BS3J6M6}},
  note         = {Machine review of arXiv:2506.08349}
}
read the original abstract

Large language models (LLMs) have demonstrated remarkable performance on various medical benchmarks, but their capabilities across different cognitive levels remain underexplored. Inspired by Bloom's Taxonomy, we propose a multi-cognitive-level evaluation framework for assessing LLMs in the medical domain in this study. The framework integrates existing medical datasets and introduces tasks targeting three cognitive levels: preliminary knowledge grasp, comprehensive knowledge application, and scenario-based problem solving. Using this framework, we systematically evaluate state-of-the-art general and medical LLMs from six prominent families: Llama, Qwen, Gemma, Phi, GPT, and DeepSeek. Our findings reveal a significant performance decline as cognitive complexity increases across evaluated models, with model size playing a more critical role in performance at higher cognitive levels. Our study highlights the need to enhance LLMs' medical capabilities at higher cognitive levels and provides insights for developing LLMs suited to real-world medical applications.

Figures

Figures reproduced from arXiv: 2506.08349 by the authors.

Figure 1
Figure 1. (a): The cognitive development process of human doctors; (b): Comparison of the existing evaluations and our proposed multi-cognitive-level evaluation framework (inspired by Bloom’s Taxonomy) regarding cognitive levels. scenario-based problem solving. In this paper, we argue that the evaluation of LLMs should also follow the cognitive development process, i.e., evalu￾ating LLMs’ medical capabilities across multiple … view at source ↗
Figure 2
Figure 2. An overview of the proposed multi-cognitive-level medical evaluation framework. tasks from different perspectives without relying on specific scenarios, thereby effectively evaluating the model’s ability in the Mid-Level. Specifically, we consider the following task types (see the middle part of [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. An overview of the full-path clinical diagnosis task ap￾plied in the proposed evaluation framework. (e.g., predefined answer choices), whereas real-world tasks involve significantly broader decision spaces. For instance, there may be hundreds of potential candidate diseases in clinical diagnosis, requiring more complex decision-making to reach the correct diagnosis. To sim￾ulate this, we design a task where the mode… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The recognize-and-retrieve procedure for constructing the High-Level dataset, ensuring alignment with previous levels in terms of knowledge coverage. real-world medical scenarios. Dataset Construction For Low-Level tasks, we directly use the questions from the MedQA (J…
Figure 5
Figure 5. Figure 5: Performance of LLMs across different parameters sizes on the proposed benchmark. The performance of Low and Mid￾Level tasks is the macro average across MedQA and MedMCQA. average performance of the three task types to compare with the other two levels (Detailed perform…
Figure 7
Figure 7. Figure 7: Average diagnosis accuracy of LLMs (Llama2 and 3, Qwen1, 2, and 2.5) across diseases in the full-path clinical diagnosis task [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Examples of Low-Level tasks. Left: a MedQA question. Right: a MedMCQA question. A.1. Low-Level Tasks As introduced in Section 3.3, we adopt the original multiple-choice questions in the MedQA and MedMCQA datasets for Low-Level tasks. MedQA is a large-scale medical exam…
Figure 9
Figure 9. Figure 9: Input prompt format of the five-shot in-context learning setting. We use five-shot in-context learning to evaluate the performance of LLMs on Low-Level tasks [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Examples of Mid-Level tasks. Left: Statement Validation tasks; Middle: Multi-step Rectification tasks; Right: Answer Existence Judgment tasks. We construct Mid-Level tasks by reformulating the original MCQs in MedQA and MedMCQA into statement verification tasks, multi…
Figure 11
Figure 11. Figure 11: Number of admission records for each disease in the constructed full-path clinical-diagnosis evaluation dataset. We then construct the full-path clinical-diagnosis evaluation dataset by extracting the history of present illness, physical examination sections from the …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 19 canonical work pages

  1. [1]

    J., Javaheripi, M., Kauffmann, P., et al

    Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., et al. Phi-4 technical report. ArXiv preprint, abs/2412.08905, 2024 a . URL https://arxiv.org/abs/2412.08905

    arXiv 2024

  2. [2]

    A., Awan, A

    Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et al. Phi-3 technical report: A highly capable language model locally on your phone. ArXiv preprint, abs/2404.14219, 2024 b . URL https://arxiv.org/abs/2404.14219

    arXiv 2024

  3. [3]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. ArXiv preprint, abs/2303.08774, 2023. URL https://arxiv.org/abs/2303.08774

    arXiv 2023

  4. [4]

    Anderson, L. W. and Krathwohl, D. R. A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives: complete edition. Addison Wesley Longman, Inc., 2001

    work page 2001

  5. [5]

    M., Hauth, A., et al

    Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. ArXiv preprint, abs/2312.11805, 2023. URL https://arxiv.org/abs/2312.11805

    arXiv 2023

  6. [6]

    Qwen technical report

    Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report. ArXiv preprint, abs/2309.16609, 2023. URL https://arxiv.org/abs/2309.16609

    arXiv 2023

  7. [7]

    Overview of the medical question answering task at trec 2017 liveqa

    Ben Abacha , A., Agichtein, E., Pinter, Y., and Demner - Fushman, D. Overview of the medical question answering task at trec 2017 liveqa. In TREC 2017, 2017

    work page 2017

  8. [8]

    S., Englehart, M

    Bloom, B. S., Englehart, M. D., Furst, E. J., Hill, W. H., Krathwohl, D. R., et al. Taxonomy of educational objectives, handbook i: the cognitive domain. new york: David mckay co, 1956

    work page 1956

Show all 40 references
  1. [9]

    The unified medical language system (umls): integrating biomedical terminology

    Bodenreider, O. The unified medical language system (umls): integrating biomedical terminology. Nucleic acids research, 32 0 (suppl\_1): 0 D267--D270, 2004

    2004

  2. [10]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert - Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E.,...

    2020

  3. [11]

    Medbench: A large-scale chinese benchmark for evaluating medical large language models

    Cai, Y., Wang, L., Wang, Y., de Melo, G., Zhang, Y., Wang, Y., and He, L. Medbench: A large-scale chinese benchmark for evaluating medical large language models. In Wooldridge, M. J., Dy, J. G., and Natarajan, S. (eds.), Thirty-Eighth AAAI Conference on Artificial Intelligence...

    2024

  4. [12]

    H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., K \"o pf, A., Mohtashami, A., et al

    Chen, Z., Cano, A. H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., K \"o pf, A., Mohtashami, A., et al. Meditron-70b: Scaling medical pretraining for large language models. ArXiv preprint, abs/2311.16079, 2023. URL https://arxiv.org/abs/2311.16079

    2023 arXiv

  5. [13]

    U., Pimentel, M

    Christophe, C., Kanithi, P., Munjal, P., Raha, T., Hayat, N., Rajan, R., Al Mahrooqi, A., Gupta, A., Salman, M. U., Pimentel, M. A., et al. Med42-evaluating fine-tuning strategies for medical llms: Full-parameter vs. parameter-efficient approaches. In AAAI 2024 Spring Symposiu...

    2024

  6. [14]

    K., Raha, T., Khan, S., and Pimentel, M

    Christophe, C., Kanithi, P. K., Raha, T., Khan, S., and Pimentel, M. A. Med42-v2: A suite of clinical llms, 2024 b . URL https://arxiv.org/abs/2408.06142

    2024 arXiv

  7. [15]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. ArXiv preprint, abs/2407.21783, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  8. [16]

    Evaluation and mitigation of the limitations of large language models in clinical decision-making

    Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature medicine, 30 0 (9): 0 2613--2622, 2024

    2024

  9. [17]

    Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Dang, K., et al. Qwen2. 5-coder technical report. ArXiv preprint, abs/2409.12186, 2024. URL https://arxiv.org/abs/2409.12186

    2024 arXiv

  10. [18]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams

    Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., and Szolovits, P. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11 0 (14): 0 6421, 2021

    2021

  11. [19]

    P ub M ed QA : A dataset for biomedical research question answering

    Jin, Q., Dhingra, B., Liu, Z., Cohen, W., and Lu, X. P ub M ed QA : A dataset for biomedical research question answering. In Inui, K., Jiang, J., Ng, V., and Wan, X. (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inte...

    2019 doi

  12. [20]

    E., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horng, S., Pollard, T

    Johnson, A. E., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horng, S., Pollard, T. J., Hao, S., Moody, B., Gow, B., et al. Mimic-iv, a freely accessible electronic health record dataset. Scientific data, 10 0 (1): 0 1, 2023

    2023

  13. [21]

    A., Roberts, A., et al

    Kraljevic, Z., Searle, T., Shek, A., Roguski, L., Noor, K., Bean, D., Mascio, A., Zhu, L., Folarin, A. A., Roberts, A., et al. Multi-domain clinical natural language processing with medcat: the medical concept annotation toolkit. Artificial intelligence in medicine, 117: 0 102...

    2021

  14. [22]

    Deepseek-v3 technical report

    Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  15. [23]

    S., Love, J., et al

    Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivi \`e re, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. ArXiv preprint, abs/2403.08295, 2024. URL https://arxiv.org/abs/2403.08295

    2024 arXiv

  16. [24]

    M., Carignan, D., and Horvitz, E

    Nori, H., King, N., McKinney, S. M., Carignan, D., and Horvitz, E. Capabilities of gpt-4 on medical challenge problems. ArXiv preprint, abs/2303.13375, 2023 a . URL https://arxiv.org/abs/2303.13375

    2023 arXiv

  17. [25]

    T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., King, N., Larson, J., Li, Y., Liu, W., et al

    Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., King, N., Larson, J., Li, Y., Liu, W., et al. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. ArXiv preprint, abs/2311.16452, 2023 b . URL https://arxiv.org/abs/2311.16452

    2023 arXiv

  18. [26]

    Gpt-4o mini: advancing cost-efficient intelligence, 2024

    OpenAI. Gpt-4o mini: advancing cost-efficient intelligence, 2024. URL https://openai.com/index/hello-gpt-4o

    2024

  19. [27]

    L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models t...

    2022

  20. [28]

    Climedbench: A large-scale chinese benchmark for evaluating medical large language models in clinical scenarios

    Ouyang, Z., Qiu, Y., Wang, L., De Melo, G., Zhang, Y., Wang, Y., and He, L. Climedbench: A large-scale chinese benchmark for evaluating medical large language models in clinical scenarios. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processin...

    2024

  21. [29]

    K., and Sankarasubbu, M

    Pal, A., Umapathi, L. K., and Sankarasubbu, M. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Flores, G., Chen, G. H., Pollard, T., Ho, J. C., and Naumann, T. (eds.), Proceedings of the Conference on Health, Inference, and L...

    2022

  22. [30]

    Towards building multilingual language model for medicine

    Qiu, P., Wu, C., Zhang, X., Lin, W., Wang, H., Zhang, Y., Wang, Y., and Xie, W. Towards building multilingual language model for medicine. Nature Communications, 15 0 (1): 0 8384, 2024

    2024

  23. [31]

    S., Wei, J., Chung, H

    Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. Large language models encode clinical knowledge. Nature, 620 0 (7972): 0 172--180, 2023 a

    2023

  24. [32]

    Towards expert-level medical question answering with large language models

    Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., et al. Towards expert-level medical question answering with large language models. ArXiv preprint, abs/2305.09617, 2023 b . URL https://arxiv.org/abs/2305.09617

    2023 arXiv

  25. [33]

    G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ram \'e , A., et al

    Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ram \'e , A., et al. Gemma 2: Improving open language models at a practical size. ArXiv preprint, abs/2408.00118, 2024. URL https://arxiv.org/abs/2408.00118

    2024 arXiv

  26. [34]

    R., Ba, J., Krishnan, R

    Toma, A., Lawler, P. R., Ba, J., Krishnan, R. G., Rubin, B. B., and Wang, B. Clinical camel: An open expert-level medical language model with dialogue-based knowledge encoding. ArXiv preprint, abs/2305.12031, 2023. URL https://arxiv.org/abs/2305.12031

    2023 arXiv

  27. [35]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. ArXiv preprint, abs/2302.13971, 2023 a . URL https://arxiv.org/abs/2302.13971

    2023 arXiv

  28. [36]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, abs/2307.09288, 2023 b . URL https://arxiv.org/abs/2307.09288

    2023 arXiv

  29. [37]

    CMB : A comprehensive medical benchmark in C hinese

    Wang, X., Chen, G., Dingjie, S., Zhiyi, Z., Chen, Z., Xiao, Q., Chen, J., Jiang, F., Li, J., Wan, X., Wang, B., and Li, H. CMB : A comprehensive medical benchmark in C hinese. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North Americ...

    2024

  30. [38]

    C., Wu, J., and Liu, X

    Yan, C., Fu, X., Xiong, Y., Wang, T., Hui, S. C., Wu, J., and Liu, X. Llm sensitivity evaluation framework for clinical diagnosis. In Proceedings of the 31st International Conference on Computational Linguistics, pp.\ 3083--3094, 2025

    2025

  31. [39]

    Qwen2 technical report

    Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. ArXiv preprint, abs/2407.10671, 2024. URL https://arxiv.org/abs/2407.10671

    2024 arXiv

  32. [40]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.