Pith. sign in

REVIEW 5 major objections 4 minor 7 cited by

GPT-5 outperforms GPT-4o and surpasses pre-licensed human experts on multimodal medical reasoning benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A benchmark study reports GPT-5 outperforming GPT-4o and pre-licensed human experts on most medical QA tasks, but not consistently on VQA-RAD.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Useful zero-shot CoT measurements of GPT-5 on existing medical benchmarks, but the 'consistently outperforms all baselines' claim is contradicted by their own VQA-RAD table and the above-human-expert result is under-supported by contamination checks and protocol comparability. the 5 major comments →

arxiv 2508.08224 v2 pith:C3UM23LL submitted 2025-08-11 cs.CL cs.AI

Capabilities of GPT-5 on Multimodal Medical Reasoning

Shansong Wang , Mingzhe Hu , Qiang Li , Mojtaba Safari , Xiaofeng Yang This is my paper
classification cs.CL cs.AI
keywords GPT-5multimodal medical reasoningvisual question answeringchain-of-thought promptingMedXpertQAclinical decision supportbenchmark evaluationhuman expert comparison
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper evaluates GPT-5 as a general-purpose multimodal reasoner for medicine, applying a unified zero-shot chain-of-thought protocol across text-only and visual medical question-answering benchmarks. It reports that GPT-5 consistently beats GPT-4o-2024-11-20 and the smaller GPT-5 variants, with particularly large gains on MedXpertQA, an expert-level benchmark that mixes images, patient narratives, and lab values. On MedXpertQA Multimodal, GPT-5's reasoning and understanding scores exceed those of pre-licensed human experts by about 24 and 29 percentage points, while GPT-4o stays mostly below human-level performance. The authors argue this marks a shift from human-comparable to above-human performance on controlled medical reasoning tasks, suggesting that one instruction-following model could serve as a reliable hub for clinical decision support.

Core claim

The paper claims that GPT-5, evaluated zero-shot with chain-of-thought prompting and a forced final answer, achieves 95.84% on MedQA (US 4-option), near-ceiling accuracy above 91% on MMLU medical subdomains, an average of 95.22% on USMLE self-assessment steps, and 69.99% reasoning and 74.37% understanding on MedXpertQA Multimodal. These MedXpertQA figures surpass pre-licensed human experts (45.76% and 44.97%) by substantial margins, whereas GPT-4o-2024-11-20 remains below human experts on most dimensions. The paper also notes that GPT-5 slightly underperforms GPT-5-mini on the radiology-specific VQA-RAD, suggesting that gains concentrate in reasoning-heavy, multimodal tasks rather than narro

What carries the argument

The evaluation protocol: a standardized zero-shot chain-of-thought prompt with a two-turn structure (first eliciting a free-form rationale, then converging to a single letter choice) applied identically to all models, together with standardized dataset splits. This design is meant to isolate the contribution of the model upgrade itself by holding prompting, answer constraints, and data splits constant across GPT-5 and GPT-4o.

Load-bearing premise

The test questions have not been seen during GPT-5's training, and the pre-licensed human expert scores were collected under the same conditions as the GPT-5 scores.

What would settle it

Evaluate GPT-5 on a fresh set of expert-level medical questions written after its training cutoff, with pre-licensed human experts answering the same questions under the same time-limited, forced-choice, image-inclusive protocol; GPT-5's accuracy falling to or below the human experts' level would refute the above-expert claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, a single generalist model can handle multimodal medical questions without task-specific fine-tuning, reducing the need for separate systems for text, images, and structured data.
  • Clinical decision-support systems could be built around zero-shot prompting of a large model, with GPT-5's gains being strongest in reasoning-intensive tasks such as USMLE Step 2 and MedXpertQA.
  • The reported improvements suggest that scaling model capability, rather than prompt engineering, drives most of the gain in medical multimodal reasoning.
  • The VQA-RAD result implies that on small, specialized benchmarks, larger models may be over-conservative; adaptive prompting or calibration may be needed to restore performance.
  • If these controlled benchmark results translate to practice, prospective clinical trials become the natural next step for validating the model's decision-support role.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 'above human expert' claim is benchmark-specific: it reflects time-limited, forced-choice, multiple-choice questions with clean images, not the full clinical workflow of history-taking, physical exam, and patient interaction; real-world transfer remains untested.
  • A concrete test of the claim would be to evaluate GPT-5 on freshly written expert-level questions authored after its training cutoff and compare its accuracy with pre-licensed human experts answering the same questions under identical timing and image conditions.
  • The authors' interpretation that GPT-5's multimodal gains come from improved cross-modal attention is plausible but not directly evidenced by benchmark accuracy alone; ablation or attention analysis would be needed to establish the mechanism.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The manuscript evaluates GPT-5, GPT-5-mini, GPT-5-nano, and GPT-4o-2024-11-20 on a set of medical QA and VQA benchmarks (MedQA, MedXpertQA text and multimodal, MMLU medical subsets, USMLE sample exams, and VQA-RAD) under zero-shot chain-of-thought prompting. It reports accuracy tables and a comparison with pre-licensed human expert scores, and claims that GPT-5 consistently outperforms all baselines, achieves state-of-the-art accuracy, and surpasses human experts on MedXpertQA by large margins. The authors make their code public and describe a unified prompting template.

Significance. If the claims were fully supported, the results would be notable: a general-purpose model exceeding pre-licensed clinicians on a challenging medical reasoning benchmark would be a meaningful data point for medical AI. The paper has some strengths: it uses a standardized prompting protocol across models, evaluates multiple benchmarks, and releases code. However, the central claims are currently undermined by an internal contradiction, missing statistical support, an underspecified human comparison, and no contamination analysis. These issues are load-bearing rather than cosmetic, so the significance cannot be assessed without substantial revision.

major comments (5)
  1. [Abstract and §3.3, Table 3] The abstract states that "GPT-5 consistently outperforms all baselines," but Table 3 shows GPT-5-mini at 74.90% versus GPT-5 at 70.92% on VQA-RAD. The discussion acknowledges this result, but the claim is not qualified in the abstract or conclusion. Because the headline assertion is directly contradicted by the paper's own data, the central claim requires correction.
  2. [§2.1 and §3.3] The VQA-RAD evaluation uses only the 251 yes/no test samples, and no confidence intervals, repeated runs, or statistical tests are reported. With 251 binary items, the observed gap between GPT-5 (70.92%) and GPT-5-mini (74.90%) may not be meaningful. The absence of uncertainty quantification affects all tables, but it is especially consequential for small subsets such as VQA-RAD and individual MMLU subjects.
  3. [§3.4, Table 4] The comparison with pre-licensed human experts is not adequately specified. The paper does not state whether the human expert scores were obtained under the same protocol (e.g., same question subset, same prompting format, same scoring rules, same allowed time) or whether they are taken from the MedXpertQA publication. If the human scores were collected under a different protocol, the "surpasses human experts by +24.23%" claim is not directly supported. This issue is central to the paper's super-human conclusion.
  4. [§2.1 and §3.1] No contamination audit is reported. The evaluated datasets (MedQA, MMLU, VQA-RAD, MedXpertQA) are public and predate GPT-5's release; MedXpertQA appeared in January 2025, about seven months before GPT-5. Near-ceiling scores such as 100% on MMLU Medical Genetics and 95.84% on MedQA are exactly the pattern expected from memorization. Without membership testing, canary evaluation, or at least a substantive discussion, the state-of-the-art and generalization claims are unsupported.
  5. [§1 and §3.1] The claim of "state-of-the-art accuracy across all QA benchmarks" is not established by the experimental design. The paper compares only four GPT-family models internally; it does not compare against published leaderboard results from other systems on the same standardized splits. The phrase "state-of-the-art" requires external comparison, which is absent.
minor comments (4)
  1. [§4] The section heading contains a typo: "Disscusion" should be "Discussion."
  2. [§2.1] The USMLE Self Assessment dataset description says sample questions are taken from official PDFs, but the processing steps (PDF parsing, question extraction, answer normalization) are not described. This limits reproducibility.
  3. [§3.3 and Figure 3] A single case study is presented as "representative." While illustrative, it does not provide quantitative evidence and should be framed as anecdotal, not as support for the aggregate claims.
  4. [Table 4] The table reports "Avg" columns for reasoning and understanding, but the averaging method is not defined, and the relationship of the average to the individual columns is not explained.

Circularity Check

0 steps flagged

No circular derivation: the paper reports external benchmark evaluations of GPT-5 against fixed public datasets and human scores; no fitted parameter is renamed as a prediction and no load-bearing self-citation appears.

full rationale

This paper is an empirical evaluation study: it runs GPT-5, GPT-5-mini, GPT-5-nano, and GPT-4o-2024-11-20 on standardized splits of MedQA, MedXpertQA, MMLU-Medical, USMLE Self Assessment, and VQA-RAD under a fixed zero-shot chain-of-thought prompting protocol, then compares accuracies. There is no derivation chain, no fitted parameter, and no quantity is defined in terms of another quantity that the paper then claims to predict. The human expert numbers in Table 4 are external inputs from the MedXpertQA benchmark, not derived from the GPT-5 outputs or from the authors' own models. The authors' self-citations (refs. [7], [8], [12]) appear only as general background on medical imaging and language models; none of them supplies a load-bearing premise for the reported GPT-5 results. The central claim that GPT-5 achieves state-of-the-art accuracy and surpasses pre-licensed human experts is a direct measurement against external public benchmarks, not a result that reduces to its own inputs. A plausible concern about benchmark contamination is a validity threat, but it is not a circularity in the paper's reasoning. Therefore the appropriate circularity score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

No free parameters or invented entities are introduced; the paper is an empirical benchmark evaluation. The load-bearing assumptions are about data leakage, comparability of human baseline scores, and representativeness of reduced subsets.

axioms (5)
  • domain assumption The benchmark test splits used were not part of GPT-5's training data.
    Unstated but required for the 'above human expert' claims. No contamination analysis is provided. Invoked in Section 3.4 and the conclusion.
  • domain assumption MedXpertQA pre-licensed human expert scores are directly comparable to GPT-5 scores under the same evaluation protocol.
    Human scores are borrowed from the MedXpertQA paper [27] with no evidence that prompts, eligible questions, or scoring rules match. Used in Table 4 and Section 3.4.
  • domain assumption Model API responses are stable enough that single-run accuracy without repeated sampling or confidence intervals is meaningful.
    No repeated runs, temperature, or seeds are reported; all tables report point estimates. Invoked in Tables 1-4.
  • domain assumption The VQA-RAD binary yes/no subset (251 samples) is representative enough to draw conclusions about VQA-RAD performance.
    Only yes/no test items are used, excluding open-ended items; the paper does not justify this subset. Invoked in Section 2.1 and Table 3.
  • domain assumption A simplified 4-option US MedQA test set is an appropriate benchmark for comparing model capabilities.
    The paper chooses the simplified version (removing one wrong answer), which is easier than the 5-option set and may inflate absolute accuracy. Invoked in Section 2.1 and Table 1.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Capabilities of GPT-5 on Multimodal Medical Reasoning." pith.science (2026). https://pith.science/paper/C3UM23LL

@misc{pith2026250808224,
  author       = {Pith},
  title        = {Pith review of: Capabilities of GPT-5 on Multimodal Medical Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C3UM23LL}},
  note         = {Machine review of arXiv:2508.08224}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent advances in large language models (LLMs) have enabled general-purpose systems to perform increasingly complex domain-specific reasoning without extensive fine-tuning. In the medical domain, decision-making often requires integrating heterogeneous information sources, including patient narratives, structured data, and medical images. This study positions GPT-5 as a generalist multimodal reasoner for medical decision support and systematically evaluates its zero-shot chain-of-thought reasoning performance on both text-based question answering and visual question answering tasks under a unified protocol. We benchmark GPT-5, GPT-5-mini, GPT-5-nano, and GPT-4o-2024-11-20 against standardized splits of MedQA, MedXpertQA (text and multimodal), MMLU medical subsets, USMLE self-assessment exams, and VQA-RAD. Results show that GPT-5 consistently outperforms all baselines, achieving state-of-the-art accuracy across all QA benchmarks and delivering substantial gains in multimodal reasoning. On MedXpertQA MM, GPT-5 improves reasoning and understanding scores by +29.26% and +26.18% over GPT-4o, respectively, and surpasses pre-licensed human experts by +24.23% in reasoning and +29.40% in understanding. In contrast, GPT-4o remains below human expert performance in most dimensions. A representative case study demonstrates GPT-5's ability to integrate visual and textual cues into a coherent diagnostic reasoning chain, recommending appropriate high-stakes interventions. Our results show that, on these controlled multimodal reasoning benchmarks, GPT-5 moves from human-comparable to above human-expert performance. This improvement may substantially inform the design of future clinical decision-support systems.

Figures

Figures reproduced from arXiv: 2508.08224 by Mingzhe Hu, Mojtaba Safari, Qiang Li, Shansong Wang, Xiaofeng Yang.

Figure 1
Figure 1. Figure 1: Prompting design for QA/VQA task 4 [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: A prompting design sample from MedXpertQA. 5 [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: GPT-5 reasoning output and final answer for MedXpertQA: case MM-1993. 4 Disscusion We evaluate the reasoning capabilities of the GPT-5 family of models on a wide range of multimodal tasks, revealing several key findings: First, GPT-5 delivers substantial gains in multimodal medical reasoning, especially in datasets like MedXpertQA MM that demand tight integration of image-derived evidence with textual pati… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MedOpenClaw and MedFlowBench: Auditing Medical Agents in Full-Study Workflows

    cs.CV 2026-03 conditional novelty 8.0

    MedFlowBench evaluates VLM agents on full radiology and pathology studies by requiring both task answers and verifiable evidence like key slices and regions of interest, revealing that answer-only scores overestimate ...

  2. Learning from Contrasts: Synthesizing Reasoning Paths from Diverse Search Trajectories

    cs.AI 2026-04 unverdicted novelty 7.0

    CRPS synthesizes reasoning paths by contrasting high- and low-quality MCTS trajectories, enabling models trained on 60K examples to match or exceed those trained on 590K standard examples with better out-of-domain gen...

  3. Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding

    cs.CV 2026-04 unverdicted novelty 7.0

    Bridge-STG decouples spatio-temporal alignment via semantic bridging and query-guided localization modules to achieve state-of-the-art m_vIoU of 34.3 on VidSTG among MLLM methods.

  4. CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement

    cs.CV 2026-08 conditional novelty 6.0

    A chest X-ray VLM co-trained with classification and grounding heads, tuned with DAPO reinforcement learning, and augmented with deterministic measurement tools outperforms prior radiology VLMs on report generation, V...

  5. CodeClinic: Evaluating Automation of Coding Skills for Clinical Reasoning Agents

    cs.AI 2026-05 unverdicted novelty 6.0

    CodeClinic benchmark demonstrates that LLM-generated Python skill libraries from clinical guidelines enhance consistency and reduce token consumption by up to 40% compared to zero-shot approaches on MIMIC-IV based tasks.

  6. RePrompT: Recurrent Prompt Tuning for Integrating Structured EHR Encoders with Large Language Models

    cs.CL 2026-04 unverdicted novelty 6.0

    RePrompT uses recurrent prompt tuning to inject prior-visit latent states and cohort-derived population prompt tokens into LLMs, yielding better performance than pure EHR or pure LLM baselines on MIMIC clinical predic...

  7. Regulating Anatomy-Aware Rewards via Trajectory-Integral Feedback for Volumetric Computed Tomography Analysis

    cs.CV 2026-05 unverdicted novelty 5.0

    TIF-GRPO uses integral feedback on pseudo-temporal trajectories to regulate anatomy-aware rewards in RL for clinical faithfulness in volumetric CT analysis.

Reference graph

Works this paper leans on

28 extracted references · 19 canonical work pages · cited by 7 Pith papers · 1 internal anchor

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Introducing gpt-5, August 7 2025

    OpenAI. Introducing gpt-5, August 7 2025

    work page 2025

  4. [4]

    Llm-mediated data extraction from patient records after radical prostatectomy

    William S Azar, Dylan M Junkin, Charles Hesswani, Christopher R Koller, Sahil H Parikh, Kyle C Schuppe, Nicholas Williams, Daniel Nethala, Neil Mendhiratta, Alexander P Kenigsberg, et al. Llm-mediated data extraction from patient records after radical prostatectomy. NEJM AI, 2(6):AIcs2400943, 2025

    work page 2025

  5. [5]

    Large language models encode clinical knowledge

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):172–180, 2023

    2023

  6. [6]

    Gpt4graph: Can large lan- guage models understand graph structured data? an empirical evaluation and benchmarking

    Jiayan Guo, Lun Du, Hengyu Liu, Mengyu Zhou, Xinyi He, and Shi Han. Gpt4graph: Can large lan- guage models understand graph structured data? an empirical evaluation and benchmarking. arXiv preprint arXiv:2305.15066, 2023

    Pith/arXiv arXiv 2023

  7. [7]

    Triad: Vision foundation model for 3d magnetic resonance imaging

    Shansong Wang, Mojtaba Safari, Qiang Li, Chih-Wei Chang, Richard LJ Qiu, Justin Roper, David S Yu, and Xiaofeng Yang. Triad: Vision foundation model for 3d magnetic resonance imaging. arXiv preprint arXiv:2502.14064, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    Unifying Biomedical Vision-Language Expertise: Towards a Generalist Foundation Model via Multi-CLIP Knowledge Distillation

    Shansong Wang, Zhecheng Jin, Mingzhe Hu, Mojtaba Safari, Feng Zhao, Chih-Wei Chang, Richard LJ Qiu, Justin Roper, David S Yu, and Xiaofeng Yang. Unifying biomedical vision-language expertise: Towards a generalist foundation model via multi-clip knowledge distillation. arXiv preprint arXiv:2506.22567, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  9. [9]

    Application of large language models in medicine

    Fenglin Liu, Hongjian Zhou, Boyang Gu, Xinyu Zou, Jinfa Huang, Jinge Wu, Yiru Li, Sam S Chen, Yining Hua, Peilin Zhou, et al. Application of large language models in medicine. Nature Reviews Bioengineering, pages 1–20, 2025

    work page 2025

  10. [10]

    Capabilities of gpt-4 on medical challenge problems

    Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    Large language models in medicine

    Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8):1930–1940, 2023

    1930

  12. [12]

    Advancing medical imaging with language models: featuring a spotlight on chatgpt

    Mingzhe Hu, Joshua Qian, Shaoyan Pan, Yuheng Li, Richard LJ Qiu, and Xiaofeng Yang. Advancing medical imaging with language models: featuring a spotlight on chatgpt. Physics in Medicine & Biology, 69(10):10TR01, 2024

    work page 2024

  13. [13]

    Introducing gpt-3.5, March 1 2023

    OpenAI. Introducing gpt-3.5, March 1 2023. 8

    work page 2023

  14. [14]

    Performance of multimodal gpt-4v on usmle with image: potential for imaging diagnostic support with explanations

    Zhichao Yang, Zonghai Yao, Mahbuba Tasmin, Parth Vashisht, Won Seok Jang, Feiyun Ouyang, Beining Wang, Dan Berlowitz, and Hong Yu. Performance of multimodal gpt-4v on usmle with image: potential for imaging diagnostic support with explanations. medRxiv, pages 2023–10, 2023

    work page 2023

  15. [15]

    Chatgpt versus the neurosurgical written boards: a comparative analysis of artificial intelligence/machine learning performance on neurosurgical board–style questions

    Benjamin S Hopkins, Vincent N Nguyen, Jonathan Dallas, Pavlos Texakalidis, Max Yang, Alex Renn, Gage Guerra, Zain Kashif, Stephanie Cheok, Gabriel Zada, et al. Chatgpt versus the neurosurgical written boards: a comparative analysis of artificial intelligence/machine learning performance on neurosurgical board–style questions. Journal of Neurosurgery, 139(...

    work page 2023

  16. [16]

    Assessing the performance of chatgpt in answering questions regarding cirrhosis and hepatocellular carcinoma

    Yee Hui Yeo, Jamil S Samaan, Wee Han Ng, Peng-Sheng Ting, Hirsh Trivedi, Aarshi Vipani, Walid Ayoub, Ju Dong Yang, Omer Liran, Brennan Spiegel, et al. Assessing the performance of chatgpt in answering questions regarding cirrhosis and hepatocellular carcinoma. Clinical and molecular hepatology, 29(3):721, 2023

    work page 2023

  17. [17]

    Assessing the accuracy and reliability of ai-generated medical responses: an evaluation of the chat-gpt model

    Douglas Johnson, Rachel Goodman, J Patrinely, Cosby Stone, Eli Zimmerman, Rebecca Donald, Sam Chang, Sean Berkowitz, Avni Finn, Eiman Jahangir, et al. Assessing the accuracy and reliability of ai-generated medical responses: an evaluation of the chat-gpt model. Research square, pages rs–3, 2023

    work page 2023

  18. [18]

    Gpt-4 in radiology: improvements in advanced reasoning

    Rajesh Bhayana, Robert R Bleakney, and Satheesh Krishna. Gpt-4 in radiology: improvements in advanced reasoning. Radiology, 307(5):e230987, 2023

    work page 2023

  19. [19]

    Chatgpt for digital pathology research

    Mohamed Omar, Varun Ullanat, Massimo Loda, Luigi Marchionni, and Renato Umeton. Chatgpt for digital pathology research. The Lancet Digital Health, 6(8):e595–e600, 2024

    work page 2024

  20. [20]

    Gizem Bozta¸ s Demir, Ya˘gızalp Süküt, Gökhan Serhat Duran, Kübra Gülnur Topsakal, and Serkan Görgülü. Enhancing systematic reviews in orthodontics: a comparative examination of gpt-3.5 and gpt-4 for generating pico-based queries with tailored prompts and configurations.European Journal of Orthodontics, 46(2):cjae011, 2024

    work page 2024

  21. [21]

    Comparison of the quality of discharge letters written by large language models and junior clinicians: single-blinded study

    Joshua Yi Min Tung, Sunil Ravinder Gill, Gerald Gui Ren Sng, Daniel Yan Zheng Lim, Yuhe Ke, Ting Fang Tan, Liyuan Jin, Kabilan Elangovan, Jasmine Chiat Ling Ong, Hairil Rizal Abdullah, et al. Comparison of the quality of discharge letters written by large language models and junior clinicians: single-blinded study. Journal of medical Internet research, 26...

    work page 2024

  22. [22]

    Ehrnoteqa: An llm benchmark for real-world clinical practice using discharge summaries

    Sunjun Kweon, Jiyoun Kim, Heeyoung Kwak, Dongchul Cha, Hangyul Yoon, Kwang Kim, Jeewon Yang, Seunghyun Won, and Edward Choi. Ehrnoteqa: An llm benchmark for real-world clinical practice using discharge summaries. Advances in Neural Information Processing Systems, 37:124575–124611, 2024

    work page 2024

  23. [23]

    Large language model integrations in cancer decision-making: a systematic review and meta- analysis

    Yuexing Hao, Zhiwen Qiu, Jason Holmes, Corinna E Löckenhoff, Wei Liu, Marzyeh Ghassemi, and Saleh Kalantari. Large language model integrations in cancer decision-making: a systematic review and meta- analysis. npj Digital Medicine, 8(1):450, 2025

    work page 2025

  24. [24]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. arXiv preprint arXiv:2009.13081, 2020

    Pith/arXiv arXiv 2009

  25. [25]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021

    work page 2021

  26. [26]

    A dataset of clinically generated visual questions and answers about radiology images

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1–10, 2018

    work page 2018

  27. [27]

    Medxpertqa: Benchmarking expert-level medical reasoning and understanding

    Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. Medxpertqa: Benchmarking expert-level medical reasoning and understanding. arXiv preprint arXiv:2501.18362, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Performance of chatgpt on usmle: potential for ai-assisted medical education using large language models

    Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. Performance of chatgpt on usmle: potential for ai-assisted medical education using large language models. PLoS digital health, 2(2):e0000198, 2023. 9

    work page 2023

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.