Doctorina MedBench-ICD10: A Dialogue-Based Benchmark and Evaluation Framework for Agent-Based Medical AI

Anna Kozlova; Hanna Plotnitskaya; Pavel Satalkin; Sergey Parfenyuk; Stanislau Salavei

REVIEW 5 major objections 6 minor 20 references

This paper claims that an interactive clinical dialogue benchmark reveals a sharp drop in base LLM performance relative to physicians, and that a structured agentic system closes most of that gap.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 17:19 UTC pith:JACOBFSN

load-bearing objection The framework is genuinely useful, but the headline claim that base LLMs crash in dialogue is built on a deliberately crippled baseline prompt; worth peer review, not as-is. the 5 major comments →

arxiv 2603.25821 v2 pith:JACOBFSN submitted 2026-03-26 cs.CL cs.AIcs.LGcs.MA

Doctorina MedBench-ICD10: A Dialogue-Based Benchmark and Evaluation Framework for Agent-Based Medical AI

Anna Kozlova , Stanislau Salavei , Pavel Satalkin , Hanna Plotnitskaya , Sergey Parfenyuk This is my paper

classification cs.CL cs.AIcs.LGcs.MA

keywords medical AI evaluationclinical dialogue simulationsimulated patienthistory-takingDOTS metricLLM vs physician performanceagentic medical systemdiagnostic accuracy

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Doctorina MedBench replaces static exam questions with a simulated doctor-patient dialogue. The paper claims that when an LLM must actively elicit a patient's history from a patient who volunteers little, base models' diagnostic and treatment accuracy drops sharply relative to practicing physicians. It further claims that wrapping a base LLM in a structured history-taking and reasoning system brings performance close to human physicians. This matters because it suggests standard medical benchmarks overstate how ready LLMs are for clinical use. The authors see the simulation format itself as a more realistic measure of clinical competence than exam-style tests.

Core claim

The central discovery is a measurement gap: on the same clinical cases, physicians and the agentic Doctorina system achieve diagnosis accuracy around 83-89 percent, while a bare LLM (GPT-5) given a one-sentence 'act as a doctor' prompt achieves roughly 81-84 percent and, more tellingly, asks very few targeted questions (about 0.66 dialogue steps on average versus 11.56 for Doctorina). The authors interpret this as evidence that competence on static tests does not transfer to interactive history-taking, and that the dialogue format exposes clinical reasoning strengths and weaknesses that multiple-choice questions hide.

What carries the argument

The measuring instrument is the simulated patient: a constrained LLM agent that answers only what is asked, volunteers nothing beyond the case record, paraphrases but never adds facts, and gives uncertain or negative replies when asked about absent information. This forces the physician model to drive the encounter. The D.O.T.S. metric (Diagnosis, Observations/Investigations, Treatment, Step count) scores both clinical correctness and dialogue efficiency, with mandatory, optional, and penalty categories for tests and a 'critical condition' override that zeroes unsafe treatment recommendations.

Load-bearing premise

The entire benchmark rests on the assumption that the simulated patient withholds and reveals information exactly like a real patient, so the measured gap is caused by the physician model's clinical skill rather than by the patient agent's behavior.

What would settle it

Replace the simulated patient with a human standardized patient (or allow the patient agent to volunteer information freely) on the same cases; if base LLM scores rise to physician levels, the reported performance gap is an artifact of the simulation rather than a measure of clinical competence.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Static exam-style benchmarks overestimate the clinical readiness of LLMs; interactive simulation adds a distinct dimension of assessment.
Medical AI systems need explicit history-taking and structured orchestration, not just raw model reasoning, to reach physician-level performance.
The same simulation and scoring can evaluate human physicians, supporting clinical training and competency monitoring.
Safety-oriented 'trap' cases and tiered regression testing can catch model degradation during development and production.
The step-count metric acts as a behavioral audit, flagging redundant or insufficient questioning independent of final diagnostic accuracy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Because the comparisons rely on a simulated patient, the observed gap might reflect the patient script's strictness rather than clinical skill; validating against human standardized patients would test this.
The GPT-5 baseline's near-zero dialogue steps suggest the comparison partly measures whether the model follows a 'conduct a dialogue' instruction, not only clinical knowledge; adding a history-taking prompt might narrow the gap.
The framework's low differential-accuracy scores for both AI and humans suggest the dialogue format is especially sensitive to hypothesis breadth, which exam scores rarely capture.
The same evaluation could be adapted for automated coaching of medical students, using step count and question coverage as feedback signals.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

The framework is genuinely useful, but the headline claim that base LLMs crash in dialogue is built on a deliberately crippled baseline prompt; worth peer review, not as-is.

read the letter

Here's my take. The framework is the real contribution: an interactive dialogue benchmark with a constrained LLM patient, the D.O.T.S. scoring scheme, trap cases, and a three-tier monitoring loop. That's genuinely useful, and the paper describes it concretely. The idea that static QA overstates clinical competence is important, and the patient-design that withholds information unless asked correctly forces history-taking to matter.

The soft spots are in the empirical claims. The headline that "base LLMs drop sharply compared to practicing physicians" is not supported by the evidence, because the baseline is deliberately crippled. GPT-5 got a one-sentence prompt with no instruction to interview; the simulated patient withholds information unless asked. So of course GPT-5, averaging 0.66 steps, scores poorly. That's a prompt artifact, not a measure of base-LLM clinical capability. Tellingly, on the 100 baseline cases, GPT-5 scored 81% vs clinicians' 83%—hardly a sharp drop. The OB/GYN table shows a bigger gap, but the same prompt problem applies. A fair baseline should at least say "ask the patient questions."

The other soft spot is circularity: the dataset, gold standards, and D.O.T.S. weights are all authored by the team building Doctorina, and cases come from Doctorina's own logs. That doesn't invalidate the framework, but it means the absolute numbers are not independent evidence. Human validation is only 3 GPs on 100 cases, with no statistical analysis. And the numbers across sections don't obviously reconcile: §6.1 reports 83.64% average diagnosis accuracy, while §6.2 reports AI Doctor at 89.3% on a 254-case sample. That may be different runs, but it's not explained.

Still, this deserves a serious referee. The engineering is coherent, the D.O.T.S. scoring is well thought out, and the monitoring architecture is a real contribution. What needs revision is the comparison methodology and the overgeneralized conclusion. If the authors release data and code and rerun the baseline with a reasonable instruction to conduct an interview, the central claim would be testable.

Referee Report

5 major / 6 minor

Summary. The paper presents Doctorina MedBench, an interactive, dialogue-based evaluation framework for medical AI systems. It replaces static test items with simulated physician-patient conversations in which an LLM-driven patient agent discloses information only in response to targeted questions, and it scores the resulting dialogues with the D.O.T.S. metric (Diagnosis, Observations/Investigations, Treatment, Step Count). The authors describe a dataset of over 1,000 clinical cases, a multi-level testing and monitoring architecture, and a comparison between the Doctorina agent and several base LLMs, along with a small human-physician validation. The paper reports that Doctorina outperforms a deliberately restricted GPT-5 baseline on diagnosis and treatment accuracy, and that it roughly matches general practitioners on 100 basic cases, while the base LLM drops sharply on specialty-specific OB/GYN cases. The central claim, stated in Section 8, is that interactive clinical simulation makes base LLMs' performance drop sharply compared with practicing physicians, implying that static exam-style benchmarks overestimate real-world clinical competence.

Significance. If the framework were independently validated and the data/code released, this would be a valuable contribution: the need for interactive, history-taking-based benchmarks for medical LLMs is real, and the hybrid design—schema-constrained LLM extraction, evidence-quote requirements, deterministic scoring, and safety-critical overrides—is a sensible way to improve auditability over pure LLM-as-a-judge scoring. The multi-level regression and trap-case architecture is also a useful engineering contribution for continuous monitoring. However, the empirical claims as presented are not yet supported: the headline comparison uses a minimal single-sentence baseline prompt that essentially prevents the base LLM from collecting history, the gold standards and case weights are authored internally, and the human validation is very small and under-analyzed. The paper is best read as a framework proposal with promising internal results, not yet as a validated public benchmark.

major comments (5)

[§6.2, §6.2.2, §7, §8] The central comparative claim is undermined by the baseline design. GPT-5 was given only the prompt "Imagine that you are a doctor. Make a diagnosis or diagnoses" (§6.2), while the simulated patient is explicitly constrained to reveal nothing unless asked (§2.2). The reported 0.66 average steps per GPT-5 run (§6.2.2) means the baseline almost never conducted an interview, so its low scores are largely an artifact of the prompt, not a measurement of base-LLM clinical capability. Section 8's conclusion that "the performance of base LLMs drops sharply compared to practicing physicians" is not supported by the 100-case GP comparison in §7, where GPT-5 scored 81% versus clinicians' 83%, nor by Table 2, which uses the same restricted prompt. A fair baseline should instruct the model to conduct a medical interview, or the paper should compare against several prompting/configurations and state t
[§5.1, §8] The gold-standard cases and per-case recommendation weights are authored by the same organization that develops Doctorina, and the cases derive from Doctorina's own consultation logs. The human validation consists of only 3 GPs on 100 selected 'baseline' cases, with no formal inter-rater reliability analysis and no statistical comparison against Doctorina's scores. The paper itself acknowledges in §8 that independent datasets are needed to minimize researcher bias, but this is a load-bearing issue, not a cosmetic one: the D.O.T.S. scores, the O/T weights, and the conclusion that Doctorina matches or outperforms physicians all rely on the internal gold standards. The authors should provide an external clinician panel to audit cases and weights, report inter-rater agreement, or at minimum release a fully independent test set.
[§2.2, §3.1] The load-bearing premise is that the LLM-driven simulated patient behaves like a real patient: it "answers only the question asked," "does not reveal clinically relevant details unless they are specifically elicited," and "does not introduce facts that are absent from the case record." No evidence is provided that this behavior matches real clinical disclosure patterns. If the simulated patient under-discloses or over-discloses relative to actual patients, every D.O.T.S. score and all model-to-model and model-to-physician comparisons are artifacts of patient-simulation fidelity. The paper should report a validation study of the patient agent—for example, comparing simulated dialogues with real patient transcripts, having clinicians rate the realism of information disclosure, or at least performing a sensitivity analysis with different patient-behavior regimes (e.g., more or less forthcom
[§6.1, §6.2.1, §6.2.3, Table 2] There are unresolved inconsistencies in the reported numbers. §6.1 reports average Diagnosis Accuracy of 83.64%, while §6.2.1 reports AI Doctor's Diagnosis Accuracy as 89.3% and §6.2.3 reports a mean improvement of 6.56 percentage points over GPT-5; these may come from different subsets or runs, but the manuscript never explains which. Table 2 reports only "Percentage of correct answers" without N per group, model versions, confidence intervals, or step counts. The 100-case human comparison in §7 (83% vs. 87%) is likewise presented without CIs or a test statistic. The step-count gold standard and the O/T weights are free parameters, and no sensitivity analysis is shown. These omissions prevent the reader from auditing the central quantitative claims and should be corrected with full case-level data or a detailed appendix.
[§5.1, §2.1] For a paper presenting a benchmark, the absence of a public release is a significant limitation: no dataset URL, no access to the case bank, no evaluator code, and no case-level scores. The paper's title and abstract describe "Doctorina MedBench" as a benchmark, but as written it is an internal evaluation system. Reproducibility and adoption by the community are impossible without at least an anonymized subset of cases, the evaluator logic, and the prompt templates. I would ask the authors to release these artifacts or clearly state what would be needed for independent replication.

minor comments (6)

[§6.2, Eq. (1)] The definition of Question Accuracy is ambiguous: the equation uses 'questions asked' in the denominator but the prose says 'mandatory questions.' Please clarify how mandatory questions are defined and how partial credit is handled.
[§6.2.3] Multiple statistical tests are reported (Wilcoxon signed-rank, McNemar) without correction for multiple comparisons or discussion of tied values in the paired differences; the large number of zero differences (e.g., 220 unchanged outcomes) should be discussed as it affects the interpretation of the tests.
[§5.2.2] The list of 'Scope' conditions and the 'Technical' category is not clearly defined; please provide concrete examples or a formal schema.
[Table 2] The base LLM identifiers 'gpt-5 (basic version)', 'grok-4 (basic version)', and 'gemini-3-pro (basic version)' need exact version strings, API/model snapshot dates, sampling parameters, and the definition of 'basic version'; otherwise the comparison cannot be reproduced.
[References] Reference [12] is a submission with no author names and no published venue; it should be updated or removed. Some other preprints also lack version/date information.
[§2.2] The text repeats the same idea in consecutive sentences: 'The patient behavior is intentionally designed...' and 'The patient's behavior is intentionally restricted...'. Please consolidate to avoid redundancy.

Circularity Check

2 steps flagged

The 'sharp drop' of base LLMs is built into the baseline definition rather than measured; the same team authors the gold standard and the evaluated system.

specific steps

self definitional [Section 6.2 / 6.2.2 / Section 8]
"To establish a rigorous baseline, GPT-5 was deployed in a restricted configuration, utilizing a concise, single-sentence system prompt: "Imagine that you are a doctor. Make a diagnosis or diagnoses". This setup intentionally excluded clinical heuristics, structured history-taking protocols, and agentic orchestration... the GPT-5 baseline averaged only 0.66 [steps per run]... "the performance of base LLMs drops sharply compared to practicing physicians.""

The term 'base LLMs' in Section 8 is defined by the Section 6.2 restricted prompt, and the simulated patient is defined in Section 2.2 to reveal nothing unless specifically elicited. A model instructed only to 'make a diagnosis' is not instructed to interview, so it averages 0.66 steps and never obtains the history needed for D.O.T.S. scoring. Its low score is a consequence of these two definitions, not an empirical property of base LLMs generally. The sharp-drop conclusion restates the construction rather than testing it. The Section 7 GP comparison (81.0% vs 83%) shows no sharp drop, underscoring that the headline gap is concentrated in the constructed baseline scenario.
self definitional [Section 5.1, Section 8, Limitations paragraph]
"The current dataset was developed by a group of physicians utilizing their expert knowledge and consultation logs from individuals who engaged with the Doctorina platform during its prototype testing phase, with their informed consent... A rigorous comparison of LLM systems with medical specialists requires the use of independently generated clinical datasets developed by external expert groups to minimize potential researcher bias."

The gold-standard diagnoses, investigation weights, treatment weights, and step limits are authored by the same organization that builds Doctorina and are partly grounded in Doctorina's own prototype consultation logs. Doctorina's claimed parity with physicians is therefore scored against a standard derived from the system's own design environment. The paper's own limitation statement admits that an externally generated dataset would be required for rigorous comparison, confirming that the current evaluation is self-referential rather than independent.

full rationale

The central derivation (D.O.T.S. scoring) is deterministic and not mathematically circular; there are no fitted parameters whose values are rediscovered as predictions, and there are no same-author self-citations doing load-bearing work. However, the paper's headline claim about base LLMs is partially built into the experiment design: the baseline was deliberately defined with no interviewing instruction while the simulated patient was defined to withhold information unless asked. The measured 0.66 average steps and the resulting low scores follow from those definitions, so Section 8's 'sharp drop' conclusion is partly a restatement of the construction rather than an independent finding. A second self-referential element is that the gold standards used to praise Doctorina come from the same physician group and Doctorina's own logs, with the paper conceding that independent datasets are needed. These problems are substantive limitations, but they do not make the metric arithmetic itself circular, so a moderate score of 6 is appropriate.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 3 invented entities

The framework is not a mathematical derivation; its outputs are determined by hand-authored gold standards and by the behavior of two LLM components (simulated patient and evaluator). The ledger shows that the central scores rest on domain assumptions that are asserted rather than independently validated.

free parameters (3)

Per-case recommendation weights for O and T scoring = not published (0-100 per item)
Section 5.1: 'weighting refers to the contribution of a specific recommendation to the overall score on a scale of 0 to 100.' These weights are hand-chosen by in-house physicians and are not in the paper; they directly determine D.O.T.S. scores.
Step-count gold standard s_limit^k = per-case integer, not published
Section 6.2: the ±25% step soft limit is defined against 'expected numsteps' from the case record; these expected values are chosen per case and not released.
Critical Condition override criteria = not published
Section 3.1: 'predefined criteria' trigger a zero score (e.g., amoxicillin in a penicillin-allergic patient); the full rule set is not published, making the scoring partly opaque.

axioms (4)

domain assumption The LLM-simulated patient's constrained disclosure behavior is a faithful model of real patient behavior.
Section 2.2: the patient agent is 'explicitly constrained to behave like a real patient... not reveal clinically relevant details unless they are specifically elicited.' No evidence is given that this matches real patients' disclosure patterns.
domain assumption LLM-as-a-judge schema-constrained extraction accurately recovers clinical entities from dialogue.
Section 3.2: the evaluator uses 'constrained LLM calls using Pydantic-based response formats' and evidence quotes; no human-annotator agreement or error analysis is reported.
domain assumption Physician-authored gold standards (diagnoses, ICD-10 codes, mandatory/optional tests, treatments, step limits) are correct and complete.
Section 5.1: cases were validated by at least two physicians from the same organization; no external expert panel or published clinical guideline is used to audit the weights.
domain assumption D.O.T.S. score components measure clinical competence rather than prompt-compliance.
The metric rewards structured history-taking and matching in-house gold standards; no correlation with patient outcomes or independent assessments is provided.

invented entities (3)

Simulated AI-patient agent no independent evidence
purpose: Acts as a realistic patient that withholds information and only answers targeted questions
No calibration against human patient behavior is reported; behavior is asserted via system-prompt constraints in Section 2.2, not measured.
D.O.T.S. scoring metric no independent evidence
purpose: Composite evaluation of Diagnosis, Observations, Treatment, and Step Count
New metric lacks validation against established clinical competence measures or external benchmarks; scores are computed from in-house gold standards.
Trap cases no independent evidence
purpose: Safety-oriented must-pass tests for continuous monitoring and regression detection
No list or evaluation results for trap cases are provided; they are described conceptually in Section 4.2.

pith-pipeline@v1.3.0-alltime-deepseek · 16511 in / 14111 out tokens · 131130 ms · 2026-08-02T17:19:55.560644+00:00 · methodology

0 comments

read the original abstract

We present Doctorina MedBench, a comprehensive evaluation framework for agent-based medical AI based on the simulation of realistic physician-patient interactions. Unlike traditional medical benchmarks that rely on solving standardized test questions, the proposed approach models a multi-step clinical dialogue in which either a physician or an AI system must collect medical history, analyze attached materials (including laboratory reports, images, and medical documents), formulate differential diagnoses, and provide personalized recommendations. System performance is evaluated using the D.O.T.S. metric, which consists of four components: Diagnosis, Observations/Investigations, Treatment, and Step Count, enabling assessment of both clinical correctness and dialogue efficiency. The system also incorporates a multi-level testing and quality monitoring architecture designed to detect model degradation during both development and deployment. The framework supports safety-oriented trap cases, category-based random sampling of clinical scenarios, and full regression testing. The dataset currently contains more than 1,000 clinical cases covering over 750 diagnoses. The universality of the evaluation metrics allows the framework to be used not only to assess medical AI systems, but also to evaluate physicians and support the development of clinical reasoning skills. Our results suggest that simulation of clinical dialogue may provide a more realistic assessment of clinical competence compared to traditional examination-style benchmarks.

Figures

Figures reproduced from arXiv: 2603.25821 by Anna Kozlova, Hanna Plotnitskaya, Pavel Satalkin, Sergey Parfenyuk, Stanislau Salavei.

**Figure 1.** Figure 1: AI Doctor evaluation workflow The orchestration of each turn is performed by a dialogue module (DoctorModel), which routes messages between the system’s functional agents. The system logs every step, preserving attachments, tracking completion flags, and recording model inputs and outputs for subsequent analysis. Upon completion of a simulated dialogue, the testing mechanism performs a multidimensional eva… view at source ↗

**Figure 2.** Figure 2: D.O.T.S. The primary metric for evaluating the performance of the AI physician model is the assessment of outcomes from a simulated dialogue between a virtual AI patient and an AI physician. The AI patient is provided with baseline data (an introductory prompt) that initiates the interaction, along with a repository of responses to potential inquiries within the simulated scenario covering secondary sympto… view at source ↗

**Figure 3.** Figure 3: Real-time Monitoring System Architecture. Short clinical ”trap” test scenarios are executed in parallel [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

**Figure 4.** Figure 4: Distribution of cases by clinical category [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

**Figure 5.** Figure 5: Distribution of cases by age group. The construction of the dataset and the balancing of categories were modeled after the official structure of the USMLE Step 2 CK exam. Work on expanding the dataset continues on an ongoing basis, while maintaining the specified balance between clinical disciplines. 5.2 Treatment Evaluation Logic. Categorization of tests. Each test has two levels of categorization. 5.2.1 … view at source ↗

**Figure 6.** Figure 6: Distribution of Medical Test Case Run Results [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗

**Figure 7.** Figure 7: Comparison of key quality metrics of GPT-5 (basic version) and AI Doctor - Part 1 [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗

**Figure 8.** Figure 8: Comparison of key quality metrics of GPT-5 (basic version) and AI Doctor - Part 2 [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗

**Figure 9.** Figure 9: Histogram of paired differences in treatment accuracy between GPT-5 (basic version) and AI Doctor [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗

**Figure 10.** Figure 10: Paired diagnostic outcomes (McNemar’s test) [PITH_FULL_IMAGE:figures/full_fig_p021_10.png] view at source ↗

**Figure 11.** Figure 11: CI comparing GPT-5 (basic version) and AI Doctor [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗

**Figure 12.** Figure 12: AI patient responses depending on the wording of the question, simulating a real conversation with a [PITH_FULL_IMAGE:figures/full_fig_p023_12.png] view at source ↗

**Figure 13.** Figure 13: Example of constructing a diagnostic reasoning chain under different initial conditions, using the ex [PITH_FULL_IMAGE:figures/full_fig_p024_13.png] view at source ↗

**Figure 14.** Figure 14: Example of constructing a diagnostic reasoning chain under different initial conditions, using the ex [PITH_FULL_IMAGE:figures/full_fig_p024_14.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

20 extracted references · 3 canonical work pages

[1]

Practice Manage- ment: Digital Health, 2025

American Medical Association.2 in 3 physicians are using health AI—up 78% from 2023. Practice Manage- ment: Digital Health, 2025. Available at:https://www.ama-assn.org/practice-management/digita l-health/2-3-physicians-are-using-health-ai-78-2023(accessed: 26.02.2025)

2023
[2]

Journal of Medical Internet Research

Zelin Yan, Jingwen Liu, Yihong Fan, Shiyuan Lu, Dingting Xu, Yun Yang, Honggang Wang, Jie Mao, Hou- Chiang Tseng, Tao-Hsing Chang, Yan Chen.Ability of ChatGPT to Replace Doctors in Patient Education: Cross-Sectional Comparative Analysis of Inflammatory Bowel Disease. Journal of Medical Internet Research. 2025;27:e62857. Available from:https://pmc.ncbi.nlm...

2025
[3]

Krichevsky, S

B. Krichevsky, S. Engeli, S. M. Bode-B ¨oger, et al.Human vs. artificial intelligence: Physicians outper- form ChatGPT in real-world pharmacotherapy counselling. British Journal of Clinical Pharmacology. 2025; (e70321). doi:10.1002/bcp.70321. Available at:https://bpspubs.onlinelibrary.wiley.com/doi/10 .1002/bcp.70321(accessed: 26.02.2025)

work page doi:10.1002/bcp.70321 2025
[4]

International Journal for Multidisciplinary Research

Veernapu K.The Implementation of AI in Clinical Decision Support System: Effects on Pa- tient Outcomes and Operational Costs. International Journal for Multidisciplinary Research. 2023;5(5). doi:10.36948/ijfmr.2023.v05i05.37225. Available at:https://www.researchgate.net/publicati on/389098016_The_Implementation_of_AI_in_Clinical_Decision_Support_System_Ef...

arXiv 2023
[5]

BMJ Open

Arvidsson R, Gunnarsson R, Entezarjou A, Sundemo D, Wikberg C.ChatGPT (GPT-4) versus doctors on complex cases of primary care: a blinded observational comparative study. BMJ Open. 2024;14(12):e086148. doi:10.1136/bmjopen-2024-086148. Available at:https://pmc.ncbi.nlm.nih.gov/articles/PMC116 83950/(accessed: 26.02.2025)

work page doi:10.1136/bmjopen-2024-086148 2024
[6]

Hayat H, et al.Toward the Autonomous AI Doctor: Quantitative Benchmarking of an Autonomous Agentic AI Versus Board-Certified Clinicians in a Real World Setting. medRxiv. 2025. doi:10.1101/2025.07.14.25331406. Available at:https://www.medrxiv.org/content/10.1101/2025.07.14.25331406v1(accessed: 26.02.2025)

work page doi:10.1101/2025.07.14.25331406 2025
[7]

T. H. Kung, M. Cheatham, A. Medenilla, et al.Performance of ChatGPT on USMLE: Potential for AI- assisted medical education using large language models. PLOS Digital Health. 2023; 2(2): e0000198. doi:10.1371/journal.pdig.0000198. Available at:https://journals.plos.org/digitalhealth/ar ticle?id=10.1371/journal.pdig.0000198(accessed: 10.03.2026)

doi:10.1371/journal.pdig.0000198 2023
[8]

A. M. Bean, R. E. Payne, G. Parsons, et al.Reliability of LLMs as medical assistants for the general public: a randomized preregistered study. Nature Medicine. 2026; 32: 609–615. doi:10.1038/s41591-025-04074-y. Available at:https://www.nature.com/articles/s41591-025-04074-y(accessed: 10.03.2026)

doi:10.1038/s41591-025-04074-y 2026
[9]

Jiang, K

Y. Jiang, K. C. Black, G. Geng, et al.MedAgentBench: A Virtual EHR Environment to Benchmark Medical LLM Agents. NEJM AI. 2025; 2(9). doi:10.1056/AIdbp2500144. Available at:https://ai.nejm.org/doi/ full/10.1056/AIdbp2500144(accessed: 10.03.2026)

doi:10.1056/aidbp2500144 2025
[10]

BMC Medical Education

Zidoun Y, El Mardi A.Artificial Intelligence (AI)-Based simulators versus simulated patients in undergrad- uate programs: A protocol for a randomized controlled trial. BMC Medical Education. 2024;24(1):1260. doi:10.1186/s12909-024-06236-x. Available at:https://pubmed.ncbi.nlm.nih.gov/39501219/(ac- cessed: 26.02.2025)

arXiv 2024
[11]

arXiv preprint arXiv:2505.08775

Arora RK, Wei J, Soskin Hicks R, Bowman P, Qui ˜nonero-Candela J, Tsimpourlas F, Sharman M, Shah M, Vallone A, Beutel A, Heidecke J, Singhal K.HealthBench: Evaluating Large Language Models Towards Improved Human Health. arXiv preprint arXiv:2505.08775. 2025. doi:10.48550/arXiv.2505.08775. Available at:https://arxiv.org/abs/2505.08775(accessed: 26.02.2025)...

Pith/arXiv arXiv doi:10.48550/arxiv.2505.08775 2025
[13]

Ben Shoham, N

O. Ben Shoham, N. Rappoport.MedConceptsQA: Open source medical concepts QA benchmark. Computers in Biology and Medicine. 2024; 182: 109089. doi:10.1016/j.compbiomed.2024.109089. Available at:https: //www.sciencedirect.com/science/article/pii/S0010482524011740(accessed: 10.03.2026)

arXiv 2024
[14]

Y. Xiao, J. Huang, R. He, et al.FairMedQA: Benchmarking Bias in Large Language Models for Medical Ques- tion Answering. arXiv preprint. 2025; arXiv:2505.19562 [cs.AI]. doi:10.48550/arXiv.2505.19562. Available at:https://arxiv.org/abs/2505.19562(accessed: 10.03.2026)

doi:10.48550/arxiv.2505.19562 2025
[15]

Y. Zuo, S. Qu, Y. Li, et al.MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding. arXiv preprint. 2025; arXiv:2501.18362 [cs.AI]. doi:10.48550/arXiv.2501.18362. Available at:https://ar xiv.org/abs/2501.18362(accessed: 10.03.2026)

Pith/arXiv arXiv doi:10.48550/arxiv.2501.18362 2025
[16]

JAMA Network Open

Bedi S, Jiang Y, Chung P, Koyejo S, Shah NH.Fidelity of Medical Reasoning in Large Language Models. JAMA Network Open. 2025;8(7):e2526021. doi:10.1001/jamanetworkopen.2025.26021. Available at:https: //jamanetwork.com/journals/jamanetworkopen/fullarticle/2837372(accessed: 26.02.2025)

arXiv 2025
[17]

Nature Medicine

Hager P, Jungmann F, Holland R, et al.Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature Medicine. 2024;30(9):2613–2622. Available at:https://www.nature .com/articles/s41591-024-03097-1(accessed: 26.02.2025)

2024
[18]

arXiv:2305.09617, 2023

Singhal K, Tu T, Gottweis J, et al.Towards Expert-Level Medical Question Answering with Large Lan- guage Models. arXiv:2305.09617, 2023. Available at:https://arxiv.org/abs/2305.09617(accessed: 26.02.2025)

Pith/arXiv arXiv 2023
[19]

B. T. Bicknell, D. Butler, S. Whalen, et al.ChatGPT-4 Omni Performance in USMLE Disciplines and Clinical Skills: Comparative Analysis. JMIR Medical Education. 2024; 10: e63430. doi:10.2196/63430. Available at: https://mededu.jmir.org/2024/1/e63430(accessed: 10.03.2026)

doi:10.2196/63430 2024
[20]

M. K. Siam, A. Varela, M. J. H. Faruk, et al.Benchmarking large language models on the United States medical licensing examination for clinical reasoning and medical licensing scenarios. Scientific Reports. 2026; 16:

2026
[1387]

Available at:https://www.nature.com/articles/s41598-025 -31010-4(accessed: 10.03.2026)

doi:10.1038/s41598-025-31010-4. Available at:https://www.nature.com/articles/s41598-025 -31010-4(accessed: 10.03.2026). 29 ANNA KOZLOVA received the M.S. degree in applied mathe- matics and computer science from Saint Pe- tersburg State University in 2021. She is a Machine Learning Engineer with over six years of R&D experience in classical ma- chine lear...

doi:10.1038/s41598-025-31010-4 2026

[1] [1]

Practice Manage- ment: Digital Health, 2025

American Medical Association.2 in 3 physicians are using health AI—up 78% from 2023. Practice Manage- ment: Digital Health, 2025. Available at:https://www.ama-assn.org/practice-management/digita l-health/2-3-physicians-are-using-health-ai-78-2023(accessed: 26.02.2025)

2023

[2] [2]

Journal of Medical Internet Research

Zelin Yan, Jingwen Liu, Yihong Fan, Shiyuan Lu, Dingting Xu, Yun Yang, Honggang Wang, Jie Mao, Hou- Chiang Tseng, Tao-Hsing Chang, Yan Chen.Ability of ChatGPT to Replace Doctors in Patient Education: Cross-Sectional Comparative Analysis of Inflammatory Bowel Disease. Journal of Medical Internet Research. 2025;27:e62857. Available from:https://pmc.ncbi.nlm...

2025

[3] [3]

Krichevsky, S

B. Krichevsky, S. Engeli, S. M. Bode-B ¨oger, et al.Human vs. artificial intelligence: Physicians outper- form ChatGPT in real-world pharmacotherapy counselling. British Journal of Clinical Pharmacology. 2025; (e70321). doi:10.1002/bcp.70321. Available at:https://bpspubs.onlinelibrary.wiley.com/doi/10 .1002/bcp.70321(accessed: 26.02.2025)

work page doi:10.1002/bcp.70321 2025

[4] [4]

International Journal for Multidisciplinary Research

Veernapu K.The Implementation of AI in Clinical Decision Support System: Effects on Pa- tient Outcomes and Operational Costs. International Journal for Multidisciplinary Research. 2023;5(5). doi:10.36948/ijfmr.2023.v05i05.37225. Available at:https://www.researchgate.net/publicati on/389098016_The_Implementation_of_AI_in_Clinical_Decision_Support_System_Ef...

arXiv 2023

[5] [5]

BMJ Open

Arvidsson R, Gunnarsson R, Entezarjou A, Sundemo D, Wikberg C.ChatGPT (GPT-4) versus doctors on complex cases of primary care: a blinded observational comparative study. BMJ Open. 2024;14(12):e086148. doi:10.1136/bmjopen-2024-086148. Available at:https://pmc.ncbi.nlm.nih.gov/articles/PMC116 83950/(accessed: 26.02.2025)

work page doi:10.1136/bmjopen-2024-086148 2024

[6] [6]

Hayat H, et al.Toward the Autonomous AI Doctor: Quantitative Benchmarking of an Autonomous Agentic AI Versus Board-Certified Clinicians in a Real World Setting. medRxiv. 2025. doi:10.1101/2025.07.14.25331406. Available at:https://www.medrxiv.org/content/10.1101/2025.07.14.25331406v1(accessed: 26.02.2025)

work page doi:10.1101/2025.07.14.25331406 2025

[7] [7]

T. H. Kung, M. Cheatham, A. Medenilla, et al.Performance of ChatGPT on USMLE: Potential for AI- assisted medical education using large language models. PLOS Digital Health. 2023; 2(2): e0000198. doi:10.1371/journal.pdig.0000198. Available at:https://journals.plos.org/digitalhealth/ar ticle?id=10.1371/journal.pdig.0000198(accessed: 10.03.2026)

doi:10.1371/journal.pdig.0000198 2023

[8] [8]

A. M. Bean, R. E. Payne, G. Parsons, et al.Reliability of LLMs as medical assistants for the general public: a randomized preregistered study. Nature Medicine. 2026; 32: 609–615. doi:10.1038/s41591-025-04074-y. Available at:https://www.nature.com/articles/s41591-025-04074-y(accessed: 10.03.2026)

doi:10.1038/s41591-025-04074-y 2026

[9] [9]

Jiang, K

Y. Jiang, K. C. Black, G. Geng, et al.MedAgentBench: A Virtual EHR Environment to Benchmark Medical LLM Agents. NEJM AI. 2025; 2(9). doi:10.1056/AIdbp2500144. Available at:https://ai.nejm.org/doi/ full/10.1056/AIdbp2500144(accessed: 10.03.2026)

doi:10.1056/aidbp2500144 2025

[10] [10]

BMC Medical Education

Zidoun Y, El Mardi A.Artificial Intelligence (AI)-Based simulators versus simulated patients in undergrad- uate programs: A protocol for a randomized controlled trial. BMC Medical Education. 2024;24(1):1260. doi:10.1186/s12909-024-06236-x. Available at:https://pubmed.ncbi.nlm.nih.gov/39501219/(ac- cessed: 26.02.2025)

arXiv 2024

[11] [11]

arXiv preprint arXiv:2505.08775

Arora RK, Wei J, Soskin Hicks R, Bowman P, Qui ˜nonero-Candela J, Tsimpourlas F, Sharman M, Shah M, Vallone A, Beutel A, Heidecke J, Singhal K.HealthBench: Evaluating Large Language Models Towards Improved Human Health. arXiv preprint arXiv:2505.08775. 2025. doi:10.48550/arXiv.2505.08775. Available at:https://arxiv.org/abs/2505.08775(accessed: 26.02.2025)...

Pith/arXiv arXiv doi:10.48550/arxiv.2505.08775 2025

[12] [13]

Ben Shoham, N

O. Ben Shoham, N. Rappoport.MedConceptsQA: Open source medical concepts QA benchmark. Computers in Biology and Medicine. 2024; 182: 109089. doi:10.1016/j.compbiomed.2024.109089. Available at:https: //www.sciencedirect.com/science/article/pii/S0010482524011740(accessed: 10.03.2026)

arXiv 2024

[13] [14]

Y. Xiao, J. Huang, R. He, et al.FairMedQA: Benchmarking Bias in Large Language Models for Medical Ques- tion Answering. arXiv preprint. 2025; arXiv:2505.19562 [cs.AI]. doi:10.48550/arXiv.2505.19562. Available at:https://arxiv.org/abs/2505.19562(accessed: 10.03.2026)

doi:10.48550/arxiv.2505.19562 2025

[14] [15]

Y. Zuo, S. Qu, Y. Li, et al.MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding. arXiv preprint. 2025; arXiv:2501.18362 [cs.AI]. doi:10.48550/arXiv.2501.18362. Available at:https://ar xiv.org/abs/2501.18362(accessed: 10.03.2026)

Pith/arXiv arXiv doi:10.48550/arxiv.2501.18362 2025

[15] [16]

JAMA Network Open

Bedi S, Jiang Y, Chung P, Koyejo S, Shah NH.Fidelity of Medical Reasoning in Large Language Models. JAMA Network Open. 2025;8(7):e2526021. doi:10.1001/jamanetworkopen.2025.26021. Available at:https: //jamanetwork.com/journals/jamanetworkopen/fullarticle/2837372(accessed: 26.02.2025)

arXiv 2025

[16] [17]

Nature Medicine

Hager P, Jungmann F, Holland R, et al.Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature Medicine. 2024;30(9):2613–2622. Available at:https://www.nature .com/articles/s41591-024-03097-1(accessed: 26.02.2025)

2024

[17] [18]

arXiv:2305.09617, 2023

Singhal K, Tu T, Gottweis J, et al.Towards Expert-Level Medical Question Answering with Large Lan- guage Models. arXiv:2305.09617, 2023. Available at:https://arxiv.org/abs/2305.09617(accessed: 26.02.2025)

Pith/arXiv arXiv 2023

[18] [19]

B. T. Bicknell, D. Butler, S. Whalen, et al.ChatGPT-4 Omni Performance in USMLE Disciplines and Clinical Skills: Comparative Analysis. JMIR Medical Education. 2024; 10: e63430. doi:10.2196/63430. Available at: https://mededu.jmir.org/2024/1/e63430(accessed: 10.03.2026)

doi:10.2196/63430 2024

[19] [20]

M. K. Siam, A. Varela, M. J. H. Faruk, et al.Benchmarking large language models on the United States medical licensing examination for clinical reasoning and medical licensing scenarios. Scientific Reports. 2026; 16:

2026

[20] [1387]

Available at:https://www.nature.com/articles/s41598-025 -31010-4(accessed: 10.03.2026)

doi:10.1038/s41598-025-31010-4. Available at:https://www.nature.com/articles/s41598-025 -31010-4(accessed: 10.03.2026). 29 ANNA KOZLOVA received the M.S. degree in applied mathe- matics and computer science from Saint Pe- tersburg State University in 2021. She is a Machine Learning Engineer with over six years of R&D experience in classical ma- chine lear...

doi:10.1038/s41598-025-31010-4 2026