Pith. sign in

REVIEW 4 major objections 5 minor 52 references

OrthoPilot, an evidence-grounded LLM agent that retrieves hospital data and medical knowledge at each step, claims to match or beat experienced orthopaedic physicians across the entire care pathway—from admission diagnosis to rehabilitation

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 06:27 UTC pith:RFZJPRET

load-bearing objection Ambitious clinical LLM evaluation undercut by a scoring rubric too close to its own training reward; worth refereeing, but the expert-beating claim is not yet credible. the 4 major comments →

arxiv 2607.12527 v3 pith:RFZJPRET submitted 2026-07-14 cs.AI

Evidence-Grounded AI for Musculoskeletal Care

Wenjie Li , Yujie Zhang , Fanrui Zhang , Haoran Sun , Renhao Yang , Junjun He , Weiran Huang , Yuanfeng Ji
show 29 more authors
Chenrun Wang Kailing Wang Hongcheng Gao Kaipeng Zhang Hanyu Wang Angela Lin Wang Xingqi He Yilin Huang Shiyi Yao Lilong Wang Yankai Jiang Yirong Chen Chenglong Ma Jiyao Liu Ming Hu Gen Li Yidong Xu Chengyu Zhuang Jiawei Liu Yin Zhang Lequan Yu Lu Chen Yinpeng Dong Lei Liu Carlos Gutierrez Sanroman Yu Qiao Weijie Ma Xiaosong Wang Lei Wang
This is my paper
classification cs.AI
keywords musculoskeletal carelarge language modelsclinical decision supportlongitudinal managementevidence-grounded reasoningOrthoPilotOrthoBenchORACLE
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Musculoskeletal disease unfolds over months to years, so the paper argues that useful clinical AI must manage the whole care pathway—admission, surgery, discharge, rehabilitation—rather than answer isolated questions. OrthoPilot is an LLM-based agent that pulls patient-specific imaging, labs, orders, and records from hospital systems, couples them with guidelines and literature, and produces staged recommendations with a traceable reasoning chain. The paper claims that on a 1,000-disease benchmark, OrthoPilot reached 81.95% on diagnostics and 85.13% on open-ended management; in a reader study with 81 physicians it scored above experts with 25 years of experience; and in prospective studies it raised full-chain management success from 16.6% to 27.2% and increased cumulative cases per bed by 9.7%. If these results hold, clinical AI moves from single predictions to longitudinal execution—a shift that would matter for any disease course that evolves over time.

Core claim

OrthoPilot's central discovery is that coupling a clinically trained 32-billion-parameter LLM (CHEESE) to a dual-evidence agent architecture—in-hospital data retrieval plus external medical knowledge—enables sustained, evidence-grounded reasoning across all 11 decision points of a musculoskeletal care pathway. Because the system actively acquires missing evidence (imaging, labs, pathology, orders, consultations) rather than answering from a static prompt, it preserves temporal coherence from admission to rehabilitation. The paper reports that this design beat 22 baseline systems on OrthoBench (1,000 disease codes, 11 tasks), generalized across 60 external centres, matched or exceeded human e

What carries the argument

The central object is the 'dual evidence world': an in-hospital information world that exposes time-stamped patient records, imaging, labs, pathology, orders, consultations and similar historical cases, and an external knowledge world that retrieves guidelines, literature, textbooks and knowledge-graph triples. Tool Plaza exposes both as callable tools; the CHEESE main agent runs a ReAct-style loop, delegates focused retrieval to subagents, and synthesizes recommendations. The load-bearing constraint is the temporal evidence envelope: only evidence timestamped before the decision node enters reasoning, and target diagnoses are masked. ORACLE turns open-ended answers into weighted coverage of

Load-bearing premise

The results assume that the masking and timestamp rules keep target diagnoses and future records out of the evidence the agent sees; if that boundary fails for any retrieval channel—especially similar-case matching by diagnosis—the benchmark, reader-study, and prospective gains would be inflated relative to unaided physicians.

What would settle it

Pull a random sample of OrthoBench admission cases and inspect the top-k outputs of the similar-case retrieval tool before the admission-diagnosis node: if those returned trajectory summaries contain or strongly imply the target diagnosis, leakage is confirmed. The cleaner test is to rerun the internal benchmark and the reader study with the similar-case tool switched off; if the 81.95% / 85.13% scores and the 27.2% full-chain success drop substantially, the reported advantage depends on that channel rather than on the agent's sequential reasoning.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • OrthoPilot as a standalone system outperforms unassisted physicians, and physicians assisted by it improve at every experience tier, with the largest gains among specialists and residents compressing the experience gap.
  • Full-chain management success in 1,870 complex cases rises from 16.6% to 27.2%, while failure-prone outcomes drop from 22.1% to 11.8%.
  • In an 8-month, 8,240-patient deployment, OrthoPilot-assisted care reduced average length of stay, raised cumulative cases per bed from 11.90 to 13.06, lowered physician and nursing documentation workload, and improved patient-reported access to health information without changing interpersonal aspects of care.
  • The same performance hierarchy held across 60 external centres on 6,796 cases, indicating the gains are not confined to the training hospital's distribution.
  • OrthoPilot retained pathway-level reasoning for rare and out-of-distribution diseases—including oncological and developmental musculoskeletal conditions—not just common trauma and degenerative disorders.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the temporal evidence envelope is the operative mechanism, the same architecture could plausibly extend to other longitudinal specialties—oncology, critical care, stroke rehabilitation—where decisions must be revised as patient state changes; the paper gestures at this but does not test it.
  • A testable extension is to measure whether OrthoPilot's gains persist when similar-case retrieval is disabled or when its returned cases are stripped of diagnosis labels; this would isolate how much of the advantage comes from historical case matching versus from the agent's sequential reasoning.
  • The 9.7% bed-turnover gain is a workflow and throughput outcome; whether it translates to better long-term functional recovery after discharge remains open, as the paper itself acknowledges.
  • ORACLE's approach of converting expert rubrics into weighted coverage scores could become a reusable evaluation standard for open-ended clinical plans beyond musculoskeletal care.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents OrthoPilot, an LLM-based agent system for longitudinal musculoskeletal care. The system uses a 32B-parameter clinical LLM (CHEESE) trained in five stages, including supervised fine-tuning, GRPO-based reinforcement learning, trajectory distillation, and agentic reinforcement learning, with access to in-hospital EHR tools and external knowledge retrieval. The authors introduce OrthoBench, a benchmark spanning 1,000 disease codes and 11 tasks along the care pathway, and ORACLE, an automated rubric-based evaluation for open-ended clinical management responses. The reported results include: superior performance over 22 baselines on OrthoBench; multicentre validation across 60 hospitals; a reader study with 81 physicians claiming OrthoPilot outperforms experts with 25 years of experience; a prospective crossover study of 1,870 cases reporting full-chain management success rising from 16.6% to 27.2%; and an 8,240-patient randomized deployment reporting increased cumulative cases per bed from 11.90 to 13.06 and improved patient-reported information access. The abstract and Section 2.3.4 make the central load-bearing claim that OrthoPilot outperforms experienced orthopaedic physicians in diagnostic reasoning, clinical decision-making, and management planning.

Significance. If the results hold, this would be a substantial contribution: it is one of the largest reported evaluations of a clinical AI agent spanning an entire care pathway, with a disease coverage breadth (1,000 codes), multicentre external validation, and prospective deployment evidence that go beyond typical single-task benchmarks. The paper also provides a new benchmark (OrthoBench) and an automated evaluation framework (ORACLE) that could be reused by the community. The authors make code available on GitHub and commit to controlled data access. These strengths are real and should be credited. However, the significance is contingent on resolving the evaluative-circularity and data-leakage concerns described below; as written, the central comparative claims against physicians are not yet established.

major comments (4)
  1. [Sections 4.4.2, 4.5.3, 4.5.5; Eqs. (15), (24)] ORACLE is the metric used to score open-ended management responses in the reader study and benchmark. Stage III uses LLM-as-a-judge accuracy rewards and Stage V uses task-accuracy rewards (Eq. 24) on the same benchmark family, with the same rubric-aligned semantic comparison. Thus OrthoPilot is partly optimized against the metric used to evaluate it, making the reported 80.0% vs 65.4% expert advantage potentially reflect rubric alignment rather than clinical superiority. The human validation of ORACLE (Section 2.2, 691 cases, 12 specialists) shows correlation with physician rankings but is far smaller than the claimed evaluation scale and does not establish that ORACLE captures clinically superior decisions at the level needed. Provide an independent, blinded expert re-scoring of a representative sample of the reader-study responses to confirm the ORACLE-based comparisons.
  2. [Section 4.2.2, Eq. (11)] The masking rule in Eq. (11) filters in-hospital evidence by timestamp and node-specific masks, but the similar-case retrieval tool returns top-k historical cases with trajectory summaries matched by ICD-10 code and diagnosis-name similarity. The text states the query is constructed from time-valid evidence, but does not state that diagnosis labels or final outcomes are masked in the returned historical cases. If those summaries contain final diagnoses, the agent can obtain target-answer information across all tasks, inflating the benchmark, reader study, and prospective results. Specify the masking applied to the similar-case retrieval output and provide an ablation or leakage test showing that the reported gains persist when this tool is disabled or when its diagnosis labels are removed.
  3. [Section 2.4.1 and Methods 4.3.2] The primary prospective endpoint (full-chain success rising from 16.6% to 27.2%) depends on diagnostic correctness for P1, P2, and P4. The main text refers to 'reference standards' but does not specify what they are; the Methods state that correctness was determined by majority vote across the three experience-tier physicians who evaluated that case. If the reference standard is the majority vote of the same physicians whose decisions are being assisted, it is not an independent gold standard, and if it is the discharge diagnosis, it may encode information that OrthoPilot—trained on the same cohort—can exploit. Specify the exact adjudication criteria, how discharge/final diagnoses were excluded, and whether the reference standard was pre-registered.
  4. [Section 2.4.2, Figure 6a-b] The randomized deployment reports hospital-level outcomes such as cumulative cases per bed and ALOS, but randomization was performed at the patient level within a single centre. The comparison of cumulative bed turnover across two arms appears to use the hospital as the unit of analysis with only four binned 2-month intervals, providing no adjustment for within-hospital clustering or contamination between arms. Provide the unit-of-analysis justification, confidence intervals, and a statistical model that accounts for the hierarchical structure, or reframe the outcome as a patient-level endpoint.
minor comments (5)
  1. [Abstract / Section 2.4.1] The abstract states full-chain management success improved 'by 10.6%', but the text reports an increase from 16.6% to 27.2%, which is a 10.6-percentage-point increase, not a 10.6% relative increase. Please correct the wording.
  2. [Section 2.2 / Figure 2c] The ORACLE validation reports a Spearman rank correlation of -0.895 without explaining the sign. If physician rank 1 denotes the best response, a negative correlation is expected, but this should be stated explicitly to avoid confusion.
  3. [Section 2.3.4 / Figure 4] The reader study design has physicians re-evaluating the same cases after a one-month washout. While a washout mitigates recall, a crossover with the same cases can introduce carryover effects. The analysis should at least discuss this and, if feasible, report a period-by-treatment interaction.
  4. [Section 3, Limitations] The Discussion acknowledges uneven evidence and hallucination risks but does not address the potential leakage through similar-case retrieval or the circularity between training rewards and ORACLE. These should be discussed explicitly in the limitations.
  5. [Section 4.3.2] The prospective crossover study description says 'all three experience-level groups followed the same protocol independently' and each group evaluated all 1,870 cases. Please clarify how the 30 physicians were distributed and whether the reported paired analyses account for the fact that each case receives three physician ratings (one per tier).

Circularity Check

0 steps flagged

No significant circularity: the evaluation rubric and training rewards are related but not identical, test sets are patient-level held out, and no load-bearing result reduces by construction.

full rationale

The paper's derivation chain is not circular at the level of its own equations or citations. ORACLE (Eq. 15) scores coverage of physician-defined rubric entities derived from gold-standard answers, while the Stage III/V training rewards (Eqs. 18 and 24) use answer-correctness signals and LLM-judge semantic comparison; the paper never states that ORACLE itself is used as a training reward, and the OrthoBench test cohort is allocated with patient-level separation from model-development data (Section 4.3.1). The reader study compares physicians and OrthoPilot on the same ORACLE rubric, but ORACLE was independently validated against physician rankings across 691 cases and 12 specialists (Section 2.2), so the comparison has external content. The prospective P1/P2/P4 endpoints are defined against predefined reference standards (Supplementary Table J11), with majority vote used as an aggregation rule rather than as the gold standard itself; the text does not show that the reference changes with the intervention. The similar-case retrieval tool is a potential data-leakage/validity risk, not a by-construction circular step. The 60-centre external validation and the 8,240-patient randomized deployment rely on independent cohorts and external measurements, and no load-bearing self-citation chain is present. Accordingly, no circular step meets the required evidentiary bar.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

The central results rest on ORACLE-based scoring, the Ruijin cohort split, and masking/retrieval rules. Several of these are not independently verifiable from the main text: no ORACLE weights, no RL reward values, no cohort split details, and similar-case retrieval may reintroduce diagnosis labels. The external 60-centre and prospective studies are the main independent checks.

free parameters (4)
  • RL reward coefficients α, β, γ = not reported
    Eq. (24) combines accuracy, format and tool-use rewards; values are chosen by hand and not given, but they shape the agentic policy producing the reported decisions.
  • ORACLE importance weights w_l = not reported
    Eq. (15) assigns weights to primary/secondary/additional rubric items; task-specific weights are described as prespecified but not published, and they determine every open-ended score.
  • Short/long horizon balance λ = not reported
    Eq. (23) mixes short- and long-horizon distillation losses; no value is reported.
  • P3 quality quartile thresholds = 25th/75th percentiles of unassisted expert scores
    Low/medium/high surgical-planning categories are defined from the unassisted-condition distribution; these thresholds are data-derived and then applied to both conditions.
axioms (5)
  • domain assumption Timestamp filtering and node-specific masking prevent future or target information from entering evaluation prompts (Eq. 11).
    The benchmark's validity depends on this; similar-case retrieval may weaken it because historical cases carry diagnosis labels.
  • domain assumption ORACLE rubric-coverage scoring is a valid proxy for clinical quality and correlates with physician judgement.
    Used for all open-ended tasks; validated against physician rankings, but the sign of the reported Spearman correlation is unexplained.
  • domain assumption Patient-level separation between model development and OrthoBench test cohort prevents training/evaluation leakage.
    Asserted in Section 4.3.1; the actual split is only in Appendix E, not included here.
  • domain assumption Physician washout periods (≥4 weeks retrospective, ≥6 weeks prospective) eliminate case-recall bias in crossover reader studies.
    Repeated evaluation of identical cases may inflate assisted gains if recall persists.
  • standard math Standard paired statistical tests (McNemar, Wilcoxon, t-tests) are applied correctly to the reported crossovers and randomized groups.
    No full statistical analysis plan or raw data is available to verify.
invented entities (1)
  • CHEESE (32B clinical LLM and agent backbone) no independent evidence
    purpose: Core reasoning engine that decides tool calls and generates clinical recommendations.
    No weights, training data or independent evaluations are released; its capabilities are evidenced only by this paper's own benchmark and studies.

pith-pipeline@v1.3.0-alltime-deepseek · 32485 in / 15552 out tokens · 148327 ms · 2026-08-02T06:27:25.211654+00:00 · methodology

0 comments
read the original abstract

Musculoskeletal diseases are among the leading causes of disability and drive the greatest global need for rehabilitation. Because recovery, remodelling and degeneration of bones, joints and related tissues unfold over months to years, care requires longitudinal management rather than isolated decisions. Clinicians must repeatedly integrate evolving patient evidence, medical knowledge and stage-specific functional goals, yet evidence is often fragmented across visits, departments and hospital systems, disrupting continuous, individualised management. Here we report OrthoPilot, a clinical artificial intelligence (AI) system powered by a large language model (LLM) that integrates hospital data streams with authoritative external knowledge for continuous musculoskeletal care. It autonomously retrieves real-time imaging, laboratory, pathology and order data and translates evolving patient states into evidence-based decisions from admission diagnosis through rehabilitation planning. We established a specialist-validated benchmark from real-world electronic health records (EHRs) spanning 1,000 disease codes. In a full-pathway reader study against 81 orthopaedic physicians, OrthoPilot outperformed experts with 25 years of experience in diagnostic reasoning, clinical decision-making and management planning. This advantage generalised across 60 external clinical centres, where OrthoPilot surpassed all evaluated intelligent systems. In a prospective physician decision-making study of 1,870 complex cases, OrthoPilot improved full-chain management success by 10.6%. In a randomised deployment involving 8,240 inpatients, integration into routine care increased cumulative cases per bed by 9.7% and improved patient-reported access to health information. These results move clinical AI from predicting isolated events toward executing longitudinal management across complete musculoskeletal care pathways.

Figures

Figures reproduced from arXiv: 2607.12527 by Angela Lin Wang, Carlos Gutierrez Sanroman, Chenglong Ma, Chengyu Zhuang, Chenrun Wang, Fanrui Zhang, Gen Li, Hanyu Wang, Haoran Sun, Hongcheng Gao, Jiawei Liu, Jiyao Liu, Junjun He, Kailing Wang, Kaipeng Zhang, Lei Liu, Lei Wang, Lequan Yu, Lilong Wang, Lu Chen, Ming Hu, Renhao Yang, Shiyi Yao, Weijie Ma, Weiran Huang, Wenjie Li, Xiaosong Wang, Xingqi He, Yankai Jiang, Yidong Xu, Yilin Huang, Yinpeng Dong, Yin Zhang, Yirong Chen, Yuanfeng Ji, Yujie Zhang, Yu Qiao.

Figure 1
Figure 1. Figure 1: OrthoPilot: an evidence-grounded agent system for full-pathway musculoskeletal care. a, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: OrthoBench and ORACLE for full-pathway clinical evaluation. a, [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Aggregate benchmark performance, external generalisability and task-level performance of OrthoPilot. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Disease-level performance and human–AI comparative reader study. a, [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Prospective crossover reader study evaluating OrthoPilot in musculoskeletal clinical decision-making. a, [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Impact of OrthoPilot on clinical efficiency, workload and patient experience across the musculoskeletal care [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Hierarchical agent workflow and training curriculum of OrthoPilot. a, [PITH_FULL_IMAGE:figures/full_fig_p028_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Frontend workflow of OrthoPilot across the musculoskeletal care pathway. [PITH_FULL_IMAGE:figures/full_fig_p029_8.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 2 canonical work pages

  1. [1]

    Musculoskeletal health.https://www.who.int/news-room/fact-sheets/detail/m usculoskeletal-conditions

    World Health Organization. Musculoskeletal health.https://www.who.int/news-room/fact-sheets/detail/m usculoskeletal-conditions. Accessed 25 June 2026

    2026

  2. [2]

    Alarcos Cieza, Kate Causey, Kaloyan Kamenov, Sarah Wulf Hanson, Somnath Chatterji, and Theo Vos. Global estimates of the need for rehabilitation based on the global burden of disease study 2019: a systematic analysis for the global burden of disease study 2019.The Lancet, 396(10267):2006–2017, 2020

    2019

  3. [3]

    Fracture healing: mechanisms and interventions.Nature Reviews Rheumatology, 11(1):45–54, 2015

    Thomas A Einhorn and Louis C Gerstenfeld. Fracture healing: mechanisms and interventions.Nature Reviews Rheumatology, 11(1):45–54, 2015

    2015

  4. [4]

    Hunter and Sita M

    David J. Hunter and Sita M. A. Bierma-Zeinstra. Osteoarthritis.The Lancet, 393:1745–1759, 2019. doi: 10.1016/S014 0-6736(19)30417-9

    work page doi:10.1016/s014 2019

  5. [5]

    Continuity of care: a multidisciplinary review.Bmj, 327(7425):1219–1221, 2003

    Jeannie L Haggerty, Robert J Reid, George K Freeman, Barbara H Starfield, Carol E Adair, and Rachael McKendry. Continuity of care: a multidisciplinary review.Bmj, 327(7425):1219–1221, 2003

    2003

  6. [6]

    Continuity and coordination of care: a practice brief to support implementation of the who framework on integrated people-centred health services

    World Health Organization et al. Continuity and coordination of care: a practice brief to support implementation of the who framework on integrated people-centred health services. 2018

    2018

  7. [7]

    Carefragmentation,carecontinuity,andcarecoordination—how they differ and why it matters.JAMA internal medicine, 184(3):236–237, 2024

    LisaMKern,JuliePWBynum,andHaroldAlanPincus. Carefragmentation,carecontinuity,andcarecoordination—how they differ and why it matters.JAMA internal medicine, 184(3):236–237, 2024. 15

    2024

  8. [8]

    Manuela L Ferreira, Katie De Luca, Lydia M Haile, Jaimie D Steinmetz, Garland T Culbreth, Marita Cross, Jacek A Kopec, Paulo H Ferreira, Fiona M Blyth, Rachelle Buchbinder, et al. Global, regional, and national burden of low back pain, 1990–2020, its attributable risk factors, and projections to 2050: a systematic analysis of the global burden of disease ...

    1990

  9. [9]

    Jaimie D Steinmetz, Garland T Culbreth, Lydia M Haile, Quinn Rafferty, Justin Lo, Kai Glenn Fukutaki, Jessica A Cruz, Amanda E Smith, Stein Emil Vollset, Peter M Brooks, et al. Global, regional, and national burden of osteoarthritis, 1990–2020 and projections to 2050: a systematic analysis for the global burden of disease study 2021.The Lancet Rheumatolog...

    1990

  10. [10]

    Management reasoning: beyond the diagnosis.Jama, 319(22): 2267–2268, 2018

    David A Cook, Jonathan Sherbino, and Steven J Durning. Management reasoning: beyond the diagnosis.Jama, 319(22): 2267–2268, 2018

    2018

  11. [11]

    Management reasoning: empirical determination of key features and a conceptual model.Academic Medicine, 98(1):80–87, 2023

    David A Cook, Christopher R Stephenson, Larry D Gruppen, and Steven J Durning. Management reasoning: empirical determination of key features and a conceptual model.Academic Medicine, 98(1):80–87, 2023

    2023

  12. [12]

    On context specificity and management reasoning: moving beyond diagnosis.Diagnosis, 12(2):217–222, 2025

    James G Boyle, Matthew R Walters, Fiona M Burton, Catherine Paton, Martin Hughes, Susan Jamieson, and Steven J Durning. On context specificity and management reasoning: moving beyond diagnosis.Diagnosis, 12(2):217–222, 2025

    2025

  13. [13]

    Maher, and PeterP.B.O’Sullivan

    Ivan Lin, Louise Wiles, Rob Waller, Roger Goucke, Yusuf Nagree, Michael Gibberd, Leon Straker, Chris G. Maher, and PeterP.B.O’Sullivan. Whatdoesbestpracticecareformusculoskeletalpainlooklike?elevenconsistentrecommendations from high-quality clinical practice guidelines: systematic review.British Journal of Sports Medicine, 54(2):79–86, 2020. doi: 10.1136/...

    work page doi:10.1136/bjsports-2018-099878 2020

  14. [14]

    Coordinating care-a perilous journey through the health care system.New England Journal of Medicine, 358(10):1064, 2008

    Thomas Bodenheimer. Coordinating care-a perilous journey through the health care system.New England Journal of Medicine, 358(10):1064, 2008

    2008

  15. [15]

    Large language models encode clinical knowledge.Nature, 620(7972): 172–180, 2023

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge.Nature, 620(7972): 172–180, 2023

    2023

  16. [16]

    Towards conversational diagnostic artificial intelligence.Nature, 642(8067):442–450, 2025

    Tao Tu, Mike Schaekermann, Anil Palepu, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Yong Cheng, et al. Towards conversational diagnostic artificial intelligence.Nature, 642(8067):442–450, 2025

    2025

  17. [17]

    Towards accurate differential diagnosis with large language models.Nature, 642 (8067):451–457, 2025

    Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang, Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, et al. Towards accurate differential diagnosis with large language models.Nature, 642 (8067):451–457, 2025

    2025

  18. [18]

    An agentic system for rare disease diagnosis with traceable reasoning.Nature, pages 1–10, 2026

    Weike Zhao, Chaoyi Wu, Yanjie Fan, Pengcheng Qiu, Xiaoman Zhang, Yuze Sun, Xiao Zhou, Shuju Zhang, Yu Peng, Yanfeng Wang, et al. An agentic system for rare disease diagnosis with traceable reasoning.Nature, pages 1–10, 2026

    2026

  19. [19]

    Txagent:anaiagentfortherapeuticreasoningacrossauniverseoftools.arXiv preprint arXiv:2503.10970, 2025

    Shanghua Gao, Richard Zhu, Zhenglun Kong, Ayush Noori, Xiaorui Su, Curtis Ginder, Theodoros Tsiligkaridis, and MarinkaZitnik. Txagent:anaiagentfortherapeuticreasoningacrossauniverseoftools.arXiv preprint arXiv:2503.10970, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Evaluationandmitigationofthelimitationsoflargelanguage models in clinical decision-making.Nature medicine, 30(9):2613–2622, 2024

    Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, MarcusMakowski,RickmerBraren,GeorgiosKaissis,etal. Evaluationandmitigationofthelimitationsoflargelanguage models in clinical decision-making.Nature medicine, 30(9):2613–2622, 2024

    2024

  21. [21]

    Gpt-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial.Nature Medicine, 31(4):1233–1238, 2025

    Ethan Goh, Robert J Gallo, Eric Strong, Yingjie Weng, Hannah Kerman, Jason A Freed, Joséphine A Cool, Zahir Kanjee, Kathleen P Lane, Andrew S Parsons, et al. Gpt-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial.Nature Medicine, 31(4):1233–1238, 2025

    2025

  22. [22]

    Foundation models for generalist medical artificial intelligence.Nature, 616(7956):259–265, 2023

    Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence.Nature, 616(7956):259–265, 2023

    2023

  23. [23]

    Healthsystem-scalelanguagemodelsareall-purposeprediction engines.Nature, 619(7969):357–362, 2023

    Lavender Yao Jiang, Xujin Chris Liu, Nima Pour Nejatian, Mustafa Nasir-Moin, Duo Wang, Anas Abidin, Kevin Eaton, HowardAntonyRiina,IlyaLaufer,PaawanPunjabi,etal. Healthsystem-scalelanguagemodelsareall-purposeprediction engines.Nature, 619(7969):357–362, 2023. 16

    2023

  24. [24]

    Deepseek-r1 incentivizes reasoning in llms through reinforcement learning.Nature, 645(8081):633–638, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning.Nature, 645(8081):633–638, 2025

    2025

  25. [25]

    The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care.Nature medicine, 24(11):1716–1720, 2018

    Matthieu Komorowski, Leo A Celi, Omar Badawi, Anthony C Gordon, and A Aldo Faisal. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care.Nature medicine, 24(11):1716–1720, 2018

    2018

  26. [26]

    React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    Pith/arXiv arXiv 2022

  27. [27]

    The practical implementation of artificial intelligence technologies in medicine.Nature medicine, 25(1):30–36, 2019

    Jianxing He, Sally L Baxter, Jie Xu, Jiming Xu, Xingtao Zhou, and Kang Zhang. The practical implementation of artificial intelligence technologies in medicine.Nature medicine, 25(1):30–36, 2019

    2019

  28. [28]

    Guidelines for clinical trial protocols for interventions involving artificial intelligence: the spirit-ai extension.The Lancet Digital Health, 2(10):e549–e560, 2020

    SamanthaCruzRivera,XiaoxuanLiu,An-WenChan,AlastairKDenniston,MelanieJCalvert,HutanAshrafian,AndrewL Beam, Gary S Collins, Ara Darzi, Jonathan J Deeks, et al. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the spirit-ai extension.The Lancet Digital Health, 2(10):e549–e560, 2020

    2020

  29. [29]

    Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the consort-ai extension.The Lancet Digital Health, 2(10):e537–e548, 2020

    XiaoxuanLiu,SamanthaCruzRivera,DavidMoher,MelanieJCalvert,AlastairKDenniston,HutanAshrafian,AndrewL Beam, An-Wen Chan, Gary S Collins, Ara DarziJonathan J Deeks, et al. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the consort-ai extension.The Lancet Digital Health, 2(10):e537–e548, 2020

    2020

  30. [30]

    Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: Decide-ai.bmj, 377, 2022

    Baptiste Vasey, Myura Nagendran, Bruce Campbell, David A Clifton, Gary S Collins, Spiros Denaxas, Alastair K Denniston, Livia Faes, Bart Geerts, Mudathir Ibrahim, et al. Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: Decide-ai.bmj, 377, 2022

    2022

  31. [31]

    Randomised controlled trials evaluating artificial intelligence in clinical practice: a scoping review.The lancet digital health, 6(5): e367–e373, 2024

    Ryan Han, Julián N Acosta, Zahra Shakeri, John PA Ioannidis, Eric J Topol, and Pranav Rajpurkar. Randomised controlled trials evaluating artificial intelligence in clinical practice: a scoping review.The lancet digital health, 6(5): e367–e373, 2024

    2024

  32. [32]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Pith/arXiv arXiv 2009

  33. [33]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11(14):6421, 2021

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11(14):6421, 2021

    2021

  34. [34]

    Medxpertqa: Benchmarking expert-level medical reasoning and understanding.arXiv preprint arXiv:2501.18362, 2025

    Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. Medxpertqa: Benchmarking expert-level medical reasoning and understanding.arXiv preprint arXiv:2501.18362, 2025

    Pith/arXiv arXiv 2025

  35. [35]

    Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. Healthbench: Evaluating large language models towards improved human health.ArXiv, abs/2505.08775, 2025. URLhttps: //api.semanticscholar.org/CorpusI...

    Pith/arXiv arXiv 2025

  36. [36]

    7 Pérez Martínez Sara 8 9 10 Seedat Soraya 11 Urbaniok Frank 12 13 Woodbridge Jayson, Panel chairs Barkun Jeffrey 14 Grocott Michael PW 15 Hurst Samia 16 Meier Christoph A

    Anja Domenghino, Carmen Walbert, Dominique Lisa Birrer, Milo A Puhan, Pierre-Alain Clavien, Outcome4Medicine consensus group Jury Heuskel Dieter 4 Man Nancy Kwan 5 Monteiro João 6 Perellon Juan F. 7 Pérez Martínez Sara 8 9 10 Seedat Soraya 11 Urbaniok Frank 12 13 Woodbridge Jayson, Panel chairs Barkun Jeffrey 14 Grocott Michael PW 15 Hurst Samia 16 Meier ...

    2023

  37. [37]

    Artificial intelligence in surgery.Nature medicine, 30(5):1257–1268, 2024

    Chris Varghese, Ewen M Harrison, Greg O’Grady, and Eric J Topol. Artificial intelligence in surgery.Nature medicine, 30(5):1257–1268, 2024

    2024

  38. [38]

    Perioperative interventions for prevention of postoperative pulmonary complications: systematic review and meta-analysis.bmj, 368, 2020

    Peter M Odor, Sohail Bampoe, David Gilhooly, Benedict Creagh-Brown, and S Ramani Moonesinghe. Perioperative interventions for prevention of postoperative pulmonary complications: systematic review and meta-analysis.bmj, 368, 2020. 17

    2020

  39. [39]

    Mapping the discharge process after surgery.JAMA surgery, 159(4):438–444, 2024

    LauraAGraham,SamanthaIllarmo,CarolinePGray,AlexHSHarris,ToddHWagner,MaryTHawn,JamesCIannuzzi, and Sherry M Wren. Mapping the discharge process after surgery.JAMA surgery, 159(4):438–444, 2024

    2024

  40. [40]

    The role of multidisciplinary team comanagement of the surgical patient: it takes a village.JAMA Network Open, 3(5):e204354, 2020

    Aditi Kashikar and Shipra Arya. The role of multidisciplinary team comanagement of the surgical patient: it takes a village.JAMA Network Open, 3(5):e204354, 2020. doi: 10.1001/jamanetworkopen.2020.4354

    arXiv 2020

  41. [41]

    Toward expert-level medical question answering with large language models.Nature medicine, 31(3):943–950, 2025

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. Toward expert-level medical question answering with large language models.Nature medicine, 31(3):943–950, 2025

    2025

  42. [42]

    Capabilities of gemini models in medicine.arXiv preprint arXiv:2404.18416, 2024

    KhaledSaab,TaoTu,Wei-HungWeng,RyutaroTanno,DavidStutz,ElleryWulczyn,FanZhang,TimStrother,Chunjong Park, Elahe Vedadi, et al. Capabilities of gemini models in medicine.arXiv preprint arXiv:2404.18416, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Mdagents:Anadaptivecollaborationofllmsformedicaldecision-making.Advances in Neural Information Processing Systems, 37:79410–79452, 2024

    Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik S Chan, Xuhai Xu, Daniel McDuff, Hyeonhoon Lee, Marzyeh Ghassemi, CynthiaBreazeal,andHaeWPark. Mdagents:Anadaptivecollaborationofllmsformedicaldecision-making.Advances in Neural Information Processing Systems, 37:79410–79452, 2024

    2024

  44. [44]

    Sutton, David Pincock, Daniel C

    Reed T. Sutton, David Pincock, Daniel C. Baumgart, Daniel C. Sadowski, Richard N. Fedorak, and Karen I. Kroeker. An overview of clinical decision support systems: benefits, risks, and strategies for success.npj Digital Medicine, 3:17,

  45. [45]

    Collaborative strategies for deploying ai-based physician decision support systems: challenges and deployment approaches.NPJ Digital Medicine, 6(1):137, 2023

    Mirja Mittermaier, Marium Raza, and Joseph C Kvedar. Collaborative strategies for deploying ai-based physician decision support systems: challenges and deployment approaches.NPJ Digital Medicine, 6(1):137, 2023

    2023

  46. [46]

    A clinically applicable approach to continuous prediction of future acute kidney injury.Nature, 572(7767):116–119, 2019

    Nenad Tomašev, Xavier Glorot, Jack W Rae, Michal Zielinski, Harry Askham, Andre Saraiva, Anne Mottram, Clemens Meyer, Suman Ravuri, Ivan Protsyuk, et al. A clinically applicable approach to continuous prediction of future acute kidney injury.Nature, 572(7767):116–119, 2019

    2019

  47. [47]

    Artificial intelligence in surgery: promises and perils.Annals of surgery, 268(1):70–76, 2018

    Daniel A Hashimoto, Guy Rosman, Daniela Rus, and Ozanan R Meireles. Artificial intelligence in surgery: promises and perils.Annals of surgery, 268(1):70–76, 2018

    2018

  48. [48]

    Probabilistic medical predictions of large language models

    Bowen Gu, Rishi J Desai, Kueiyu Joshua Lin, and Jie Yang. Probabilistic medical predictions of large language models. npj Digital Medicine, 7(1):367, 2024. 18 4 Methods 4.1 Problem Formulation 4.1.1 Full-pathway task definition We formulate musculoskeletal care reasoning as a longitudinal sequence of clinically grounded evaluation nodes distributed across...

    2024

  49. [50]

    Benchmarking retrieval-augmented generation for medicine

    Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking retrieval-augmented generation for medicine. InFindings of the Association for Computational Linguistics: ACL 2024, pages 6233–6251, 2024

    2024

  50. [51]

    Robertson and Hugo Zaragoza

    Stephen E. Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond.Found. Trends Inf. Retr., 3:333–389, 2009. URLhttps://api.semanticscholar.org/CorpusID:207178704

    2009

  51. [52]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024. Acknowledgements This work was supported by the National Key Research and Development Program of China (2023YF...

    Pith/arXiv arXiv 2024

  52. [2020]

    doi: 10.1038/s41746-020-0221-y

    doi:10.1038/s41746-020-0221-y