Pith. sign in

REVIEW 5 major objections 5 minor 90 references

Patient-facing health AI agents, tested in realistic tool-using conversations, still fall short: triage pass rates range from 32% to 88%, and the strongest model scores 4.25 out of 5.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:16 UTC pith:ORCB2TY2

load-bearing objection A genuinely useful benchmark for patient-facing health agents, but the headline gap numbers rest on an under-validated LLM jury and need revision before I'd trust the magnitudes. the 5 major comments →

arxiv 2607.25485 v1 pith:ORCB2TY2 submitted 2026-07-28 cs.AI cs.CL

PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents

Korosh Vatanparvar , Ashutosh Joshi , Maria Xenochristou , Mohammad Abuzar Hashemi , Prasad Kasu , Deepak Bansal , Daniel Lopez-Martinez , Anchal Nema
show 6 more authors
Ramya Ganesan Will Kimbrough Alex Woody Yadunandana Rao Dilek Hakkani-Tur Wilko Schulz-Mahlendorf
This is my paper
classification cs.AI cs.CL
keywords patient-facing health AIagentic AI benchmarkLLM-as-a-Juryclinical safetytriage qualityhealthcare workflow automationsimulated patientclinical gap analysis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

PatientAgentBench sets out to evaluate AI agents that talk to patients and act on their behalf—booking appointments, refilling prescriptions, triaging symptoms—rather than merely answering medical questions. Using 1,200 generated patient scenarios, 10 models from four families, and 12,000 simulated conversations, it claims to surface clinically meaningful gaps in patient-facing agentic AI, with an LLM-as-a-Jury that aligns with licensed clinicians at 79–93% adjacent agreement. Triage quality is the most discriminating dimension, with pass rates from 32% to 88%; agents often act on administrative requests without clinical screening; and even the strongest model scores only 4.25 out of 5 overall. The takeaway is that current frontier models, even inside a shared baseline harness, are not yet safe to autonomously triage or act for patients without engineered safeguards, and static Q&A benchmarks cannot see these failures.

Core claim

The paper's central claim is that evaluating a patient-facing health AI agent requires watching it do sustained, multi-turn, tool-using work against a realistic patient record, and that on such a benchmark every current model falls short in clinically meaningful ways. On 1,200 scenarios, the strongest model earns 4.25/5 overall, with a triage pass rate of 88%; the weakest models pass triage only 32% of the time. The failures cluster into recognizable patterns: agents acting on administrative requests before any clinical screening, fabricating tool executions they never performed, issuing prescriptive medication directives without provider caveats, and omitting crisis resources during mental

What carries the argument

The load-bearing machinery is PatientAgentBench's three-phase pipeline: distribution-controlled seed selection plus LLM-based scenario enrichment produces the 1,200 patient records with comorbidities and polypharmacy; a dual-agent loop pairs a simulated patient (persona-driven, no tools) with the model under evaluation running a ReAct (reason-then-act) agent over a stateful sandbox of 15 healthcare tools—appointments, prescriptions, telehealth, profile management—that persists data across turns; and each conversation trace, including internal tool calls and results, is scored by an LLM-as-a-Jury of two frontier models against 102 conversation-agnostic, clinician-grounded criteria across six

Load-bearing premise

The paper's central numbers rest on the assumption that the automated judge's scores on all 12,000 conversations are as reliable as the scores licensed clinicians gave on the 75 conversations used for validation; if the automated judge is biased on the unannotated conversations—especially on clinical safety, where agreement was lowest—the reported gap sizes and rankings could change.

What would settle it

Have licensed clinicians score a fresh stratified sample of, say, 300 conversations drawn from the same 12,000 (none used in the 75-conversation validation), under the same six rubrics, and compare exact scores with the jury's; if exact agreement on clinical safety is materially below the reported 79% adjacent agreement, or if the jury's conservative bias reverses on the unannotated conversations, the headline triage and safety gaps are not established.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Triage is the most discriminating dimension: pass rates span 32–88%, and even frontier models pass only 76–88%, so autonomous triage is not yet safe without engineered safeguards.
  • Agents often process administrative requests without clinical screening, which for complex patients (polypharmacy, comorbidities) risks missing drug interactions and worsening conditions.
  • The weakest models fabricate unexecuted tool actions and omit crisis resources; frontier models fail on 1–3% of cases via unverified tool outputs and missing hotlines in emergencies.
  • A clinically grounded agent design around the same base model raised aggregate scores by 0.21 to near-99% pass rates, showing that part of the gap is agent design rather than model capability alone.
  • Static medical Q&A benchmarks are insufficient: the failures surface only in sustained, tool-using conversations against realistic patient records.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the LLM-as-a-Jury was validated on only 75 conversations (about 500 rubric scores); its scores on the other roughly 12,000 conversations could carry bias that the reported 79% adjacent agreement on clinical safety does not bound, so an independent clinician audit on fresh conversations is the natural next check.
  • Editorial inference: the framework's case-agnostic rubrics and stateful sandbox generalize beyond primary care; the same machinery could evaluate agents in mental health crisis lines, chronic care navigation, or pharmacy workflows by reweighting scenarios and adding tools.
  • Editorial inference: the 'hidden complexity' pattern—routine administrative requests from complex patients are where safety fails—suggests deployment guards should make pre-action clinical screening a deterministic rule, not a model's in-the-moment choice.
  • Editorial inference: a deployment gate that tracks triage quality on administrative requests separately would be a better safety signal than the aggregate score, since the aggregate masks the frontier's 76–88% triage pass rate.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces PatientAgentBench, a benchmark framework for evaluating patient-facing agentic health AI. It generates synthetic patient scenarios from configurable seeds, runs multi-turn dual-agent conversations between a simulated patient and a tool-using assistant in a stateful healthcare sandbox, and scores each conversation with an LLM-as-a-Jury across six clinician-grounded rubric dimensions (task completion, clinical safety, workflow accuracy, triage quality, clinical helpfulness, conversational quality). Ten foundation models are evaluated on 1,200 scenarios (12,000 conversations). The headline results are that triage quality is the most discriminating dimension (pass rates 32–88%), frontier models reach only 4.25/5 aggregate, and recurring failures include crisis-resource omission, fabricated tool execution, and insufficient pre-action clinical screening. A clinician annotation study on 75 conversations reports 79–93% adjacent agreement between the LLM jury and clinicians, together with a systematic conservative jury bias on safety-critical dimensions.

Significance. If the evaluation is valid, the paper fills a real gap: there is currently no benchmark that jointly evaluates patient-facing multi-turn conversation, tool use, and clinically meaningful scoring. The strengths are substantial: the framework is open-source and reproducible; the rubrics are conversation-agnostic and grounded in clinician input; synthetic patient profiles and conversations were rated realistic by clinicians; all models share a common baseline harness; and the disaggregated analyses by personality, complexity, severity, demographics, and task category are valuable. The Appendix A case study, showing that agent design changes can improve scores while holding the base model fixed, is a useful demonstration of the benchmark's diagnostic utility. However, the central quantitative claims — the size of the cross-model gaps and the 'residual gap' at the frontier — rest on an LLM-as-a-Jury whose validation is currently thin: only 75 of 12,000 conversations, adjacent rather than exact agreement, no per-model/per-tier agreement, and a jury drawn from the same model families as several evaluated systems.

major comments (5)
  1. [§4.2/§5.5] The LLMJ validation is not sufficient to support the quantitative residual-gap claims. The validation set is 75 conversations (~500 rubric scores) out of 12,000, and agreement is reported only as adjacent ±1 (79–93%); exact agreement is not reported. The systematic conservative biases (−0.95 on clinical safety, −0.67 on workflow accuracy, −0.52 on triage) are large relative to the inter-model differences reported in Table 4. Because the paper's central claim is that stronger models 'narrow but do not close' the gap, the paper must show that jury–clinician agreement holds per model or per tier, and that exact disagreement does not alter rankings. Please report exact agreement, per-tier agreement, and a robustness check, e.g., a clinician-scored held-out set or a per-evaluator bias analysis.
  2. [§4.1/§6.3] The jury comprises Claude Opus 4.8 and GPT-5.5, the same families as several evaluated models. The paper acknowledges self-preference in Section 6.3 but does not test for it. Since per-evaluator scores are preserved, it is feasible and necessary to report whether each evaluator scores same-family models more favorably. Without such an analysis, the reported frontier-tier separation and the size of the cross-model gaps could be partly an artifact of evaluator model-family overlap. This is a correctness risk, not merely a caveat.
  3. [§3.3] The simulated patient ('user agent') model is not disclosed. All conversations, and therefore all scores, depend on the behavior of this model. Reproducibility and fair cross-model comparison require identifying the model and version used for patient simulation. If the same model family is used for patient simulation and for evaluation, this is another channel for family-specific bias that should be discussed and, if possible, controlled.
  4. [Abstract/§7] The claim that the results 'confirm that static benchmarks are insufficient' is not directly tested. The paper demonstrates that this agentic benchmark surfaces failures, but there is no paired comparison showing that the same models perform better on static QA or single-turn tasks. Please either add such a comparison or soften the claim from 'confirming' to 'motivating' or 'suggesting'.
  5. [§4] The default-to-4 rule for dimensions with little to exercise may systematically inflate pass rates and compress differences, particularly for task completion and clinical helpfulness, where pass rates are near-universal. The paper should report how often this default was used per dimension and per model, and analyze sensitivity to this rule (e.g., treating the dimension as missing rather than scoring 4). This affects the conclusion that these dimensions are 'least discriminating.'
minor comments (5)
  1. [§5.5] The comparison between LLMJ–human and human–human agreement mixes metrics: LLMJ is compared to averaged clinician scores, while clinician inter-annotator agreement is pairwise. Please report both measures using the same metric (e.g., exact and adjacent agreement, and weighted kappa) so the comparison is apples-to-apples.
  2. [Figure 16] The scatter plot would be easier to interpret if the exact agreement percentages were reported numerically alongside the adjacent agreement percentages for each dimension.
  3. [§3.1/§5.4] The 'infeasible' scenarios are deliberately retained, and the safety degradation on them is interesting. Please report how many of these scenarios were included in the 75-conversation clinician annotation sample, since this affects the validity of the LLMJ scores on that subgroup.
  4. [Table 4] Pass rates are reported as whole percentages with no uncertainty. With N=1,200 per model, a 95% CI for a proportion would add useful precision, especially for the triage pass rates in the 32–88% range.
  5. [§4.1] Minor formatting: the denominator notation 'P wi = 8.3' should be written as 'Σ wi = 8.3' for clarity. Also, the abstract's phrase 'on par with or exceeding clinician inter-rater agreement' should be qualified as 'adjacent agreement' to avoid overstatement.

Circularity Check

0 steps flagged

No significant circularity: the benchmark's evaluations are anchored by external clinician annotation, and the acknowledged evaluator-family overlap is a validity concern, not a constructional reduction of the results to their inputs.

full rationale

Walking the derivation chain, PatientAgentBench does not derive its headline results from its own inputs by construction. Scenario generation, conversation collection, and scoring are separate phases: seed tuples are sampled from a configurable distribution, enriched by Claude Sonnet 4.6 into patient profiles, and conversations are produced by a dual-agent loop with a shared ReAct harness; scores are given by a two-model LLM-as-a-Jury using 102 conversation-agnostic criteria. The aggregate score is a transparent weighted formula (Eq. 1) with clinician-set weights, not a fitted parameter. The key external anchor is the clinician annotation study (Sections 4.2 and 5.5): 75 conversations, ~500 rubric scores, 79–93% adjacent agreement. This is independent evidence, even if limited and reported only as adjacent agreement. The paper explicitly flags the evaluator-family overlap in Section 6.3: 'Because the benchmark scenarios and the simulated patient’s responses are generated by a Claude model, and models from the same families as the evaluators are among the models under evaluation, self-preference bias is a concern.' That is an honest validity limitation, not a circular derivation: the scores are not defined in terms of the evaluated models, and no equation or construction makes a model's score equal to the evaluator's own output. Likewise, the CGA case study uses the same rubrics to measure an intervention, but the comparison is between two agent configurations under a fixed evaluation, so it does not reduce to the rubric definition. There are no load-bearing self-citations, no imported uniqueness theorems, and no fitted quantity renamed as a prediction. Weaknesses in LLMJ validation, exact-agreement reporting, and per-tier bias analysis are correctness and robustness concerns, but they do not constitute circularity under the defined criteria.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The benchmark is a measurement framework, not a derivation, so there are no fitted physical parameters. The main free parameters are the rubric weights, pass threshold, the default-to-4 rule, and the configurable seed distribution; each directly shapes the reported aggregate scores and pass rates. The key assumptions are the generalizability of LLMJ validation and the fidelity of the synthetic patient and tool sandbox.

free parameters (4)
  • Rubric weights w_i = safety 2.0, workflow 1.6, triage 1.4, helpfulness 1.4, task completion 1.0, conversational quality 0.9
    Hand-selected via clinician discussion; the aggregate scores and model rankings depend on these weights, and the paper notes different priorities could change rankings.
  • Pass threshold = 3.0 on 1–5 scale
    Chosen by clinician assessment of minimum acceptable standard; pass rates are a headline result and directly depend on this threshold.
  • Default score for non-applicable dimensions = 4
    When a dimension is considered not exercised, a score of 4 is assigned; this inflates average scores for many administrative conversations.
  • Seed distribution weights = 67% complicated, 30% severe, 15% pediatric, etc. (Table 2)
    Deliberate oversampling of clinically demanding cases; all reported model gap sizes depend on this configurable distribution.
axioms (4)
  • domain assumption LLMJ scores on all 12,000 conversations generalize from the 75-conversation clinician alignment sample
    The entire reporting of model scores rests on the assumption that the LLM jury's agreement with clinicians (79–93% adjacent) extends to unannotated conversations, including those with different failure modes.
  • domain assumption Simulated patient behavior is a valid proxy for real patient interactions
    The benchmark generates conversations via an LLM user agent; high clinician realism ratings for a sample do not guarantee validity for safety-critical behaviors like triage and crisis detection.
  • domain assumption The tool sandbox accurately emulates real-world healthcare workflows and error conditions
    Workflow accuracy and fabrication findings depend on sandbox fidelity: if real EHRs have different failure modes, the measured gaps may not transfer.
  • domain assumption The 102 clinician-grounded criteria form a valid and complete operationalization of clinical safety and quality
    Rubric validity is supported by clinician involvement but not independently established; the criteria may omit important patient-facing risks.

pith-pipeline@v1.3.0-alltime-deepseek · 43606 in / 11079 out tokens · 116143 ms · 2026-08-01T02:16:26.775671+00:00 · methodology

0 comments
read the original abstract

Health AI is evolving from answering questions to agentic systems that converse with patients, reason about health records, and act on their behalf. Primary care guards against diagnostic errors and unsafe care; agents assisting in this domain warrant evaluation against the same risks. Current benchmarks focus on medical knowledge, assessed through isolated question-answering or clinician-facing tasks. PatientAgentBench benchmarks patient-facing agentic healthcare; it evaluates a foundation model, wrapped in an agent with a sandbox of healthcare tools, conversing with a simulated patient. Each conversation is scored by an LLM-as-a-Jury across six dimensions via over a hundred conversation-agnostic, clinician-grounded criteria. To validate alignment, licensed clinicians annotated shared conversations, yielding 79-93% adjacent agreement between jury and expert raters, on par with or exceeding clinician inter-rater agreement. We benchmarked 10 models across four families on the same 1,200 scenarios and found clinical gaps. Triage quality is the most discriminating dimension: pass rates rise from 32% for the weakest models to 88% for the strongest, with agents often acting on administrative requests without clinical screening. Clinical safety and workflow accuracy follow the same pattern: the weakest models fail often, fabricating unexecuted actions, while frontier models fail on only 1-3% of cases, from unverified tool outputs and omitted crisis resources in an emergency. More capable models narrow these gaps but do not close them; the strongest scores only 4.25 of 5 overall. These failures surface only in sustained, tool-using conversations against realistic patient records, confirming that static benchmarks are insufficient as healthcare agentic systems gain autonomy. We release the framework as a reproducible, clinician-validated evaluation standard to help the field close this gap.

Figures

Figures reproduced from arXiv: 2607.25485 by Alex Woody, Anchal Nema, Ashutosh Joshi, Daniel Lopez-Martinez, Deepak Bansal, Dilek Hakkani-Tur, Korosh Vatanparvar, Maria Xenochristou, Mohammad Abuzar Hashemi, Prasad Kasu, Ramya Ganesan, Wilko Schulz-Mahlendorf, Will Kimbrough, Yadunandana Rao.

Figure 1
Figure 1. Figure 1: Two hypothetical agents receive the same input but produce different clinical outcomes. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: End-to-end PatientAgentBench architecture. The three-phase pipeline generates a benchmark case from a [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Dual-agent conversation loop architecture showing benchmark entry inputs, the conversation loop with ReAct [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: LLM-as-a-Jury evaluation framework. Each of [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Aggregate score distributions and per-rubric pass rates across the 12 health agents. Each row shows one [PITH_FULL_IMAGE:figures/full_fig_p026_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Triage quality by patient personality type across the 12 health agents. Skeptical and terse patients, who [PITH_FULL_IMAGE:figures/full_fig_p027_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Clinical safety by patient personality type across the 12 health agents. The Qwen3-Next-80B agent shows a [PITH_FULL_IMAGE:figures/full_fig_p027_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Conversational quality by patient personality type across the 12 health agents. Anxious patients receive the [PITH_FULL_IMAGE:figures/full_fig_p027_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Clinical safety by scenario complexity across the 12 health agents. Infeasible scenarios, those with a [PITH_FULL_IMAGE:figures/full_fig_p028_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Aggregate score by scenario complexity across the 12 health agents. Top-tier agents perform [PITH_FULL_IMAGE:figures/full_fig_p028_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Aggregate score by condition severity level across the 12 health agents. A severity paradox emerges: [PITH_FULL_IMAGE:figures/full_fig_p028_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Clinical safety by patient age group across the 12 health agents. Most agents score lowest on senior or [PITH_FULL_IMAGE:figures/full_fig_p029_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Clinical safety by patient gender identity across the 12 health agents. Several agents score lowest on [PITH_FULL_IMAGE:figures/full_fig_p029_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Clinical helpfulness by patient gender identity across the 12 health agents. Most agents score lowest on a [PITH_FULL_IMAGE:figures/full_fig_p029_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Aggregate score by task category across the 12 health agents. Profile management, the purely administrative [PITH_FULL_IMAGE:figures/full_fig_p030_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: LLMJ vs. averaged licensed-clinician scores per rubric dimension. Each point plots one conversation’s [PITH_FULL_IMAGE:figures/full_fig_p030_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

90 extracted references · 12 linked inside Pith

  1. [1]

    2001 , publisher =

    Crossing the Quality Chasm: A New Health System for the 21st Century , author =. 2001 , publisher =

    2001

  2. [2]

    2021 , url =

    Global Patient Safety Action Plan 2021--2030: Towards Eliminating Avoidable Harm in Health Care , author =. 2021 , url =

    2021

  3. [3]

    2022 , note =

    Patient Safety Assessment Manual for Primary Care , author =. 2022 , note =

    2022

  4. [4]

    Technical Series on Safer Primary Care , author =

  5. [5]

    BMJ Quality & Safety , volume =

    The global burden of diagnostic errors in primary care , author =. BMJ Quality & Safety , volume =. 2017 , doi =

    2017

  6. [6]

    2023 , howpublished =

    Patient Safety , author =. 2023 , howpublished =

    2023

  7. [7]

    2015 , publisher =

    Improving Diagnosis in Health Care , author =. 2015 , publisher =

    2015

  8. [8]

    arXiv preprint arXiv:2604.27470 , year=

    HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats , author=. arXiv preprint arXiv:2604.27470 , year=

    Pith/arXiv arXiv

  9. [9]

    Liu, Qianchu and Zhang, Sheng and Qin, Guanghui and others , journal=

  10. [10]

    2024 , url=

    Agentic Healthcare Communications (AHC): The Rise of Intelligent Healthcare Automation , author=. 2024 , url=

    2024

  11. [11]

    , journal=

    Shehab, Mohammed A. , journal=. Agentic-AI Healthcare: Multilingual, Privacy-First Framework with. 2025 , doi=

    2025

  12. [12]

    To Err Is Human: Building a Safer Health System , author=

  13. [13]

    npj Digital Medicine , volume=

    The Potential of Artificial Intelligence to Improve Patient Safety: A Scoping Review , author=. npj Digital Medicine , volume=. 2021 , doi=

    2021

  14. [14]

    Nature Biomedical Engineering , year=

    Synthetic Data in Machine Learning for Medicine and Healthcare , author=. Nature Biomedical Engineering , year=

  15. [15]

    NPJ Digital Medicine , year=

    Generating High-Fidelity Synthetic Patient Data for Assessing Machine Learning Healthcare Software , author=. NPJ Digital Medicine , year=

  16. [16]

    Zheng, Lianmin and others , booktitle=. Judging

  17. [17]

    AlpacaEval: An Automatic Evaluator of Instruction-following Models , author=

  18. [18]

    Doctors Talking with Patients/Patients Talking with Doctors: Improving Communication in Medical Visits , author=

  19. [19]

    Patient Education and Counseling , year=

    How Does Communication Heal? Pathways Linking Clinician-Patient Communication to Health Outcomes , author=. Patient Education and Counseling , year=

  20. [20]

    BMJ , year=

    Evaluation of Symptom Checkers for Self Diagnosis and Triage: Audit Study , author=. BMJ , year=

  21. [21]

    Nature Medicine , year=

    High-Performance Medicine: The Convergence of Human and Artificial Intelligence , author=. Nature Medicine , year=

  22. [22]

    Artificial Intelligence Review , volume=

    Large language models in medical and healthcare fields: applications, advances, and challenges , author=. Artificial Intelligence Review , volume=. 2024 , doi=

    2024

  23. [23]

    JAMA , year=

    Big Data and Machine Learning in Health Care , author=. JAMA , year=

  24. [24]

    Nature , year=

    Foundation Models for Generalist Medical Artificial Intelligence , author=. Nature , year=

  25. [25]

    NEJM AI , year=

    Towards Generalist Biomedical AI , author=. NEJM AI , year=

  26. [26]

    Science , year=

    Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations , author=. Science , year=

  27. [27]

    The Lancet Digital Health , year=

    AI Recognition of Patient Race in Medical Imaging: A Modelling Study , author=. The Lancet Digital Health , year=

  28. [28]

    NPJ Digital Medicine , year=

    Scalable and Accurate Deep Learning with Electronic Health Records , author=. NPJ Digital Medicine , year=

  29. [29]

    Journal of the American Medical Informatics Association , year=

    Opportunities and Challenges in Developing Risk Prediction Models with Electronic Health Records Data , author=. Journal of the American Medical Informatics Association , year=

  30. [30]

    Health Affairs , volume=

    Developing Physician Communication Skills for Patient-Centered Care , author=. Health Affairs , volume=. 2010 , doi=

    2010

  31. [31]

    2007 , doi=

    Preventing Medication Errors , author=. 2007 , doi=

    2007

  32. [32]

    arXiv preprint arXiv:2311.16452 , year=

    Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine , author=. arXiv preprint arXiv:2311.16452 , year=

    Pith/arXiv arXiv

  33. [33]

    Nature Medicine , volume=

    Large Language Models in Medicine , author=. Nature Medicine , volume=. 2023 , doi=

    2023

  34. [34]

    Chase, Harrison , year=

  35. [35]

    2025 , url=

    The Claude Model Family: Claude Opus 4.5, Claude Sonnet 4.5, and Claude Haiku 4.5 , author=. 2025 , url=

    2025

  36. [36]

    2024 , url=

    Amazon Nova Foundation Models: Nova Premier, Nova Pro, and Nova Lite , author=. 2024 , url=

    2024

  37. [37]

    2025 , eprint=

    ^2 -Bench: Evaluating Conversational Agents in a Dual-Control Environment , author=. 2025 , eprint=

    2025

  38. [38]

    Preference Leakage: A Contamination Problem in

    Dawei Li and Renliang Sun and Yue Huang and Ming Zhong and Bohan Jiang and Jiawei Han and Xiangliang Zhang and Wei Wang and Huan Liu , year=. Preference Leakage: A Contamination Problem in. 2502.01534 , archivePrefix=

    arXiv

  39. [39]

    Self-Preference Bias in

    Wataoka, Koki and Takahashi, Tsubasa and Ri, Ryokan , year=. Self-Preference Bias in. 2410.21819 , archivePrefix=

    Pith/arXiv arXiv

  40. [40]

    and Feng, Shi , year=

    Panickssery, Arjun and Bowman, Samuel R. and Feng, Shi , year=. 2404.13076 , archivePrefix=

    Pith/arXiv arXiv

  41. [41]

    International Conference on Learning Representations (ICLR) , year=

    ReAct: Synergizing Reasoning and Acting in Language Models , author=. International Conference on Learning Representations (ICLR) , year=

  42. [42]

    arXiv preprint arXiv:2409.07314 , year=

    Kanithi, Praveenkumar and Christophe, Cl\'. arXiv preprint arXiv:2409.07314 , year=

    Pith/arXiv arXiv

  43. [43]

    Scientific Data , volume=

    MIMIC-III, a Freely Accessible Critical Care Database , author=. Scientific Data , volume=. 2016 , publisher=

    2016

  44. [44]

    Johnson, Alistair E. W. and Bulgarelli, Lucas and Shen, Lu and Gayles, Alvin and Shammout, Ayad and Horng, Steven and Pollard, Tom J. and Hao, Sicheng and Moody, Benjamin and Gow, Brian and others , journal=. 2023 , publisher=

    2023

  45. [45]

    Thomas and Jessica M

    Shuyue Stella Li and Jimin Mun and Faeze Brahman and Pedram Hosseini and Bryceton G. Thomas and Jessica M. Sin and Bing Ren and Jonathan S. Ilgen and Yulia Tsvetkov and Maarten Sap , year=. 2502.14860 , archivePrefix=

    Pith/arXiv arXiv

  46. [46]

    Applied Sciences , volume=

    What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams , author=. Applied Sciences , volume=. 2021 , publisher=

    2021

  47. [47]

    International Conference on Learning Representations (ICLR) , year=

    WebArena: A Realistic Web Environment for Building Autonomous Agents , author=. International Conference on Learning Representations (ICLR) , year=

  48. [48]

    International Conference on Learning Representations (ICLR) , year=

    SWE-bench: Can Language Models Resolve Real-World GitHub Issues? , author=. International Conference on Learning Representations (ICLR) , year=

  49. [49]

    PARADISE : a framework for evaluating spoken dialogue agents

    Walker, Marilyn A and Litman, Diane J and Kamm, Candace A and Abella, Alicia. PARADISE : a framework for evaluating spoken dialogue agents. Proceedings of the 35th annual meeting on Association for Computational Linguistics -

  50. [50]

    Towards an automatic Turing test: Learning to evaluate dialogue responses

    Lowe, Ryan and Noseworthy, Michael and Serban, Iulian Vlad and Angelard-Gontier, Nicolas and Bengio, Yoshua and Pineau, Joelle. Towards an automatic Turing test: Learning to evaluate dialogue responses. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

  51. [51]

    Proceedings of the 42nd International Conference on Machine Learning (ICML) , year =

    MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding , author =. Proceedings of the 42nd International Conference on Machine Learning (ICML) , year =. 2501.18362 , archivePrefix =

    Pith/arXiv arXiv

  52. [52]

    Proceedings of the 31st International Conference on Computational Linguistics (COLING) , pages =

    AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator , author =. Proceedings of the 31st International Conference on Computational Linguistics (COLING) , pages =

  53. [53]

    arXiv preprint arXiv:2601.03023 , year =

    MedDialogRubrics: A Comprehensive Benchmark and Evaluation Framework for Multi-turn Medical Consultations in Large Language Models , author =. arXiv preprint arXiv:2601.03023 , year =

    arXiv

  54. [54]

    and Geng, Gloria and Park, Danny and Zou, James and Ng, Andrew Y

    Jiang, Yixing and Black, Kameron C. and Geng, Gloria and Park, Danny and Zou, James and Ng, Andrew Y. and Chen, Jonathan H. , journal =. MedAgentBench: A Virtual. 2025 , publisher =

    2025

  55. [55]

    Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track , volume =

    MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models , author =. Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track , volume =. 2024 , eprint =

    2024

  56. [56]

    and He, Junjun and Qiao, Yu , booktitle =

    Chen, Pengcheng and Ye, Jin and Wang, Guoan and Li, Yanjun and Deng, Zhongying and Li, Wei and Li, Tianbin and Duan, Haodong and Huang, Ziyan and Su, Yanzhou and Wang, Benyou and Zhang, Shaoting and Fu, Bin and Cai, Jianfei and Zhuang, Bohan and Seibel, Eric J. and He, Junjun and Qiao, Yu , booktitle =. 2024 , eprint =

    2024

  57. [57]

    ClinicalBench: Can

    Chen, Canyu and Yu, Jian and Chen, Shan and Liu, Che and Wan, Zhongwei and Zhou, Shuang and Luo, Yuan and Zhang, Rui and Bitterman, Danielle and Wang, Fei and Shu, Kai , journal =. ClinicalBench: Can

  58. [58]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages =

    3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark , author =. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages =

    2025

  59. [59]

    International Conference on Learning Representations (ICLR) , year =

    -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains , author =. International Conference on Learning Representations (ICLR) , year =. 2406.12045 , archivePrefix =

    Pith/arXiv arXiv

  60. [60]

    Replacing Judges with Juries: Evaluating

    Verga, Pat and Hofst. Replacing Judges with Juries: Evaluating. arXiv preprint arXiv:2404.18796 , year =

    Pith/arXiv arXiv

  61. [61]

    Schmidgall, Samuel and Ziaei, Rojin and Harris, Carl and Reis, Eduardo and Jopling, Jeffrey and Moor, Michael , journal =

  62. [62]

    and Mao, Huanzhi and Yan, Fanjia and Ji, Charlie Cheng-Jie and Suresh, Vishnu and Stoica, Ion and Gonzalez, Joseph E

    Patil, Shishir G. and Mao, Huanzhi and Yan, Fanjia and Ji, Charlie Cheng-Jie and Suresh, Vishnu and Stoica, Ion and Gonzalez, Joseph E. , booktitle =. The Berkeley Function Calling Leaderboard (

  63. [63]

    Journal of Medical Internet Research , volume =

    The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review , author =. Journal of Medical Internet Research , volume =. 2020 , doi =

    2020

  64. [64]

    Journal of Medical Internet Research , volume =

    Conversational Agents in Health Care: Scoping Review and Conceptual Analysis , author =. Journal of Medical Internet Research , volume =. 2020 , doi =

    2020

  65. [65]

    Journal of the American Medical Informatics Association , volume =

    Conversational Agents in Healthcare: A Systematic Review , author =. Journal of the American Medical Informatics Association , volume =. 2018 , doi =

    2018

  66. [66]

    Journal of Medical Internet Research , volume =

    Language Use in Conversational Agent-Based Health Communication: Systematic Review , author =. Journal of Medical Internet Research , volume =. 2022 , doi =

    2022

  67. [67]

    npj Digital Medicine , volume =

    The Diagnostic and Triage Accuracy of Digital and Online Symptom Checker Tools: A Systematic Review , author =. npj Digital Medicine , volume =. 2022 , doi =

    2022

  68. [68]

    Journal of Medical Internet Research , volume =

    Triage and Diagnostic Accuracy of Online Symptom Checkers: Systematic Review , author =. Journal of Medical Internet Research , volume =. 2023 , doi =

    2023

  69. [69]

    PLOS ONE , volume =

    Accuracy of Online Symptom Checkers and the Potential Impact on Service Utilisation , author =. PLOS ONE , volume =. 2021 , doi =

    2021

  70. [70]

    BMJ Open , volume =

    Digital and Online Symptom Checkers and Health Assessment/Triage Services for Urgent Health Problems: Systematic Review , author =. BMJ Open , volume =. 2019 , doi =

    2019

  71. [71]

    arXiv preprint arXiv:2310.19736 , year =

    Evaluating Large Language Models: A Comprehensive Survey , author =. arXiv preprint arXiv:2310.19736 , year =

    Pith/arXiv arXiv

  72. [72]

    Nature , volume =

    Large Language Models Encode Clinical Knowledge , author =. Nature , volume =. 2023 , doi =

    2023

  73. [73]

    Nature Medicine , volume =

    Toward Expert-Level Medical Question Answering with Large Language Models , author =. Nature Medicine , volume =. 2025 , doi =

    2025

  74. [74]

    arXiv preprint arXiv:2505.08775 , year =

    HealthBench: Evaluating Large Language Models Towards Improved Human Health , author =. arXiv preprint arXiv:2505.08775 , year =

    Pith/arXiv arXiv

  75. [75]

    Annals of Internal Medicine , volume =

    Ensuring Fairness in Machine Learning to Advance Health Equity , author =. Annals of Internal Medicine , volume =. 2018 , doi =

    2018

  76. [76]

    Annals of Family Medicine , volume =

    The Values and Value of Patient-Centered Care , author =. Annals of Family Medicine , volume =. 2011 , doi =

    2011

  77. [77]

    2026 , note =

    The Gemini Model Family: Gemini 3 Flash and Gemini 3.1 Pro , author =. 2026 , note =

    2026

  78. [78]

    2025 , eprint =

    Qwen3 Technical Report , author =. 2025 , eprint =

    2025

  79. [79]

    arXiv preprint arXiv:2511.18491 , year =

    MindEval: Benchmarking Language Models on Multi-turn Mental Health Support , author =. arXiv preprint arXiv:2511.18491 , year =

    arXiv

  80. [80]

    Nature , volume =

    Towards Conversational Diagnostic Artificial Intelligence , author =. Nature , volume =. 2025 , doi =

    2025

Showing first 80 references.