OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries

Michiel van der Heijden; Miguel Romano; Olivier Fail; Reinhard Berkels; Rogerio Corga Da Silva; Sandhanakrishnan Ravichandran; Shivesh Kumar; Valentine Emmanuel Gnanapragasam

REVIEW 4 major objections 4 minor 3 cited by

OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A retrieval-augmented clinical assistant, DR.INFO, is reported to outperform GPT-5 and o3 on HealthBench's Hard subset, with a body-table score of 0.51 against GPT-5's 0.46 and o3's 0.32.

desk verdict The paper's headline claim—DR.INFO beats GPT-5 and o3 on HealthBench Hard—is not reproducible as written because the abstract and body report different scores (0.68 vs 0.51); the underlying evaluation is real and fixable, but not publishable as is. read the letter →

arxiv 2509.02594 v3 pith:7WXP3YXU submitted 2025-08-29 q-bio.QM cs.AIcs.ETcs.IR

Sandhanakrishnan Ravichandran , Shivesh Kumar , Rogerio Corga Da Silva , Miguel Romano , Reinhard Berkels , Michiel van der Heijden , Olivier Fail , Valentine Emmanuel Gnanapragasam This is my paper

classification q-bio.QMcs.AIcs.ETcs.IR

keywords HealthBenchclinicalLLMevaluationretrieval-augmentedgenerationagenticRAGrubric-basedscoringmedicalAIassistantGPT-5comparisonopen-endedconversations

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that DR.INFO, an agentic retrieval-augmented clinical assistant, outperforms both frontier general-purpose LLMs and comparable clinical assistants on open-ended, rubric-scored health conversations. On the 1,000-item HealthBench Hard subset, the body reports a DR.INFO score of 0.51, above the reported GPT-5 score of 0.46 and o3's 0.32; on a 100-item comparison it reports 0.54 versus 0.49 for OpenEvidence and 0.48 for Pathway.md. The abstract gives different, higher numbers (0.68 and 0.72) for the same evaluations, an inconsistency that would need resolving before any specific figure is relied on. The broader point the authors are trying to establish is that behavior-level, rubric-based evaluation—measuring accuracy, completeness, context awareness, communication, and instruction following—reveals real clinical competence that multiple-choice benchmarks miss. If the central claim holds, retrieval-augmented architectures that deliberately seek context and hedge under uncertainty offer a safer path to deploying LLMs in medicine.

What carries the argument

The load-bearing mechanism is HealthBench's rubric scoring. Each conversation comes with physician-authored criteria, each assigned a point value from -10 to +10; a response is judged against each criterion, the earned points are summed and divided by the sum of absolute point values, and the result is clipped to [0, 1]. This turns open-ended conversations into a comparable behavioral score across five axes: accuracy, completeness, context awareness, communication, and instruction following. The system under test, DR.INFO, is an agentic RAG assistant: it retrieves relevant medical passages and takes conversational steps such as hedging and recommending escalation; the paper contrasts that ar

What would settle it

Score DR.INFO, GPT-5, o3, OpenEvidence, and Pathway.md on the same 1,000 HealthBench Hard conversations using one independent grader (human physician rubric reviewers or a fixed third-party judge), with model identities hidden; if DR.INFO's score does not exceed GPT-5's, or if the 0.68 abstract figure cannot be reproduced from the body's protocol, the central outperformance claim fails. Also check whether visual approximation of published baseline figures understates GPT-5 or o3 scores by more than the claimed margin.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that DR.INFO, the authors' agentic retrieval-augmented clinical assistant, sets a new high mark on HealthBench Hard, a physician-authored rubric evaluation of 1,000 open-ended, high-stakes health conversations. In the body of the paper, DR.INFO receives a HealthBench score of 0.51, above reported scores for GPT-5 (0.46 in thinking mode), o3 (0.32), and several other frontier models, with higher axis scores on accuracy, communication, instruction following, completeness, and context awareness. In a separate 100-conversation sample scored against two other agentic RAG assistants, DR.INFO scores 0.54 versus 0.49 for OpenEvidence and 0.48 for Pathwa

Load-bearing premise

The headline result stands or falls on the assumption that DR.INFO's rubric scores, produced by the authors' own protocol, are directly comparable to competitor scores that were visually approximated from published figures and to GPT-5's official reported number—and that whichever of the paper's inconsistent DR.INFO scores (abstract 0.68/0.72 vs body 0.51/0.54) is meant to be the headline is the reproducible one.

Editorial extensions

If this is right

If the reported scores are correct, retrieval-augmented agentic design can lift open-ended clinical safety behaviors without relying on a single model's raw knowledge.
Rubric-based, behavior-level benchmarking becomes a viable deployment gate because it surfaces axis-specific deficits—such as DR.INFO's context-seeking score of 0.35—instead of collapsing quality into one accuracy number.
The authors' own bootstrapped analysis indicates the lead over the two other clinical assistants is robust only at the 90% confidence level, so a full 1,000-item head-to-head would be needed to confirm superiority at the stricter 95% level.
Low scores on context awareness and completeness point to a concrete next improvement: teaching the assistant to ask for missing clinical details before answering, rather than over-answering from retrieved documents.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The comparison's reliability depends on whether DR.INFO's rubric scores, produced by the authors' own protocol, are directly comparable to competitor scores that were visually approximated from published figures and to GPT-5's official reported number; if a single independent grader scored all systems, the ranking could shift.
The abstract/body discrepancy (0.68/0.72 vs 0.51/0.54) suggests the evaluation may have been updated after the abstract was written; readers should treat 'state of the art' as tied to the exact scoring run, grader, and protocol used.
A testable extension of the paper's approach is to add an explicit 'gather missing information' step to an agentic RAG assistant and measure whether the context-awareness and completeness axes rise by more than a few points on the same HealthBench Hard subset.
HealthBench-style positive-and-negative rubrics could transfer to other high-stakes domains such as legal or financial advice, where open-ended conversations, hedging, and instruction following matter; domain experts would need to author the criteria.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. First, the paper's central claim rests on a number that appears twice in incompatible forms. The abstract says 0.68 on the 1,000-item Hard set; the body, Tables 1 and 7, says 0.51. The 100-sample comparison is similarly mismatched: 0.72 abstract, 0.54 body. That's not rounding; it's a 33% relative shift in the headline result. Any conclusion about outperforming frontier models depends entirely on which number you believe. Second, the comparison itself is partly built on visually approximated baseline scores from the HealthBench paper, and the grader for DR.INFO's responses is never identified. So the evaluation has a reproducibility problem even after you pick a number.

What's genuinely here: a real application of an established rubric-based benchmark to a proprietary agentic RAG system, with worked examples and an attempt at a head-to-head against two commercial clinical assistants. The bootstrapped 90% CI significance test is honest—they admit the 95% threshold fails. The limitations section is candid about text-only evaluation and subjective rubrics. That's more than many vendor reports do.

The soft spots are proportional. The abstract inconsistency is load-bearing and must be fixed before anything else. The visual approximation of baseline scores is a real methodological weakness, but it's not fatal if the published HealthBench figures are accurate and the approximation is close—still, the authors should obtain the exact numbers or re-run the baselines. The unidentified grader matters because HealthBench is rubric-based and the scorer affects the score; a sentence naming the model or protocol would help. No code or data is released, which limits independent checking. These are all fixable with enough detail.

Who is this for? Someone interested in how a real clinical RAG assistant performs on HealthBench, and in the pitfalls of vendor-run evaluation. It's not a methods paper. As a scientific claim, it currently fails because the headline number is ambiguous. As a case study, it's useful.

I'd send it to peer review rather than desk reject—the underlying question is timely and the work is salvageable—but any reviewer should anchor on the abstract/body inconsistency and demand the exact baseline numbers and grader identity. With those fixed, the paper could be a modest but legitimate evaluation report.

Referee Report

4 major / 4 minor

Summary. The manuscript reports a vendor evaluation of DR.INFO, a proprietary agentic RAG-based clinical assistant, on OpenAI's HealthBench Hard subset. The body claims DR.INFO scores 0.51 on 1,000 examples and 0.54 on a 100-example comparison, outperforming frontier LLMs and similar RAG assistants. The paper describes the HealthBench rubric framework, gives worked examples, and uses bootstrapping on the 100-sample comparison. A separate abstract version in the paper reports scores of 0.68 and 0.72. The central claim is that DR.INFO is state-of-the-art on this benchmark, particularly in communication, instruction following, and accuracy, with weaknesses in context awareness and completeness.

Significance. If the reported scores are accurate and the comparisons are fair, the result would be significant: a medical assistant outperforming GPT-5 and o3 on a safety-relevant, rubric-based benchmark is a useful data point for clinical LLM deployment. The use of an external benchmark and the inclusion of bootstrapped confidence intervals for the n=100 comparison are strengths. However, the significance is curtailed because the evaluation is a vendor self-assessment with an unidentified rubric grader, competitor scores are visually approximated, and the headline score appears in two incompatible versions.

major comments (4)

[Abstract vs. §6.1–6.2, Tables 1–2, §7–8] The headline result is internally inconsistent. The abstract at the top of the manuscript reports DR.INFO scores of 0.68 on the n=1000 Hard subset and 0.72 on the n=100 subset. The body abstract and Sections 6.1/6.2, Tables 1 and 2, report 0.51 and 0.54, and Sections 7–8 use 0.51. The difference between 0.51 and 0.68 is far beyond rounding (a ~33% relative change) and changes the comparative claims: 'surpassing GPT-5 ... outperforming o3 by nearly 60%' is derived from 0.51. As written, no single authoritative measurement is given, so the central result is not reproducible. All occurrences must be reconciled, with the definitive final score stated once.
[§6.1, §6.2.1] The scoring protocol is underspecified. The paper states that HealthBench uses physician-authored rubrics (§3), but does not state who applied the rubrics to DR.INFO's responses: human physicians, an LLM-based grader, or a hybrid. Section 6.2.1 refers to 'LLM-based scoring' and 'stochasticity' without identifying the model, prompt, temperature, or adjudication procedure. The axis-level scores in Table 1 and the aggregate score 0.51 therefore cannot be independently verified. This is load-bearing because the entire ranking claim rests on this grader's outputs.
[§6.1, Figure 1] The comparison against frontier LLMs is based on 'visually approximated' axis-wise scores from published HealthBench figures, not exact values. The overall scores for GPT-5 (0.46) and o3 (0.32) are quoted from secondary reports without a precise reference to the exact HealthBench numbers. This makes the claimed margin (e.g., 'nearly 60%') unreliable. The authors should either obtain exact baseline values from the HealthBench authors or reframe the comparison as qualitative/reported, with margins removed.
[§6.2.1, Table 3, §7–8] The statistical support is weaker than the conclusions suggest. The bootstrapped analysis for the n=100 comparison shows that DR.INFO's advantage over OpenEvidence and Pathway.md is significant at 90% but not at 95% CI (Table 3). The Discussion nonetheless says these results 'indicate that DR.INFO is likely to outperform' and the Conclusion calls it 'state-of-the-art.' Moreover, the n=1000 score of 0.51 has no confidence interval or error estimate. The claims should be tempered accordingly.

minor comments (4)

[Abstract, §1] Typos and formatting errors: 'high-stake clincal' in the abstract, 'themese' in §6.1, 'F ollowing' in Table 1, 'Y es' in Table 3. The body abstract also contains a different score from the header abstract, which must be reconciled.
[References] Reference [14] is a product URL rather than a citable system description; a technical description of DR.INFO's architecture and retrieval pipeline should be provided or cited. Reference [22] is a marketing page; if exact benchmark numbers are used, a primary source is preferable.
[§5.2] The worked examples are described first as 'from the health bench consensus dataset' and later as selected from 'HealthBench hard dataset.' Clarify which subset each example belongs to and how representative they are of the aggregate result.
[Figures 1–2] Figures 1 and 2 are referenced with captions but the actual plots are not visible in the manuscript text. If they are included as images, ensure they are legible; if not, add them.

Circularity Check

0 steps flagged · score 0.0 of 10

No definitional circularity: DR.INFO's HealthBench scores are external-benchmark measurements, not predictions derived from the comparison baselines or from self-citations.

full rationale

The paper's central claim is an empirical benchmark comparison. DR.INFO is evaluated on OpenAI's HealthBench Hard subset [13], an external dataset with physician-authored rubrics; its scores (0.51/0.54 in the body; 0.68/0.72 in the abstract) are outputs of an evaluation protocol, not parameters fitted to the reported competitor numbers. The baselines (GPT-5 0.46, o3 0.32, etc.) are externally published; even where the authors say they were 'visually approximated from the published HealthBench figures' (Section 6.1), those numbers are not constructed from DR.INFO's inputs. The comparative claims are simple arithmetic (0.51 vs 0.46 and 0.32), not a derivation that reduces to its own assumptions. There is no load-bearing self-citation: [13] is external, and [14] merely identifies the system under test; the USMLE 95.4% statement in the Introduction cites [6]/[14], but that claim is not part of the HealthBench scoring chain. The unverified 'LLM rubric' grader mentioned in Section 6.2.1 ('axis-level scores generated by the LLM rubric') is a serious transparency/external-validity concern, and the abstract/body score mismatch (0.68/0.72 vs 0.51/0.54) is a reproducibility defect, but neither is a definitional circularity: the paper nowhere states that the grader is DR.INFO itself, that the rubric is defined to match DR.INFO, or that the competitor scores are fitted inputs. Because the benchmark is external and the comparison is a measurement, the derivation chain is self-contained against an external standard. Score 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No free parameters are fitted: the report compares scores rather than fitting a model. The axioms are the unstated trust assumptions in the evaluation chain: HealthBench validity, LLM grader fidelity, visual approximation of baselines, subset representativeness, and absence of test-set contamination.

assumptions (5)

domain assumption HealthBench rubric criteria and physician annotations are a valid and complete measure of clinical response quality.
The entire evaluation (Sections 2 and 5) treats HealthBench scores as ground truth and does not independently validate the rubrics or annotations.
domain assumption LLM-based rubric scoring correctly reproduces physician judgments.
Section 6.2.1 refers to 'axis-level scores generated by the LLM rubric'; the identity, calibration, and bias of the LLM grader are not described.
domain assumption Scores for competing models were visually approximated from published HealthBench figures with sufficient accuracy for ranking comparisons.
Section 6.1 states competitor axis scores were 'visually approximated from the published HealthBench figures', a method not validated for precision.
domain assumption The 100-sample subset is representative of the HealthBench Hard set.
Section 6.2 describes random sampling with even distribution across seven themes, but provides no stratification, seed, or power analysis.
domain assumption DR.INFO was not tuned on HealthBench and its scoring protocol is unbiased.
No evidence is given that the vendor avoided test-set contamination or that the rubric grader was blind to which system produced the response.

how reviews work

0 comments

Cite this review

Pith. "Pith review of OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries." pith.science (2026). https://pith.science/paper/7WXP3YXU

@misc{pith2026250902594,
  author       = {Pith},
  title        = {Pith review of: OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7WXP3YXU}},
  note         = {Machine review of arXiv:2509.02594}
}

read the original abstract

Evaluating large language models (LLMs) on their ability to generate high-quality, accurate, situationally aware answers to clinical questions requires going beyond conventional benchmarks to assess how these systems behave in complex, high-stakes clinical scenarios. Traditional evaluations are often limited to multiple-choice questions that fail to capture essential competencies such as contextual reasoning, contextual awareness, and uncertainty handling. To address these limitations, we evaluate our agentic RAG-based clinical support assistant, DR. INFO, using HealthBench, a rubric-driven benchmark composed of open-ended, expert-annotated health conversations. On the Hard subset of 1,000 challenging examples, DR. INFO achieves a HealthBench Hard score of 0.68, outperforming leading frontier LLMs including the GPT-5 model family (GPT-5: 0.46, GPT-5.2: 0.42, GPT-5.1: 0.40), Grok 3 (0.23), Gemini 2.5 Pro (0.19), and Claude 3.7 Sonnet (0.02) across all behavioral axes (accuracy, completeness, instruction following, etc.). In a separate 100-sample evaluation against similar agentic RAG assistants (OpenEvidence and Pathway.md, now DoxGPT by Doximity), it maintains a performance lead with a HealthBench Hard score of 0.72. These results highlight the strengths of DR. INFO in communication, instruction following, and accuracy, while also revealing areas for improvement in context awareness and response completeness. Overall, the findings underscore the utility of behavior-level, rubric-based evaluation for building reliable and trustworthy AI-enabled clinical support systems.

Figures

Figures reproduced from arXiv: 2509.02594 by the authors.

**Figure 2.** HealthBench axis-wise scores for 100-sample subset: DR.INFO vs. OpenEvidence vs. Pathway.md [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Generalistic or Specific Embeddings, Which is Better? An Empirical Study on Search for Clinical Coding in Non-English Languages
cs.CL 2026-05 unverdicted novelty 5.0 of 10

Fine-tuning a Spanish biomedical encoder on Gemini-generated synthetic data for multiple languages yields a bi-encoder that matches or exceeds BioBERT-ST on clinical code retrieval metrics, with further gains from cro...
MDIA: A Multi-Agent Diagnostic Intelligence Pipeline on HealthBench Professional
cs.AI 2026-05 unverdicted novelty 4.0 of 10

MDIA, a specialty-routed 7-node multi-agent system, reports 0.6272 accuracy on 525 HealthBench Professional cases using GPT-5.4, outperforming the ChatGPT for Clinicians baseline by 3.72 points and attributing the lif...
DR. INFO at the Point of Care: A Prospective Pilot Study of Physician-Perceived Value of an Agentic AI Clinical Assistant
cs.HC 2026-03 conditional novelty 3.0 of 10

A pilot study of 29 physicians found high perceived time savings (4.27/5) and decision support (4.16/5) from DR. INFO, yielding an NPS of 81.2 that dropped to 44.8 under conservative assumptions.

Reference graph

Works this paper leans on

20 extracted references · 8 canonical work pages · cited by 3 Pith papers

[1]

A systematic review of large language model (LLM) evaluations in clinical medicine

Sina Shool et al. “A systematic review of large language model (LLM) evaluations in clinical medicine”. In: BMC Medical Informatics and Decision Making 25 (2025), p. 117. doi: 10.1186/s12911- 025- 02954-4 . url: https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911- 025-02954-4

doi:10.1186/s12911- 2025
[3]

Scientific Evidence for Clinical Text Summarization Using Large Language Models: Scoping Review

Lydie Bednarczyk et al. “Scientific Evidence for Clinical Text Summarization Using Large Language Models: Scoping Review”. In: Journal of Medical Internet Research (2025). PMCID: PMC12123242. doi: 10.2196/68998. url: https://www.jmir.org/2025/1/e68998

doi:10.2196/68998 2025
[5]

AI Consult: Real-World Evaluation of Large Language Model-Based Clinical Decision Support in Primary Care

Robert Korom et al. AI Consult: Real-World Evaluation of Large Language Model-Based Clinical Decision Support in Primary Care . Preprint. Quality improvement study across 39,849 visits in 15 clinics; 16% fewer diagnostic errors and 13% fewer treatment errors. 2025

work page 2025
[6]

Capabilities of GPT-4 on Medical Challenge Problems

Harsha Nori et al. “Capabilities of GPT-4 on Medical Challenge Problems”. In: arXiv preprint arXiv:2303.13375 (2023). doi: 10.48550/arXiv.2303.13375. url: https://doi.org/10.48550/arXiv.2303.13375

arXiv doi:10.48550/arxiv.2303.13375 2023
[7]

Large Language Models Encode Clinical Knowledge

Karan Singhal et al. “Large Language Models Encode Clinical Knowledge”. In: Nature 620 (2023), pp. 172–180. doi: 10.1038/s41586-023-06291-2 . url: https://www.nature.com/articles/s41586- 023-06291-2

doi:10.1038/s41586-023-06291-2 2023
[8]

Clinical Reasoning of a Generative Artificial Intelligence Model Compared With Physi- cians

Cabral et al. “Clinical Reasoning of a Generative Artificial Intelligence Model Compared With Physi- cians”. In: JAMA Internal Medicine 184.5 (2024), pp. 581–583. doi: 10.1001/jamainternmed.2024

work page doi:10.1001/jamainternmed.2024 2024
[9]

What disease does this patient have? A large-scale open domain question answering dataset from medical exams

Di Jin et al. “What disease does this patient have? A large-scale open domain question answering dataset from medical exams”. In: Applied Sciences 11.14 (2021), p. 6421. doi: 10.3390/app11146421. url: https://doi.org/10.3390/app11146421

doi:10.3390/app11146421 2021
[10]

MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering

Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. “MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering”. In: Proceedings of the Conference on Health, Inference, and Learning . Ed. by Gerardo Flores et al. Vol. 174. Proceedings of Machine Learning Research. PMLR, July 2022, pp. 248–260. url: https:/...

work page 2022

Show all 20 references

[11]

PubMedQA: A Dataset for Biomedical Research Question Answering

Qiao Jin et al. “PubMedQA: A Dataset for Biomedical Research Question Answering”. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) . 2019, pp. 256...

2019 arXiv
[12]

Sequential Diagnosis with Language Models

Harsha Nori et al. “Sequential Diagnosis with Language Models”. In: arXiv preprint arXiv:2506.22405 (2025). url: https://arxiv.org/abs/2506.22405

2025 arXiv
[13]

HealthBench: Evaluating Large Language Models Towards Improved Human Health

Rahul K. Arora et al. “HealthBench: Evaluating Large Language Models Towards Improved Human Health”. In: arXiv preprint arXiv:2505.08775 (2025). doi: 10.48550/arXiv.2505.08775. url: https: //doi.org/10.48550/arXiv.2505.08775

2025 doi arXiv
[14]

Dr.INFO: Agentic Clinical Assistant

Synduct. Dr.INFO: Agentic Clinical Assistant. Accessed: 2025-07-21. 2025. url: https://app.drinfo. ai. 12

2025
[15]

Leveraging long context in retrieval augmented language models for medical question answering

Gongbo Zhang et al. “Leveraging long context in retrieval augmented language models for medical question answering”. In: npj Digital Medicine 8 (2025), p. 239. doi: 10.1038/s41746-025-01651-w . url: https://www.nature.com/articles/s41746-025-01651-w

2025 doi
[17]

VISTA: A Rubric-based Visual Task Assessment Instruction-Specific Task Assessments

Scale AI. VISTA: A Rubric-based Visual Task Assessment Instruction-Specific Task Assessments . url: https://scale.com/leaderboard/visual_language_understanding
[18]

PaperBench: A Rubric-based Benchmark for Scientific Paper Summarization

Giulio Starace et al. “PaperBench: A Rubric-based Benchmark for Scientific Paper Summarization”. In: arXiv preprint arXiv:2504.01848v1 (2025). url: https://arxiv.org/html/2504.01848v1

2025 arXiv
[19]

WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild

Bill Yuchen Lin et al. “WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild”. In: arXiv preprint arXiv:2406.04770 (2024). url: https://doi.org/10.48550/arXiv. 2406.04770

2024 doi arXiv
[20]

Autonomous medical evaluation for guideline adherence of large language models

Ethan Fast et al. “Autonomous medical evaluation for guideline adherence of large language models”. In: npj Digital Medicine 7 (2024), p. 358. doi: 10.1038/s41746- 024- 01356- 6. url: https://doi. org/10.1038/s41746-024-01356-6

2024 doi
[21]

MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs

Ved Sirdeshmukh et al. “MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs”. In: arXiv preprint arXiv:2501.17399 (2025). url: https://doi.org/ 10.48550/arXiv.2501.17399

2025 doi arXiv
[22]

Introducing GPT-5

OpenAI. Introducing GPT-5. Accessed: 2025-08-15. 2025. url:https://openai.com/index/introducing- gpt-5/. 13

2025
[295]

url: https://doi.org/10.1001/jamainternmed.2024.0295

2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

A systematic review of large language model (LLM) evaluations in clinical medicine

Sina Shool et al. “A systematic review of large language model (LLM) evaluations in clinical medicine”. In: BMC Medical Informatics and Decision Making 25 (2025), p. 117. doi: 10.1186/s12911- 025- 02954-4 . url: https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911- 025-02954-4

doi:10.1186/s12911- 2025

[2] [3]

Scientific Evidence for Clinical Text Summarization Using Large Language Models: Scoping Review

Lydie Bednarczyk et al. “Scientific Evidence for Clinical Text Summarization Using Large Language Models: Scoping Review”. In: Journal of Medical Internet Research (2025). PMCID: PMC12123242. doi: 10.2196/68998. url: https://www.jmir.org/2025/1/e68998

doi:10.2196/68998 2025

[3] [5]

AI Consult: Real-World Evaluation of Large Language Model-Based Clinical Decision Support in Primary Care

Robert Korom et al. AI Consult: Real-World Evaluation of Large Language Model-Based Clinical Decision Support in Primary Care . Preprint. Quality improvement study across 39,849 visits in 15 clinics; 16% fewer diagnostic errors and 13% fewer treatment errors. 2025

work page 2025

[4] [6]

Capabilities of GPT-4 on Medical Challenge Problems

Harsha Nori et al. “Capabilities of GPT-4 on Medical Challenge Problems”. In: arXiv preprint arXiv:2303.13375 (2023). doi: 10.48550/arXiv.2303.13375. url: https://doi.org/10.48550/arXiv.2303.13375

arXiv doi:10.48550/arxiv.2303.13375 2023

[5] [7]

Large Language Models Encode Clinical Knowledge

Karan Singhal et al. “Large Language Models Encode Clinical Knowledge”. In: Nature 620 (2023), pp. 172–180. doi: 10.1038/s41586-023-06291-2 . url: https://www.nature.com/articles/s41586- 023-06291-2

doi:10.1038/s41586-023-06291-2 2023

[6] [8]

Clinical Reasoning of a Generative Artificial Intelligence Model Compared With Physi- cians

Cabral et al. “Clinical Reasoning of a Generative Artificial Intelligence Model Compared With Physi- cians”. In: JAMA Internal Medicine 184.5 (2024), pp. 581–583. doi: 10.1001/jamainternmed.2024

work page doi:10.1001/jamainternmed.2024 2024

[7] [9]

What disease does this patient have? A large-scale open domain question answering dataset from medical exams

Di Jin et al. “What disease does this patient have? A large-scale open domain question answering dataset from medical exams”. In: Applied Sciences 11.14 (2021), p. 6421. doi: 10.3390/app11146421. url: https://doi.org/10.3390/app11146421

doi:10.3390/app11146421 2021

[8] [10]

MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering

Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. “MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering”. In: Proceedings of the Conference on Health, Inference, and Learning . Ed. by Gerardo Flores et al. Vol. 174. Proceedings of Machine Learning Research. PMLR, July 2022, pp. 248–260. url: https:/...

work page 2022