Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

A retrieval-augmented clinical assistant, DR.INFO, is reported to outperform GPT-5 and o3 on HealthBench's Hard subset, with a body-table score of 0.51 against GPT-5's 0.46 and o3's 0.32.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

DR.INFO, a vendor-built RAG clinical assistant, is reported to beat frontier LLMs on OpenAI's HealthBench, but the paper's own scores contradict its abstract.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection The paper's headline claim—DR.INFO beats GPT-5 and o3 on HealthBench Hard—is not reproducible as written because the abstract and body report different scores (0.68 vs 0.51); the underlying evaluation is real and fixable, but not publishable as is. the 4 major comments →

arxiv 2509.02594 v3 pith:7WXP3YXU submitted 2025-08-29 q-bio.QM cs.AIcs.ETcs.IR

OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries

Sandhanakrishnan Ravichandran , Shivesh Kumar , Rogerio Corga Da Silva , Miguel Romano , Reinhard Berkels , Michiel van der Heijden , Olivier Fail , Valentine Emmanuel Gnanapragasam This is my paper
classification q-bio.QM cs.AIcs.ETcs.IR
keywords HealthBenchclinical LLM evaluationretrieval-augmented generationagentic RAGrubric-based scoringmedical AI assistantGPT-5 comparisonopen-ended health conversations
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that DR.INFO, an agentic retrieval-augmented clinical assistant, outperforms both frontier general-purpose LLMs and comparable clinical assistants on open-ended, rubric-scored health conversations. On the 1,000-item HealthBench Hard subset, the body reports a DR.INFO score of 0.51, above the reported GPT-5 score of 0.46 and o3's 0.32; on a 100-item comparison it reports 0.54 versus 0.49 for OpenEvidence and 0.48 for Pathway.md. The abstract gives different, higher numbers (0.68 and 0.72) for the same evaluations, an inconsistency that would need resolving before any specific figure is relied on. The broader point the authors are trying to establish is that behavior-level, rubric-based evaluation—measuring accuracy, completeness, context awareness, communication, and instruction following—reveals real clinical competence that multiple-choice benchmarks miss. If the central claim holds, retrieval-augmented architectures that deliberately seek context and hedge under uncertainty offer a safer path to deploying LLMs in medicine.

Core claim

On its own terms, the paper's central claim is that DR.INFO, the authors' agentic retrieval-augmented clinical assistant, sets a new high mark on HealthBench Hard, a physician-authored rubric evaluation of 1,000 open-ended, high-stakes health conversations. In the body of the paper, DR.INFO receives a HealthBench score of 0.51, above reported scores for GPT-5 (0.46 in thinking mode), o3 (0.32), and several other frontier models, with higher axis scores on accuracy, communication, instruction following, completeness, and context awareness. In a separate 100-conversation sample scored against two other agentic RAG assistants, DR.INFO scores 0.54 versus 0.49 for OpenEvidence and 0.48 for Pathwa

What carries the argument

The load-bearing mechanism is HealthBench's rubric scoring. Each conversation comes with physician-authored criteria, each assigned a point value from -10 to +10; a response is judged against each criterion, the earned points are summed and divided by the sum of absolute point values, and the result is clipped to [0, 1]. This turns open-ended conversations into a comparable behavioral score across five axes: accuracy, completeness, context awareness, communication, and instruction following. The system under test, DR.INFO, is an agentic RAG assistant: it retrieves relevant medical passages and takes conversational steps such as hedging and recommending escalation; the paper contrasts that ar

Load-bearing premise

The headline result stands or falls on the assumption that DR.INFO's rubric scores, produced by the authors' own protocol, are directly comparable to competitor scores that were visually approximated from published figures and to GPT-5's official reported number—and that whichever of the paper's inconsistent DR.INFO scores (abstract 0.68/0.72 vs body 0.51/0.54) is meant to be the headline is the reproducible one.

What would settle it

Score DR.INFO, GPT-5, o3, OpenEvidence, and Pathway.md on the same 1,000 HealthBench Hard conversations using one independent grader (human physician rubric reviewers or a fixed third-party judge), with model identities hidden; if DR.INFO's score does not exceed GPT-5's, or if the 0.68 abstract figure cannot be reproduced from the body's protocol, the central outperformance claim fails. Also check whether visual approximation of published baseline figures understates GPT-5 or o3 scores by more than the claimed margin.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the reported scores are correct, retrieval-augmented agentic design can lift open-ended clinical safety behaviors without relying on a single model's raw knowledge.
  • Rubric-based, behavior-level benchmarking becomes a viable deployment gate because it surfaces axis-specific deficits—such as DR.INFO's context-seeking score of 0.35—instead of collapsing quality into one accuracy number.
  • The authors' own bootstrapped analysis indicates the lead over the two other clinical assistants is robust only at the 90% confidence level, so a full 1,000-item head-to-head would be needed to confirm superiority at the stricter 95% level.
  • Low scores on context awareness and completeness point to a concrete next improvement: teaching the assistant to ask for missing clinical details before answering, rather than over-answering from retrieved documents.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The comparison's reliability depends on whether DR.INFO's rubric scores, produced by the authors' own protocol, are directly comparable to competitor scores that were visually approximated from published figures and to GPT-5's official reported number; if a single independent grader scored all systems, the ranking could shift.
  • The abstract/body discrepancy (0.68/0.72 vs 0.51/0.54) suggests the evaluation may have been updated after the abstract was written; readers should treat 'state of the art' as tied to the exact scoring run, grader, and protocol used.
  • A testable extension of the paper's approach is to add an explicit 'gather missing information' step to an agentic RAG assistant and measure whether the context-awareness and completeness axes rise by more than a few points on the same HealthBench Hard subset.
  • HealthBench-style positive-and-negative rubrics could transfer to other high-stakes domains such as legal or financial advice, where open-ended conversations, hedging, and instruction following matter; domain experts would need to author the criteria.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript reports a vendor evaluation of DR.INFO, a proprietary agentic RAG-based clinical assistant, on OpenAI's HealthBench Hard subset. The body claims DR.INFO scores 0.51 on 1,000 examples and 0.54 on a 100-example comparison, outperforming frontier LLMs and similar RAG assistants. The paper describes the HealthBench rubric framework, gives worked examples, and uses bootstrapping on the 100-sample comparison. A separate abstract version in the paper reports scores of 0.68 and 0.72. The central claim is that DR.INFO is state-of-the-art on this benchmark, particularly in communication, instruction following, and accuracy, with weaknesses in context awareness and completeness.

Significance. If the reported scores are accurate and the comparisons are fair, the result would be significant: a medical assistant outperforming GPT-5 and o3 on a safety-relevant, rubric-based benchmark is a useful data point for clinical LLM deployment. The use of an external benchmark and the inclusion of bootstrapped confidence intervals for the n=100 comparison are strengths. However, the significance is curtailed because the evaluation is a vendor self-assessment with an unidentified rubric grader, competitor scores are visually approximated, and the headline score appears in two incompatible versions.

major comments (4)
  1. [Abstract vs. §6.1–6.2, Tables 1–2, §7–8] The headline result is internally inconsistent. The abstract at the top of the manuscript reports DR.INFO scores of 0.68 on the n=1000 Hard subset and 0.72 on the n=100 subset. The body abstract and Sections 6.1/6.2, Tables 1 and 2, report 0.51 and 0.54, and Sections 7–8 use 0.51. The difference between 0.51 and 0.68 is far beyond rounding (a ~33% relative change) and changes the comparative claims: 'surpassing GPT-5 ... outperforming o3 by nearly 60%' is derived from 0.51. As written, no single authoritative measurement is given, so the central result is not reproducible. All occurrences must be reconciled, with the definitive final score stated once.
  2. [§6.1, §6.2.1] The scoring protocol is underspecified. The paper states that HealthBench uses physician-authored rubrics (§3), but does not state who applied the rubrics to DR.INFO's responses: human physicians, an LLM-based grader, or a hybrid. Section 6.2.1 refers to 'LLM-based scoring' and 'stochasticity' without identifying the model, prompt, temperature, or adjudication procedure. The axis-level scores in Table 1 and the aggregate score 0.51 therefore cannot be independently verified. This is load-bearing because the entire ranking claim rests on this grader's outputs.
  3. [§6.1, Figure 1] The comparison against frontier LLMs is based on 'visually approximated' axis-wise scores from published HealthBench figures, not exact values. The overall scores for GPT-5 (0.46) and o3 (0.32) are quoted from secondary reports without a precise reference to the exact HealthBench numbers. This makes the claimed margin (e.g., 'nearly 60%') unreliable. The authors should either obtain exact baseline values from the HealthBench authors or reframe the comparison as qualitative/reported, with margins removed.
  4. [§6.2.1, Table 3, §7–8] The statistical support is weaker than the conclusions suggest. The bootstrapped analysis for the n=100 comparison shows that DR.INFO's advantage over OpenEvidence and Pathway.md is significant at 90% but not at 95% CI (Table 3). The Discussion nonetheless says these results 'indicate that DR.INFO is likely to outperform' and the Conclusion calls it 'state-of-the-art.' Moreover, the n=1000 score of 0.51 has no confidence interval or error estimate. The claims should be tempered accordingly.
minor comments (4)
  1. [Abstract, §1] Typos and formatting errors: 'high-stake clincal' in the abstract, 'themese' in §6.1, 'F ollowing' in Table 1, 'Y es' in Table 3. The body abstract also contains a different score from the header abstract, which must be reconciled.
  2. [References] Reference [14] is a product URL rather than a citable system description; a technical description of DR.INFO's architecture and retrieval pipeline should be provided or cited. Reference [22] is a marketing page; if exact benchmark numbers are used, a primary source is preferable.
  3. [§5.2] The worked examples are described first as 'from the health bench consensus dataset' and later as selected from 'HealthBench hard dataset.' Clarify which subset each example belongs to and how representative they are of the aggregate result.
  4. [Figures 1–2] Figures 1 and 2 are referenced with captions but the actual plots are not visible in the manuscript text. If they are included as images, ensure they are legible; if not, add them.

Circularity Check

0 steps flagged

No definitional circularity: DR.INFO's HealthBench scores are external-benchmark measurements, not predictions derived from the comparison baselines or from self-citations.

full rationale

The paper's central claim is an empirical benchmark comparison. DR.INFO is evaluated on OpenAI's HealthBench Hard subset [13], an external dataset with physician-authored rubrics; its scores (0.51/0.54 in the body; 0.68/0.72 in the abstract) are outputs of an evaluation protocol, not parameters fitted to the reported competitor numbers. The baselines (GPT-5 0.46, o3 0.32, etc.) are externally published; even where the authors say they were 'visually approximated from the published HealthBench figures' (Section 6.1), those numbers are not constructed from DR.INFO's inputs. The comparative claims are simple arithmetic (0.51 vs 0.46 and 0.32), not a derivation that reduces to its own assumptions. There is no load-bearing self-citation: [13] is external, and [14] merely identifies the system under test; the USMLE 95.4% statement in the Introduction cites [6]/[14], but that claim is not part of the HealthBench scoring chain. The unverified 'LLM rubric' grader mentioned in Section 6.2.1 ('axis-level scores generated by the LLM rubric') is a serious transparency/external-validity concern, and the abstract/body score mismatch (0.68/0.72 vs 0.51/0.54) is a reproducibility defect, but neither is a definitional circularity: the paper nowhere states that the grader is DR.INFO itself, that the rubric is defined to match DR.INFO, or that the competitor scores are fitted inputs. Because the benchmark is external and the comparison is a measurement, the derivation chain is self-contained against an external standard. Score 0.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

No free parameters are fitted: the report compares scores rather than fitting a model. The axioms are the unstated trust assumptions in the evaluation chain: HealthBench validity, LLM grader fidelity, visual approximation of baselines, subset representativeness, and absence of test-set contamination.

axioms (5)
  • domain assumption HealthBench rubric criteria and physician annotations are a valid and complete measure of clinical response quality.
    The entire evaluation (Sections 2 and 5) treats HealthBench scores as ground truth and does not independently validate the rubrics or annotations.
  • domain assumption LLM-based rubric scoring correctly reproduces physician judgments.
    Section 6.2.1 refers to 'axis-level scores generated by the LLM rubric'; the identity, calibration, and bias of the LLM grader are not described.
  • domain assumption Scores for competing models were visually approximated from published HealthBench figures with sufficient accuracy for ranking comparisons.
    Section 6.1 states competitor axis scores were 'visually approximated from the published HealthBench figures', a method not validated for precision.
  • domain assumption The 100-sample subset is representative of the HealthBench Hard set.
    Section 6.2 describes random sampling with even distribution across seven themes, but provides no stratification, seed, or power analysis.
  • domain assumption DR.INFO was not tuned on HealthBench and its scoring protocol is unbiased.
    No evidence is given that the vendor avoided test-set contamination or that the rubric grader was blind to which system produced the response.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries." pith.science (2026). https://pith.science/paper/7WXP3YXU

@misc{pith2026250902594,
  author       = {Pith},
  title        = {Pith review of: OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7WXP3YXU}},
  note         = {Machine review of arXiv:2509.02594}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Evaluating large language models (LLMs) on their ability to generate high-quality, accurate, situationally aware answers to clinical questions requires going beyond conventional benchmarks to assess how these systems behave in complex, high-stakes clinical scenarios. Traditional evaluations are often limited to multiple-choice questions that fail to capture essential competencies such as contextual reasoning, contextual awareness, and uncertainty handling. To address these limitations, we evaluate our agentic RAG-based clinical support assistant, DR. INFO, using HealthBench, a rubric-driven benchmark composed of open-ended, expert-annotated health conversations. On the Hard subset of 1,000 challenging examples, DR. INFO achieves a HealthBench Hard score of 0.68, outperforming leading frontier LLMs including the GPT-5 model family (GPT-5: 0.46, GPT-5.2: 0.42, GPT-5.1: 0.40), Grok 3 (0.23), Gemini 2.5 Pro (0.19), and Claude 3.7 Sonnet (0.02) across all behavioral axes (accuracy, completeness, instruction following, etc.). In a separate 100-sample evaluation against similar agentic RAG assistants (OpenEvidence and Pathway.md, now DoxGPT by Doximity), it maintains a performance lead with a HealthBench Hard score of 0.72. These results highlight the strengths of DR. INFO in communication, instruction following, and accuracy, while also revealing areas for improvement in context awareness and response completeness. Overall, the findings underscore the utility of behavior-level, rubric-based evaluation for building reliable and trustworthy AI-enabled clinical support systems.

Figures

Figures reproduced from arXiv: 2509.02594 by Michiel van der Heijden, Miguel Romano, Olivier Fail, Reinhard Berkels, Rogerio Corga Da Silva, Sandhanakrishnan Ravichandran, Shivesh Kumar, Valentine Emmanuel Gnanapragasam.

Figure 1
Figure 1. Figure 1: Comparison of axis-wise scores for DR.INFO and other frontier LLMs on the HealthBench Hard [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: HealthBench axis-wise scores for 100-sample subset: DR.INFO vs. OpenEvidence vs. Pathway.md [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Generalistic or Specific Embeddings, Which is Better? An Empirical Study on Search for Clinical Coding in Non-English Languages

    cs.CL 2026-05 unverdicted novelty 5.0

    Fine-tuning a Spanish biomedical encoder on Gemini-generated synthetic data for multiple languages yields a bi-encoder that matches or exceeds BioBERT-ST on clinical code retrieval metrics, with further gains from cro...

  2. MDIA: A Multi-Agent Diagnostic Intelligence Pipeline on HealthBench Professional

    cs.AI 2026-05 unverdicted novelty 4.0

    MDIA, a specialty-routed 7-node multi-agent system, reports 0.6272 accuracy on 525 HealthBench Professional cases using GPT-5.4, outperforming the ChatGPT for Clinicians baseline by 3.72 points and attributing the lif...

  3. DR. INFO at the Point of Care: A Prospective Pilot Study of Physician-Perceived Value of an Agentic AI Clinical Assistant

    cs.HC 2026-03 conditional novelty 3.0

    A pilot study of 29 physicians found high perceived time savings (4.27/5) and decision support (4.16/5) from DR. INFO, yielding an NPS of 81.2 that dropped to 44.8 under conservative assumptions.

Reference graph

Works this paper leans on

20 extracted references · 8 canonical work pages · cited by 3 Pith papers

  1. [1]

    A systematic review of large language model (LLM) evaluations in clinical medicine

    Sina Shool et al. “A systematic review of large language model (LLM) evaluations in clinical medicine”. In: BMC Medical Informatics and Decision Making 25 (2025), p. 117. doi: 10.1186/s12911- 025- 02954-4 . url: https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911- 025-02954-4

    doi:10.1186/s12911- 2025

  2. [3]

    Scientific Evidence for Clinical Text Summarization Using Large Language Models: Scoping Review

    Lydie Bednarczyk et al. “Scientific Evidence for Clinical Text Summarization Using Large Language Models: Scoping Review”. In: Journal of Medical Internet Research (2025). PMCID: PMC12123242. doi: 10.2196/68998. url: https://www.jmir.org/2025/1/e68998

    doi:10.2196/68998 2025

  3. [5]

    AI Consult: Real-World Evaluation of Large Language Model-Based Clinical Decision Support in Primary Care

    Robert Korom et al. AI Consult: Real-World Evaluation of Large Language Model-Based Clinical Decision Support in Primary Care . Preprint. Quality improvement study across 39,849 visits in 15 clinics; 16% fewer diagnostic errors and 13% fewer treatment errors. 2025

    work page 2025

  4. [6]

    Capabilities of GPT-4 on Medical Challenge Problems

    Harsha Nori et al. “Capabilities of GPT-4 on Medical Challenge Problems”. In: arXiv preprint arXiv:2303.13375 (2023). doi: 10.48550/arXiv.2303.13375. url: https://doi.org/10.48550/arXiv.2303.13375

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.13375 2023

  5. [7]

    Large Language Models Encode Clinical Knowledge

    Karan Singhal et al. “Large Language Models Encode Clinical Knowledge”. In: Nature 620 (2023), pp. 172–180. doi: 10.1038/s41586-023-06291-2 . url: https://www.nature.com/articles/s41586- 023-06291-2

    doi:10.1038/s41586-023-06291-2 2023

  6. [8]

    Clinical Reasoning of a Generative Artificial Intelligence Model Compared With Physi- cians

    Cabral et al. “Clinical Reasoning of a Generative Artificial Intelligence Model Compared With Physi- cians”. In: JAMA Internal Medicine 184.5 (2024), pp. 581–583. doi: 10.1001/jamainternmed.2024

    work page doi:10.1001/jamainternmed.2024 2024

  7. [9]

    What disease does this patient have? A large-scale open domain question answering dataset from medical exams

    Di Jin et al. “What disease does this patient have? A large-scale open domain question answering dataset from medical exams”. In: Applied Sciences 11.14 (2021), p. 6421. doi: 10.3390/app11146421. url: https://doi.org/10.3390/app11146421

    doi:10.3390/app11146421 2021

  8. [10]

    MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. “MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering”. In: Proceedings of the Conference on Health, Inference, and Learning . Ed. by Gerardo Flores et al. Vol. 174. Proceedings of Machine Learning Research. PMLR, July 2022, pp. 248–260. url: https:/...

    work page 2022

  9. [11]

    PubMedQA: A Dataset for Biomedical Research Question Answering

    Qiao Jin et al. “PubMedQA: A Dataset for Biomedical Research Question Answering”. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) . 2019, pp. 2567–2577. doi: 10. 18653/v1/D19-1259. url: https://arxiv.org/abs/1909.06146

    Pith/arXiv arXiv 2019

  10. [12]

    Sequential Diagnosis with Language Models

    Harsha Nori et al. “Sequential Diagnosis with Language Models”. In: arXiv preprint arXiv:2506.22405 (2025). url: https://arxiv.org/abs/2506.22405

    Pith/arXiv arXiv 2025

  11. [13]

    HealthBench: Evaluating Large Language Models Towards Improved Human Health

    Rahul K. Arora et al. “HealthBench: Evaluating Large Language Models Towards Improved Human Health”. In: arXiv preprint arXiv:2505.08775 (2025). doi: 10.48550/arXiv.2505.08775. url: https: //doi.org/10.48550/arXiv.2505.08775

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.08775 2025

  12. [14]

    Dr.INFO: Agentic Clinical Assistant

    Synduct. Dr.INFO: Agentic Clinical Assistant. Accessed: 2025-07-21. 2025. url: https://app.drinfo. ai. 12

    work page 2025

  13. [15]

    Leveraging long context in retrieval augmented language models for medical question answering

    Gongbo Zhang et al. “Leveraging long context in retrieval augmented language models for medical question answering”. In: npj Digital Medicine 8 (2025), p. 239. doi: 10.1038/s41746-025-01651-w . url: https://www.nature.com/articles/s41746-025-01651-w

    work page doi:10.1038/s41746-025-01651-w 2025

  14. [17]

    VISTA: A Rubric-based Visual Task Assessment Instruction-Specific Task Assessments

    Scale AI. VISTA: A Rubric-based Visual Task Assessment Instruction-Specific Task Assessments . url: https://scale.com/leaderboard/visual_language_understanding

    work page

  15. [18]

    PaperBench: A Rubric-based Benchmark for Scientific Paper Summarization

    Giulio Starace et al. “PaperBench: A Rubric-based Benchmark for Scientific Paper Summarization”. In: arXiv preprint arXiv:2504.01848v1 (2025). url: https://arxiv.org/html/2504.01848v1

    Pith/arXiv arXiv 2025

  16. [19]

    WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild

    Bill Yuchen Lin et al. “WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild”. In: arXiv preprint arXiv:2406.04770 (2024). url: https://doi.org/10.48550/arXiv. 2406.04770

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.04770 2024

  17. [20]

    Autonomous medical evaluation for guideline adherence of large language models

    Ethan Fast et al. “Autonomous medical evaluation for guideline adherence of large language models”. In: npj Digital Medicine 7 (2024), p. 358. doi: 10.1038/s41746- 024- 01356- 6. url: https://doi. org/10.1038/s41746-024-01356-6

    doi:10.1038/s41746- 2024

  18. [21]

    MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs

    Ved Sirdeshmukh et al. “MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs”. In: arXiv preprint arXiv:2501.17399 (2025). url: https://doi.org/ 10.48550/arXiv.2501.17399

    Pith/arXiv arXiv doi:10.48550/arxiv.2501.17399 2025

  19. [22]

    Introducing GPT-5

    OpenAI. Introducing GPT-5. Accessed: 2025-08-15. 2025. url:https://openai.com/index/introducing- gpt-5/. 13

    work page 2025

  20. [295]

    url: https://doi.org/10.1001/jamainternmed.2024.0295

    work page arXiv 2024

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.