Pith. sign in

REVIEW 5 major objections 5 minor 69 references

A Scalable Approach to Benchmarking the In-Conversation Differential Diagnostic Accuracy of a Health AI

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A conversational health AI reaches 81.8% top-one diagnostic accuracy by interviewing AI-simulated patients built from 400 validated clinical vignettes.

desk verdict The benchmark framework is a plausible, modest step, but the headline 'significantly outperforming' is unsupported because the comparator scores in Section 5.1 have no stated measurement protocol. read the letter →

arxiv 2412.12538 v1 pith:AYDIEMMV submitted 2024-12-17 cs.AI cs.HC

Deep Bhatt , Surya Ayyagari , Anuruddh Mishra This is my paper · ORCID
classification cs.AIcs.HC
keywords diagnosticaccuracyhealthAIbenchmarkingclinicalvignettespatientactorsconversationaldifferentialdiagnosissymptomcheckerslargelanguagemodelsinmedicinereferral
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the right way to evaluate a conversational health AI is to let it conduct a live interview with a simulated patient, rather than feeding it a written case or a multiple-choice exam. To demonstrate this, the authors build AI patient actors from 400 validated clinical vignettes across 14 specialties, have their own health AI August converse with them, and ask physician judges to score the diagnoses August states aloud. On this protocol August reaches 81.8% top-one accuracy, 85.0% top-two accuracy, 95.8% correct specialist referrals, and a mean of 16 questions per consultation. The paper reports that these figures exceed published symptom-checker scores and the scores of three family physicians, and it offers the protocol as a reproducible, scalable benchmark for a field that has lacked one.

What carries the argument

The load-bearing object is the AI patient actor: an LLM initialized with a validated clinical vignette and prompted to speak in simple lay language, present the most distressing symptom first, volunteer nothing beyond what is asked, and stay strictly in character. This turns a static written case into a live, repeatable consultation that can be run at scale. The second mechanism is the matching rubric [41], adapted to be stricter by excluding near-matches, umbrella terms, and conditions with mere symptomatic overlap; human physician judges apply it to the diagnoses the health AI explicitly states in the conversation. Together these convert 'did the AI get the right answer' into 'did the AI elicit the right information and name the right condition from a genuine back-and-forth history.'

What would settle it

Re-run Avey and Ada on the same 400 vignettes through exactly the AI-patient-actor conversations and the same physician-judge rubric; if their top-one accuracies do not reproduce near 67.5% and 54.2%, or if August's margin shrinks, the outperformance claim is an artifact of mismatched baselines. A second check is to have trained humans act out the same vignettes with August: if top-one accuracy falls well below 81.8%, the LLM patient actors are inflating the score.

Watch

Extended reading notes

Core claim

The central discovery, on the paper's own terms, is that a health AI built for conversation can be held to a standardized diagnostic standard: when August interviews an LLM-based patient actor that is bound to the facts of a validated clinical vignette, a physician judge can reliably determine whether the AI's stated diagnosis matches the gold standard. Across 400 cases August's first diagnosis matched in 327 cases (81.8%) and one of its top two matched in 340 cases (85.0%). The paper also reports 95.8% accuracy in recommending the right specialist, and 47% fewer questions than symptom checkers (16 vs 29 on average). These numbers are presented as evidence both that August performs well and that the benchmark itself works as a scalable evaluation method for conversational diagnostic AI.

Load-bearing premise

The headline comparison assumes that the symptom-checker and doctor scores were produced under the same conversational protocol and the same matching rules; the paper describes August's protocol in detail but gives no such description for the baselines.

Editorial extensions

If this is right

  • A conversational diagnostic AI can reach 81.8% top-one accuracy on validated vignettes while averaging 16 questions, suggesting that scripted short interviews can be both efficient and diagnostically productive.
  • Other health AI developers can run the same patient-actor protocol on the same vignette corpus, making head-to-head diagnostic comparisons possible without recruiting human standardized patients.
  • The choice of matching rubric materially changes reported accuracy; the paper's stricter exclusions make its 81.8% figure conservative relative to rubrics that count near-matches.
  • The 95.8% specialist-referral accuracy suggests the benchmark also captures triage quality, not just diagnosis naming, which matters for the AI's real-world role.
  • If the benchmark reflects real consultation skill, it implies a carefully prompted conversational AI can outperform practicing family physicians on vignette-based differential diagnosis in this specific setting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The comparative claim is only as strong as the comparators' protocol: the paper does not describe how the Avey, Ada, or physician baselines were measured, so a fair reading treats the outperformance as provisional until those systems are re-run under the same patient-actor and judge rules.
  • Because the authors' own patient actors answer only when asked and use grammatically correct English, the 81.8% figure is likely an upper bound for performance with real users; a small study with human standardized patients on the same vignettes would test this directly.
  • The judge step still depends on human physicians, so the framework's scalability claim will fully stand only once the proposed automated judge reproduces human-rubric scores on a held-out set of conversations.
  • If the vignette corpus and automated judge are released, the benchmark becomes a regression suite: future health AIs could be compared on identical conversations and identical scoring, which would give regulators and purchasers a common yardstick.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a conversational benchmarking framework for health AI that combines 400 clinical vignettes from Hammoud et al. with LLM-based patient actors and human judges. It applies this framework to August, a health AI developed by the authors' own company, and reports a top-one diagnostic accuracy of 327/400 (81.8%), top-two accuracy of 340/400 (85.0%), specialist referral accuracy of 383/400 (95.8%), and a mean of 16 questions per consultation. The authors claim that August significantly outperforms two symptom checkers (Avey 67.5%, Ada 54.2%) and three family medicine physicians (49.7%, 61.3%, 72.5%). The paper also describes the benchmark design, outcome measures, strengths and limitations, and future plans for real-world validation.

Significance. If the methodology were independent and the comparator measurements were valid, the framework would address a genuine gap: there is no widely accepted, scalable method for evaluating conversational diagnostic AI. The use of 400 validated clinical vignettes, explicit matching criteria, and internally consistent tabulations are useful starting points. The paper also takes some care to report limitations, including the fidelity of AI patient actors and the absence of physical examination data. However, the empirical contribution as presented does not support the headline comparative claim: the comparator scores lack a described measurement protocol, no confidence intervals or significance tests are provided, and the benchmark has closed-loop features that favor the evaluated system. The framework itself is potentially valuable but is not yet independently reproducible from the manuscript.

major comments (5)
  1. [Section 5.1] The central claim that August 'significantly outperforms' Avey, Ada, and the three clinicians is unsupported because the manuscript never states how the comparator scores were measured. It is not reported whether Avey, Ada, and the clinicians were run on the same 400 vignettes, through the same AI patient actors, with the same conversational input format, the same judge instructions, or the same modified Gilbert matching rubric; the number of clinicians and their testing conditions are also absent. If these figures were taken from Hammoud et al. or Gilbert et al. rather than re-measured under this benchmark's protocol, then the comparison is invalid. The authors must either report the full comparator protocol, re-measure the comparators under the benchmark's exact conditions, or remove the comparative superiority claim.
  2. [Section 5.1 and Tables 1-2] No confidence intervals or significance tests appear anywhere in the paper, so the word 'significantly' is not backed by statistical inference. With 400 cases, a binomial confidence interval should accompany the overall 81.8% and 85.0% estimates, and pairwise comparisons to each comparator should be tested. This is especially important because several specialty rows are based on very small counts (e.g., Dermatology 13/13, Hematology 16/23), where the noise floor is high and the reported percentages cannot support strong conclusions.
  3. [Section 3.2] The benchmark is closed-loop in a way that threatens the comparative claim. The patient actors' communication style was explicitly derived from 'patterns from anonymized conversations between a health AI and actual users,' and the health AI in question appears to be August itself. This means the test distribution was shaped by August's own interaction patterns, potentially giving August an advantage over comparators that were not developed or tuned under the same conditions. The authors should construct patient actors independently of the evaluated system, or at minimum report a sensitivity analysis with patient actors not informed by August's conversations.
  4. [Section 3.3] The scoring rubric is a vendor-modified version of the Gilbert et al. criteria, but the paper does not report inter-rater reliability among the human judges, whether the judges were blinded to the identity of the system, or how the modified rubric affects comparability with previously published symptom-checker scores. Because August's outputs were judged under this modified rubric while external comparator scores may come from a different rubric, observed differences can be an artifact of the scoring criteria. The authors should provide inter-rater reliability statistics, describe judge blinding, and demonstrate that the modified rubric does not change the rank ordering of systems.
  5. [Sections 3 and 9] The paper claims a 'standardized and scalable framework' and states an intention to make benchmarking accessible to others, but it does not provide the patient-actor prompts, the judge instructions, the vignette selection criteria, or the code used to run the benchmark. The data availability statement only directs readers to a contact email. Without these artifacts, the framework is not reproducible and the scalability claim cannot be independently assessed. The authors should release the benchmark artifacts or provide a detailed specification sufficient for independent reimplementation.
minor comments (5)
  1. [References] References [50] and [63] are the same work (Hammoud et al., JMIR AI 2024) and should be merged into a single citation.
  2. [Section 1.1] The sentence about symptom checkers relying on 'generative adversarial networks' is not accurate for the systems under discussion and is not supported by the cited literature; this should be corrected or removed.
  3. [Section 4.2 and Table 4] The 'Correct Specialty Identified' outcome lacks an explicit definition of what counts as the correct specialty and how the referral was judged; Table 4 also does not define 'Common' versus 'Less Common' incidence or report case counts for those subgroups.
  4. [Abstract and Section 7] The abstract's unqualified 'diagnostic accuracy' and 'real-world impact' wording should be tempered, since Section 7 concedes that the patient actors do not capture incomplete or inaccurate patient input, linguistic diversity, and physical examination data; the results should be described as accuracy on simulated conversational vignettes.
  5. [Throughout] There are minor language issues, including 'there exists no standardized and scalable framework' in the abstract, 'AIs training data' in Section 1, and 'we aim examine' in Section 4; a careful proofread is needed.

Circularity Check

1 steps flagged · score 4.0 of 10

August's 81.8% accuracy is grounded in external vignettes, but the benchmark's patient-actor communication style is explicitly built from the same health AI family as the system under test, creating a self-referential conversational test distribution; the comparator claims are an evidentiary gap rather than a circular step.

  1. self definitional [Section 3.2 (Patient-Actor Design), in the context of Section 2 (Background) where August is identified as the health AI under test]
    "Drawing from our internal research, we incorporated patterns from anonymized conversations between a health AI and actual users to inform the patient-actor’s communication style. The AI patient-actor was then precisely prompted to follow these communication patterns, ensuring consistent and realistic patient interactions with the health AI."

    The benchmark evaluates August, the paper's own health AI, by conversing with an LLM-based patient actor. That actor's communication style is constructed from anonymized conversations of a health AI with actual users; in the paper, the health AI being evaluated is August. The test-time input distribution is therefore derived from the target system's own conversational behavior. Accuracy on this distribution can reflect that the patient actor was built to talk the way August already talks, making the novel conversational component of the benchmark a self-consistency check rather than an independent, externally specified patient simulation. This is circular by construction: the test input is defined using the system under test.

full rationale

The core accuracy number, 81.8% (327/400), is not circular in isolation: the vignettes are external (Hammoud et al.), the gold-standard diagnoses come from those vignettes, and human physicians applied a stated matching rubric adapted from Gilbert et al. There is no load-bearing self-citation chain; the rubric and vignette sources are external prior work, not author self-citations. The main circularity is the patient-actor design in Section 3.2: the conversational input distribution is built from anonymized conversations of the same health AI family as August, so the evaluation's novel layer is shaped by the system under test. This does not force the 81.8% value, because many other factors determine the score, but it is a genuine self-reference in the measurement chain. Separately, Section 5.1 reports comparator accuracies for Avey, Ada, and three clinicians without any stated measurement protocol, so a reader cannot verify that these baselines were run on the same 400 vignettes, through the same patient actors, under the same modified Gilbert rubric. That is an unsupported comparability assumption rather than a circular derivation, so I do not count it as a circular step. Section 7's own limitations, including grammatically correct English and the absence of real patient unpredictability, further limit external validity but are not circularity. Overall: partial circularity from the patient-actor construction, with independent content in the core accuracy measurement; score 4.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The benchmark introduces no fitted numeric constants, but the difficulty of the test distribution is set by hand-authored choices that function like free parameters: patient-actor response rules derived from August's own user conversations (Section 3.2), a non-random selection of 400 vignettes from the Hammoud et al. database (Section 3.1), and a vendor-authored modification of the Gilbert et al. matching rubric including a 'Direct Causation' match rule (Section 3.3). The load-bearing domain assumptions are that AI patient actors emulate real patients, that the comparator baselines were measured under comparable conditions, and that the selected vignettes are representative; the authors themselves qualify the first two in Section 7. No new physical or scientific entities are invented; the patient actor is a tool and August is the product under test.

free parameters (3)
  • Patient-actor behavioral rules = n/a
    Hand-authored rules in Section 3.2 (present the most distressing symptom first, answer only when asked, stick strictly to the vignette) set test difficulty; they were drawn from August's own anonymized conversations and iterated until outputs adhered, so they act as a fitted tuning of the test distribution to the system under test.
  • Vignette selection (400 from the Hammoud et al. database) = 400 of unspecified total
    Selection 'for their diversity' (Section 3.1) with no sampling or exclusion protocol; retaining easier or more classic presentations would inflate measured accuracy.
  • Matching rubric modifications = n/a
    The vendor authors modified Gilbert et al.'s criteria (Section 3.3), including a 'Direct Causation' match rule that counts Urethritis as matched by Gonorrhoea; the rubric determines what counts as correct and is authored by the vendor whose product is scored.
assumptions (3)
  • domain assumption AI patient actors faithfully emulate real patient conversational behavior
    The entire benchmark rests on Section 3.2's claim that an LLM prompted with a vignette 'simulate[s] conversations as a real patient would'; Section 7 concedes patient actors reliably provide pertinent positives and negatives on prompting, unlike real patients, so the difficulty distribution is likely easier than reality.
  • domain assumption Baseline comparators were measured under conditions comparable to August's
    Section 5.1's comparison to Avey, Ada Health, and three clinicians assumes the same vignettes, same input format, and same matching rubric; no protocol is given, so the comparison could be apples-to-oranges.
  • domain assumption The 400 selected vignettes are representative of diagnostic practice
    Section 3.1 selects vignettes 'for their diversity' without a sampling or exclusion protocol; the paper's own Section 7 notes vignette homogenization limits generalizability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Scalable Approach to Benchmarking the In-Conversation Differential Diagnostic Accuracy of a Health AI." pith.science (2026). https://pith.science/paper/AYDIEMMV

@misc{pith2026241212538,
  author       = {Pith},
  title        = {Pith review of: A Scalable Approach to Benchmarking the In-Conversation Differential Diagnostic Accuracy of a Health AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AYDIEMMV}},
  note         = {Machine review of arXiv:2412.12538}
}
read the original abstract

Diagnostic errors in healthcare persist as a critical challenge, with increasing numbers of patients turning to online resources for health information. While AI-powered healthcare chatbots show promise, there exists no standardized and scalable framework for evaluating their diagnostic capabilities. This study introduces a scalable benchmarking methodology for assessing health AI systems and demonstrates its application through August, an AI-driven conversational chatbot. Our methodology employs 400 validated clinical vignettes across 14 medical specialties, using AI-powered patient actors to simulate realistic clinical interactions. In systematic testing, August achieved a top-one diagnostic accuracy of 81.8% (327/400 cases) and a top-two accuracy of 85.0% (340/400 cases), significantly outperforming traditional symptom checkers. The system demonstrated 95.8% accuracy in specialist referrals and required 47% fewer questions compared to conventional symptom checkers (mean 16 vs 29 questions), while maintaining empathetic dialogue throughout consultations. These findings demonstrate the potential of AI chatbots to enhance healthcare delivery, though implementation challenges remain regarding real-world validation and integration of objective clinical data. This research provides a reproducible framework for evaluating healthcare AI systems, contributing to the responsible development and deployment of AI in clinical settings.

Figures

Figures reproduced from arXiv: 2412.12538 by the authors.

Figure 1
Figure 1. Mayo Clinic’s Symptom Checker. Online symptom checkers typically require users to input symptoms using specific terminology that aligns with the designers’ expectations. This rigid input method often means that if users describe their symptoms differently—perhaps using layman’s terms, vaguely describing a sequence of symptoms or expressing multiple symptoms simultaneously — the system may fail to interpret their con… view at source ↗
Figure 2
Figure 2. Schematic of an Interaction to assess the diagnostic accuracy of health AIs. It assess the AIs information-gathering ability through conversation and its capacity to generate accurate diagnostic hypotheses based solely on subjective patient information, mimicking initial consultations or remote telehealth encounters. 3.1. Selection of Clinical Vignettes The selection of representative clinical vignettes is crucial f… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 50 canonical work pages

  1. [1]

    The theory and practice of clinical decision-making

    Croskerry P. The theory and practice of clinical decision-making. Can J Anesth. 2005;52(6):R1-R8. doi:10.1007/BF03023077

    work page doi:10.1007/bf03023077 2005

  2. [2]

    Online health: untangling the web

    McDaid D, Park A-L. Online health: untangling the web. 2011

    work page 2011

  3. [3]

    Changes in rates of autopsy-detected diagnostic errors over time: a system- atic review

    Shojania KG, Burton EC, McDonald KM, Gold- man L. Changes in rates of autopsy-detected diagnostic errors over time: a system- atic review. JAMA. 2003;289(21):2849-2856. doi:10.1001/jama.289.21.2849

    work page doi:10.1001/jama.289.21.2849 2003

  4. [4]

    Types and origins of diagnostic errors in primary care set- tings

    Singh H, Giardina TD, Meyer AND, Forjuoh SN, Reis MD, Thomas EJ. Types and origins of diagnostic errors in primary care set- tings. JAMA Intern Med . 2013;173(6):418-425. doi:10.1001/jamainternmed.2013.2777

    work page arXiv 2013

  5. [5]

    Diagnostic errors in hospital- ized adults who died or were transferred to intensive care

    Auerbach AD, Lee TM, Hubbard CC, et al; UPSIDE Research Group. Diagnostic errors in hospital- ized adults who died or were transferred to intensive care. JAMA Intern Med. 2024;184(2):164-

    work page 2024

  6. [6]

    National Academies Press; December 29, 2015

    Balogh EP, Miller BT, Ball JR, eds; Improving Diagnosis in Health Care. National Academies Press; December 29, 2015. doi:10.17226/21794

    doi:10.17226/21794 2015

  7. [7]

    Diagnostic errors in the emergency depart- ment: a systematic review

    Newman-Toker DE, Peterson SM, Badihian S, et al. Diagnostic errors in the emergency depart- ment: a systematic review. Agency for Health- care Research and Quality. December 2022 re- port No.:22(23)-EHC043. Accessed November 15, 2024

    work page 2022

  8. [8]

    The ef- fect of Dr Google on doctor-patient encounters in primary care: a quantitative, observational, cross-sectional study

    Van Riel N, Auwerx K, Debbaut P, et al. The ef- fect of Dr Google on doctor-patient encounters in primary care: a quantitative, observational, cross-sectional study. BJGP Open . 2017;1:bjg- popen17X100833

    work page 2017

Show all 69 references
  1. [9]

    Health Online 2013

    Fox S, Duggan M. Health Online 2013. Internet and American life project. Pew Research Center and California Health Care Foundation, 2013:4

    2013

  2. [10]

    Assessment of Diagnosis and Triage in Validated Case Vignettes Among Nonphysicians Before and After Internet Search

    Levine DM, Mehrotra A. Assessment of Diagnosis and Triage in Validated Case Vignettes Among Nonphysicians Before and After Internet Search. JAMA Network Open. 2021;4(3):e213287

    2021

  3. [11]

    A random- ized controlled trial of online symptom search- ing to inform diagnosis

    Martin SS, Quaye E, Schultz S, et al. A random- ized controlled trial of online symptom search- ing to inform diagnosis

  4. [12]

    Benchmarking triage capability of symptom checkers against that of medical laypersons: survey study

    Schmieding ML, M¨orgeli R, Schmieding MAL, et al. Benchmarking triage capability of symptom checkers against that of medical laypersons: survey study. Journal of Medical Internet Re- search. 2021;23(3):e24475

    2021

  5. [13]

    Should you search the Internet for information about your acute symptoms? Telemed J E Health

    North F, Ward WJ, Varkey P, et al. Should you search the Internet for information about your acute symptoms? Telemed J E Health . 2012;18:213-8

    2012

  6. [14]

    Internet health infor- mation seeking and the patient-physician re- lationship: A systematic review

    Tan SS, Goonawardene N. Internet health infor- mation seeking and the patient-physician re- lationship: A systematic review. J Med Internet Res. 2017;19:e9. doi:10.2196/jmir.5729

    2017 doi

  7. [15]

    KFF Health Mis- information Tracking Poll: Artificial In- telligence and Health Information

    Kaiser Family Foundation. KFF Health Mis- information Tracking Poll: Artificial In- telligence and Health Information . Pub- lished: August 15, 2024. Accessed Novem- ber 11, 2024. https://www.kff.org/health- misinformation-and-trust/poll-finding/kff- health-misinformation-trac...

    2024

  8. [16]

    Appropriateness of Arti- ficial Intelligence Chatbots in Diabetic Foot Ulcer Management

    Shiraishi M, Lee H, Kanayama K, Moriwaki Y, Okazaki M. Appropriateness of Arti- ficial Intelligence Chatbots in Diabetic Foot Ulcer Management. The International Journal of Lower Extremity Wounds . 2024. doi:10.1177/15347346241236811

    2024 doi

  9. [17]

    GPT-based chatbot tools are still unreliable in the man- agement of prosthetic joint infections

    Sambri A, Fiore M, Tedeschi S, et al. GPT-based chatbot tools are still unreliable in the man- agement of prosthetic joint infections. Mus- culoskeletal Surgery. 2024. doi:10.1007/s12306- 024-00846-w

    2024 doi

  10. [18]

    Ethical concerns and re- sponsible use of ChatGPT in healthcare

    Dhane AS, Sarode SC. Ethical concerns and re- sponsible use of ChatGPT in healthcare. Oral Oncology. 2023. doi:10.1016/j.oor.2023.100113. Deep Bhatt, Surya Ayyagari and Anuruddh Mishra December 18, 2024 16

    2023

  11. [19]

    Protocol For Human Evaluation of Artificial Intelligence Chat- bots in Clinical Consultations

    Chiu E, Chung TW. Protocol For Human Evaluation of Artificial Intelligence Chat- bots in Clinical Consultations. 2024. doi:10.1101/2024.03.01.24303593

    2024 doi

  12. [20]

    Balancing Innovation and Professionalism: The Emerging Role of AI-Powered Chatbots in Medical Consultation

    Itelman E, Golovchiner G, Barsheshet A, et al. Balancing Innovation and Professionalism: The Emerging Role of AI-Powered Chatbots in Medical Consultation. Heart Rhythm. 2024. doi:10.1016/j.hrthm.2024.04.010

    2024 doi

  13. [21]

    Azmin SA, Sen A, Bain C. Evaluating the Po- tential and Pitfalls of AI-Powered Conversa- tional Agents as Human-like Virtual Health Carers in the Remote Management of Non- Communicable Diseases: A Scoping Review (Preprint). Journal of Medical Internet Research

  14. [22]

    Chatbots in Health Care: Connecting Patients to Information: Emerging Health Technologies

    Clark M, Bailey S; Authors. Chatbots in Health Care: Connecting Patients to Information: Emerging Health Technologies . [Internet]. Ottawa (ON): Canadian Agency for Drugs and Technologies in Health; 2024 Jan. Available from: https://www.ncbi.nlm.nih.gov/books/NBK602381/

    2024

  15. [23]

    Un- derstanding Large Language Models

    Sreerakuvandana S, Pappachan P, Arya V. Un- derstanding Large Language Models. Advances in Computational Intelligence and Robotics Book Series. 2024. doi:10.4018/979-8-3693-3860- 5.ch001

    2024 doi

  16. [24]

    The Breakthrough of Large Language Models Re- lease for Medical Applications: 1-Year Timeline and Perspectives

    Cascella M, Semeraro F, Montomoli J, et al. The Breakthrough of Large Language Models Re- lease for Medical Applications: 1-Year Timeline and Perspectives. Journal of Medical Systems

  17. [25]

    The future landscape of large language models in medicine

    Clusmann J, Kolbinger FR, Muti HS, et al. The future landscape of large language models in medicine. Communications Medicine. 2023. doi:10.1038/s43856-023-00370-1

    2023 doi

  18. [26]

    Large Lan- guage Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study

    Wilhelm TI, Roos J, Kaczmarczyk R. Large Lan- guage Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study. Journal of Medical Internet Research . 2023. doi:10.2196/49324

    2023 doi

  19. [27]

    Large lan- guage models for science and medicine

    Telenti A, Auli M, Hie BL, et al. Large lan- guage models for science and medicine. Eu- ropean Journal of Clinical Investigation. 2024. doi:10.1111/eci.14183

    2024 doi

  20. [28]

    Evaluation of large language models in breast can- cer clinical scenarios: A comparative analy- sis based on ChatGPT-3.5, ChatGPT-4.0, and Claude2

    Deng L, Wang T, Zhai Y, et al. Evaluation of large language models in breast can- cer clinical scenarios: A comparative analy- sis based on ChatGPT-3.5, ChatGPT-4.0, and Claude2. International Journal of Surgery. 2024. doi:10.1097/js9.0000000000001066

    2024 doi

  21. [29]

    Towards A Deep Learning Question-Answering Specialized Chat- bot for Objective Structured Clinical Examina- tions

    Zini J, Rizk Y, Awad M, Antoun J. Towards A Deep Learning Question-Answering Specialized Chat- bot for Objective Structured Clinical Examina- tions. International Joint Conference on Neural Network. 2019. doi:10.1109/IJCNN.2019.8851729

    2019

  22. [30]

    Chatbots for Symptom Screening and Patient Education: A Pilot Study on Patient Acceptability in Autoim- mune Inflammatory Diseases

    Tan TC, Roslan NE, Li JW, et al. Chatbots for Symptom Screening and Patient Education: A Pilot Study on Patient Acceptability in Autoim- mune Inflammatory Diseases. Journal of Medi- cal Internet Research. 2023. doi:10.2196/49239

    2023 doi

  23. [31]

    Diagnostic reasoning: where we’ve been, where we’re going

    Monteiro SM, Norman G. Diagnostic reasoning: where we’ve been, where we’re going. Teach- ing and Learning in Medicine . 2013;25 Suppl 1:S26–S32. doi:10.1080/10401334.2013.842911

    2013

  24. [32]

    Clinical decision making

    Berman S. Clinical decision making. In: Ba- jaj L, Hambidge SJ, Kerby G, Nyquist A-C, edi- tors. Berman’s Pediatric Decision Making. 5th ed. 2011. p. 1–6. doi:10.1016/B978-0-323-05405- 8.00010-3

    2011 doi

  25. [33]

    what is likely to happen

    Croft P, Altman DG, Deeks JJ, et al. The science of clinical practice: disease diagnosis or patient prognosis? Evidence about “what is likely to happen” should shape clinical practice. BMC Med. 2015;13:20. doi:10.1186/s12916-014-0265-4

    2015 doi

  26. [34]

    Medicine information needs of patients: the relationships between informa- tion needs, diagnosis and disease

    Duggan C, Bates I. Medicine information needs of patients: the relationships between informa- tion needs, diagnosis and disease. BMJ Quality & Safety. 2008;17:85-89

    2008

  27. [35]

    Diagnostic rea- soning prompts reveal the potential for large language model interpretability in medicine

    Savage T, Nayak A, Gallo R, et al. Diagnostic rea- soning prompts reveal the potential for large language model interpretability in medicine. Deep Bhatt, Surya Ayyagari and Anuruddh Mishra December 18, 2024 17 NPJ Digit Med. 2024;7(1):20. doi:10.1038/s41746- 024-01010-1

    2024 doi

  28. [36]

    Evaluation of symptom checkers for self diagno- sis and triage: audit study

    Semigran HL, Linder JA, Gidengil C, Mehrotra A. Evaluation of symptom checkers for self diagno- sis and triage: audit study. BMJ. 2015;351:h3480. doi:10.1136/bmj.h3480

    2015 doi

  29. [37]

    Patients don’t present with five choices: an alternative to multiple-choice tests in assessing physicians’ competence

    Veloski JJ, Rabinowitz HK, Robeson MR, Young PR. Patients don’t present with five choices: an alternative to multiple-choice tests in assessing physicians’ competence. Academic Medicine . 1999;74(5):539–546. doi:10.1097/00001888- 199905000-00022

    1999 doi

  30. [38]

    Am- bient artificial intelligence scribes to alle- viate the burden of clinical documentation

    Tierney AA, Gayre G, Hoberman B, et al. Am- bient artificial intelligence scribes to alle- viate the burden of clinical documentation. NEJM Catal. Published online February 21, 2024. doi:10.1056/CAT.23.0404

    2024 doi

  31. [39]

    Clinical reasoning assessment meth- ods: a scoping review and practical guidance

    Daniel M, Rencic J, Durning SJ, et al. Clinical reasoning assessment meth- ods: a scoping review and practical guidance. Acad Med . 2019;94(6):902-912. doi:10.1097/ACM.0000000000002618

    2019 doi

  32. [40]

    Accuracy of a gen- erative artificial intelligence model in a complex diagnostic challenge

    Kanjee Z, Crowe B, Rodman A. Accuracy of a gen- erative artificial intelligence model in a complex diagnostic challenge. JAMA. 2023;330(1):78-80. doi:10.1001/jama.2023.8288

    2023

  33. [41]

    How accurate are digital symptom assessment apps for suggesting conditions and urgency advice? A clinical vignettes comparison to GPs

    Gilbert S, Mehl A, Baluch A, et al. How accurate are digital symptom assessment apps for suggesting conditions and urgency advice? A clinical vignettes comparison to GPs. BMJ Open. 2020;10(12):e040269. doi:10.1136/bmjopen- 2020-040269

    2020 doi

  34. [42]

    Chat- GPT influence on medical decision-making, bias, and equity: a randomized study of clin- icians evaluating clinical vignettes

    Goh E, Bunning B, Khoong E, et al. Chat- GPT influence on medical decision-making, bias, and equity: a randomized study of clin- icians evaluating clinical vignettes. medRxiv. Preprint posted online November 27, 2023. doi:10.1101/2023.11.24.23298844

    2023 doi

  35. [43]

    Com- paring physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum

    Ayers JW, Poliak A, Dredze M, et al. Com- paring physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Intern Med . 2023;183(6):589-596. doi:10.1001/jamainternmed.2023.1838

    2023

  36. [44]

    Patient- centered decision making and health care out- comes: an observational study

    Weiner SJ, Schwartz A, Sharma G, et al. Patient- centered decision making and health care out- comes: an observational study. Ann Intern Med. 2013;158(8):573-579. doi:10.7326/0003-4819-158- 8-201304160-00001

    2013 doi

  37. [45]

    To- wards conversational diagnostic AI

    Tu T, Palepu A, Schaekermann M, et al. To- wards conversational diagnostic AI. arXiv. Preprint posted online January 11, 2024. doi:10.48550/arXiv.2401.05654

    2024 doi arXiv

  38. [46]

    Large language models in medicine: the poten- tials and pitfalls: a narrative review

    Omiye JA, Gui H, Rezaei SJ, Zou J, Daneshjou R. Large language models in medicine: the poten- tials and pitfalls: a narrative review. Ann Intern Med. 2024;177(2):210-220. doi:10.7326/M23-2772

    2024 doi

  39. [47]

    Appropriateness of cardiovas- cular disease prevention recommendations ob- tained from a popular online chat-based ar- tificial intelligence model

    Sarraju A, et al. Appropriateness of cardiovas- cular disease prevention recommendations ob- tained from a popular online chat-based ar- tificial intelligence model. JAMA. 2023;329:10. doi:10.1001/jama.2023.1044

    2023

  40. [48]

    Young Adults’ Perspectives on the Use of Symptom Checkers for Self-Triage and Self-Diagnosis: Qualitative Study

    Aboueid S, Meyer S, Wallace JR, Mahajan S, Chaurasia A. Young Adults’ Perspectives on the Use of Symptom Checkers for Self-Triage and Self-Diagnosis: Qualitative Study. JMIR Pub- lic Health and Surveillance . 2021;7(1):e22637. doi:10.2196/22637

    2021 doi

  41. [49]

    Ensuring Fairness in Machine Learning to Advance Health Equity

    Rajkomar A, Hardt M, Howell MD, et al. Ensuring Fairness in Machine Learning to Advance Health Equity. Ann Intern Med . 2018;169(12):866–872. doi:10.7326/M18-1990

    2018 doi

  42. [51]

    Search Engines vs

    Cross S, Mourad A, Zuccon G, Koopman B. Search Engines vs. Symptom Checkers: A Comparison of their Effectiveness for On- line Health Advice. The Web Conference. 2021. doi:10.1145/3442381.3450140

    2021

  43. [52]

    ‘Next please!’ Psychological and practical consequences of an inconclusive diag- Deep Bhatt, Surya Ayyagari and Anuruddh Mishra December 18, 2024 18 nosis

    Hussain FA. ‘Next please!’ Psychological and practical consequences of an inconclusive diag- Deep Bhatt, Surya Ayyagari and Anuruddh Mishra December 18, 2024 18 nosis. British Journal of General Practice. 2020. doi:10.3399/BJGP20X707885

    2024 doi

  44. [53]

    The use of vignettes for conducting healthcare research

    Keane D, Lang A, Craven MP, Sharples S. The use of vignettes for conducting healthcare research

  45. [54]

    Do case vignettes accurately reflect antibiotic prescription? Infection Control and Hospital Epidemiology

    Lucet J-C, Nicolas-Chanoine M-H, Lefort A, et al. Do case vignettes accurately reflect antibiotic prescription? Infection Control and Hospital Epidemiology. 2011. doi:10.1086/661914

    2011 doi

  46. [55]

    Vignettes as research tools in global health communication: a systematic review of the literature from 2000 to 2020

    Riley AH, Critchlow E, Birkenstock L, et al. Vignettes as research tools in global health communication: a systematic review of the literature from 2000 to 2020. Jour- nal of Communication in Healthcare . 2021. doi:10.1080/17538068.2021.1945766

  47. [56]

    Com- munication with Diverse Patients: Addressing Culture and Language.Pediatric Clinics of North America

    O’Toole JK, Alvarado-Little W, Ledford CJW. Com- munication with Diverse Patients: Addressing Culture and Language.Pediatric Clinics of North America. 2019. doi:10.1016/J.PCL.2019.03.006

    2019 doi

  48. [57]

    Ex- ploring the role of communication barriers in healthcare

    Shrivastava SR, Shrivastava PS, Ramasamy J. Ex- ploring the role of communication barriers in healthcare. South African Family Practice. 2016. doi:10.1080/20786190.2014.977039

    2016

  49. [58]

    Recognition of patients with medically un- explained physical symptoms by family physi- cians: results of a focus group study.BMC Family Practice

    den Boeft M, Huisman D, van der Wouden JC, et al. Recognition of patients with medically un- explained physical symptoms by family physi- cians: results of a focus group study.BMC Family Practice. 2016. doi:10.1186/S12875-016-0451-X

    2016 doi

  50. [59]

    Do Patients Under- stand? The Permanente Journal

    Graham S, Brookey J. Do Patients Under- stand? The Permanente Journal . 2008. doi:10.7812/TPP/07-144

    2008 doi

  51. [60]

    The importance of the history and physical in diagnosis

    Muhrer JC. The importance of the history and physical in diagnosis. Gender & Development

  52. [61]

    In- ternational variations in primary care physi- cian consultation time: a systematic review of 67 countries

    Irving G, Neves AL, Dambha-Miller H, et al. In- ternational variations in primary care physi- cian consultation time: a systematic review of 67 countries. BMJ Open . 2017;7:e017902. doi:10.1136/bmjopen-2016-017902

    2017 doi

  53. [62]

    Preva- lence of Occupational Burnout among Resident Doctors Working in Public Sector Hospitals in Mumbai

    Dhusia AH, Dhaimade PA, Jain AA, et al. Preva- lence of Occupational Burnout among Resident Doctors Working in Public Sector Hospitals in Mumbai. Indian Journal of Community Medicine. 2019;44(4):352–356. doi:10.4103/ijcm.IJCM7819

    2019 doi

  54. [63]

    Eval- uating the Diagnostic Performance of Symp- tom Checkers: Clinical Vignette Study

    Hammoud M, Douglas S, Darmach M, et al. Eval- uating the Diagnostic Performance of Symp- tom Checkers: Clinical Vignette Study. JMIR AI. 2024;3:e46875. doi:10.2196/46875

    2024 doi

  55. [64]

    A systematic review of trustworthy and explainable artificial intelligence in healthcare: Assessment of quality, bias risk, and data fusion

    Albahri AS, Mohammed AF, Fadhel MA, et al. A systematic review of trustworthy and explainable artificial intelligence in healthcare: Assessment of quality, bias risk, and data fusion. Information Fusion . 2023. doi:10.1016/j.inffus.2023.03.008

    2023 doi

  56. [65]

    Evalu- ation framework to guide implementation of AI systems into healthcare settings

    Reddy S, Rogers WA, Makinen V.-P., et al. Evalu- ation framework to guide implementation of AI systems into healthcare settings. BMJ Health & Care Informatics. 2021. doi:10.1136/BMJHCI-2021- 100444

    2021 doi

  57. [66]

    What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Med- ical Exams

    Jin D, Pan E, Oufattole N, et al. What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Med- ical Exams. Applied Sciences. 2021;11(14):6421. doi:10.3390/APP11146421

    2021 doi

  58. [67]

    WHO and ITU establish benchmarking process for artificial intelligence in health

    Wiegand T, Krishnamurthy R, Kuglitsch M, et al. WHO and ITU establish benchmarking process for artificial intelligence in health. The Lancet. 2019;393(10191):2425–2427. doi:10.1016/S0140- 6736(19)30762-7. Deep Bhatt, Surya Ayyagari and Anuruddh Mishra December 18, 2024

    2019 doi

  59. [173]

    doi:10.1001/jamainternmed.2023.7347

    2023

  60. [2014]

    doi:10.1097/01.NPR.0000444648.20444.E6

  61. [2024]

    doi:10.1007/s10916-024-02045-3

    doi

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.