REVIEW 22 cited by
Towards Conversational Diagnostic AI
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
At the heart of medicine lies the physician-patient dialogue, where skillful history-taking paves the way for accurate diagnosis, effective management, and enduring trust. Artificial Intelligence (AI) systems capable of diagnostic dialogue could increase accessibility, consistency, and quality of care. However, approximating clinicians' expertise is an outstanding grand challenge. Here, we introduce AMIE (Articulate Medical Intelligence Explorer), a Large Language Model (LLM) based AI system optimized for diagnostic dialogue. AMIE uses a novel self-play based simulated environment with automated feedback mechanisms for scaling learning across diverse disease conditions, specialties, and contexts. We designed a framework for evaluating clinically-meaningful axes of performance including history-taking, diagnostic accuracy, management reasoning, communication skills, and empathy. We compared AMIE's performance to that of primary care physicians (PCPs) in a randomized, double-blind crossover study of text-based consultations with validated patient actors in the style of an Objective Structured Clinical Examination (OSCE). The study included 149 case scenarios from clinical providers in Canada, the UK, and India, 20 PCPs for comparison with AMIE, and evaluations by specialist physicians and patient actors. AMIE demonstrated greater diagnostic accuracy and superior performance on 28 of 32 axes according to specialist physicians and 24 of 26 axes according to patient actors. Our research has several limitations and should be interpreted with appropriate caution. Clinicians were limited to unfamiliar synchronous text-chat which permits large-scale LLM-patient interactions but is not representative of usual clinical practice. While further research is required before AMIE could be translated to real-world settings, the results represent a milestone towards conversational diagnostic AI.
Forward citations
Cited by 22 Pith papers
-
Conversational Medical AI: Ready for Practice
A real-world randomized trial found that a physician-supervised medical chatbot improved patient-rated clarity and satisfaction without overall safety failures.
-
Social Chain of Thought: A Multi-Agent Architecture Grounded in Medical Differential Diagnosis Methodology
A seven-round multi-agent LLM pipeline improves differential diagnosis recall on the Open-XDDx benchmark, with gains concentrated in the hardest cases.
-
Guideline-as-Oracle: Zero-Annotation Training of an Ophthalmic Telephone Triage Agent
A 70-row table compiled from ophthalmology guidelines can label 3,000 synthetic dialogues, and fine-tuning a 9B model on them improves agreement with an author-defined reference from 61.7% to 74.1% and emergent recall...
-
Active Learning for Neurosymbolic Program Synthesis
The abstract claims a new active learning technique, constrained conformal evaluation (tool SmartLabel), that finds the ground-truth program in 98% of benchmarks, but the delivered full text is a different paper, leav...
-
BEnchmarking LLMs for Ophthalmology (BELO) for Ophthalmological Knowledge and Reasoning
BELO is a new ophthalmology benchmark of 900 expert-checked multiple-choice questions with reasoning, used to evaluate six LLMs on accuracy and explanation quality.
-
BehaviorSFT: Behavioral Token Conditioning for Clinical Agents Across the Proactivity Spectrum
BehaviorSFT uses `reactive` and `proactive` control tokens to fine-tune clinical LLM agents, improving their scores on the authors' new BehaviorBench dataset, though the benchmark is AI-generated and minimally clinici...
-
MedBrowseComp: Benchmarking Medical Deep Research and Computer Use
A new benchmark of more than 1,000 multi-hop medical browsing questions shows that even the best deep-research and computer-use AI agents answer fewer than half correctly.
-
MM-Agent: LLM as Agents for Real-world Mathematical Modeling Problem
MM-Agent, a multi-stage LLM pipeline with a hierarchical modeling method library, is claimed to outperform prior agents and award-winning human solutions on a new 111-problem MCM/ICM-based mathematical modeling benchmark.
-
Advancing Conversational Diagnostic AI with Multimodal Reasoning
A multimodal state-aware version of the AMIE diagnostic AI outperformed primary care physicians on most axes in a blinded OSCE-style chat study with 105 scenarios.
-
CAMI: A Counselor Agent Supporting Motivational Interviewing through State Inference and Topic Exploration
CAMI's combination of client-state inference and topic-tree exploration improves motivational-interviewing counseling performance over four LLM-based baselines in simulated sessions.
-
Exploring the Inquiry-Diagnosis Relationship with Advanced Patient Simulators
A dialogue-strategy-trained patient simulator improves realism in AI medical consultations and shows that inquiry quality and diagnostic skill jointly limit diagnostic accuracy.
-
MATRIX: Multi-Agent simulaTion fRamework for safe Interactions and conteXtual clinical conversational evaluation
MATRIX combines a structured safety taxonomy, an LLM hazard judge, and a patient simulator to benchmark clinical dialogue agents, claiming expert-level hazard detection and revealing weak emergency handling in current LLMs.
-
A Novel Evaluation Benchmark for Medical LLMs: Illuminating Safety and Effectiveness in Clinical Domains
A new 2,069-item, 30-criterion benchmark of clinical scenarios shows six LLMs average 57.2%, with safety (54.7%) below effectiveness (62.3%) and a 13.3% drop in high-risk cases.
-
AI Agents for Conversational Patient Triage: Preliminary Simulation-Based Evaluation with Real-World EHR Data
A patient simulator built from EHR vignettes was rated consistent with those vignettes in 97.7% of 519 conversations by two clinicians, while the AI triage system's top three diagnoses contained the most likely diagno...
-
Embedding-Driven Diversity Sampling to Improve Few-Shot Synthetic Data Generation
Embedding-driven diversity sampling of few-shot examples improves downstream classification with synthetic clinical text over random and zero-shot baselines on CheXpert radiology reports.
-
Enhanced Large Language Models for Effective Screening of Depression and Anxiety
A synthetic-data pipeline and fine-tuned LLM system, EmoScan, screens for depressive and anxiety disorders from generated clinical interviews, outperforming GPT-4 on synthetic test data but only slightly on an externa...
-
RobustFT: Robust Supervised Fine-tuning for Large Language Models under Noisy Response
A noise-robust SFT framework that detects noisy responses via multi-expert LLM consensus, relabels them with context-enhanced reasoning, and filters low-confidence samples, improving LLM performance on five benchmarks.
-
The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomy
Medical agents should be scaled mainly by richer clinical environments and self-evolution loops, not parameter growth alone, under a three-level autonomy taxonomy.
-
Hallucinations in medical devices
AI hallucinations in medical devices are defined as plausible errors, either impactful or benign, to guide device evaluation.
-
A Scalable Approach to Benchmarking the In-Conversation Differential Diagnostic Accuracy of a Health AI
The authors report a scalable AI-patient-actor benchmark and claim their own chatbot August reaches 81.8% top-one diagnostic accuracy on 400 vignettes, but the evaluation is self-run and the comparison baselines are u...
-
Performance of a large language model-Artificial Intelligence based chatbot for counseling patients with sexually transmitted infections and genital diseases
A GPT-4-based STI counseling chatbot scored reasonably well in a small expert role-play evaluation, especially on information correctness and empathy, but showed redundancy and weaker non-STI diagnosis.
-
Multimodal Large Language Models for Medicine: A Comprehensive Survey
A comprehensive review cataloging medical MLLMs, their uses in report generation, diagnosis, and treatment, and the challenges of accuracy, hallucination, fairness, privacy, and deployment.
Discussion (0). Continue with ORCID to comment.