REVIEW 9 cited by
Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator
read the original abstract
Large Language Models (LLMs) have demonstrated remarkable proficiency in human interactions, yet their application within the medical field remains insufficiently explored. Previous works mainly focus on the performance of medical knowledge with examinations, which is far from the realistic scenarios, falling short in assessing the abilities of LLMs on clinical tasks. In the quest to enhance the application of Large Language Models (LLMs) in healthcare, this paper introduces the Automated Interactive Evaluation (AIE) framework and the State-Aware Patient Simulator (SAPS), targeting the gap between traditional LLM evaluations and the nuanced demands of clinical practice. Unlike prior methods that rely on static medical knowledge assessments, AIE and SAPS provide a dynamic, realistic platform for assessing LLMs through multi-turn doctor-patient simulations. This approach offers a closer approximation to real clinical scenarios and allows for a detailed analysis of LLM behaviors in response to complex patient interactions. Our extensive experimental validation demonstrates the effectiveness of the AIE framework, with outcomes that align well with human evaluations, underscoring its potential to revolutionize medical LLM testing for improved healthcare delivery.
Forward citations
Cited by 9 Pith papers
-
AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments
AgentClinic is a multimodal agent benchmark demonstrating that LLM diagnostic accuracy on MedQA drops to below one-tenth in sequential clinical simulations, with Claude-3.5 leading and large tool-use differences acros...
-
Patients With Personality: Realistic Patient Simulation through Controlled Diversity and Selective Disclosure
PWP generates realistic, diverse virtual patients via HEXACO personality parametrization, achieving clinician-rated realism close to human actors, wider behavioral range than baselines, and reduced oversharing while a...
-
Simulating Couple Conflict: Designing A Multi-Agent System for Therapy Training and Practice
A stateful multi-agent system simulates demand-withdraw couple conflicts across six stages for therapist training and outperforms prompt-based baselines in realism and state detection.
-
MoBayes: A Modular Bayesian Framework for Separating Reasoning from Language in Conversational Clinical Decision Support
MoBayes separates LLM language parsing from Bayesian probabilistic reasoning in conversational clinical decision support and reports performance gains over standalone frontier LLMs across multiple knowledge bases and ...
-
MoBayes: A Modular Bayesian Framework for Separating Reasoning from Language in Conversational Clinical Decision Support
Separating language from Bayesian reasoning lets cheap LLM sensors beat larger standalone LLM doctors on conversational diagnosis with controllable abstention and lower cost.
-
ClinConsensus: A Physician-Calibrated Benchmark for Evaluating Clinical Rubric Coverage in Chinese Medical LLMs
ClinConsensus, a Chinese medical benchmark with physician-calibrated thresholded rubric coverage, shows LLMs achieve 39.6–52.1% rubric accuracy but only 17.8–32.9% CACS@10, exposing a large coverage gap.
-
A Patient Simulation Framework for Risk Assessment of Conversational Healthcare AI: Evaluation of an Antidepressant Decision Aid
A patient simulator integrating medical, linguistic, and behavioral profiles exposes a monotonic performance decline in an antidepressant decision aid as simulated health literacy decreases.
-
Active Evidence-Seeking and Diagnostic Reasoning in Large Language Models for Clinical Decision Support
Multi-turn evidence seeking reduces LLM diagnostic accuracy by 12.75% and supporting-evidence quality by 24.36% versus full-context evaluation in a new OSCE-inspired benchmark across 468 cases and 15 models.
-
MoBayes: A Modular Bayesian Framework for Separating Reasoning from Language in Conversational Clinical Decision Support
BMBE separates LLM language handling from a standalone Bayesian diagnostic engine, producing calibrated selective diagnosis, a performance gap over frontier LLMs, and robustness to adversarial inputs.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.