Pith. sign in

REVIEW 4 major objections 5 minor 41 references

Are LLMs Ready to Assist Physicians? PhysAssistBench for Interactive Doctor-Patient-EHR Assistance

T0 review · 4 major / 5 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Leading LLMs still fail as physician assistants when they must coordinate knowledge, ambiguous patient talk, and precise EHR tool use in the same multi-turn encounter.

desk verdict Solid integrated assistance benchmark with real empirical bite; synthetic patients and a single LLM judge are real soft spots but do not erase the contribution. read the letter →

arxiv 2606.18613 v3 pith:KDSXSM53 submitted 2026-06-17 cs.CL cs.AI

Tianming Du , Peijie Yu , Sihan Shang , Danli Shi , My Linh Nguyen , Shengbo Gao , Guangyuan Li , Yinghong Yu
show 7 more authors
Yan Jiang Qianlong Zhao Behzad Bozorgtabar Shaoxiong Ji Jiazhen Pan Daniel Rueckert Jiancheng Yang
This is my paper
classification cs.CLcs.AI
keywords medicalLLMsphysicianassistanceEHRFHIRmulti-turninteractionagenticpatientsimplicitqueriesbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the realistic near-term job for medical language models is not to replace doctors but to assist them under oversight. That job is not the same as scoring well on medical quizzes, EHR lookup alone, or patient chat alone: a physician issues underspecified, multi-turn requests; the patient answers in colloquial, incomplete language; and the EHR demands exact structured tool calls. The authors introduce PhysAssistBench, built from real MIMIC-IV records via a pipeline that creates record-grounded “agentic patients,” yielding 324 bilingual four-turn sessions (1,296 turns) with turn-level rubrics. Across many strong models, mean turn scores look middling, but full-session reliability collapses—especially on multi-source data gathering, clinical reasoning, patient-interview turns, and abstract event anaphora—showing that coordination, not isolated skill, is the bottleneck.

What carries the argument

PhysAssistBench and its agentic-patient pipeline: a multi-agent process that plans clinically plausible four-turn arcs from static MIMIC-IV admissions, grounds patient dialogue and FHIR tools in the same record, filters unsupported cases, and scores each turn with physician-validated rubrics (including implicitness types NA/PE/AE and tasks IL/DG/CR/WU).

What would settle it

Run the same 324 sessions (or a clinician-held-out subset) with live physicians issuing the queries and real patients or live chart review for gold answers; if top models then sustain high Pass@Session at τ=0.75, or if human assistants score no better under the same rubrics, the claim that models specifically fail this coordination bottleneck would be undermined.

Watch

Extended reading notes

Core claim

Current leading LLMs are not yet reliable physician assistants in interactive doctor–patient–EHR settings: they must resolve implicit physician intent, interview ambiguous patients, issue grounded FHIR tool calls, and integrate evidence across sources in one multi-turn workflow, and they remain unreliable at that joint task even when they look strong on isolated clinical knowledge or tool use.

Load-bearing premise

That record-grounded synthetic patients with fixed interview replies, plus an automated rubric judge, are faithful enough stand-ins for real patient ambiguity and clinical correctness that low scores mainly reflect assistant failure rather than simulation or judge artifacts.

Editorial extensions

If this is right

  • Gains on medical exams or single-turn EHR agents will not transfer to physician assistance without joint multi-turn evaluation of intent resolution, patient dialogue, and FHIR actions.
  • Session-level reliability (all turns pass) is a stricter and more clinically meaningful target than mean turn score.
  • Data gathering across multiple tools and patient-interview turns are primary training and evaluation priorities; abstract event anaphora is especially hard.
  • Language-conditioned tool-use bias (e.g., more refusals or skipped EHR calls in English) must be measured separately in bilingual clinical agents.
  • Benchmarks and training environments need record-grounded interactive patients, not unconstrained role-play or static notes alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If coordination is the bottleneck, fine-tuning only on medical QA or isolated tool traces may plateau; curricula that force multi-source composition under ellipsis may move Pass@Session more than scale alone.
  • The same agentic-patient construction could stress-test referral, critical care, or post-discharge workflows the paper lists as out of scope.
  • Low Pass@Session implies clinical deployments should keep humans in the loop on every multi-turn thread, not only final recommendations.
  • Judge–human agreement on a small sample leaves open a measurement risk: future work could audit failures by task type to separate true reasoning errors from rubric noise.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces PhysAssistBench, a bilingual multi-turn benchmark for LLMs as physician assistants that must coordinate clinical knowledge, FHIR-based EHR tool use, and patient communication under underspecified physician requests. Built from MIMIC-IV via a multi-agent pipeline that constructs record-grounded “agentic patients,” the evaluation set comprises 324 four-turn sessions (1,296 turns) spanning four clinical scenarios, four task types (IL, DG, CR, WU), three implicitness subtypes (NA, PE, AE), and three data-richness tiers. Models are scored with turn-level rubrics (mRS, Pass@Turn, Pass@Session). Experiments on 14 leading LLMs report moderate mean rubric scores but low session-level reliability (e.g., best Pass@Session ≈23–27% at τ=0.60), with Data Gathering and patient-interview turns as consistent bottlenecks, and a measurable penalty for implicit vs. explicit queries. The authors conclude that reliable assistance requires coordination across knowledge, communication, and systems, not isolated gains.

Significance. If the evaluation environment is a sufficiently faithful proxy, this is a timely and useful contribution: it fills a clear gap relative to prior knowledge-, system-, or dialogue-only medical benchmarks (Table 1) by jointly testing implicit physician intent, FHIR tool use, and patient interaction in multi-turn sessions. Strengths include a scalable, evidence-filtered construction pipeline; balanced scenario/tier design; explicit-vs-implicit and EHR-vs-patient ablations; bilingual reporting; and open release plans under appropriate MIMIC licensing. The empirical pattern—IL/WU relatively strong, DG and patient turns weak, Pass@Session far below Pass@Turn—is informative for clinical agent research even if absolute numbers depend on rubrics and simulation choices. The work is significant as a benchmark and diagnostic study of coordination failures, not as a definitive clinical readiness verdict.

major comments (4)
  1. Abstract and §1 claim a “curated bilingual evaluation set of 1,296 manually reviewed and physician-validated turns,” but Appendix C.2–C.3 and Limitations describe only partial expert coverage: clinical plausibility on a 25% sample (95% after regeneration) and 94% judge–human item agreement on 128 turns / 32 sessions, with full turn-level physician validation of all 1,296 turns not completed. This overstates validation relative to the body text and weakens confidence in the central “unreliable assistants” claim. Please align abstract/intro wording with the actual review protocol and report stratified agreement (EHR-only vs patient-interview; IL/DG/CR/WU).
  2. §3.2 and §4 treat patient interaction as fixed, pre-generated, record-grounded replies replayed via patient.* tools, not live multi-turn patient ambiguity. Figure 4’s ~26-point drop on patient-interview turns is a main pillar of the “communication bottleneck / coordination” conclusion (§5.4, §6). Without an error audit separating rigid replay/persona constraints, missing follow-up flexibility, and judge/rubric strictness from genuine assistant failures, attribution of low scores primarily to model coordination is not fully supported. Add a stratified failure analysis of patient-interview turns and qualify claims about patient communication accordingly.
  3. §3.3 and §5.1 rely on a single fixed GPT-5.4-mini LLM judge for all rubric scoring, with human agreement only on a small sample (App. C.3). Because gold answers and rubrics are themselves pipeline-generated (§4, App. H), judge–pipeline correlation could systematically penalize valid alternative tool trajectories or clinically acceptable paraphrases—especially on DG/CR. Report inter-judge reliability with a second model or clinician subsample, and sensitivity of mRS/Pass@Session to judge choice or τ, so the low Pass@Session result is not judge-specific.
  4. §3.3 states tool use is evaluated only implicitly via answer rubrics (“correct values imply correct tools”) and allows deviation from gold trajectories. That is reasonable for clinical correctness, but it under-specifies whether failures are wrong tools, wrong arguments, skipped EHR calls (cf. language-conditioned tool bias in §5.2), or integration errors after correct retrieval. For a benchmark whose contribution is doctor–patient–EHR coordination, add at least coarse tool metrics (invocation of required resource types; empty vs non-empty results; write-parameter match on WU) alongside rubrics so the coordination diagnosis is actionable.
minor comments (5)
  1. Table 1 and related work are clear; consider briefly discussing how PhysAssistBench relates to concurrent “clinical environment simulator” visions (e.g., Luo et al. 2026, already cited) so novelty is framed as operationalized evaluation rather than only conceptual gap.
  2. Figure 3 / I2 heatmaps are dense; ensure color scales and cell values remain legible in print, and state n per cell so sparse subtype×task combinations are not over-interpreted.
  3. Appendix G vs H.6 give slightly different turn-type counts (e.g., IL 540 vs 444); reconcile final statistics so the released dataset matches the paper.
  4. Language-conditioned tool-invocation bias (§5.2) is interesting but currently anecdotal for Gemini; if retained as a finding, quantify across more models or move to discussion as a hypothesis.
  5. Minor consistency: abstract “physician-validated turns” vs Limitations “sampled subset”; fix throughout for precision.

Circularity Check

0 steps flagged · score 1.0 of 10

Empirical benchmark paper with no derivation that reduces predictions to fitted inputs or self-definitional identities; only mild dependence via LLM-assisted synthesis/judging that is not load-bearing by construction.

full rationale

PhysAssistBench is a new multi-turn evaluation resource built from MIMIC-IV via a multi-agent pipeline, with fixed gold tool trajectories, pre-generated patient replies, and turn-level rubrics. Leading models are then scored on held-out sessions (mRS, Pass@Turn, Pass@Session). There is no first-principles derivation, uniqueness theorem, or fitted parameter that is renamed as a prediction of a closely related quantity. Success is not defined as matching the generative prior of the synthesis models; gold answers and rubrics are constrained by real EHR evidence, quality gates, and staged human/physician review (sampled clinical plausibility + 94% judge-human agreement on 128 turns). Self-citations are absent as load-bearing premises. The mild residual dependence (LLM patient agents, rubric generator, GPT-5.4-mini judge) is a validity/artifact concern, not circularity of the kind where Eq. X equals the input by construction. Score 1 reflects only that residual non-load-bearing model-in-the-loop dependence; the central unreliability claim remains an external empirical measurement on the constructed benchmark.

Assumptions & free parameters 4 free parameters · 6 assumptions · 2 invented entities

This is an empirical benchmark paper, not a derivation. Load-bearing premises are domain and evaluation assumptions: MIMIC-IV as a usable proxy for assistance workflows; FHIR R4 tool abstraction as the right system interface; LLM-simulated patients grounded in records as adequate stand-ins for patient ambiguity; rubric items and an LLM judge as valid clinical correctness measures; and four scenarios / three richness tiers as representative of physician assistance. Free parameters are evaluation thresholds and generation/review design choices rather than fitted physical constants. No new physical entities are postulated.

free parameters (4)
  • Pass threshold τ ∈ {0.60, 0.75}
    Binarization thresholds for Pass@Turn/Pass@Session are chosen by the authors; rankings and “unreliable” rhetoric depend on these cutoffs even though continuous mRS is also reported.
  • Session design constants (4 turns; 4 scenarios; 3 richness tiers; 27 sessions per cell → 324)
    Structural sampling choices determine coverage and difficulty mix; not fitted to maximize a scientific constant, but they shape all headline statistics.
  • Judge model and scoring protocol (GPT-5.4-mini, temperature 0.2, max 16 tools/turn)
    Evaluation outcomes depend on a fixed proprietary judge and decoding/tool budget settings that are not uniquely determined by clinical ground truth.
  • Pipeline retry/skip and quality-gate rejection rates
    Gate failure rates (~12%/5%/10%) and max retries determine which patients/sessions enter the final set, acting as selection knobs on the evaluation distribution.
assumptions (6)
  • domain assumption Near-term medical LLMs should be evaluated as physician assistants under human oversight rather than as autonomous physicians.
    Stated in abstract/introduction and used to justify the assistance framing versus knowledge-only or doctor-replacement benchmarks.
  • domain assumption Real physician requests are often multi-turn, elliptical, and anaphoric (NA/PE/AE), so explicit single-turn QA understates deployment difficulty.
    Linguistic and clinical communication premise motivating implicit-query design (§2.4, §3.2).
  • ad hoc to paper Record-grounded agentic patients with fixed interview replies preserve clinical factuality sufficiently for evaluation when unsupported cases are filtered.
    Core construction principle of the multi-agent pipeline (§4); not independently validated against live patient dialogues.
  • ad hoc to paper Rubric item pass fractions and LLM-judge binary decisions are adequate proxies for clinical answer quality, including tool-grounded factuality.
    Scoring formalization in §3.3; human agreement reported only on a 128-turn sample (Appendix C.3).
  • domain assumption FHIR R4 read/write plus patient.* tools form a representative EHR/patient interface for assistance tasks.
    Tool inventory and scenario design (§3.2, Appendix D); standard in interoperability research but still a modeling choice.
  • domain assumption MIMIC-IV de-identified US academic/ICU-heavy records are an acceptable source for bilingual assistance scenarios after translation/verification.
    Data source and language limitations acknowledged in Limitations/Ethics; still foundational to all results.
invented entities (2)
  • PhysAssistBench sessions (agentic patient environment)
    purpose: Turn static MIMIC-IV admissions into multi-turn interactive doctor–patient–EHR evaluation units with gold tools, answers, and rubrics.
    The benchmark environment is the paper’s main constructed object; it is not a physical entity but a synthetic evaluation world whose fidelity is assumed.
  • Physician-query implicitness typology (NA, PE, AE) as operational labels for clinical assistant difficulty independent evidence
    purpose: Taxonomize underspecified physician requests and enable explicit/implicit ablations.
    Drawn from linguistic notions of ellipsis/anaphora but operationalized specifically for this benchmark’s turns 1–3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are LLMs Ready to Assist Physicians? PhysAssistBench for Interactive Doctor-Patient-EHR Assistance." pith.science (2026). https://pith.science/paper/KDSXSM53

@misc{pith2026260618613,
  author       = {Pith},
  title        = {Pith review of: Are LLMs Ready to Assist Physicians? PhysAssistBench for Interactive Doctor-Patient-EHR Assistance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KDSXSM53}},
  note         = {Machine review of arXiv:2606.18613}
}
read the original abstract

The most plausible near-term role of medical LLMs is to assist rather than replace physicians, yet current evaluations often test isolated capabilities: clinical knowledge, EHR system interaction, or patient communication. Physician assistance instead requires coordinating these capabilities within the same interaction, where physicians issue underspecified requests, patients describe symptoms ambiguously, and EHR systems demand precise tool use. We introduce PhysAssistBench, a benchmark for interactive doctor-patient-EHR assistance. Built from real MIMIC-IV cases, PhysAssistBench uses a scalable pipeline to construct agentic patients: interactive, record-grounded agents that turn static EHR records into multi-turn clinical scenarios while preserving clinical factuality. PhysAssistBench provides a curated bilingual evaluation set of 1,296 manually reviewed and physician-validated turns. Experiments with leading LLMs show that current models remain unreliable in this setting, which exposes a key bottleneck for clinical LLMs: reliable assistance requires coordination across knowledge, communication, and systems, not isolated gains in any of them.

Figures

Figures reproduced from arXiv: 2606.18613 by the authors.

Figure 1
Figure 1. Left: PHYSASSISTBENCH evaluates LLMs as physician assistants, not physicians: the assistant follows physician requests while interacting with a record-grounded FHIR-based EHR system and a dialogue patient. A multi-agent pipeline transforms static MIMIC-IV records into this agentic patient environment, exposing them through standardized FHIR interfaces rather than direct record access. Right: A representative hyperte… view at source ↗
Figure 2
Figure 2. Multi-agent data synthesis pipeline. A static MIMIC-IV record is transformed into a grounded, multi-turn [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Rubric score (%) by implicitness type × task type (EN), for 14 models and their average. Informa￾tion Lookup stays robust across all implicitness types, while Data Gathering (multi-tool composition) and Clin￾ical Reasoning (knowledge-grounded inference) are the consistent weak points across models. Both demand reasoning beyond single-point retrieval. Results for Chi￾nese (ZH) are presented in Appendix I2 (63.9 vs. 6… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The comparison of rubric scores between EHR-only and Patient-interview turns [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Review history (2 revisions) →

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 1 linked inside Pith

  1. [1]

    Er-reason: A benchmark dataset for llm-based clinical reasoning in the emergency room.arXiv preprint arXiv:2505.22919. MiniMax. 2026. MiniMax M2.7: Early echoes of self-evolution. https://www.minimax.io/news/ minimax-m27-en. Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M. Krumholz, Jure Leskovec, Eric J. Topol, and Pranav Rajpurkar. 20...

    arXiv 2026

  2. [2]

    Data or Specimens Only Research

    Lefusion: Controllable pathology synthesis via lesion-focused diffusion models. InInternational Conference on Learning Representations, volume 2025, pages 13232–13253. Shuang Zhou, Wenya Xie, Jiaxi Li, Zaifu Zhan, Mei- jia Song, Han Yang, Cheyenna Espinoza, Lindsay Welton, Xinnie Mai, Yanwei Jin, and 1 others. 2025. Automating expert-level medical reasoni...

    2025

  3. [3]

    CheckerAgent(ValidatePlans) [LLM]IL:1 read tool DG: ≥ 2read toolsCR:1read tool and clinical reasoningWU: 1 writetool MIMIC-IVRaw Data Admissions LabEvents Prescriptions Microbiology· · · Apply Per-Scenario Clinical data requirements to the MIMIC-IV Pool DiagnosticWorkup≥ 2 organ systems+imaging study+≥ 3 diagnoses Meddical Safetymonitored drugs+monitoring...

  4. [4]

    Session PlanAgent(LLM, once per entry)Inputs: Patient EHR Summary + Scenario Rules + 4-turn Arc TemplateOutput: Session Plan: Turn Intent, topic and tool hint for each of the 4 turns Per-Turn Loop ×4 (turn_idx∈{0,1,2,3})(Any failed check will result in the rerun of entire turn)

  5. [5]

    User Agent(Implicit Reformulation)[LLM] Selects an implicit Query Subtype (NA, PE, or AE) and rewrites the Explicit query accordingly

  6. [6]

    User Agent[LLM]Inputs: Turn Intent, Dialogue History, Failed Questions, and Current EHR Snapshot.Output: Explicit Physician Query

  7. [7]

    Tool ExecutorFHIR:Actions querythe local MIMIC-IV-backed FHIR store

    Planner Agent [LLM]Inputs: User Query, Available tools, Task Type, Tool Hints, and current EHR Snapshot.Output: Ordered Action List consisting of tool invocations with Arguments. Tool ExecutorFHIR:Actions querythe local MIMIC-IV-backed FHIR store. Patient:Actions invoke the Patient Agent. Write:Actions simulate resource creation. 5 Patient Agent [LLM]The ...

  8. [8]

    Given that

    Answer Agent[LLM]Inputs: User Query, Executed Actions, Dialogue History, and Task Type.Output: Task-specific Response Formatted according to Task Type (IL, DG, CR, or WU). ·····4. Checker Agent (Validate Answer) [LLM]FormationCompliance +DeterministicAnswers 8.Rubric Generation Agent[LLM]Generates several EHR-anchored evaluation criteria for the current t...

Show all 41 references
  1. [9]

    An explicit user question (from Stage 1)

  2. [10]

    The conversation history (prior turns)

  3. [11]

    "" [User Prompt]=

    A transformation rule specifying which implicitness subtype to apply Your task: rewrite the explicit question into its elliptic/anaphoric form. Rules: • Keep the clinical meaningidentical. • Applyonlythe transformation described — do not add new information. • Maintain the cas...

  4. [12]

    Items with1 resultare restricted to Information Lookup turns only

    Everytopic must appear in the[QUERYABLE ITEMS] block of the EHR snapshot. Items with1 resultare restricted to Information Lookup turns only

  5. [13]

    Topics must not repeat across turns — each turn adds new information

  6. [14]

    4.turn_intents[i]must matchtopicandtool_hintinturns[i]

    Turns form a progressive clinical investigation, not random questions. 4.turn_intents[i]must matchtopicandtool_hintinturns[i]

  7. [15]

    Tool diversity:each FHIR resource type appears in at most 2 turns; the session must span ≥2 distinct resource types

  8. [16]

    T{i}[R] retrieve {item} — {clinical purpose}

    Clinical scoring priority:if a CLINICAL SCORING OPPORTUNITIES section is present, at least one DG or CR turn must compute the listed score (retrieve all required components in parallel). [arc / intent formats] • IL:"T{i}[R] retrieve {item} — {clinical purpose}" • DG:"T{i}[W] {...

  9. [17]

    Respond in natural spoken language as the patient

  10. [18]

    Stay strictly in character based on the personality above

  11. [19]

    Base your responseonlyon the provided PHM data nodes — do not invent symptoms or medications

  12. [20]

    Donotuse medical jargon ifhealth_literacy=low

  13. [21]

    If asked about a medication you never filled, express this naturally

  14. [22]

    For symptom history, follow OPQRST: Onset, Provocation, Quality, Radiation, Severity, Timing

  15. [23]

    Keep responses concise (2–5 sentences) unless probed for details

  16. [24]

    "" [User Prompt]=

    Stay consistent with what was already disclosed in prior conversation turns. [WithheldFlags —critical_withheldpersona only] Critical information (e.g. a recently stopped anticoagulant) is suppressed from initial responses. It is revealedonlywhen the physician’s follow-up query...

  17. [25]

    No read/search tools allowed

    Write/Update:exactly 1 write tool ( MedicationRequest.create,ServiceRequest.create, orFlag.create) + prepare_to_answer. No read/search tools allowed

  18. [26]

    6.subject_idmust be present in EHR tool arguments when the patient is known

    All tools must exist in theAvailable Toolslist. 6.subject_idmust be present in EHR tool arguments when the patient is known

  19. [27]

    Patient tools (patient.xxx): bothsubject_idandsession_idrequired

  20. [28]

    Tool arguments must match their schema (no missing required parameters)

  21. [29]

    "" [User Prompt]=

    The tools chosen must berelevantto the question asked. 10.Action_Listmust end withprepare_to_answer. Outputonlythe JSON, no other text.""" [User Prompt]=""" User question:"{{user_question}}" Task type:{{Information Lookup / Data Gathering / Clinical Reasoning / Write/Update}} ...

  22. [30]

    The answer correctly cites creatinine as 0.9 mg/dL

    Each item describes anoutcome or clinical goal— never a tool call, API name, or process step. 2.Ground items in actual EHR values. Write:“The answer correctly cites creatinine as 0.9 mg/dL”— not“mentions the creatinine value”

  23. [31]

    Each item must be independently evaluable asYESorNO

  24. [32]

    Include≥1 reasoning or recommendation item (not just fact retrieval)

  25. [33]

    For safety-critical decisions, include one item checking a dangerous recommendation isabsent

  26. [34]

    Donotmention tool names, function names, or system internals

  27. [35]

    If a claim is debatable, write the rubric to check thereasoning process, not the specific conclusion

    Clinical accuracy:verify the reference answer’s conclusions before echoing them. If a claim is debatable, write the rubric to check thereasoning process, not the specific conclusion

  28. [36]

    The answer cites creatinine as 0.9 mg/dL

    Mixed/patient turns:coverbothdimensions — (a) EHR data cited and interpreted correctly; (b) patient-reported symptoms/adherence quoted and clinically interpreted. Item count by task type: •IL:3 items — value cited, value interpreted, conclusion stated •DG:4–5 items — each valu...

  29. [37]

    medication, dose, route, frequency, indication)

    Each item names aspecific fieldof tool_call.arguments (e.g. medication, dose, route, frequency, indication)

  30. [38]

    Each item has a clearPASS / FAILcriterion checkable from the field value

  31. [39]

    Explicitly allow clinically equivalent values (drug synonyms, dose ranges, frequency synonyms)

  32. [40]

    dose ≥ contraindicated threshold, wrong drug class, unjustifiedstatpriority)

    Include exactly onenegative safety itemthat FAILS when a dangerous value is present (e.g. dose ≥ contraindicated threshold, wrong drug class, unjustifiedstatpriority)

  33. [41]

    "" [User Prompt]=

    Donotwrite items about clinical reasoning or prose justification. """ [User Prompt]="""(shared across all turn types) Clinical question:"{{user_question}}" [EHR Data Retrieved] [{{tool_name}}]: {{actual FHIR R4 Bundle values used in the gold answer}} [Reference Answer] {{gold_...

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.