{"id":"e45b7f5b-3429-4bc2-8c52-3d53aa0e6e7d","arxiv_id":"2608.07796","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new 750-case EHR benchmark that jointly scores verdicts, evidence grounding, policy use, process adherence, and abstention finds that all 16 tested agents under-abstain and that defect-free accuracy reorders the leaderboard.","lead":"This paper introduces CliniCARE-Bench, a 750-case benchmark that tests AI agents doing retrospective clinical audits over real electronic health records, scoring not only verdicts but also the evidence and process behind them. Sixteen agent systems scored 65.3 to 76.1 percent accuracy, and that accuracy hid process defects: clean investigations were rarer and reordered the leaderboard.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reference labels for most of 750 cases rest on an ensemble majority, with only 100 clinician-calibrated; 25% disagreement on split-vote cases could flip the under-abstention and defect-gap findings.","rationale":"I agree with the reader's weakest-assumption identification. The paper is transparent about the calibration protocol and its limits; it reports clinician agreement and double-review statistics honestly, and the process-judge reliability checks are a genuine strength. However, the scale of unverified labels (650/750) and the low agreement on split-vote cases makes label correctness the single point on which the headline findings hinge. The under-abstention asymmetry is presented as a property of every system, but it is computed against reference labels that are themselves model-generated on exactly the Indeterminate classes where systems are weakest. A modest label error rate on those classes could erase the asymmetry. The defect-free gap is similarly label-dependent. The proposed test—independent clinician adjudication of all split-vote cases—would settle the question directly and is feasible because the traces and data are replayable. Unless or until that test is run, the CONDITIONAL verdict with medium correctness risk is appropriate; no change in verdict is needed.","tokens_in":41841,"tokens_out":4301,"duration_ms":39228,"concrete_test":"Re-adjudicate all 183 split-vote cases and a random sample of 50 unanimous cases with two independent clinicians per case, working directly from raw MIMIC-IV records and scenario specifications, blinded to ensemble reports. Replace the reference labels and recompute over-commitment, over-abstention, and defect-free accuracy for all 16 systems. If over-commitment no longer exceeds over-abstention for every system, or if the mean defect gap falls by more than half, the central empirical findings are artifacts of label error.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is the correctness of the reference verdicts. Section 4.4 shows 556 of 750 cases were unanimous, 183 split 2-vs-1, and only 11 three-distinct cases were escalated to clinicians. Section 4.5 reports 91% clinician agreement on the 75-case calibration sample, but only 75% on the 24 split-vote cases, implying roughly a quarter of the 183 split-vote labels may be wrong. Because the headline claims—that every system under-abstains and that defect-free accuracy reorders the leaderboard—are computed against these labels, systematic label error on the Indeterminate classes would directly manufacture both findings. Over-commitment is measured as definitive predictions on reference-Indeterminate cases; if many of those labels should be definitive, the asymmetry shrinks or reverses. Defect-free accuracy credits a verdict only when it matches the reference, so label errors depress or inflate it. Furthermore, the calibration clinicians reviewed model-surfaced evidence rather than the raw record (Section 4.5), so extraction errors by the ensemble are never independently checked. This is not a hypothetical: the ensemble includes GPT-5.5, which is also an evaluated system, and its failures would propagate into the reference standard.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces CliniCARE-Bench, a benchmark for retrospective clinical audit over real longitudinal EHR data. It consists of 25 clinician-validated scenarios instantiated as 750 patient-specific cases from MIMIC-IV, with each case requiring one of four verdicts: Yes, No, Indeterminate: Lack of Data, or Indeterminate: Medically Ambiguous. Systems interact with a governed, logged tool environment for record retrieval, computation, and policy access, and are scored not only on verdict accuracy but also on defect-free accuracy, process adherence, evidence and policy grounding, abstention behavior, reliability, and resource use. Reference verdicts are produced by an ensemble of three frontier model harnesses and calibrated against a Clinical Board review on a stratified subset of 100 cases. The paper evaluates 16 agentic systems, reporting four-way accuracy between 65.3% and 76.1%, a defect-free accuracy that is 4.8–14.8 percentage points lower and reorders the leaderboard, and a universal over-commitment pattern in which every system issues definitive verdicts on reference-abstention cases at a higher rate than it abstains on reference-definitive cases.","tokens_in":42070,"tokens_out":8872,"duration_ms":78985,"significance":"If the benchmark and its reference labels are reliable, this is a valuable and timely contribution. It moves clinical agent evaluation from static knowledge tests and simulated encounters to replayable, process-aware audit over real longitudinal records, and it jointly scores investigation quality, grounding, policy use, and calibrated abstention. The paper is unusually thorough in its validation efforts: the Clinical Board calibration, the judge-reliability panels, the repeated-run analyses, the policy-corpus ablation, and the test-time-compute comparison are all reported with appropriate caveats. The headline dissociations, particularly the gap between raw accuracy and defect-free accuracy and the systematic under-abstention, are clinically important and, if correct, would justify the benchmark's deployment-oriented framing. The main risk to these claims is the validity of the reference verdicts for the majority of cases that were not directly clinician-reviewed, together with the self-referential role of GPT-5.5 as ensemble member, evaluated system, and judge; these issues are acknowledged in the paper but are not fully resolved.","major_comments":[{"comment":"The reference verdicts for 650 of 750 cases are not directly clinician-reviewed; only 75 cases were used for verdict-level calibration, with an additional 25 double-reviewed. On the 2-vs-1 split-vote cases, clinician agreement with the reference was 18/24 (75%), and the paper states that every disagreement resolved to the dissenting model rather than an outside label. Because the over-commitment rate is defined over reference-abstention runs and defect-free accuracy requires exact label match, systematic label errors on the 183 un-reviewed split-vote cases, especially on the Indeterminate classes, would directly bias both headline findings. The paper acknowledges this limitation in Section 7 but does not quantify its impact. I recommend a sensitivity analysis that recomputes over-commitment and defect-free accuracy under plausible label flips for split-vote cases, or direct clinician review of all 183 split-vote cases; without this, the claim that every system under-abstains is not robustly established.","section":"Sections 4.4, 4.5, and 6.2 (Tables 3 and Figure 6)"},{"comment":"The calibration review is explicitly not independent: clinicians were given the scenario specification and the three ensemble reports and did not interface with the patient record themselves. This means the ensemble's retrieval and extraction errors are never independently checked. This is particularly consequential for Indeterminate: Lack of Data verdicts, which depend on the absence of evidence: if the ensemble failed to retrieve a decisive record, the reference label would be systematically wrong. The paper states this design is intentional, but it limits the calibration to medical judgment over model-surfaced evidence. I recommend a supplementary audit in which clinicians directly query the raw MIMIC record for a sample of split-vote Lack-of-Data cases, in order to rule out extraction-driven label error.","section":"Section 4.5 (Clinical Review and Label Calibration)"},{"comment":"Defect-free accuracy, the second headline metric, depends on the LLM judge's grading of MUST-NOT process criteria. The production judge is GPT-5.5, which is also an evaluated system and a member of the reference-verdict ensemble. The paper's judge-reliability validation reports moderate agreement for the process rubric (Fleiss kappa approximately 0.65, with 16% of criteria contested) and finds no own-family favoritism only for policy support, not for process criteria. Because the defect gap reorders the leaderboard, a systematic grading bias in MUST-NOT criteria for one system family could change the central conclusion. Please report process-grading agreement broken down by evaluated system family, or run the process rubric with an independent judge from a different model family, to rule out this source of bias.","section":"Sections 5.5, 5.7, and 6.7 (process rubric and judge)"}],"minor_comments":[{"comment":"The paper should state explicitly whether the GPT-5.5 model used in the reference-verdict ensemble is the same checkpoint as the evaluated GPT-5.5 system, and should provide version or date information for all ensemble members, so that the degree of overlap between the labeling process and the evaluated systems is fully transparent.","section":"Section 4.4 and Table 2"},{"comment":"The caption of Figure 5 and the definition of the defect gap should clarify that both raw accuracy and defect-free accuracy in the figure are computed over report-present runs, matching the Gap column in Table 3, so that readers do not compare the figure's raw-accuracy values directly with the all-run Acc column.","section":"Figure 5 and Table 3"},{"comment":"The development-subset experiments use Opus 4.8 for paired ablations (Table 8) but Opus 5 for the sliced full-cohort runs reported in Appendix E.1 (Table 16); while the text explains this, a single sentence in Section 6.6 pointing to the two different sources would prevent readers from mistakenly treating Table 8 and Table 16 as directly comparable.","section":"Section 6.6 and Appendix E"},{"comment":"The process score formula floors negative totals at zero, so a run that violates one MUST-NOT criterion and a run that violates several are indistinguishable once the total is zero; the text should state this limitation explicitly, since the defect-free metric counts any MUST-NOT violation equally.","section":"Section 5.5 (process score)"},{"comment":"The paper repeatedly states that benchmark data and evaluation code will be released, but no repository or release timeline is provided; for a benchmark paper, a concrete availability statement with an anonymized URL would strengthen reproducibility claims.","section":"Section 7 and release plans"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is unusually transparent about its limitations, which is commendable, but the central empirical claims rest on reference labels that are mostly model-generated and only partially clinician-calibrated. The self-referential role of GPT-5.5 as ensemble member, evaluated system, and judge is a notable integrity risk that the paper acknowledges but does not eliminate. If the authors add the requested sensitivity analyses and independent process-judge checks, the paper could become a strong benchmark contribution; in its current form, the headline findings are not yet fully robust to label error."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CliniCARE-Bench is the first clinical-agent benchmark I've seen that scores the whole investigation—four-way verdicts, claim-level grounding, policy citation, process adherence, and calibrated abstention—in one patient-level framework over real MIMIC-IV longitudinal records. That joint framing is the new contribution, and it's well executed. The scenario specifications in the appendix are unusually detailed, the division between rule-based and LLM-judged metrics is thoughtful, and the paper is refreshingly honest about its own limitations. Credit where due: they report judge-reliability panels (Fleiss kappa 0.65-0.86), repeated-run analyses showing systems are stably wrong, and a development-subset replication of the abstention asymmetry. The empirical pattern—defect-free accuracy 4.8-14.8 points below raw accuracy, reordering the leaderboard—is internally consistent.\n\nThe soft spot is the reference labels. Only 100 of 750 cases got clinician review; the rest rest on a three-model ensemble majority. On the 24 split-vote cases in the calibration sample, clinician agreement with the reference falls to 75%, versus 91% overall. Since the under-abstention finding is measured as definitive predictions on reference-Indeterminate cases, systematic label error on those classes would directly manufacture the asymmetry. The paper acknowledges this, and the mitigations are real—the asymmetry reproduces on a dev subset with a different label mix, and unanimous cases are more reliable—but the clinicians reviewed model-surfaced evidence rather than the raw record, so extraction errors propagate. The judge (GPT-5.5) is also an evaluated system, though the multi-judge panels and independence checks blunt that concern.\n\nNone of this is fatal; the limitations section is unusually candid. But the headline numbers should be treated as conditional until independent judge replication or a larger clinician-reviewed label set appears. The paper deserves a serious referee; I'd send it out with instructions to focus on label construction and the abstention metrics. It'll be a useful benchmark for the community even if some numbers shift.","headline":"A genuinely useful benchmark with a real label-construction soft spot; worth serious refereeing.","tokens_in":42702,"tokens_out":2678,"would_cite":true,"duration_ms":25301,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces a 750-case EHR benchmark showing that correct verdicts can mask indefensible investigations, with defect-free accuracy 4.8-14.8 points lower than raw accuracy.","keywords":["clinical AI benchmark","electronic health records","agentic evaluation","calibrated abstention","evidence grounding","process adherence","defect-free accuracy","MIMIC-IV"],"falsifier":"Independently re-adjudicate all 750 cases with blinded clinicians, or a large stratified sample such as 200 cases, and compare their verdicts to the reference labels; if clinician agreement on the Indeterminate classes dropped well below the 91% seen on the 75-case calibration sample, or if clinicians assigned Yes or No to a substantial share of cases the ensemble called Indeterminate, the under-abstention finding would be an artifact of the reference labels rather than a property of the systems.","tokens_in":41635,"feed_emoji":"🩺","tokens_out":7239,"duration_ms":61497,"temperature":0.7,"pith_summary":"This paper argues that clinical AI should be judged not by whether it returns the right verdict, but by whether it conducted a defensible investigation. To test that, it introduces CliniCARE-Bench: 25 clinician-validated audit scenarios instantiated as 750 patient-specific cases over real longitudinal MIMIC-IV records, where agents retrieve evidence through governed, logged tools and must return one of four verdicts, including two deferral classes that separate missing evidence from medical ambiguity. Across 16 systems, raw four-way accuracy reaches 65.3-76.1%, but defect-free accuracy, which credits a correct verdict only when the investigation violated no prohibited shortcut, is 4.8-14.8 percentage points lower and reorders the leaderboard. Every system also under-abstains, committing to definitive verdicts on cases whose reference verdict requires deferral. The paper's point is that a correct answer is not evidence of correct reasoning, and only a trace-level audit can tell the difference.","feed_headline":"Correct verdicts mask bad EHR investigations in every agent tested","feed_subtitle":"Benchmark over 750 MIMIC-IV cases: defect-free accuracy falls 4.8-14.8 points and reorders the leaderboard.","key_machinery":"The load-bearing machinery is the process rubric: per-scenario MUST-DO and MUST-NOT criteria, weighted by clinician authors, graded by an LLM judge against the final report together with the logged tool-call trajectory. From this rubric the benchmark derives defect-free accuracy, which credits a run only when the verdict is correct and no MUST-NOT criterion is sprung. The complementary mechanism is the four-way verdict space, whose two Indeterminate classes - Lack of Data versus Medically Ambiguous - make calibrated abstention a first-class, scorable outcome rather than a post hoc confidence threshold. Policy grounding is enforced by a fixed corpus of governing documents with line-level provenance, so policy citations can be checked for resolution and support by rule or by judge. Together these let the benchmark separate outcome from process: a correct label, a sound investigation, and a defensible abstention are each measured separately.","core_discovery":"The central claim is that raw verdict accuracy systematically overstates the quality of agentic clinical investigation over EHRs. Using a benchmark built from real patient records and clinician-validated adjudication criteria, the paper shows that when a verdict is correct but the investigation uses a prohibited shortcut - for example, asserting a condition is excluded without the test result that would settle it - the run is defective. Defect-free accuracy, which requires both a correct verdict and no MUST-NOT violation, falls 4.8-14.8 points below accuracy on the same runs, and this correction changes the system ordering. In addition, all sixteen evaluated systems over-commit: on cases where the reference verdict is Indeterminate, they return Yes or No at rates (21.3-55.3%) far above their over-abstention rates on definitive cases (7.3-16.5%). The benchmark is offered as a deployment-oriented standard for measuring whether a clinical agent can retrieve needed evidence, ground claims in it, apply the governing policy, and defer when the record cannot support a unique conclusion.","pith_inferences":["If the defect-free gap is driven by a small set of recurring shortcuts, then a targeted intervention - such as requiring confirmation of an order time against a radiology report or an ascitic-fluid result before ruling out peritonitis - could raise defect-free accuracy without changing raw accuracy; this is testable by adding such requirements to a harness.","The two Indeterminate classes may index different failure mechanisms: over-committing on Lack of Data cases suggests retrieval or coverage failures, while over-committing on Medically Ambiguous cases suggests criterion-application failures; separating them could guide where to invest in an agent.","Because the case mix is intentionally stratified, the accuracy numbers are not prevalence estimates; a deployment cohort with a different base rate of indeterminate cases would likely show different accuracy and different over-commitment rates.","The same evaluation recipe - governed logged tools, explicit adjudication criteria, traceable evidence, and scored deferral - transfers to other audit settings such as financial or legal review, where the defensibility of a decision matters as much as its conclusion; a concrete port would require instantiating a policy corpus and process rubric for that domain."],"forward_implications":["No evaluated system comes close to autonomous deployment: the best resolves only 76.1% of the 750 cases, so human-supervised review remains necessary for retrospective clinical audit.","A correct verdict is not a reliable signal of a sound investigation: up to one in five correct verdicts for the most affected system violates a prohibited shortcut, so quality controls should audit the trace, not just the label.","The under-abstention asymmetry is systematic: all sixteen systems over-commit on deferral cases more than they over-abstain on definitive ones, so interventions should push agents toward deferral when evidence is missing or ambiguous.","Repeated attempts do not fix this: three-attempt reliability shows systems are mostly stably wrong rather than randomly wrong, so running a query again will not surface the error.","Policy omission is the norm: every system cites genuinely governing documents but omits roughly half of the other governing documents, meaning verdicts can be correct for the wrong reasons."],"supporting_citations":[{"why":"MIMIC-IV v3.1 with its Note and ED modules is the real-patient-derived substrate from which all 750 cases are instantiated.","marker":"[14]"},{"why":"Supplies the trace-level clinical auditing approach this benchmark extends to score process, grounding, and abstention within the investigation trace.","marker":"[19]"},{"why":"Documents the outcome-process gap for medical LLMs that the defect-free accuracy finding extends to agentic EHR investigation.","marker":"[20]"},{"why":"Also documents the outcome-process gap and robustness concerns, providing prior evidence that correct output does not imply correct reasoning.","marker":"[21]"},{"why":"Governs a representative scenario by supplying the adjudication thresholds, illustrating how the policy corpus provides the decision criteria.","marker":"[15]"},{"why":"Provides the shared open medical-evaluation ecosystem the benchmark is designed to join, grounding its positioning among other clinical benchmarks.","marker":"[6]"}],"fun_headline_variants":["Right verdicts, wrong process in all 16 EHR agents","Benchmark: raw accuracy flatters; defect-free scores reorder clinical agents","All 16 clinical agents cut corners on EHR cases, new benchmark finds","Correct answers score high but hide flawed investigations in EHR agents","EHR agents over-commit on ambiguous cases, says CliniCARE-Bench"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline findings assume the 750 reference verdicts are correct, but only 75 cases plus 25 double-reviewed cases were checked by the Clinical Board; the remaining roughly 650 labels come from a three-model adjudication ensemble without direct clinician verification, so if those model-assisted labels are systematically wrong - especially on the two Indeterminate classes - the measured under-abstention and the defect-free gap would be artifacts of labeling rather than properties of the systems.","fun_headline_variants_meta":{"raw":{"variants":["Right verdicts, wrong process in all 16 EHR agents","Benchmark: raw accuracy flatters; defect-free scores reorder clinical agents","All 16 clinical agents cut corners on EHR cases, new benchmark finds","Correct answers score high but hide flawed investigations in EHR agents","EHR agents over-commit on ambiguous cases, says CliniCARE-Bench"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001755,"raw_usage":{"total_tokens":7006,"prompt_tokens":1099,"completion_tokens":5907,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":715,"completion_tokens_details":{"reasoning_tokens":5812}},"tokens_in":715,"tokens_out":5907,"duration_ms":40308,"temperature":1.0,"reasoning_tokens":5812,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:12:34.756187+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently re-adjudicate all 750 cases with blinded clinicians, or a large stratified sample such as 200 cases, and compare their verdicts to the reference labels; if clinician agreement on the Indeterminate classes dropped well below the 91% seen on the 75-case calibration sample, or if clinicians assigned Yes or No to a substantial share of cases the ensemble called Indeterminate, the under-abstention finding would be an artifact of the reference labels rather than a property of the systems.","supporting_citations":[{"cited_title":"Fidelity of medical reasoning in large language models.JAMA Network Open, 8(8):e2526021, 2025","cited_arxiv_id":null,"evidence_quote":"Documents the outcome-process gap for medical LLMs that the defect-free accuracy finding extends to agentic EHR investigation."},{"cited_title":"KDIGO clinical practice guideline for acute kidney injury.Kidney International Supplements, 2012","cited_arxiv_id":null,"evidence_quote":"Governs a representative scenario by supplying the adjudication thresholds, illustrating how the policy corpus provides the decision criteria."}],"review_version":1}