Judge-dependent safety gains and model-specific helpfulness costs of evidence-sufficiency prompting in clinical LLMs

Koyar Afrasyab

REVIEW 2 major objections 5 minor 12 references

The paper shows that an evidence-sufficiency prompt genuinely changes clinical LLM behavior, but the measured safety gain is judge-dependent: a second judge records half the effect, and clinicians show the first over-labels; LLM-judged safe

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 16:01 UTC pith:ZFD4QSOH

load-bearing objection A careful, well-controlled benchmark showing LLM-judged safety effects are judge-relative; the genuine-behavior-change claim is plausible but rests on one judge. the 2 major comments →

arxiv 2607.18086 v1 pith:ZFD4QSOH submitted 2026-07-20 cs.AI

Judge-dependent safety gains and model-specific helpfulness costs of evidence-sufficiency prompting in clinical LLMs

Koyar Afrasyab This is my paper

classification cs.AI

keywords LLM-as-judgejudge dependenceclinical decision supportevidence sufficiencyunsafe overconfidencesafety-helpfulness trade-offbenchmark robustnessabstention

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether a 'safety gain' measured by an LLM judge reflects real behavior change or the judge's calibration, using a structured evidence-sufficiency prompt as a test case on 1,200 fully paired clinical question-answers across four models. It finds the prompt genuinely reduces unsafe overconfidence—the primary judge scores a 24.7-point paired drop—but an independent different-family judge scores only a 13.1-point drop on the same outputs, and blinded clinician review shows the primary judge is a high-sensitivity, low-specificity screen that over-labels. The safety gain also carries a model-specific helpfulness cost, from near-free for GPT-5.5 to a -58-point collapse in correct diagnosis for Gemini 3.5 Flash. The paper concludes that LLM-judged clinical safety numbers are directional and relative, not calibrated absolute rates, and should be anchored to human review and read jointly with helpfulness.

Core claim

The paper claims that the evidence-sufficiency wrapper reduces unsafe overconfident answers in four clinical LLMs, but that the size of the reduction is judge-dependent: the pre-specified judge measured a 24.7-point paired reduction, an independent cross-family judge measured 13.1 points on the same responses, and blinded clinicians characterized the primary judge as a high-sensitivity (1.00), low-specificity (0.55) screen. The behavioral effect is genuine, not a token-reward artifact, because matched scaffold controls with identical section labels produced unsafe rates from 23% to 80% depending on the instruction, and the effect decomposes into a 9.0-point structural component and a 14.8-po

What carries the argument

The paper's central machinery is a fully paired common panel combined with matched scaffold-control arms: the same four section labels (EVIDENCE PRESENT, EVIDENCE MISSING, SUFFICIENCY JUDGMENT, ANSWER) are paired with neutral, abstain, and forced-commit instructions, isolating scaffold structure from abstention content and providing an adversarial bound. This is supplemented by a cross-family judge re-scoring all cells, a positive-enriched blinded clinician review set, and a per-model helpfulness analysis on answerable complete-information cases. Together they separate genuine behavior change from judge circularity and anchor the automated label to human judgment.

Load-bearing premise

The conclusion that the measured reduction reflects genuine behavior change rather than judge circularity rests on the assumption that the LLM judge is not biased by prompt-condition-specific response style beyond the matched scaffold tokens; the paper itself concedes that a condition-dependent bias could still distort the structural/content decomposition.

What would settle it

Present the primary judge with wrapper outputs whose four section labels (EVIDENCE PRESENT, EVIDENCE MISSING, SUFFICIENCY JUDGMENT, ANSWER) have been stripped while preserving the answer text, and re-measure the paired reduction; if the 24.7-point drop shrinks to near zero rather than remaining around 15 points, the judge was responding to the scaffold's formatting rather than to genuine behavioral change.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

LLM-judged clinical safety effects should be reported as directional and relative, not as calibrated absolute rates, because the same responses yield a 24.7-point reduction under one judge and a 13.1-point reduction under another.
Safety gains from evidence-sufficiency prompting carry a model-specific helpfulness cost; net value must be evaluated per model and jointly with accuracy, from near-free for GPT-5.5 to a -58-point drop in correct diagnosis for Gemini 3.5 Flash.
The primary judge behaves as a high-sensitivity, low-specificity screen against clinician truth, so absolute unsafe rates are over-estimated; direction and ranking are more trustworthy than absolute levels, and clinician anchoring is necessary before deployment claims.
The wrapper's effect decomposes additively into a 9.0-point scaffold-structure component and a 14.8-point abstention-content component, and the same scaffold tokens can be steered to a 42.1-point increase in unsafe overconfidence under forced commitment, confirming the judge is scoring behavior rather than tokens.
Effects are heterogeneous across models and concentrated in genuinely incomplete-information inputs; full-information diagnostic cases show no significant reduction, so the wrapper acts mainly when information is truly missing.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

An implicit consequence of this result is that any single-LLM-judged safety benchmark without a second-family judge or human anchoring can overstate effect sizes by roughly two-fold; the paper's protocol—paired panel, cross-judge check, positive-enriched clinician set—could serve as a reusable template for clinical LLM safety evaluations.
A testable extension would be to re-score wrapper outputs with the four section labels stripped while preserving answer text; if the 14.8-point abstention-content component survives, the behavioral effect is robust to the hypothesized condition-dependent judge bias, and if it collapses, the format confound is real.
The same directional/relative logic likely applies to other LLM-judged safety endpoints such as toxicity, sycophancy, or hallucination, because any automated judge is a screen with its own sensitivity–specificity trade-off; cross-study absolute rates should not be compared without a common human anchor.
The inverse relationship between safety gain and accuracy collapse across models suggests prompt interventions should be tuned per model rather than deployed uniformly, and could be formalized as a Pareto frontier that separates models with acceptable safety–helpfulness trade-offs from those without.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A careful, well-controlled benchmark showing LLM-judged safety effects are judge-relative; the genuine-behavior-change claim is plausible but rests on one judge.

read the letter

Colleague,

This paper is worth your time: it is a rare example of an LLM-judged safety benchmark that takes its own measurement problems seriously. The core result — that a prompt intervention's measured safety gain changes from 24.7 to 13.1 points depending on which LLM judge scores it, with one-directional disagreement — is new and important. It supports the conclusion that such effects should be reported as directional, judge-relative, per-model results, not calibrated absolute rates.

What the paper does well: the fully paired design (1,200 cells), the two-family cross-judge scoring with kappa 0.19, the matched scaffold controls showing the same section tokens produce unsafe rates from 23% to 80%, the per-model helpfulness cost (Gemini collapses, GPT-5.5 near-free), and the clinician review anchoring the primary judge as a high-sensitivity/low-specificity screen. These are not decorative extras; they directly address the circularity and calibration threats that plague most LLM-as-judge papers. The authors also pre-specify the primary endpoint and report multiplicity corrections, heterogeneity tests, and power. That is more rigor than most.

Soft spots: First, the genuine-behavior-change claim — that the wrapper actually makes models safer, not just that the judge prefers its output style — still rests largely on one judge's labels. The scaffold controls refute the crude token-rewarding hypothesis but not a subtler condition-dependent bias: the wrapper elicits explicit 'EVIDENCE MISSING' and 'SUFFICIENCY JUDGMENT: insufficient' statements, which are exactly what the rubric rewards. The clinicians characterized the judge but did not directly score paired wrapper-vs-standard responses. The authors concede this in the Discussion, but the concession is buried. Second, the clinician sensitivity estimate rests on 6-8 cases; the 95% CI is wide, and the PPV (~15%) is heavily base-rate dependent. Third, the helpfulness cost is scored by a single correctness judge, unvalidated against clinicians — the authors flag this as provisional, but it is central to the per-model trade-off claims. Fourth, at review time the repository was request-only with no DOI, so I could not inspect artifacts. That is an availability problem, not necessarily a validity problem, but it matters.

Who is this for? Any researcher using LLM judges to evaluate clinical safety, and anyone building safety benchmarks more broadly. It deserves a serious referee, not desk rejection. The right outcome is probably accept with revisions: the repository needs a persistent identifier, the correctness judge needs a human spot-check, and the genuine-behavior-change claim should be either re-framed as judge-relative behavior change or supported by a direct human-scored paired subset.

I would take it to reading group, and I would cite it for the measurement point, though I would phrase the behavior-change claim with a caveat.

Referee Report

2 major / 5 minor

Summary. This paper investigates whether a structured evidence-sufficiency wrapper reduces unsafe overconfident responses in clinical LLMs, how far the measured effect depends on the scoring judge, and what it costs in helpfulness. In a fully paired common panel of 1,200 model-item cells across four models, the pre-specified primary judge (GPT-5.4-nano) labels a 24.7-point paired reduction in unsafe overconfidence (49.3% to 24.7%); a different-family judge (Claude Sonnet 5) finds a directionally consistent but much smaller 13.1-point reduction. Blinded clinicians characterize the primary judge as a high-sensitivity (1.00) but low-specificity (0.55) screen. The authors use matched scaffold controls, two wrapper paraphrases, decode replicates, and a decomposition into scaffold-structure and abstention-content components to argue that the effect reflects genuine behavior change, and they report a strongly model-specific helpfulness cost (correct diagnosis falls from 80.3% to 50.3%). The paper concludes that LLM-judged clinical safety effects should be treated as directional and relative, anchored to human review, and evaluated jointly with helpfulness, not as calibrated absolute rates.

Significance. If the results hold, the paper makes a valuable methodological contribution: it demonstrates that LLM-judged clinical-safety effect sizes are judge-relative, and it provides a concrete template for reporting such effects with paired designs, cross-judge checks, scaffold controls, human anchoring, and helpfulness trade-offs. The design is unusually careful: fully paired data, per-model reporting, a pre-specified primary endpoint, bootstrap and McNemar/GEE inference, matched scaffold controls (including an adversarial forced-commitment arm), two independent paraphrases, stochastic decode replicates, and a blinded clinician review. The authors are also commendably explicit about the limitations, including the small human sample, the single-judge helpfulness endpoint, and the possibility of condition-dependent judge bias. The paper is a solid contribution to benchmark methodology, although the claim of 'genuine behavior change' is somewhat stronger than the evidence currently supports.

major comments (2)

[Abstract; Discussion, Mechanism of Avoided Overconfidence] The assertion that the wrapper produces 'genuine behavior change, not judge circularity' is stronger than the evidence. The clinician review characterized the primary judge's overall sensitivity/specificity, but it did not directly score paired wrapper-vs-standard responses under human labels. The high sensitivity (1.00) was estimated on only 6/6 clinician-unsafe cases in a positive-enriched set and does not rule out a condition-dependent bias in which the judge rewards wrapper-specific phrases such as 'EVIDENCE MISSING' and 'SUFFICIENCY JUDGMENT: insufficient'. The authors themselves concede that 'a condition-dependent bias could still distort the split' (Discussion, Mechanism paragraph). Because the wrapper condition systematically elicits exactly the lexical features the rubric rewards, the scaffold controls, while strong, do not fully exclude this confound. Please either add a human-
[Results: Helpfulness and Accuracy Trade-off; Limitations point 5] The helpfulness cost (correct diagnosis -30.0 points overall; Gemini 3.5 Flash -58 points) is scored by a single correctness judge (GPT-5.4-mini) with no clinician validation. Given the paper's own central finding that a single LLM judge can be severely miscalibrated, these magnitudes are load-bearing for the 'model-specific helpfulness costs' claim. The Discussion calls this provisional, but the Abstract reports the 80.3% to 50.3% drop without the same caveat. Please either provide a blinded human spot-check of the correctness labels on a paired subset or clearly mark all per-model accuracy magnitudes as unvalidated in the Abstract and Results, not only in a later limitation paragraph.

minor comments (5)

[Tables 3 and 7] Table 3 has broken column headers (e.g., 'prompt_co ndition') and irregular spacing; Table 7 also has inconsistent number alignment. These should be cleaned up for readability.
[Table 2] n_outputs differs between models (1000 for GPT-5.5 and Claude Opus 4.8; 840 for Gemini 3.5 Flash and Grok 4.3). Add a footnote explaining that the extra outputs come from the paraphrase and decode-robustness subsets, so readers do not infer differential panel sizes.
[Methods vs Protocol statistical plan] The protocol lists answer_length_words as a covariate in the GEE model, but the Methods and Results describe adjustment for model, dataset, and perturbation type only. Clarify whether answer length was included in the final sensitivity model and, if not, why.
[Results: Mechanism decomposition] The statement that the sum of the scaffold-structure (9.0) and abstention-content (14.8) components 'matched' the full wrapper effect (23.8) is algebraically guaranteed when the same subset is used for all contrasts. This is not an empirical check; the component bootstrap CIs are the informative part. Please rephrase to avoid suggesting that the additive agreement provides independent confirmation.
[Results: Decode stability] The text reports replicate results for 'two lower-cost models' but does not name them. Name the models in the text or table so the robustness analysis is reproducible.

Circularity Check

0 steps flagged

No significant circularity: the primary endpoint is explicitly judge-conditional, scaffold controls directly test the token-reward threat, and cross-judge plus clinician anchoring provide external checks.

full rationale

The paper's derivation chain does not reduce to its inputs. The primary endpoint is a pre-specified paired risk difference under a fixed LLM judge, explicitly framed as 'a computational quantity conditional on that judge, not a direct measure of clinical safety' (Statistical Analysis). No parameter is fitted to the outcome and then reported as a prediction; the 24.7-point reduction is a measured contrast. The main circularity threat—that the judge simply rewards the wrapper's section tokens—is directly tested by the matched scaffold arms: identical tokens (EVIDENCE PRESENT, EVIDENCE MISSING, SUFFICIENCY JUDGMENT) yield unsafe rates of 22.9% (wrapper), 37.7% (neutral scaffold), and 79.8% (format scaffold), so the label is not determined by token presence (Results, Mechanism and Circularity Control). The conclusion is further anchored by a different-family judge (Claude Sonnet 5) that reproduces the direction on the same 1,200 pairs, and by a blinded clinician review that characterizes the primary judge as high-sensitivity/low-specificity; these are external checks, not self-citations. The references are to public benchmarks and prior unrelated groups; no load-bearing self-citation or imported uniqueness theorem appears. The additive decomposition (9.0 + 14.8 = 23.8 points) is an arithmetic identity of contrasts, but it is presented as a breakdown of the measured effect, not as evidence that predicts the effect. The paper itself concedes that 'a condition-dependent bias could still distort the split' (Discussion), meaning a subtle judge-stylistic bias remains untested by the scaffold controls. That is a genuine validity limitation—the skeptic's concern is legitimate—but it is not a construction-level circularity: the headline claims are already qualified as judge-relative and direction/ranking claims, and the main result would not be true by definition even if the bias existed. Hence the circularity score is low.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

No fitted free parameters or invented entities are introduced; the paper's results are empirical. The central claim rests on construct validity of the judge/rubric, the human reference standard, the correctness judge, and the scaffold-control assumption.

axioms (5)

domain assumption The rubric-based LLM judge label is a valid screen for the construct of unsafe overconfidence.
The primary endpoint is operationalized by GPT-5.4-nano's binary rubric output; the paper explicitly calls this a computational first-pass label, not clinical ground truth (Methods: Scoring).
domain assumption The blinded clinician majority is an adequate reference standard for judge sensitivity/specificity.
Only two reliable raters and 6-8 clinician-unsafe cases support the sensitivity estimate; acknowledged in Results: Clinician Review and Limitations point 3.
domain assumption GPT-5.4-mini correctly labels diagnostic correctness on the MedRBench answerable subset.
The helpfulness trade-off (80.3% to 50.3% correct) depends on this correctness judge, which has not been validated against humans (Limitations point 5).
domain assumption The matched scaffold controls fully isolate wrapper structure from abstention content; no condition-dependent judge bias distorts the contrasts.
The authors concede that "a condition-dependent bias could still distort the split" (Discussion, Mechanism and Circularity Control), so the additive 9.0/14.8 decomposition depends on this assumption.
domain assumption Synthetic input perturbations are representative of clinically meaningful incomplete-information settings.
The paper notes perturbations are synthetic and may not capture all forms of real clinical ambiguity (Limitations point 6).

pith-pipeline@v1.3.0-alltime-deepseek · 16562 in / 12444 out tokens · 96791 ms · 2026-08-01T16:01:11.319603+00:00 · methodology

0 comments

read the original abstract

Background: LLM judges increasingly score whether clinical language models give overconfident answers under incomplete evidence, yet whether a measured "safety gain" reflects real behavior change or the judge's calibration is unresolved. Using a structured evidence-sufficiency prompt as a test case, we asked whether it reduces unsafe overconfident answers, how far that effect depends on the scoring judge, and what it costs in helpfulness. Methods: In a retrospective public-data benchmark (Real-POCQi, HealthBench, MedRBench), four models (GPT-5.5, Claude Opus 4.8, Gemini 3.5 Flash, Grok 4.3) answered a fully paired common panel (1,200 cells) with a standard prompt and the wrapper. The pre-specified endpoint was the paired reduction in unsafe overconfidence scored by the primary judge (GPT-5.4-nano); secondary analyses added a different-family judge (Claude Sonnet 5), a correctness judge, matched scaffold controls, and a blinded three-clinician review. Results: Unsafe overconfidence fell from 49.3% to 24.7%, a paired reduction of 24.7 points (95% CI 21.8-27.7; p<0.001), robust in direction across models and paraphrases. Magnitude was judge-dependent: Sonnet agreed on direction but nearly halved the effect (+13.1 points), with one-directional disagreement. Blinded clinicians characterized the primary judge as a high-sensitivity (1.00), low-specificity (0.55) screen, not a calibrated rate. The gain carried a model-specific helpfulness cost (correct diagnosis 80.3% to 50.3%): near-free for GPT-5.5, near-total for Gemini (-58 points). Matched scaffold controls showed genuine behavior change, not judge circularity. Conclusions: LLM-judged clinical safety effects should be reported as directional and relative, anchored to human review and evaluated jointly with helpfulness, not as calibrated absolute rates. This does not establish clinical deployment readiness.

discussion (0)

Reference graph

Works this paper leans on

12 extracted references · 1 canonical work pages

[1]

Evaluation and mitigation of the limitations of large language models in clinical decision-making

Hager P, Jungmann F, Holland R, Bhagat K, Hubrecht I, Knauer M, Vielhauer J, Makowski M, Braren R, Kaissis G, Rueckert D. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature Medicine. 2024. doi:10.1038/s41591-024-03097-1

doi:10.1038/s41591-024-03097-1 2024
[2]

Health AI readiness evaluation and robustness stress-testing resource

Gu Y, et al. Health AI readiness evaluation and robustness stress-testing resource. Nature Medicine
[3]

Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries

Feng JJ, et al. Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries. arXiv:2606.28960. 2026. https://arxiv.org/abs/2606.28960

Pith/arXiv arXiv 2026
[4]

Real-POCQi dataset

Feng Lab. Real-POCQi dataset. Hugging Face. 2026. https://huggingface.co/datasets/jjfenglab/Real- POCQi

2026
[5]

HealthBench

OpenAI. HealthBench. 2025. https://openai.com/index/healthbench/

2025
[6]

Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases

Qiu P, Wu C, Liu S, Zhao W, Chen Z, Gu H, Peng C, Zhang Y, Wang Y, Xie W. Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases. arXiv:2503.04691. 2025. https://arxiv.org/abs/2503.04691

Pith/arXiv arXiv 2025
[7]

MedRBench

MAGIC-AI4Med. MedRBench. GitHub. 2025. https://github.com/MAGIC-AI4Med/MedRBench

2025
[8]

MIMIC-IV-Ext Clinical Decision Making Dataset

Hager P, et al. MIMIC-IV-Ext Clinical Decision Making Dataset. PhysioNet. Version 1.1. https://physionet.org/content/mimic-iv-ext-cdm/1.1/
[9]

DECIDE-AI: Developmental and Exploratory Clinical Investigations of Decision support systems driven by Artiﬁcial Intelligence

Vasey B, Nagendran M, Campbell B, Clifton DA, Collins GS, Denaxas S, et al. DECIDE-AI: Developmental and Exploratory Clinical Investigations of Decision support systems driven by Artiﬁcial Intelligence. Nature Medicine. 2022. doi:10.1038/s41591-022-01772-9

doi:10.1038/s41591-022-01772-9 2022
[10]

Reporting guidelines for clinical trials evaluating artiﬁcial intelligence interventions: the CONSORT-AI and SPIRIT-AI extensions

Liu X, Cruz Rivera S, Moher D, Calvert MJ, Denniston AK, SPIRIT-AI and CONSORT-AI Working Group. Reporting guidelines for clinical trials evaluating artiﬁcial intelligence interventions: the CONSORT-AI and SPIRIT-AI extensions. Nature Medicine. 2020. doi:10.1038/s41591-020-1037-7

doi:10.1038/s41591-020-1037-7 2020
[11]

STARD-AI reporting guideline for diagnostic accuracy studies involving artiﬁcial intelligence

STARD-AI Steering Group. STARD-AI reporting guideline for diagnostic accuracy studies involving artiﬁcial intelligence. Nature Medicine. 2025. doi:10.1038/s41591-025-03953-8

doi:10.1038/s41591-025-03953-8 2025
[2026]

Code and data release: https://github.com/aiden-ygu/health-ai- readiness-eval/tree/v1.0.0

doi:10.1038/s41591-026-04501-8. Code and data release: https://github.com/aiden-ygu/health-ai- readiness-eval/tree/v1.0.0. Zenodo: https://doi.org/10.5281/zenodo.20047288

work page doi:10.1038/s41591-026-04501-8

[1] [1]

Evaluation and mitigation of the limitations of large language models in clinical decision-making

Hager P, Jungmann F, Holland R, Bhagat K, Hubrecht I, Knauer M, Vielhauer J, Makowski M, Braren R, Kaissis G, Rueckert D. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature Medicine. 2024. doi:10.1038/s41591-024-03097-1

doi:10.1038/s41591-024-03097-1 2024

[2] [2]

Health AI readiness evaluation and robustness stress-testing resource

Gu Y, et al. Health AI readiness evaluation and robustness stress-testing resource. Nature Medicine

[3] [3]

Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries

Feng JJ, et al. Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries. arXiv:2606.28960. 2026. https://arxiv.org/abs/2606.28960

Pith/arXiv arXiv 2026

[4] [4]

Real-POCQi dataset

Feng Lab. Real-POCQi dataset. Hugging Face. 2026. https://huggingface.co/datasets/jjfenglab/Real- POCQi

2026

[5] [5]

HealthBench

OpenAI. HealthBench. 2025. https://openai.com/index/healthbench/

2025

[6] [6]

Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases

Qiu P, Wu C, Liu S, Zhao W, Chen Z, Gu H, Peng C, Zhang Y, Wang Y, Xie W. Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases. arXiv:2503.04691. 2025. https://arxiv.org/abs/2503.04691

Pith/arXiv arXiv 2025

[7] [7]

MedRBench

MAGIC-AI4Med. MedRBench. GitHub. 2025. https://github.com/MAGIC-AI4Med/MedRBench

2025

[8] [8]

MIMIC-IV-Ext Clinical Decision Making Dataset

Hager P, et al. MIMIC-IV-Ext Clinical Decision Making Dataset. PhysioNet. Version 1.1. https://physionet.org/content/mimic-iv-ext-cdm/1.1/

[9] [9]

DECIDE-AI: Developmental and Exploratory Clinical Investigations of Decision support systems driven by Artiﬁcial Intelligence

Vasey B, Nagendran M, Campbell B, Clifton DA, Collins GS, Denaxas S, et al. DECIDE-AI: Developmental and Exploratory Clinical Investigations of Decision support systems driven by Artiﬁcial Intelligence. Nature Medicine. 2022. doi:10.1038/s41591-022-01772-9

doi:10.1038/s41591-022-01772-9 2022

[10] [10]

Reporting guidelines for clinical trials evaluating artiﬁcial intelligence interventions: the CONSORT-AI and SPIRIT-AI extensions

Liu X, Cruz Rivera S, Moher D, Calvert MJ, Denniston AK, SPIRIT-AI and CONSORT-AI Working Group. Reporting guidelines for clinical trials evaluating artiﬁcial intelligence interventions: the CONSORT-AI and SPIRIT-AI extensions. Nature Medicine. 2020. doi:10.1038/s41591-020-1037-7

doi:10.1038/s41591-020-1037-7 2020

[11] [11]

STARD-AI reporting guideline for diagnostic accuracy studies involving artiﬁcial intelligence

STARD-AI Steering Group. STARD-AI reporting guideline for diagnostic accuracy studies involving artiﬁcial intelligence. Nature Medicine. 2025. doi:10.1038/s41591-025-03953-8

doi:10.1038/s41591-025-03953-8 2025

[12] [2026]

Code and data release: https://github.com/aiden-ygu/health-ai- readiness-eval/tree/v1.0.0

doi:10.1038/s41591-026-04501-8. Code and data release: https://github.com/aiden-ygu/health-ai- readiness-eval/tree/v1.0.0. Zenodo: https://doi.org/10.5281/zenodo.20047288

work page doi:10.1038/s41591-026-04501-8