Do Clinical Models Change Treatment Decisions?

Dongkyu Cho; Miao Zhang; Rumi Chunara

arxiv: 2605.28129 · v1 · pith:XIWG2NYCnew · submitted 2026-05-27 · 💻 cs.AI

Do Clinical Models Change Treatment Decisions?

Dongkyu Cho , Miao Zhang , Rumi Chunara This is my paper

Pith reviewed 2026-06-29 12:03 UTC · model grok-4.3

classification 💻 cs.AI

keywords clinical decision makingtreatment adaptationmedical QA evaluationbenchmark constructionmodel supervisioncontext pivotingfrontier models

0 comments

The pith

Strong medical QA performance does not reliably predict whether models change treatment decisions when patient context changes.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces ClinPivot, a benchmark that tests whether clinical models adjust treatment recommendations when new patient constraints are introduced through pivoted contexts derived from biomedical relations. It demonstrates that high scores on factual medical QA do not guarantee correct decision changes, causing model rankings to shift between evaluation regimes. Decision-structured supervision improves performance on both adaptive decisions and QA under matched knowledge budgets, while replay techniques help retain general capabilities. A sympathetic reader would care because treatment decisions in medicine depend on adapting to shifting contexts rather than static factual recall.

Core claim

Strong performance on medical QA benchmarks does not reliably predict decision-making performance on treatment choices that must change with new clinical constraints. Frontier models and task-adapted Qwen variants often fail to change decisions correctly in response to pivoted patient contexts, and model rankings shift across evaluation regimes. Decision-structured supervision improves pivot-sensitive decision-making and medical QA under matched knowledge budgets, while lightweight replay reduces losses in general assistant ability.

What carries the argument

ClinPivot, an auditable benchmark that constructs pivoted patient contexts from biomedical relations to test whether models change treatment choices when the action space shifts.

If this is right

Models that excel at medical QA may still fail to adjust treatments when patient constraints change.
Model rankings on decision-making tasks differ from those on standard QA evaluations.
Decision-structured supervision during training improves both adaptive decision-making and QA performance.
Lightweight replay during specialized training can preserve general assistant abilities.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Evaluation of clinical AI should incorporate explicit tests of context-driven adaptation rather than relying solely on QA accuracy.
Training approaches that emphasize decision pivots may transfer to other adaptive reasoning tasks outside medicine.
Benchmarks like ClinPivot could be extended to measure correlation with actual clinical outcomes in prospective studies.

Load-bearing premise

The pivoted patient contexts constructed in ClinPivot from biomedical relations accurately represent real clinical scenarios in which treatment decisions should change.

What would settle it

A controlled study in which expert clinicians review model outputs on real patient cases with documented context shifts and find that models failing ClinPivot still make the correct treatment changes would falsify the central claim.

Figures

Figures reproduced from arXiv: 2605.28129 by Dongkyu Cho, Miao Zhang, Rumi Chunara.

**Figure 1.** Figure 1: Models with high MedQA accuracy still show large drops on CLINPIVOT. Unmarked frontiermodel accuracies are from [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

**Figure 2.** Figure 2: Decision-structured supervision improves CLINPIVOT more than QA-only supervision. Panels A and B report Qwen-3-4B and Qwen-3-8B results from [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: Replay recovers much of the general reten [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

read the original abstract

Clinical foundation models are evaluated with factual or exam-style medical QA, but treatment decisions must change when patient context changes. We introduce ClinPivot, an auditable treatment-decision benchmark built from biomedical relations and pivoted patient contexts. ClinPivot asks whether models change treatment choices when new clinical constraints shift the action space. We find that strong medical QA performance does not reliably predict decision-making performance: frontier models and task-adapted Qwen variants often fail to change decisions correctly, and model rankings shift across evaluation regimes. Decision-structured supervision improves pivot-sensitive decision-making and medical QA under matched knowledge budgets, while lightweight replay reduces losses in general assistant ability.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

ClinPivot tests whether QA scores predict if models update treatment decisions on context shifts, and finds they often don't, but the pivots' realism is the open question.

read the letter

The new piece is ClinPivot, a benchmark that starts from biomedical relations, applies pivots to change patient context and the available actions, then measures whether models actually switch their treatment choice. The main result is that high medical QA scores do not track this behavior: frontier models and fine-tuned Qwen variants frequently fail to make the right change, and the model ordering changes when you switch from QA to this decision task. Decision-structured supervision improves the pivot performance without hurting general ability as much.

That framing is useful because it directly asks whether current evaluation regimes capture what matters for treatment decisions. The paper gives a concrete way to surface the gap.

The soft spot is the pivots themselves. They are built synthetically by altering relations to shift the action space, but the abstract gives no evidence that these scenarios match real clinical constraints or that clinicians would agree the decision should change. Without that check, the observed failures and ranking shifts could be specific to how the benchmark was constructed rather than a general property of the models.

Methods details are also thin: no sample sizes, no error breakdown, no description of how the relations were chosen or validated. That makes it hard to know how stable the findings are.

This is for people building or evaluating clinical decision systems who want to move past pure QA tests. It deserves peer review so the benchmark construction can be examined and the results replicated with more controls.

Referee Report

2 major / 2 minor

Summary. The paper introduces ClinPivot, an auditable benchmark constructed from biomedical relations and pivoted patient contexts, to test whether clinical models appropriately alter treatment decisions when new constraints shift the action space. It claims that strong medical QA performance does not reliably predict decision-making performance, that frontier models and task-adapted Qwen variants frequently fail to change decisions correctly, that model rankings shift across evaluation regimes, and that decision-structured supervision improves pivot-sensitive decision-making while preserving general capabilities under matched knowledge budgets.

Significance. If the benchmark construction is shown to produce valid instances of treatment change, the result would demonstrate a meaningful gap between standard medical QA evaluations and clinical decision-making requirements, with implications for how clinical foundation models are assessed and trained. The use of relation-based pivots for an auditable benchmark is a constructive methodological contribution.

major comments (2)

[ClinPivot benchmark construction] ClinPivot benchmark construction (abstract and methods): the central claim that QA performance fails to predict decision-making performance rests on the assumption that pivoted contexts constitute realistic scenarios where treatment should change due to clinical constraints; however, the construction extracts relations and applies pivots without described independent validation against expert judgment or real patient trajectories, raising the risk that observed failures and ranking shifts are artifacts of synthetic benchmark design rather than generalizable findings.
[Evaluation and results] Evaluation and results sections: the reported shifts in model rankings across QA and decision-making regimes, and the claim that decision-structured supervision improves performance, require explicit details on sample sizes, statistical tests for ranking changes, error analysis, and controls for knowledge overlap between QA and pivot tasks to establish that the disconnect is not driven by mismatched difficulty or data leakage.

minor comments (2)

[Abstract] The abstract states findings on frontier models and Qwen variants but does not specify the exact models, number of pivots, or decision categories evaluated; adding these details would improve clarity.
[Introduction] Notation for 'pivoted patient contexts' and 'action space' shifts could be defined more explicitly on first use to aid readers unfamiliar with the benchmark.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive report. We address the two major comments point-by-point below. We agree that both points require additional detail and will revise the manuscript accordingly.

read point-by-point responses

Referee: [ClinPivot benchmark construction] ClinPivot benchmark construction (abstract and methods): the central claim that QA performance fails to predict decision-making performance rests on the assumption that pivoted contexts constitute realistic scenarios where treatment should change due to clinical constraints; however, the construction extracts relations and applies pivots without described independent validation against expert judgment or real patient trajectories, raising the risk that observed failures and ranking shifts are artifacts of synthetic benchmark design rather than generalizable findings.

Authors: The benchmark is deliberately synthetic and constructed from publicly verifiable biomedical relations (e.g., drug-disease, contraindication triples drawn from established KBs) so that every pivot instance remains fully auditable. We acknowledge that the current manuscript does not describe an independent expert validation step. In revision we will add (i) an explicit description of the pivot-generation algorithm, (ii) the criteria used to select clinically meaningful constraints, and (iii) results of a small-scale expert review (n=50 instances) confirming that the pivoted contexts are plausible and that the expected treatment change is clinically justified. This will be reported without claiming equivalence to real patient trajectories. revision: yes
Referee: [Evaluation and results] Evaluation and results sections: the reported shifts in model rankings across QA and decision-making regimes, and the claim that decision-structured supervision improves performance, require explicit details on sample sizes, statistical tests for ranking changes, error analysis, and controls for knowledge overlap between QA and pivot tasks to establish that the disconnect is not driven by mismatched difficulty or data leakage.

Authors: We agree that these details are necessary. The revised manuscript will report: exact sample sizes per regime, Spearman rank correlation and paired statistical tests (McNemar / Wilcoxon) for ranking shifts, a categorized error analysis of failure modes, and explicit controls showing that QA and pivot tasks use disjoint relation sets with no entity overlap. These additions will be placed in the Evaluation and Results sections and will be accompanied by the corresponding numbers and p-values. revision: yes

Circularity Check

0 steps flagged

No circularity: benchmark construction and evaluation are independent of tested models

full rationale

The paper constructs ClinPivot from external biomedical relations and pivoted contexts, then empirically evaluates frontier and adapted models on decision-change tasks versus QA. No equations or steps reduce by construction to fitted inputs, self-definitions, or self-citation chains; the benchmark is presented as an external test set whose validity is separate from model outputs. Central findings (ranking shifts, failure to change decisions) are reported as observed results rather than tautologies. This matches the default case of a self-contained empirical study with no load-bearing circular reductions.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 1 invented entities

Abstract-only review; no free parameters, axioms, or invented entities beyond the benchmark itself can be audited from available text.

invented entities (1)

ClinPivot benchmark no independent evidence
purpose: Auditable treatment-decision benchmark using pivoted patient contexts
Newly introduced in the paper to test decision change.

pith-pipeline@v0.9.1-grok · 5624 in / 992 out tokens · 37753 ms · 2026-06-29T12:03:23.445920+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages

[1]

Forget forgetting: Continual learning in a world of abundant memory, 2026

Building a knowledge graph to enable pre- cision medicine.Scientific Data, 10(1):67. Dongkyu Cho, Taesup Moon, Rumi Chunara, Kyunghyun Cho, and Sungmin Cha. 2026a. Forget forgetting: Continual learning in a world of abundant memory.Preprint, arXiv:2502.07274. Dongkyu Cho, Amy B. Z. Zhang, Bilel Fehri, Sheng Wang, Rumi Chunara, Hengrui Cai, and Rui Song. 2...

work page arXiv 2025
[2]

Towards Generalist Biomedical AI,

Towards generalist biomedical ai.Preprint, arXiv:2307.14334. Hang Wu, Wenqi Shi, Anirudh Choudhary, and May D Wang. 2024a. Clinical decision making under un- certainty: a bootstrapped counterfactual inference approach.BMC Medical Informatics and Decision Making, 24(1):275. Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Ha...

work page arXiv 2025

[1] [1]

Forget forgetting: Continual learning in a world of abundant memory, 2026

Building a knowledge graph to enable pre- cision medicine.Scientific Data, 10(1):67. Dongkyu Cho, Taesup Moon, Rumi Chunara, Kyunghyun Cho, and Sungmin Cha. 2026a. Forget forgetting: Continual learning in a world of abundant memory.Preprint, arXiv:2502.07274. Dongkyu Cho, Amy B. Z. Zhang, Bilel Fehri, Sheng Wang, Rumi Chunara, Hengrui Cai, and Rui Song. 2...

work page arXiv 2025

[2] [2]

Towards Generalist Biomedical AI,

Towards generalist biomedical ai.Preprint, arXiv:2307.14334. Hang Wu, Wenqi Shi, Anirudh Choudhary, and May D Wang. 2024a. Clinical decision making under un- certainty: a bootstrapped counterfactual inference approach.BMC Medical Informatics and Decision Making, 24(1):275. Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Ha...

work page arXiv 2025