REVIEW 3 major objections 5 minor 14 references
Pragmatic Policy Development via Interpretable Behavior Cloning
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper claims that treatment policies built from the most frequently chosen actions in each patient state—estimated by a tree-based behavior policy model—are interpretable, reliably evaluable, and on rheumatoid arthritis and sepsis…
desk verdict A genuinely useful, interpretable alternative to offline RL for clinical policies, with a real OPE credibility gap that the authors should be pushed to close. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Top-k behavior-cloning policy: renormalize the behavior-policy model's probabilities over the k most likely actions in a state and zero out the rest, so that k=1 yields a deterministic majority-treatment policy and k=K recovers the behavior policy itself. The behavior-policy model is a tree-based meta-estimator that separates a binary switch-treatment tree from a multi-class which-treatment tree, capturing the clinical tendency to stay on the current drug, with a separate baseline tree for first decisions. Leaves of these trees group patients with similar treatment propensities, and within-leaf average outcomes drive the outcome-guided variant. Weighted importance sampling with effective sample size is the evaluation machinery that quantifies how much overlap the target policy retains with the behavior policy.
What would settle it
A concrete check is to rerun the WIS evaluation with the reward replaced by a pre-treatment negative-control outcome that treatment cannot influence; if the MC or MC+O policies then appear to have nonzero value differences, the OPE pipeline is biased by confounding or other misspecification. A more decisive test would compare the k=1 majority-treatment policy against usual care in a randomized trial and see whether the trial confidence interval contains the paper's OPE estimate.
Extended reading notes
Core claim
The central discovery is that a target policy defined by renormalizing the behavior-policy model's probabilities over the k most probable actions in a state—with outcomes optionally steering selection among those k—inherits enough overlap with the behavior policy to make weighted importance sampling trustworthy. In the RA and sepsis experiments, the deterministic k=1 policy (always give the most common treatment in the patient's leaf) is estimated to beat current practice, with effective sample sizes of 406 and 64 versus 1.7–19.3 for the RL policies; increasing k trades away some estimated value for much larger effective sample sizes, especially in sepsis. The outcome-guided MC+O policies show larger estimated gains but with high variance, so the authors present them as promising but not decisive. The paper's position is that evaluation feasibility, not optimality, should drive policy construction in safety-critical settings.
Load-bearing premise
The value estimates are causal only if the observed state captures every variable that affects both treatment choice and outcome; the authors explicitly note that unmeasured confounders may remain, and any such confounder would bias the estimated advantage of the proposed policies, especially the outcome-guided ones.
Editorial extensions
If this is right
- In domains where an interpretable behavior-policy model fits well, reliable value estimates can be obtained for simple standardized-care policies without training any RL agent, lowering the barrier to clinical use.
- The knob k gives practitioners a direct bias-variance trade-off: smaller k targets larger potential improvements but fewer effective samples, while larger k approaches current practice, making evaluation feasibility a design choice rather than an afterthought.
- Offline RL policies in these two settings had effective sample sizes too small to support their value estimates, so claims of superiority from such black-box policies should be treated as unverified until the overlap problem is addressed.
- Outcome-guided top-k policies offer a route to improve on the majority treatment when within-leaf outcome data are informative, but the high variance of their OPE estimates limits current conclusions.
- The structure-aware meta-model improves behavior-policy accuracy in RA by exploiting the strong tendency to continue existing treatment, which also yields simpler and more interpretable trees.
Reading between the lines
- The framework does not depend on healthcare: any domain with repeated decisions made by many practitioners—education, social services, credit decisions—could derive standardizable, interpretable policies this way, provided a behavior-policy model that naturally groups states exists.
- The k=1 policy is a form of mode cloning; when clinician disagreement within a leaf is high, the most common treatment may be barely more frequent than alternatives, so reporting within-leaf treatment entropy could guide when a stochastic top-k policy is more appropriate than a deterministic one.
- The within-leaf average outcome used for MC+O is a simple estimator; pairing top-k selection with doubly robust or leaf-level weighted regression could reduce the variance the authors observe while preserving tree interpretability, though the paper does not test this.
- A natural external test is a randomized comparison of the k=1 majority-treatment protocol against usual care in one of the two settings, since the paper's causal claim depends on no unmeasured confounding.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a framework it calls pragmatic policy development: an interpretable behavior policy is estimated with a tree-based meta-model (separate trees for treatment switching, treatment choice, and, in the rheumatoid arthritis application, baseline choice), and candidate target policies are derived as the Top-k most frequent treatments in each leaf, optionally refined by leaf-level average outcomes. The target policies are evaluated with weighted importance sampling (WIS) on held-out data from two clinical registries (rheumatoid arthritis and sepsis) and compared against offline RL baselines. The paper reports that the most-common-treatment (MC) policies achieve higher estimated values than observed practice in RA with well-separated IQRs, and smaller, less certain improvements in sepsis, while attaining substantially larger effective sample sizes than the RL policies.
Significance. The framework is a sensible, practical alternative to black-box offline RL for clinical policy development, and the paper's emphasis on constructing policies that are evaluable and interpretable is valuable. The experimental protocol is careful in several respects: 50 replicated data splits, held-out test sets for OPE, calibration of the behavior model, and a structure-aware meta-model that exploits treatment persistence. The comparison with offline RL baselines, showing their very low effective sample sizes under WIS, is a useful practical point. However, the central empirical claim of outperforming current practice is not yet established: the sepsis evidence is weak, and the WIS estimates show an internal-control anomaly that the paper does not address. If the OPE concerns are resolved, the paper would be a useful contribution; as it stands, the evidence is suggestive rather than conclusive.
major comments (3)
- [Table 2, Random row; §5.2] The WIS estimate for the random sepsis policy is 97.9 (IQR 59.7–99.6) with ESS 1.6, the highest value in the sepsis column; in RA the random policy is estimated at 1.7 (IQR −4.8 to 5.1) with ESS 1.3, also above the behavior policy. These results are never discussed. A uniformly random treatment policy being near-optimal in sepsis is not clinically plausible, and the tiny ESS values show that these estimates are dominated by a small number of trajectories. This internal control failure indicates that the WIS estimator is not reliable for low-overlap policies in these data. Because the MC k=1 sepsis estimate (74.1, ESS 64.1) is produced by the same estimator and its IQR (66.8–82.9) overlaps substantially with the behavior-policy IQR (70.7–72.2), the paper's claim that MC policies are estimated to outperform current practice in sepsis is not supported.
- [Table 2, sepsis MC(k) rows] For sepsis, the estimated MC policy values increase with k: 74.1 for k=1, 74.6 for k=2, and 75.1 for k=3, even though increasing k makes the target policy closer to the behavior policy, whose value is 71.6. If the WIS estimates were tracking the true value, one would expect the estimates to move toward 71.6 as k approaches K. The observed trend is inconsistent with that expectation and suggests that the sepsis WIS estimates for the MC policies are either noisy or biased. The paper should explain this non-monotonicity or temper the claim that MC policies outperform the behavior policy in sepsis.
- [§5.2 and §B.3] The behavior policy model p̂μ used to construct the MC target policies is the same model used in the denominator of the WIS importance weights. Since the target policy is defined through Top-k of p̂μ, any systematic error in p̂μ affects both which actions are selected and the weights assigned to those actions; for example, if the model overestimates the probability of the selected action, the WIS weights are deflated and the effective sample size is inflated. The paper should either use cross-fitting with an independently estimated behavior model for OPE, or provide a sensitivity analysis showing that the value estimates and the random-policy anomaly are robust to the choice and estimation of p̂μ. Without this, the reported ESS values and value estimates may overstate the reliability of the evaluation.
minor comments (5)
- [§4, Eq. (3)] The notation for p_t^μ(k|S_t) is confusing: it is first defined as p̂μ(A_t=k|S_t), but the classifier is trained only on treatment-switch events, so it should be defined as p̂μ(A_t=k | C_t=1, S_t); as written, the indicator in the definition of tilde p_t appears redundant and the relationship between the different quantities is hard to follow.
- [§5.2 and §B.2] The random policy is said to be 'made deterministic' in RA and softened in sepsis; please clarify what 'deterministic random policy' means (e.g., a fixed state-to-action mapping drawn once) and describe how it is implemented.
- [Table 2] The paper should report the number of evaluation trajectories in the test sets, since ESS values such as 64.1 in sepsis are only interpretable relative to the total number of trajectories; the behavior-policy ESS in sepsis is reported as a constant 2297.0 with a degenerate IQR, which is better replaced by the trajectory count.
- [Abstract] The abstract states that policies derived under the framework 'can outperform current practice,' while the body more carefully says 'estimated to'; the abstract should match the evidence and say 'are estimated to' to avoid overclaiming.
- [Figure 3] Figure 3 shows the MC policy value for k=1,...,8 in RA, where k=8 recovers the behavior policy; consider adding this point explicitly to the figure or Table 2 so that the convergence check is transparent.
Circularity Check
No significant circularity; the value estimates depend on held-out rewards and are not forced by the behavior-policy fit, though self-citations and the random-policy control warrant scrutiny.
full rationale
The paper's central derivation constructs target policies from the fitted behavior model p̂μ (Eq. 1) and evaluates them with WIS using the same p̂μ in the denominator. This shared model does not make the evaluation circular: the WIS estimate is a reward-weighted average over held-out test trajectories, and the MC policies' values are not determined by p̂μ alone. The k=8 RA case, where MC recovers the behavior policy, is an explicit consequence of Eq. (1) and is presented as such, not as a prediction. Self-citations (Matsson et al. 2024a, 2024b; Matsson and Johansson 2022) support state-representation choices and pattern interpretation, but the outperformance claim is tested on held-out data with external reward definitions, so no load-bearing step reduces to a self-citation. The undiscussed random-policy control in Table 2 (sepsis 97.9 vs. 71.6 for behavior, ESS 1.6) suggests the WIS estimates may be unreliable, but unreliability is a correctness/validity issue, not a circularity of the derivation chain.
Assumptions & free parameters
free parameters (4)
- k (number of most common treatments) =
k ∈ {1,...,8} in RA, {1,2,3} in sepsis
- p1 (additional switching probability) =
0 to 0.5
- Proportion of pre-b/tsDMARD observations in RA training data =
chosen from {0, 0.25, 0.5, 0.75, 1}
- Fraction of RA patients with pre-enrollment b/tsDMARD use =
chosen from {0, 0.25, 0.5, 0.75, 1}
assumptions (5)
- domain assumption No unmeasured confounding: St includes all variables affecting both treatment and outcome
- domain assumption Behavior policy model p̂μ is a well-specified and well-calibrated estimate of the true behavior policy
- domain assumption Overlap/positivity: pμ(a|s) > 0 whenever pπ(a|s) > 0
- domain assumption State representation (current covariates, previous action, previous reward, history aggregates) is Markovian
- domain assumption Reward functions reflect clinically meaningful outcomes (CDAI change, survival)
Cite this review
Pith. "Pith review of Pragmatic Policy Development via Interpretable Behavior Cloning." pith.science (2026). https://pith.science/paper/M3YUDMNL
@misc{pith2026250717056,
author = {Pith},
title = {Pith review of: Pragmatic Policy Development via Interpretable Behavior Cloning},
year = {2026},
howpublished = {\url{https://pith.science/paper/M3YUDMNL}},
note = {Machine review of arXiv:2507.17056}
}
read the original abstract
Offline reinforcement learning (RL) holds great promise for deriving optimal policies from observational data, but challenges related to interpretability and evaluation limit its practical use in safety-critical domains. Interpretability is hindered by the black-box nature of unconstrained RL policies, while evaluation -- typically performed off-policy -- is sensitive to large deviations from the data-collecting behavior policy, especially when using methods based on importance sampling. To address these challenges, we propose a simple yet practical alternative: deriving treatment policies from the most frequently chosen actions in each patient state, as estimated by an interpretable model of the behavior policy. By using a tree-based model, which is specifically designed to exploit patterns in the data, we obtain a natural grouping of states with respect to treatment. The tree structure ensures interpretability by design, while varying the number of actions considered controls the degree of overlap with the behavior policy, enabling reliable off-policy evaluation. This pragmatic approach to policy development standardizes frequent treatment patterns, capturing the collective clinical judgment embedded in the data. Using real-world examples in rheumatoid arthritis and sepsis care, we demonstrate that policies derived under this framework can outperform current practice, offering interpretable alternatives to those obtained via offline RL.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Only the subtrajectory up to the point of an excluded visit was considered for off-policy evaluation. Following (Matsson et al., 2024a), we restricted the analysis to classes of DMARDs rather than individ- ual drugs. Specifically, we divided bDMARDs into TNF inhibitors and non-TNF inhibitors, resulting in the following classes of drugs: csDMARDs, TNF in- ...
work page 2018
-
[5]
The doctor just won’t accept that! arXiv preprint arXiv:1711.08037 ,
Zachary C Lipton. The doctor just won’t accept that! arXiv preprint arXiv:1711.08037 ,
-
[6]
For DT-S and DT-BLS, we used the same hyperparameters as for DT for each decision tree in these meta-models. As noted above, in the RA set- ting, we introduced two additional hyperparameters: the proportion of observations collected before the first b/tsDMARD initiation, and the fraction of ob- servations from patients who initiated b/tsDMARD therapy prio...
work page 2015
-
[8]
For the other methods—DQN, BCQ, and CQL— we used the implementation provided by Luo et al
Gender, n (%) 1,562 Male 389 (24.9) Female 1,173 (75.1) BMI, n (%) 1,535 Underweight 14 (0.9) Healthy weight 356 (23.2) Overweight 492 (32.1) Obesity 673 (43.8) Blood pressure, n (%) 1,561 Elevated 247 (15.8) Hypertension stage 1 531 (34.0) Hypertension stage 2 390 (25.0) Normal 393 (25.2) Currently pregnant, n (%) 1,041 3 (0.3) Pregnant since last visit,...
work page 2024
-
[9]
Behavioural cloning for driving robots over rough terrain
Raymond Sheh, Bernhard Hengst, and Claude Sam- mut. Behavioural cloning for driving robots over rough terrain. In Proceedings of the 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 732–737,
work page 2011
-
[1983]
Conservative Q- improvement: Reinforcement learning for an interpretable decision-tree policy
Aaron M Roth, Nicholay Topin, Pooyan Jamshidi, and Manuela Veloso. Conservative Q- improvement: Reinforcement learning for an interpretable decision-tree policy. arXiv preprint arXiv:1907.01180,
arXiv 1907
-
[2000]
Deep reinforcement learning for sepsis treatment
Aniruddh Raghu, Matthieu Komorowski, Imran Ahmed, Leo Celi, Peter Szolovits, and Marzyeh Ghassemi. Deep reinforcement learning for sepsis treatment. arXiv preprint arXiv:1711.09602,
-
[2014]
Distilling deep rein- forcement learning policies in soft decision trees
Youri Coppens, Kyriakos Efthymiadis, Tom Lenaerts, Ann Now´ e, Tim Miller, Rosina Weber, and Daniele Magazzeni. Distilling deep rein- forcement learning policies in soft decision trees. In Proceedings of the IJCAI 2019 Workshop on Explainable Artificial Intelligence, pages 1–6,
work page 2019
Show all 14 references
-
[2016]
Smolen, Robert B.M
Josef S. Smolen, Robert B.M. Landew´ e, Sytske Anne Bergstra, Andreas Kerschbaumer, Alexandre Sepriano, Daniel Aletaha, Roberto Caporali, Christopher John Edwards, Kimme L. Hyrich, Janet E. Pope, et al. EULAR recommenda- tions for the management of rheumatoid arthri- tis with ...
2022
-
[2018]
Off-policy deep reinforcement learning without ex- ploration
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without ex- ploration. In Proceedings of the 36th Interna- tional conference on machine learning , volume 97 of PMLR, pages 2052–2062,
-
[2019]
Offline reinforcement learning: Tutorial, review, and perspectives on open problems
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643,
2005 arXiv
-
[2020]
Measuring calibration in deep learning
Jeremy Nixon, Mike Dusenberry, Ghassen Jerfel, Timothy Nguyen, Jeremiah Liu, Linchuan Zhang, and Dustin Tran. Measuring calibration in deep learning. arXiv preprint arXiv:1904.01685 ,
1904 arXiv
-
[2021]
Datasets Our experimental evaluation was based on two dis- tinct datasets related to the management of rheuma- toid arthritis (RA) and sepsis
Appendix A. Datasets Our experimental evaluation was based on two dis- tinct datasets related to the management of rheuma- toid arthritis (RA) and sepsis. Here, we provide ad- ditional details on the datasets, including the full set 12 Pragmatic Policy Development via Interpre...
2023
-
[2024]
Evaluating reinforcement learning al- gorithms in observational health settings
Omer Gottesman, Fredrik Johansson, Joshua Meier, Jack Dent, Donghun Lee, Srivatsan Srinivasan, Linying Zhang, Yi Ding, David Wihl, Xuefeng Peng, et al. Evaluating reinforcement learning al- gorithms in observational health settings. arXiv preprint arXiv:1805.12298,
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.