{"id":"55e3fb53-8f4b-45d3-b2c6-60018c78066b","arxiv_id":"2507.17056","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A decision-tree behavior cloning approach that chooses the most frequent treatments per patient state yields interpretable policies with reliable off-policy evaluation in two clinical datasets.","lead":"The authors propose a simple, interpretable method for deriving treatment policies from clinical data by cloning the most common decisions of clinicians, modeled with decision trees. They show that such policies can be evaluated reliably and, in rheumatoid arthritis and sepsis datasets, may improve patient outcomes compared with current practice.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"WIS reliability is not established: the random-policy control in Table 2 is estimated to outperform both current practice and the proposed MC policies, indicating the off-policy estimates supporting the outperform claim may be artifacts.","rationale":"The contribution is best read as a framework for constructing interpretable, overlap-friendly policies plus an empirical demonstration of their value. The ESS advantage over RL policies is structurally sound and likely holds. The load-bearing part is the claim of outperformance, which depends on WIS estimates. Table 2 provides a direct internal check: a uniformly random policy is a control with no plausible clinical superiority, yet WIS assigns it the highest sepsis value (97.9) and an above-practice RA value (1.7). With ESS around 1.6 the estimate is obviously noisy, but its presence in the same table and the absence of discussion is a red flag: the estimator has not been validated against a known-bad policy. The sepsis MC k=1 estimate has ESS 64.1, a modest improvement, and its IQR overlaps the behavior policy's IQR. The k=3 estimate has better ESS, but the same WIS/behavior-model machinery produced the random anomaly, so ESS alone is not enough. Unmeasured confounding is a valid concern (Section 6) and would also bias the outcome-guided variant, but because the authors disclose it and it is standard in observational studies, the random-policy failure is the more immediate, manuscript-internal reason to doubt the central claim. I also note that Eq. (3) as written is circular or ambiguous in defining p-tilde in terms of p, which should be fixed with a re-derivation and code release, though this is secondary to the OPE reliability concern. Conditional acceptance is appropriate: the authors should validate OPE with a negative control and a more robust estimator, report significance tests, and release code.","tokens_in":18804,"tokens_out":18214,"duration_ms":200834,"concrete_test":"Rerun Table 2 with a doubly robust or model-based estimator (e.g., DR or MAGIC) using the same behavior model, and add a negative-control policy (e.g., always the least common treatment). If the random policy's value collapses to at most the behavior policy and the MC advantage over practice shrinks, the WIS-based outperformance is an artifact; if DR still shows random near 97.9 and MC clearly above behavior, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 2 includes a policy that chooses treatments uniformly at random. Under the same WIS procedure, the random policy is estimated at 97.9 (IQR 59.7-99.6) in sepsis versus 71.6 for observed practice and 74.1-75.1 for the MC policies, and at 1.7 versus -1.1 for RA practice, despite ESS of roughly 1.3-1.6. A random sepsis dosing policy being near-optimal is not clinically plausible, and the paper never discusses this internal control failure. The central claim that the proposed policies are 'estimated to outperform current practice' relies on WIS estimates built from the same estimated behavior policy, and the MC k=1 sepsis estimate (ESS 64.1) uses the same estimator with only moderately larger support. Absent an explanation or a validated OPE check, the outperform claim is not supported; the anomaly suggests the WIS estimates are fragile and potentially optimistically biased for low-overlap policies, including the sepsis k=1 claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework it calls pragmatic policy development: an interpretable behavior policy is estimated with a tree-based meta-model (separate trees for treatment switching, treatment choice, and, in the rheumatoid arthritis application, baseline choice), and candidate target policies are derived as the Top-k most frequent treatments in each leaf, optionally refined by leaf-level average outcomes. The target policies are evaluated with weighted importance sampling (WIS) on held-out data from two clinical registries (rheumatoid arthritis and sepsis) and compared against offline RL baselines. The paper reports that the most-common-treatment (MC) policies achieve higher estimated values than observed practice in RA with well-separated IQRs, and smaller, less certain improvements in sepsis, while attaining substantially larger effective sample sizes than the RL policies.","tokens_in":19006,"tokens_out":9921,"duration_ms":110248,"significance":"The framework is a sensible, practical alternative to black-box offline RL for clinical policy development, and the paper's emphasis on constructing policies that are evaluable and interpretable is valuable. The experimental protocol is careful in several respects: 50 replicated data splits, held-out test sets for OPE, calibration of the behavior model, and a structure-aware meta-model that exploits treatment persistence. The comparison with offline RL baselines, showing their very low effective sample sizes under WIS, is a useful practical point. However, the central empirical claim of outperforming current practice is not yet established: the sepsis evidence is weak, and the WIS estimates show an internal-control anomaly that the paper does not address. If the OPE concerns are resolved, the paper would be a useful contribution; as it stands, the evidence is suggestive rather than conclusive.","major_comments":[{"comment":"The WIS estimate for the random sepsis policy is 97.9 (IQR 59.7–99.6) with ESS 1.6, the highest value in the sepsis column; in RA the random policy is estimated at 1.7 (IQR −4.8 to 5.1) with ESS 1.3, also above the behavior policy. These results are never discussed. A uniformly random treatment policy being near-optimal in sepsis is not clinically plausible, and the tiny ESS values show that these estimates are dominated by a small number of trajectories. This internal control failure indicates that the WIS estimator is not reliable for low-overlap policies in these data. Because the MC k=1 sepsis estimate (74.1, ESS 64.1) is produced by the same estimator and its IQR (66.8–82.9) overlaps substantially with the behavior-policy IQR (70.7–72.2), the paper's claim that MC policies are estimated to outperform current practice in sepsis is not supported.","section":"Table 2, Random row; §5.2"},{"comment":"For sepsis, the estimated MC policy values increase with k: 74.1 for k=1, 74.6 for k=2, and 75.1 for k=3, even though increasing k makes the target policy closer to the behavior policy, whose value is 71.6. If the WIS estimates were tracking the true value, one would expect the estimates to move toward 71.6 as k approaches K. The observed trend is inconsistent with that expectation and suggests that the sepsis WIS estimates for the MC policies are either noisy or biased. The paper should explain this non-monotonicity or temper the claim that MC policies outperform the behavior policy in sepsis.","section":"Table 2, sepsis MC(k) rows"},{"comment":"The behavior policy model p̂μ used to construct the MC target policies is the same model used in the denominator of the WIS importance weights. Since the target policy is defined through Top-k of p̂μ, any systematic error in p̂μ affects both which actions are selected and the weights assigned to those actions; for example, if the model overestimates the probability of the selected action, the WIS weights are deflated and the effective sample size is inflated. The paper should either use cross-fitting with an independently estimated behavior model for OPE, or provide a sensitivity analysis showing that the value estimates and the random-policy anomaly are robust to the choice and estimation of p̂μ. Without this, the reported ESS values and value estimates may overstate the reliability of the evaluation.","section":"§5.2 and §B.3"}],"minor_comments":[{"comment":"The notation for p_t^μ(k|S_t) is confusing: it is first defined as p̂μ(A_t=k|S_t), but the classifier is trained only on treatment-switch events, so it should be defined as p̂μ(A_t=k | C_t=1, S_t); as written, the indicator in the definition of tilde p_t appears redundant and the relationship between the different quantities is hard to follow.","section":"§4, Eq. (3)"},{"comment":"The random policy is said to be 'made deterministic' in RA and softened in sepsis; please clarify what 'deterministic random policy' means (e.g., a fixed state-to-action mapping drawn once) and describe how it is implemented.","section":"§5.2 and §B.2"},{"comment":"The paper should report the number of evaluation trajectories in the test sets, since ESS values such as 64.1 in sepsis are only interpretable relative to the total number of trajectories; the behavior-policy ESS in sepsis is reported as a constant 2297.0 with a degenerate IQR, which is better replaced by the trajectory count.","section":"Table 2"},{"comment":"The abstract states that policies derived under the framework 'can outperform current practice,' while the body more carefully says 'estimated to'; the abstract should match the evidence and say 'are estimated to' to avoid overclaiming.","section":"Abstract"},{"comment":"Figure 3 shows the MC policy value for k=1,...,8 in RA, where k=8 recovers the behavior policy; consider adding this point explicitly to the figure or Table 2 so that the convergence check is transparent.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The random-policy row in Table 2 is the key issue: it is an internal control that fails in a way the paper does not discuss. If the authors can show that the sepsis MC(k=1) estimate is robust to alternative OPE estimators (e.g., doubly robust or per-decision WIS), report diagnostics for the largest importance weights, and address the non-monotonic trend in the sepsis MC rows, I would be willing to consider a revised version. The manuscript is within the journal's scope, and the framework is simple enough that the practical contribution could justify publication despite being methodologically modest."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a serious look. The core idea is simple and practical: instead of optimizing a black-box policy, fit a tree-based model of the behavior policy and define the target policy as the top-k most common treatments, optionally guided by leaf-level outcomes. The separate switch and treatment trees are a nice trick for chronic care data, and the paper is honest that this is not optimal but evaluable. The RA results are fairly clean, with well-separated value estimates and ESS numbers that make sense.\n\nThe main soft spot is the OPE sanity check. Table 2 includes a random policy that under WIS is estimated to beat both the behavior policy and all MC policies in sepsis (97.9 vs 71.6). The ESS is around 1.3, so the point estimate is essentially noise, but the paper does not discuss this at all. A reader is left wondering whether WIS is systematically optimistic for low-overlap policies, and the sepsis MC k=1 estimate (ESS 64) uses the same estimator with only moderately more support. The paper's claim that MC policies 'can outperform current practice' is not strongly supported for sepsis; the IQRs overlap substantially. The RA results are more believable.\n\nThe unmeasured confounding issue is acknowledged, which is good, but it is particularly damaging for the MC+O variants, which are the ones showing the largest gains. The high variance in those estimates also limits their usefulness.\n\nI'd like to see the random policy control discussed and, ideally, an additional OPE method for cross-checking, plus some kind of statistical significance test or bootstrap CI rather than just IQRs. The paper would also be stronger with code release, given the reproducibility issues in this literature.\n\nOverall: the method is a solid, interpretable alternative that deserves airing. The authors know the limitations and are transparent. This should go to peer review, not be desk-rejected.","headline":"A genuinely useful, interpretable alternative to offline RL for clinical policies, with a real OPE credibility gap that the authors should be pushed to close.","tokens_in":19524,"tokens_out":2344,"would_cite":true,"duration_ms":40750,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that treatment policies built from the most frequently chosen actions in each patient state—estimated by a tree-based behavior policy model—are interpretable, reliably evaluable, and on rheumatoid arthritis and sepsis…","keywords":["clinical decision-making","reinforcement learning","off-policy evaluation","interpretability","behavior cloning","decision trees","importance sampling","effective sample size"],"falsifier":"A concrete check is to rerun the WIS evaluation with the reward replaced by a pre-treatment negative-control outcome that treatment cannot influence; if the MC or MC+O policies then appear to have nonzero value differences, the OPE pipeline is biased by confounding or other misspecification. A more decisive test would compare the k=1 majority-treatment policy against usual care in a randomized trial and see whether the trial confidence interval contains the paper's OPE estimate.","tokens_in":18591,"feed_emoji":"🩺","tokens_out":6093,"duration_ms":61854,"temperature":0.7,"pith_summary":"Offline reinforcement learning promises data-driven treatment policies but is hampered by black-box policies and noisy off-policy evaluation. This paper proposes a simpler route: clone the behavior policy with an interpretable decision-tree model, then act on the most frequent treatment in each leaf, or on the outcome-best among the top k treatments. Because every recommended action appears in the observed data, importance-sampling evaluation keeps a large effective sample size, and the tree structure makes the policy readable by clinicians. On rheumatoid arthritis and sepsis data, the resulting policies are estimated to outperform current practice on average, with far larger effective sample sizes than four offline RL baselines. The authors frame the approach as standardizing frequent treatment patterns rather than chasing optimality, and they caution that the causal interpretation of the value estimates relies on no unmeasured confounding.","feed_headline":"Most-common-treatment policies beat offline RL in two care datasets","feed_subtitle":"Tree-based policies stay readable while yielding far more effective samples than offline RL in evaluation.","key_machinery":"The central object is the Top-k behavior-cloning policy: renormalize the behavior-policy model's probabilities over the k most likely actions in a state and zero out the rest, so that k=1 yields a deterministic majority-treatment policy and k=K recovers the behavior policy itself. The behavior-policy model is a tree-based meta-estimator that separates a binary switch-treatment tree from a multi-class which-treatment tree, capturing the clinical tendency to stay on the current drug, with a separate baseline tree for first decisions. Leaves of these trees group patients with similar treatment propensities, and within-leaf average outcomes drive the outcome-guided variant. Weighted importance sampling with effective sample size is the evaluation machinery that quantifies how much overlap the target policy retains with the behavior policy.","core_discovery":"The central discovery is that a target policy defined by renormalizing the behavior-policy model's probabilities over the k most probable actions in a state—with outcomes optionally steering selection among those k—inherits enough overlap with the behavior policy to make weighted importance sampling trustworthy. In the RA and sepsis experiments, the deterministic k=1 policy (always give the most common treatment in the patient's leaf) is estimated to beat current practice, with effective sample sizes of 406 and 64 versus 1.7–19.3 for the RL policies; increasing k trades away some estimated value for much larger effective sample sizes, especially in sepsis. The outcome-guided MC+O policies show larger estimated gains but with high variance, so the authors present them as promising but not decisive. The paper's position is that evaluation feasibility, not optimality, should drive policy construction in safety-critical settings.","pith_inferences":["The framework does not depend on healthcare: any domain with repeated decisions made by many practitioners—education, social services, credit decisions—could derive standardizable, interpretable policies this way, provided a behavior-policy model that naturally groups states exists.","The k=1 policy is a form of mode cloning; when clinician disagreement within a leaf is high, the most common treatment may be barely more frequent than alternatives, so reporting within-leaf treatment entropy could guide when a stochastic top-k policy is more appropriate than a deterministic one.","The within-leaf average outcome used for MC+O is a simple estimator; pairing top-k selection with doubly robust or leaf-level weighted regression could reduce the variance the authors observe while preserving tree interpretability, though the paper does not test this.","A natural external test is a randomized comparison of the k=1 majority-treatment protocol against usual care in one of the two settings, since the paper's causal claim depends on no unmeasured confounding."],"forward_implications":["In domains where an interpretable behavior-policy model fits well, reliable value estimates can be obtained for simple standardized-care policies without training any RL agent, lowering the barrier to clinical use.","The knob k gives practitioners a direct bias-variance trade-off: smaller k targets larger potential improvements but fewer effective samples, while larger k approaches current practice, making evaluation feasibility a design choice rather than an afterthought.","Offline RL policies in these two settings had effective sample sizes too small to support their value estimates, so claims of superiority from such black-box policies should be treated as unverified until the overlap problem is addressed.","Outcome-guided top-k policies offer a route to improve on the majority treatment when within-leaf outcome data are informative, but the high variance of their OPE estimates limits current conclusions.","The structure-aware meta-model improves behavior-policy accuracy in RA by exploiting the strong tendency to continue existing treatment, which also yields simpler and more interpretable trees."],"supporting_citations":[{"why":"Supplies the weighted importance sampling estimator used for all off-policy value estimates.","marker":"(Precup et al., 2000)"},{"why":"Defines the effective sample size used to quantify the reliability of the importance-sampling estimates.","marker":"(Owen, 2013)"},{"why":"Provides the behavior-cloning framing that the proposed approach builds on.","marker":"(Torabi et al., 2018)"},{"why":"Justifies matching on propensity scores within tree leaves as a way to adjust for confounding.","marker":"(Rosenbaum and Rubin, 1983)"},{"why":"BCQ is a baseline offline RL method explicitly designed to stay close to the behavior policy.","marker":"(Fujimoto et al., 2019)"},{"why":"CQL is a baseline conservative offline RL method used in the comparisons.","marker":"(Kumar et al., 2020)"},{"why":"Provides the sepsis data collection and preprocessing code as well as the Q-learning-style clinical decision baseline.","marker":"(Komorowski et al., 2018)"},{"why":"Supplies patient exclusions and the RL implementations used for the DQN, BCQ, and CQL baselines.","marker":"(Luo et al., 2024)"},{"why":"Supports the sufficient state representation and the use of interpretable models for sequential clinical policies.","marker":"(Matsson et al., 2024b)"},{"why":"Supports reasoning about which policies can be evaluated with statistical support and the use of prototype-based behavior models.","marker":"(Matsson and Johansson, 2022)"}],"fun_headline_variants":["Frequent-treatment policies beat offline RL in clinical care","Simple common-treatment policies outperform offline RL in trials","Interpretable care policies from frequent actions top offline RL","Using most common treatments yields better policies than RL","Pragmatic behavior-cloning policies edge out offline RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The value estimates are causal only if the observed state captures every variable that affects both treatment choice and outcome; the authors explicitly note that unmeasured confounders may remain, and any such confounder would bias the estimated advantage of the proposed policies, especially the outcome-guided ones.","fun_headline_variants_meta":{"raw":{"variants":["Frequent-treatment policies beat offline RL in clinical care","Simple common-treatment policies outperform offline RL in trials","Interpretable care policies from frequent actions top offline RL","Using most common treatments yields better policies than RL","Pragmatic behavior-cloning policies edge out offline RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1345,"prompt_tokens":922,"completion_tokens":423,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":348}},"tokens_in":538,"tokens_out":423,"duration_ms":4951,"temperature":1.0,"reasoning_tokens":348,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:56:51.183347+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check is to rerun the WIS evaluation with the reward replaced by a pre-treatment negative-control outcome that treatment cannot influence; if the MC or MC+O policies then appear to have nonzero value differences, the OPE pipeline is biased by confounding or other misspecification. A more decisive test would compare the k=1 majority-treatment policy against usual care in a randomized trial and see whether the trial confidence interval contains the paper's OPE estimate.","supporting_citations":[],"review_version":1}