{"id":"cc22fc95-8759-4816-b229-7ea22857c045","arxiv_id":"2505.08378","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A split-sample conformal method certifies, with probability at least 1-alpha, that a learned treatment policy's treatment risk stays below tau under bounded miscalibration of assignment or selection odds.","lead":"This paper proposes a method for learning treatment allocation policies that keep the risk of harm among treated patients below a chosen threshold, with a finite-sample guarantee, even when unmeasured factors make the true risk non-identifiable. It combines sensitivity bounds on confounding or selection with split-sample conformal risk control, and demonstrates the approach on simulated and real clinical trial data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4.3's proof conflates the in-sample constraint with the out-of-sample risk; the finite-sample guarantee is only shown for risk averaged over the training split, not for the returned policy.","rationale":"The reader's verdict is CONDITIONAL and identifies two issues: the equality assumption in Theorem 4.3 is not guaranteed by the algorithm, and the RCT denominator in Remark 4.2 is not identified under the paper's own selection model. I agree with both, but the more fundamental problem is in the proof of Theorem 4.3 itself, specifically Eq. (18). Even if the RCT denominator were fixed and an equality assumption were enforced, the theorem as written would still prove control of the marginal, D_m-averaged treatment-risk bound rather than the conditional risk of the returned policy. The calibration step (13) can be made valid for the conditional risk by split conformality, but then step (18) must hold conditional on D_m, and it does not. The equality assumption in Theorem 4.3 concerns the empirical constraint in (12), not the conditional risk, so it does not close the gap. This is an internal inconsistency, not a disagreement with consensus. The miscalibration-bounded importance weights and the split-conformal calibration idea are genuinely promising, and the experiments suggest the method can work in the tested regimes; the problem is that the central certification proof is missing an argument that controls the in-sample selection bias of the solution to (12). The RCT denominator issue in Remark 4.2 is also real and should be corrected, but it affects only the trial-data branch, whereas the Eq. (18) issue affects the observational branch too. The verdict should remain CONDITIONAL: a revision that replaces Eq. (18) with a valid conditional or uniform bound, plus a corrected RCT denominator, would be required before the stated finite-sample certification is established.","tokens_in":15652,"tokens_out":20738,"duration_ms":214117,"concrete_test":"Re-derive Eq. (18) with conditioning on D_m made explicit. For pi(X;t) from (12), the constraint gives (1/m) sum_i V_i(t) <= t, and since each V_i is an in-sample loss of the fitted policy, E[V_i(t)|D_m] = V_i(t); the equality (1/m) sum_i E[V_i(t)] = E[V(t)|D_m] is therefore false. If the expectation in (18) is instead taken over D_m, then T(t) is the D_m-averaged risk and Theorem 4.3 no longer establishes the stated guarantee (8) for the risk of the realized policy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central certification in Theorem 4.3 is not established for the risk of the policy actually returned. In Eq. (18), the authors infer from the empirical constraint bE_m[V(t)] <= t in (12) that t >= (1/m) sum_i E[V_i(t)] = E[V(t)] = T(t). The equality E[V_i(t)] = E[V(t)] is a marginal exchangeability statement: it averages over the training split D_m. But the quantity T(t) that the calibration set D_n bounds in (13) must be the risk of the policy pi(X;t) conditional on D_m; otherwise (13) is not a bound on the risk of the realized policy and (8) is not a statement about the returned pi. Conditional on D_m, the in-sample constraint gives only (1/m) sum_i V_i(t) <= t; it gives no bound on E[V(t)|D_m]. Thus the chain 'T(t_n)>tau implies t_n>tau' is invalid for the conditional risk, and the event in (16) is not the event whose probability (8) controls. The theorem proves a weaker statement: with probability at least 1-alpha, the risk of the selected policy averaged over D_m is below tau. The equality assumption on pi(X;tau) does not repair this, because it is an equality of the empirical constraint in (12), not an equality of the conditional risk.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method for learning treatment-allocation policies that minimize population risk subject to a constraint on the treatment risk, defined as the probability of a non-beneficial outcome among treated patients. The method is designed for settings in which the treatment risk is not point-identifiable, either because of unmeasured confounding in observational data or because of unmeasured selection into a randomized trial. The authors model the degree of miscalibration of the propensity or selection odds by a parameter Gamma, derive upper bounds on the population and treatment risks (Lemma 4.1), and then use sample splitting: the first half of the data, D_m, is used to learn a family of policies indexed by a nominal tolerance t; the second half, D_n, is used to construct an upper confidence bound on the treatment risk of each policy and to select an empirical tolerance t_n (Algorithm 1, Theorem 4.3). The paper also reports simulation experiments and applications to the STAR and International Stroke Trial datasets.","tokens_in":15963,"tokens_out":14494,"duration_ms":147683,"significance":"If the central finite-sample guarantee were established, the paper would make a useful contribution: it extends distribution-free risk-control ideas from prediction sets to policy learning under partial identifiability, while keeping the policy class interpretable. The miscalibration model via odds-ratio bounds is clearly stated and the empirical evaluation is extensive, including both confounding and selection-bias settings. The authors also correctly identify that the treatment risk is not point-identifiable in either data regime and that a certification statement must account for this. However, the main theoretical result, Theorem 4.3, has a proof gap that is load-bearing, and the randomized-trial extension has an identifiability problem. These issues affect the paper's central claim, so the contribution is currently not established.","major_comments":[{"comment":"The proof of Theorem 4.3 relies on the inequality t ≥ T(t), derived in Eq. (18) by taking expectations of the empirical constraint in (12). This step is valid only if T(t) is defined as the marginal expectation E[V(t)] averaged over the training split D_m. But the guarantee (8) is about the treatment risk of the realized policy conditional on the data, i.e., E[V(t)|D_m]. For the conditional risk, the in-sample constraint bE_m[V(t)] ≤ t gives no bound; conditional on D_m, the sample mean can be below t while the population mean is above t. Consequently the chain 'T(t_n) > τ implies t_n ≥ T(t_n) > τ' is invalid. If T(t) is instead read as the marginal expectation, then (13) and the final event control the risk averaged over D_m, not the risk of the returned policy, which is a strictly weaker statement than (8). The equality assumption on π(X;τ) does not repair this, because it is an equality of the empirical constraint, not of the conditional risk.","section":"Theorem 4.3, Eq. (18)"},{"comment":"For randomized trial data, the denominator pπ(A=1|S=0) in Lemma 4.1 and in Eq. (11) is not identifiable from trial data under the paper's own model (4), because the trial covariate distribution p(x|S=1) need not equal the target distribution p(x|S=0). Remark 4.2 writes pπ(A=1|S=0) = ∫1(π(x)=1)p(x|S=s)dx; when s=1, the right-hand side is the treatment probability under the trial covariate distribution, not the target-population quantity. The weight W^Γ in (11) also contains p(S=1)/p(S=0), which is not specified or bounded. As a result, the upper bound (9) for the RCT case is not computable from the available data, and the STAR and IST experiments in Section 5.2 and Appendices A.4–A.5 do not have a supported finite-sample guarantee unless an additional assumption equating target and trial covariate distributions is introduced.","section":"Remark 4.2 and Lemma 4.1 (RCT case)"},{"comment":"Theorem 4.3 is conditional on the nominal policy π(X;τ) achieving the constraint in (12) with equality. Since (12) is an inequality constraint, equality is a data-dependent event whose probability is neither controlled nor guaranteed by Algorithm 1. If the constraint is inactive, the proof's step T(τ)=τ fails, and the theorem gives no certificate. The authors should either justify that equality can be enforced by construction (e.g., by choosing τ on a grid where the constraint binds) or provide a proof that does not require this condition.","section":"Theorem 4.3, equality assumption"}],"minor_comments":[{"comment":"There are typos in the text describing the DAGs: 'were' should be 'where' and 'unbserved' should be 'unobserved'.","section":"Section 2"},{"comment":"Step 2 loops over t in the continuum (0,1); the practical implementation uses a finite grid, but this discretization and its effect on the guarantee should be stated explicitly.","section":"Algorithm 1"},{"comment":"The definition of t_n as an arg min subject to a constraint involving all t' ≤ t presumes a well-defined feasible set; since T_n^α(t) need not be monotone, the authors should discuss existence, uniqueness, and computation of t_n.","section":"Eq. (14)"},{"comment":"The proof concludes that E[V_{n+1}(t_n)|E=1] ≤ τ and says this holds with probability at least 1−α; the probabilistic statement should be made precise, since E[V_{n+1}|E=1] is a conditional expectation over the data, not a random variable with an obvious coverage interpretation.","section":"Appendix B, Theorem B.1"},{"comment":"The Bentkus bound is described for binary losses, but the paper does not discuss whether the bound in (13) needs to hold simultaneously over t for the proof of Theorem 4.3; a clarifying sentence would help.","section":"Remark 4.4"}],"recommendation":"reject","confidential_remarks":"The main theorem's proof is not a minor gap: the inequality t ≥ T(t) is used pathwise but is only available in expectation over the training split, so the finite-sample guarantee for the returned policy is not established. The RCT extension also has an identifiability problem in the denominator pπ(A=1|S=0). These are load-bearing issues that would require a substantially revised theorem and a re-scoped claim; I cannot recommend publication in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read Ek & Zachariah's paper on certifiable treatment allocations under partial identifiability. The short version: the problem is real and the combination of miscalibration-bounded weights with split-conformal risk control is a natural and promising idea, but as written the central certification theorem doesn't go through, and the RCT case has an identification error.\n\nWhat's new: they extend conformal risk control to policy learning with partially identified treatment risk, using Tan-style miscalibration bounds to get valid upper bounds on the risk, then select a policy threshold t_n on a separate split. The experiments with fast-and-frugal trees on synthetic and STAR/IST data are clear and honest. The appendix on benchmarking Gamma is practical and useful.\n\nThe soft spots are serious, not minor. First, Theorem 4.3's proof. In (18) they infer from the in-sample constraint bEm[V(t)] <= t that t >= (1/m) sum E[V_i(t)] = E[V(t)] = T(t). That equality is at best an unconditional expectation over the training split; for the realized D_m, the empirical constraint gives no bound on the conditional risk T(t) of the learned policy. The chain t_n >= T(t_n) > tau is therefore unjustified, and the event whose probability (8) controls is not the event they bound. The assumption that pi(X;tau) is constraint-active doesn't repair this, because the equality is about the empirical constraint, not the conditional risk. The theorem proves a weaker statement about risk averaged over D_m.\n\nSecond, Remark 4.2. For RCT data, p_pi(A=1|S=0) = integral 1(pi(x)=1) p(x|S=0) dx, and that is not identified from trial data under their own selection model (4) where U affects S. They seem to slip in p(x|S=s) with the target population, but for s=1 that's the trial covariate distribution. The denominator of the treatment-risk bound is therefore not identifiable from the trial, and the certified guarantee for the RCT case is unsupported. They also need p(S=1)/p(S=0) in the weights, which is another unknown.\n\nOtherwise the observational part is mostly coherent if you're willing to condition on a fixed D_m in (13); the split-conformal ingredient is standard. But these two issues are load-bearing. The paper deserves a serious referee because the approach is worth fixing; I'd send it out, but any referee should insist on a corrected proof and a clear treatment of the RCT target-population quantities.\n\nFor my own work, I wouldn't cite the guarantee as a theorem until the proof is fixed.\n\nRecommendation: engage with it as promising work in progress; ask for major revision.","headline":"Promising combination of sensitivity weights and conformal risk control, but Theorem 4.3's proof has a load-bearing in-sample/out-of-sample gap and the RCT setting has an identification error.","tokens_in":16469,"tokens_out":5978,"would_cite":false,"duration_ms":58906,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A policy-learning method certifies, with finite samples, that treatment risk stays below a chosen tolerance even when the risk is only partially identified.","keywords":["treatment allocation","risk control","partial identifiability","finite-sample guarantees","miscalibration bounds","policy learning","unmeasured confounding","randomized trials"],"falsifier":"Simulate a randomized trial in which an unobserved variable shifts both trial selection and outcome, so the trial covariate distribution differs from the target population. Run the method with $\\Gamma=1$, evaluate the true treatment risk $T(\\pi)$ on the target distribution over many datasets, and check whether fewer than $1-\\alpha$ of the runs satisfy $T(\\pi)\\le\\tau$; if not, the RCT certification in Theorem 4.3 fails.","tokens_in":15386,"feed_emoji":"🩺","tokens_out":8356,"duration_ms":77502,"temperature":0.7,"pith_summary":"The paper proposes a way to learn treatment-allocation policies that keep the \"treatment risk\"—the proportion of treated patients who suffer a non-beneficial outcome rather than benefit—below a chosen tolerance, with a finite-sample probabilistic guarantee. The setting is deliberately hard: the risk need not be point-identifiable, because observational data may have unmeasured confounders and randomized trials may recruit a population different from the target. The method assumes only a user-specified bound on how miscalibrated the nominal treatment-assignment or trial-selection odds can be, and then certifies the policy against all degrees of miscalibration up to that bound. If the paper is right, a clinician can be told that, with confidence at least $1-\\alpha$, no more than a fraction $\\tau$ of treated patients will experience the non-beneficial outcome, even though the exact risk is unknown.","feed_headline":"Certified treatment-risk control works without point identification","feed_subtitle":"New policies cap the share of treated patients who do not benefit, with a confidence guarantee, even when risks are only partly known.","key_machinery":"The load-bearing object is the miscalibration weight $W^\\Gamma$. It is built from the nominal assignment odds (observational case) or selection odds (trial case) together with the assumed miscalibration factor $\\Gamma$, and Lemma 4.1 shows that multiplying the observed loss $L$ by $W^\\Gamma$ gives an upper bound on the unidentifiable population and treatment risks. The method then learns a policy from one split of the data by minimizing an empirical estimate of the population-risk bound subject to a nominal treatment-risk constraint, and uses a second split to compute a finite-sample upper confidence bound on the treatment risk (for instance via the Bentkus bound), choosing the tightest tolerance $t_n$ that still satisfies $\\tau>T^\\alpha_n(t')$ for all smaller tolerances. The proof of Theorem 4.3 runs the confidence-bound event and the equality assumption for $\\pi(X;\\tau)$ together to show that violation of the target risk has probability at most $\\alpha$.","core_discovery":"The central claim is Theorem 4.3: if the nominal policy $\\pi(X;\\tau)$ obtained from the empirical constrained problem (12) satisfies its treatment-risk constraint with equality, then the policy $\\pi(X;t_n)$ selected by the empirical tolerance (14) satisfies $P(T(\\pi)\\le \\tau \\mid S=s)\\ge 1-\\alpha$ for any degree of miscalibration up to a prespecified $\\Gamma$. Here $T(\\pi)=P_\\pi(L=1\\mid A=1,S=0)$ is the treatment risk, and $S=s$ is the sampling condition (observational data for $s=0$ or randomized trial data for $s=1$). The argument converts an unidentifiable risk into an upper bound via importance weights $W^\\Gamma$, then controls that upper bound in finite samples. The experiments illustrate the resulting trade-off between lowering population risk and keeping treatment risk under $\\tau$, on simulated data and on the STAR and IST trials.","pith_inferences":["Beyond the paper: for randomized trial data, the certification inherits the assumption that the trial covariate distribution matches the target population, because the policy's treatment-probability denominator $p_\\pi(A=1\\mid S=0)$ is identified from trial data only when $p(x\\mid S=1)=p(x\\mid S=0)$; if selection into the trial shifts the covariate mix, the claimed RCT guarantee is not supported by","Beyond the paper: the finite-sample guarantee is conditional on $\\Gamma$ actually bounding the unknown odds; the paper benchmarks $\\Gamma$ by omitting covariates, but if the true unmeasured factor has a larger effect than those benchmarks, the guarantee has no force.","Beyond the paper: the same split-sample risk-control logic could be applied subgroup-wise; the paper notes that aggregate control may be insufficient for fairness, so stratifying by sensitive covariates is a natural next step.","Beyond the paper: the Bentkus bound is tight for binary losses, but for non-binary or high-variance losses other confidence bounds could tighten the method, as Remark 4.4 hints."],"forward_implications":["With finite sample sizes, the learned policy is certified to keep treatment risk below $\\tau$ with probability at least $1-\\alpha$, without assuming point identification.","The user trades off population benefit against non-maleficence: smaller $\\tau$ treats fewer patients and raises population risk, as shown in the synthetic and STAR results.","The guarantee holds simultaneously for all miscalibration levels up to $\\Gamma$, so a defensible $\\Gamma$ (for example, benchmarked by omitting covariates) makes the policy robust to unmeasured confounding or selection.","The same procedure handles observational data, randomized trial data, and the mixed case where an observational study is conducted on a study population that differs from the target.","Applying the method to STAR and IST data yields simple fast-and-frugal decision-tree policies whose estimated treatment risk stays below $\\tau$ across random splits."],"supporting_citations":[{"why":"Supplies the split-sample finite-sample confidence-bound technique (including the Bentkus bound) and the proof strategy used in Theorem 4.3.","marker":"[Bates et al., 2021]"},{"why":"Provides the odds-ratio miscalibration model in condition (6) that bounds the unknown treatment-assignment odds.","marker":"[Tan, 2006]"},{"why":"Frames partial identification of treatment effects and motivates benchmarking a range of plausible miscalibration.","marker":"[Manski, 2007]"},{"why":"Establishes that randomized trial and target populations can differ through selection, the trial-data challenge handled by condition (7).","marker":"[Westreich, 2019]"},{"why":"Supplies the counterfactual 'fraction negatively affected' harm bound that the paper contrasts with its observable treatment-risk guarantee.","marker":"[Kallus, 2022]"},{"why":"Formulates benefit-risk constrained policy learning, the problem setting the paper extends to certified finite-sample control.","marker":"[Wang et al., 2018]"},{"why":"Provides the omitted-covariate benchmarking procedure used to choose a credible $\\Gamma$ in experiments.","marker":"[Huang et al., 2021]"}],"fun_headline_variants":["Certified risk control without point identification","Guaranteed treatment risk caps from partial data","Finite-sample guarantee for risk-constrained policies","Learning safe allocations with partially known risks","Partial identifiability no barrier to certified risk control"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central load-bearing premise is that the nominal odds are miscalibrated by at most the chosen $\\Gamma$ and that, in the trial case, the trial covariate distribution matches the target population; if either fails, the certification is unsupported.","fun_headline_variants_meta":{"raw":{"variants":["Certified risk control without point identification","Guaranteed treatment risk caps from partial data","Finite-sample guarantee for risk-constrained policies","Learning safe allocations with partially known risks","Partial identifiability no barrier to certified risk control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001239,"raw_usage":{"total_tokens":5034,"prompt_tokens":840,"completion_tokens":4194,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":456,"completion_tokens_details":{"reasoning_tokens":4126}},"tokens_in":456,"tokens_out":4194,"duration_ms":31448,"temperature":1.0,"reasoning_tokens":4126,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:57:19.244427+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate a randomized trial in which an unobserved variable shifts both trial selection and outcome, so the trial covariate distribution differs from the target population. Run the method with $\\Gamma=1$, evaluate the true treatment risk $T(\\pi)$ on the target distribution over many datasets, and check whether fewer than $1-\\alpha$ of the runs satisfy $T(\\pi)\\le\\tau$; if not, the RCT certification in Theorem 4.3 fails.","supporting_citations":[{"cited_title":"Distribution-free, risk-controlling prediction sets","cited_arxiv_id":null,"evidence_quote":"Supplies the split-sample finite-sample confidence-bound technique (including the Bentkus bound) and the proof strategy used in Theorem 4.3."},{"cited_title":"Identification for prediction and decision","cited_arxiv_id":null,"evidence_quote":"Frames partial identification of treatment effects and motivates benchmarking a range of plausible miscalibration."},{"cited_title":"Epidemiology by Design: A Causal Approach to the Health Sciences","cited_arxiv_id":null,"evidence_quote":"Establishes that randomized trial and target populations can differ through selection, the trial-data challenge handled by condition (7)."},{"cited_title":"What's the harm? sharp bounds on the fraction negatively affected by treatment","cited_arxiv_id":null,"evidence_quote":"Supplies the counterfactual 'fraction negatively affected' harm bound that the paper contrasts with its observable treatment-risk guarantee."},{"cited_title":"Off-policy risk assessment in contextual bandits","cited_arxiv_id":null,"evidence_quote":"Provides the omitted-covariate benchmarking procedure used to choose a credible $\\Gamma$ in experiments."}],"review_version":1}