{"id":"5ad739b9-2949-48c2-bf87-faa1b072f067","arxiv_id":"2502.09467","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Under monotonicity of outcomes in model performance, a new ML model's causal impact can be tightly bounded using only prior RCT data, with worst-case bounds that cannot be improved without more assumptions.","lead":"This paper shows how data from one randomized trial of an ML model can be used to bound the effect of deploying an updated, untrialed model, under assumptions about user trust and model performance. It gives tight bounds, tests for the assumptions, and design advice, potentially saving the cost of rerunning trials for every model update.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Assumption 3.1 is load-bearing; the proposed falsification tests only an aggregate agreement-region condition, so local monotonicity violations can pass the test and still break the Theorem 3.1 bounds.","rationale":"The reader's weakest assumption is also the one I regard as most load-bearing. The cluster-randomization and interference concerns raised elsewhere are real but secondary: they affect the variance estimator and the breadth of the DGP, whereas a failure of Assumption 3.1 breaks the identification argument itself. I agree with the reader's identification of the problem, and the proposed counterexample sharpens it: Proposition 3.1 is an aggregate test, so local violations that cancel can evade it while still overturning the bound. This does not contradict Theorem 3.1, which is valid conditionally on the assumption; it shows that the paper's framing that the assumption can be checked with multi-model RCT data is too strong. The bounds remain useful when monotonicity is independently credible, but a revised paper should recommend stratified or subgroup falsification where feasible and plainly state that one-sided agreement regions are assumption-dependent. Because the reader's CONDITIONAL verdict already reflects the need for such caveats, I recommend no change.","tokens_in":32450,"tokens_out":16974,"duration_ms":162356,"concrete_test":"Run the following simulation with the paper's framework. Let X have two equally likely values x1,x2; let f_M(πL)=0, f_M(πe)=0.5, f_M(πH)=1; all three policies alert on both x values, so X_agree is the whole space. Set Y∈[0,1] with conditional means: on x1, E[Y|x1,πL]=0.9, E[Y|x1,πH]=0.4, E[Y(A=alert,M=0.5)|x1]=0.1; on x2, E[Y|x2,πL]=0.1, E[Y|x2,πH]=0.9, E[Y(A=alert,M=0.5)|x2]=0.5. Then Proposition 3.1 gives E[Y|X_agree,πH]=0.65 ≥ E[Y|X_agree,πL]=0.5, so the falsification test does not reject. But Theorem 3.1 gives L=(0.9+0.1)/2=0.5 while the true policy value is (0.1+0.5)/2=0.3, violating Eq. (3). If this simulation yields L>true, the paper's falsification strategy is insufficient to protect the central bound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 3.1's lower and upper bounds are obtained through Lemma D.2 inequalities (22)-(23), which replace E[Y|X=x,A=πe,M=f_M(πe)] with the conditional mean under a next-worst or next-best agreeing trial model. The direction of the inequality is supplied entirely by Assumption 3.1, the pointwise monotonicity of Y(A=a,M=m) in m for fixed a. If a higher-accuracy model changes user trust or behavior even when the output is unchanged, this ordering can invert and the claimed bound fails. The paper's own check, Proposition 3.1, does not secure the assumption: it tests only an average over X_agree, comparing E[Y|X∈X_agree,Π=π_high] with E[Y|X∈X_agree,Π=π_low]. That is a necessary condition, not sufficient. A violation on one subset of X_agree can be offset by larger opposite differences elsewhere, so the test can pass while monotonicity is violated exactly where Lemma D.2 applies. Moreover, on regions where only one side of the performance ordering has an agreeing trial model (Eqs. (10)-(11) of Proposition 3.3), no falsification data exist, yet those regions contribute conditional means or Ymin/Ymax to the bounds. Thus the central claim's practical guarantee is not established by the tests the paper offers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a method for evaluating the causal impact of deploying a new machine-learning model πe using only data from a prior randomized controlled trial that trialed other models. The authors formalize a structural causal model in which model outputs are deterministic functions of covariates, model performance M is a scalar property of the deployed model, and outcomes depend on both the output and M. Under assumptions of performance monotonicity (Assumption 3.1), existence of a neutral action whose outcome is invariant to M (Assumption 3.2), and bounded outcomes (Assumption 3.3), they derive upper and lower bounds on the policy value (Theorem 3.1), prove these bounds are tight (Theorem 3.2), provide an estimator with asymptotically valid confidence intervals (Proposition 3.4), and propose falsification tests for the first two assumptions (Propositions 3.1 and 3.2). A simulation study illustrates that the most accurate model need not have the largest causal impact.","tokens_in":32682,"tokens_out":20089,"duration_ms":171001,"significance":"If the results are correct, this is a useful contribution to the emerging literature on evaluating ML decision-support systems after deployment. The formal bounds are derived cleanly, and the tightness construction in Theorem 3.2 is a notable strength: it shows precisely where unobserved (x,a,m) combinations force the use of Ymin and Ymax. The paper also ships reproducible code for the simulation and gives concrete trial-design recommendations. The main caveat is that the practical applicability hinges on Assumption 3.1 (performance monotonicity), and the proposed falsification tests provide only weak, aggregate necessary conditions for it; this limits the strength of the claim that the assumptions 'can be checked' from the RCT data.","major_comments":[{"comment":"In the Related literature paragraph, the paper states that 'we allow for the fact that actions taken for one patient can influence outcomes for other patients.' This is not supported by the model. In Assumption 2.1 the outcome is generated by Y=f_Y(A,X,M,ϵ_Y), and the DAG in Fig. 2 contains no edges between outcomes or actions of different patients; D enters only through Π. All results in Section 3 therefore assume no interference (SUTVA). The sentence should be removed or the model and proofs should be extended to include interference; as written, the scope claim contradicts the formal setup.","section":"Section 1, Related literature"},{"comment":"Assumption 3.1 is the central ordering assumption used in Lemma D.2 (inequalities (22)–(23)), and therefore in every step of Theorem 3.1. The falsification test in Proposition 3.1 only compares the aggregate means E[Y|X∈X_agree,Π=π2] and E[Y|X∈X_agree,Π=π1]. This is a necessary condition: if the test rejects, Assumption 3.1 is false. However, a violation of monotonicity on one subset of X_agree can be offset by opposite differences on other subsets, so the test can pass even when the pointwise inequality required by Lemma D.2 fails. Moreover, in the regions described by equations (10)–(11) of Proposition 3.3, there is no agreeing model on one side of the performance ordering, so no falsification data exist at all, yet those regions enter the bounds through Ymin or Ymax. The authors should either develop stronger (e.g., conditional-on-subset) tests or revise the claim in the introduction that 'both of our main assumptions can be checked using RCT data'; as stated, the testable implications do not secure the key assumption on the regions where the bounds are most fragile.","section":"Assumption 3.1 / Proposition 3.1"}],"minor_comments":[{"comment":"In the proof after Eq. (58), the statement that f_L(x,a,m_e)=F^{-1}_{Y|x,a,hprev(m_e)} for all x with \\tildeΠ^e_≤(X) non-empty is incomplete: when m_e belongs to the observed performance set M_a(x), the fourth line of Eq. (56) applies instead of the hprev case. The final equality (62) still holds, but the proof should split into the cases m_e∈M_a(x) and m_e∉M_a(x).","section":"Appendix D, proof of Theorem 3.2"},{"comment":"There is a typo: 'related tooff-policy policy evaluation' should read 'related to off-policy policy evaluation.'","section":"Section 1, Related literature"},{"comment":"The notation '1{π_e ≠ a0}' uses the policy π_e as shorthand for π_e(X); although this is acknowledged in footnote 3, the displayed formulas would be clearer if the covariate argument were written explicitly as 1{π_e(X) ≠ a0}.","section":"Theorem 3.1 and Proposition 3.3"},{"comment":"The piecewise definitions of ψ_L and ψ_U mix indicator notation in the first and third cases with bare Ymin/Ymax in the second and fourth cases; making all cases uniform would improve readability.","section":"Proposition 3.4"},{"comment":"The claim that the lower bound for patient outcomes under πe0 exceeds outcomes under all trialed models is visible in Fig. 5, but the text would benefit from reporting the estimated bound values and confidence intervals explicitly.","section":"Section 5, simulation study"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid borderline submission: the main theorem is sound and the tightness proof is a genuine contribution. The two major issues—the unsupported interference claim and the mismatch between the strength of the falsification tests and the claims made for them—are fixable without changing the core results. I would be willing to re-review a revised version. The paper's fit for the journal is acceptable if the authors clarify the scope and limitations of the assumptions; the core result is publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dave,\n\nThis paper is a genuine advance on a practical problem: evaluating the causal effect of deploying an untrialed ML model using data from a prior RCT. The core theoretical result, Theorem 3.1, derives tight bounds on the policy value under explicit assumptions, and Theorem 3.2 gives a constructive proof of tightness. The proofs check out, the estimator in Proposition 3.4 is simple, and the simulation is well-specified with public code. The idea of using next-best/worst agreeing trial models to bound counterfactual outcomes under performance monotonicity is clean and, in this combination, new. The falsification tests are useful as a first pass.\n\nThe main soft spot is exactly what the stress-test note flags. Assumption 3.1 (performance monotonicity) is load-bearing: the direction of the inequalities in Lemma D.2, and hence the entire bound, comes from a pointwise monotonicity of Y in M for fixed A. Proposition 3.1 tests only E[Y|X∈X_agree,Π=π_high] ≥ E[Y|X∈X_agree,Π=π_low] — an aggregate average over the agreement region. Local violations can cancel, so the test can pass while monotonicity fails on the very subsets where Lemma D.2 applies. The paper is careful to call these 'falsification' tests in the theorem statements, but the introduction says the assumptions 'can be checked' using RCT data. That is an overstatement; they are necessary conditions, not sufficient checks. The authors should say so explicitly and, where possible, suggest conditional or stratified tests. This does not sink the theorem, but it changes the practical guarantee.\n\nTwo smaller issues. First, the Related Literature section claims the framework allows actions for one patient to influence outcomes for other patients, but Assumption 2.1 has no interference terms. Either add a model for interference or drop the claim. Second, Proposition 3.4 uses an iid central limit theorem, while the paper's motivating examples are cluster RCTs. Outcomes within a cluster are likely correlated; the confidence intervals need clustered standard errors or an explicit independence assumption. The abstract's 'precisely bounded' also overstates how often the bounds are informative — Proposition 3.3 shows many new models will land on the Ymin/Ymax regions. The conclusion acknowledges this, so it is a framing fix, not a technical error.\n\nWho gets value from this? Statisticians and ML researchers who have RCT data on an older model and want a defensible answer about an update. It deserves a serious referee. I would send it out and ask for revisions along the lines above; the core result is sound and worth publishing.\n\nBest","headline":"A clean partial-identification result for evaluating untrialed ML models from prior RCT data, with a load-bearing monotonicity assumption that its falsification test does not actually secure.","tokens_in":33218,"tokens_out":4025,"would_cite":true,"duration_ms":34744,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62D20"],"pacs":[],"model":"deepseek-v4-flash","headline":"Given data from a single prior randomized trial, the causal impact of deploying any new ML model—including models never trialed—can be tightly bounded under three stated assumptions, and the bounds cannot be improved without further…","keywords":["causal inference","policy value bounds","randomized controlled trial","model updates","decision support","off-policy evaluation","performance monotonicity","tight bounds"],"falsifier":"Run the paper's Proposition 3.1 test on actual trial data: identify a subpopulation where two trialed models with different measured performance produce the same output, and compare mean outcomes; if the higher-performing model's mean outcome is significantly lower, Assumption 3.1 is false. The analogous Proposition 3.2 test, comparing outcomes where both models output the neutral action, would falsify Assumption 3.2 if the means differ.","tokens_in":32216,"feed_emoji":"📊","tokens_out":6120,"duration_ms":54574,"temperature":0.7,"pith_summary":"The paper asks whether a single randomized trial of one or a few ML decision-support models can also tell you the causal effect of deploying a different, updated model that was never in the trial. Its answer is yes, up to bounds: under three assumptions—outcomes are monotone in model performance, there is a neutral output whose effect does not depend on performance, and outcomes are bounded—the policy value of any new model is sandwiched between a lower bound using the worst agreeing trialed model and an upper bound using the best agreeing trialed model, with $Y_{\\min}$ and $Y_{\\max}$ where no trialed model agrees. The bounds are provably tight, so no additional assumption-free tightening is possible, and simple weighted-mean estimators give asymptotically valid confidence intervals. The practical point is that expensive repeated trials may be unnecessary for validating model updates, and that trial designs with multiple models are worth running because they make future updates evaluable.","feed_headline":"One RCT can bound the impact of AI model updates.","feed_subtitle":"Proven-tight bounds let clinics assess updated alerting models from old trial data alone.","key_machinery":"The load-bearing object is the family of agreement sets built from the trial's models: for each $x$, $\\Pi^e(x)$ collects trialed policies whose output matches $\\pi_e(x)$, and $\\Pi^e_{\\le}(x)$, $\\Pi^e_{\\ge}(x)$ restrict to those with performance at most or at least that of $\\pi_e$; the extremal sets $\\widetilde{\\Pi}^e_{\\le}(x)$ and $\\widetilde{\\Pi}^e_{\\ge}(x)$ pick the next-worst and next-best performing agreeing models. The bounds replace the unobserved outcome under $\\pi_e$ with observed outcomes under these extremal agreeing models, relying on Performance Monotonicity (Assumption 3.1) to order outcomes by model performance, Neutral Actions (Assumption 3.2) to reuse control-arm data, and Bounded Outcomes (Assumption 3.3) to cap the residual uncertainty where no trialed model agrees.","core_discovery":"On the paper's own terms, the central discovery is Theorem 3.1: under the structural model of Assumption 2.1 and Assumptions 3.1 to 3.3, the expected outcome under a new policy $\\pi_e$ satisfies $L(\\pi_e) \\le \\mathbb{E}[Y(A=\\pi_e, M=f_M(\\pi_e))] \\le U(\\pi_e)$, where for each covariate value $x$ the lower bound uses outcome data from trialed models that agree with $\\pi_e(x)$ and have next-worst performance (or $Y_{\\min}$ if none exists), and the upper bound uses next-best performance (or $Y_{\\max}$). Theorem 3.2 shows these bounds are tight in that two structural causal models matching the same observed data can realize $L$ and $U$ respectively, so no sharper bounds follow from the assumptions alone. Proposition 3.4 turns the bounds into inverse-probability-weighted sample means with normal confidence intervals, and Propositions 3.1 and 3.2 give observable checks that can falsify the two behavioral assumptions from the trial data itself.","pith_inferences":["If the same bounds were applied to observational deployment data rather than RCT data, the independence of policy assignment and covariates would fail, so one would need propensity-style adjustment and the falsification tests would lose their clean interpretation; the RCT requirement is doing real work.","The bounds are pessimistic by design on never-agreeing regions, and a natural testable extension is to replace $Y_{\\min}$ and $Y_{\\max}$ with domain-specific constraints such as “alerts are never harmful when correct,” which would tighten the intervals without contradicting the tightness result.","Falsification via Proposition 3.1 only inspects agreement regions, so performance monotonicity remains untested precisely on the covariate regions where no trialed model matches $\\pi_e$; trial designers who want to validate future models should therefore maximize overlap between candidate models' action regions."],"forward_implications":["A practitioner with one completed multi-arm cluster RCT can report an interval for the causal effect of any proposed model update without running a new trial, as long as the three assumptions hold.","Trial designs that randomize several models with genuinely different outputs and a spread of performance levels produce narrower bounds for future updates, because more $x$-values have agreeing next-worst and next-best trialed models.","Exact point identification occurs only in the special case where the agreement and performance structure makes the bound gap zero, such as a trialed model with equal performance on every case, or only neutral outputs; otherwise the honest answer is an interval.","Choosing updates by raw accuracy can be wrong: the simulation shows a less accurate model can have a higher lower bound on patient outcomes than a more accurate rival, so outcome-based bounds, not performance metrics, should drive deployment decisions."],"supporting_citations":[{"why":"Supplies the single-world intervention graph framework used to formalize potential outcomes and justify conditional ignorability in the proofs.","marker":"[Richardson and Robins, 2013]"},{"why":"Supplies the potential-outcome calculus rule used to connect counterfactual outcome distributions to observed conditional distributions.","marker":"[Malinsky et al., 2019]"},{"why":"Provides the comparative causal-evaluation setting for AI-assisted decision-making that this work extends from observable predictions to downstream outcomes.","marker":"[Imai et al., 2023]"},{"why":"Formalizes transportability of causal effects from trial data to new settings, the broader problem this paper addresses for unseen interventions.","marker":"[Pearl and Bareinboim, 2011]"},{"why":"Gives generalizability and propensity-score methods for transporting trial results, contextualizing the inference-from-RCT goal.","marker":"[Stuart et al., 2011]"},{"why":"Positions the paper relative to off-policy evaluation in reinforcement learning, where deterministic policies and lack of coverage are key distinctions.","marker":"[Uehara et al., 2022]"},{"why":"Provides the central limit theorem basis for the asymptotically valid confidence intervals in Proposition 3.4.","marker":"[Vaart, 1998]"}],"fun_headline_variants":["One trial can bound future ML model impacts","Tight bounds for AI updates from a single RCT","One RCT suffices to bound new model causal effects","No new trial needed to validate AI model updates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"If a better-performing model ever makes clinicians less attentive to its outputs, so that for a fixed output a higher-performance model leads to worse outcomes than a lower-performance one, the monotonicity assumption fails and the direction of the bounds can flip.","fun_headline_variants_meta":{"raw":{"variants":["One trial can bound future ML model impacts","Tight bounds for AI updates from a single RCT","One RCT suffices to bound new model causal effects","No new trial needed to validate AI model updates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1333,"prompt_tokens":965,"completion_tokens":368,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":320}},"tokens_in":581,"tokens_out":368,"duration_ms":33227,"temperature":1.0,"reasoning_tokens":320,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T21:22:26.972173+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's Proposition 3.1 test on actual trial data: identify a subpopulation where two trialed models with different measured performance produce the same output, and compare mean outcomes; if the higher-performing model's mean outcome is significantly lower, Assumption 3.1 is false. The analogous Proposition 3.2 test, comparing outcomes where both models output the neutral action, would falsify Assumption 3.2 if the means differ.","supporting_citations":[{"cited_title":"Single world intervention graphs ( SWIG s): A unification of the counterfactual and graphical approaches to causality","cited_arxiv_id":null,"evidence_quote":"Supplies the single-world intervention graph framework used to formalize potential outcomes and justify conditional ignorability in the proofs."},{"cited_title":"A potential outcomes calculus for identifying conditional path-specific effects","cited_arxiv_id":null,"evidence_quote":"Supplies the potential-outcome calculus rule used to connect counterfactual outcome distributions to observed conditional distributions."},{"cited_title":"Experimental evaluation of algorithm-assisted human decision-making: A pplication to pretrial public safety assessment","cited_arxiv_id":null,"evidence_quote":"Provides the comparative causal-evaluation setting for AI-assisted decision-making that this work extends from observable predictions to downstream outcomes."},{"cited_title":"Transportability of causal and statistical relations: A formal approach","cited_arxiv_id":null,"evidence_quote":"Formalizes transportability of causal effects from trial data to new settings, the broader problem this paper addresses for unseen interventions."},{"cited_title":"The use of propensity scores to assess the generalizability of results from randomized trials","cited_arxiv_id":null,"evidence_quote":"Gives generalizability and propensity-score methods for transporting trial results, contextualizing the inference-from-RCT goal."}],"review_version":1}