{"id":"dfc25f06-576f-498e-9371-42d2d07ca51c","arxiv_id":"2501.07722","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A/B testing with ML: a randomization test whose statistic is the cross-validation error difference between models with and without treatment, with finite-sample validity and power theory.","lead":"This paper builds hypothesis tests for A/B experiments that use machine learning models to detect complex treatment effects while keeping the finite-sample guarantees of randomization tests. A statistic based on cross-validation error differences between models with and without treatment is shown to have valid p-values, and power analyses and simulations suggest it can detect effects that average-treatment-effect tests miss.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 2's power guarantee is conditional on Assumption 2.2, an unverifiable sign condition; Example 4 shows misspecified model classes can make Δ<0 and kill the exponential bound, so the central 'ML improves power' claim is not unconditional.","rationale":"I agree with the reader's weakest-assumption identification. The core validity theorem (Theorem 1) is standard and sound. Theorem 2 is conditional on Assumption 2, and among its conditions the sign condition (Condition 2) is the uniquely fragile one: boundedness and vanishing Rademacher complexity are standard, but Condition 2 mixes the unknown h with the model class and can fail for plausible misspecified classes, as the authors' own Example 4 shows. The tie-handling gap in the proof of Lemma 4 (the m_R=0 step) is real but patchable by replacing the event t(r)>T with t(r)≥T and using the fact U≤1. The grid-based heterogeneity claim in Section 4 is also overstated without a confidence-set correction, but it is peripheral to the main global-null power result. Proposition 2's relative-efficiency conclusion is not entailed by the proved lower-bound inequality, since a lower bound on the ratio that tends to zero is compatible with equal error rates; this weakens the paper's comparative claim but does not invalidate Theorem 2. Overall the main theorems are conditionally correct, and the issues are addressable, so the reader's CONDITIONAL verdict remains appropriate.","tokens_in":32759,"tokens_out":26549,"duration_ms":258586,"concrete_test":"Simulate the Example 4 data-generating process: F1={c z}, baseline mean E[Y(0)]=−x0 with x0 large (e.g., 10), h(x)=1, n=500, Bernoulli design π=0.5, R=1000 randomizations, α=0.05. Record the rejection rate of Procedure 1. If the rejection rate remains at or below 5% while the treatment has a strong constant effect, this confirms that when Assumption 2.2 fails the exponential bound of Theorem 2 does not apply, making the power claim genuinely conditional on an unverifiable sign condition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the central power result, Theorem 2 requires Δ in Eq. (7) to be positive under H_1^glob. The proof of Δ≥0 relies entirely on Condition 2 in Assumption 2: E[h(X)(f*_r(X,1)−f*_r(X,0))]≥0. This condition couples the unknown true CATE h(X) with the user-chosen model class F1 through the best in-class predictor f*_r under the randomized treatment distribution. It is not a boundedness or Rademacher-complexity condition; it is a directional alignment condition that cannot be checked from observed data because h is unidentified. Example 4 provides a simple, plausible misspecification (F1={c z} with a large baseline mean) where the condition fails and Δ<0, making the exponential type II error bound vacuous. Thus the paper's advertised benefit—flexible ML models yield exponentially smaller type II error—holds only under an assumption the practitioner cannot verify. Theorem 1's finite-sample validity is unaffected, but the power theorem's practical scope is narrower than the 'regardless of ML models' framing suggests.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ML-assisted randomization tests for A/B experiments. The main procedure (Procedure 1) uses as a test statistic the difference between k-fold cross-validation errors of two ML models, one including the treatment variable and one excluding it, and computes an FRT-style p-value by re-randomizing treatments. The paper claims finite-sample validity of this p-value under the global null regardless of the ML models used (Theorem 1), and an exponential type II error bound under an alternative when the ML class predicts well (Theorem 2). It also extends the framework to covariate balance, heterogeneity testing via p-value maximization, and spillover tests under network interference (Theorem 3). A power comparison with a residualized FRT (Proposition 1 and 2) and a sample-size procedure are presented, followed by simulations for constant, heterogeneous, and spillover effects.","tokens_in":33078,"tokens_out":5908,"duration_ms":62199,"significance":"If the results were fully established, the paper would be a useful contribution: Theorem 1 gives a simple, finite-sample valid way to combine black-box ML prediction with randomization inference, and the Delta-based power analysis in Eq. (7) is an appealing way to connect predictive power to testing power. The spillover test in Section 5 is also a clean and correct application of conditional randomization tests. However, the central power claim is more fragile than the abstract suggests: Theorem 2 depends on an unverifiable sign condition (Assumption 2, Condition 2), the proof assumes exact empirical risk minimization that random forest implementations do not satisfy, and Proposition 2's efficiency conclusion does not follow from the displayed inequality. The finite-sample validity results (Theorem 1, Theorem 3, Theorem 5) are correct and are the most solid part of the paper. With a careful reframing of the power results, the paper would be publishable; in its current form the gap between the advertised 'ML improves power' claim and what is actually proven is too large.","major_comments":[{"comment":"The assertion in Theorem 2 that Delta >= 0, and hence the exponential type II error bound, relies entirely on Condition 2: E[h(X)(f*_r(X,1)-f*_r(X,0))] >= 0. This condition couples the unidentified true CATE h(X) with the user-chosen model class F1, and it cannot be verified from observed data because h is not identified. Example 4 shows a simple misspecified class with a large baseline where the condition fails and Delta < 0, making the bound vacuous. The abstract and Section 1 advertise power improvements from flexible ML 'regardless of the choice of ML models', but the power theorem is really conditional on an unverifiable alignment condition. Theorem 1's validity is unaffected, but the power result should be stated and interpreted conditionally.","section":"Section 3, Assumption 2, Condition 2, and Example 4"},{"comment":"The proof of Theorem 2 assumes that in each cross-validation fold the fitted model is an exact empirical risk minimizer over a fixed class F: 'we assume that the fitted models minimize the empirical loss in each cross-validation step'. Random forests, which are the main ML method in the simulations (Sections 6.1-6.3 via caret/randomForest/ranger), do not perform exact ERM over any explicit function class. Consequently, Theorem 2 as proven does not cover the paper's own flagship implementation. The theorem should be stated for an idealized ERM oracle, with the simulations explicitly presented as heuristic evidence, or the implementation should be characterized by a function class and an approximate ERM guarantee.","section":"Appendix B, Additional regularity conditions"},{"comment":"The displayed result in Proposition 2 is a lower bound on lim inf (1/n) log( P(phi_ML=0) / P(phi_RES=0) ), and Eq. (10) states that the type II error ratio is at least exp(-n[I(Delta_ML)-I(Delta_RES)]). Because I is increasing and Delta_ML >= Delta_RES under Proposition 1, this lower bound tends to zero. A lower bound that tends to zero is vacuous and cannot imply that the ratio itself tends to zero. The conclusion in the paragraph after Eq. (10), that 'the type II error of Procedure 1 is exponentially smaller than that of the residualized method', requires an upper bound of the form P(phi_ML=0)/P(phi_RES=0) <= exp(-n[...]) or a direct large-deviation comparison of the two error probabilities. As written, the efficiency claim does not follow from Proposition 2.","section":"Section 3.1, Proposition 2 and Eq. (10)"}],"minor_comments":[{"comment":"The proof of Lemma 4 uses the condition (R+1)alpha > 1 (equivalently R > 1/alpha - 1) and assumes m_R = 0 with probability one, but these conditions are not stated in Theorem 2. They should be included in the theorem statement or the proof should be adjusted.","section":"Section 3, Theorem 2 / Appendix B, Lemma 4"},{"comment":"There is a typographical error: 'ANOV A' should be 'ANOVA'.","section":"Section 2.2"},{"comment":"The text says tau_S is varied over {0,0.1,...,1} and {0,0.2,...,2} in Setups (A) and (B), respectively, but the x-axis of Figure 3 is labeled from 0 to 5. The caption or the axis should be made consistent with the reported ranges.","section":"Section 6.3"},{"comment":"The sample size formula in Eq. (11) uses L and M0 estimated from the same sample and omits the factor R as a heuristic. This should be flagged more explicitly as a heuristic without finite-sample guarantee, since the authors themselves note the worst-case bound is conservative.","section":"Section 3.2"},{"comment":"The simulations use Gaussian errors, while Assumption 1 requires bounded outcomes with probability one. The numerical validation of Theorem 2 is therefore only approximate; this is acceptable if stated, but it should be acknowledged in the simulation section.","section":"Section 6.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is in scope for a statistics journal and the finite-sample validity results are solid. The main revision needed is an honest reframing of Theorem 2 and Proposition 2: the power claims are conditional on unverifiable alignment and exact-ERM conditions, and the efficiency comparison in Proposition 2 is not established by the current inequality. The authors should also consider adding code/data or a reproducibility statement for the simulations, since the simulation section reports substantial computational effort but no accompanying artifacts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core validity result, Theorem 1, is genuine and the paper is worth reading. The idea of using a CV-error-difference statistic inside a Fisherian randomization test is new, and it cleanly separates the ML model from the validity guarantee. The spillover extension in Theorem 3 is also sound. I don't think there's a circularity problem: the p-value validity does not depend on the model being correct, and the power analysis treats Delta as a population quantity rather than a fitted parameter.\n\nThe paper also does some things well. It relates the proposed statistic to Breiman's MDA and to Sobol indices, which is useful context. The power theorem, Theorem 2, is an interesting step: it gives an exponential type II error bound in terms of Delta, the difference in predictive risk between randomized and observed treatment assignments. That framing is genuinely novel for the randomization-test literature. The simulations back up the main qualitative claim that richer model classes yield higher power.\n\nNow the soft spots. The big one is Theorem 2's Condition 2 in Assumption 2: E[h(X)(f*_r(X,1)-f*_r(X,0))] >= 0. This is a sign-alignment condition coupling the unidentified true effect h with the user's chosen model class. It is not verifiable from data, and Example 4 shows a plausible misspecification where it fails and Delta < 0, making the exponential bound vacuous. The authors are honest about this, but the abstract's 'regardless of the ML models used' applies to validity, not to the power guarantee. That distinction is easy to miss and should be made prominent.\n\nMore serious, in my view, is Proposition 2. The displayed inequality is a lower bound on the lim inf of the log-ratio of type II errors, and it is bounded below by a quantity that is <= 0. That does not imply the ratio decays exponentially; it is a one-sided bound in the wrong direction. The text then interprets the result as 'type II error of ML-FRT is exponentially smaller,' which is not supported by the proof. This needs correction, either by proving an upper bound on the ratio or by weakening the claim.\n\nMinor: the heterogeneity test's plain grid version (without the Berger-Boos correction) is said to be valid 'by majorization' but no proof is included in Appendix C; only the gamma-corrected version is proved. The ERM assumption on the fitted models is not exactly satisfied by random forests, but the authors flag this themselves.\n\nWho benefits: applied statisticians and experimenters who want finite-sample valid tests for complex effects and are willing to treat power as heuristic under misspecification. It deserves a serious referee. My recommendation: send to peer review, but require the authors to fix Proposition 2 and to reframe the power claims as conditional on an unverifiable sign condition.","headline":"The finite-sample validity result is real and worth engaging, but the power claims are more conditional than the abstract suggests; send it out, with a request to fix Proposition 2.","tokens_in":33488,"tokens_out":2384,"would_cite":true,"duration_ms":25249,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F03","62F40","62G10"],"pacs":[],"model":"deepseek-v4-flash","headline":"A cross-validation gap between two ML models is a finite-sample valid test for treatment effects; better predictors make it exponentially more powerful.","keywords":["randomization test","Fisher randomization test","machine learning","treatment effect heterogeneity","spillover effects","cross-validation","finite-sample inference","A/B testing"],"falsifier":"Simulate data of the form $Y_i = B X_i + Z_i h(X_i) + \\varepsilon_i$ with a large negative $B E[X_i]$ and use the small model class $F_1=\\{(x,z)\\mapsto cz\\}$ as in Example 4; if the observed rejection rate of Procedure 1 stays near the nominal level while the sample version of $\\Delta$ is negative, Condition 2 fails and the exponential power bound of Theorem 2 does not apply.","tokens_in":32542,"feed_emoji":"📊","tokens_out":8941,"duration_ms":75634,"temperature":0.7,"pith_summary":"The paper proposes a way to test whether a treatment has any effect in an A/B experiment by using the difference between two machine-learning prediction errors as a randomization-test statistic. One ML model predicts the outcome from covariates alone; the other also uses the treatment assignment. Because treatments are randomized, the null distribution of this statistic can be generated by re-randomizing treatments, which makes the test valid in finite samples no matter which ML models are used. The paper's power analysis shows that better predictive models give exponentially smaller type II error, and that the advantage grows when effects are heterogeneous or spill over a network. The finite-sample validity needs no assumptions, but the power result rests on a direction-agreement condition that the user's chosen model class must satisfy.","feed_headline":"ML prediction gap yields exact A/B tests at any sample size","feed_subtitle":"A cross-validation error difference becomes a valid p-value, and flexible models make the test more powerful.","key_machinery":"The engine is the test statistic $t_n(Y,Z,X)=CV_{n,k}(M_0)-CV_{n,k}(M_1)$: the difference in $k$-fold cross-validation squared errors between an ML model fitted on covariates alone and one that also includes the treatment. Under the null, re-assigning treatments to the same outcomes produces exchangeable copies of this statistic, so the p-value in Procedure 1 is finite-sample valid by standard randomization arguments. The power analysis is carried by $\\Delta$, the excess prediction error of the randomized-treatment distribution over the observed-treatment distribution; a larger $\\Delta$ separates the sampling and randomization distributions and produces the exponential type II error bound.","core_discovery":"The paper's central claim is that the randomization p-value of Procedure 1 is exactly valid under the global null $H_0^{glob}$ for every sample size and every choice of ML models (Theorem 1), and that under the alternative its type II error decays as $O(k\\exp(-Cn\\Delta^2/(kM^4)))$ (Theorem 2). The quantity $\\Delta$ is the predictive gain from using the real treatment instead of a randomly re-assigned treatment, and Condition 2 of Assumption 2 ensures $\\Delta\\ge 0$ by requiring the in-class best predictor under randomization to agree on average with the sign of the true effect. In the comparison with a residualized Fisher randomization test, the paper shows $\\Delta_{ML}=\\pi(1-\\pi)E[h^2(X)]$ whereas $\\Delta_{RES}=\\pi(1-\\pi)(E[h(X)])^2$, so effects that cancel on average still produce signal for the new test. The same logic is extended to test effect homogeneity by maximizing p-values over a grid of constant effects, and to test spillovers by conditioning the randomization distribution on focal units' treatments.","pith_inferences":["The paper leaves implicit that Theorem 1's validity is agnostic to how the ML models are trained, so the procedure could be applied to unstable or non-convex predictors; the real question is whether the model class makes $\\Delta$ large.","A practical diagnostic suggested by the theory: estimate $\\Delta$ from the sample; a near-zero or negative estimate flags a model class that will be powerless even though the test remains valid.","The covariate-balance adaptation in Section 2.3 points to a general use: the same statistic can serve as a finite-sample check on whether randomization actually balanced pre-treatment covariates.","The power comparison with residualized tests suggests a testable extension to factorial designs, where $\\Delta$ would compare a full model with all treatment arms against a reduced model and Condition 2 would require sign agreement for each contrast."],"forward_implications":["Any randomized experiment, regardless of which black-box predictors are used, obtains an exact level-$\\alpha$ test for the presence of any treatment effect at any sample size.","Using flexible ML models instead of linear models enlarges $\\Delta$ from $\\pi(1-\\pi)(E[h(X)])^2$ to $\\pi(1-\\pi)E[h^2(X)]$, so heterogeneous effects that cancel in the average become detectable.","The same CV-difference construction detects network spillovers when the randomization distribution is conditioned on focal units' treatments, with finite-sample validity.","The proof yields a sample-size formula (Equation 11) that experiment designers can use to hit a target power, given estimates of $\\Delta$ and the loss bound."],"supporting_citations":[{"why":"Supplies the Fisher randomization test foundation that Procedure 1 generalizes to ML-based statistics.","marker":"(Fisher, 1935)"},{"why":"Provides Theorem 15.2.1, the standard result used to prove finite-sample validity of the conditional randomization p-value.","marker":"(Lehmann and Romano, 2005)"},{"why":"Appendix I is cited alongside Lehmann and Romano for the validity of randomization p-values with ties.","marker":"(Young, 2019)"},{"why":"Gives the randomization-inference framework for treatment effect variation that Section 4 extends by maximizing p-values over a grid.","marker":"(Ding et al., 2016)"},{"why":"Supplies exact p-values for network interference, a baseline and background for the spillover test in Section 5.","marker":"(Athey et al., 2018)"},{"why":"Provides the conditional randomization test under interference that the spillover procedure adapts via focal units.","marker":"(Basse et al., 2019)"},{"why":"Defines the covariate-adjusted Fisher randomization test that serves as the residualized comparison in Section 3.1.","marker":"(Zhao and Ding, 2021)"},{"why":"Supplies the Rademacher complexity and concentration inequalities used in the proof of Theorem 2.","marker":"(Wainwright, 2019)"},{"why":"Gives the confidence-set p-value maximization used to construct the validated heterogeneity test pval^{het,γ}_n.","marker":"(Berger and Boos, 1994)"}],"fun_headline_variants":["ML cross-validation gap gives exact A/B p-values","Finite-sample valid A/B tests via ML randomization","ML gap test detects heterogeneity and spillovers","Exact A/B inference from ML prediction gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The power guarantee depends on Condition 2 of Assumption 2: the best model inside the user's chosen model class, fitted with randomized treatments, must on average give the treatment effect the same sign as the true effect; this condition cannot be checked from data and fails when a poor model class is paired with a dominant baseline effect.","fun_headline_variants_meta":{"raw":{"variants":["ML cross-validation gap gives exact A/B p-values","Finite-sample valid A/B tests via ML randomization","ML gap test detects heterogeneity and spillovers","Exact A/B inference from ML prediction gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000455,"raw_usage":{"total_tokens":2271,"prompt_tokens":916,"completion_tokens":1355,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":1295}},"tokens_in":532,"tokens_out":1355,"duration_ms":9294,"temperature":1.0,"reasoning_tokens":1295,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:36:43.409341+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate data of the form $Y_i = B X_i + Z_i h(X_i) + \\varepsilon_i$ with a large negative $B E[X_i]$ and use the small model class $F_1=\\{(x,z)\\mapsto cz\\}$ as in Example 4; if the observed rejection rate of Procedure 1 stays near the nominal level while the sample version of $\\Delta$ is negative, Condition 2 fails and the exponential power bound of Theorem 2 does not apply.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Fisher randomization test foundation that Procedure 1 generalizes to ML-based statistics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Theorem 15.2.1, the standard result used to prove finite-sample validity of the conditional randomization p-value."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Appendix I is cited alongside Lehmann and Romano for the validity of randomization p-values with ties."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the conditional randomization test under interference that the spillover procedure adapts via focal units."},{"cited_title":"and Ding, P","cited_arxiv_id":null,"evidence_quote":"Defines the covariate-adjusted Fisher randomization test that serves as the residualized comparison in Section 3.1."}],"review_version":1}