Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

ML-assisted Randomization Tests for Detecting Treatment Effects in A/B Experiments

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A cross-validation gap between two ML models is a finite-sample valid test for treatment effects; better predictors make it exponentially more powerful.

desk verdict The finite-sample validity result is real and worth engaging, but the power claims are more conditional than the abstract suggests; send it out, with a request to fix Proposition 2. read the letter →

arxiv 2501.07722 v1 pith:KPGCCCEO submitted 2025-01-13 stat.ME stat.ML

Wenxuan Guo , JungHo Lee , Panos Toulis This is my paper · ORCID
classification stat.MEstat.ML MSC 62F0362F4062G10
keywords randomizationtestFishermachinelearningtreatmenteffectheterogeneityspillovereffectscross-validationfinite-sampleinferenceA/Btesting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a way to test whether a treatment has any effect in an A/B experiment by using the difference between two machine-learning prediction errors as a randomization-test statistic. One ML model predicts the outcome from covariates alone; the other also uses the treatment assignment. Because treatments are randomized, the null distribution of this statistic can be generated by re-randomizing treatments, which makes the test valid in finite samples no matter which ML models are used. The paper's power analysis shows that better predictive models give exponentially smaller type II error, and that the advantage grows when effects are heterogeneous or spill over a network. The finite-sample validity needs no assumptions, but the power result rests on a direction-agreement condition that the user's chosen model class must satisfy.

What carries the argument

The engine is the test statistic $t_n(Y,Z,X)=CV_{n,k}(M_0)-CV_{n,k}(M_1)$: the difference in $k$-fold cross-validation squared errors between an ML model fitted on covariates alone and one that also includes the treatment. Under the null, re-assigning treatments to the same outcomes produces exchangeable copies of this statistic, so the p-value in Procedure 1 is finite-sample valid by standard randomization arguments. The power analysis is carried by $\Delta$, the excess prediction error of the randomized-treatment distribution over the observed-treatment distribution; a larger $\Delta$ separates the sampling and randomization distributions and produces the exponential type II error bound.

What would settle it

Simulate data of the form $Y_i = B X_i + Z_i h(X_i) + \varepsilon_i$ with a large negative $B E[X_i]$ and use the small model class $F_1=\{(x,z)\mapsto cz\}$ as in Example 4; if the observed rejection rate of Procedure 1 stays near the nominal level while the sample version of $\Delta$ is negative, Condition 2 fails and the exponential power bound of Theorem 2 does not apply.

Watch

Extended reading notes

Core claim

The paper's central claim is that the randomization p-value of Procedure 1 is exactly valid under the global null $H_0^{glob}$ for every sample size and every choice of ML models (Theorem 1), and that under the alternative its type II error decays as $O(k\exp(-Cn\Delta^2/(kM^4)))$ (Theorem 2). The quantity $\Delta$ is the predictive gain from using the real treatment instead of a randomly re-assigned treatment, and Condition 2 of Assumption 2 ensures $\Delta\ge 0$ by requiring the in-class best predictor under randomization to agree on average with the sign of the true effect. In the comparison with a residualized Fisher randomization test, the paper shows $\Delta_{ML}=\pi(1-\pi)E[h^2(X)]$ whereas $\Delta_{RES}=\pi(1-\pi)(E[h(X)])^2$, so effects that cancel on average still produce signal for the new test. The same logic is extended to test effect homogeneity by maximizing p-values over a grid of constant effects, and to test spillovers by conditioning the randomization distribution on focal units' treatments.

Load-bearing premise

The power guarantee depends on Condition 2 of Assumption 2: the best model inside the user's chosen model class, fitted with randomized treatments, must on average give the treatment effect the same sign as the true effect; this condition cannot be checked from data and fails when a poor model class is paired with a dominant baseline effect.

Editorial extensions

If this is right

  • Any randomized experiment, regardless of which black-box predictors are used, obtains an exact level-$\alpha$ test for the presence of any treatment effect at any sample size.
  • Using flexible ML models instead of linear models enlarges $\Delta$ from $\pi(1-\pi)(E[h(X)])^2$ to $\pi(1-\pi)E[h^2(X)]$, so heterogeneous effects that cancel in the average become detectable.
  • The same CV-difference construction detects network spillovers when the randomization distribution is conditioned on focal units' treatments, with finite-sample validity.
  • The proof yields a sample-size formula (Equation 11) that experiment designers can use to hit a target power, given estimates of $\Delta$ and the loss bound.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that Theorem 1's validity is agnostic to how the ML models are trained, so the procedure could be applied to unstable or non-convex predictors; the real question is whether the model class makes $\Delta$ large.
  • A practical diagnostic suggested by the theory: estimate $\Delta$ from the sample; a near-zero or negative estimate flags a model class that will be powerless even though the test remains valid.
  • The covariate-balance adaptation in Section 2.3 points to a general use: the same statistic can serve as a finite-sample check on whether randomization actually balanced pre-treatment covariates.
  • The power comparison with residualized tests suggests a testable extension to factorial designs, where $\Delta$ would compare a full model with all treatment arms against a reduced model and Condition 2 would require sign agreement for each contrast.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ML-assisted randomization tests for A/B experiments. The main procedure (Procedure 1) uses as a test statistic the difference between k-fold cross-validation errors of two ML models, one including the treatment variable and one excluding it, and computes an FRT-style p-value by re-randomizing treatments. The paper claims finite-sample validity of this p-value under the global null regardless of the ML models used (Theorem 1), and an exponential type II error bound under an alternative when the ML class predicts well (Theorem 2). It also extends the framework to covariate balance, heterogeneity testing via p-value maximization, and spillover tests under network interference (Theorem 3). A power comparison with a residualized FRT (Proposition 1 and 2) and a sample-size procedure are presented, followed by simulations for constant, heterogeneous, and spillover effects.

Significance. If the results were fully established, the paper would be a useful contribution: Theorem 1 gives a simple, finite-sample valid way to combine black-box ML prediction with randomization inference, and the Delta-based power analysis in Eq. (7) is an appealing way to connect predictive power to testing power. The spillover test in Section 5 is also a clean and correct application of conditional randomization tests. However, the central power claim is more fragile than the abstract suggests: Theorem 2 depends on an unverifiable sign condition (Assumption 2, Condition 2), the proof assumes exact empirical risk minimization that random forest implementations do not satisfy, and Proposition 2's efficiency conclusion does not follow from the displayed inequality. The finite-sample validity results (Theorem 1, Theorem 3, Theorem 5) are correct and are the most solid part of the paper. With a careful reframing of the power results, the paper would be publishable; in its current form the gap between the advertised 'ML improves power' claim and what is actually proven is too large.

major comments (3)
  1. [Section 3, Assumption 2, Condition 2, and Example 4] The assertion in Theorem 2 that Delta >= 0, and hence the exponential type II error bound, relies entirely on Condition 2: E[h(X)(f*_r(X,1)-f*_r(X,0))] >= 0. This condition couples the unidentified true CATE h(X) with the user-chosen model class F1, and it cannot be verified from observed data because h is not identified. Example 4 shows a simple misspecified class with a large baseline where the condition fails and Delta < 0, making the bound vacuous. The abstract and Section 1 advertise power improvements from flexible ML 'regardless of the choice of ML models', but the power theorem is really conditional on an unverifiable alignment condition. Theorem 1's validity is unaffected, but the power result should be stated and interpreted conditionally.
  2. [Appendix B, Additional regularity conditions] The proof of Theorem 2 assumes that in each cross-validation fold the fitted model is an exact empirical risk minimizer over a fixed class F: 'we assume that the fitted models minimize the empirical loss in each cross-validation step'. Random forests, which are the main ML method in the simulations (Sections 6.1-6.3 via caret/randomForest/ranger), do not perform exact ERM over any explicit function class. Consequently, Theorem 2 as proven does not cover the paper's own flagship implementation. The theorem should be stated for an idealized ERM oracle, with the simulations explicitly presented as heuristic evidence, or the implementation should be characterized by a function class and an approximate ERM guarantee.
  3. [Section 3.1, Proposition 2 and Eq. (10)] The displayed result in Proposition 2 is a lower bound on lim inf (1/n) log( P(phi_ML=0) / P(phi_RES=0) ), and Eq. (10) states that the type II error ratio is at least exp(-n[I(Delta_ML)-I(Delta_RES)]). Because I is increasing and Delta_ML >= Delta_RES under Proposition 1, this lower bound tends to zero. A lower bound that tends to zero is vacuous and cannot imply that the ratio itself tends to zero. The conclusion in the paragraph after Eq. (10), that 'the type II error of Procedure 1 is exponentially smaller than that of the residualized method', requires an upper bound of the form P(phi_ML=0)/P(phi_RES=0) <= exp(-n[...]) or a direct large-deviation comparison of the two error probabilities. As written, the efficiency claim does not follow from Proposition 2.
minor comments (5)
  1. [Section 3, Theorem 2 / Appendix B, Lemma 4] The proof of Lemma 4 uses the condition (R+1)alpha > 1 (equivalently R > 1/alpha - 1) and assumes m_R = 0 with probability one, but these conditions are not stated in Theorem 2. They should be included in the theorem statement or the proof should be adjusted.
  2. [Section 2.2] There is a typographical error: 'ANOV A' should be 'ANOVA'.
  3. [Section 6.3] The text says tau_S is varied over {0,0.1,...,1} and {0,0.2,...,2} in Setups (A) and (B), respectively, but the x-axis of Figure 3 is labeled from 0 to 5. The caption or the axis should be made consistent with the reported ranges.
  4. [Section 3.2] The sample size formula in Eq. (11) uses L and M0 estimated from the same sample and omits the factor R as a heuristic. This should be flagged more explicitly as a heuristic without finite-sample guarantee, since the authors themselves note the worst-case bound is conservative.
  5. [Section 6.1] The simulations use Gaussian errors, while Assumption 1 requires bounded outcomes with probability one. The numerical validation of Theorem 2 is therefore only approximate; this is acceptable if stated, but it should be acknowledged in the simulation section.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: validity is a standard Fisherian randomization argument and the power bound is a conditional population-level inequality, not a fitted input.

full rationale

The central validity claim (Theorem 1, Procedure 1) is a direct Fisherian randomization argument: under H_0^glob, (Y, Z, X) and (Y, Z^(r), X) are equal in law conditional on (Y, X), so the p-value is exactly valid for any ML model. No fitted parameter, no estimated constant, and no self-citation enters this step. The power result (Theorem 2) is a conditional mathematical bound: under Assumptions 1 and 2, the type II error is O(k exp(-C n Delta^2/(k M^4))). Delta in Eq. (7) is a population difference of prediction errors under the randomized versus observed treatment assignment; it is not estimated from the data, and the theorem does not claim to predict Delta. The proof that Delta >= 0 uses Condition 2 of Assumption 2, which is an explicitly stated sign condition on the best in-class predictor, not a conclusion smuggled in from a self-citation or from the test statistic. The claim that better predictive models give better power is a substantive quantification, not a tautology, because the exponential bound requires concentration of the CV statistic around its population mean, which is proved via Hoeffding and Rademacher bounds rather than assumed. The sample-size section uses pilot estimates of L and M0 heuristically and explicitly labels the in-sample version as a heuristic; this is a practical approximation, not a derivation on which the main theorems depend. Self-citations (e.g., Guo and Toulis 2023, Basse et al. 2024, Puelz et al. 2021) are contextual or methodological background and do not carry the load of the proofs. The unverifiable nature of Assumption 2.2, illustrated by Example 4, is a real assumption-sensitivity and correctness-risk concern about the scope of the power guarantee, but it is not circular: the paper states the condition, does not derive it, and does not rename a fitted quantity as a prediction. No equation in the paper reduces to its own input, so no circular step is present.

Assumptions & free parameters 0 free parameters · 9 assumptions · 0 invented entities

The theory rests on standard learning-theoretic assumptions and two ad hoc conditions (Condition 2 and ERM) that the authors flag. No free parameters are fitted to make the derivation work; the method's validity does not depend on model tuning.

assumptions (9)
  • domain assumption Potential outcome model Yi(z) = mu + b(Xi) + zi h(Xi) + g(X, z_-i) + eps_i with additive mean-zero independent noise and orthogonal functions.
    This model underlies all hypothesis definitions; independence of errors is not universal in causal ML and may fail in clustered designs, though simulations relax it.
  • domain assumption Assumption 1: Bernoulli treatment assignment, i.i.d. (Xi, eps_i), and bounded outcomes |Yi| <= M.
    Simplifies the power analysis; finite-sample validity does not need it, but the power theorem does.
  • ad hoc to paper Assumption 2, Condition 2: E[h(X)(f*_r(X,1) - f*_r(X,0))] >= 0.
    Needed to guarantee Delta >= 0 and to prove the type II error bound; authors show it can fail (Example 4), so it is a substantive unverifiable condition.
  • standard math Assumption 2, Condition 3: Rademacher complexities of F0, F1 and F1 under randomized assignment are o(1).
    Standard in statistical learning theory and justified in Section B.4 via pseudo-dimension; holds for many function classes.
  • ad hoc to paper Empirical risk minimization for fitted models in each CV fold, as stated in Appendix B, Additional regularity conditions.
    Required for the concentration bounds in Lemma 3; not satisfied exactly by random forests or neural networks, so the theory is approximate for the implemented method.
  • domain assumption Assumption 3: the residualized model fit bm(x) converges uniformly to m*(x) = b + mu(x) + pi h(x).
    Used in Proposition 1 to bridge the idealized residualized method with the practical one.
  • domain assumption Assumption 4: |h(Xi)| <= M/2 and |eps_i| <= M/2 almost surely.
    Ensures residual boundedness so the residualized method falls inside the power-analysis framework.
  • ad hoc to paper Assumption 5: F1 contains the best predictors mu+b(x)+pi h(x) and mu+b(x)+z h(x).
    Needed to simplify Delta_ML to pi(1-pi)E[h^2]; requires the user's model class to nest the truth.
  • ad hoc to paper Assumption 6: existence of a rate function I satisfying tail inequalities for the ML-FRT and RES test statistics.
    Needed for Proposition 2; authors state verifying it is challenging and leave it as future work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ML-assisted Randomization Tests for Detecting Treatment Effects in A/B Experiments." pith.science (2026). https://pith.science/paper/KPGCCCEO

@misc{pith2026250107722,
  author       = {Pith},
  title        = {Pith review of: ML-assisted Randomization Tests for Detecting Treatment Effects in A/B Experiments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KPGCCCEO}},
  note         = {Machine review of arXiv:2501.07722}
}
read the original abstract

Experimentation is widely utilized for causal inference and data-driven decision-making across disciplines. In an A/B experiment, for example, an online business randomizes two different treatments (e.g., website designs) to their customers and then aims to infer which treatment is better. In this paper, we construct randomization tests for complex treatment effects, including heterogeneity and interference. A key feature of our approach is the use of flexible machine learning (ML) models, where the test statistic is defined as the difference between the cross-validation errors from two ML models, one including the treatment variable and the other without it. This approach combines the predictive power of modern ML tools with the finite-sample validity of randomization procedures, enabling a robust and efficient way to detect complex treatment effects in experimental settings. We demonstrate this combined benefit both theoretically and empirically through applied examples.

Figures

Figures reproduced from arXiv: 2501.07722 by the authors.

Figure 1
Figure 1. Rejection rates (in black) and signal strength [PITH_FULL_IMAGE:figures/full_fig_p026_1.png] view at source ↗
Figure 2
Figure 2. Rejection rates for heterogeneous treatment effect across [PITH_FULL_IMAGE:figures/full_fig_p028_2.png] view at source ↗
Figure 3
Figure 3. Rejection rates for spillover effects across [PITH_FULL_IMAGE:figures/full_fig_p030_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Rejection rates for testing constant treatment effects with different baseline effects [PITH_FULL_IMAGE:figures/full_fig_p040_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Design-Based Minimax Theory for Network Experiments

    math.ST 2026-08 conditional novelty 7.0 of 10

    The minimax risk of any network experiment under arbitrary neighborhood interference is a function of the conflict graph of observable exposures, with rates bounded by the graph's independence number, critical degree,...

  2. Adaptive sample splitting for randomization tests

    stat.ME 2025-04 conditional novelty 7.0 of 10

    AdaSplit adaptively divides a trial sample into estimation and testing folds, prioritizing units with uncertain treatment assignments for estimation, to boost the power of subgroup randomization tests.

Reference graph

Works this paper leans on

72 extracted references · 45 canonical work pages · cited by 2 Pith papers

  1. [1]

    L., Bartlett, P

    Anthony, M., Bartlett, P. L., Bartlett, P. L., et al. (1999). Neural network learning: Theoretical foundations , volume 9. cambridge university press Cambridge

    1999

  2. [2]

    Aronow, P. M. (2012). A general method for detecting interference between units in randomized experiments. Sociological Methods & Research , 41(1):3--16

    work page 2012

  3. [3]

    Athey, S., Eckles, D., and Imbens, G. W. (2018). Exact p-values for network interference. Journal of the American Statistical Association , 113(521):230--240

    2018

  4. [4]

    and Robins, J

    Bang, H. and Robins, J. M. (2005). Doubly robust estimation in missing data and causal inference models. Biometrics , 61(4):962--973

    2005

  5. [5]

    L., Harvey, N., Liaw, C., and Mehrabian, A

    Bartlett, P. L., Harvey, N., Liaw, C., and Mehrabian, A. (2019). Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks. Journal of Machine Learning Research , 20(63):1--17

    2019

  6. [6]

    Bartlett, P. L. and Mendelson, S. (2002). Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research , 3(Nov):463--482

    2002

  7. [7]

    Basse, G., Ding, P., Feller, A., and Toulis, P. (2024). Randomization tests for peer effects in group formation experiments. Econometrica , 92(2):567--590

    2024

  8. [8]

    and Feller, A

    Basse, G. and Feller, A. (2018). Analyzing two-stage experiments in the presence of interference. Journal of the American Statistical Association , 113(521):41--55

    2018

Show all 72 references
  1. [9]

    Basse, G., Feller, A., and Toulis, P. (2019). Randomization tests of causal effects under interference . Biometrika , 106(2):487--494

    2019

  2. [10]

    B \'e nard, C., Da Veiga, S., and Scornet, E. (2022). Mean decrease accuracy for random forests: inconsistency, and a practical solution via the sobol-mda. Biometrika , 109(4):881--900

    2022

  3. [11]

    Berger, R. L. and Boos, D. D. (1994). P values maximized over a confidence set for the nuisance parameter. Journal of the American Statistical Association , 89(427):1012--1016

    1994

  4. [12]

    Boucheron, S., Lugosi, G., and Massart, P. (2013). Concentration Inequalities: A Nonasymptotic Theory of Independence . OUP Oxford

    2013

  5. [13]

    Breiman, L. (2001). Random forests. Machine learning , 45:5--32

    2001

  6. [14]

    Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters

    2018

  7. [15]

    Cox, D. R. and Reid, N. (2000). The theory of the design of experiments . Chapman and Hall/CRC

    2000

  8. [16]

    Ding, P., Feller, A., and Miratrix, L. (2016). Randomization inference for treatment effect variation. Journal of the Royal Statistical Society Series B: Statistical Methodology , 78(3):655--671

    2016

  9. [17]

    Dobriban, E. (2022). Consistency of invariance-based randomization tests. The Annals of Statistics , 50(4):2443--2466

    2022

  10. [18]

    Fisher, R. A. (1935). The Design of Experiments . Oliver and Boyd, Edinburgh

    1935

  11. [19]

    Gail, M., Tan, W.-Y., and Piantadosi, S. (1988). Tests for no treatment effect in randomized clinical trials. Biometrika , 75(1):57--64

    1988

  12. [20]

    and Green, D

    Gerber, A. and Green, D. (2012). Field Experiments: Design, Analysis, and Interpretation . W. W. Norton

    2012

  13. [21]

    and Nickl, R

    Gin \'e , E. and Nickl, R. (2021). Mathematical foundations of infinite-dimensional statistical models . Cambridge university press

    2021

  14. [22]

    Granger, C. W. J. (1969). Investigating causal relations by econometric models and cross-spectral methods. Econometrica: journal of the Econometric Society , pages 424--438

    1969

  15. [23]

    Green, D. P. and Kern, H. L. (2012). Modeling Heterogeneous Treatment Effects in Survey Experiments with Bayesian Additive Regression Trees . Public Opinion Quarterly , 76(3):491--511

    2012

  16. [24]

    Grimmer, J., Messing, S., and Westwood, S. J. (2017). Estimating heterogeneous treatment effects and the effects of heterogeneous treatments with ensemble methods. Political Analysis , 25(4):413–434

    2017

  17. [25]

    and Toulis, P

    Guo, W. and Toulis, P. (2023). Invariance-based inference in high-dimensional regression with finite-sample guarantees. arXiv preprint arXiv:2312.15079

    2023 arXiv

  18. [26]

    R., Murray, J

    Hahn, P. R., Murray, J. S., and Carvalho, C. M. (2020). Bayesian Regression Tree Models for Causal Inference: Regularization, Confounding, and Heterogeneous Effects (with Discussion) . Bayesian Analysis , 15(3):965 -- 2020

    2020

  19. [27]

    Hennessy, J., Dasgupta, T., Miratrix, L., Pattanayak, C., and Sarkar, P. (2016). A conditional randomization test to account for covariate imbalance in randomized experiments. Journal of Causal Inference , 4(1):61--80

    2016

  20. [28]

    Hill, J. L. (2011). Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics , 20(1):217--240

    2011

  21. [29]

    Hur, Y., Guo, W., and Liang, T. (2024). Reversible gromov--monge sampler for simulation-based inference. SIAM Journal on Mathematics of Data Science , 6(2):283--310

    2024

  22. [30]

    and Li, M

    Imai, K. and Li, M. L. (2025). Statistical inference for heterogeneous treatment effects discovered by generic machine learning in randomized experiments. Journal of Business & Economic Statistics , 43(1):256--268

    2025

  23. [31]

    Imbens, G. W. and Rubin, D. B. (2015). Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction . Cambridge University Press

    2015

  24. [32]

    and Kogalur, U

    Ishwaran, H. and Kogalur, U. (2024). Fast Unified Random Forests for Survival, Regression, and Classification (RF-SRC) . R package version 3.3.1

    2024

  25. [33]

    Kennedy, E. H. (2022). Semiparametric doubly robust targeted double machine learning: a review. arXiv preprint arXiv:2203.06469

    2022 arXiv

  26. [34]

    Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y., and Pohlmann, N. (2013). Online controlled experiments at large scale. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '13, page 1168–1176, New York, NY, USA. Asso...

    2013

  27. [35]

    Kohavi, R., Tang, D., and Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing . Cambridge University Press

    2020

  28. [36]

    Building predictive models in r using the caret package

    Kuhn and Max (2008). Building predictive models in r using the caret package. Journal of Statistical Software , 28(5):1–26

    2008

  29. [37]

    R., Sekhon, J

    K \"u nzel, S. R., Sekhon, J. S., Bickel, P. J., and Yu, B. (2019). Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the National Academy of Sciences , 116(10):4156--4165

    2019

  30. [38]

    Lehmann, E. L. and Romano, J. P. (2005). Testing statistical hypotheses . Springer Texts in Statistics. Springer, New York, third edition

    2005

  31. [39]

    and Cand \`e s, E

    Lei, L. and Cand \`e s, E. J. (2021). Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Society Series B: Statistical Methodology , 83(5):911--938

    2021

  32. [40]

    Li, M. L. and Imai, K. (2024). Neyman meets causal machine learning: Experimental evaluation of individualized treatment rules. Journal of Causal Inference , 12(1):20230072

    2024

  33. [41]

    and Wiener, M

    Liaw, A. and Wiener, M. (2002). Classification and regression by randomforest. R News , 2(3):18--22

    2002

  34. [42]

    A., Muir, I., and Sun, G

    List, J. A., Muir, I., and Sun, G. (2024). Using machine learning for efficient flexible regression adjustment in economic experiments. Econometric Reviews , 44(1):2--40

    2024

  35. [43]

    A., Shaikh, A

    List, J. A., Shaikh, A. M., and Xu, Y. (2019). Multiple hypothesis testing in experimental economics. Experimental Economics , 22:773--793

    2019

  36. [44]

    and Schmidt, D

    Makalic, E. and Schmidt, D. F. (2022). An efficient algorithm for sampling from sin k (x) for generating random correlation matrices. Communications in Statistics-Simulation and Computation , 51(5):2731--2735

    2022

  37. [45]

    Neyman, J. (1923). Sur les applications de la th \'e orie des probabilit \'e s aux experiences agricoles: Essai des principes. Roczniki Nauk Rolniczych , 10(1):1--51

    1923

  38. [46]

    Oprescu, M., Syrgkanis, V., and Wu, Z. S. (2019). Orthogonal random forest for causal inference. In International Conference on Machine Learning , pages 4932--4941. PMLR

    2019

  39. [47]

    and Wang, X

    Pourahmadi, M. and Wang, X. (2015). Distribution of random correlation matrices: Hyperspherical parameterization of the cholesky factor. Statistics & Probability Letters , 106:5--12

    2015

  40. [48]

    Poyarkov, A., Drutsa, A., Khalyavin, A., Gusev, G., and Serdyukov, P. (2016). Boosted decision tree regression adjustment for variance reduction in online controlled experiments. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Min...

    2016

  41. [49]

    Puelz, D., Basse, G., Feller, A., and Toulis, P. (2021). A Graph-Theoretic Approach to Randomization Tests of Causal Effects under General Interference . Journal of the Royal Statistical Society Series B: Statistical Methodology , 84(1):174--204

    2021

  42. [50]

    Raz, J. (1990). Testing for no effect when estimating a smooth function by nonparametric regression: a randomization approach. Journal of the American Statistical Association , 85(409):132--138

    1990

  43. [51]

    Robins, J. M. and Rotnitzky, A. (1995). Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association , 90(429):122--129

    1995

  44. [52]

    Rosenbaum, P. R. (2002). Covariance adjustment in randomized experiments and observational studies. Statistical Science , 17(3):286--327

    2002

  45. [53]

    Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology , 66(5):688

    1974

  46. [54]

    Schuler, A. (2021). Designing efficient randomized trials: Power and sample size calculation when using semiparametric efficient estimators. The International Journal of Biostatistics , 18:151--171

    2021

  47. [55]

    R., and Fisher, C

    Schuler, A., Walsh, D., Hall, D., Walsh, J. R., and Fisher, C. K. (2020). Increasing the efficiency of randomized trial estimates via linear adjustment for a prognostic score. The International Journal of Biostatistics , 18:329 -- 356

    2020

  48. [56]

    Sobel, M. E. (2006). What do randomized studies of housing mobility demonstrate? causal inference in the face of interference. Journal of the American Statistical Association , 101(476):1398--1407

    2006

  49. [57]

    Syrgkanis, V., Lei, V., Oprescu, M., Hei, M., Battocchi, K., and Lewis, G. (2019). Machine learning estimation of heterogeneous treatment effects with instruments. Advances in Neural Information Processing Systems , 32

    2019

  50. [58]

    Taddy, M., Gardner, M., Chen, L., and Draper, D. (2016). A nonparametric bayesian analysis of heterogenous treatment effects in digital experimentation. Journal of Business & Economic Statistics , 34(4):661--672

    2016

  51. [59]

    J., Foygel Barber, R., Candes, E., and Ramdas, A

    Tibshirani, R. J., Foygel Barber, R., Candes, E., and Ramdas, A. (2019). Conformal prediction under covariate shift. Advances in neural information processing systems , 32

    2019

  52. [60]

    and Kao, E

    Toulis, P. and Kao, E. (2013). Estimation of causal peer influence effects. In International conference on machine learning , pages 1489--1497. PMLR

    2013

  53. [61]

    Tukey, J. W. (1993). Tightening the clinical trial. Controlled clinical trials , 14(4):266--285

    1993

  54. [62]

    L., Perel, P., Clayton, T., Edwards, P., Hernández, A

    Turner, E. L., Perel, P., Clayton, T., Edwards, P., Hernández, A. V., Roberts, I., Shakur, H., and Steyerberg, E. W. (2012). Covariate adjustment increased power in randomized controlled trials: an example in traumatic brain injury. Journal of Clinical Epidemiology , 65(5):474--481

    2012

  55. [63]

    Van der Laan, M. J. and Rose, S. (2011). Targeted learning . Springer

    2011

  56. [64]

    Vovk, V., Gammerman, A., and Shafer, G. (2005). Algorithmic learning in a random world , volume 29. Springer

    2005

  57. [65]

    and Athey, S

    Wager, S. and Athey, S. (2018). Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association , 113(523):1228--1242

    2018

  58. [66]

    Wainwright, M. J. (2019). High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press

    2019

  59. [67]

    and Feng, J

    Williamson, B. and Feng, J. (2020). Efficient nonparametric statistical inference on population feature importance using shapley values. In International conference on machine learning , pages 10282--10291. PMLR

    2020

  60. [68]

    D., Gilbert, P

    Williamson, B. D., Gilbert, P. B., Carone, M., and Simon, N. (2021). Nonparametric variable importance assessment using machine learning techniques. Biometrics , 77(1):9--22

    2021

  61. [69]

    D., Gilbert, P

    Williamson, B. D., Gilbert, P. B., Simon, N. R., and Carone, M. (2023). A general framework for inference on algorithm-agnostic variable importance. Journal of the American Statistical Association , 118(543):1645--1658

    2023

  62. [70]

    Wright, M. N. and Ziegler, A. (2017). Ranger: A fast implementation of random forests for high dimensional data in c++ and R . J. Stat. Softw. , 77(1)

    2017

  63. [71]

    Young, A. (2019). Channeling fisher: Randomization tests and the statistical insignificance of seemingly significant experimental results. The quarterly journal of economics , 134(2):557--598

    2019

  64. [72]

    and Ding, P

    Zhao, A. and Ding, P. (2021). Covariate-adjusted fisher randomization tests for the average treatment effect. Journal of Econometrics , 225(2):278--294

    2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.