REVIEW 4 major objections 4 minor 17 references
Adaptive sample splitting for randomization tests
T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Randomization tests for subgroups gain power when sample splitting follows a certainty score that sends easy-to-predict units to testing and hard-to-predict units to estimation.
desk verdict Solid validity result and a genuinely new splitting idea, but the power comparison doesn't isolate the adaptive allocation from the final negative-CATE removal, so the headline gain is not as clean as claimed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the certainty score $C_i=|2e(X_i,Y_i)-1|$, where $e(X_i,Y_i)$ is the posterior probability that unit $i$ received treatment given its covariates and outcome; under the paper's Gaussian model this equals the sigmoid $\sigma([Y_i-\mu(X_i)]\tau(X_i)/\nu^2)$. Algorithm 1 starts with a small nuisance fold selected by leverage and moves units from inference to estimation one at a time, choosing the unit that minimizes $\mathrm{sign}(\hat\tau(X_j))\cdot \hat C_j$, until the CATE predictions converge or each subgroup keeps at least a preset proportion $\rho$ of its units. The testing side uses an augmented inverse-probability-weighted statistic built from estimated $\mu$ and $\tau$; the estimation side uses BaR-learner, which extends the residual-regression R-learner loss for CATE estimation by imputing treatment assignments for units whose outcomes make the assignment nearly certain. This allocation mechanism is what makes high-certainty units enlarge the observed statistic relative to the randomized reference distribution without inflating its variance.
What would settle it
Simulate the paper's default five-subgroup design with $n=500$ and a linear CATE, but draw $\epsilon_i$ from a centered exponential or $t_3$ distribution so that Assumption 3 fails; if AdaSplit's mean subgroup p-values are not uniformly smaller than random splitting across 100 runs, the Gaussian model foundation of the allocation rule would be falsified. Alternatively, compute the exact conditional p-value in (9) for a small subgroup and compare it with the Gaussian formula in Theorem 1; a discrepancy larger than the claimed $O(|J_k|^{-1/2})$ would undermine the threshold allocation.
Extended reading notes
Core claim
AdaSplit's central discovery is a division of labor: under the Gaussian outcome model $Y_i=\mu(X_i)+(Z_i-1/2)\tau(X_i)+\epsilon_i$, the posterior assignment probability is $e(X_i,Y_i)=\sigma([Y_i-\mu(X_i)]\tau(X_i)/\nu^2)$, so the certainty score $C_i=|2e(X_i,Y_i)-1|$ is high exactly when the unit's treatment arm is imputable from covariates and outcome. Such high-certainty units dominate the mean-variance ratio of the AIPW statistic and should be kept in the inference fold; low-certainty units, whose assignments are needed to train the CATE estimator, should be put in the nuisance fold. The paper formalizes this tradeoff through a Gaussian approximation of the conditional p-value (Theorem 1), a threshold solution for the relaxed allocation (Proposition 2), and a BaR-learner that imputes held-out assignments with inverse-probability reweighting. Theorem 2 shows that the adaptive folds preserve finite-sample validity and make null p-values independent across subgroups, so closed testing with an exact combination method for independent p-values controls the family-wise error rate.
Load-bearing premise
The allocation rule derives from a certainty-score formula that assumes Gaussian, homoscedastic outcome errors, so when that model is misspecified (as in the binary-outcome trial) the score is only heuristic and the power advantage over random splitting is not theoretically guaranteed.
Editorial extensions
If this is right
- Subgroup randomization tests can gain power from an estimated CATE without losing finite-sample validity, as long as the estimation fold is chosen adaptively but independently of the inference-fold assignments.
- Because null p-values are conditionally independent across subgroups, closed testing can use an exact combination method for independent p-values, giving stronger family-wise error control than conservative union-style corrections.
- The deterministic splitting rule makes the procedure reproducible and lets it stop early, so subgroups with strong effects keep a larger inference fold and hence smaller p-values.
- BaR-learner's assignment imputation recovers most of the efficiency of fitting the CATE on the full sample, which is what lets the shared estimator help all subgroups at once.
- In the paper's real-data analysis, the procedure identifies interpretable patient subgroups with significant treatment effects where difference-in-means and random-split AIPW tests reject none.
Reading between the lines
- The qualitative split—hard-to-predict units fit the model, easy-to-predict units run the test—does not obviously require Gaussian errors; replacing the sigmoid score with a nonparametric estimate of $P(Z=1\mid X,Y)$ is a natural robustness check, though the paper does not make this claim.
- The same certainty-score logic transfers to other estimation-plus-testing pipelines, such as change-point localization or conditional independence testing, where the paper only sketches the direction.
- The procedure can be viewed as active learning for hypothesis tests: it spends observed treatment assignments on the units whose labels most reduce CATE estimation error, the opposite of uncertainty sampling for classification; a formal comparison to uncertainty sampling would be an editorial extension, not a paper result.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes AdaSplit, an adaptive sample-splitting procedure for randomization tests in subgroup analysis. It defines a certainty score based on the posterior assignment probability e(X_i,Y_i), argues that high-certainty units are more valuable for testing while low-certainty units are more important for estimating the conditional average treatment effect (CATE), and develops a greedy algorithm that moves low-certainty and negative-CATE units into a nuisance fold used to fit a CATE estimator (the BaR-learner). The paper proves finite-sample Type I error control for each subgroup and conditional independence of null p-values (Theorem 2), and gives asymptotic Gaussian approximations for power under a known-nuisance model (Theorem 1, Propositions 2 and 4). Simulation experiments compare AdaSplit with difference-in-means and random-split baselines, and an application to the SPRINT trial identifies three subgroups with significant effects. The central claims are that AdaSplit preserves validity while improving power over random splitting via its adaptive allocation.
Significance. If the power claim can be properly isolated, the paper makes a useful contribution: it gives a rigorous finite-sample validity guarantee for a data-adaptive sample-splitting scheme in randomization tests, a practically relevant setting where cross-fitting and random splitting are standard. The conditional independence of null p-values (Theorem 2) is a nice property that enables simple combination methods such as Fisher's method in closed testing. The BaR-learner, which uses imputed assignments for held-out units, is an interesting and potentially reusable tool. The authors provide code for reproducibility, and the validity theorem is stated with a clear proof. However, the current experimental evidence for the power advantage is confounded with test-fold enrichment and fold-size differences, and the asymptotic power theory is derived in a regime that the algorithm deliberately leaves; these issues must be addressed before the central contribution is supported.
major comments (4)
- [Section 4.1.3 and Algorithm 1 (final step)] The headline power comparison in Figure 4 and Table 4 does not isolate the proposed allocation rule. The final step of Algorithm 1 (Section 3.3) removes from J_k the units with the most negative estimated CATEs before computing p-values, while the RT(RandomSplit) baseline in Section 4.1.3 tests a random 50/50 split and performs no such removal. The gain reported for AdaSplit could therefore be produced entirely by this final test-fold enrichment, independently of the certainty-score allocation or BaR-learner. To support the central claim, the authors should add an ablation that applies the same negative-CATE removal (and, ideally, the same final inference-fold size) to the random-split baseline, or otherwise vary only the allocation criterion while holding the test-fold composition fixed.
- [Section 3.1 (Theorem 1, Assumption 5) and Section 3.3] The asymptotic power theory and the algorithm operate in incompatible regimes. Theorem 1 and Proposition 2 are proved under Assumption 5, which requires e(X_j,Y_j) ∈ [δ,1−δ] for all j in J_k, i.e., certainty scores bounded away from 1. The AdaSplit algorithm, by contrast, deliberately reserves for the inference fold the units with the highest certainty scores, for which e is close to 0 or 1 (see the discussion after Proposition 2 and Step 1 of Algorithm 1). The approximation in (11) therefore does not apply to the selected folds, so the theoretical motivation for the allocation rule is not established in the setting where AdaSplit is designed to operate. The authors should either extend the Berry–Esseen argument to accommodate e near the boundary or state explicitly that the power analysis is heuristic for the selected folds.
- [Section 4.2 (SPRINT application) and Eq. (13)] The real-data demonstration uses a binary outcome while the score is derived from the Gaussian model. The certainty score in (13) follows from the Gaussian homoscedastic model in Assumption 3, but the SPRINT primary outcome analysed in Section 4.2 is a binary indicator. The paper does not provide a misspecification analysis or a binary-outcome simulation, so the three rejected subgroups rest on a heuristic whose power and validity properties in this setting are unexamined. A binary-outcome simulation, or an argument for why the score remains an appropriate allocation criterion under non-Gaussian outcomes, is needed before the SPRINT result can be interpreted.
- [Section 4.1.3 and Figure 5] The experimental setup confounds the adaptive allocation with the size of the inference fold. Because AdaSplit's stopping rule can terminate before the nuisance proportion reaches the maximum ρ, it often tests on a larger inference fold than the 50/50 random-split baseline (Figure 5b). Since a larger test fold mechanically increases randomization test power, the advantage in Figure 4 could be partly due to the larger test fold rather than to the certainty-score allocation. The authors should match the final inference-fold sizes across methods, or condition on the realized π_k, to separate these effects.
minor comments (4)
- [Algorithm 1] The while condition 'max{π[K]}≤ρ' in Algorithm 1 appears to be a typographical error: with an initial inference proportion near 1−p and ρ=0.5, the loop would never run. The text indicates the loop should continue while every subgroup still has inference proportion at least ρ, i.e., the condition should be min_k π_k ≥ ρ.
- [Section 3.2] There is a typo, 'R-leanrer', which should be 'R-learner'.
- [Section 3.3 and Algorithm 1] The final removal step is written as 'until π_k < ρ' in the pseudocode, but the text says the removal happens 'if ... π_k ≥ ρ after termination'. Please make the stopping boundary consistent: removal should stop when the inference proportion reaches ρ, not drops below it.
- [Section 4.1.3] The statement that 'all methods control their type I errors at the nominal level 0.2' is based on Table 2, where the Monte-Carlo standard errors are about 0.04; a sentence clarifying how the reported values relate to the 0.2 level given this uncertainty would be helpful.
Circularity Check
Self-contained derivation; no circular reduction found.
full rationale
AdaSplit's derivation is self-contained. The certainty score in eq. (1) is defined through the posterior assignment probability e(X_i, Y_i), and Proposition 1 derives that probability as a sigmoid of (Y_i - mu(X_i))tau(X_i)/nu^2 under the explicit Gaussian assumption (Assumption 3). This is a Bayes-rule identity in which tau is a model parameter, not a fitted output renamed as a prediction. The allocation in Algorithm 1 uses only assignments in the nuisance fold I for fitting tau and e, and Theorem 2 proves validity by conditioning on the selected fold I(O[n]) and showing null assignments in the inference fold remain independent Bernoulli; no step in that proof substitutes the final p-value back into the selection criterion. The power analysis (Theorem 1, Propositions 2-4) is explicitly an oracle calculation: Section 3.1 says it 'assumes mu and tau are known' and the practical algorithm replaces them by estimates, so the theoretical optimality statement does not reduce to the fitted values. The fact that the score depends on the very tau that the tests target is a modeling choice for the power analysis, not a validity loop, since the estimate of tau comes only from the nuisance fold. The final removal of units with the most negative estimated CATEs is a deliberate, theory-motivated enrichment of the test fold; because it uses only nuisance-fold information it does not create a circularity, though it means the headline comparison with RT(RandomSplit) does not isolate the certainty-score ordering as the sole cause of the power gain. That is a benchmarking/identification caveat, not an equation-level circularity. The self-citation to Zhang and Zhao (2023) for the standard randomization-test validity argument is not load-bearing in a circular way: the paper supplies its own partition-invariance proof, and the cited result is a published general theorem. No claimed prediction or first-principles result is equivalent by construction to its inputs.
Assumptions & free parameters
free parameters (6)
- Initial nuisance proportion p =
0.05
- Maximum nuisance proportion rho =
0.5
- Stopping threshold epsilon_l =
0.01
- Convergence window n0 =
50
- Monte-Carlo draws =
1000
- Noise variance nu^2 =
estimated in (25)
assumptions (7)
- domain assumption Treatment assignments are independent Bernoulli draws with probability 1/2, e(X_i)=1/2 for all i.
- domain assumption Observed outcome Y_i = Z_i Y_i(1) + (1-Z_i) Y_i(0).
- domain assumption Y_i = mu_0(X_i) + Z_i tau(X_i) + epsilon_i with iid N(0, nu^2) errors.
- ad hoc to paper The third-moment to second-moment ratio of weights W_j is O_P(|J_k|^{-1/2}).
- ad hoc to paper e(X_j,Y_j) in [delta, 1-delta] for all j in J_k.
- ad hoc to paper Bounded covariates and outcomes, conditional independence (epsilon_i, Z_i) indep B_i | X_i,Y_i, and selection probability p(x,y) > c_p.
- standard math Berry-Esseen bounds, matrix Bernstein inequality, Neyman-Pearson lemma.
Cite this review
Pith. "Pith review of Adaptive sample splitting for randomization tests." pith.science (2026). https://pith.science/paper/VLRI6QH6
@misc{pith2026250421572,
author = {Pith},
title = {Pith review of: Adaptive sample splitting for randomization tests},
year = {2026},
howpublished = {\url{https://pith.science/paper/VLRI6QH6}},
note = {Machine review of arXiv:2504.21572}
}
abstract
Randomization tests are widely used to generate finite-sample valid $p$-values for causal inference on experimental data. However, when applied to subgroup analysis, these tests may lack power due to small subgroup sizes. Incorporating a shared estimator of the conditional average treatment effect (CATE) can substantially improve power across subgroups but requires sample splitting to preserve validity. To this end, we quantify each unit's contribution to estimation and testing using a certainty score, which measures how certain the unit's treatment assignment is given its covariates and outcome. We show that units with higher certainty scores are more valuable for testing but less important for CATE estimation, since their treatment assignments can be accurately imputed. Building on this insight, we propose AdaSplit, a sample splitting procedure that adaptively allocates units between estimation and testing to maximize their overall contribution across tasks. We evaluate AdaSplit through simulation studies, demonstrating that it yields more powerful randomization tests than baselines that omit CATE estimation or rely on random sample splitting. Finally, we apply AdaSplit to a blood pressure intervention trial, identifying patient subgroups with significant treatment effects.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[2]
Leta,b∈ Rm with bi > 0 for alli, and let B∈ (0,Pm i=1bi]
Lemma 1(Neyman and Pearson [1933]). Leta,b∈ Rm with bi > 0 for alli, and let B∈ (0,Pm i=1bi]. Then the optimization problem max w∈[0,1]m mX i=1 wiai, subject to mX i=1 wibi≤B admits an optimal solution of the form w∗ i = 1, if ai/bi >c and ai≥ 0, c′, if ai/bi =c and ai≥ 0, 0, if ai/bi <c or ai < 0, for somec≥ 0 and c′∈ [0, 1). Proof of Lemma 1.As ...
work page 1933
-
[3]
We next introduce Robinson [1988]’s transformation of the linear model in(3). Taking an expectation of both sides of the model conditional onXi, µ(Xi) = E[Yi|Xi] =µ0(Xi) +e(Xi)τ(Xi). Subtracting this from the original model yields Yi =µ(Xi) + [Zi−e(Xi)]τ(Xi) +ϵi. By the normal assumption ofϵi in (3) and e(Xi) = 1/2, the outcomeYi conditional on Xi =x and ...
work page 1988
-
[5]
By Ek− ˜Ek =P j∈Jk ˆWj e(Xj,Yj)−e(Xj) , we can rewrite (28) as ˆPk(OJk) = 1− Φ ˜V−1/2 k V 1/2 k V−1/2 k Ak + ˜V−1/2 k h Ek− ˜Ek i + OP 1/ p |Jk| . Conditional onX[n],Y [n] and ZI, the remaining randomness in thep-value comes from the observed treatment assignmentsZJk in the summationAk defined above. Similar to the proof above, we can first check thatˆWj[...
work page 2010
-
[6]
C Paratore, C Zichi, M Audisio, M Bungaro, A Caglio, R Di Liello, T Gamba, P Gargiulo, A Mariniello, ML Reale, et al. Subgroup analyses in randomized phase iii trials of systemic treatments in patients with advanced solid tumours: a systematic review of trials published between 2017 and 2020.ESMO open, 7(6): 100593,
work page 2017
-
[7]
Active learning for convolutional neural networks: A core-set approach
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489,
-
[9]
Define the least-squares estimatorˆτ OLS(x) =x⊤ ˆβOLS I from R-learner as ˆβOLS I = Σ−1 I ϕI with ϕI =n−1 I nX i=1 BiXi ˆRi. (22) Let Ui := E[Xiϵi|Xi,Bi]. Due to the selection bias inI, ˆβOLS I −β = Σ−1 I 1 nI nX i=1 BiXiUi +BiXi(ϵi−Ui) +µ(Xi)− ˆµ(Xi) 1−e(Xi) p → (E[X⊤ i Xi|Bi = 1])−1E[Xiϵi|Bi = 1]. (23) In contrast, without conditioning onBi, we have E[X...
work page 1952
-
[11]
By the Berry-Essen theorem in Chapter 3 of Chen et al. [2010], we have sup t∈R P ˜ZJk n ˜V−1/2 k ˜Ak≤t o − Φ(t) ≤ C ˜V 3/2 k X j∈Jk ˆW 3 j E ˜Zj n | ˜Zj−e(Xj)|3 o =OP 1/ p |Jk| . where C is a universal constant. The equality holds under Assumptions 4 and
work page 2010
-
[14]
to provide the validity of our randomization tests. Here, we partition the spaceZ of assignments Z[n] into disjoint subsets based on the selected nuisance foldI =I(X[n],Y [n],Z [n]): AI(X[n],Y [n]) ={z[n]∈Z :I(X[n],Y [n],z [n]) =I}, for all valuesI ofI. Every subsetAI satisfies an invariance property: ifzI =z′ I, z[n]∈A I(X[n],Y [n]) =⇒ z′ [n]∈A I(X[n],Y ...
work page 2023
Show all 17 references
-
[16]
B.8 Proof of Proposition 6 Proof
Marginalizing outXi proves the claim. B.8 Proof of Proposition 6 Proof. We first divide the errorˆβI−β into two error terms. ˆβI−β = (Σ−1 [n]− Σ−1)ˆϕ[n] + Σ−1(ˆϕ[n]− Σβ). As in the last subsection, by the matrix Bernstein inequality in Vershynin [2018], E[(Σ−1 [n]− Σ−1)ˆϕ[n]] ...
2018
-
[17]
43 C Additional simulations C.1 Details of Figure 3 To generate Figure 3, we simulate a sample ofn = 200 i.i.d
Combining the bounds for both error terms proves the claim. 43 C Additional simulations C.1 Details of Figure 3 To generate Figure 3, we simulate a sample ofn = 200 i.i.d. units. For each unit i, we generate a covariateXi ∼ Unif(0, 5), and a treatment assignment variable Zi∼ B...
2016
-
[1935]
Prepivoted permutation tests.arXiv preprint arXiv:2102.04423,
Colin B Fogarty. Prepivoted permutation tests.arXiv preprint arXiv:2102.04423,
-
[1964]
Submodularity in data subset selection and active learning
29 Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. InInternational conference on machine learning, pages 1954–1963. PMLR,
1954
-
[2006]
Statistical performance guarantee for subgroup identification with generic machine learning.arXiv preprint arXiv:2310.07973,
Michael Lingzhi Li and Kosuke Imai. Statistical performance guarantee for subgroup identification with generic machine learning.arXiv preprint arXiv:2310.07973,
-
[2016]
Enhanced inference for distributions and quantiles of individual treatment effects in various experiments.arXiv preprint arXiv:2407.13261,
Zhe Chen and Xinran Li. Enhanced inference for distributions and quantiles of individual treatment effects in various experiments.arXiv preprint arXiv:2407.13261,
-
[2018]
and Ramdas et al. [2023]. Conditioning on the selected nuisance foldI(O[n]), the assignmentsZJK0 in the null inference fold are independent Bernoulli random variables. Moreover, conditioning on X[n],YI∪JK1 , and ZI∪JK1 fixes all other sources of randomness, including the estim...
2023
-
[2021]
Deep batch active learning by diverse, uncertain gradient lower bounds
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv:1906.03671,
1906 arXiv
-
[2023]
Ml-assisted randomization tests for detecting treatment effects in a/b experiments.arXiv preprint arXiv:2501.07722,
Wenxuan Guo, JungHo Lee, and Panos Toulis. Ml-assisted randomization tests for detecting treatment effects in a/b experiments.arXiv preprint arXiv:2501.07722,
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.