{"id":"d4c54fd1-3a08-4142-867e-8984f2019816","arxiv_id":"2504.21572","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"AdaSplit adaptively divides a trial sample into estimation and testing folds, prioritizing units with uncertain treatment assignments for estimation, to boost the power of subgroup randomization tests.","lead":"This paper proposes AdaSplit, a way to split a clinical trial's patients into two groups: one to estimate treatment effects and one to run valid hypothesis tests. It shows that adapting the split to each patient's information content produces more powerful subgroup tests than random splitting.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Power gain is not isolated from final test-fold enrichment: AdaSplit removes negative-CATE units before testing while RT(RandomSplit) does not, so the headline comparison may reflect selection rather than the certainty-score allocation.","rationale":"The reader's verdict of CONDITIONAL is reasonable: Theorem 2, which gives finite-sample Type I error control and conditional independence of null p-values, appears sound, and the simulations show consistent gains. The reader's weakest-assumption concern, that the certainty score is derived from Gaussian homoscedastic errors (Assumption 3) and is only heuristic for binary outcomes, is legitimate but is a generalization issue. My concern is more fundamental: even in the Gaussian setting where the score is correctly specified, the experiments do not isolate the certainty-based allocation from the final selection of a positive-CATE test fold. AdaSplit's final step removes units with the most negative estimated CATEs before computing p-values, while RT(RandomSplit) tests a random 50% subset. Thus the reported power advantage could be driven by this enrichment rather than by the adaptive allocation or BaR-learner. A matched-random-split experiment with the same final fold sizes and the same CATE estimator would settle whether the allocation criterion itself matters. If AdaSplit still wins, the reader's misspecification concern remains and should be probed with binary-outcome simulations. If it does not win, the central power claim is unsupported even under the paper's ideal model. This does not overturn the validity result, so the conditional acceptance stands, but the matched-split experiment should be added to the revision requirements.","tokens_in":29703,"tokens_out":22836,"duration_ms":270077,"concrete_test":"Re-run the default and increased-noise simulations with an additional baseline: after each AdaSplit trial, record the per-subgroup final inference-fold sizes |J_k|; then generate a random nuisance fold of exactly those sizes, refit mu and tau with the same routines (linear mu; BaR-learner), and compute the subgroup p-values with the same Monte Carlo draws. Average over 20 random splits per trial. If RT(RandomSplit-matched) produces p-values no larger than RT(AdaSplit), the reported power advantage is not evidence for certainty-based allocation. Optionally, restrict the random test fold to the |J_k| units with the largest CATE estimated from an initial 5% random nuisance fit, to see whether final positive-CATE enrichment alone reproduces the gain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The power claim rests on a comparison that does not isolate the adaptive allocation mechanism. In Algorithm 1, after the greedy loop stops, the final step removes from J_k the units with the most negative estimated CATEs until the inference proportion drops below rho, and only then are p-values computed (Section 3.3). RT(RandomSplit), in contrast, tests a 50/50 random split with no such selection. Figure 4 therefore shows AdaSplit against a baseline whose test fold contains low- and negative-CATE units that AdaSplit has deliberately removed; the gain could come from this final enrichment alone, with the certainty score and BaR-learner playing little role. The supporting theory (Theorem 1 and Propositions 2-4) assumes mu and tau are known or analyzes individual contributions asymptotically; it does not characterize the full greedy procedure, so it cannot rule out this explanation. The misspecification of the certainty score on binary outcomes is a separate, secondary concern: even in the Gaussian simulations where the score is correctly specified, no experiment holds the test-fold size, estimator, and selection rule fixed while varying only the allocation criterion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AdaSplit, an adaptive sample-splitting procedure for randomization tests in subgroup analysis. It defines a certainty score based on the posterior assignment probability e(X_i,Y_i), argues that high-certainty units are more valuable for testing while low-certainty units are more important for estimating the conditional average treatment effect (CATE), and develops a greedy algorithm that moves low-certainty and negative-CATE units into a nuisance fold used to fit a CATE estimator (the BaR-learner). The paper proves finite-sample Type I error control for each subgroup and conditional independence of null p-values (Theorem 2), and gives asymptotic Gaussian approximations for power under a known-nuisance model (Theorem 1, Propositions 2 and 4). Simulation experiments compare AdaSplit with difference-in-means and random-split baselines, and an application to the SPRINT trial identifies three subgroups with significant effects. The central claims are that AdaSplit preserves validity while improving power over random splitting via its adaptive allocation.","tokens_in":30050,"tokens_out":9830,"duration_ms":104608,"significance":"If the power claim can be properly isolated, the paper makes a useful contribution: it gives a rigorous finite-sample validity guarantee for a data-adaptive sample-splitting scheme in randomization tests, a practically relevant setting where cross-fitting and random splitting are standard. The conditional independence of null p-values (Theorem 2) is a nice property that enables simple combination methods such as Fisher's method in closed testing. The BaR-learner, which uses imputed assignments for held-out units, is an interesting and potentially reusable tool. The authors provide code for reproducibility, and the validity theorem is stated with a clear proof. However, the current experimental evidence for the power advantage is confounded with test-fold enrichment and fold-size differences, and the asymptotic power theory is derived in a regime that the algorithm deliberately leaves; these issues must be addressed before the central contribution is supported.","major_comments":[{"comment":"The headline power comparison in Figure 4 and Table 4 does not isolate the proposed allocation rule. The final step of Algorithm 1 (Section 3.3) removes from J_k the units with the most negative estimated CATEs before computing p-values, while the RT(RandomSplit) baseline in Section 4.1.3 tests a random 50/50 split and performs no such removal. The gain reported for AdaSplit could therefore be produced entirely by this final test-fold enrichment, independently of the certainty-score allocation or BaR-learner. To support the central claim, the authors should add an ablation that applies the same negative-CATE removal (and, ideally, the same final inference-fold size) to the random-split baseline, or otherwise vary only the allocation criterion while holding the test-fold composition fixed.","section":"Section 4.1.3 and Algorithm 1 (final step)"},{"comment":"The asymptotic power theory and the algorithm operate in incompatible regimes. Theorem 1 and Proposition 2 are proved under Assumption 5, which requires e(X_j,Y_j) ∈ [δ,1−δ] for all j in J_k, i.e., certainty scores bounded away from 1. The AdaSplit algorithm, by contrast, deliberately reserves for the inference fold the units with the highest certainty scores, for which e is close to 0 or 1 (see the discussion after Proposition 2 and Step 1 of Algorithm 1). The approximation in (11) therefore does not apply to the selected folds, so the theoretical motivation for the allocation rule is not established in the setting where AdaSplit is designed to operate. The authors should either extend the Berry–Esseen argument to accommodate e near the boundary or state explicitly that the power analysis is heuristic for the selected folds.","section":"Section 3.1 (Theorem 1, Assumption 5) and Section 3.3"},{"comment":"The real-data demonstration uses a binary outcome while the score is derived from the Gaussian model. The certainty score in (13) follows from the Gaussian homoscedastic model in Assumption 3, but the SPRINT primary outcome analysed in Section 4.2 is a binary indicator. The paper does not provide a misspecification analysis or a binary-outcome simulation, so the three rejected subgroups rest on a heuristic whose power and validity properties in this setting are unexamined. A binary-outcome simulation, or an argument for why the score remains an appropriate allocation criterion under non-Gaussian outcomes, is needed before the SPRINT result can be interpreted.","section":"Section 4.2 (SPRINT application) and Eq. (13)"},{"comment":"The experimental setup confounds the adaptive allocation with the size of the inference fold. Because AdaSplit's stopping rule can terminate before the nuisance proportion reaches the maximum ρ, it often tests on a larger inference fold than the 50/50 random-split baseline (Figure 5b). Since a larger test fold mechanically increases randomization test power, the advantage in Figure 4 could be partly due to the larger test fold rather than to the certainty-score allocation. The authors should match the final inference-fold sizes across methods, or condition on the realized π_k, to separate these effects.","section":"Section 4.1.3 and Figure 5"}],"minor_comments":[{"comment":"The while condition 'max{π[K]}≤ρ' in Algorithm 1 appears to be a typographical error: with an initial inference proportion near 1−p and ρ=0.5, the loop would never run. The text indicates the loop should continue while every subgroup still has inference proportion at least ρ, i.e., the condition should be min_k π_k ≥ ρ.","section":"Algorithm 1"},{"comment":"There is a typo, 'R-leanrer', which should be 'R-learner'.","section":"Section 3.2"},{"comment":"The final removal step is written as 'until π_k < ρ' in the pseudocode, but the text says the removal happens 'if ... π_k ≥ ρ after termination'. Please make the stopping boundary consistent: removal should stop when the inference proportion reaches ρ, not drops below it.","section":"Section 3.3 and Algorithm 1"},{"comment":"The statement that 'all methods control their type I errors at the nominal level 0.2' is based on Table 2, where the Monte-Carlo standard errors are about 0.04; a sentence clarifying how the reported values relate to the 0.2 level given this uncertainty would be helpful.","section":"Section 4.1.3"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of stat.ME and the validity result is a genuine strength. The main risk is that the power advantage is not yet convincingly isolated; I would ask the handling editor to ensure the revised version contains an experiment that holds the test-fold composition and size fixed and varies only the allocation mechanism, together with a discussion of the Assumption 5 boundary issue. If these are addressed, the paper could be a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper deserves a serious referee. The validity theorem is real, the adaptive splitting idea is genuinely new, and the BaR-learner is a clever touch. But the power comparison in Figure 4 is not a clean test of the allocation mechanism. Algorithm 1 ends by removing the most negative-CATE units from the inference fold; RT(RandomSplit) doesn't do that. So AdaSplit's smaller p-values may come largely from that final enrichment, not from the certainty-score ordering or the imputation step. The authors never run the ablation that would settle it: hold test-fold size, estimator, and selection rule fixed, and vary only the allocation criterion.\n\nWhat's actually new: the certainty score in (1), the proof that high-certainty units are better for testing and worse for estimation (Propositions 2 and 4), the BaR-learner with imputed assignments, and the conditional independence result for null p-values in Theorem 2. The validity argument is clean—it conditions on the selected nuisance fold and shows the null assignments are still uniform over the inference fold. That part holds up.\n\nSoft spots, in order of size. First, the confounded power comparison I just named. It's not a validity problem, but it means the abstract's 'more powerful' claim is overstated relative to what's demonstrated. Second, the theory behind the allocation rule assumes mu and tau are known and e(x,y) bounded away from 0 and 1; the greedy algorithm is a heuristic, and the high-certainty units the method most wants to test are exactly the ones violating Assumption 5. That's acceptable for a proposal but worth stating. Third, the certainty score comes from a Gaussian homoscedastic model, while the SPRINT application has a binary outcome. The authors apply the score anyway; it may work, but there is no misspecification analysis.\n\nBottom line: the central validity result is solid, and the ideas have legs. The paper needs an ablation and a bit more humility about where the power comes from. Who's it for? People building regression-assisted randomization tests and anyone doing confirmatory subgroup analysis in trials. I'd be happy to referee it, and I'd ask for the ablation as a major revision point.","headline":"Solid validity result and a genuinely new splitting idea, but the power comparison doesn't isolate the adaptive allocation from the final negative-CATE removal, so the headline gain is not as clean as claimed.","tokens_in":30440,"tokens_out":3186,"would_cite":true,"duration_ms":34429,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G09","62P10"],"pacs":[],"model":"deepseek-v4-flash","headline":"Randomization tests for subgroups gain power when sample splitting follows a certainty score that sends easy-to-predict units to testing and hard-to-predict units to estimation.","keywords":["adaptive sample splitting","randomization tests","subgroup analysis","conditional average treatment effect","certainty score","AIPW","finite-sample validity","family-wise error rate"],"falsifier":"Simulate the paper's default five-subgroup design with $n=500$ and a linear CATE, but draw $\\epsilon_i$ from a centered exponential or $t_3$ distribution so that Assumption 3 fails; if AdaSplit's mean subgroup p-values are not uniformly smaller than random splitting across 100 runs, the Gaussian model foundation of the allocation rule would be falsified. Alternatively, compute the exact conditional p-value in (9) for a small subgroup and compare it with the Gaussian formula in Theorem 1; a discrepancy larger than the claimed $O(|J_k|^{-1/2})$ would undermine the threshold allocation.","tokens_in":29517,"feed_emoji":"📊","tokens_out":13784,"duration_ms":130149,"temperature":0.7,"pith_summary":"Subgroup randomization tests in clinical trials are valid but often underpowered because each subgroup is small. The paper argues that a shared estimate of the conditional average treatment effect (CATE), fed into an augmented inverse-probability-weighted test statistic, can rescue power, provided the estimation and testing use separate units. It proposes AdaSplit, an adaptive sample-splitting rule that assigns each unit to estimation or testing according to a certainty score $C_i=|2e(X_i,Y_i)-1|$, the distance of the posterior probability of treatment from $1/2$. Units whose assignments are easy to predict are reserved for testing; units whose assignments are hard to predict are given to CATE estimation, where they matter most. The paper proves finite-sample Type I error control and conditional independence of null p-values, and shows in simulations and in a blood-pressure intervention trial that AdaSplit produces smaller subgroup p-values than random splitting or no splitting at all.","feed_headline":"Adaptive splitting makes subgroup randomization tests more powerful","feed_subtitle":"Units whose treatment arms are easy to predict go to testing; the rest train the treatment-effect model.","key_machinery":"The load-bearing object is the certainty score $C_i=|2e(X_i,Y_i)-1|$, where $e(X_i,Y_i)$ is the posterior probability that unit $i$ received treatment given its covariates and outcome; under the paper's Gaussian model this equals the sigmoid $\\sigma([Y_i-\\mu(X_i)]\\tau(X_i)/\\nu^2)$. Algorithm 1 starts with a small nuisance fold selected by leverage and moves units from inference to estimation one at a time, choosing the unit that minimizes $\\mathrm{sign}(\\hat\\tau(X_j))\\cdot \\hat C_j$, until the CATE predictions converge or each subgroup keeps at least a preset proportion $\\rho$ of its units. The testing side uses an augmented inverse-probability-weighted statistic built from estimated $\\mu$ and $\\tau$; the estimation side uses BaR-learner, which extends the residual-regression R-learner loss for CATE estimation by imputing treatment assignments for units whose outcomes make the assignment nearly certain. This allocation mechanism is what makes high-certainty units enlarge the observed statistic relative to the randomized reference distribution without inflating its variance.","core_discovery":"AdaSplit's central discovery is a division of labor: under the Gaussian outcome model $Y_i=\\mu(X_i)+(Z_i-1/2)\\tau(X_i)+\\epsilon_i$, the posterior assignment probability is $e(X_i,Y_i)=\\sigma([Y_i-\\mu(X_i)]\\tau(X_i)/\\nu^2)$, so the certainty score $C_i=|2e(X_i,Y_i)-1|$ is high exactly when the unit's treatment arm is imputable from covariates and outcome. Such high-certainty units dominate the mean-variance ratio of the AIPW statistic and should be kept in the inference fold; low-certainty units, whose assignments are needed to train the CATE estimator, should be put in the nuisance fold. The paper formalizes this tradeoff through a Gaussian approximation of the conditional p-value (Theorem 1), a threshold solution for the relaxed allocation (Proposition 2), and a BaR-learner that imputes held-out assignments with inverse-probability reweighting. Theorem 2 shows that the adaptive folds preserve finite-sample validity and make null p-values independent across subgroups, so closed testing with an exact combination method for independent p-values controls the family-wise error rate.","pith_inferences":["The qualitative split—hard-to-predict units fit the model, easy-to-predict units run the test—does not obviously require Gaussian errors; replacing the sigmoid score with a nonparametric estimate of $P(Z=1\\mid X,Y)$ is a natural robustness check, though the paper does not make this claim.","The same certainty-score logic transfers to other estimation-plus-testing pipelines, such as change-point localization or conditional independence testing, where the paper only sketches the direction.","The procedure can be viewed as active learning for hypothesis tests: it spends observed treatment assignments on the units whose labels most reduce CATE estimation error, the opposite of uncertainty sampling for classification; a formal comparison to uncertainty sampling would be an editorial extension, not a paper result."],"forward_implications":["Subgroup randomization tests can gain power from an estimated CATE without losing finite-sample validity, as long as the estimation fold is chosen adaptively but independently of the inference-fold assignments.","Because null p-values are conditionally independent across subgroups, closed testing can use an exact combination method for independent p-values, giving stronger family-wise error control than conservative union-style corrections.","The deterministic splitting rule makes the procedure reproducible and lets it stop early, so subgroups with strong effects keep a larger inference fold and hence smaller p-values.","BaR-learner's assignment imputation recovers most of the efficiency of fitting the CATE on the full sample, which is what lets the shared estimator help all subgroups at once.","In the paper's real-data analysis, the procedure identifies interpretable patient subgroups with significant treatment effects where difference-in-means and random-split AIPW tests reject none."],"supporting_citations":[{"why":"Supplies the residualized-statistic idea that motivates model-assisted randomization tests.","marker":"Rosenbaum [2002]"},{"why":"Gives the augmented inverse probability weighting formula used in the subgroup test statistic.","marker":"[Robins et al., 1994]"},{"why":"Defines the R-learner loss and residual transformation that BaR-learner extends with imputed assignments.","marker":"[Robinson, 1988, Nie and Wager, 2021]"},{"why":"Provides the inverse-probability reweighting that corrects selection bias in the nuisance fold.","marker":"[Horvitz and Thompson, 1952]"},{"why":"Supplies closed testing, the multiple-testing framework that benefits from independent null p-values.","marker":"[Marcus et al., 1976]"},{"why":"Supplies the partition-and-invariance argument used to prove validity of the adaptive randomization test.","marker":"[Zhang and Zhao, 2023]"},{"why":"Establishes cross-fitting as the standard alternative that AdaSplit is intended to improve on.","marker":"[Chernozhukov et al., 2018]"},{"why":"Provides the Berry-Esseen bounds used in the Gaussian approximation of the conditional p-value.","marker":"[Chen et al., 2010]"}],"fun_headline_variants":["AdaSplit: certainty-driven splitting for stronger subgroup tests","Certainty-based splits yield sharper subgroup randomization tests","Randomization tests get a power boost from certainty splits","Certainty score splits units to strengthen subgroup tests","Adaptive splitting by certainty sharpens subgroup inference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The allocation rule derives from a certainty-score formula that assumes Gaussian, homoscedastic outcome errors, so when that model is misspecified (as in the binary-outcome trial) the score is only heuristic and the power advantage over random splitting is not theoretically guaranteed.","fun_headline_variants_meta":{"raw":{"variants":["AdaSplit: certainty-driven splitting for stronger subgroup tests","Certainty-based splits yield sharper subgroup randomization tests","Randomization tests get a power boost from certainty splits","Certainty score splits units to strengthen subgroup tests","Adaptive splitting by certainty sharpens subgroup inference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001098,"raw_usage":{"total_tokens":4596,"prompt_tokens":972,"completion_tokens":3624,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":3551}},"tokens_in":588,"tokens_out":3624,"duration_ms":26560,"temperature":1.0,"reasoning_tokens":3551,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:00:16.201460+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate the paper's default five-subgroup design with $n=500$ and a linear CATE, but draw $\\epsilon_i$ from a centered exponential or $t_3$ distribution so that Assumption 3 fails; if AdaSplit's mean subgroup p-values are not uniformly smaller than random splitting across 100 runs, the Gaussian model foundation of the allocation rule would be falsified. Alternatively, compute the exact conditional p-value in (9) for a small subgroup and compare it with the Gaussian formula in Theorem 1; a discrepancy larger than the claimed $O(|J_k|^{-1/2})$ would undermine the threshold allocation.","supporting_citations":[{"cited_title":"Taking an expectation of both sides of the model conditional onXi, µ(Xi) = E[Yi|Xi] =µ0(Xi) +e(Xi)τ(Xi)","cited_arxiv_id":null,"evidence_quote":"Defines the R-learner loss and residual transformation that BaR-learner extends with imputed assignments."},{"cited_title":"[2010], we have sup t∈R P ˜ZJk n ˜V−1/2 k ˜Ak≤t o − Φ(t) ≤ C ˜V 3/2 k X j∈Jk ˆW 3 j E ˜Zj n | ˜Zj−e(Xj)|3 o =OP 1/ p |Jk|","cited_arxiv_id":null,"evidence_quote":"Provides the Berry-Esseen bounds used in the Gaussian approximation of the conditional p-value."}],"review_version":1}