REVIEW 3 major objections 5 minor 1 cited by
Demonstration Experiments
T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read This paper proves that two test statistics—a pooled statistic and a time-uniform max statistic—can test whether any arm's mean exceeds a threshold under nearly arbitrary adaptive sampling, provided each arm is sampled twice initially, and i
desk verdict Solid framework with a real gap between the advertised 'many arms' regime and the proven conditions. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The two load-bearing devices are (1) the regularized pooled statistic, which replaces unknown variances by padding or thresholding so that a martingale central-limit theorem gives a distribution-free normal limit regardless of the sampling strategy, and (2) a quantitative strong-coupling result that embeds each arm's sequential t-statistic in a Brownian path, so that a union-bound-corrected maximum over arms inherits the boundary-crossing probabilities of Brownian motion. The SN-UCB algorithm uses a self-normalized confidence interval for the signal-to-noise ratio μ_g/σ_g to allocate samples to promising arms, with a regret bound derived from that confidence bound.
What would settle it
Run the max-linear test under the null at (k,T)=(250,1000) (same k/T ratio as the paper's (50,200) case) and compare the empirical rejection rate to 0.05; the paper's Theorem 3 predicts validity only when T/[k log^3(T k)] → ∞, so a rejection rate far above 5% in this regime would falsify the advertised claim that the max tests are valid under general adaptive sampling with many arms.
Extended reading notes
Core claim
Under the null that all arm means are at or below their thresholds, the infeasible pooled statistic H_T is a supermartingale, and with regularized variance estimates (padding or thresholding) it satisfies a finite-sample Gaussian approximation with error O(k log^{3/2}(kT)/√T). Consequently the pooled test has asymptotic size α under the sharp null and is non-conservative. Independently, each arm's sequential t-statistic Ẑ_g(q) can, over q ∈ [T/k, T], be coupled uniformly to a Brownian motion; this yields a moderate-deviations bound on boundary crossing, which makes the max tests A_lin and A_log valid at level α for any sampling strategy satisfying the minimal two-samples-per-arm condition,
Load-bearing premise
The max tests' type-I error guarantee rests on the growth condition T/[k_T log^3(T k_T)] → ∞, so the number of arms must be much smaller than the horizon; when k is a sizable fraction of T—as in the paper's own (k=50, T=200) simulation, where the max-linear test rejects at 0.114 instead of 0.05—that premise fails.
Editorial extensions
If this is right
- Experimenters can use almost any adaptive allocation rule—subject only to drawing each arm twice upfront—and still get a level-α test of the global null that some arm's mean exceeds threshold.
- The pooled test is non-conservative under the sharp null and works well when several arms have moderate effects; the max test is conservative but allows the experimenter to stop early or peek without invalidating the test.
- Adaptive allocation can substantially improve power over uniform designs; SN-UCB specifically targets the signal-to-noise ratio that drives both test statistics, and its regret grows only logarithmically in the horizon.
- The moderate-deviations result for the sequential t-statistic extends time-uniform inference to settings where the number of hypotheses grows with T (sub-logarithmically in the stated theorem), which matters for many-armed bandit experiments.
- Since the max tests are based on infinite-horizon boundaries, they remain valid if the experimenter continues beyond the planned horizon while monitoring the data.
Reading between the lines
- The abstract advertises the 'number of arms large relative to sample size' regime, but Theorem 3 requires T/[k_T log^3(T k_T)] → ∞, i.e., k = o(T/log^3 T). In the paper's own simulation at k=50, T=200, the max-linear test's type-I error is 0.114, more than double the nominal 0.05; this suggests the advertised regime falls outside the theory, and experimenters with k comparable to T should prefer t
- The pooled test's empirical type-I error remains near 0.05 even at k=50, T=200 where the theoretical condition (k^2 log k/T → 0) fails; a sharper finite-sample analysis of the pooled statistic in the dense-arm regime would be a natural follow-up.
- The SN-UCB regret bound depends on the inverse gaps between arms; adapting the allocation to a gap-free or local-alternative setting would make the power guarantees more directly actionable for practitioners.
- Because power is expressed through the signal-to-noise ratio, other standard bandit exploration rules (e.g., probability-matching) could be redirected to that objective and likely enjoy similar guarantees, though the paper only proves the bound for SN-UCB.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper formalizes "demonstration experiments" as testing the global null that no arm's mean exceeds a threshold, under fully adaptive sampling in a multi-armed bandit. It proposes two families of tests: a pooled statistic that aggregates standardized arm sums, and max statistics based on time-uniform boundaries for individual sequential t-statistics. The main theoretical claims are a finite-sample CLT for the pooled statistic (Theorems 1-2), asymptotic validity of pooled testing (Corollary 1), type-I error control of the max tests under a growth condition on the number of arms (Theorem 3), and a logarithmic regret bound for the proposed SN-UCB allocation rule (Theorem 4). The paper also reports simulations comparing SN-UCB, standard UCB, Thompson sampling, uniform allocation, and an oracle on type-I error and power.
Significance. If the validity results hold, the paper makes a useful contribution to anytime-valid inference under adaptive sampling: it identifies test statistics that are robust to strategic allocation, and it connects experimental design to bandit optimization of signal-to-noise ratios. The proof machinery, combining martingale CLTs, self-normalized concentration, and Sakhanenko coupling, is substantive and clearly documented. The paper is also candid about several open problems. However, the advertised central regime — a "large number of arms relative to the sample size" — is not what the theorems deliver. The max-test theorem requires k_T to be much smaller than T, and its proof requires an extra logarithmic factor beyond the stated condition. The simulation table itself shows size inflation for the linear max test exactly in the k-comparable-to-T regime. These issues are load-bearing for the paper's main claims and need to be fixed before publication.
major comments (3)
- [Theorem 3 / Appendix A.3.2] The theorem's hypothesis T/[k_T log^3(T k_T)] -> infinity is not sufficient for the proof as written. In the proof of Theorem 3, r_T = w_alpha(k_T) and convergence of xi_T^2 = [r_T^2 log^3(k_T T) + r_T^8]/(T/k_T) is required. Since w_alpha(k_T) is of order sqrt(log k_T), this is k_T log^4(k_T T)/T -> 0, i.e. T/[k_T log^4(T k_T)] -> infinity. The stated condition allows k_T = T/log^4 T, for which T/(k_T log^3(T k_T)) ~ log T -> infinity but k_T log^4(T k_T)/T is bounded away from 0. Thus Theorem 3(i) is not established by the supplied argument; either strengthen the theorem's condition or extend the proof.
- [Abstract and Sections 1.1.2, 3.2] The advertised regime is not what is proved. Propositions 2 and 3 establish only one-sided upper bounds; the phrase "moderate-deviations principle" overstates these results. More importantly, the abstract and introduction describe the setting as one where the number of arms is large relative to the sample size, but Theorem 3 requires k_T = o(T/log^3 T) (and, after the correction above, o(T/log^4 T)). This excludes k comparable to T, which is exactly the regime where Table 1 shows the linear max test with type-I error 0.114 at k=50, T=200. Please revise the claims and explicitly state the sublinear-arm regime.
- [Section 1 vs Section 3.2, Eqs. (6)-(7)] The paper claims validity under "strategic termination" for the proposed statistics. Section 3.2 explicitly says the pooled test does not support early stopping. For the max tests, the events A_lin and A_log are defined as maxima over t>=1; at a finite stopping time T the experimenter cannot determine these events without observing the future. If the intended procedure is the finite-horizon clipped version (max over observed t<=T), this should be stated; if the experimenter may continue indefinitely, the stopping rule should be described. Otherwise the claim about strategic termination is ambiguous.
minor comments (5)
- [Eq. (7)] In the definition of A_log, the argument of \hat Z_g is N_g(T), but the outer maximum is over t. Presumably this should be N_g(t), matching A_lin.
- [Appendix A.3.2 / Appendix B.1] Lemma 24 is numbered twice: once in the proof of Proposition 2 and once in the confidence-bound section. Please renumber the later lemma.
- [Theorem 3, definition of z_alpha(k) and w_alpha(k)] The display defining z_alpha(k) and w_alpha(k) contains a stray "4." and writes 2k[1-Phi(z_alpha)] = k Psi_+(w_alpha) = alpha. Please clarify that z_alpha and w_alpha are indexed by k, and check that the linear boundary indeed uses the 2k Bonferroni factor.
- [Assumption 2 / Theorem 3] Theorem 3 uses the set K(t,zeta) = {g: N_g(t) >= zeta T/k}, while Propositions 2 and 3 are stated for q >= T/k (i.e., zeta=1). The proof does not explain how a general zeta enters the moderate-deviation bounds, even though simulations use zeta=2. Please state the role of zeta explicitly.
- [Future Work / Sections 3.1, 5.2] The Future Work section admits that the pooled statistic's strong empirical performance when k is large relative to T lacks theoretical justification. This caveat is important enough to appear in the introduction or abstract, since the current wording suggests the methods cover that regime.
Circularity Check
No significant circularity: the paper's validity claims rest on external analytic results and a priori tuning constants, not on fitted inputs or self-citations.
full rationale
The paper's central claims (Corollary 1 and Theorem 3) are derived analytically from martingale CLTs, moderate-deviations couplings, and boundary-crossing probabilities of Robbins and Siegmund, Sakhanenko, Chernozhukov et al., Fan et al., and Waudby-Smith et al. These are external, previously established results with no author overlap with the present paper. The test statistics and critical values are defined by closed-form formulas (e.g., $z_\alpha(k)$ and $w_\alpha(k)$ solve $2k[1-\Phi(z_\alpha)] = k\Psi_+(w_\alpha)=\alpha$) rather than calibrated to data; tuning constants such as $\lambda_{k,T}=\sqrt{\log(kT)}$, $\rho_{k,T}=C\nu\log(kT)$, and $\zeta$ are chosen a priori. No parameter is fitted to a subset of the data and then renamed a prediction. The pooled-statistic supermartingale property (Lemma 1) and the Gaussian approximation bounds are proved from the stated assumptions, not assumed by construction. The simulation study is an external check, and the type-I error inflation at $(k,T)=(50,200)$ is explicitly attributed to the known breakdown of the Gaussian approximation in that regime, which is a correctness/robustness concern rather than a circularity. The proof of Theorem 3 may require a slightly stronger growth condition than stated, but that is a gap or error in the proof, not a reduction of the result to its own inputs. No load-bearing step in the derivation chain is equivalent by construction to an input, and there are no self-citations used to justify the main results.
Assumptions & free parameters
free parameters (5)
- lambda (padding regularization) =
sqrt(log(kT))
- rho (threshold regularization) =
Cν log(kT)
- beta (SN-UCB exploration exponent) =
β>2 (unspecified)
- zeta (max-test sample cutoff) =
ζ=2 in simulations
- nu (sub-Gaussian scale) =
unknown/assumed
assumptions (4)
- domain assumption Assumption 1: i.i.d. potential outcome vectors across rounds
- domain assumption Assumption 2: each arm is sampled exactly twice at the outset
- domain assumption Assumption 3: sub-Gaussian tails with parameter νσ_g
- standard math Moderate-deviations upper bounds use Sakhanenko's quantitative invariance principle
Cite this review
Pith. "Pith review of Demonstration Experiments." pith.science (2026). https://pith.science/paper/7PODKSE2
@misc{pith2026260306941,
author = {Pith},
title = {Pith review of: Demonstration Experiments},
year = {2026},
howpublished = {\url{https://pith.science/paper/7PODKSE2}},
note = {Machine review of arXiv:2603.06941}
}
abstract
Adaptive experiments are used extensively in online platforms, healthcare and biotechnology, and the social sciences. Often, the primary goal is not to precisely estimate a treatment effect but to demonstrate that at least one candidate intervention yields a positive effect, for some subpopulation and on some measured outcome. We formalize this objective as testing the global null in a threshold bandit framework, and develop two inference procedures that are valid under general adaptive sampling: one that pools information across promising arms, and one based on time-uniform multiple testing of individual arm means. To support the latter, we establish a moderate-deviations principle for the sequential $t$-statistic, justifying asymptotic confidence sequences in settings where the number of arms is large relative to the sample size. To illustrate how adaptive designs can target the proposed statistics, we recast experimental design as bandit optimization with an arm's reward given by its signal-to-noise ratio, and analyze an allocation rule for which we establish a logarithmic regret bound. We apply the methods in a simulation study of targeting unconditional cash transfer programs.
Figures
Forward citations
Cited by 1 Pith paper
-
Confidence Horizons
A new family of 'asymptotic confidence horizons' provides large-sample anytime-valid coverage on bounded time windows, with closed-form boundary quantiles and connections to group sequential methods.
Reference graph
Works this paper leans on
-
[1]
Exploration--exploitation tradeoff using variance estimates in multi-armed bandits
Jean-Yves Audibert, R \'e mi Munos, and Csaba Szepesv \'a ri. Exploration--exploitation tradeoff using variance estimates in multi-armed bandits. Theoretical Computer Science, 410 0 (19): 0 1876--1902, 2009
1902
-
[2]
Finite-time analysis of the multiarmed bandit problem
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47 0 (2): 0 235--256, 2002
2002
-
[3]
Regret analysis of stochastic and nonstochastic multi-armed bandit problems
S \'e bastien Bubeck, Nicolo Cesa-Bianchi, et al. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends in Machine Learning , 5 0 (1): 0 1--122, 2012
2012
-
[4]
Sequential testing for elicitable functionals via supermartingales
Philippe Casgrain, Martin Larsson, and Johanna Ziegel. Sequential testing for elicitable functionals via supermartingales. Bernoulli, 30 0 (2): 0 1347--1374, 2024
2024
-
[5]
Statistical inference for online decision making: In a contextual bandit setting
Haoyu Chen, Wenbin Lu, and Rui Song. Statistical inference for online decision making: In a contextual bandit setting. Journal of the American Statistical Association, 116 0 (533): 0 240--255, 2021
2021
-
[6]
Optimal conditional inference in adaptive experiments
Jiafeng Chen and Isaiah Andrews. Optimal conditional inference in adaptive experiments. arXiv preprint arXiv:2309.12162, 2023
arXiv 2023
-
[7]
Empirical and multiplier bootstraps for suprema of empirical processes of increasing complexity, and related G aussian couplings
Victor Chernozhukov, Denis Chetverikov, and Kengo Kato. Empirical and multiplier bootstraps for suprema of empirical processes of increasing complexity, and related G aussian couplings. Stochastic Processes and their Applications, 126 0 (12): 0 3632--3651, 2016
2016
-
[8]
Brownian Motion, chapter 8, pages 379--441
Erhan C nlar. Brownian Motion, chapter 8, pages 379--441. Springer New York, New York, NY, 2011. ISBN 978-0-387-87859-1
2011
Show all 34 references
-
[9]
de la Pe \ n a, Michael J
Victor H. de la Pe \ n a, Michael J. Klass, and Tze Leung Lai. Self-normalized processes: exponential inequalities, moment bounds and iterated logarithm laws . The Annals of Probability, 32 0 (3): 0 1902 -- 1933, 2004. doi:10.1214/009117904000000397. URL https://doi.org/10.121...
1902 doi
-
[10]
Self-normalized processes: Limit theory and Statistical Applications
Victor H de la Pe \ n a, Tze Leung Lai, and Qi-Man Shao. Self-normalized processes: Limit theory and Statistical Applications. Springer, 2009
2009
-
[11]
Distances of probability measures and random variables
RM Dudley. Distances of probability measures and random variables. The Annals of Mathematical Statistics, 39 0 (5): 0 1563--1572, 1968
1968
-
[12]
Cram \'e r large deviation expansions for martingales under B ernstein’s condition
Xiequan Fan, Ion Grama, and Quansheng Liu. Cram \'e r large deviation expansions for martingales under B ernstein’s condition. Stochastic Processes and their Applications, 123 0 (11): 0 3919--3942, 2013
2013
-
[13]
Confidence intervals for policy evaluation in adaptive experiments
Vitor Hadad, David A Hirshberg, Ruohan Zhan, Stefan Wager, and Susan Athey. Confidence intervals for policy evaluation in adaptive experiments. Proceedings of the National Academy of Sciences, 118 0 (15): 0 e2014602118, 2021
2021
-
[14]
Martingale limit theory and its application
Peter Hall and Christopher C Heyde. Martingale limit theory and its application. Academic press, 2014
2014
-
[15]
Tight bounds for the expected risk of linear classifiers and pac-bayes finite-sample guarantees
Jean Honorio and Tommi Jaakkola. Tight bounds for the expected risk of linear classifiers and pac-bayes finite-sample guarantees. In Artificial Intelligence and Statistics, pages 384--392. PMLR, 2014
2014
-
[16]
Sequential estimation of quantiles with applications to A / B testing and best-arm identification
Steven R Howard and Aaditya Ramdas. Sequential estimation of quantiles with applications to A / B testing and best-arm identification. Bernoulli, 28 0 (3): 0 1704--1728, 2022
2022
-
[17]
Time-uniform C hernoff bounds via nonnegative supermartingales
Steven R Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform C hernoff bounds via nonnegative supermartingales. 2020
2020
-
[18]
Sequential test for the lowest mean: From T hompson to M urphy sampling
Emilie Kaufmann, Wouter M Koolen, and Aur \'e lien Garivier. Sequential test for the lowest mean: From T hompson to M urphy sampling. Advances in Neural Information Processing Systems, 31, 2018
2018
-
[19]
An experimental design for anytime-valid causal inference on multi-armed bandits
Biyonka Liang and Iavor Bojinov. An experimental design for anytime-valid causal inference on multi-armed bandits. arXiv preprint arXiv:2311.05794, 2023
2023 arXiv
-
[20]
An optimal algorithm for the thresholding bandit problem
Andrea Locatelli, Maurilio Gutzeit, and Alexandra Carpentier. An optimal algorithm for the thresholding bandit problem. In International Conference on Machine Learning, pages 1690--1698. PMLR, 2016
2016
-
[21]
On the rate of convergence in the martingale central limit theorem
Jean-Christophe Mourrat. On the rate of convergence in the martingale central limit theorem . Bernoulli, 19 0 (2): 0 633 -- 645, 2013. doi:10.3150/12-BEJ417. URL https://doi.org/10.3150/12-BEJ417
2013 doi
-
[22]
Game-theoretic statistics and safe anytime-valid inference
Aaditya Ramdas, Peter Gr \"u nwald, Vladimir Vovk, and Glenn Shafer. Game-theoretic statistics and safe anytime-valid inference. Statistical Science, 38 0 (4): 0 576--601, 2023
2023
-
[23]
Boundary crossing probabilities for the W iener process and sample sums
Herbert Robbins and David Siegmund. Boundary crossing probabilities for the W iener process and sample sums. The Annals of Mathematical Statistics, pages 1410--1429, 1970
1970
-
[24]
Rate of convergence in the invariance principle for variables with exponential moments that are not identically distributed
Aleksandr Ivanovich Sakhanenko. Rate of convergence in the invariance principle for variables with exponential moments that are not identically distributed. Matematicheskie Trudy, 3: 0 4--49, 1984
1984
-
[25]
Testing by betting: A strategy for statistical and scientific communication
Glenn Shafer. Testing by betting: A strategy for statistical and scientific communication. Journal of the Royal Statistical Society Series A: Statistics in Society, 184 0 (2): 0 407--431, 2021
2021
-
[26]
Strong approximation theorems for independent random variables and their applications
Qi-Man Shao. Strong approximation theorems for independent random variables and their applications. Journal of multivariate analysis, 52 0 (1): 0 107--130, 1995
1995
-
[27]
Nonparametric two-sample testing by betting
Shubhanshu Shekhar and Aaditya Ramdas. Nonparametric two-sample testing by betting. IEEE Transactions on Information Theory, 70 0 (2): 0 1178--1203, 2023
2023
-
[28]
Multi-armed bandit experimental design: Online decision-making and adaptive inference
David Simchi-Levi and Chonghuan Wang. Multi-armed bandit experimental design: Online decision-making and adaptive inference. In International Conference on Artificial Intelligence and Statistics, pages 3086--3097. PMLR, 2023
2023
-
[29]
On the likelihood that one unknown probability exceeds another in view of the evidence of two samples
William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25 0 (3/4): 0 285--294, 1933
1933
-
[30]
High-Dimensional Probability: An Introduction with Applications in Data Science
Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2 edition, 2026
2026
-
[31]
High-dimensional statistics: A non-asymptotic viewpoint, volume 48
Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge University Press, 2019
2019
-
[32]
Distribution-uniform anytime-valid sequential inference
Ian Waudby-Smith, Edward H Kennedy, and Aaditya Ramdas. Distribution-uniform anytime-valid sequential inference. arXiv preprint arXiv:2311.03343, 2023
2023
-
[33]
Time-uniform central limit theory and asymptotic confidence sequences
Ian Waudby-Smith, David Arbour, Ritwik Sinha, Edward H Kennedy, and Aaditya Ramdas. Time-uniform central limit theory and asymptotic confidence sequences. The Annals of Statistics, 52 0 (6): 0 2613--2640, 2024
2024
-
[34]
The accuracy of strong gaussian approximation for sums of independent random vectors
A Yu Zaitsev. The accuracy of strong gaussian approximation for sums of independent random vectors. Russian Mathematical Surveys, 68 0 (4): 0 721, 2013
2013
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.