Pith. sign in

REVIEW 2 major objections 4 minor 128 references

When and How to Pilot: Design Rules for Two-Wave Experiments

T0 review · 2 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper's central claim is that pilot-based assignment should be governed by a Conditional Minimax Regret rule that acts on the confidence set for the variances rather than on point estimates, and that this rule delivers finite-sample cer

desk verdict A well-motivated, clearly-derived decision rule for pilot-based assignment, with one load-bearing concentration inequality that is cited rather than proved. read the letter →

arxiv 2607.16982 v1 pith:7ODJW3H5 submitted 2026-07-18 econ.EM

Juan C. Yamin This is my paper · ORCID
classification econ.EM MSC 62C2062K0562F25
keywords pilotstudyadaptiveexperimentaldesignNeymanallocationminimaxregretconfidencerectangletwo-waveexperimentstreatmentassignmentprobabilityfinite-sampledecisiontheory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper addresses a practical question in two-wave experiments: when a small pilot estimates the treatment and control variances, how much should the main-wave assignment probability tilt away from 50-50? It argues that the two textbook answers are wrong in finite samples: balance ignores the pilot, while plugging pilot variances into the Neyman allocation can overreact and produce arbitrarily large precision losses. The proposed middle path, the Conditional Minimax Regret (CMR) rule, builds a finite-sample confidence rectangle for the two variances and chooses the treatment share that minimizes worst-case regret over that rectangle. CMR stays at balance when the pilot is uninformative, converges to the Neyman allocation as the pilot grows, and comes with a certificate bounding the realized regret with probability at least 1−α. A reader should care because the paper supplies a computable design rule with both finite-sample safety and asymptotic efficiency, plus guidance on when a pilot is worth running at all.

What carries the argument

The central object is a 1−α confidence rectangle for the pair (treatment variance, control variance), built by Bonferroni-combining two one-sided finite-sample bounds from an empirical Bernstein inequality for each arm's standard deviation. The regret identity r(π,θ)=((1−π)σ1−πσ0)^2/[π(1−π)] makes regret affine in the standard deviations, so the worst case over the rectangle occurs at one of two off-diagonal corners; the minimizer is the Neyman formula applied to the midpoints of the two standard-deviation intervals, and the certificate is the common regret at those two corners. This rectangle-plus-corner structure is what turns an intractable minimax problem over distributions into a closed

What would settle it

Run a Monte Carlo with M=30 (15 per arm) from a Bernoulli(p=0.9) outcome, compute the claimed 95% rectangle for 10,000 pilots, and check whether the true variance pair is inside at least 95% of draws. If coverage falls below 95%, the paper's Theorem 4.1 certificate does not hold for that distribution. A more direct check is to verify empirically whether each one-sided standard-deviation bound holds with probability at least 1−α/4.

Watch

Extended reading notes

Core claim

The paper establishes that the right response to pilot data is neither to ignore it nor to follow it blindly, but to act only on what the pilot has not ruled out. Concretely, CMR forms a 1−α confidence rectangle for the pair of arm variances, then picks the main-wave treatment share that minimizes worst-case regret over that rectangle. In the two-arm case the rule has a closed form: apply the Neyman formula to the midpoints of the two standard-deviation confidence intervals, with a certificate equal to the common worst-case regret at the two off-diagonal corners. The paper proves that this rule never has worse certificate than balance, converges to the infeasible Neyman allocation at the inv

Load-bearing premise

Everything rests on the empirical Bernstein inequality that gives one-sided confidence bounds for each arm's standard deviation with radius sqrt(2 log(1/b)/(M_d−1)); if that concentration radius is not valid for the outcome distribution, the 1−α rectangle coverage fails and the certificate guarantee collapses.

Editorial extensions

If this is right

  • Balanced assignment remains the right default for very small pilots: below a threshold (e.g., 72 observations at α=0.05 with the baseline bounds), CMR returns exactly balance, so a pilot cannot hurt under the pooled estimator and is strictly wasteful if the pilot data are discarded.
  • When the pilot is informative, CMR captures most of the precision gain of full adaptation while avoiding feasible Neyman's boundary failures; in simulations calibrated to four field experiments, CMR's efficiency loss stayed at or near balance's value at small pilots and fell to near-zero as pilots grew.
  • CMR's worst-case expected regret decays at the same M^{-1/2} rate as the exact minimax-regret value, so no pilot-based rule can beat it in rate, and at interior variance pairs both assignment error and realized regret shrink at M^{-1}.
  • The rule extends to multi-arm shared-control and stratified designs by replacing the scalar assignment with an allocation vector and solving the same rectangle-minimax problem as a finite convex program.
  • The paper's pilot-size analysis says a budget's two-thirds power is approximately optimal: a balanced pilot of that size followed by CMR attains the best possible worst-case rate n^{-4/3}, and any near-optimal design must size its pilot in that way.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because the rule's conservatism lives entirely in the confidence set, using tighter or model-specific sets (e.g., exact binomial inversion for binary outcomes) makes CMR activate at much smaller pilots; a natural extension is to choose the confidence construction itself as part of the design problem, trading coverage sharpness against pilot cost.
  • Beyond the paper: the same rectangle-minimax recipe should carry over to cluster-randomized, sequential, or imperfect-compliance experiments whenever the design loss is convex in a low-dimensional variance vector; the paper names these as open directions but does not develop them.
  • Beyond the paper: an experimenter with a prior on the variance pair could plausibly beat CMR at small pilots, since CMR hedges against the worst surviving configuration rather than the most likely one; the paper does not compare against Bayesian or shrinkage rules.
  • Beyond the paper: the break-even pilot-size bound suggests a practical heuristic—if the planned pilot share exceeds (σ1−σ0)^2/[2(σ1^2+σ0^2)] using planning values, the pilot is unlikely to repay its cost under design-only accounting; this can be checked before running.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper studies two-wave experiments in which a pilot sample is used to choose the main-wave treatment assignment probability. It formalizes the problem as a finite-sample decision problem with regret relative to the infeasible Neyman allocation, shows that balanced assignment is minimax-risk optimal but ignores the pilot, and that feasible Neyman allocation adapts but has unbounded worst-case risk. The proposed Conditional Minimax Regret (CMR) rule builds a finite-sample confidence rectangle for the two arm variances, then chooses the assignment minimizing worst-case regret over that rectangle. Section 4 derives a closed-form assignment and certificate, proves finite-sample coverage of the certificate, Neyman convergence at rate M^{-1/2} in assignment and M^{-1} in regret, and a worst-case expected regret bound matching the minimax-regret lower bound up to constants. Section 5 extends the construction to multi-arm shared-control and stratified designs; Appendix D treats binary, unbounded, multiple, and delayed outcomes; Appendix E analyzes whether and how large a pilot should be, proposing an n^{2/3} pilot size rule; Section 6 reports simulations calibrated to four published field experiments. The central claims are that CMR is closed-form, finite-sample safe, asymptotically efficient at interior points, and minimax-rate optimal.

Significance. If the results hold, this is a substantial contribution to the design of pilot-based experiments. The paper does not fit parameters or rely on simulation-tuned constants: the rule is derived from a minimax regret problem over a confidence rectangle, and the main guarantees are theorems rather than heuristics. The certificate idea, in the spirit of Andrews and Chen, gives experimenters a finite-sample bound on the precision loss of the chosen design. The convergence and minimax-rate results are derived with proofs, and the extensions to multi-arm, stratified, binary, unbounded, multiple, and delayed outcomes make the framework broadly applicable. The calibrated simulations are a genuine strength: they use published microdata, show realistic pilot sizes, and document that feasible Neyman can have infinite mean loss while CMR remains near balance. The practical message — that the upside of adaptation is modest in two-arm designs but the downside of plug-in rules is large — is clearly supported.

major comments (2)
  1. [§4.2, Proposition 4.1] Proposition 4.1 assumes strict positivity of both lower standard-deviation endpoints, σbar_1>0 and σbar_0>0. But the endpoint construction in §4.3, eq. (4.6), yields σbar_d = (σhat_d − η_d(b))_+, which is zero with positive probability whenever a pilot arm is homogeneous (e.g., binary outcomes all equal). The closed form (4.4) and certificate (4.5) are used for every realized rectangle, including these zero-lower-endpoint cases, and Theorem 4.1 and the simulations rely on that use. The paper should either extend Proposition 4.1 to weak inequalities, or explicitly state that the formula is extended by continuity and verify, with a proof, that the extended formula still solves (4.2). As written, the implemented rule is not exactly the rule covered by the proposition at pilot realizations where its hypothesis fails.
  2. [§4.3, Lemma 4.1] All finite-sample guarantees — Theorem 4.1(i), Theorem 4.2, Theorem 4.3 — rest on the exact two one-sided inequalities Pr(σ_d ≤ σhat_d + η_d(b)) ≥ 1−b and Pr(σ_d ≥ σhat_d − η_d(b)) ≥ 1−b with η_d(b) = sqrt(2 log(1/b)/(M_d−1)). The proof cites 'Theorem 10 of Maurer and Pontil (2009)' without reproducing the theorem or its normalization. Because the radius and the denominator M_d−1 are load-bearing, the paper should state the cited theorem verbatim and verify the translation to the unbiased sample variance V_Md; a small mismatch in constants would invalidate Theorem 4.1. I checked the cited result and it does give the asserted one-sided bounds, so this is a documentation and verification issue rather than an identified error, but it should be fixed for the paper to be self-contained.
minor comments (4)
  1. [§4, notation] The lower and upper standard-deviation endpoints are denoted with overline/underline variants of σ, which is visually easy to confuse, especially in Proposition 4.1 and the proof of Theorem 4.1. The paper should introduce distinct symbols or a clear typographical convention once in §4.1 and use them consistently.
  2. [§4.3, Lemma 4.1] The paragraph before eq. (4.6) says the Maurer–Pontil inequality gives the two one-sided statements 'uniformly over F∈F'. It would help to state explicitly that this uniformity is over the class of [0,1]-bounded distributions and that the projection onto [0,1/4] does not break the one-sided coverage, as is done in the proof but not in the main text.
  3. [§6, Tables and Figures] Table C.2 reports coverage as 100.0% in every design and pilot size. Since 500 replications cannot establish exact 100% coverage, it would be useful to report Monte Carlo standard errors or at least state that coverage is so far above the nominal 95% level that the rounding to 100.0% is not informative about the actual finite-sample coverage of the distribution-free rectangle.
  4. [Appendix D.1] The exact-inversion folded-binomial construction for binary outcomes is a valuable refinement. The paper should emphasize in the main text that the binary CMR becomes informative with as few as four pilot observations, whereas the Maurer–Pontil rectangle requires much larger pilots; this contrast is currently buried in Appendix D.1 and Appendix E.5.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all central claims are derived from pre-specified external concentration bounds and the paper's own finite-sample decision problem, not from fitted inputs or self-citations.

full rationale

The derivation chain is self-contained against external benchmarks. The CMR rule is defined as the minimizer of worst-case regret over a confidence rectangle for the two arm variances; Proposition 4.1 derives the closed form from the regret identity and the corner geometry, not from an assumed conclusion. The finite-sample certificate (Theorem 4.1) follows from the rectangle's coverage, which in turn rests on Lemma 4.1 and the cited Maurer-Pontil empirical Bernstein bound: 'By Theorem 10 of Maurer and Pontil (2009)' — an external, machine-verifiable theorem, not a self-citation. No fitted constants enter: alpha=0.05 and the concentration radius eta_d(b) are pre-specified, and the infeasible Neyman benchmark is defined independently. The convergence and regret-rate results (Theorems 4.2-4.3) are proved from the contraction of the rectangle, and the simulation DGPs are calibrated to published field microdata. The one concern raised by the skeptic — that the exact form of the Maurer-Pontil radius might differ, e.g., require an empirical-variance term — is a correctness risk about an external input fact, not a circularity: the paper does not define the radius in terms of the target result, nor does it fit the radius to make the certificate hold. The only self-reference is the rule's own definition, which is definitional rather than circular. Accordingly, no circular step can be exhibited, and the score is 0.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claims rest on: (1) the maintained bounded-outcome model with i.i.d. sampling in pilot and main wave; (2) the validity of the empirical-Bernstein concentration inequality used to build the confidence rectangle; (3) the variance-of-difference-in-means objective. These are standard domain assumptions plus a cited mathematical result. There are no invented entities; the only user-chosen input is the coverage level α (set to 0.05 in simulations), which is not fitted to data.

free parameters (1)
  • α (coverage level) = 0.05 (default in simulations; user-chosen)
    Controls the width of the confidence rectangle and hence how quickly CMR moves away from balance. Not fitted to data; a standard confidence level. All theorems are stated for fixed α∈(0,1).
assumptions (5)
  • domain assumption Potential outcomes take values in [0,1] (Y(d)∈[0,1] for d∈{0,1}), so all variances are bounded by 1/4.
    Maintained throughout Sections 2–6; relaxed via kurtosis bound in Appendix D.2.
  • domain assumption Pilot and main-wave units are i.i.d. draws from the same population distribution F; pilot uses CRD with fixed arm sizes M_d ≥ 2.
    Section 2.1; if the pilot population differs from the main wave, the variance estimates are irrelevant to the design loss.
  • domain assumption The design objective is the variance of the difference-in-means ATE estimator, V(π, σ_1^2, σ_0^2) = σ_1^2/π + σ_0^2/(1−π).
    Section 2.2; the whole regret framework is built on this criterion.
  • standard math Maurer–Pontil empirical Bernstein inequality: for bounded random variables, Pr(σ_d ≤ σ̂_d + η_d(b)) ≥ 1−b and Pr(σ_d ≥ σ̂_d − η_d(b)) ≥ 1−b with η_d(b)=sqrt(2 log(1/b)/(M_d−1)).
    Lemma 4.1 and eq. (4.6). The paper cites Theorem 10 of Maurer and Pontil (2009); this is the key external mathematical input.
  • standard math Standard decision-theoretic criteria: minimax risk, minimax regret, and the no-pilot benchmark.
    Sections 2.3–3; used to define balance as the no-pilot minimax-regret action (Prop. 3.3(i)).

how reviews work

0 comments
Cite this review

Pith. "Pith review of When and How to Pilot: Design Rules for Two-Wave Experiments." pith.science (2026). https://pith.science/paper/7ODJW3H5

@misc{pith2026260716982,
  author       = {Pith},
  title        = {Pith review of: When and How to Pilot: Design Rules for Two-Wave Experiments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7ODJW3H5}},
  note         = {Machine review of arXiv:2607.16982}
}
read the original abstract

Experimenters often run pilots, but how much a small pilot should shape the main-wave design has no settled answer. This paper shows how noisy pilot evidence should guide treatment assignment probabilities in two-wave experiments. Two canonical rules mark the extremes. Balanced assignment guards against worst cases but ignores evidence that one arm is noisier. Feasible Neyman allocation adapts, but with a finite pilot it can overreact to noise, producing arbitrarily large precision losses. We propose a Conditional Minimax Regret (CMR) rule that minimizes worst-case regret over a finite-sample confidence set for the treatment and control variances. CMR retains balance's worst-case protection with high probability, converges to the Neyman allocation as the pilot grows, and attains the minimax-regret rate up to constants. It extends to multi-arm and stratified designs, and simulations calibrated to four field experiments show it avoids feasible Neyman's severe small-pilot losses while capturing most of its large-pilot gains.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

128 extracted references · 21 canonical work pages

  1. [1]

    and Chan, Claire L

    Eldridge, Sandra M. and Chan, Claire L. and Campbell, Michael J. and Bond, Christine M. and Hopewell, Sally and Thabane, Lehana and Lancaster, Gillian A. , title =. BMJ , year =

  2. [3]

    Comment on ``An Essay on the Logical Foundations of Survey Sampling, Part One'' by

    H. Comment on ``An Essay on the Logical Foundations of Survey Sampling, Part One'' by. Foundations of Statistical Inference , editor =

  3. [4]

    Efficient estimation of average treatment effects using the estimated propensity score , year =

    Hirano, Keisuke and Imbens, Guido W and Ridder, Geert , journal =. Efficient estimation of average treatment effects using the estimated propensity score , year =

  4. [5]

    On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection , year =

    Neyman, Jerzy , journal =. On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection , year =

  5. [10]

    On the method of bounded differences , year =

    McDiarmid, Colin and others , journal =. On the method of bounded differences , year =

  6. [11]

    Empirical

    Maurer, Andreas and Pontil, Massimiliano , booktitle =. Empirical. 2009 , eprint =

    2009

  7. [14]

    Econometric Theory , title =

    Stoye, J. Econometric Theory , title =. 2007 , number =

    2007

  8. [15]

    How to Minimize Maximum Regret in Repeated Decision\# Making , year =

    Schlag, Karl H , journal =. How to Minimize Maximum Regret in Repeated Decision\# Making , year =

Show all 128 references
  1. [16]

    Who should be treated? empirical welfare maximization methods for treatment choice , year =

    Kitagawa, Toru and Tetenov, Aleksey , journal =. Who should be treated? empirical welfare maximization methods for treatment choice , year =

  2. [17]

    Minimax-regret treatment choice with missing outcome data , year =

    Manski, Charles F , journal =. Minimax-regret treatment choice with missing outcome data , year =

  3. [19]

    Minimax-regret treatment rules with many treatments , year =

    Masten, Matthew A , journal =. Minimax-regret treatment rules with many treatments , year =

  4. [20]

    Asymptotics for statistical treatment rules , year =

    Hirano, Keisuke and Porter, Jack R , journal =. Asymptotics for statistical treatment rules , year =

  5. [22]

    Statistical decision rules in econometrics , year =

    Hirano, Keisuke and Porter, J , journal =. Statistical decision rules in econometrics , year =

  6. [23]

    Policy Learning with Confidence , year =

    Chernozhukov, Victor and Lee, Sokbae and Rosen, Adam M and Sun, Liyang , journal =. Policy Learning with Confidence , year =

  7. [26]

    2021 , publisher =

    Giacomini, Raffaella and Kitagawa, Toru and Read, Matthew , title =. 2021 , publisher =

    2021

  8. [28]

    Gambling, Saving, and Lumpy Liquidity Needs , year =

    Herskowitz, Sylvan , journal =. Gambling, Saving, and Lumpy Liquidity Needs , year =. doi:10.1257/app.20180177 , publisher =

    doi

  9. [29]

    Bridging the Intention-Behavior Gap?

    Abel, Martin and Burger, Rulof and Carranza, Eliana and Piraino, Patrizio , journal =. Bridging the Intention-Behavior Gap?. 2019 , number =. doi:10.1257/app.20170566 , publisher =

    2019 doi

  10. [31]

    Medium- and Long-Term Educational Consequences of Alternative Conditional Cash Transfer Designs: Experimental Evidence from

    Barrera-Osorio, Felipe and Linden, Leigh L and Saavedra, Juan E , journal =. Medium- and Long-Term Educational Consequences of Alternative Conditional Cash Transfer Designs: Experimental Evidence from. 2019 , number =. doi:10.1257/app.20170008 , publisher =

    2019 doi

  11. [32]

    Labor Drops: Experimental Evidence on the Return to Additional Labor in Microenterprises , year =

    De Mel, Suresh and McKenzie, David and Woodruff, Christopher , journal =. Labor Drops: Experimental Evidence on the Return to Additional Labor in Microenterprises , year =. doi:10.1257/app.20170497 , publisher =

    doi

  12. [33]

    Leader Selection and Service Delivery in Community Groups: Experimental Evidence from

    Deserranno, Erika and Stryjan, Miri and Sulaiman, Munshi , journal =. Leader Selection and Service Delivery in Community Groups: Experimental Evidence from. 2019 , number =. doi:10.1257/app.20180248 , publisher =

    2019 doi

  13. [34]

    One in a Million: Field Experiments on Perceived Closeness of the Election and Voter Turnout , year =

    Gerber, Alan and Hoffman, Mitchell and Morgan, John and Raymond, Collin , journal =. One in a Million: Field Experiments on Perceived Closeness of the Election and Voter Turnout , year =. doi:10.1257/app.20180574 , publisher =

    doi

  14. [35]

    American Economic Journal: Applied Economics , title =

    Himmler, Oliver and J. American Economic Journal: Applied Economics , title =. 2019 , number =. doi:10.1257/app.20170288 , publisher =

    2019 doi

  15. [36]

    American Economic Journal: Applied Economics , title =

    Lafortune, Jeanne and Riutort, Julio and Tessada, Jos. American Economic Journal: Applied Economics , title =. 2018 , number =. doi:10.1257/app.20170077 , publisher =

    2018 doi

  16. [37]

    Poverty and Migration in the Digital Age: Experimental Evidence on Mobile Banking in

    Lee, Jean N and Morduch, Jonathan and Ravindran, Saravana and Shonchoy, Abu and Zaman, Hassan , journal =. Poverty and Migration in the Digital Age: Experimental Evidence on Mobile Banking in. 2021 , number =. doi:10.1257/app.20190067 , publisher =

    2021 doi

  17. [40]

    Higbee, Samuel D , title =

  18. [41]

    Minimax-Regret Sample Selection in Randomized Experiments , year =

    Hu, Yuchen and Zhu, Henry and Brunskill, Emma and Wager, Stefan , booktitle =. Minimax-Regret Sample Selection in Randomized Experiments , year =

  19. [42]

    Optimal data collection for randomized control trials , year =

    Carneiro, Pedro and Lee, Sokbae and Wilhelm, Daniel , journal =. Optimal data collection for randomized control trials , year =

  20. [43]

    So you want to run an experiment, now what? Some simple rules of thumb for optimal experimental design , year =

    List, John A and Sadoff, Sally and Wagner, Mathis , journal =. So you want to run an experiment, now what? Some simple rules of thumb for optimal experimental design , year =

  21. [44]

    2022 , publisher =

    Blackwell, Matthew and Pashley, Nicole E and Valentino, Dominic , title =. 2022 , publisher =

    2022

  22. [45]

    Experimental design under network interference , year =

    Viviano, Davide , journal =. Experimental design under network interference , year =

  23. [46]

    and Savage, Leonard J

    Bahadur, Raghu R. and Savage, Leonard J. , journal =. The nonexistence of certain statistical procedures in nonparametric problems , year =

  24. [49]

    Safe policy learning through extrapolation: Application to pre-trial risk assessment , year =

    Ben-Michael, Eli and Greiner, D James and Imai, Kosuke and Jiang, Zhichao , journal =. Safe policy learning through extrapolation: Application to pre-trial risk assessment , year =

  25. [50]

    Asymptotic statistics , year =

    Van der Vaart, Aad W , publisher =. Asymptotic statistics , year =

  26. [51]

    Approximation theorems of mathematical statistics , year =

    Serfling, Robert J , publisher =. Approximation theorems of mathematical statistics , year =

  27. [52]

    Principles of mathematical analysis , year =

    Rudin, Walter , journal =. Principles of mathematical analysis , year =

  28. [53]

    Microcredit Impacts: Evidence from a Randomized Microcredit Program Placement Experiment by Compartamos Banco , year =

    Angelucci, Manuela and Karlan, Dean and Zinman, Jonathan , journal =. Microcredit Impacts: Evidence from a Randomized Microcredit Program Placement Experiment by Compartamos Banco , year =

  29. [54]

    Teoria statistica delle classi e calcolo delle probabilit

    Bonferroni, Carlo , journal =. Teoria statistica delle classi e calcolo delle probabilit. 1936 , pages =

    1936

  30. [55]

    Testing statistical hypotheses , year =

    Lehmann, Erich Leo and Romano, Joseph P , publisher =. Testing statistical hypotheses , year =

  31. [56]

    1967 , number =

    Journal of the American statistical association , title =. 1967 , number =

    1967

  32. [57]

    2025 , archiveprefix =

    Martinez-Taboada, Diego and Ramdas, Aaditya , title =. 2025 , archiveprefix =. doi:10.48550/arXiv.2505.01987 , eprint =

    2025 doi arXiv

  33. [58]

    Causal inference in statistics, social, and biomedical sciences , year =

    Imbens, Guido W and Rubin, Donald B , publisher =. Causal inference in statistics, social, and biomedical sciences , year =

  34. [59]

    Statistical inference , year =

    Casella, George and Berger, Roger , publisher =. Statistical inference , year =

  35. [60]

    The experimental approach to development economics , year =

    Banerjee, Abhijit V and Duflo, Esther , journal =. The experimental approach to development economics , year =

  36. [61]

    1947 , publisher =

    Von Neumann, John and Morgenstern, Oskar , title =. 1947 , publisher =

    1947

  37. [62]

    Statistical decision functions , year =

    Wald, Abraham , booktitle =. Statistical decision functions , year =

  38. [63]

    Sion, Maurice , title =

  39. [64]

    Maxmin expected utility with non-unique prior , year =

    Gilboa, Itzhak and Schmeidler, David , journal =. Maxmin expected utility with non-unique prior , year =

  40. [65]

    A better bound on the variance , year =

    Bhatia, Rajendra and Davis, Chandler , journal =. A better bound on the variance , year =. doi:10.1080/00029890.2000.12005203 , publisher =

    2000

  41. [66]

    Convex Analysis , year =

    Ralph Tyrell Rockafellar , publisher =. Convex Analysis , year =

  42. [68]

    Risk, uncertainty and profit , year =

    Knight, Frank Hyneman , publisher =. Risk, uncertainty and profit , year =

  43. [70]

    Remarks on statistical inference for statistical decisions , year =

    Manski, Charles F , institution =. Remarks on statistical inference for statistical decisions , year =. doi:10.1920/wp.cem.2019.06 , number =

    1920 doi

  44. [71]

    Monotone operator theory in convex optimization , year =

    Combettes, Patrick L , journal =. Monotone operator theory in convex optimization , year =

  45. [72]

    Topological Spaces , year =

    Berge, Claude , publisher =. Topological Spaces , year =

  46. [73]

    , journal =

    Gart, John J. , journal =. A Locally Most Powerful Test for the Symmetric Folded Binomial Distribution , year =

  47. [74]

    An Alternative Test for the Symmetric Folded Binomial Distribution , year =

    Mantel, Nathan , journal =. An Alternative Test for the Symmetric Folded Binomial Distribution , year =

  48. [75]

    and Ragozini, Giancarlo , journal =

    Porzio, Giovanni C. and Ragozini, Giancarlo , journal =. On the Stochastic Ordering of Folded Binomials , year =

  49. [76]

    Probability Inequalities for Sums of Bounded Random Variables , year =

    Hoeffding, Wassily , journal =. Probability Inequalities for Sums of Bounded Random Variables , year =

  50. [77]

    Foundations of Computational Mathematics , title =

    Lugosi, G. Foundations of Computational Mathematics , title =. 2019 , number =

    2019

  51. [79]

    Clopper, C. J. and Pearson, E. S. , journal =. The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial , year =

  52. [80]

    , institution =

    Schlag, Karl H. , institution =. 2006 , number =

    2006

  53. [81]

    Dai, Jessica and Gradu, Paula and Harshaw, Christopher , booktitle =. Clip-. 2023 , volume =

    2023

  54. [82]

    Stronger

    Noarov, Georgy and Fogliato, Riccardo and Bertran, Martin and Roth, Aaron , booktitle =. Stronger. 2025 , note =

    2025

  55. [83]

    Logarithmic

    Neopane, Ojash and Ramdas, Aaditya and Singh, Aarti , booktitle =. Logarithmic. 2025 , series =

    2025

  56. [84]

    2025 , note =

    Neopane, Ojash and Ramdas, Aaditya and Singh, Aarti , title =. 2025 , note =

    2025

  57. [85]

    Adaptive

    Zhao, Jinglong , booktitle =. Adaptive. 2024 , note =

    2024

  58. [86]

    and Border, Kim C

    Aliprantis, Charalambos D. and Border, Kim C. , edition =. Infinite Dimensional Analysis: A Hitchhiker's Guide , year =

  59. [87]

    , publisher =

    Tsybakov, Alexandre B. , publisher =. Introduction to Nonparametric Estimation , year =

  60. [88]

    Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities , year =

    Miguel, Edward and Kremer, Michael , journal =. Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities , year =

  61. [89]

    Bertrand, Marianne and Mullainathan, Sendhil , journal =. Are. 2004 , number =

    2004

  62. [90]

    , journal =

    Thornton, Rebecca L. , journal =. The Demand for, and Impact of, Learning. 2008 , number =

    2008

  63. [91]

    Challenging the empirical mean and empirical variance: A deviation study , year =

    Catoni, Olivier , journal =. Challenging the empirical mean and empirical variance: A deviation study , year =

  64. [92]

    and Liebman, Jeffrey B

    Kling, Jeffrey R. and Liebman, Jeffrey B. and Katz, Lawrence F. , journal =. Experimental Analysis of Neighborhood Effects , year =

  65. [93]

    The Fitting of Straight Lines if Both Variables Are Subject to Error , year =

    Wald, Abraham , journal =. The Fitting of Straight Lines if Both Variables Are Subject to Error , year =

  66. [94]

    and Rubin, Herman , journal =

    Anderson, Theodore W. and Rubin, Herman , journal =. Estimator of the Parameters of a Single Equation in a Complete System of Stochastic Equations , year =

  67. [95]

    and Angrist, Joshua D

    Imbens, Guido W. and Angrist, Joshua D. , journal =. Identification and Estimation of Local Average Treatment Effects , year =

  68. [96]

    and Pischke, J

    Angrist, Joshua D. and Pischke, J. Mostly Harmless Econometrics: An Empiricist's Companion , year =

  69. [97]

    and Tabord-Meehan, Max , journal =

    Bai, Yuehao and Guo, Hongchang and Shaikh, Azeem M. and Tabord-Meehan, Max , journal =. Inference in Experiments with Matched Pairs and Imperfect Compliance , year =

  70. [98]

    , journal =

    Chetty, Raj and Hendren, Nathaniel and Katz, Lawrence F. , journal =. The Effects of Exposure to Better Neighborhoods on Children: New Evidence from the Moving to Opportunity Experiment , year =

  71. [99]

    and Kang, Hyunseung , note =

    Athey, Susan and Chetty, Raj and Imbens, Guido W. and Kang, Hyunseung , note =. The Surrogate Index: Combining Short-Term Proxies to Estimate Long-Term Treatment Effects More Rapidly and Precisely , year =

  72. [100]

    Aliprantis and Kim C

    Charalambos D. Aliprantis and Kim C. Border. Infinite Dimensional Analysis: A Hitchhiker's Guide. Springer, 3rd edition, 2006

    2006

  73. [101]

    Registrations in the AEA RCT registry

    American Economic Association . Registrations in the AEA RCT registry. Harvard Dataverse, 2026. URL https://doi.org/10.7910/DVN/PWNOZZ. April 2026 monthly snapshot

    2026 doi

  74. [102]

    Imbens, and Hyunseung Kang

    Susan Athey, Raj Chetty, Guido W. Imbens, and Hyunseung Kang. The surrogate index: Combining short-term proxies to estimate long-term treatment effects more rapidly and precisely. Working paper, 2024

    2024

  75. [103]

    Bahadur and Leonard J

    Raghu R. Bahadur and Leonard J. Savage. The nonexistence of certain statistical procedures in nonparametric problems. The Annals of Mathematical Statistics, 27 0 (4): 0 1115--1122, 1956. doi:10.1214/aoms/1177728077

    1956

  76. [104]

    On the performance of the Neyman allocation with small pilots

    Yong Cai and Ahnaf Rafi. On the performance of the Neyman allocation with small pilots. Journal of Econometrics, 242 0 (1): 0 105793, 2024. doi:10.1016/j.jeconom.2024.105793

    2024

  77. [105]

    Raj Chetty, Nathaniel Hendren, and Lawrence F. Katz. The effects of exposure to better neighborhoods on children: New evidence from the moving to opportunity experiment. American Economic Review, 106 0 (4): 0 855--902, 2016

    2016

  78. [106]

    C. J. Clopper and E. S. Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26 0 (4): 0 404--413, 1934. doi:10.1093/biomet/26.4.404

    1934 doi

  79. [107]

    Eldridge, Claire L

    Sandra M. Eldridge, Claire L. Chan, Michael J. Campbell, Christine M. Bond, Sally Hopewell, Lehana Thabane, and Gillian A. Lancaster. CONSORT 2010 statement: extension to randomised pilot and feasibility trials. BMJ, 355: 0 i5239, 2016. doi:10.1136/bmj.i5239

    2010 doi

  80. [108]

    John J. Gart. A locally most powerful test for the symmetric folded binomial distribution. Biometrics, 26 0 (1): 0 129--138, 1970. doi:10.2307/2529049

    1970 doi

  81. [109]

    Adaptive experimental design using the propensity score

    Jinyong Hahn, Keisuke Hirano, and Dean Karlan. Adaptive experimental design using the propensity score. Journal of Business & Economic Statistics, 29 0 (1): 0 96--108, 2011. doi:10.1198/jbes.2009.08161

    2011 arXiv

  82. [110]

    Probability inequalities for sums of bounded random variables

    Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58 0 (301): 0 13--30, 1963. doi:10.1080/01621459.1963.10500830

    1963

  83. [111]

    Kling, Jeffrey B

    Jeffrey R. Kling, Jeffrey B. Liebman, and Lawrence F. Katz. Experimental analysis of neighborhood effects. Econometrica, 75 0 (1): 0 83--119, 2007

    2007

  84. [112]

    Testing statistical hypotheses

    Erich Leo Lehmann and Joseph P Romano. Testing statistical hypotheses. Springer, 2005

    2005

  85. [113]

    Mean estimation and regression under heavy-tailed distributions: A survey

    G \'a bor Lugosi and Shahar Mendelson. Mean estimation and regression under heavy-tailed distributions: A survey. Foundations of Computational Mathematics, 19 0 (5): 0 1145--1190, 2019. doi:10.1007/s10208-019-09427-x

    2019 doi

  86. [114]

    An alternative test for the symmetric folded binomial distribution

    Nathan Mantel. An alternative test for the symmetric folded binomial distribution. Biometrics, 26 0 (4): 0 848--851, 1970. doi:10.2307/2528731

    1970 doi

  87. [115]

    Empirical Bernstein bounds and sample variance penalization

    Andreas Maurer and Massimiliano Pontil. Empirical Bernstein bounds and sample variance penalization. In Proceedings of the 22nd Annual Conference on Learning Theory, 2009. URL https://arxiv.org/abs/0907.3740

    2009 arXiv

  88. [116]

    Outline of a theory of statistical estimation based on the classical theory of probability

    Jerzy Neyman. Outline of a theory of statistical estimation based on the classical theory of probability. Philosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences, 236 0 (767): 0 333--380, 1937. doi:10.1098/rsta.1937.0005

    1937

  89. [117]

    Porzio and Giancarlo Ragozini

    Giovanni C. Porzio and Giancarlo Ragozini. On the stochastic ordering of folded binomials. Statistics & Probability Letters, 79 0 (9): 0 1299--1304, 2009. doi:10.1016/j.spl.2009.01.021

    2009 doi

  90. [118]

    Stratification trees for adaptive randomisation in randomised controlled trials

    Max Tabord-Meehan. Stratification trees for adaptive randomisation in randomised controlled trials. Review of Economic Studies, 90 0 (5): 0 2646--2673, 2023. doi:10.1093/restud/rdac083

    2023 doi

  91. [119]

    Tsybakov

    Alexandre B. Tsybakov. Introduction to Nonparametric Estimation. Springer, 2009

    2009

  92. [120]

    The value of reference letters: Experimental evidence from South Africa

    Martin Abel, Rulof Burger, and Patrizio Piraino. The value of reference letters: Experimental evidence from South Africa . American Economic Journal: Applied Economics, 12 0 (3): 0 40--71, 2020. doi:10.1257/app.20180666

    2020 doi

  93. [121]

    Certified decisions, 2025

    Isaiah Andrews and Jiafeng Chen. Certified decisions, 2025. URL https://arxiv.org/abs/2502.17830

    2025 arXiv

  94. [122]

    Armstrong

    Timothy B. Armstrong. Asymptotic efficiency bounds for a class of experimental designs, 2026. URL https://arxiv.org/abs/2205.02726

    2026 arXiv

  95. [123]

    Optimality of matched-pair designs in randomized controlled trials

    Yuehao Bai. Optimality of matched-pair designs in randomized controlled trials. American Economic Review, 112 0 (12): 0 3911--3940, 2022. doi:10.1257/aer.20201856

    2022 doi

  96. [124]

    James O. Berger. Statistical Decision Theory and Bayesian Analysis. Springer Series in Statistics. Springer, New York, NY, 2 edition, 1985. ISBN 978-0-387-96098-2. doi:10.1007/978-1-4757-4286-2

    1985 doi

  97. [125]

    Are Emily and Greg more employable than Lakisha and Jamal ? A field experiment on labor market discrimination

    Marianne Bertrand and Sendhil Mullainathan. Are Emily and Greg more employable than Lakisha and Jamal ? A field experiment on labor market discrimination. American Economic Review, 94 0 (4): 0 991--1013, 2004. doi:10.1257/0002828042002561

    2004 doi

  98. [126]

    Teoria statistica delle classi e calcolo delle probabilit \`a

    Carlo Bonferroni. Teoria statistica delle classi e calcolo delle probabilit \`a . Pubblicazioni del R. Istituto Superiore di Scienze Economiche e Commerciali di Firenze, 8: 0 3--62, 1936

    1936

  99. [127]

    Policy learning with confidence

    Victor Chernozhukov, Sokbae Lee, Adam M Rosen, and Liyang Sun. Policy learning with confidence. arXiv preprint arXiv:2502.10653, 2025

    2025

  100. [128]

    Fine stratification of survey experiments, 2026

    Max Cytrynbaum. Fine stratification of survey experiments, 2026. URL https://arxiv.org/abs/2111.08157

    2026 arXiv

  101. [129]

    Clip- OGD : An experimental design for adaptive Neyman allocation in sequential experiments

    Jessica Dai, Paula Gradu, and Christopher Harshaw. Clip- OGD : An experimental design for adaptive Neyman allocation in sequential experiments. In Advances in Neural Information Processing Systems, volume 36, 2023. arXiv:2305.17187

    2023 arXiv

  102. [130]

    On the role of the propensity score in efficient semiparametric estimation of average treatment effects

    Jinyong Hahn. On the role of the propensity score in efficient semiparametric estimation of average treatment effects. Econometrica, 66 0 (2): 0 315--331, 1998. doi:10.2307/2998560

    1998 doi

  103. [131]

    Comment on ``an essay on the logical foundations of survey sampling, part one'' by D

    Jaroslav H \'a jek. Comment on ``an essay on the logical foundations of survey sampling, part one'' by D. Basu . In V. P. Godambe and D. A. Sprott, editors, Foundations of Statistical Inference, page 236. Holt, Rinehart and Winston, Toronto, 1971

    1971

  104. [132]

    Minimax-regret sample selection in randomized experiments

    Yuchen Hu, Henry Zhu, Emma Brunskill, and Stefan Wager. Minimax-regret sample selection in randomized experiments. In Proceedings of the 25th ACM Conference on Economics and Computation, pages 1209--1235, 2024

    2024

  105. [133]

    Causal inference in statistics, social, and biomedical sciences

    Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge university press, 2015

    2015

  106. [134]

    Evidence aggregation for treatment choice, 2024

    Takuya Ishihara and Toru Kitagawa. Evidence aggregation for treatment choice, 2024. URL https://arxiv.org/abs/2108.06473

    2024 arXiv

  107. [136]

    Econometrics for decision making: Building foundations sketched by Haavelmo and Wald

    Charles F Manski. Econometrics for decision making: Building foundations sketched by Haavelmo and Wald . Econometrica, 89 0 (6): 0 2827--2853, 2021. doi:10.3982/ecta17985

    2021 doi

  108. [137]

    Sufficient trial size to inform clinical practice

    Charles F Manski and Aleksey Tetenov. Sufficient trial size to inform clinical practice. Proceedings of the National Academy of Sciences, 113 0 (38): 0 10518--10523, 2016. doi:10.1073/pnas.1612174113

    2016 doi

  109. [138]

    Sharp empirical Bernstein bounds for the variance of bounded random variables, 2025

    Diego Martinez-Taboada and Aaditya Ramdas. Sharp empirical Bernstein bounds for the variance of bounded random variables, 2025. URL https://arxiv.org/abs/2505.01987. arXiv:2505.01987v2

    2025 arXiv

  110. [139]

    On the method of bounded differences

    Colin McDiarmid et al. On the method of bounded differences. Surveys in combinatorics, 141 0 (1): 0 148--188, 1989

    1989

  111. [140]

    Worms: Identifying impacts on education and health in the presence of treatment externalities

    Edward Miguel and Michael Kremer. Worms: Identifying impacts on education and health in the presence of treatment externalities. Econometrica, 72 0 (1): 0 159--217, 2004. doi:10.1111/j.1468-0262.2004.00481.x

    2004

  112. [141]

    On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection

    Jerzy Neyman. On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection. Journal of the Royal Statistical Society, 97 0 (4): 0 558--625, 1934. doi:10.2307/2342192

    1934 doi

  113. [142]

    The theory of statistical decision

    Leonard J Savage. The theory of statistical decision. Journal of the American Statistical Association, 46 0 (253): 0 55--67, 1951. doi:10.1080/01621459.1951.10500768

    1951

  114. [143]

    Karl H. Schlag. ELEVEN : Tests needed for a recommendation. Economics Working Paper ECO 2006/2, European University Institute, 2006

    2006

  115. [144]

    Rectangular confidence regions for the means of multivariate normal distributions

    Zbyn e k S id \'a k. Rectangular confidence regions for the means of multivariate normal distributions. Journal of the American statistical association, 62 0 (318): 0 626--633, 1967

    1967

  116. [145]

    Minimax regret treatment choice with finite samples

    J \"o rg Stoye. Minimax regret treatment choice with finite samples. Journal of Econometrics, 151 0 (1): 0 70--81, 2009. doi:10.1016/j.jeconom.2009.02.013

    2009 doi

  117. [146]

    Minimax regret treatment choice with covariates or with limited validity of experiments

    J \"o rg Stoye. Minimax regret treatment choice with covariates or with limited validity of experiments. Journal of Econometrics, 166 0 (1): 0 138--156, 2012. doi:10.1016/j.jeconom.2011.06.012

    2012 doi

  118. [147]

    Statistical treatment choice based on asymmetric minimax regret criteria

    Aleksey Tetenov. Statistical treatment choice based on asymmetric minimax regret criteria. Journal of Econometrics, 166 0 (1): 0 157--165, 2012. doi:10.1016/j.jeconom.2011.06.013

    2012 doi

  119. [148]

    Thornton

    Rebecca L. Thornton. The demand for, and impact of, learning HIV status. American Economic Review, 98 0 (5): 0 1829--1863, 2008. doi:10.1257/aer.98.5.1829

    2008 doi

  120. [149]

    Adaptive Neyman allocation

    Jinglong Zhao. Adaptive Neyman allocation. In Proceedings of the 25th ACM Conference on Economics and Computation, 2024. arXiv:2309.08808

    2024

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.