pith. sign in

arxiv: 2205.02726 · v3 · pith:WMXUSFEDnew · submitted 2022-05-05 · 📊 stat.ME

Asymptotic Efficiency Bounds for a Class of Experimental Designs

Timothy B. Armstrong This is my paper

Pith reviewed 2026-05-24 11:13 UTC · model grok-4.3

classification 📊 stat.ME
keywords asymptotic efficiencyexperimental designaverage treatment effectadaptive designspropensity scoresemiparametric boundsHahn bound
0
0 comments X

The pith

Adaptive treatment assignment based on past data cannot improve first-order asymptotic efficiency for average treatment effect estimation beyond the best fixed propensity score design.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper derives efficiency bounds for experiments that sample units sequentially and assign treatment possibly depending on covariates and past outcomes. It shows that for binary treatment average effect estimation, these bounds match the minimum of the Hahn semiparametric efficiency bound over possible propensity scores. This implies that adaptive or stratified designs offer no additional first-order efficiency gains. The results extend to multiple treatments with constraints and covariate-based outcome sampling. A reader would care because it bounds the potential benefits of complex adaptive experimentation in large samples.

Core claim

In sequential sampling from an infinite population, the asymptotic efficiency bound for any design that assigns treatment as a function of covariates and past outcomes is given by the minimized Hahn (1998) bound, so no further first order improvement is possible relative to the optimal fixed propensity score design.

What carries the argument

The minimized Hahn (1998) semiparametric efficiency bound over propensity scores, which serves as the lower bound even for adaptive assignment rules.

If this is right

  • Adaptive designs achieve at most the efficiency of the optimal fixed propensity score design.
  • The bound applies to designs with multiple treatments and possible constraints on treatment probabilities.
  • Covariate based sampling of a single outcome is also covered by the same efficiency results.
  • No first-order asymptotic improvement comes from using past outcome data in treatment assignment.
  • The results hold for any randomized function of covariates and past data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Researchers may prefer simpler non-adaptive designs for their computational ease if asymptotic efficiency is the goal.
  • Higher-order terms or finite-sample behavior might still favor adaptive approaches despite matching first-order bounds.
  • The framework could extend to other estimands like conditional average treatment effects.
  • Testing the bound in finite samples with specific adaptive rules would be a natural next step.

Load-bearing premise

The derivation assumes sequential sampling from an infinite population where the Hahn semiparametric efficiency bound remains the relevant benchmark for adaptive designs.

What would settle it

An adaptive design and corresponding estimator that attains an asymptotic variance strictly below the minimized Hahn bound in a large-sample simulation or theoretical construction would falsify the claim.

read the original abstract

We consider an experimental design setting in which units are assigned to treatment after being sampled sequentially from an infinite population. We derive asymptotic efficiency bounds that apply to data from any experiment that assigns treatment as a (possibly randomized) function of covariates and past outcome data, including stratification on covariates and adaptive designs. For estimating the average treatment effect of a binary treatment, our results show that no further first order asymptotic efficiency improvement is possible relative to an estimator that achieves the Hahn (1998) bound in an experimental design where the propensity score is chosen to minimize this bound. Our results also apply to settings with multiple treatments with possible constraints on treatment, as well as covariate based sampling of a single outcome.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The paper derives asymptotic efficiency bounds applicable to any sequential experimental design (including adaptive ones) in which treatment assignment is a possibly randomized function of covariates and past outcomes. For the ATE of a binary treatment, the central claim is that the first-order asymptotic efficiency bound equals the minimized Hahn (1998) bound and that no further improvement is possible; analogous results are stated for multiple treatments under constraints and for covariate-based sampling of a single outcome.

Significance. If the derivations are correct, the result is significant for semiparametric efficiency theory and experimental design: it implies that adaptive dependence on past outcomes cannot improve the first-order asymptotic precision of ATE estimators beyond the best non-adaptive propensity-score design. This supplies a theoretical benchmark that can guide design choice and clarifies the scope of Hahn (1998) under sequential sampling from an infinite population.

major comments (1)
  1. [Abstract / main theorem] Abstract and introduction: the claim that the Hahn (1998) bound remains the semiparametric efficiency lower bound under outcome-dependent assignment requires an explicit verification that the tangent space (and hence the efficient influence function) is unchanged. Adaptation introduces dependence between the treatment indicator and previous responses, which can alter score functions; without this calculation the central claim that no further first-order improvement is possible does not follow from the stated conditions.
minor comments (1)
  1. [Abstract] The abstract states the infinite-population sequential sampling assumption but does not list the precise regularity conditions (e.g., bounded moments, positivity) under which the bounds are derived; these should be stated explicitly in the main text.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their thoughtful review and for highlighting the need for explicit verification of the tangent space under sequential designs. We address the major comment below and will revise the manuscript accordingly to strengthen the presentation of the main result.

read point-by-point responses

  1. Referee: [Abstract / main theorem] Abstract and introduction: the claim that the Hahn (1998) bound remains the semiparametric efficiency lower bound under outcome-dependent assignment requires an explicit verification that the tangent space (and hence the efficient influence function) is unchanged. Adaptation introduces dependence between the treatment indicator and previous responses, which can alter score functions; without this calculation the central claim that no further first-order improvement is possible does not follow from the stated conditions.

    Authors: We agree that an explicit derivation of the tangent space is required to rigorously establish that the efficient influence function is unchanged. In the revised version we will insert a new subsection (immediately preceding the proof of Theorem 1) that computes the tangent space for the sequential model. The argument proceeds by noting that the observed-data likelihood factors as the product of the known assignment mechanism (which is measurable with respect to the filtration generated by past data) and the conditional outcome densities. Because the assignment probabilities are known and the outcome model is nonparametric, the scores for the conditional outcome distributions coincide with those in the non-adaptive case; the additional dependence induced by adaptation does not enlarge the tangent space for the ATE functional. Consequently the efficient influence function remains identical to the one derived by Hahn (1998), and the first-order bound is unchanged. We will also add a brief remark in the introduction and abstract clarifying that this tangent-space calculation underpins the central claim. revision: yes

Circularity Check

0 steps flagged

No circularity; derivation extends external Hahn (1998) semiparametric bound to adaptive designs

full rationale

The paper's central result states that adaptive designs (treatment assignment depending on past outcomes) cannot improve first-order asymptotic efficiency for the ATE beyond the minimized Hahn (1998) bound. This is framed as an extension of the external Hahn result rather than a reduction to any quantity defined inside the paper. No self-citations are load-bearing for the tangent-space or efficiency-bound argument, no parameters are fitted to data and then relabeled as predictions, and no ansatz or uniqueness claim is smuggled via prior work by the same author. The derivation is therefore self-contained against the cited external benchmark.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only the abstract is available; no explicit free parameters, axioms, or invented entities can be extracted. The result appears to rest on the external Hahn (1998) bound and standard asymptotic regularity conditions for sequential sampling.

pith-pipeline@v0.9.0 · 5631 in / 1113 out tokens · 19101 ms · 2026-05-24T11:13:18.409079+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 19 canonical work pages

  1. [1]

    Abadie, A. and G. W. Imbens (2012): A Martingale Representation for Matching Estimators , Journal of the American Statistical Association, 107, 833--843

    work page 2012

  2. [2]

    Andrews, D. W. K. (1988): Laws of Large Numbers for Dependent Non - Identically Distributed Random Variables , Econometric Theory, 4, 458--467

    work page 1988

  3. [3]

    Bai, Y., J. P. Romano, and A. M. Shaikh (2021): Inference in Experiments With Matched Pairs , Journal of the American Statistical Association, 0, 1--12, publisher: Taylor & Francis \_eprint: https://doi.org/10.1080/01621459.2021.1883437

    work page doi:10.1080/01621459.2021.1883437 2021

  4. [4]

    (1995): Probability and Measure , 3rd Edition , Wiley-Interscience, 3 ed

    Billingsley, P. (1995): Probability and Measure , 3rd Edition , Wiley-Interscience, 3 ed

    work page 1995

  5. [5]

    Blackwell, D. and M. A. Girshick (1954): Theory of Games and Statistical Decisions , John Wiley & Sons, Incorporated

    work page 1954

  6. [6]

    Bruhn, M. and D. McKenzie (2009): In Pursuit of Balance : Randomization in Practice in Development Field Experiments , American Economic Journal: Applied Economics, 1, 200--232

    work page 2009

  7. [7]

    Bugni, F. A., I. A. Canay, and A. M. Shaikh (2018): Inference Under Covariate - Adaptive Randomization , Journal of the American Statistical Association, 113, 1784--1796

    work page 2018

  8. [8]

    (2021): Designing Representative and Balanced Experiments by Local Randomization , arXiv:2111.08157 [econ, math, stat]

    Cytrynbaum, M. (2021): Designing Representative and Balanced Experiments by Local Randomization , arXiv:2111.08157 [econ, math, stat]

    work page arXiv 2021

  9. [9]

    Glennerster, and M

    Duflo, E., R. Glennerster, and M. Kremer (2007): Chapter 61 Using Randomization in Development Economics Research : A Toolkit , in Handbook of Development Economics , ed. by T. P. Schultz and J. A. Strauss, Elsevier, vol. 4, 3895--3962

    work page 2007

  10. [10]

    (1998): On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects , Econometrica, 66, 315--331

    Hahn, J. (1998): On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects , Econometrica, 66, 315--331

    work page 1998

  11. [11]

    Hirano, and D

    Hahn, J., K. Hirano, and D. Karlan (2011): Adaptive Experimental Design Using the Propensity Score , Journal of Business & Economic Statistics, 29, 96--108

    work page 2011

  12. [12]

    Imbens, G., G. King, D. McKenzie, and G. Ridder (2009): On the finite sample benefits of stratification in randomized experiments,

    work page 2009

  13. [13]

    Imbens, G. W. and D. B. Rubin (2015): Causal Inference for Statistics , Social , and Biomedical Sciences : An Introduction , New York: Cambridge University Press, 1 edition ed

    work page 2015

  14. [14]

    Neyman, J. (1934): On the Two Different Aspects of the Representative Method : The Method of Stratified Sampling and the Method of Purposive Selection , Journal of the Royal Statistical Society, 97, 558--625

    work page 1934

  15. [15]

    Rosenberger, W. F. and J. M. Lachin (2015): Randomization in Clinical Trials : Theory and Practice , John Wiley & Sons, google-Books-ID: ZJEvCgAAQBAJ

    work page 2015

  16. [16]

    Savage, L. J. (1972): The Foundations of Statistics , New York, NY: Dover, 2 ed

    work page 1972

  17. [17]

    (2018): Stratification Trees for Adaptive Randomization in Randomized Controlled Trials , arXiv:1806.05127 [econ, stat], arXiv: 1806.05127

    Tabord-Meehan, M. (2018): Stratification Trees for Adaptive Randomization in Randomized Controlled Trials , arXiv:1806.05127 [econ, stat], arXiv: 1806.05127

    work page arXiv 2018

  18. [18]

    van der Vaart, A. W. (1998): Asymptotic Statistics , Cambridge, UK ; New York, NY, USA: Cambridge University Press

    work page 1998

  19. [19]

    van der Vaart, A. W. and J. A. Wellner (1996): Weak convergence and empirical processes, Springer

    work page 1996