Pith. sign in

REVIEW 4 major objections 3 minor 1 cited by

Learning What to Learn: Experimental Design when Combining Experimental with Observational Evidence

T0 review · 4 major / 3 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper establishes that worst-case adaptation regret in designs combining experiments with observational evidence equals the larger of two normalized ratios, turning design choice into tractable optimization.

desk verdict The core minimax-regret characterization is right and worth taking seriously, but the application numbers and the local-asymptotics condition need fixing before this is publishable. read the letter →

arxiv 2510.23434 v5 pith:SH2OHCDW submitted 2025-10-27 econ.EM math.STstat.MEstat.TH

Aristotelis Epanomeritakis , Davide Viviano This is my paper
classification econ.EMmath.STstat.MEstat.TH MSC 62K0562C2090C11
keywords experimentaldesignobservationalevidenceadaptationregretminimaxbias-variancetrade-offmixed-integerquadraticprogrammingexternalvaliditysensitivityanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks how to choose which experiment to run, how to allocate sample, and how to combine experimental estimates with observational evidence that may be biased in unknown ways. It measures a design by its adaptation regret: the worst-case ratio of the design's mean-squared error to the error of an oracle that knows the bias bound and picks both design and estimator. The central claim is that this regret equals the larger of two normalized components — a variance regret and a bias regret — so the bias-variance trade-off becomes explicit and no bias bound is needed as input. The result reduces the joint choice of experiments, sample sizes, and shrinkage weights to a mixed-integer quadratic program, using only information standard power calculations already require. If correct, this gives a practical, pre-analysis-plan-friendly way to design trials meant to answer broader policy questions such as general-equilibrium effects.

What carries the argument

The load-bearing object is the adaptation regret R(E,Σ,γ), a minimax proportional-regret criterion in which a candidate design is compared to an oracle that knows the observational bias bound. With linearly combined experimental and observational estimates, the regret simplifies to the maximum of two normalized terms: variance regret α/α⋆ and bias regret β/β⋆, where the bias term is driven by the sensitivity vector ω = ∂τ/∂θ and the shrinkage weights γ. The quasi-convexity of this max form turns design choice into a mixed-integer quadratic program, giving a closed-form variance-optimal allocation conditional on the chosen arms and weights.

What would settle it

Implement a small two-parameter instance with fixed observational variances σ², experimental variances v², sensitivity weights ω, and a candidate design; for this design compute sup_{B≥0} (α + B²β)/inf_{feasible designs}(α' + B²β') by enumerating B, and compare it to max{α/α⋆, β/β⋆}. A single design where the two differ would refute Theorem 1.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is Theorem 1: for any feasible design (E, Σ, γ) satisfying a first-order linearity condition, the adaptation regret R(E,Σ,γ) equals max{α(E,Σ,γ)/α⋆, β(E,γ)/β⋆}. Here α is the estimator's variance, α⋆ the smallest variance attainable over the feasible class; β is the worst-case squared bias from using observational evidence, β⋆ the smallest attainable such bias. The same decomposition extends from linear shrinkage estimators to GMM-style moment selection under arbitrary normed bias sets and to confidence-interval length in partially identified models. This identity is what makes the design problem computationally tractable and is used to produce concret

Load-bearing premise

The whole decomposition rests on Assumption 2 — that the target error is exactly the weighted sum of parameter errors with known sensitivity weights ω — and on the local-misspecification condition ||b_n||₂√n → 0 in Section 5.3; the paper itself notes (Remark 5) that using the same data to estimate the Jacobian and choose the design can invalidate standard inference unless samples are split.

Editorial extensions

If this is right

  • Researchers can choose experiments, sample allocations, and shrinkage weights without specifying a bias bound, using only observational estimates, their covariance, and sensitivity weights — the same inputs as a conventional power analysis.
  • At the optimum, the two normalized components tend to be equalized, so designs automatically invest in high-sensitivity parameters when bias dominates and in high-variance arms when precision dominates.
  • The characterization extends to GMM moment selection, arbitrary norm ambiguity sets, partially identified targets, and confidence-interval length; in point-identified models, minimizing MSE regret and interval-length regret give the same design.
  • In the paper's cash-transfer application, the proposed two-arm design has roughly 4.5 times lower worst-case bias than the variance-optimal allocation with only a modest variance increase; in the site-selection application, its worst-case MSE stays within about 20% of the infeasible oracle.
  • The output is a complete pre-analysis plan: chosen treatment arms, sample sizes, and a prespecified rule for combining experimental and observational estimates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The variance/bias regret split suggests a reusable reporting format: any design could be summarized by the pair (α/α⋆, β/β⋆), letting reviewers see immediately which term drives the design and how far the feasible frontier sacrifices one for the other.
  • The local-misspecification condition ||b_n||₂√n → 0 (Section 5.3) implies the robustness guarantee is asymptotic in spirit; if real-world biases are fixed rather than local, the regret formula may misstate the true MSE trade-off, so users should check sensitivity of the chosen design to non-local biases.
  • The framework's dependence on a known sensitivity vector ω suggests a natural robustness extension: compute the optimal design under a range of plausible ω values, or treat ω as uncertain with a normed ambiguity set, and check whether the chosen experiment changes.
  • Because the paper itself warns (Remark 5) that estimating the Jacobian from the same sample used to choose the design can distort inference, an immediate practical rule is to split the observational sample into design and estimation parts before applying the method.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper studies experimental design when a researcher combines a randomized experiment with observational estimates that may be biased. The target is τ(θ), a known smooth function of parameters. The authors define a minimax proportional-regret criterion relative to an oracle that knows the worst-case bias bound and can choose both design and estimator. Theorem 1 (and its GMM generalization, Theorem 2) claims that the regret equals max{α/α⋆, β/β⋆}, i.e., the maximum of a normalized variance component and a normalized bias component. The paper also treats partially identified targets, confidence-interval length, prior information on the bias bound, and nonlinear estimands through local asymptotics. Two applications are developed: a cash-transfer experiment for general-equilibrium effects and a microfinance site-selection problem. The proposed algorithm is a mixed-integer quadratic program solved with off-the-shelf software.

Significance. If the characterization in Theorem 1 is correct, this is a valuable and potentially influential contribution. It turns a seemingly intractable joint design-and-estimation problem under unknown misspecification into a transparent max of two normalized criteria, requiring only inputs that are standard for power calculations. The proof of the central theorem is concise and plausible, and the MIQP formulation is implementable with standard software. The empirical applications are concrete and use externally calibrated inputs, which is a strength. However, the nonlinear local-asymptotic extension and the headline empirical numbers contain inconsistencies that currently prevent the paper from being accepted as is.

major comments (4)
  1. [§5.3, Eq. (23)] The stated local condition ||b_n||_2√n → 0 is incompatible with the text immediately below, which claims the framework covers b_n ∼ n^{-α} for α > 1/4. For α = 1/4, √n||b_n||_2 = n^{1/4}, which violates (23); the condition actually requires α > 1/2. The natural corrected condition, needed for the Taylor remainder and for the plug-in gradient error to be negligible, is √n||b_n||_2² → 0. Please correct (23) and provide a proof of the expansion under the corrected condition.
  2. [§5.3, displayed GMM expansion] The expansion √n(θ̂ − θ) = −(Λ⊤WΛ)^{−1}Λ⊤W√n(ḡΣ − b_n) + o_p(1) is not the standard GMM expansion when E[ḡΣ] = b_n. The usual expansion contains an additional term −(Λ⊤WΛ)^{−1}Λ⊤W√n b_n, which is non-negligible unless √n b_n → 0. Dropping this term changes the asymptotic bias and therefore the bias component of the regret formula in Theorem 2. This is load-bearing for the nonlinear/GMM extension and for the empirical applications that rely on it. The authors should either rescale the parameterization so that the bias enters as a fixed vector in the local asymptotics, or derive the correct MSE expression with the bias term included.
  3. [§1 and §6.1] The headline empirical claims in the introduction are inconsistent with the results reported in Section 6.1. The introduction states that at n_tot = 1000 the two-arm design delivers 'more than a 400% reduction in bias (and adaptation regret) with variance no larger than 30% of the Neyman variance', and that a single-arm design yields a '140% reduction in bias at the cost of only about a 12% increase in variance'. But Section 6.1 (Figure 4 and surrounding text) reports that the Neyman variance is only about 14% smaller for one arm and 35% smaller for two arms, and that the Neyman bias is about 1.4 times larger (one arm) and 4.5 times larger (two arms). These numbers imply a variance ratio of roughly 1.16 and 1.54 relative to Neyman, not 30%, and bias reductions of roughly 29% and 78%, not 140% and 400%. The microfinance summary in the introduction ('reduce MSE by more than 250%') is simil
  4. [§5.3, Remark 5] The discussion of estimating ω at the biased observational estimate is only heuristic. The text argues that b_n → 0 makes Jacobians and gradients consistently estimable, but the regret formula involves a supremum over B and over designs. Pointwise op(1) convergence of the first-order expansion does not automatically preserve the uniform regret characterization. A formal lemma is needed showing that, under the corrected condition √n||b_n||_2² → 0, the error from plugging in ω(θ̂_obs) is uniformly negligible in the regret objective over the feasible class D′ and over B ≥ 0. Without such a lemma, the claim that the same regret-optimal design is valid for nonlinear τ and GMM is not rigorously supported.
minor comments (3)
  1. [§6.1, Figure 4] The y-axis label 'Neyman/Regret Optimal' is ambiguous. It appears to plot the ratio of Neyman regret to the proposed design's regret, but the text should state this explicitly.
  2. [§5.3, Eq. (23)] If the corrected condition is √n||b_n||_2² → 0, the statement 'encompassing ||b_n||_2 = 1/n as a special case' remains true (since √n·n^{-2} → 0), but the phrase 'grow faster ... than the standard error' should be reconciled with the condition in a revised sentence.
  3. [Appendix A.3, Corollary 1] The line 'Because Ṙ_l(W,Σ) = R_l(W,Σ)^{1/2}' is correct only after squaring the regret definitions consistently; please add a brief clarification.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the regret decomposition is derived from stated assumptions, not fitted or defined into existence.

full rationale

I traced the derivation chain. Theorem 1 is proven in Appendix A.1 from the MSE decomposition sup_b∈B(B) MSE_b = α + B^2 β, the oracle envelope δ(t) = inf(α + tβ), and the quasi-convexity of (α + tβ)/δ(t); α⋆ and β⋆ are the t = 0 and t → ∞ limits of the same envelope, so the max-form is a mathematical consequence rather than a restatement of the inputs. The same argument structure proves Theorems 2–4. The empirical sections calibrate ω, Σ_obs, y0, d from the observational sample and prior literature, but these are inputs to the optimization; the paper does not fit a parameter and then relabel its fitted value as a prediction. In the microfinance application, the bias bound B is calibrated ex post from Banerjee et al. (2024) and explicitly not used in the design, so the MSE comparison functions as an external benchmark. The only flagged issue, Section 5.3's condition ||b_n||_2 √n → 0 alongside the claim that b_n ∼ n^{-α} for α > 1/4, is an internal consistency/correctness concern about the local asymptotics, not a circular step: Assumption 2 is stated, not derived from the regret formula. Self-citations (Viviano 2020; Breza et al. 2025; Rambachan et al. 2024) appear only in the literature review and are not load-bearing for the central result.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central theorem rests on standard decision-theoretic assumptions (bounded bias in a norm ball, linear estimators, known covariance, first-order linearity), plus application-specific calibration choices for variances and model constants. No new physical or conceptual entities are introduced beyond the conceptual oracle benchmark.

free parameters (4)
  • y₀ (income multiplier in GE application) = 1.5
    Set to Egger et al. (2022) total income/consumption multiplier minus 1; used in the GE estimand (28) to define θ₁ and affect ω.
  • d (demand slope parameter) = 0.5·(1−S₀)/W₀
    Calibrated from meta-analysis in Espey and Thilmany (2000) and PROGRESA baseline data; appears in w₀(θ) in (28) and affects the sensitivity ω.
  • Per-unit experimental variances vⱼ² = Calibrated to corresponding diagonal of Σ_obs times n_obs
    In Section 6.1, the experimental variance per unit is set equal to the observational study's variance scaled by sample sizes; this choice determines α and the optimal sample allocation.
  • Microfinance area experimental variance 2·v_pre,a² = 2·v_pre,a²
    In Section 6.2, the variance of a treated-control difference is assumed to be twice the pre-intervention variance from Table 2; this assumption calibrates α for the site-selection application.
assumptions (6)
  • domain assumption Setting 1: observational estimate satisfies E[θ̃_obs]−θ = b for unknown b; experimental estimates are unbiased; joint covariance Σ(E) is known.
    Defines the data-generating model in Section 2; the whole regret analysis is built on it.
  • domain assumption Assumption 1: feasible set D has uniformly bounded, strictly positive definite Σ(E), and γ=1 is admissible.
    Ensures the variance and bias regrets are finite and the optimization is well-posed; used in Theorem 1.
  • domain assumption Assumption 2: first-order linearity τ(θ)−τ(θ̂) = Σ ωⱼ(θⱼ−θ̂ⱼ) with known ω, |ωⱼ|∈(0,1).
    Central to the regret decomposition; the paper argues it is a first-order approximation, formalized in Section 5.3.
  • domain assumption Assumption 3 (GMM version): Λ⊤WΛ is invertible and ΓΛ(W)ΣΓΛ(W)⊤ has uniformly bounded entries and is positive definite.
    Required for Theorem 2 to guarantee finite, nondegenerate variance of the reweighted moments.
  • domain assumption Local misspecification condition ||bₙ||₂√n → 0 (Section 5.3, (23)).
    Justifies ignoring higher-order terms in the linear expansion of τ and g; the paper's own statement of the rate (α>1/4) appears to be a typo (should be α>1/2).
  • domain assumption The estimand τ and sensitivity ω are known/specified a priori.
    The framework requires the researcher to pre-specify τ(θ) and compute ω at the observational estimates; if this parameterization is wrong, the design target is wrong.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning What to Learn: Experimental Design when Combining Experimental with Observational Evidence." pith.science (2026). https://pith.science/paper/SH2OHCDW

@misc{pith2026251023434,
  author       = {Pith},
  title        = {Pith review of: Learning What to Learn: Experimental Design when Combining Experimental with Observational Evidence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SH2OHCDW}},
  note         = {Machine review of arXiv:2510.23434}
}
read the original abstract

Experiments deliver credible treatment-effect estimates but, because they are costly, are often restricted to specific sites, small populations, or particular mechanisms. A common practice across several fields is therefore to combine experimental estimates with reduced-form or structural external (observational) evidence to answer broader policy questions, such as those involving general equilibrium effects or external validity. We develop a unified framework for the design of experiments when combined with external evidence, i.e., choosing which experiment(s) to run and how to allocate sample size under arbitrary budget constraints. Because observational evidence may suffer bias unknown ex-ante, we evaluate designs using a minimax proportional-regret criterion that compares any candidate design to an oracle with knowledge about the observational study bias that jointly chooses the design and estimator. This yields a transparent bias-variance trade-off that does not require the researcher to specify a bias bound and relies only on information already needed for conventional power calculations. We illustrate the framework for studying the general equilibrium effects of cash transfer programs.

Figures

Figures reproduced from arXiv: 2510.23434 by the authors.

Figure 1
Figure 1. Each feasible design (E, Σ, γ) maps to a point α/α⋆ , β/β⋆  : the x–axis is the variance ratio and the y–axis is the worst–case bias ratio. The blue curve depicts the attainable frontier as we vary shrinkage γ and precision Σ. Level sets of the objective R = max{α/α⋆ , β/β⋆} are axis–aligned squares (the dotted inverted “L” shows the smallest such square touching the frontier). The minimizer outside boundary soluti… view at source ↗
Figure 2
Figure 2. Timeline of design, reporting, audience confidence set, and loss. [PITH_FULL_IMAGE:figures/full_fig_p023_2.png] view at source ↗
Figure 3
Figure 3. Optimal experimental sample sizes nj allocated to each arm (UCT, CCT, and Job program) as a function of the total experimental sample size ntot. The left panel considers designs with only one treatment arm, while the right panel allows two treatment arms. metric Variance regret Bias regret Adaptive regret 0.75 1.00 1.25 1.50 1.75 250 500 750 1000 ntot Neyman/Regret Optimal One feasible experiment 1 2 3 4 5 250 500 7… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Variance, bias, and adaptive regret of the Neyman (variance–optimal) allocation [PITH_FULL_IMAGE:figures/full_fig_p032_4.png]
Figure 5
Figure 5. Figure 5: Observational villages and area-level DiD effects in Karnataka. The background [PITH_FULL_IMAGE:figures/full_fig_p035_5.png]
Figure 6
Figure 6. Figure 6: Area-level allocations as total treated villages [PITH_FULL_IMAGE:figures/full_fig_p036_6.png]
Figure 7
Figure 7. Figure 7: Relative worst-case MSE across designs under a calibrated bias bound [PITH_FULL_IMAGE:figures/full_fig_p036_7.png]
Figure 8
Figure 8. Figure 8: Example for j = 2. Top row: relative bias β/β⋆ (dashed) and relative variance α/α⋆ (solid), both evaluated at the optimal weight γ ⋆ 2 . Bottom row: the optimal weight γ ⋆ 2 . Columns vary, respectively, (a) ω2 with v1 = 0.5, v2 = 1, σ1 = σ2 = 1, ω1 = 1; (b) v2 with ω …
Figure 9
Figure 9. Figure 9: Example for j = 2 (regret). Different colors correspond to the regret for choosing either experiment. Figure reports max{α(j, γ⋆ j )/α⋆ , β(j, γ⋆ j )/β⋆} for j ∈ {1, 2} as the x–axis parameter varies (columns: ω2, v2, σ2). The vertical dashed line marks indifference, w…
Figure 10
Figure 10. Figure 10: Regret comparisons. Top row: max{α(j, γ⋆ j )/α⋆ , β(j, γ⋆ j )/β⋆} for j ∈ {1, 2} as the x–axis parameter varies (columns: ω2, v2, σ2). The vertical dashed line marks indiffer￾ence, where the two curves intersect; to its left/right, the optimal experiment is the one wi…
Figure 11
Figure 11. Figure 11: Share of experimental papers published in AEA journals also presenting experi [PITH_FULL_IMAGE:figures/full_fig_p055_11.png]
Figure 12
Figure 12. Figure 12: Experimental design for General Equilibrium Effects: Variance and bias regret [PITH_FULL_IMAGE:figures/full_fig_p055_12.png]
Figure 13
Figure 13. Figure 13: Experimental Design for General Equilibrium Effects: Optimal shrinkage weights [PITH_FULL_IMAGE:figures/full_fig_p056_13.png]
Figure 14
Figure 14. Figure 14: Site selection for external validity: Optimal shrinkage weight [PITH_FULL_IMAGE:figures/full_fig_p056_14.png]
Figure 15
Figure 15. Figure 15: Site selection for external validity: Variance, bias, and adaptive regret of the [PITH_FULL_IMAGE:figures/full_fig_p057_15.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Better Measurement or Larger Samples? Data Collection for Policy Learning with Unobserved Heterogeneity

    econ.EM 2026-04 unverdicted novelty 7.0 of 10

    The paper derives rate-sharp regret bounds showing how estimate precision affects policy performance with latent traits and provides a sufficient condition for minimax-optimal data collection plans balancing measureme...

Reference graph

Works this paper leans on

90 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    Abadie, A. and J. Zhao (2021). Synthetic controls for experimental design. arXiv preprint arXiv:2108.02196\/

    arXiv 2021

  2. [2]

    Allcott, H., J. C. Castillo, M. Gentzkow, L. Musolff, and T. Salz (2025). Sources of market power in web search: Evidence from a field experiment. Technical report, National Bureau of Economic Research

    2025

  3. [3]

    Gallego, and C

    Allende, C., F. Gallego, and C. Neilson (2019). Approximating the equilibrium effects of informed school choice. Technical report

    2019

  4. [4]

    Barahona, M

    Andrews, I., N. Barahona, M. Gentzkow, A. Rambachan, and J. M. Shapiro (2025). Structural estimation under misspecification: Theory and implications for practice. The Quarterly Journal of Economics\/ , qjaf018

    2025

  5. [5]

    Gentzkow, and J

    Andrews, I., M. Gentzkow, and J. M. Shapiro (2017, 06). Measuring the sensitivity of parameter estimates to estimation moments. The Quarterly Journal of Economics\/ 132\/ (4), 1553--1592

    2017

  6. [6]

    Gentzkow, and J

    Andrews, I., M. Gentzkow, and J. M. Shapiro (2020). Transparency in structural research. Journal of Business & Economic Statistics\/ 38\/ (4), 711--722

    2020

  7. [7]

    Kitagawa, and A

    Andrews, I., T. Kitagawa, and A. McCloskey (2024). Inference on winners. The Quarterly Journal of Economics\/ 139\/ (1), 305--358

    2024

  8. [8]

    Andrews, I. and J. M. Shapiro (2021). A model of scientific communication. Econometrica\/ 89\/ (5), 2117--2142

    2021

Show all 90 references
  1. [9]

    Armstrong, T. B., P. Kline, and L. Sun (2024). Adapting to misspecification

    2024

  2. [10]

    Armstrong, T. B. and M. Koles \'a r (2018). Optimal inference in a class of regression models. Econometrica\/ 86\/ (2), 655--683

    2018

  3. [11]

    Armstrong, T. B. and M. Koles \'a r (2021). Sensitivity analysis using approximate moment condition models. Quantitative Economics\/ 12\/ (1), 77--108

    2021

  4. [12]

    Aronow, P. M., C. Samii, et al. (2017). Estimating average causal effects under general interference, with application to a social network experiment. The Annals of Applied Statistics\/ 11\/ (4), 1912--1947

    2017

  5. [13]

    Asher, S., T. Lunt, R. Matsuura, and P. Novosad (2019). The socioeconomic high-resolution rural-urban geographic dataset on india (shrug). URL: https://doi. org/10.7910/DVN/DPESAK\/

    2019 doi

  6. [14]

    Chetty, and G

    Athey, S., R. Chetty, and G. Imbens (2020). Combining experimental and observational data to estimate treatment effects on long term outcomes. arXiv preprint arXiv:2006.09676\/

    2020 arXiv

  7. [15]

    Chetty, and G

    Athey, S., R. Chetty, and G. Imbens (2025). The experimental selection correction estimator: Using experiments to remove biases in observational estimates. Technical report, National Bureau of Economic Research

    2025

  8. [16]

    Athey, S. and G. W. Imbens (2017). The econometrics of randomized experiments. In Handbook of economic field experiments , Volume 1, pp.\ 73--140. Elsevier

    2017

  9. [17]

    Atkinson, A. C. and V. Fedorov (1975). The design of experiments for discriminating between two rival models. Biometrika\/ 62\/ (1), 57--70

    1975

  10. [18]

    Attanasio, O. P., C. Meghir, and A. Santiago (2012). Education choices in mexico: using a structural model and a randomized experiment to evaluate progresa. The Review of Economic Studies\/ 79\/ (1), 37--66

    2012

  11. [19]

    Bai, Y. (2019). Optimality of matched-pair designs in randomized controlled trials. Available at SSRN 3483834\/

    2019

  12. [20]

    Jalal, and N

    Bandiera, O., A. Jalal, and N. Roussille (2025). The illusion of time: Gender gaps in job search and employment. Technical report, National Bureau of Economic Research

    2025

  13. [21]

    Breza, A

    Banerjee, A., E. Breza, A. G. Chandrasekhar, E. Duflo, M. O. Jackson, and C. Kinnan (2024). Changes in social network structure in response to exposure to formal credit markets. Review of Economic Studies\/ 91\/ (3), 1331--1372

    2024

  14. [22]

    Banerjee, A. V., S. Chassang, S. Montero, and E. Snowberg (2020). A theory of experimenters: Robustness, randomization, and balance. American Economic Review\/ 110\/ (4), 1206--1230

    2020

  15. [23]

    Bergquist, L. F. and M. Dinerstein (2020). Competition and entry in agricultural markets: Experimental evidence from kenya. American Economic Review\/ 110\/ (12), 3705--3747

    2020

  16. [24]

    Johnson, and N

    Bertsimas, D., M. Johnson, and N. Kallus (2015). The power of optimization over randomization in designing experiments involving small samples. Operations Research\/ 63\/ (4), 868--876

    2015

  17. [25]

    Bickel, P. (1984). Parametric robustness: small biases can be worthwhile. The Annals of Statistics\/ 12\/ (3), 864--879

    1984

  18. [26]

    Bonhomme, S. and M. Weidner (2022). Minimizing sensitivity to model misspecification. Quantitative Economics\/ 13\/ (3), 907--954

    2022

  19. [27]

    Box, G. E. and N. R. Draper (1959). A basis for the selection of a response surface design. Journal of the American Statistical Association\/ 54\/ (287), 622--654

    1959

  20. [28]

    Breza, E., A. G. Chandrasekhar, and D. Viviano (2025). Generalizability with ignorance in mind: learning what we do (not) know for archetypes discovery. arXiv preprint arXiv:2501.13355\/

    2025 arXiv

  21. [29]

    Colomboni, and M

    Cesa-Bianchi, N., R. Colomboni, and M. Kasy (2025). Adaptive maximization of social welfare. Econometrica\/ 93\/ (3), 1073--1104

    2025

  22. [30]

    Chaloner, K. and I. Verdinelli (1995). Bayesian experimental design: A review. Statistical science\/ , 273--304

    1995

  23. [31]

    Chamberlain, G. (2000). Econometric applications of maxmin expected utility. Journal of Applied Econometrics\/ 15\/ (6), 625--644

    2000

  24. [32]

    Chaudhuri, P. and P. A. Mykland (1993). Nonlinear experiments: Optimal design and inference based on likelihood. Journal of the American Statistical Association\/ 88\/ (422), 538--546

    1993

  25. [33]

    Hendren, and L

    Chetty, R., N. Hendren, and L. F. Katz (2016). The effects of exposure to better neighborhoods on children: New evidence from the moving to opportunity experiment. American Economic Review\/ 106\/ (4), 855--902

    2016

  26. [34]

    Cytrynbaum, M. (2021). Optimal stratification of survey experiments. arXiv preprint arXiv:2111.08157\/

    2021 arXiv

  27. [35]

    Finan, A

    de Albuquerque, A., F. Finan, A. Jha, L. Karpuska, and F. Trebbi (2025). Decoupling taste-based versus statistical discrimination in elections. Technical report, National Bureau of Economic Research

    2025

  28. [36]

    de Chaisemartin, C. and X. D'Haultf uille (2020). Empirical mse minimization to estimate a scalar parameter. arXiv preprint arXiv:2006.14667\/

    2020 arXiv

  29. [37]

    Dominitz, J. and C. F. Manski (2017). More data or better data? a statistical decision problem. The Review of Economic Studies\/ 84\/ (4), 1583--1605

    2017

  30. [38]

    Donoho, D. L. (1994). Statistical estimation and optimal recovery. The Annals of Statistics\/ 22\/ (1), 238--270

    1994

  31. [39]

    Donoho, D. L., R. C. Liu, and B. MacGibbon (1990). Minimax risk over hyperrectangles, and implications. The Annals of Statistics\/ , 1416--1437

    1990

  32. [40]

    Glennerster, and M

    Duflo, E., R. Glennerster, and M. Kremer (2007). Using randomization in development economics research: A toolkit. Handbook of development economics\/ 4 , 3895--3962

    2007

  33. [41]

    Huitfeldt, S

    Dutz, D., I. Huitfeldt, S. Lacouture, M. Mogstad, A. Torgovitsky, and W. Van Dijk (2021). Selection in surveys: Using randomized incentives to detect and account for nonresponse bias. Technical report, National Bureau of Economic Research

    2021

  34. [42]

    Haushofer, E

    Egger, D., J. Haushofer, E. Miguel, P. Niehaus, and M. Walker (2022). General equilibrium effects of cash transfers: Experimental evidence from K enya. Econometrica\/ 90\/ (6), 2603--2643

    2022

  35. [43]

    Espey, M. and D. D. Thilmany (2000). Farm labor demand: A meta-regression analysis of wage elasticities. Journal of Agricultural and Resource Economics\/ , 252--266

    2000

  36. [44]

    Gechter, M. (2022). Combining experimental and observational studies in meta-analysis: A debiasing approach. Working paper, Pennylvania State University and London School of Economics

    2022

  37. [45]

    Hirano, J

    Gechter, M., K. Hirano, J. Lee, M. Mahmud, O. Mondal, J. Morduch, S. Ravindran, and A. S. Shonchoy (2024). Selecting experimental sites for external validity. arXiv preprint arXiv:2405.13241\/

    2024 arXiv

  38. [46]

    Gerber, A. S. and D. P. Green (2012). Field Experiments: Design, Analysis, and Interpretation . New York: W. W. Norton & Company

    2012

  39. [47]

    Ghanem, D., P. H. Sant'Anna, and K. W \"u thrich (2022). Selection and parallel trends. arXiv preprint arXiv:2203.09001\/

    2022 arXiv

  40. [48]

    Higbee, S. D. (2024). Experimental design for policy choice

    2024

  41. [49]

    Hu, Y., H. Zhu, E. Brunskil, and S. Wager (2024). Minimax-regret sample selection in randomized experiments. In Proceedings of the 25th ACM Conference on Economics and Computation , pp.\ 1209--1235

    2024

  42. [50]

    Kallus, N. (2018). Optimal a priori balance in the design of controlled experiments. Journal of the Royal Statistical Society Series B: Statistical Methodology\/ 80\/ (1), 85--112

    2018

  43. [51]

    Kallus, N., A. M. Puli, and U. Shalit (2018). Removing hidden confounding by experimental grounding. Advances in neural information processing systems\/ 31

    2018

  44. [52]

    Kasy, M. (2016). Why experimenters might not always want to randomize, and what they could do instead. Political Analysis\/ 24\/ (3), 324--338

    2016

  45. [53]

    Kasy, M. and A. Sautmann (2019). Adaptive treatment assignment in experiments for policy choice

    2019

  46. [54]

    Katz, J. and H. Allcott (2025). Digital media mergers: Theory and application to facebook-instagram. Technical report, Working paper

    2025

  47. [55]

    Kempthorne, P. J. (1988). Controlling risks under different loss functions: The compromise decision problem. The Annals of Statistics\/ 16\/ (4), 1594--1608

    1988

  48. [56]

    Kiefer, J. and J. Wolfowitz (1959). Optimum designs in regression problems. The annals of mathematical statistics\/ 30\/ (2), 271--294

    1959

  49. [57]

    Kitagawa, T. and A. Tetenov (2018). Who should be treated? E mpirical welfare maximization methods for treatment choice. Econometrica\/ 86\/ (2), 591--616

    2018

  50. [58]

    Gaduh, T

    Kreindler, G., A. Gaduh, T. Graff, R. Hanna, and B. A. Olken (2023). Optimal public transportation networks: Evidence from the world's largest bus rapid transit system in jakarta. Technical report, National Bureau of Economic Research

    2023

  51. [59]

    Kwon, S. and L. Sun (2025). Estimating treatment effects under bounded heterogeneity. arXiv preprint arXiv:2510.05454\/

    2025 arXiv

  52. [60]

    Larroucau, T., I. Rios, A. Fabre, and C. Neilson (2024). College application mistakes and the design of information policies at scale. Unpublished paper, Arizona State University, Tempe\/

    2024

  53. [61]

    List, J. A., S. Sadoff, and M. Wagner (2011). So you want to run an experiment, now what? some simple rules of thumb for optimal experimental design. Experimental Economics\/ 14\/ (4), 439--457

    2011

  54. [62]

    Tommasi, and P

    L \'o pez-Fidalgo, J., C. Tommasi, and P. C. Trandafir (2007). An optimal experimental design criterion for discriminating between non-normal models. Journal of the Royal Statistical Society Series B: Statistical Methodology\/ 69\/ (2), 231--242

    2007

  55. [63]

    Manski, C. (2004). Statistical treatment rules for heterogeneous populations. Econometrica\/ 72\/ (4), 1221--1246

    2004

  56. [64]

    Manski, C. F. (1990). Nonparametric bounds on treatment effects. The American Economic Review\/ 80\/ (2), 319--323

    1990

  57. [65]

    Manski, C. F. (2025). Using limited trial evidence to credibly choose treatment dosage when efficacy and adverse effects weakly increase with dose. Epidemiology\/ 36\/ (1), 60--65

    2025

  58. [66]

    Manski, C. F. and A. Tetenov (2007). Admissible treatment rules for a risk-averse planner with experimental data on an innovation. Journal of Statistical Planning and Inference\/ 137\/ (6), 1998--2010

    2007

  59. [67]

    Manski, C. F. and A. Tetenov (2016). Sufficient trial size to inform clinical practice. Proceedings of the National Academy of Sciences\/ 113\/ (38), 10518--10523

    2016

  60. [68]

    Meghir, C., A. M. Mobarak, C. Mommaerts, and M. Morten (2022). Migration and informal insurance: Evidence from a randomized controlled trial and a structural model. The Review of Economic Studies\/ 89\/ (1), 452--480

    2022

  61. [69]

    Qiu, and J

    Montiel Olea, J., C. Qiu, and J. Stoye (2023). Decision theory for treatment choice with partial identification. Preprint\/

    2023

  62. [70]

    Morita, S., P. F. Thall, and K. Takeda (2017). A simulation study of methods for selecting subgroup-specific doses in phase 1 trials. Pharmaceutical statistics\/ 16\/ (2), 143--156

    2017

  63. [71]

    Muralidharan, K. and P. Niehaus (2017). Experimentation at scale. Journal of Economic Perspectives\/ 31\/ (4), 103--24

    2017

  64. [72]

    Romero, and K

    Muralidharan, K., M. Romero, and K. W \"u thrich (2020). Factorial designs, model selection, and (incorrect) inference in randomized experiments. NBER Working Paper

    2020

  65. [73]

    Niederle, M. (2025). Experiments: Why, how, and a users guide for producers as well as consumers. Technical report, National Bureau of Economic Research

    2025

  66. [74]

    Olea, J. L. M., B. Prallon, C. Qiu, J. Stoye, and Y. Sun (2024). Externally valid selection of experimental sites via the k-median problem. arXiv preprint arXiv:2408.09187\/

    2024 arXiv

  67. [75]

    Porter, S., T. A. Murray, and A. Eaton (2024). Phase i/ii design for selecting subgroup-specific optimal biological doses for prespecified subgroups. Statistics in Medicine\/ 43\/ (28), 5401--5411

    2024

  68. [76]

    Singh, and D

    Rambachan, A., R. Singh, and D. Viviano (2024). Program evaluation with remotely sensed outcomes. arXiv preprint arXiv:2411.10959\/

    2024 arXiv

  69. [77]

    Ravallion, M. (2012). Fighting poverty one experiment at a time: A review of abhijit banerjee and esther duflo's poor economics: A radical rethinking of the way to fight global poverty. Journal of Economic Literature\/ 50\/ (1), 103--114

    2012

  70. [78]

    Reeves, S. W., S. Lubold, A. G. Chandrasekhar, and T. H. McCormick (2024). Model-based inference and experimental design for interference using partial network data. arXiv preprint arXiv:2406.11940\/

    2024 arXiv

  71. [79]

    Rosenman, E. T. and A. B. Owen (2021). Designing experiments informed by observational studies. Journal of Causal Inference\/ 9\/ (1), 147--171

    2021

  72. [80]

    Rosenman, E. T., A. B. Owen, M. Baiocchi, and H. R. Banack (2022). Propensity score methods for merging observational and experimental datasets. Statistics in Medicine\/ 41\/ (1), 65--86

    2022

  73. [81]

    Russo, D. J., B. Van Roy, A. Kazerouni, I. Osband, Z. Wen, et al. (2018). A tutorial on thompson sampling. Foundations and Trends in Machine Learning\/ 11\/ (1), 1--96

    2018

  74. [82]

    Sacks, J. and D. Ylvisaker (1984). Some model robust designs in regression. The Annals of Statistics\/ , 1324--1348

    1984

  75. [83]

    Silvey, S. (2013). Optimal design: an introduction to the theory for parameter estimation , Volume 1. Springer Science & Business Media

    2013

  76. [84]

    Tabord-Meehan, M. (2018). Stratification trees for adaptive randomization in randomized controlled trials. arXiv preprint arXiv:1806.05127\/

    2018 arXiv

  77. [85]

    Todd, P. E. and K. I. Wolpin (2006). Assessing the impact of a school subsidy program in mexico: Using a social experiment to validate a dynamic behavioral model of child schooling and fertility. American Economic Review\/ 96\/ (5), 1384--1417

    2006

  78. [86]

    Tsirpitzi, R. E., F. Miller, and C.-F. Burman (2023). Robust optimal designs using a model misspecification term. Metrika\/ 86\/ (7), 781--804

    2023

  79. [87]

    Tsybakov, A. B. (1998). Pointwise and sup-norm sharp adaptive estimation of functions on the sobolev classes. The Annals of Statistics\/ 26\/ (6), 2420--2469

    1998

  80. [88]

    Viviano, D. (2020). Experimental design under network interference. arXiv preprint arXiv:2003.08421\/

    2020 arXiv

  81. [89]

    Wiens, D. P. (1998). Minimax robust designs and weights for approximately specified regression models with heteroscedastic errors. Journal of the American Statistical Association\/ 93\/ (444), 1440--1450

    1998

  82. [90]

    Kirubi, K

    Wingfield, T., B. Kirubi, K. Viney, D. Boccia, S. Atkins, et al. (2023). Experiences of conditional and unconditional cash transfers intended for improving health outcomes and health service use: a qualitative evidence synthesis. Cochrane Database of Systematic Reviews\/ (3)

    2023

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.