Pith. sign in

REVIEW 3 major objections 4 minor 69 references

Off-policy estimation with adaptively collected data: the power of online learning

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read With adaptively collected data, an AIPW estimator driven by a no-regret online learner achieves the instance-dependent optimal mean-squared error up to constants.

desk verdict A solid finite-sample oracle inequality and lower bound for OPE with adaptive data, but the advertised instance-dependent optimality is only proven under uniform overlap and the tabular analysis misses a convexity condition. read the letter →

arxiv 2411.12786 v1 pith:O2KJLZ6Q submitted 2024-11-19 stat.ML cs.LGmath.OCmath.STstat.TH

Jeonghwan Lee , Cong Ma This is my paper · ORCID
classification stat.MLcs.LGmath.OCmath.STstat.TH
keywords off-policyevaluationaveragetreatmenteffectadaptivedatacollectionAIPWestimatoronlinelearningnon-parametricregressionlocalminimaxlowerboundcontextualbandits
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Estimating a linear functional of the treatment effect — the average treatment effect or the value of a target policy — becomes delicate when the data were collected adaptively, because the behavior policy changes with what has been seen. This paper proves finite-sample mean-squared-error bounds for augmented inverse propensity weighting (AIPW) estimators in that setting, showing the excess error is a sequentially weighted discrepancy between the true treatment effect and the estimates plugged into the scores. The central move is a reduction: obtain those estimates from an online non-parametric regression algorithm, so the discrepancy becomes the algorithm's regret plus an approximation error measured in a weighted norm. With a no-regret learner and a well-specified model class, the AIPW estimator matches the local minimax lower bound up to a constant, establishing that online learning is the right way to handle nuisance estimation under adaptivity.

What carries the argument

The load-bearing object is the sequentially weighted $\ell^2$-norm $\|\varphi\|_{(n)}^2 := \frac{1}{n}\sum_{i=1}^n \mathbb{E}_{I^*}\left[ \frac{g^2(X_i,A_i)\,\varphi^2(X_i,A_i)}{(\pi_i^*)^2(X_i,O_{i-1};A_i)} \right]$, which weights each round's squared error by the squared inverse propensity of the behavior policy. The AIPW scores are $\hat\Gamma_i := \frac{g}{\pi_i^*}(Y_i - \hat\mu_i) + \langle g, \hat\mu_i\rangle$, with $\hat\mu_i$ depending only on the past trajectory; Lemma B.1 shows these scores are unbiased and uncorrelated across rounds. The key reduction (Eqs. B.11–B.13) rewrites the MSE's excess term as the expected cumulative loss of the online learner minus the noise variance, then upper-bounds it by the learner's static regret plus the approximation error $\inf_{\mu\in\mathcal{F}} \|\mu-\mu^*\|_{(n)}^2$. Plugging in online gradient descent for the tabular and linear cases yields regret $O(B^2\sqrt{n})$, and the general case follows the online non-parametric regression framework with sequential Rademacher complexity.

What would settle it

Simulate the tabular two-armed bandit with known propensities, bounded outcomes, and a fixed behavior policy so that $K=1$ in Eq. (4.1); at $n \ge 10^5$, compare the empirical MSE of the OGD-based AIPW estimator (averaged over many replications) with the lower bound $C(K) v_*^2/n$ and the right-hand side of the oracle inequality (3.13). If the empirical MSE consistently falls below the minimax lower bound, the claimed optimality is wrong; if it exceeds the upper bound, the oracle inequality is wrong.

Watch

Extended reading notes

Core claim

The paper's central claim is an oracle inequality (Theorem 3.2) for the AIPW estimator: when the treatment-effect estimates $\{\hat\mu_i\}$ are produced by an online regression algorithm $\mathcal{A}$ over a function class $\mathcal{F}$, the mean-squared error satisfies $\mathbb{E}_{I^*}[\{\hat\tau_n - \tau\}^2] \le \frac{1}{n}\left( v_*^2 + \frac{1}{n}\mathbb{E}_{I^*}[\mathrm{Regret}(n,\mathcal{F};\mathcal{A})] + \inf_{\mu\in\mathcal{F}} \|\mu-\mu^*\|_{(n)}^2 \right)$, where $v_*^2$ is the variance of an oracle AIPW estimator that knows $\mu^*$, Regret is the static regret of the online learner against $\mathcal{F}$, and the norm is the sequentially weighted $\ell^2$-norm. Complementing this, Theorem 4.1 gives the local minimax lower bound $M_n(C_\delta(I^*)) \ge C(K)\, v_*^2 / n$ over a neighborhood of problem instances. Together they show that no-regret online learning of the outcome model (Regret $= o(n)$) with a well-specified class achieves the instance-dependent optimal MSE up to a universal constant, generalizing the i.i.d. theory of [42] to adaptive data collection without cross-fitting. The paper instantiates the reduction in three cases: tabular models and linear function approximation via online gradient descent, and general function approximation via online non-parametric regression.

Load-bearing premise

All finite-sample guarantees assume strict overlap — the known propensities keep $|g/\pi_i^*| \le B$ almost surely — and the lower bound further assumes the adaptive policy stays within a constant factor $K$ of some reference Markov policy; if either constant degrades, the bounds blow up or the optimality claim loses its force.

Editorial extensions

If this is right

  • An AIPW estimator using any no-regret online regression algorithm for the outcome model achieves mean-squared error $v_*^2/n$ plus $o(1/n)$ when the model class is well specified, so efficient off-policy estimation is possible with adaptively collected data.
  • The estimator does not require cross-fitting; the online nature of the nuisance estimates automatically preserves the martingale structure of the AIPW scores, which are uncorrelated across rounds.
  • For tabular and linear outcome models, concrete rates follow: with OGD, the excess MSE is $O(B^2/\sqrt{n})$ in the tabular case and $O(B^2 R(L+R)/\sqrt{n})$ in the linear case.
  • For general function classes with sequential metric entropy exponent $p$, the excess MSE vanishes at rate $n^{-2/(p+2)}$ for $p<2$ and $n^{-1/p}$ for $p>2$, matching the minimax online-regression rates.
  • The local minimax lower bound identifies $v_*^2/n$ as the fundamental benchmark, so no alternative estimator can do better on all instances in a local neighborhood, up to a constant depending on the coverage $K$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the weighting norm $\|\cdot\|_{(n)}$ is defined through the actual adaptive behavior policy, the optimal variance $v_*^2$ is instance-dependent: the same estimator automatically adapts to how much exploration actually occurred, suggesting a principled way to account for the cost of exploration in adaptive experiments.
  • The uncorrelated-score property in Lemma B.1 suggests a direct route to anytime-valid inference: combined with time-uniform concentration bounds, the AIPW scores should yield confidence sequences for the ATE under adaptivity, an extension the paper does not develop.
  • The paper notes that the general function-approximation forecaster from the relaxation recipe is not directly computationally implementable in full generality; the reduction is therefore most immediately actionable through the OGD-based tabular and linear instantiations and through the constructive finite-class algorithms from the online regression literature.
  • The reduction only relies on regret and approximation error in the weighted norm, so other online learners (e.g., online mirror descent or online Newton step) should plug into the same oracle inequality unchanged, with their own regret rates determining the resulting MSE bound.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies estimation of a linear functional of the treatment effect (e.g., off-policy evaluation or average treatment effect) from adaptively collected contextual-bandit data. It analyzes a class of augmented inverse propensity weighting (AIPW) estimators whose nuisance treatment-effect estimates are produced by online learning algorithms. The main contributions are: (i) Theorem 3.1, a generic finite-sample MSE bound for AIPW estimators that exposes a sequentially weighted estimation error; (ii) Theorem 3.2, an oracle inequality connecting the MSE to the expected regret of an online non-parametric regression algorithm plus an approximation error; (iii) concrete regret bounds for tabular OGD (Theorem 3.3), linear OGD (Theorem 3.4), and general function approximation via online non-parametric regression (Appendix B.6); and (iv) Theorem 4.1, a local minimax lower bound of order v2*/n. The authors conclude that no-regret online learning attains instance-dependent optimality of AIPW in the large-sample regime.

Significance. If the main claims hold, the paper provides a clean and useful reduction: the estimation of a linear functional with adaptively collected data is controlled by a sequential weighted loss, and no-regret online regression yields finite-sample MSE bounds that match a local minimax lower bound up to lower-order terms. The strongest assets are Theorem 3.2, which is a compact and provable oracle inequality; the explicit tabular and linear OGD regret bounds; and the Le Cam lower bound that captures the variance v2*/n. The paper also ships proofs for the variance decomposition and for the local minimax argument. However, the optimality claim is only established under a strict-overlap assumption with a fixed constant B, and the tabular OGD result needs a convexity condition on F; these qualifications must be handled before the contribution can be assessed as fully valid.

major comments (3)
  1. [Section 3.5.1, Algorithm 3, Theorem 3.3] The tabular OGD result does not state that the function class F is convex. Online gradient descent with projection requires F to be a closed convex set; for an arbitrary subset F of [-L,L]^{X×A}, the projection map Π_F is not well-defined in the sense used in Algorithm 3, and the standard OGD regret bound invoked in the proof (Theorem 3.1 of [18]) does not apply. The theorem and algorithm should explicitly assume F is closed and convex, or specify an alternative projection-based argument that handles nonconvex F.
  2. [Abstract, Section 1.1, and Assumption 1 (Eq. 3.14)] The claimed instance-dependent optimality of no-regret online learning is only justified when the overlap constant B in Assumption 1 is fixed independently of n. If B is allowed to grow with n, the bounds degrade polynomially. For example, with binary actions, g(1)=1, g(0)=0, µ*=0, σ=1, and an adaptive policy with π*_1(1)=n^{-2} and π*_i(1)=1/2 for i≥2, Assumption 1 holds with B=n^2, v2*/n is of order n^2, while (3.13) with the regret bound (3.17) gives an MSE upper bound of order n^{2.5}. Thus the upper bound can be polynomially larger than the lower bound in the drifting-overlap regime that the introduction cites as a key motivation. The authors should qualify the optimality claim in the abstract and contributions to the strict-overlap regime, or extend the analysis to time-varying overlap.
  3. [Appendix C.3, Lemma C.2, Eq. (C.23)] The KL-divergence computation in the proof of Lemma C.2 is not correct as written. The perturbation Γ_i(s) changes the outcome kernel at a single round i, so the log-likelihood ratio is nonzero only at that round; Eq. (C.23) displays a sum over all n rounds and uses the same index i for the perturbation and the summation. The argument is repairable because the single surviving term is bounded by n||σ||^2_(n), but the display should be corrected and the notation should distinguish the perturbed round from the summation index.
minor comments (4)
  1. [Assumption 1, Eq. (3.14)] The statement says B is a 'universal constant' but does not explicitly say B is independent of n and known to the algorithm. Please state this explicitly, since the later optimality discussion depends on B being fixed.
  2. [Assumption 4, Eq. (4.3)] Assumption 4 divides by ||σ||_(n). The degenerate case ||σ||_(n)=0 should be handled separately or excluded, otherwise the assumption and Lemma C.2 are not well-defined in that case.
  3. [Section 3.5.3 and Appendix B.6] The 'general function approximation' instantiation is deferred entirely to the appendix, and Algorithm 5 is non-constructive because the relaxation R_n in Eq. (B.44) is generally not computable. The main text should state clearly what is and is not provided for this case, to avoid overclaiming an algorithmic instantiation.
  4. [Appendix C.3, Eq. (C.28)] The constant in the final lower bound of Lemma C.2 appears to be off by a factor of two relative to the standard Le Cam constant used in the proof (1/16 vs. 1/8). Please check the constants in the derivation from (C.27) to (C.28).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the proof chain is self-contained and the relied-upon regret and minimax results are external standard results.

full rationale

The derivation is not circular. Theorem 3.1 is a direct variance decomposition of the AIPW scores; the only nontrivial step is Lemma B.1 showing conditional unbiasedness and uncorrelatedness, which follows from the definition of the propensities and the adaptive sampling scheme. Theorem 3.2 then converts the estimation-error term (3.9) into the expected cumulative loss of the online regression algorithm plus the best-in-class approximation error, using the identity E[l_i(µ)|...] = g^2/(π*_i)^2 [σ^2 + (µ-µ*)^2] (Eq. 3.10). This is an algebraic reduction, not an assumption of the conclusion. The regret bounds in Theorems 3.3 and 3.4 are quoted from standard OCO results (Hazan 2016) with explicit Lipschitz constants computed under the stated strict-overlap assumption; the general-function-approximation bounds are from Rakhlin and Sridharan (2014), an external source. The lower bound Theorem 4.1 is proved by Le Cam's two-point method with explicit Gaussian perturbations (Eq. C.18) and does not invoke the upper bound; Assumptions 3 and 4 are explicitly stated conditions on the local neighborhood, not derived from the desired conclusion. No parameter is fitted and then relabeled as a prediction, and no load-bearing claim rests on a self-citation by the authors. The dependence on the overlap constant B is an explicitly acknowledged scope restriction (Section 3.5), not a circular step: it limits the regime in which the stated optimality is established, but does not make the stated theorems equivalent to their inputs.

Assumptions & free parameters 0 free parameters · 8 assumptions · 0 invented entities

The central results rely on standard OPE assumptions: known propensities, strict overlap, bounded outcomes, finite variance, and coverage conditions for the lower bound. No free parameters are fitted to data; the only numerical inputs are problem constants such as B, L, R, K. The unstated convexity of F in the tabular OGD theorem is the most fragile implicit premise.

assumptions (8)
  • domain assumption Contexts are i.i.d. from a fixed distribution; actions are drawn from a known, history-dependent behavior policy; outcomes have conditional mean µ*(x,a) and finite conditional variance.
    Section 2 data generating procedure; this defines the adaptive data model and the estimand.
  • domain assumption Propensity scores π*_i are known exactly to the statistician.
    Used to build AIPW scores and weighted ℓ2 norms; if propensities are estimated, the bounds do not apply.
  • domain assumption Strict overlap: |g(X_i,A_i)/π*_i(X_i,O_{i-1};A_i)| ≤ B almost surely (Assumption 1).
    Controls importance weights and Lipschitz constants in Theorems 3.3, 3.4, and Appendix B.6; without it the regret bounds are vacuous.
  • domain assumption Outcomes are bounded: Y ⊆ [-L,L].
    Section 3.5; used for Lipschitz gradient bounds and online regression losses.
  • domain assumption For tabular case, X×A is finite (Assumption 2); for linear case, ‖φ‖≤1 and θ∈B(0,R); for general case, F⊆[-L,L]^{X×A}.
    These structural assumptions define the three instantiations and the approximation classes.
  • domain assumption For lower bounds: H_{2→4}<∞ (Assumption 3), neighborhood lower bound on δ (Assumption 4), and existence of reference Markov policies with 1/K ≤ π_i/π*_i ≤ K (Eq. 4.1).
    Used in Le Cam constructions to keep perturbed instances inside the local neighborhood.
  • standard math Standard regret bounds: Theorem 3.1 of Hazan (2016) for OGD, and the online non-parametric regression results and relaxation recipe of Rakhlin and Sridharan (2014) and Rakhlin, Shamir and Sridharan (2012).
    The finite-sample MSE bounds inherit these external regret guarantees.
  • ad hoc to paper The function class F in the tabular OGD result is convex, or at least supports projection in the sense used by OGD.
    Algorithm 3 projects onto F; standard OGD analysis requires a convex decision set, but the paper does not state or prove this condition in Theorem 3.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Off-policy estimation with adaptively collected data: the power of online learning." pith.science (2026). https://pith.science/paper/O2KJLZ6Q

@misc{pith2026241112786,
  author       = {Pith},
  title        = {Pith review of: Off-policy estimation with adaptively collected data: the power of online learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O2KJLZ6Q}},
  note         = {Machine review of arXiv:2411.12786}
}
read the original abstract

We consider estimation of a linear functional of the treatment effect using adaptively collected data. This task finds a variety of applications including the off-policy evaluation (\textsf{OPE}) in contextual bandits, and estimation of the average treatment effect (\textsf{ATE}) in causal inference. While a certain class of augmented inverse propensity weighting (\textsf{AIPW}) estimators enjoys desirable asymptotic properties including the semi-parametric efficiency, much less is known about their non-asymptotic theory with adaptively collected data. To fill in the gap, we first establish generic upper bounds on the mean-squared error of the class of AIPW estimators that crucially depends on a sequentially weighted error between the treatment effect and its estimates. Motivated by this, we also propose a general reduction scheme that allows one to produce a sequence of estimates for the treatment effect via online learning to minimize the sequentially weighted estimation error. To illustrate this, we provide three concrete instantiations in (\romannumeral 1) the tabular case; (\romannumeral 2) the case of linear function approximation; and (\romannumeral 3) the case of general function approximation for the outcome model. We then provide a local minimax lower bound to show the instance-dependent optimality of the \textsf{AIPW} estimator using no-regret online learning algorithms.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 62 canonical work pages

  1. [18]

    Introduction to online convex optimization

    Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends® in Optimiza- tion, 2(3-4):157–325, 2016

    work page 2016

  2. [1]

    Effective evaluation using logged bandit feedback from multiple loggers

    Aman Agarwal, Soumya Basu, Tobias Schnabel, and Thorsten Joa chims. Effective evaluation using logged bandit feedback from multiple loggers. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining , pages 687–696, 2017

    work page 2017

  3. [2]

    Thompson sampling for contextu al bandits with linear payoffs

    Shipra Agrawal and Navin Goyal. Thompson sampling for contextu al bandits with linear payoffs. In International conference on machine learning , pages 127–135. PMLR, 2013

    2013

  4. [3]

    Finite-sample optimal e stimation and inference on average treatment effects under unconfoundedness

    Timothy B Armstrong and Michal Koles´ ar. Finite-sample optimal e stimation and inference on average treatment effects under unconfoundedness. Econometrica, 89(3):1141–1177, 2021

    work page 2021

  5. [4]

    Counter factual reasoning and learning systems: The example of computational advertising

    L´ eon Bottou, Jonas Peters, Joaquin Qui˜ nonero-Candela, D enis X Charles, D Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counter factual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research , 14(11), 2013

    work page 2013

  6. [5]

    Double/debiased/neyman machine learning of treatment eff ects

    Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esthe r Duflo, Christian Hansen, and Whitney Newey. Double/debiased/neyman machine learning of treatment eff ects. American Economic Review , 107(5):261–265, 2017

    work page 2017

  7. [6]

    Double/debiased machine learning for tre atment and structural parameters, 2018

    Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esthe r Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for tre atment and structural parameters, 2018

    work page 2018

  8. [7]

    Semiparametric e fficient inference in adaptive experiments

    Thomas Cook, Alan Mishler, and Aaditya Ramdas. Semiparametric e fficient inference in adaptive experiments. In Causal Learning and Reasoning , pages 1033–1064. PMLR, 2024

    work page 2024

Show all 69 references
  1. [8]

    Clip-ogd: An experimental design for adaptive neyman allocation in sequential experiments

    Jessica Dai, Paula Gradu, and Christopher Harshaw. Clip-ogd: An experimental design for adaptive neyman allocation in sequential experiments. Advances in Neural Information Processing Systems , 36, 2024

    2024

  2. [9]

    Do ubly Robust Policy Evaluation and Optimization

    Miroslav Dud ´ ık, Dumitru Erhan, John Langford, and Lihong Li. Do ubly Robust Policy Evaluation and Optimization. Statistical Science, 29(4):485 – 511, 2014

    2014

  3. [10]

    Doubly robust po licy evaluation and learning

    Miroslav Dud ´ ık, John Langford, and Lihong Li. Doubly robust po licy evaluation and learning. arXiv preprint arXiv:1103.4601, 2011

    2011 arXiv

  4. [11]

    Overlap in observational studies with high-dimensional covariates

    Alexander D’Amour, Peng Ding, Avi Feller, Lihua Lei, and Jasjeet Sekhon. Overlap in observational studies with high-dimensional covariates. Journal of Econometrics , 221(2):644–654, 2021

    2021

  5. [12]

    More robust doubly robust off- policy evaluation

    Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzade h. More robust doubly robust off- policy evaluation. In International Conference on Machine Learning , pages 1447–1456. PMLR, 2018

    2018

  6. [13]

    Off-policy evalua- tion with deficient support using side information

    Nicol` o Felicioni, Maurizio Ferrari Dacrema, Marcello Restelli, and P aolo Cremonesi. Off-policy evalua- tion with deficient support using side information. Advances in Neural Information Processing Systems , 35:30250–30264, 2022

    2022

  7. [14]

    On choosing and bounding pr obability metrics

    Alison L Gibbs and Francis Edward Su. On choosing and bounding pr obability metrics. International statistical review, 70(3):419–435, 2002. 13

    2002

  8. [15]

    Some limit theorems for empirical pro cesses

    Evarist Gin´ e and Joel Zinn. Some limit theorems for empirical pro cesses. The Annals of Probability , pages 929–989, 1984

    1984

  9. [16]

    Confidence intervals for policy evaluation in adaptive experiments.” arxiv e-prints

    Vitor Hadad, David A Hirshberg, Ruohan Zhan, Stefan Wager, a nd Susan Athey. Confidence intervals for policy evaluation in adaptive experiments.” arxiv e-prints. arXiv preprint arXiv:1911.02768 , 2019

    1911 arXiv

  10. [17]

    Confidence inter- vals for policy evaluation in adaptive experiments

    Vitor Hadad, David A Hirshberg, Ruohan Zhan, Stefan Wager, a nd Susan Athey. Confidence inter- vals for policy evaluation in adaptive experiments. Proceedings of the national academy of sciences , 118(15):e2014602118, 2021

    2021

  11. [19]

    Weighted average importance sampling and def ensive mixture distributions

    Tim Hesterberg. Weighted average importance sampling and def ensive mixture distributions. Techno- metrics, 37(2):185–194, 1995

    1995

  12. [20]

    Efficient estim ation of average treatment effects using the estimated propensity score

    Keisuke Hirano, Guido W Imbens, and Geert Ridder. Efficient estim ation of average treatment effects using the estimated propensity score. Econometrica, 71(4):1161–1189, 2003

    2003

  13. [21]

    A generalization of sa mpling without replacement from a finite universe

    Daniel G Horvitz and Donovan J Thompson. A generalization of sa mpling without replacement from a finite universe. Journal of the American statistical Association , 47(260):663–685, 1952

    1952

  14. [22]

    Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon

    Steven R. Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics , 49(2):1055 – 1080, 2021

    2021

  15. [23]

    Nonparametric estimation of average treatme nt effects under exogeneity: A review

    Guido W Imbens. Nonparametric estimation of average treatme nt effects under exogeneity: A review. Review of Economics and statistics , 86(1):4–29, 2004

    2004

  16. [24]

    Causal inference in statistics, social, and biomedical sci ences

    Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sci ences. Cambridge University Press, 2015

    2015

  17. [25]

    Truncated importance sampling

    Edward L Ionides. Truncated importance sampling. Journal of Computational and Graphical Statistics , 17(2):295–311, 2008

    2008

  18. [26]

    Policy lea rning” without”overlap: Pessimism and generalized empirical bernstein’s inequality

    Ying Jin, Zhimei Ren, Zhuoran Yang, and Zhaoran Wang. Policy lea rning” without”overlap: Pessimism and generalized empirical bernstein’s inequality. arXiv preprint arXiv:2212.09900 , 2022

    2022 arXiv

  19. [27]

    Optimal off- policy evaluation from multiple logging policies

    Nathan Kallus, Yuta Saito, and Masatoshi Uehara. Optimal off- policy evaluation from multiple logging policies. In International Conference on Machine Learning , pages 5247–5256. PMLR, 2021

    2021

  20. [28]

    Policy evaluation and optimization w ith continuous treatments

    Nathan Kallus and Angela Zhou. Policy evaluation and optimization w ith continuous treatments. In International conference on artificial intelligence and st atistics, pages 1243–1251. PMLR, 2018

    2018

  21. [29]

    Joseph D. Y. Kang and Joseph L. Schafer. Demystifying Double Robustness: A Comparison of Alterna- tive Strategies for Estimating a Population Mean from Incomplete Da ta. Statistical Science, 22(4):523 – 539, 2007

    2007

  22. [30]

    Off-po licy confidence sequences

    Nikos Karampatziakis, Paul Mineiro, and Aaditya Ramdas. Off-po licy confidence sequences. In Inter- national Conference on Machine Learning , pages 5301–5310. PMLR, 2021

    2021

  23. [31]

    Efficient adaptive experimental design for average treatment effect estimation

    Masahiro Kato, Takuya Ishihara, Junya Honda, and Yusuke Na rita. Efficient adaptive experimental design for average treatment effect estimation. arXiv preprint arXiv:2002.05308 , 2020. 14

    2002 arXiv

  24. [32]

    Irregular identification, support conditions, and inverse weight estima- tion

    Shakeeb Khan and Elie Tamer. Irregular identification, support conditions, and inverse weight estima- tion. Econometrica, 78(6):2021–2042, 2010

    2021

  25. [33]

    Asymptotically efficient ada ptive allocation rules

    Tze Leung Lai and Herbert Robbins. Asymptotically efficient ada ptive allocation rules. Advances in applied mathematics, 6(1):4–22, 1985

    1985

  26. [34]

    Bandit algorithms

    Tor Lattimore and Csaba Szepesv´ ari. Bandit algorithms . Cambridge University Press, 2020

    2020

  27. [35]

    Local metric learning for off-policy evaluation in contextua l bandits with continuous actions

    Haanvid Lee, Jongmin Lee, Yunseon Choi, Wonseok Jeon, Byung -Jun Lee, Yung-Kyun Noh, and Kee- Eung Kim. Local metric learning for off-policy evaluation in contextua l bandits with continuous actions. Advances in Neural Information Processing Systems , 35:3913–3925, 2022

    2022

  28. [36]

    Distribution-free assessment of population overlap in observational studies

    Lihua Lei, Alexander D’Amour, Peng Ding, Avi Feller, and Jasjeet Sekhon. Distribution-free assessment of population overlap in observational studies. Technical report, Working paper, Stanford University, 2021

    2021

  29. [37]

    Sharp high-probability sample complexities for policy evaluation with linear function approximation

    Gen Li and Weichen Wu. Sharp high-probability sample complexities for policy evaluation with linear function approximation. arXivorg, 2023

    2023

  30. [38]

    Toward minimaxoff-policy value estimation

    Lihong Li, R´ emi Munos, and Csaba Szepesv´ ari. Toward minimaxoff-policy value estimation. In Artificial Intelligence and Statistics , pages 608–616. PMLR, 2015

    2015

  31. [39]

    Statistical analysis with missing data , volume 793

    Roderick JA Little and Donald B Rubin. Statistical analysis with missing data , volume 793. John Wiley & Sons, 2019

    2019

  32. [40]

    Statistical infer ence for the mean outcome under a possibly non-unique optimal treatment strategy

    Alexander R Luedtke and Mark J Van Der Laan. Statistical infer ence for the mean outcome under a possibly non-unique optimal treatment strategy. Annals of statistics , 44(2):713, 2016

    2016

  33. [41]

    Min imax off-policy evaluation for multi-armed bandits

    Cong Ma, Banghua Zhu, Jiantao Jiao, and Martin J Wainwright. Min imax off-policy evaluation for multi-armed bandits. IEEE Transactions on Information Theory , 68(8):5314–5339, 2022

    2022

  34. [42]

    Off-polic y estimation of linear functionals: Non-asymptotic theory for semi-parametric efficiency

    Wenlong Mou, Martin J Wainwright, and Peter L Bartlett. Off-polic y estimation of linear functionals: Non-asymptotic theory for semi-parametric efficiency. arXiv preprint arXiv:2209.13075 , 2022

    2022 arXiv

  35. [43]

    Efficient counter factual learning from bandit feedback

    Yusuke Narita, Shota Yasui, and Kohei Yata. Efficient counter factual learning from bandit feedback. In Proceedings of the AAAI Conference on Artificial Intelligen ce, volume 33, pages 4634–4641, 2019

    2019

  36. [44]

    Offline policy evaluation in large action spaces via outcome-oriented action group ing

    Jie Peng, Hao Zou, Jiashuo Liu, Shaoming Li, Yibao Jiang, Jian Pei, a nd Peng Cui. Offline policy evaluation in large action spaces via outcome-oriented action group ing. In Proceedings of the ACM Web Conference 2023, pages 1220–1230, 2023

    2023

  37. [45]

    Online non-parametric regression

    Alexander Rakhlin and Karthik Sridharan. Online non-parametric regression. In Conference on Learning Theory, pages 1232–1264. PMLR, 2014

    2014

  38. [46]

    Seque ntial complexities and uniform mar- tingale laws of large numbers

    Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Seque ntial complexities and uniform mar- tingale laws of large numbers. Probability theory and related fields , 161:111–153, 2015

    2015

  39. [47]

    Relax and ra ndomize: From value to algorithms

    Sasha Rakhlin, Ohad Shamir, and Karthik Sridharan. Relax and ra ndomize: From value to algorithms. Advances in Neural Information Processing Systems , 25, 2012. 15

    2012

  40. [48]

    Comment: Performance of double-robust estimators when” inverse probability” weights ar e highly variable

    James Robins, Mariela Sued, Quanhong Lei-Gomez, and Andrea R otnitzky. Comment: Performance of double-robust estimators when” inverse probability” weights ar e highly variable. Statistical Science, 22(4):544–559, 2007

    2007

  41. [49]

    Semiparametric efficiency in multivariate regression models with missing data

    James M Robins and Andrea Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association , 90(429):122–129, 1995

    1995

  42. [50]

    Estimatio n of regression coefficients when some regressors are not always observed

    James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimatio n of regression coefficients when some regressors are not always observed. Journal of the American statistical Association , 89(427):846–866, 1994

    1994

  43. [51]

    Analysis o f semiparametric regression models for repeated outcomes in the presence of missing data

    James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. Analysis o f semiparametric regression models for repeated outcomes in the presence of missing data. Journal of the american statistical association , 90(429):106–121, 1995

    1995

  44. [52]

    A tutorial on thompson sampling

    Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osban d, Zheng Wen, et al. A tutorial on thompson sampling. Foundations and Trends ® in Machine Learning , 11(1):1–96, 2018

    2018

  45. [53]

    Off-policy evaluation for larg e action spaces via embeddings

    Yuta Saito and Thorsten Joachims. Off-policy evaluation for larg e action spaces via embeddings. arXiv preprint arXiv:2202.06317, 2022

    2022 arXiv

  46. [54]

    Off-policy ev aluation for large action spaces via conjunct effect modeling

    Yuta Saito, Qingyang Ren, and Thorsten Joachims. Off-policy ev aluation for large action spaces via conjunct effect modeling. In international conference on Machine learning , pages 29734–29759. PMLR, 2023

    2023

  47. [55]

    Lear ning from logged implicit exploration data

    Alex Strehl, John Langford, Lihong Li, and Sham M Kakade. Lear ning from logged implicit exploration data. Advances in neural information processing systems , 23, 2010

    2010

  48. [56]

    Doubly robust off-policy evaluation with shrinkage

    Yi Su, Maria Dimakopoulou, Akshay Krishnamurthy, and Miroslav Dud ´ ık. Doubly robust off-policy evaluation with shrinkage. In International Conference on Machine Learning , pages 9167–9176. PMLR, 2020

    2020

  49. [57]

    Cab: Continuous adaptive blend- ing for policy evaluation and learning

    Yi Su, Lequn Wang, Michele Santacatterina, and Thorsten Joa chims. Cab: Continuous adaptive blend- ing for policy evaluation and learning. In International Conference on Machine Learning , pages 6005–

  50. [58]

    The self-normalized estimator for counterfactual learning

    Adith Swaminathan and Thorsten Joachims. The self-normalized estimator for counterfactual learning. advances in neural information processing systems , 28, 2015

    2015

  51. [59]

    Data-efficient off-policy policy eva luation for reinforcement learn- ing

    Philip Thomas and Emma Brunskill. Data-efficient off-policy policy eva luation for reinforcement learn- ing. In International Conference on Machine Learning , pages 2139–2148. PMLR, 2016

    2016

  52. [60]

    On the likelihood that one unknown probability ex ceeds another in view of the evidence of two samples

    William R Thompson. On the likelihood that one unknown probability ex ceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285–294, 1933

    1933

  53. [61]

    The construction and analysis of adaptive group sequential designs

    Mark J van der Laan. The construction and analysis of adaptive group sequential designs. 2008

    2008

  54. [62]

    High-dimensional statistics: A non-asymptotic viewpoint , volume 48

    Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press, 2019. 16

    2019

  55. [63]

    Oracle-e fficient pessimism: Offline policy optimization in contextual bandits

    Lequn Wang, Akshay Krishnamurthy, and Alex Slivkins. Oracle-e fficient pessimism: Offline policy optimization in contextual bandits. In International Conference on Artificial Intelligence and St atistics, pages 766–774. PMLR, 2024

    2024

  56. [64]

    Optimal and ad aptive off-policy evaluation in contextual bandits

    Yu-Xiang Wang, Alekh Agarwal, and Miroslav Dudık. Optimal and ad aptive off-policy evaluation in contextual bandits. In International Conference on Machine Learning , pages 3589–3597. PMLR, 2017

    2017

  57. [65]

    Anytime-valid off-policy inference for contextual bandits

    Ian Waudby-Smith, Lili Wu, Aaditya Ramdas, Nikos Karampatziak is, and Paul Mineiro. Anytime-valid off-policy inference for contextual bandits. ACM/JMS Journal of Data Science , 1(3):1–42, 2024

    2024

  58. [66]

    Asymptotic inference of causal effects with o bservational studies trimmed by the estimated propensity scores

    S Yang and P Ding. Asymptotic inference of causal effects with o bservational studies trimmed by the estimated propensity scores. Biometrika, 105(2):487–493, 03 2018

    2018

  59. [67]

    Off-policy evaluation via adaptive weighting with data from contextual bandits

    Ruohan Zhan, Vitor Hadad, David A Hirshberg, and Susan Athey . Off-policy evaluation via adaptive weighting with data from contextual bandits. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 2125–2135, 2021

    2021

  60. [68]

    Policy learning with adaptively collected data

    Ruohan Zhan, Zhimei Ren, Susan Athey, and Zhengyuan Zhou. Policy learning with adaptively collected data. Management Science , 2023

    2023

  61. [69]

    Inference for batched bandits

    Kelly Zhang, Lucas Janson, and Susan Murphy. Inference for batched bandits. Advances in neural information processing systems , 33:9818–9829, 2020. 17 A Some elementary inequalities and their proofs The following lemma plays a key role in the truncation arguments used in estab...

    2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.