Pith. sign in

REVIEW 2 major objections 3 minor 71 references

Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes

T0 review · 2 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new estimator, DRL, is proven to attain the best possible accuracy for off-policy evaluation under Markov decision processes.

desk verdict The MDP efficiency bound and DRL construction are genuinely new, but the main efficiency theorem as stated has a real cross-time product gap that the current proof does not close. read the letter →

arxiv 1908.08526 v3 pith:J56PJ3VI submitted 2019-08-22 cs.LG cs.AIstat.ML

Nathan Kallus , Masatoshi Uehara This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords off-policyevaluationsemiparametricefficiencyMarkovdecisionprocessdoublereinforcementlearningmarginalizeddensityratiomachineefficientinfluencefunction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Off-policy evaluation asks how well a new decision policy would have performed using only data logged under a different behavior policy. This paper pins down the best possible statistical accuracy for that problem when the environment is a Markov decision process, and then constructs an estimator that attains that accuracy. The key move is to replace cumulative importance weights with marginalized density ratios, which exploit the memoryless structure of MDPs and make the variance bound polynomial rather than exponential in horizon. The resulting estimator, double reinforcement learning (DRL), is proven efficient when its two learned components converge at fourth-root rates, and it remains consistent if only one of those components is right.

What carries the argument

The machine that carries the argument is the efficient influence function of the policy value under the MDP model, $$\phi_{M2}(H)=-\$rho^{{\pi_e}}$+\sum_{t=0}^T\left(\mu_t(s_t,a_t)(r_t-q_t(s_t,a_t))+\mu_{t-1}(s_{t-1},a_{t-1})v_t(s_t)\right),$$ with $\mu_t(s_t,a_t)=E[\lambda_t\mid s_t,a_t]$ the marginalized density ratio, the conditional expectation of the cumulative importance weight given the current state and action. DRL is the cross-fold plug-in estimator of this functional: estimates of $q_t$ and $\mu_t$ are formed on one half of the data and scored on the other half. The identity doing the work is that the plug-in error decomposes into a term vanishing through the conditional moment equation $E[r_t+q_{t+1}-q_t\mid s_t,a_t]=0$ and a symmetric term, so only the product of the two nuisance errors survives; that is what lets a product condition replace a Donsker condition.

What would settle it

Simulate an MDP with known transition and reward probabilities and a known behavior policy, estimate $\mu_t$ and $q_t$ by kernel regression so the product condition holds, and compute the normalized squared error of DRL(M2) as $n$ grows; if it does not converge to the theoretical efficiency bound $E[\phi_{M2}^2]$, the efficiency claim is contradicted. Alternatively, add a hidden state variable to the same simulator so transitions depend on the past: as the hidden-state influence grows, DRL(M2)'s error should fail to track the MDP bound, confirming that the Markov assumption is the load-bearing premise.

Watch

Extended reading notes

Core claim

The paper derives, for the first time, the semiparametric efficiency bound and efficient influence function for off-policy evaluation under the Markov decision process model. The efficient influence function is $\phi_{M2}(H)=-\rho^{\pi_e}+\sum_{t=0}^T(\mu_t(r_t-q_t)+\mu_{t-1}v_t)$, with the cumulative importance weight $\lambda_t$ replaced by the marginalized density ratio $\mu_t=E[\lambda_t\mid s_t,a_t]$. The paper then builds DRL by cross-fold plug-in of estimated q-functions and marginalized density ratios into this influence function. Theorem 12 states that if the product of the L2 errors of the two nuisance estimates is $o_p(n^{-1/2})$ and each error is $o_p(1)$, DRL achieves the efficiency bound under the MDP model, making it the first globally efficient off-policy evaluation estimator for MDPs. The same construction also gives double robustness: if either the q-model or the density-ratio model is correctly specified, DRL is at least consistent, and under rate conditions it is $\sqrt{n}$-consistent.

Load-bearing premise

The data truly come from a Markov decision process: transitions, rewards, and the behavior policy depend only on the current state and action, with no hidden state or history dependence.

Editorial extensions

If this is right

  • For long-horizon problems modeled as MDPs, DRL's variance grows polynomially with horizon rather than exponentially, making evaluation feasible where cumulative importance sampling and marginalized importance sampling fail.
  • DRL gives a template for using flexible machine learning in efficient estimation: any pair of nuisance learners whose product error is $o_p(n^{-1/2})$ yields efficiency, with no Donsker or bounded-entropy restriction on the learners.
  • If only the q-function model or only the marginalized density-ratio model is correctly specified, DRL remains consistent and, under rate conditions, $\sqrt{n}$-consistent, protecting against misspecification of one component.
  • In tabular MDPs, DRL is efficient under minimal conditions, so it can serve as a gold-standard estimator for finite-state benchmark problems.
  • When the behavior policy is known, consistency of the q-function alone is enough for DRL(M1) to be $\sqrt{n}$-consistent even under an MDP, though its asymptotic variance is generally larger than DRL(M2).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension is a hybrid estimator that checks whether the data are approximately Markov and then selects DRL(M2) or DRL(M1), because the paper's Remark 11 already notes the M1 version is more robust while the M2 version is more efficient.
  • The polynomial-versus-exponential variance gap suggests that in MDP-like environments, DRL should be preferred over importance sampling even at moderate horizons; this can be verified empirically by measuring RMSE ratios at increasing horizon while holding the process fixed.
  • The product-rate condition implies that slow convergence in one nuisance can be compensated by fast convergence in the other, which could guide practical tuning of q-estimators versus density-ratio estimators in a given application.
  • The same marginalization idea may transfer to other sequential estimands, such as dynamic treatment effects, whenever conditional independence given current state holds, though the paper does not develop that application.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. This paper studies off-policy evaluation (OPE) in reinforcement learning and derives semiparametric efficiency bounds and efficient influence functions for the target policy value under non-Markov (NMDP) and Markov (MDP) decision process models, with unknown or known behavior policy. It then constructs DRL, a cross-fitted estimator that plugs estimated q-functions and (marginalized) density ratios into the efficient influence function. The paper claims that DRL achieves the efficiency bound under MDP (and NMDP) when per-time products of L2 errors of the two nuisance estimates are o_p(n^{-1/2}), and that it is doubly robust when one nuisance is consistent; finite-sample guarantees and experiments in OpenAI Gym are also provided.

Significance. The efficiency-bound results are a significant contribution: the M2 influence function and bound are new, the comparison showing the MDP bound is polynomial in horizon while the NMDP bound is generally exponential is an important insight, and the tangent-space proofs in the appendix are careful. The DRL construction is elegant and the reproducibility of the experiments (code provided) is a strength. However, the main efficiency theorem (Theorem 12) and its NMDP counterpart (Theorem 5) are not established as stated because the proofs do not control cross-time products of the nuisance errors; a repair is available by strengthening the rate condition to a uniform-over-time product condition. The paper's central claim is therefore plausible and likely fixable, but the current version overclaims relative to its proof.

major comments (2)
  1. [§3.2, Theorem 12, proof of Lemma 38] The rate condition in Theorem 12 is insufficient for the claimed conclusion. The proof of Lemma 38 bounds the plug-in bias by √n Σ_t O(‖μ̂_t−μ_t‖_2‖q̂_t−q_t‖_2), but the second summand in the displayed decomposition is (μ̂_{t−1}−μ_{t−1})(−v̂_t+v_t), which is only controlled by ‖μ̂_{t−1}−μ_{t−1}‖_2‖q̂_t−q_t‖_2, a cross-time product that the theorem's per-time hypothesis does not constrain. For example, with T=1 and ‖μ̂_0−μ_0‖_2=n^{-0.1}, ‖q̂_1−q_1‖_2=n^{-0.1}, ‖μ̂_1−μ_1‖_2=n^{-0.4}, ‖q̂_0−q_0‖_2=n^{-0.4}, every same-time product is n^{-0.5}=o_p(n^{-1/2}) but the cross-time product ‖μ̂_0−μ_0‖_2‖q̂_1−q_1‖_2=n^{-0.2}, so √n times the bias diverges. The same gap affects Theorem 5 (via Lemma 33), Theorem 8, and Theorem 14. To repair the proof, strengthen the hypothesis to a uniform condition such as (max_t ‖μ̂_t−μ_t‖_2)(max_t ‖q̂_t−q_t‖_2)=o_p(n^{-1/2}) (and analogously for λ); this is also the natural reading of the abstract's 'fourth-root rates' statement when rates are uniform in t.
  2. [Abstract and §3.1–§3.2, Theorems 5, 8, 12, 14] The formal rate conditions in the efficiency theorems are weaker than the abstract's informal 'fourth-root rates' claim, and the proof actually requires the stronger uniform-over-time version. As written, the theorems state only per-time product conditions, which the proof does not establish. Please either provide a proof under the stated per-time products (which the counterexample above shows is impossible in general) or restate the theorems with the uniform condition, and adjust the abstract and remarks to match the formal statements.
minor comments (3)
  1. [§3.2, Theorem 13] Theorem 13 contains a typo: the condition '‖q(j)_t − λ_t‖_2^2 ≤ κ2' should read '‖q(j)_t − q_t‖_2^2 ≤ κ2'.
  2. [§2, Theorem 4] The notation in the text following Theorem 4, 'Eπe[log(var(rt +vt+1 | H at))] = E [log(var(rt +vt+1 | H at))]', is confusing because the marginal distributions of H_at under πe and πb differ; please clarify whether this is an identity or an assumption.
  3. [§3, proof of Theorem 5] In the proof of Theorem 5, the decomposition of D1 has a sign inconsistency: the second term is written as (λ̂_{t−1}−λ_{t−1})(v̂_t−v_t) in one place and as (λ̂_{t−1}−λ_{t−1})(−v̂_t+v_t) in Lemma 33; this is harmless but should be made consistent.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: the MDP efficiency bound and DRL efficiency theorem rest on independent tangent-space derivations, not on fitted inputs or self-citational uniqueness.

full rationale

The paper's central derivation chain is non-circular. The semiparametric efficiency bound under M2 (Theorem 2) is obtained by direct pathwise-derivative and tangent-space calculations in Appendix B.2: the paper computes a gradient for the policy value, characterizes the tangent space, verifies orthogonality to the orthogonal tangent space, and then computes the variance of the efficient influence function. The DRL(M2) estimator is then defined by plugging cross-fitted estimates of the marginalized density ratio and q-function into that independently derived efficient influence function (Section 3.2), and Theorem 12 imposes product rate conditions on the nuisance error terms. There is no parameter fitted to a subset and then renamed a prediction, and no definition of the estimand in terms of the estimator. The self-citation to Kallus and Uehara (2019) is background on an NMDP-style doubly robust estimator and is not used to justify the MDP efficiency bound or the DRL(M2) construction. The cited rate results for nonparametric nuisance estimation (e.g., Ai and Chen 2003; Chen 2007) are external to the paper's own fitted values. The horizon-dependence comparison (Theorem 4) follows from Jensen's inequality and the overlap/reward assumptions, not from an assumed conclusion. One caveat outside the circularity dimension: the proof of Lemma 38 appears to bound a cross-time bias term, (μ̂_{t−1}−μ_{t−1})(v̂_t−v_t), by the same-time product ‖μ̂_t−μ_t‖‖q̂_t−q_t‖; if valid, this is a correctness gap in the proof of Theorem 12, not a circularity, because the theorem's conclusion is not an input to the derivation. Accordingly, the circularity score is minimal.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. It relies on standard semiparametric theory and on domain assumptions about the MDP structure, boundedness, and nuisance estimation rates, all stated explicitly.

assumptions (5)
  • standard math Standard semiparametric inference theory (pathwise differentiability, tangent space, regularity).
    Used throughout to define influence functions and efficiency bounds; see Section 1.2 and Appendix B.1.
  • domain assumption The data-generating process is a Markov decision process (transitions and rewards depend only on current state and action) and the behavior policy is Markovian.
    Defines model M2 in Section 1.1; required for Theorem 2 and the DRL(M2) efficiency result.
  • domain assumption Assumption 1 (sequential overlap): density ratios η_t and μ_t are bounded.
    Stated in Section 1.1; needed for the finite-sample bounds and to ensure the efficiency bound is finite; also used in Assumptions 3 and 4.
  • domain assumption Assumption 2 (bounded rewards): rewards are bounded by R_max.
    Stated in Section 1.1; used in the efficiency bound calculations and finite-sample guarantees.
  • domain assumption Nuisance estimators (q-functions and density ratios) satisfy rate conditions, e.g., product of L2 errors is o_p(n^{-1/2}).
    Assumed in Theorems 5 and 12; this is the key condition for achieving efficiency.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes." pith.science (2026). https://pith.science/paper/J56PJ3VI

@misc{pith2026190808526,
  author       = {Pith},
  title        = {Pith review of: Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J56PJ3VI}},
  note         = {Machine review of arXiv:1908.08526}
}
abstract

Off-policy evaluation (OPE) in reinforcement learning allows one to evaluate novel decision policies without needing to conduct exploration, which is often costly or otherwise infeasible. We consider for the first time the semiparametric efficiency limits of OPE in Markov decision processes (MDPs), where actions, rewards, and states are memoryless. We show existing OPE estimators may fail to be efficient in this setting. We develop a new estimator based on cross-fold estimation of $q$-functions and marginalized density ratios, which we term double reinforcement learning (DRL). We show that DRL is efficient when both components are estimated at fourth-root rates and is also doubly robust when only one component is consistent. We investigate these properties empirically and demonstrate the performance benefits due to harnessing memorylessness.

Figures

Figures reproduced from arXiv: 1908.08526 by the authors.

Figure 1
Figure 1. M1: Non-Markov decision process (NMDP) s0 a0 r0 s1 a1 r1 s2 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. M2: Markov decision process (MDP) O(T 2 ) = EffBd(M2) = EffBd(M2b) > EffBd(M2q) > > 2 Ω(T) = EffBd(M1) = EffBd(M1b) > EffBd(M1q) [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Relationship between the semiparametric efficienc [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

71 extracted references · 65 canonical work pages

  1. [1]

    Ai and X

    C. Ai and X. Chen. Efficient estimation of models with conditional moment restrictions containing unknown functions. Econometrica, 71: 0 1795--1843, 2003

    work page 2003

  2. [2]

    Ai and X

    C. Ai and X. Chen. Semiparametric efficiency bound for models of sequential moment restrictions containing unknown functions. Journal of Econometrics, 170: 0 442--457, 2012

    work page 2012

  3. [3]

    Antos, C

    A. Antos, C. Szepesv \'a ri, and R. Munos. Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path. Machine Learning, 71: 0 89--129, 2008

    work page 2008

  4. [4]

    P. L. Bartlett, O. Bousquet, and S. Mendelson. Local rademacher complexities. The Annals of Statistics, 33: 0 1497--1537, 2005

    work page 2005

  5. [5]

    Benkeser and M

    D. Benkeser and M. van der Laan. The highly adaptive lasso estimator. In 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA), volume 2016, pages 689--696. IEEE, 2016

    work page 2016

  6. [6]

    D. P. Bertsekas. Dynamic programming and optimal control. Athena Scientific optimization and computation series. Athena Scientific, Belmont, Mass, 4th ed. edition, 2012

    work page 2012

  7. [8]

    Bibaut, I

    A. Bibaut, I. Malenica, N. Vlassis, and M. van der Laan. More efficient off-policy evaluation through regularized targeted learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 654--663, 2019

    work page 2019

  8. [9]

    A. F. Bibaut and M. J. van der Laan. Fast rates for empirical risk minimization over c adl ag functions with bounded sectional variation norm. arXiv preprint arXiv:1907.09244, 2019 b

    arXiv 1907

Show all 71 references
  1. [10]

    P. J. Bickel, C. A. J. Klaassen, Y. Ritov, and J. A. Wellner. Efficient and Adaptive Estimation for Semiparametric Models. Springer, 1998

    1998

  2. [11]

    C. G. Bowsher and P. S. Swain. Identifying sources of variation and the flow of information in biochemical networks. Proceedings of the National Academy of Sciences, 109, 2012

    2012

  3. [12]

    Brockman, V

    G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, , and W. Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016

    2016 arXiv

  4. [13]

    Chakraborty and E

    B. Chakraborty and E. Moodie. Statistical methods for dynamic treatment regimes. Springer, 2013

    2013

  5. [14]

    Chamberlain

    G. Chamberlain. Comment: Sequential moment restrictions in panel data. Journal of Business & Economic Statistics, 10: 0 20--26, 1992

    1992

  6. [15]

    X. Chen. Chapter 76 large sample sieve estimation of semi-nonparametric models. Handbook of Econometrics, 6: 0 5549--5632, 2007

    2007

  7. [16]

    Chernozhukov, D

    V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins. Double/debiased machine learning for treatment and structural parameters. Econometrics Journal, 21: 0 C1--C68, 2018

    2018

  8. [17]

    I. n. D\'iaz. Machine learning in the estimation of causal effects: targeted minimum loss-based estimation and double/debiased machine learning. Biostatistics, 2019

    2019

  9. [18]

    Dudik, D

    M. Dudik, D. Erhan, J. Langford, and L. Li. Doubly robust policy evaluation and optimization. Statistical Science, 29: 0 485--511, 2014

    2014

  10. [19]

    Ertefaie and R

    A. Ertefaie and R. L. Strawderman. Constructing dynamic treatment regimes over indefinite time horizons . Biometrika, 105: 0 963--977, 2018

    2018

  11. [20]

    Farajtabar, Y

    M. Farajtabar, Y. Chow, and M. Ghavamzadeh. More robust doubly robust off-policy evaluation. In Proceedings of the 35th International Conference on Machine Learning, pages 1447--1456, 2018

    2018

  12. [21]

    Gottesman, F

    O. Gottesman, F. Johansson, M. Komorowski, A. Faisal, D. Sontag, F. Doshi-Velez, and L. A. Celi. Guidelines for reinforcement learning in healthcare. Nat Med, 25: 0 16--18, 2019

    2019

  13. [22]

    Gy \"o rfi, M

    L. Gy \"o rfi, M. Kohler, A. Krzyzak, and H. Walk. A distribution-free theory of nonparametric regression. Springer Science & Business Media, 2006

    2006

  14. [23]

    J. Hahn. Efficient estimation of panel data models with sequential moment restrictions. Journal of Econometrics, 79: 0 1--21, 1997

    1997

  15. [24]

    J. Hahn. On the role of the propensity score in efficient semiparametric estimation of average treatment effects. Econometrica, 66: 0 315--331, 1998

    1998

  16. [25]

    L. P. Hansen. Large sample properties of generalized method of moments estimators. Econometrica, 50: 0 1029--1054, 1982

    1982

  17. [26]

    L. P. Hansen, J. Heaton, and A. Yaron. Finite-sample properties of some alternative gmm estimators. Journal of Business & Economic Statistics, 14: 0 262--280, 1996

    1996

  18. [27]

    Hernan and J

    M. Hernan and J. Robins. Causal Inference. Boca Raton: Chapman & Hall/CRC, 2019

    2019

  19. [28]

    Hirano, G

    K. Hirano, G. W. Imbens, and G. Ridder. Efficient estimation of average treatment effects using the estimated propensity score. Econometrica, 71: 0 1161--1189, 2003

    2003

  20. [29]

    Imaizumi and K

    M. Imaizumi and K. Fukumizu. Deep neural networks learn non-smooth functions effectively. arXiv preprint arXiv:1802.04474, 2018

    2018 arXiv

  21. [30]

    Jiang and L

    N. Jiang and L. Li. Doubly robust off-policy value evaluation for reinforcement learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning, pages 652--661, 2016

    2016

  22. [31]

    Kallus and M

    N. Kallus and M. Uehara. Intrinsically efficient, stable, and bounded off-policy evaluation for reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019

    2019

  23. [32]

    Khosravi, G

    K. Khosravi, G. Lewis, and V. Syrgkanis. Non-parametric inference adaptive to intrinsic dimension. arXiv preprint arXiv:1901.03719, 2019

    1901 arXiv

  24. [33]

    C. A. J. Klaassen. Consistent estimation of the influence function of locally asymptotically linear estimators. The Annals of Statistics, 15: 0 1548--1562, 1987

    1987

  25. [34]

    M. R. Kosorok. Introduction to Empirical Processes and Semiparametric Inference. Springer Series in Statistics. Springer New York, New York, NY, 2008

    2008

  26. [35]

    Lagoudakis and R

    M. Lagoudakis and R. Parr. Least-squares policy iteration. Journal of Machine Learning Research, 4: 0 1107--1149, 2004

    2004

  27. [36]

    H. Le, C. Voloshin, and Y. Yue. Batch policy learning under constraints. In International Conference on Machine Learning, pages 3703--3712, 2019

    2019

  28. [37]

    L. Li, R. Munos, and C. Szepesvari. Toward minimax off-policy value estimation. In Proceedings of the 18th International Conference on Artificial Intelligence and Statistics, pages 608--616, 2015

    2015

  29. [38]

    Li and J

    Q. Li and J. S. Racine. Nonparametric econometrics : theory and practice. Princeton University Press, Princeton, N.J., 2007

    2007

  30. [39]

    Q. Liu, L. Li, Z. Tang, and D. Zhou. Breaking the curse of horizon: Infinite-horizon off-policy estimation. In Advances in Neural Information Processing Systems 31, pages 5356--5366. 2018

    2018

  31. [40]

    D. J. Luckett, E. B. Laber, A. R. Kahkoska, D. M. Maahs, E. Mayer-Davis, and M. R. Kosorok. Estimating dynamic treatment regimes in mobile health using v-learning. Journal of the American Statistical Association, pages 1--34, 2018

    2018

  32. [41]

    A. R. Mahmood, H. P. van Hasselt, and R. S. Sutton. Weighted importance sampling for off-policy learning with linear function approximation. In Advances in Neural Information Processing Systems 27, pages 3014--3022. 2014

    2014

  33. [42]

    Mandel, Y

    T. Mandel, Y. Liu, S. Levine, E. Brunskill, and Z. Popovic. Off-policy evaluation across representations with applications to educational games. In Proceedings of the 13th International Conference on Autonomous Agentsand Multi-agent Systems, pages 1077--1084, 2014

    2014

  34. [43]

    Mannor, D

    S. Mannor, D. Simester, P. Sun, and J. N. Tsitsiklis. Bias and variance approximation in value function estimates. Management Science, 53 0 (2): 0 308--322, 2007

    2007

  35. [44]

    Munos, T

    R. Munos, T. Stepleton, A. Harutyunyan, and M. Bellemare. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems 29, pages 1054--1062. 2016

    2016

  36. [45]

    S. A. Murphy. Optimal dynamic treatment regimes. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 65: 0 331--355, 2003

    2003

  37. [46]

    S. A. Murphy, M. J. van der Laan, and J. M. Robins. Marginal mean models for dynamic regimes. Journal of the American Statistical Association, 96: 0 1410--1423, 2001

    2001

  38. [47]

    W. K. Newey and D. L. Mcfadden. Large sample estimation and hypothesis testing. Handbook of Econometrics, IV: 0 2113--2245, 1994

    1994

  39. [48]

    Precup, R

    D. Precup, R. Sutton, and S. Singh. Eligibility traces for off-policy policy evaluation. In Proceedings of the 17th International Conference on Machine Learning, pages 759--766, 2000

    2000

  40. [49]

    Rahimi and B

    A. Rahimi and B. Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems 20, pages 1177--1184. 2008

    2008

  41. [50]

    J. Robins. A new approach to causal inference in mortality studies with a sustained exposure period—application to control of the healthy worker survivor effect. Mathematical modelling, 7 0 (9-12): 0 1393--1512, 1986

    1986

  42. [51]

    J. M. Robins. Marginal structural models versus structural nested models as tools for causal inference. In Statistical models in epidemiology, the environment, and clinical trials, pages 95--133. Springer, 2000

    2000

  43. [52]

    J. M. Robins, M. A. Hern \'a n, and B. Brumback. Marginal structural models and causal inference in epidemiology. Epidemiology, 11: 0 551, 2000

    2000

  44. [53]

    Rotnitzky and S

    A. Rotnitzky and S. Vansteelandt. Double-robust methods. In Handbook of missing data methodology. In Handbooks of Modern Statistical Methods, pages 185--212. Chapman and Hall/CRC, 2014

    2014

  45. [54]

    Rotnitzky, E

    A. Rotnitzky, E. Smucler, and J. Robins. Characterization of parameters with a mixed bias property. arXiv preprint arXiv:1509.02556, 2019

    2019 arXiv

  46. [55]

    Scharfstein, A

    D. Scharfstein, A. Rotnizky, and J. M. Robins. Adjusting for nonignorable dropout using semi-parametric models. Journal of the American Statistical Association, 94: 0 1096--1146, 1999

    1999

  47. [56]

    X. Shen. On methods of sieves and penalization. The Annals of Statistics, 25: 0 2555--2591, 1997

    1997

  48. [57]

    C. J. Stone. Rejoinder: The use of polynomial splines and their tensor products in multivariate function estimation. The Annals of Statistics, 22: 0 179--184, 1994

    1994

  49. [58]

    R. S. Sutton. Reinforcement learning : an introduction. MIT Press, Cambridge, Mass., 2018

    2018

  50. [59]

    Swaminathan and T

    A. Swaminathan and T. Joachims. The self-normalized estimator for counterfactual learning. In Advances in Neural Information Processing Systems 28, pages 3231--3239. 2015

    2015

  51. [60]

    Thomas and E

    P. Thomas and E. Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning, pages 2139--2148, 2016

    2016

  52. [61]

    Tripathi

    G. Tripathi. A matrix extension of the Cauchy-Schwarz inequality . Economics Letters, 63: 0 1--3, 1999

    1999

  53. [62]

    A. A. Tsiatis. Semiparametric Theory and Missing Data. Springer Series in Statistics. Springer New York, New York, NY, 2006

    2006

  54. [63]

    M. J. van der Laan and J. M. Robins. Unified Methods for Censored Longitudinal Data and Causality. Springer Series in Statistics,. Springer New York, New York, NY, 2003

    2003

  55. [64]

    A. W. van der Vaart. On differentiable functionals. Ann. Statist., 19: 0 178--204, 03 1991

    1991

  56. [65]

    A. W. van der Vaart. Asymptotic statistics. Cambridge University Press, Cambridge, UK, 1998

    1998

  57. [66]

    A. W. van der Vaart. Semiparametric Statistics. Lecture Notes in Mathematics ; 1781. Springer Berlin Heidelberg, Berlin, Heidelberg, 2002

    2002

  58. [67]

    Vermeulen

    K. Vermeulen. Semiparametric efficiency. Gent, Faculteit WetenschappenVakgroep Toegepaste Wiskunde en Informatica, 2010

    2010

  59. [68]

    Wager and G

    S. Wager and G. Walther. Adaptive concentration of regression trees, with application to random forests. arXiv preprint arXiv:1503.06388, 2016

    2016 arXiv

  60. [69]

    T. Xie, Y. Ma, and Y.-X. Wang. Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling. In Advances in Neural Information Processing Systems 32, pages 9665--9675. 2019

    2019

  61. [70]

    Yin and Y.-X

    M. Yin and Y.-X. Wang. Asymptotically efficient off-policy evaluation for tabular reinforcement learning. In Proceedings of the 23nd International Workshop on Artificial Intelligence and Statistics (To appear), 2020

    2020

  62. [71]

    Zhang, A

    B. Zhang, A. A. Tsiatis, E. B. Laber, and M. Davidian. Robust estimation of optimal dynamic treatment regimes for sequential treatment decisions. Biometrika, 100: 0 681--694, 2013

    2013

  63. [72]

    Zheng and M

    W. Zheng and M. J. van der Laan. Cross-validated targeted minimum-loss-based estimation. In Targeted Learning: Causal Inference for Observational and Experimental Data, Springer Series in Statistics, pages 459--474. Springer New York, New York, NY, 2011

    2011

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.