Pith. sign in

REVIEW 3 major objections 5 minor 78 references

Off-Policy Evaluation Under Nonignorable Missing Data

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Nonignorable missingness makes ordinary off-policy value estimates biased, and an inverse-probability-weighted estimator built on a shadow variable restores consistency and valid confidence intervals.

desk verdict Real contribution for fixed-T OPE under MNAR, but the claimed T→∞ consistency and asymptotic normality are invalid under monotone dropout; needs theorem restatement and a linked codebase before I would sign off. read the letter →

arxiv 2507.06961 v1 pith:4AGNHOLO submitted 2025-07-09 stat.ML cs.LG

Han Wang , Yang Xu , Wenbin Lu , Rui Song This is my paper · ORCID
classification stat.MLcs.LG MSC 62D0562F12
keywords off-policyevaluationnonignorablemissingnessmissingnotatrandomshadowvariableinverseprobabilityweightingdropoutpropensityvaluefunctioninferencereinforcementlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Off-policy evaluation estimates the value of a target policy from logged trajectories, and logged trajectories are often cut short by dropout. The paper proves that when dropout is ignorable (missing at random), the usual complete-case estimator remains consistent; when dropout is nonignorable, because it depends on the unobserved reward or next state, the usual estimator is biased and the bias does not disappear as the sample grows. To fix that regime, the paper weights every observed transition by the inverse probability of observing it, estimating that probability through a moment equation that uses a shadow variable, an auxiliary covariate that predicts the unobserved outcome but is unrelated to dropout. It then proves the inverse-probability-weighted value estimator is consistent and asymptotically normal as either the number of trajectories or the horizon grows, and demonstrates in simulations and on a sepsis dataset that its confidence intervals attain nominal coverage.

What carries the argument

The engine is the inverse-probability-weighted Bellman estimating equation $\mathbb{E}[\eta_{t+1}\{1-\lambda_{t+1}(\psi)\}^{-1} M_t(\beta^\pi)]=0$, where $M_t(\beta^\pi)=\xi_t(R_{t+1}+\gamma V^\pi(S_{t+1})-Q^\pi(S_t,A_t))$ is the Bellman residual and $\xi_t$ is a linear sieve basis for the state-action features. The dropout propensity $\lambda_{t+1}(\psi)=\lambda(S_t,A_t,R_{t+1},S_{t+1};\psi)$ is identified through a shadow variable $Z_t$, a covariate that is conditionally independent of dropout given $(S_t,A_t,R_{t+1},S_{t+1})$ but associated with the unobserved outcome, and estimated from $\mathbb{E}[(\eta_{t+1}/\{1-\lambda_{t+1}(\psi)\}-1)h(S_t,A_t,Z_t)]=0$, in parametric or semiparametric exponential-tilting form. The load-bearing identity is $\mathbb{E}[\eta_{t+1}/\{1-\lambda_{t+1}(\psi^\ast)\} \mid \mathcal{F}_t,R_{t+1},S_{t+1},\eta_t=1]=1$, which restores the zero expectation of the Bellman residual at the true parameter and gives the estimator its consistency.

What would settle it

Simulate MNAR data with a true dropout probability outside the fitted model class, for example a non-logistic function of the unobserved reward while fitting a logistic propensity, or generate the shadow variable so it also depends on dropout; under either misspecification, the IPW value estimate's bias should fail to vanish as $nT\to\infty$, contradicting Theorem 4.6.

Watch

Extended reading notes

Core claim

The paper's central claim is that monotone nonignorable missingness breaks standard off-policy evaluation while ignorable missingness does not. Under Assumption A.1, the complete-case estimator $\widehat V^\pi_{\mathrm{CC}}(G)$ is consistent under MAR (Theorem 4.5), but under MNAR it is biased because $\mathbb{E}[\eta_{t+1} M_t(\beta^\ast)] \neq 0$. The proposed estimator weights each transition by $\eta_{t+1}/\{1-\lambda(S_t,A_t,R_{t+1},S_{t+1};\widehat\psi)\}$, with $\widehat\psi$ obtained from a shadow-variable moment equation; Theorem 4.6 gives consistency, and Theorem 4.7 gives $\sqrt{nT}\,\widehat\sigma^{-1}\{\widehat V^\pi_{\mathrm{IPW}}(G)-V^\pi(G)\} \xrightarrow{d} N(0,1)$, where the limit holds as either $n\to\infty$ or $T\to\infty$. The paper also shows that the same inverse-weighting idea can be inserted into other OPE losses, such as fitted Q-evaluation.

Load-bearing premise

The consistency proof collapses if the dropout model is not the true mechanism or if the chosen shadow variable is not conditionally independent of dropout given the unobserved outcome, because then equation (4) does not recover the dropout probabilities.

Editorial extensions

If this is right

  • Under missing-at-random dropout, complete-case value estimation needs no missingness correction: it remains consistent.
  • Under nonignorable dropout, complete-case estimates are biased and confidence intervals under-cover, with the bias persisting as $nT$ grows.
  • A correctly specified dropout propensity plus a valid shadow variable makes the IPW value estimator consistent and asymptotically normal, so uncertainty quantification is available.
  • The IPW correction is modular and can be combined with other value-estimation losses, including fitted Q-evaluation, rather than only the linear-sieve estimator in the main theorems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the weighting identity only needs the conditional expectation of the response indicator, the same correction should carry over to fitted Q-evaluation and marginalized-importance-sampling estimators, but the paper only sketches those extensions and does not prove their asymptotics.
  • Editorial inference: the variance estimator used with a semi-parametric dropout model drops the uncertainty from estimating $\psi$; users should expect mild under-coverage in small samples and could add a bootstrap or influence-function correction.
  • Editorial inference: when no credible shadow variable exists, the identification failure described in the paper implies that no MNAR-robust value estimate is possible without extra assumptions; a sensitivity analysis over candidate shadow variables is the practical counterpart.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies off-policy evaluation (OPE) in infinite-horizon MDPs when logged trajectories are subject to monotone missingness. It argues that complete-case (CC) value estimation remains consistent under ignorable missingness (MAR) but can be biased under nonignorable missingness (MNAR). To restore consistency, the paper proposes an inverse probability weighted (IPW) value estimator, with the dropout propensity estimated through a shadow-variable estimating equation, and claims bidirectional consistency and asymptotic normality as either the number of trajectories n or the horizon T tends to infinity. Supporting results include simulations, a synthetic sepsis experiment, and an application to MIMIC-III data.

Significance. The paper addresses a genuine gap: OPE under MNAR is rarely studied, and the proposed IPW framework with shadow variables is a natural and potentially useful extension of the missing-data literature to sequential decision problems. The negative result that CC estimators can be biased under MNAR is valuable, and the MAR consistency argument is plausible under the stated conditional independence. The authors provide extensive simulations, a real-data application, and released code. However, the headline bidirectional theorems are not supported by the current proofs in the T→∞ direction under the formulated dropout model, and the dropout propensity estimating equation is not well defined for time points after dropout. These issues are load-bearing for the paper's central claims and require substantive correction.

major comments (3)
  1. [Section 4.2, Theorems 4.5-4.7; Appendix G] The claim that consistency and √(nT)-asymptotic normality hold 'as either n→∞ or T→∞' is not supported by the proof under the paper's monotone dropout setup. The proofs in Appendix G treat nT as the number of terms in the moment sums (e.g., Eqs. (29)-(31) and the martingale construction in Step 3 of the proof of Theorem 4.7), but under monotone missingness the number of observed transitions is N_obs = Σ_i (C_i ∧ T), not nT. If dropout is eventually certain, as in the paper's simulation models where the dropout probability is bounded away from zero, N_obs = O_p(n) for fixed n and the estimator cannot converge as T→∞ at the claimed √(nT) rate; if dropout is not eventually certain, an additional condition and a different variance normalization are needed. The theorems should be restricted to n→∞ with T fixed, or reformulated with N_obs and an explicit assumption that the effective sample size diverges in the stated asymptotic regime.
  2. [Section 4.3, Eq. (4); Appendix B] The estimating equation for ψ is not well defined as written for subjects who have dropped out. For a subject with dropout time C_i and for all t > C_i, we have η_{i,t+1}=0 and hence m_i,t = -h(S_i,t, A_i,t, Z_i,t), which requires the state, action, and shadow variable at time t even though these are not observed (and are not generated if the trajectory terminates at dropout). The paper never restricts the sum in Eq. (4) to person-time with η_{i,t}=1, nor does it specify that h is set to zero after dropout. Consequently, the left-hand side of Eq. (4) cannot be computed from the observed data, and the identifiability of ψ through the shadow variable is not established for the actual observed data. This also affects the expansion of √(nT)(bψ - ψ*) in Appendix G.2. Please rewrite Eq. (4) over observed person-time and re-derive the subsequent theory under that definition.
  3. [Appendix G.3, Eqs. (33)-(35)] For the semi-parametric dropout model, the implemented variance estimator replaces Ω_IPW by eΩ_IPW, which drops the H2 term that accounts for estimation of ψ. Theorem 4.7, however, is stated for bσ^2 in Eq. (34), and no result in the paper shows that the omitted term is asymptotically negligible. Since Algorithm 1 and all real-data and semi-parametric simulation results use the approximation (35), the coverage guarantee for the IPW(SP) estimator is not proven. Either prove that the approximation error is o_p(1) under the semi-parametric model, or state Theorem 4.7 only for the parametric dropout model with the full variance estimator and present eΩ_IPW as a heuristic approximation.
minor comments (5)
  1. [Section 5, Table 1] The column heading says 'standard error in parenthesis', but the numbers appear to be Monte Carlo standard deviations of the bias rather than standard errors of the mean; please clarify the definition and report standard errors of the Monte Carlo averages if that is the intent.
  2. [Figure 2] The x-axis label 'Confidence level 1' should read 'Nominal confidence level 1−α'.
  3. [Equation (3)] The typeset equation contains the LaTeX underbrace annotation '| {z }' inside the matrix expression; this should be removed.
  4. [Section 6 and Appendix E.3] The shadow variable is described in Section 6 as the previous GCS score, S^GCS_{t−1}, but Appendix E.3 writes Z_{t+1} = 1(S^GCS_{t+1} ≥ 14); please clarify the timing and explain whether the shadow variable is part of the state vector used in the value estimation.
  5. [Appendix B.3] The paper correctly acknowledges that when conditions for a chosen shadow variable are only partially satisfied, the observed likelihood is non-identifiable and the estimating equation is likely to fail; this important caveat should be connected more explicitly to Assumption A.2(c) and to the MIMIC-III application, where the validity of the chosen shadow variable is not empirically verified.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the main derivation is an extension of published external foundations, with the dropout-propensity fit explicitly propagated into the inference.

full rationale

The paper's central claim is that under Assumptions A.1 and A.2 (correctly specified dropout propensity model, positivity, and a valid shadow variable), the IPW value estimator is consistent and asymptotically normal (Theorems 4.6-4.7). This is proved rather than assumed: the estimating equation (4) identifies psi from the shadow-variable moment condition, and equation (3) uses the resulting weights in a distinct value estimating equation. The proof of Theorem 4.6 in Appendix G.2 shows consistency from E{omega_{t+1}(psi*) | ...} = 1 and standard sieve-regression bounds, and the asymptotic variance in equation (31) explicitly carries the uncertainty of the psi fit through the H1 and H2 terms. The self-citations (Shi et al. 2021b, Shi et al. 2020, Xu et al. 2023) overlap with the present authors, and Assumption A.1 plus Lemmas G.1-G.6 are imported from Shi et al. (2021b); however, Shi et al. (2021b) is an externally published, peer-reviewed result whose assumptions do not include the target MNAR/IPW claim, and the present paper supplies its own proofs for the missing-data adaptation. The shadow-variable identification results (Wang et al. 2014; Shao & Wang 2016; Miao et al. 2024) are external, not self-citations. The bootstrap/Monte-Carlo validation and the synthetic sepsis experiment use a known dropout mechanism as ground truth, which is an independent check rather than a renamed prediction. A separate, non-circular correctness concern: Theorems 4.6-4.7 claim bidirectional asymptotics 'as either n -> infinity or T -> infinity', but under monotone dropout with positivity (A.2(b), 1 - lambda >= c_lambda), each trajectory drops out with geometric tail, so the observed transition count per trajectory is bounded in probability and the effective sample size is O_p(n) rather than nT; the T -> infinity leg of the theorem and the sqrt(nT) normalization are therefore questionable as stated. This is a correctness/scope issue, not a circularity: it does not involve the conclusion being assumed in the assumptions or the prediction being fitted from its own target. Overall, no load-bearing circular reduction was found, so the circularity score is low.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on a standard stack of MDP regularity conditions, on correct specification of the dropout model, and on identifiability of that model via a shadow variable. No new entities are introduced. The free parameters are the dropout propensity parameters, the sieve dimension, bandwidth choices, and numerical stabilization constants.

free parameters (5)
  • Dropout propensity model parameters psi = Estimated from data; true values in simulations are [2, 0.15, -0.3]
    The IPW weights 1/(1-lambda_{t+1}(psi)) depend on the estimated psi. In the simulation, the true psi is [2,0.15,-0.3] but in real data it is estimated from the logged data.
  • Sieve basis dimension L = 36 in simulation, 56 in real data
    Controls the approximation error of the Q-function; chosen by the user, fixed across sample sizes.
  • Semi-parametric bandwidth constant c / h_l = c=7.5 in simulation, h_l=10*sigma_l*n_l^(-1/3) in real data
    Tuned by inspection of the objective function curve (Appendix E.2.3); affects the kernel estimate of the nonparametric component g.
  • Missing propensity truncation bound = 0.01
    Weights are truncated to avoid extreme inverse probabilities; a practical stabilizer.
  • Ridge penalty for matrix inversion = 10^(-5)
    Improves numerical stability of the estimated moment matrix.
assumptions (6)
  • domain assumption The MDP is time-homogeneous and rewards are conditionally mean independent (Assumptions 3.1 and 3.2)
    Standard RL assumptions. Used throughout to ensure the Bellman equation holds, e.g., E{M_t(beta*)|S_t,A_t}=0.
  • standard math Standard sieve regularity conditions (Assumption A.1), including smoothness of reward and transition, geometric ergodicity, and growth conditions on the number of basis functions
    Borrowed from Shi et al. (2021b). Needed for consistency and asymptotic normality of the value estimator under complete data.
  • domain assumption The dropout propensity model is correctly specified (Assumption A.2(a))
    Required for consistent estimation of psi and thus for the IPW weights. The paper's own simulations assume this.
  • standard math Positivity: probability of observing a transition is bounded away from zero (Assumption A.2(b))
    Standard IPW assumption, ensures weights are bounded.
  • domain assumption There exists a shadow variable Z_t that is independent of missingness given (S_t,A_t,R_{t+1},S_{t+1}) and predictive of (R_{t+1},S_{t+1}) (Assumption A.2(c))
    Identifiability of the dropout propensity under MNAR. The paper relies on prior results (Wang et al. 2014, Miao et al. 2024) that this identifies the model.
  • standard math Prior identifiability results for nonignorable missing data with shadow variables (Wang et al. 2014, Shao and Wang 2016, Miao et al. 2024)
    The paper takes these as proved background. If these fail, the estimating equation for psi may not identify the truth.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Off-Policy Evaluation Under Nonignorable Missing Data." pith.science (2026). https://pith.science/paper/4AGNHOLO

@misc{pith2026250706961,
  author       = {Pith},
  title        = {Pith review of: Off-Policy Evaluation Under Nonignorable Missing Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4AGNHOLO}},
  note         = {Machine review of arXiv:2507.06961}
}
read the original abstract

Off-Policy Evaluation (OPE) aims to estimate the value of a target policy using offline data collected from potentially different policies. In real-world applications, however, logged data often suffers from missingness. While OPE has been extensively studied in the literature, a theoretical understanding of how missing data affects OPE results remains unclear. In this paper, we investigate OPE in the presence of monotone missingness and theoretically demonstrate that the value estimates remain unbiased under ignorable missingness but can be biased under nonignorable (informative) missingness. To retain the consistency of value estimation, we propose an inverse probability weighted value estimator and conduct statistical inference to quantify the uncertainty of the estimates. Through a series of numerical experiments, we empirically demonstrate that our proposed estimator yields a more reliable value inference under missing data.

Figures

Figures reproduced from arXiv: 2507.06961 by the authors.

Figure 1
Figure 1. The average SOFA scores for patients remaining in the dataset (blue) and patients who died during ICU stay (red). The shadow represents the 25% to 75% quantile. In the RL literature, missingness is sometimes addressed by manually defining it as a special event within the reward framework. For example, in the Gridworld environment, “missingness” might be represented by the agent hitting a wall. In such cases, RL algo… view at source ↗
Figure 2
Figure 2. Empirical coverage probability with respect to different values of α under (n, T) = (1000, 10) and target policy π. CC estimator exhibits high bias, resulting in poor coverage probability of the associated confidence intervals. This under-coverage issue gets worse as the sample size grows. In contrast, the proposed IPW estimator (the last two lines, with a ‘P’ or ‘SP’ in parentheses to distinguish between the parame… view at source ↗
Figure 3
Figure 3. [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

78 extracted references · 70 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Birmingham, J., Rotnitzky, A., and Fitzmaurice, G. M. Pattern--mixture and selection models for analysing longitudinal data with monotone missing patterns. Journal of the Royal Statistical Society Series B: Statistical Methodology, 65 0 (1): 0 275--297, 2003

    work page 2003

  3. [3]

    Bradley, R. C. Basic properties of strong mixing conditions. a survey and some open questions. Probability Surveys, 2: 0 107--144, 2005

    work page 2005

  4. [4]

    Bradtke, S. J. and Barto, A. G. Linear least-squares algorithms for temporal difference learning. Machine Learning, 22 0 (1): 0 33--57, 1996

    work page 1996

  5. [5]

    Identifiability and estimation of causal effects in randomized trials with noncompliance and completely nonignorable missing data

    Chen, H., Geng, Z., and Zhou, X.-H. Identifiability and estimation of causal effects in randomized trials with noncompliance and completely nonignorable missing data. Biometrics, 65 0 (3): 0 675--682, 2009

    work page 2009

  6. [6]

    Multiply robust off-policy evaluation and learning under truncation by death

    Chu, J., Yang, S., and Lu, W. Multiply robust off-policy evaluation and learning under truncation by death. In International Conference on Machine Learning, pp.\ 6195--6227. PMLR, 2023

    work page 2023

  7. [7]

    Coindice: Off-policy confidence interval estimation

    Dai, B., Nachum, O., Chow, Y., Li, L., Szepesv \'a ri, C., and Schuurmans, D. Coindice: Off-policy confidence interval estimation. Advances in Neural Information Processing Systems, 33: 0 9398--9411, 2020

    work page 2020

  8. [8]

    Splines as linear combinations of b-splines

    De Boor, C. Splines as linear combinations of b-splines. a survey. Technical report, Wisconsin Univ Madison Mathematics Research Center, 1976

    work page 1976

Show all 78 references
  1. [9]

    and Kenward, M

    Diggle, P. and Kenward, M. G. Informative drop-out in longitudinal data analysis. Journal of the Royal Statistical Society: Series C (Applied Statistics), 43 0 (1): 0 49--73, 1994

    1994

  2. [10]

    Ascertaining properties of weighting in the estimation of optimal treatment regimes under monotone missingness

    Dong, L., Laber, E., Goldberg, Y., Song, R., and Yang, S. Ascertaining properties of weighting in the estimation of optimal treatment regimes under monotone missingness. Statistics in Medicine, 39 0 (25): 0 3503--3520, 2020

    2020

  3. [11]

    Off-policy deep reinforcement learning without exploration

    Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp.\ 2052--2062. PMLR, 2019

    2019

  4. [12]

    and Kosorok, M

    Goldberg, Y. and Kosorok, M. R. Q-learning with censored data. Annals of Statistics, 40 0 (1): 0 529, 2012

    2012

  5. [13]

    Graham, J. W. et al. Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60 0 (1): 0 549--576, 2009

    2009

  6. [14]

    Hansen, L. P. Large sample properties of generalized method of moments estimators. Econometrica: Journal of the Econometric Society, pp.\ 1029--1054, 1982

    1982

  7. [15]

    Harper, F. M. and Konstan, J. A. The movielens datasets: History and context. ACM Transactions on Interactive Intelligent Systems (TiiS), 5 0 (4): 0 1--19, 2015

    2015

  8. [16]

    Huang, J. Z. et al. Projection estimation in multiple regression with application to functional anova models. Annals of Statistics, 26 0 (1): 0 242--272, 1998

    1998

  9. [17]

    and Li, L

    Jiang, N. and Li, L. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning, pp.\ 652--661. PMLR, 2016

    2016

  10. [18]

    Real-time bidding with multi-agent reinforcement learning in display advertising

    Jin, J., Song, C., Li, H., Gai, K., Wang, J., and Zhang, W. Real-time bidding with multi-agent reinforcement learning in display advertising. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pp.\ 2193--2201, 2018

    2018

  11. [19]

    E., Pollard, T

    Johnson, A. E., Pollard, T. J., Shen, L., Li-Wei, H. L., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Celi, L. A., and Mark, R. G. MIMIC-III , a freely accessible critical care database. Scientific Data, 3 0 (1): 0 1--9, 2016

    2016

  12. [20]

    and Uehara, M

    Kallus, N. and Uehara, M. Efficiently breaking the curse of horizon in off-policy evaluation with double reinforcement learning. Operations Research, 2022

    2022

  13. [21]

    Kim, J. K. and Yu, C. L. A semiparametric estimation of mean functionals with nonignorable missing data. Journal of the American Statistical Association, 106 0 (493): 0 157--165, 2011

    2011

  14. [22]

    Nurse-led discharge from high dependency unit

    Knight, G. Nurse-led discharge from high dependency unit. Nursing in Critical Care, 8 0 (2): 0 56--61, 2003

    2003

  15. [23]

    A., and Peters, J

    Kober, J., Bagnell, J. A., and Peters, J. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32 0 (11): 0 1238--1274, 2013

    2013

  16. [24]

    A., Badawi, O., Gordon, A

    Komorowski, M., Celi, L. A., Badawi, O., Gordon, A. C., and Faisal, A. A. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature Medicine, 24 0 (11): 0 1716--1720, 2018

    2018

  17. [25]

    Kramer, A. A. and Zimmerman, J. E. A predictive model for the early identification of patients at risk for a prolonged intensive care unit length of stay. BMC Medical Informatics and Decision Making, 10 0 (1): 0 1--16, 2010

    2010

  18. [26]

    Lagoudakis, M. G. and Parr, R. Least-squares policy iteration. The Journal of Machine Learning Research, 4: 0 1107--1149, 2003

    2003

  19. [27]

    Batch policy learning under constraints

    Le, H., Voloshin, C., and Yue, Y. Batch policy learning under constraints. In International Conference on Machine Learning, pp.\ 3703--3712. PMLR, 2019

    2019

  20. [28]

    Li, L., Albert-Smet, I., and Faisal, A. A. Optimizing medical treatment for sepsis in intensive care: from reinforcement learning to pre-trial evaluation. arXiv preprint arXiv:2003.06474, 2020

    2003 arXiv

  21. [29]

    Linero, A. R. and Daniels, M. J. A flexible bayesian approach to monotone missing data in longitudinal studies with nonignorable missingness with application to an acute schizophrenia clinical trial. Journal of the American Statistical Association, 110 0 (509): 0 45--55, 2015

    2015

  22. [30]

    Liu, D. C. and Nocedal, J. On the limited memory bfgs method for large scale optimization. Mathematical Programming, 45 0 (1): 0 503--528, 1989

    1989

  23. [31]

    Breaking the curse of horizon: Infinite-horizon off-policy estimation

    Liu, Q., Li, L., Tang, Z., and Zhou, D. Breaking the curse of horizon: Infinite-horizon off-policy estimation. Advances in Neural Information Processing Systems, 31, 2018

    2018

  24. [32]

    J., Laber, E

    Luckett, D. J., Laber, E. B., Kahkoska, A. R., Maahs, D. M., Mayer-Davis, E., and Kosorok, M. R. Estimating dynamic treatment regimes in mobile health using v-learning. Journal of the American Statistical Association, 2019

    2019

  25. [33]

    McLeish, D. L. Dependent central limit theorems and invariance principles. The Annals of Probability, 2 0 (4): 0 620--628, 1974

    1974

  26. [34]

    J., Lawson, D

    McWilliams, C. J., Lawson, D. J., Santos-Rodriguez, R., Gilchrist, I. D., Champneys, A., Gould, T. H., Thomas, M. J., and Bourdeaux, C. P. Towards a decision support tool for intensive care discharge: machine learning algorithm development using electronic healthcare data from...

    2019

  27. [35]

    Identifiability of normal and normal mixture models with nonignorable missing data

    Miao, W., Ding, P., and Geng, Z. Identifiability of normal and normal mixture models with nonignorable missing data. Journal of the American Statistical Association, 111 0 (516): 0 1673--1683, 2016

    2016

  28. [36]

    J., and Geng, Z

    Miao, W., Liu, L., Li, Y., Tchetgen Tchetgen, E. J., and Geng, Z. Identification and semiparametric efficiency theory of nonignorable missing data with a shadow variable. ACM/JMS Journal of Data Science, 1 0 (2): 0 1--23, 2024

    2024

  29. [37]

    Playing atari with deep reinforcement learning

    Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013

    2013 arXiv

  30. [38]

    A., Veness, J., Bellemare, M

    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature, 518 0 (7540): 0 529--533, 2015

    2015

  31. [39]

    Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections

    Nachum, O., Chow, Y., Dai, B., and Li, L. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems, 32, 2019

    2019

  32. [40]

    H., Ross, A., Faisal, A., and Doshi-Velez, F

    Peng, X., Ding, Y., Wihl, D., Gottesman, O., Komorowski, M., Lehman, L.-w. H., Ross, A., Faisal, A., and Doshi-Velez, F. Improving sepsis treatment strategies by combining deep and kernel-based reinforcement learning. In AMIA Annual Symposium Proceedings, volume 2018, pp.\ 887...

    2018

  33. [41]

    Prasad, N., Cheng, L.-F., Chivers, C., Draugelis, M., and Engelhardt, B. E. A reinforcement learning approach to weaning of mechanical ventilation in intensive care units. In 33rd Conference on Uncertainty in Artificial Intelligence, 2017

    2017

  34. [42]

    Eligibility traces for off-policy policy evaluation

    Precup, D. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, pp.\ 80, 2000

    2000

  35. [43]

    Puterman, M. L. Markov decision processes: Discrete stochastic dynamic programming, 1994

    1994

  36. [44]

    Deep reinforcement learning for sepsis treatment

    Raghu, A., Komorowski, M., Ahmed, I., Celi, L., Szolovits, P., and Ghassemi, M. Deep reinforcement learning for sepsis treatment. arXiv preprint arXiv:1711.09602, 2017 a

    2017 arXiv

  37. [45]

    A., Szolovits, P., and Ghassemi, M

    Raghu, A., Komorowski, M., Celi, L. A., Szolovits, P., and Ghassemi, M. Continuous state-space models for optimal sepsis treatment: a deep reinforcement learning approach. In Machine Learning for Healthcare Conference, pp.\ 147--163. PMLR, 2017 b

    2017

  38. [46]

    Model-based reinforcement learning for sepsis treatment

    Raghu, A., Komorowski, M., and Singh, S. Model-based reinforcement learning for sepsis treatment. arXiv preprint arXiv:1811.09602, 2018

    2018 arXiv

  39. [47]

    M., Rotnitzky, A., and Scharfstein, D

    Robins, J. M., Rotnitzky, A., and Scharfstein, D. O. Sensitivity analysis for selection bias and unmeasured confounding in missing data and causal inference models. In Statistical Models in Epidemiology, the Environment, and Clinical Trials, pp.\ 1--94. Springer, 2000

    2000

  40. [48]

    Rosenbaum, P. R. and Rubin, D. B. The central role of the propensity score in observational studies for causal effects. Biometrika, 70 0 (1): 0 41--55, 1983

    1983

  41. [49]

    and Robins, J

    Rotnitzky, A. and Robins, J. Analysis of semi-parametric regression models with non-ignorable non-response. Statistics in Medicine, 16 0 (1): 0 81--102, 1997

    1997

  42. [50]

    and Wang, L

    Shao, J. and Wang, L. Semiparametric inverse propensity weighting for nonignorable missing data. Biometrika, 103 0 (1): 0 175--187, 2016

    2016

  43. [51]

    Does the markov decision process fit the data: Testing for the markov property in sequential decision making

    Shi, C., Wan, R., Song, R., Lu, W., and Leng, L. Does the markov decision process fit the data: Testing for the markov property in sequential decision making. In International Conference on Machine Learning, pp.\ 8807--8817. PMLR, 2020

    2020

  44. [52]

    Deeply-debiased off-policy interval estimation

    Shi, C., Wan, R., Chernozhukov, V., and Song, R. Deeply-debiased off-policy interval estimation. In International Conference on Machine Learning, pp.\ 9580--9591. PMLR, 2021 a

    2021

  45. [53]

    Statistical inference of the value function for reinforcement learning in infinite-horizon settings

    Shi, C., Zhang, S., Lu, W., and Song, R. Statistical inference of the value function for reinforcement learning in infinite-horizon settings. Journal of the Royal Statistical Society. Series B: Statistical Methodology, 2021 b

    2021

  46. [54]

    J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al

    Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. Nature, 529 0 (7587): 0 484--489, 2016

    2016

  47. [55]

    S., Seymour, C

    Singer, M., Deutschman, C. S., Seymour, C. W., Shankar-Hari, M., Annane, D., Bauer, M., Bellomo, R., Bernard, G. R., Chiche, J.-D., Coopersmith, C. M., et al. The third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA, 315 0 (8): 0 801--810, 2016

    2016

  48. [56]

    Bias in reinforcement learning: A review in healthcare applications

    Smith, B., Khojandi, A., and Vasudevan, R. Bias in reinforcement learning: A review in healthcare applications. ACM Computing Surveys, 56 0 (2): 0 1--17, 2023

    2023

  49. [57]

    A., Cai, T., and Szolovits, P

    Sonabend, A., Lu, J., Celi, L. A., Cai, T., and Szolovits, P. Expert-supervised reinforcement learning for offline policy learning and evaluation. Advances in Neural Information Processing Systems, 33: 0 18967--18977, 2020

    2020

  50. [58]

    Sun, B., Liu, L., Miao, W., Wirth, K., Robins, J., and Tchetgen, E. J. T. Semiparametric estimation with data missing not at random using an instrumental variable. Statistica Sinica, 28 0 (4): 0 1965, 2018

    1965

  51. [59]

    Doubly robust bias reduction in infinite horizon off-policy estimation

    Tang, Z., Feng, Y., Li, L., Zhou, D., and Liu, Q. Doubly robust bias reduction in infinite horizon off-policy estimation. arXiv preprint arXiv:1910.07186, 2019

    1910 arXiv

  52. [60]

    Tchetgen Tchetgen, E. J. and Wirth, K. E. A general instrumental variable framework for regression analysis with outcome missing not at random. Biometrics, 73 0 (4): 0 1123--1131, 2017

    2017

  53. [61]

    and Murphy, S

    Tewari, A. and Murphy, S. A. From ads to interventions: Contextual bandits in mobile health. Mobile Health: Sensors, Analytic Methods, and Applications, pp.\ 495--517, 2017

    2017

  54. [62]

    High-confidence off-policy evaluation

    Thomas, P., Theocharous, G., and Ghavamzadeh, M. High-confidence off-policy evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2015

    2015

  55. [63]

    Minimax weight and q-function learning for off-policy evaluation

    Uehara, M., Huang, J., and Jiang, N. Minimax weight and q-function learning for off-policy evaluation. In International Conference on Machine Learning, pp.\ 9659--9668. PMLR, 2020

    2020

  56. [64]

    L., Moreno, R., Takala, J., Willatts, S., De Mendonça, A., Bruining, H., Reinhart, C

    Vincent, J. L., Moreno, R., Takala, J., Willatts, S., De Mendonça, A., Bruining, H., Reinhart, C. K., Suter, P. M., and Thijs, L. G. The sofa (sepsis-related organ failure assessment) score to describe organ dysfunction/failure. Intensive Care Medicine, 22 0 (7): 0 707--710, 1996

    1996

  57. [65]

    M., Jiang, N., and Yue, Y

    Voloshin, C., Le, H. M., Jiang, N., and Yue, Y. Empirical study of off-policy policy evaluation for reinforcement learning. arXiv preprint arXiv:1911.06854, 2019

    1911 arXiv

  58. [66]

    Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation

    Wang, L., Zhang, W., He, X., and Zha, H. Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp.\ 2447--2456, 2018

    2018

  59. [67]

    Wang, S., Shao, J., and Kim, J. K. An instrumental variable approach for identification and estimation with nonignorable nonresponse. Statistica Sinica, pp.\ 1097--1116, 2014

    2014

  60. [68]

    Robust reinforcement learning in pomdps with incomplete and noisy observations

    Wang, Y., He, H., and Tan, X. Robust reinforcement learning in pomdps with incomplete and noisy observations. arXiv preprint arXiv:1902.05795, 2019

    1902 arXiv

  61. [69]

    Dueling network architectures for deep reinforcement learning

    Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., and Freitas, N. Dueling network architectures for deep reinforcement learning. In International Conference on Machine Learning, pp.\ 1995--2003. PMLR, 2016

    1995

  62. [70]

    Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling

    Xie, T., Ma, Y., and Wang, Y.-X. Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling. Advances in Neural Information Processing Systems, 32, 2019

    2019

  63. [71]

    An instrumental variable approach to confounded off-policy evaluation

    Xu, Y., Zhu, J., Shi, C., Luo, S., and Song, R. An instrumental variable approach to confounded off-policy evaluation. In International Conference on Machine Learning, pp.\ 38848--38880. PMLR, 2023

    2023

  64. [72]

    Large-scale order dispatch in on-demand ride-hailing platforms: A learning and planning approach

    Xu, Z., Li, Z., Guan, Q., Zhang, D., Li, Q., Nan, J., Liu, C., Bian, W., and Ye, J. Large-scale order dispatch in on-demand ride-hailing platforms: A learning and planning approach. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mi...

    2018

  65. [73]

    Partially observable reinforcement learning for blood glucose control under missing data

    Yu, H., Xiang, J., Kong, N., Luo, L., and Yang, C.-C. Partially observable reinforcement learning for blood glucose control under missing data. Available at SSRN 4647947

  66. [74]

    E., Pawelkiewicz, W., DeFrancesco, J

    Zahner, G. E., Pawelkiewicz, W., DeFrancesco, J. J., and Adnopoz, J. Children's mental health service needs and utilization patterns in an urban community: An epidemiological assessment. Journal of the American Academy of Child & Adolescent Psychiatry, 31 0 (5): 0 951--960, 1992

    1992

  67. [75]

    and Ma, Y

    Zhao, J. and Ma, Y. Optimal pseudolikelihood estimation in the analysis of multivariate missing data with nonignorable nonresponse. Biometrika, 105 0 (2): 0 479--486, 2018

    2018

  68. [76]

    and Ma, Y

    Zhao, J. and Ma, Y. A versatile estimation procedure without estimating the nonignorable missingness mechanism. Journal of the American Statistical Association, 117 0 (540): 0 1916--1930, 2022

    1916

  69. [77]

    and Shao, J

    Zhao, J. and Shao, J. Semiparametric pseudo-likelihoods in generalized linear models with nonignorable missing data. Journal of the American Statistical Association, 110 0 (512): 0 1577--1590, 2015

    2015

  70. [78]

    and Kim, J

    Zhou, M. and Kim, J. K. An efficient method of estimation for longitudinal surveys with monotone missing data. Biometrika, 99 0 (3): 0 631--648, 2012

    2012

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.