Pith. sign in

REVIEW 4 major objections 5 minor 80 references

Under a conditional-independence condition, the value of a target policy in offline reinforcement learning is identifiable and estimable even when the recorded actions are only noisy proxies for hidden true actions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 03:00 UTC pith:BPDYIAW3

load-bearing objection Worth engaging with, but the core identification is outsourced to a self-cited preprint and needs to be brought into the paper before I trust the theorems. the 4 major comments →

arxiv 2607.25241 v1 pith:BPDYIAW3 submitted 2026-07-28 stat.ML cs.LG

Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions

Zeyu Bian , Ying Zhou , Yifan Cui This is my paper
classification stat.ML cs.LG
keywords off-policy evaluationhidden actionslatent actionmultiply robust estimationinfluence functionoffline reinforcement learningaction misclassificationsequential decision making
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper takes on offline reinforcement learning in the setting where the actions that actually drove the system were never recorded; only noisy surrogates are available. It claims that, under a conditional-independence condition, the value of a target policy is uniquely identified from the observed states, surrogate actions, rewards, and next states. The proposed estimator, LURE, replaces the missing action indicators with contrast weights built from the surrogate action, reward, and next state, and is multiply robust and asymptotically normal. If correct, this gives practitioners a way to evaluate policies from error-prone logs such as electronic health records without assuming the recorded action is the delivered action.

Core claim

The paper establishes that in an infinite-horizon discounted Markov decision process with binary hidden actions, the policy value V(π) is identified from the observed distribution of (S, \tilde A, R, S') under coverage, relevance, conditional independence, and a surrogate-informativeness assumption. The next state S' serves as the crucial second proxy: together with the reward R and the surrogate action \tilde A, it permits construction of functions g_a and g'_a whose conditional expectations equal the unobserved indicators 1(A=a). The estimator LURE is the sample mean of the corresponding observed-data influence function, with nuisance functions estimated by a weighted EM-style procedure an

What carries the argument

The central object is the observed-data influence function φπ(O) for the policy value. It replaces the missing action indicator 1(A=a) with product contrast weights: g_a uses the surrogate action and reward, and g'_a uses the surrogate action and a feature of the next state, each normalized by the difference between action-specific conditional means. These weights are constructed so that, conditional on the true state and action, their expectation is exactly 1(A=a); the next state plays the role of the extra proxy variable that makes this replacement possible. A weighted EM-style algorithm iterates between estimating latent-action posterior probabilities and re-estimating the nuisance functi

Load-bearing premise

The load-bearing premise is Assumption 3.1(iii): conditional on the true state and action, the recorded surrogate action, the reward, and the next state are mutually independent.

What would settle it

Simulate an MDP in which the misclassification probability Pr(\tilde A ≠ A | S, A) depends on an unobserved acuity variable that also shifts the reward, while keeping the other assumptions intact; if LURE's estimate is biased or its confidence intervals undercover, the central claim fails. Concretely, set R = θ_R(S,A) + U and logit Pr(\tilde A ≠ A) = ν U with U a latent state correlate, then check whether the estimator still centers on the true value as ν grows.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • LURE gives asymptotically valid confidence intervals for the value of a target policy when only surrogate actions are recorded, enabling hypothesis tests and policy comparisons with quantified uncertainty.
  • The multiple-consistency structure means a practitioner can obtain consistent estimates from several different combinations of correctly specified nuisance components, such as the density ratio and surrogate-action model without a correct reward model.
  • The framework extends standard doubly robust off-policy evaluation to a measurement-error setting, replacing the assumption of perfectly recorded actions with a conditional-independence assumption about the proxies.
  • In clinical datasets where documented treatments may differ from delivered ones, using LURE can change which treatment policy is preferred relative to methods that treat the documented action as the delivered action, as the sepsis application illustrates.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • My inference: the conditional-independence assumption — that reward, surrogate action, and next state are independent given the true state and action — is the spot most likely to fail in practice, for instance when documentation delay is correlated with patient severity. A sensitivity analysis that perturbs this independence and reports how LURE's estimate drifts would be a natural companion to th
  • My inference: the same contrast-weight construction could be adapted to other error structures, such as a coarsened or aggregated version of a continuous action, or misclassification probabilities that depend on an auxiliary recorded covariate.
  • My inference: the label-alignment step relies on a separation condition; with limited data or weak separation, labels could switch across cross-fitting folds and inflate variance. Diagnostics for label-switching would be a practical safeguard for real deployments.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies off-policy evaluation (OPE) in infinite-horizon discounted Markov decision processes when the true action A is unobserved and only a noisy surrogate \tilde A is recorded. Under Assumption 3.1 — in particular the conditional independence R ⊥⊥ \tilde A ⊥⊥ S' | (S,A) — the paper claims identification of the policy value V(π) using the next state as a proxy. It then derives an observed-data influence function (Theorem 2), constructs the cross-fitted LURE estimator, proves a second-order error bound and multiple robustness (Theorem 3, Corollary 1), and establishes asymptotic normality (Theorem 4). A novel EM-style algorithm is proposed to estimate the latent-action nuisance components. The method is evaluated in simulations (tabular, continuous-state, CartPole) and on a MIMIC-III sepsis analysis. The central claim is that LURE is the first OPE method that is valid when recorded actions are only noisy proxies for the true actions.

Significance. If the identification and asymptotic results are correct, the paper addresses a genuinely important gap: standard OPE methods can be severely biased when actions are misclassified, and the paper provides a principled alternative. The influence-function construction is nontrivial and the multiple-robustness decomposition in Theorem 3 is a strength: the remainder is explicitly bounded by products of nuisance-error terms, and the four consistency cases in Corollary 1 are clearly delineated. The paper also provides extensive simulations and a real-data application. However, the central identification theorem is not proved in the manuscript: it delegates the key step to an external self-cited preprint, and the practical nuisance estimator (Algorithm 2) is not given any convergence analysis. These are load-bearing gaps that prevent the results from being fully verified as stated.

major comments (4)
  1. [Appendix A.2 (Proof of Theorem 1)] The proof of Theorem 1 does not contain an identification argument. It states that 'Zhou and Tchetgen Tchetgen (2024) proved that Pr(˜A=a'|A=a,S=s) is identifiable' and then inverts the 2×2 misclassification matrix. No conditions from that external preprint are stated, no mapping to Assumption 3.1 is provided, and no argument is reproduced. If the cited result requires an additional proxy or a completeness condition not implied by Assumption 3.1, then V(π) may not be identifiable from O=(S,˜A,R,S'). Since Theorem 1 underpins every subsequent theorem, this is a load-bearing omission. The authors should either give a self-contained proof of identifiability of the misclassification matrix under Assumption 3.1, or state and prove a precise theorem from Zhou and Tchetgen Tchetgen (2024) and verify all its conditions.
  2. [Lemma 2 (Appendix A.1)] Lemma 2 is the core identity behind the influence function: E[g_a(R,˜A,S)|S,A] = E[g'_a(S',˜A,S)|S,A] = 1(A=a). The proof is explicitly 'based on the proof of Theorem 4' of the same self-cited preprint and does not show the calculation for the product contrast. While the identity is plausible and can be verified by direct algebra under Assumption 3.1, the manuscript does not actually prove it. Given that Lemma 2 is essential for Theorem 2 and for the cancellation of the linear drift in Theorem 3, the proof should be self-contained.
  3. [Section 5 / Algorithm 2] Theorems 3 and 4 assume rate conditions on the nuisance estimators (α_min > 1/4, label-selection condition, etc.), but no theory is supplied for the proposed EM-style algorithm (Algorithm 2). It is not shown that the EM iterates converge to the true posterior η(O;a), nor that the resulting weighted regressions achieve the assumed rates. Since LURE is implemented with this algorithm, the asymptotic theory is conditional on an unverified premise about a central algorithmic component. The authors should either prove convergence/rates under explicit conditions, or clearly state the rate conditions as assumptions on the generic nuisance estimators and provide guidance on when Algorithm 2 satisfies them. At minimum, a consistency analysis of the EM procedure is needed.
  4. [Assumption 3.1(iii)] The conditional independence R ⊥⊥ ˜A ⊥⊥ S' | (S,A) is strong and is a key premise for Lemma 2 and Theorem 1. The paper's motivating EHR example (delayed charting, documentation errors) naturally raises the possibility that recording error is correlated with unmeasured patient acuity or with the reward/transition beyond the true action. The paper offers no sensitivity analysis or discussion of how violations of this assumption affect LURE. Given that this assumption is load-bearing for identification, the authors should at least provide a sensitivity model (e.g., allowing bounded residual dependence) or a clear statement of the limits of the method. Without this, the practical scope of the central claim is not quantified.
minor comments (5)
  1. [Section 6.1, Table 1 and Figures 3–5] Coverage is reported only for LURE because the baselines are biased. It would be helpful to also report bias and RMSE for all methods in a table, so readers can assess the magnitude of the bias reduction and the relative efficiency of LURE across misclassification rates.
  2. [Section 4, proxy selection] The choice of l(S') is not formalized in the theory. The simulations select the coordinate of S' with largest partial correlation with ˜A given S. This data-dependent selection is not accounted for in the influence-function derivation. If l is treated as fixed in the theory, state this explicitly; if it is selected from data, discuss the impact on inference.
  3. [Appendix A.2, Equation (10)] The matrix P_{˜A,A}(s) is defined with columns corresponding to A=0 and A=1. The inversion step is correct under Assumption 3.1(iv), but the notation could be clearer about the column ordering to avoid confusion.
  4. [Theorem 3 and Appendix A.4] Theorem 3 defines L_n = max_a ||δθ~A(·,a)||^κ, but the proof of label-selection error uses the L2 norm and derives O_p(||δθ~A||^κ). The role of κ is not pinned down; in Theorem 4 the condition is stated as max_a ||δθ~A||^κ = o_p(n^{-1/2}) for some κ>0. Since this condition is effectively a rate condition on ||δθ~A||, it would be clearer to state it directly in terms of the L2 error rate and avoid the unspecified κ.
  5. [Section 8 / Discussion] The discussion does not mention the conditional-independence assumption's limitations or the EM convergence issue. A short paragraph on these two points would help readers understand the scope of the claims.

Circularity Check

1 steps flagged

Theorem 1's identification is outsourced to a self-cited preprint; the rest of the LURE derivation is otherwise self-contained.

specific steps
  1. uniqueness imported from authors [Appendix A.2, Proof of Theorem 1; Section 3 Theorem 1]
    "Under Assumption 3.1, Zhou and Tchetgen Tchetgen (2024) proved that Pr(˜A=a'|A=a,S=s) is identifiable for all s,a' and a. ... Since P^{-1}_{˜A,A}(s) is identifiable, and P_˜A(s) is a function of the observed data, P_A(s) is identifiable for all s."

    The paper's foundational identification claim—that V(π) is identifiable from O=(S,˜A,R,S')—is not derived from Assumption 3.1 in the paper. Instead, the proof imports the identifiability of the misclassification matrix Pr(˜A|A,S) from a preprint co-authored by one of the present authors. The remaining algebra only inverts that matrix. Every subsequent theorem (Lemma 2, Theorem 2, Theorem 3, Corollary 1, Theorem 4) presupposes Theorem 1, so the central result is conditional on an unverified self-citation rather than on a self-contained proof. The paper does not state which conditions of the cited preprint correspond to Assumption 3.1, so an omitted completeness or extra-proxy condition there would invalidate the whole chain.

full rationale

The derivation chain is largely self-contained after Theorem 1: Lemma 1, Lemma 2's algebra, Lemma 3, Theorem 2's influence-function projection, Theorem 3's error decomposition, Corollary 1's multiple robustness, Theorem 4's asymptotic normality, and Proposition 1's variance consistency are all proved from the stated assumptions and the standard Kallus–Uehara full-data representation. The EM-based nuisance estimation and label-alignment step are not fitted parameters renamed as predictions. The only load-bearing circularity is the identification theorem: its proof reduces Pr(Ã|A,S) identifiability to a self-cited preprint by co-author Zhou, without reproducing or verifying the argument. Lemma 2 also cites the same preprint but its proof is actually shown in the paper, so it is a minor self-citation rather than a separate circular step. Because the estimator and its asymptotic theory do not reduce to the fitted nuisance constants, the circularity is partial and localized to the identification premise. Score 4 reflects some self-citation with the central claim still having substantial independent content.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claim rests on structural conditional-independence assumptions, a borrowed identification theorem, and an assumed convergence of an EM-based nuisance pipeline. The paper contributes the influence-function combination and asymptotic expansion, but the identification and algorithm-convergence parts are imported or assumed.

free parameters (3)
  • Proxy coordinate selection j* for l(S') = data-dependent (max |partial corr(S'^(j), \tilde A | S)|)
    In Sections 6-7, the feature l(S') is chosen as the state component with the largest absolute partial correlation with the observed surrogate conditional on S. This data-driven choice affects θ_{S'} and hence LURE, and the theoretical results do not model this selection step.
  • EM initialization η(0) and convergence tolerances
    Algorithm 2 requires an initial posterior η(0) with probabilities summing to 1 and a convergence rule, but the paper specifies neither. EM can converge to different local solutions depending on initialization, and label alignment only fixes labels, not the quality of the fitted mixture.
  • Nuisance model specifications = linear (MIMIC-III, General MDP), additive cubic B-spline (CartPole), nonparametric cell means (tabular)
    The asymptotic theory is silent on approximation error; consistency of the EM-weighted regressions is only plausible if these function classes are correctly specified, and no guarantees are derived for them.
axioms (5)
  • domain assumption Stationary infinite-horizon discounted MDP with known initial distribution p_e and stationary behavior policy b.
    Section 2; V(π) is defined with respect to this stationarity; if b or the environment changes over time, the density-ratio identification fails.
  • domain assumption Assumption 3.1(i)-(iv): coverage, relevance of R, \tilde A, S' for A given S, conditional independence R⊥\tilde A⊥S' | (S,A), and surrogate ordering Pr(\tilde A=0|A=1,S) < Pr(\tilde A=0|A=0,S).
    This is the structural basis for identification and for the E[g_a|S,A]=1(A=a) property in Lemma 2.
  • domain assumption Assumption 5.1: bounded nuisances and uniform separation of contrasts θ_\tilde A(s,0)-θ_\tilde A(s,1), θ_R(s,0)-θ_R(s,1), θ_{S'}(s,0)-θ_{S'}(s,1) (true and estimated) away from 0.
    Needed for denominators in g_a/g'_a, label alignment, and the CLT; can be violated by weak proxies.
  • ad hoc to paper The Zhou and Tchetgen Tchetgen (2024) identification theorem applies to the RL observed data, making Pr(\tilde A|A,S) identifiable.
    Theorem 1 proof invokes this external self-cited result without stating or verifying its conditions on the MDP data.
  • ad hoc to paper EM and weighted-regression nuisance estimators achieve the rate conditions assumed in Theorems 3-4 (α_min > 1/4, label-selection error o_p(n^{-1/2})).
    Section 5 assumes 'suitable convergence rate conditions'; no proof is given for the EM procedure of Algorithm 2, especially under misspecified densities q and h.

pith-pipeline@v1.3.0-alltime-deepseek · 39220 in / 21094 out tokens · 231631 ms · 2026-08-01T03:00:55.894417+00:00 · methodology

0 comments
read the original abstract

Standard offline reinforcement learning (RL) algorithms typically assume that the actions in the dataset are observed without error. However, in many real-world applications, the true actions are unobserved and only noisy proxies are available, causing existing RL methods to yield biased and potentially misleading conclusions. We study off-policy evaluation in infinite-horizon discounted Markov decision processes with hidden actions. By leveraging the next-state variable as a natural proxy for the unobserved action, we establish identification of the policy value and propose an influence-function-based estimator called LURE (Learning from the Unseen: Robust Estimator). LURE is multiply robust, remaining consistent under several combinations of correctly specified nuisance components, and is asymptotically normal, enabling valid statistical inference. To our knowledge, this is the first work to address offline RL with hidden actions. We demonstrate LURE's effectiveness through simulations and a sepsis management application using the MIMIC-III database.

Figures

Figures reproduced from arXiv: 2607.25241 by Yifan Cui, Ying Zhou, Zeyu Bian.

Figure 1
Figure 1. Figure 1: Data-generating process in standard MDP. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Data-generating process with latent actions. The underlying latent true action [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Empirical distributions of the estimated policy value under Scenarios 1 ( [PITH_FULL_IMAGE:figures/full_fig_p024_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Empirical distributions of the estimated policy value under Scenarios 1 ( [PITH_FULL_IMAGE:figures/full_fig_p025_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Empirical distributions of the estimated policy value under Scenarios 1 ( [PITH_FULL_IMAGE:figures/full_fig_p027_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

80 extracted references · 7 linked inside Pith

  1. [1]

    Operations Research , volume=

    Efficiently breaking the curse of horizon in off-policy evaluation with double reinforcement learning , author=. Operations Research , volume=. 2022 , publisher=

    2022

  2. [2]

    arXiv preprint arXiv:1606.01540 , year=

    Openai gym , author=. arXiv preprint arXiv:1606.01540 , year=

    Pith/arXiv arXiv

  3. [3]

    Journal of the American Statistical Association , pages=

    Data-driven knowledge transfer in batch q* learning , author=. Journal of the American Statistical Association , pages=. 2026 , publisher=

    2026

  4. [4]

    The Annals of Statistics , volume=

    Off-policy evaluation in partially observed Markov decision processes under sequential ignorability , author=. The Annals of Statistics , volume=. 2023 , publisher=

    2023

  5. [5]

    International conference on machine learning , pages=

    Distributional reinforcement learning for efficient exploration , author=. International conference on machine learning , pages=. 2019 , organization=

    2019

  6. [6]

    The Annals of Statistics , volume=

    Projected state-action balancing weights for offline reinforcement learning , author=. The Annals of Statistics , volume=. 2023 , publisher=

    2023

  7. [7]

    Machine learning for health , pages=

    An extensive data processing pipeline for mimic-iv , author=. Machine learning for health , pages=. 2022 , organization=

    2022

  8. [8]

    Scientific data , volume=

    MIMIC-III, a freely accessible critical care database , author=. Scientific data , volume=. 2016 , publisher=

    2016

  9. [9]

    Biostatistics , volume=

    Biases in electronic health record data due to sampling and limited follow-up , author=. Biostatistics , volume=. 2018 , publisher=

    2018

  10. [10]

    Journal of the American Medical Informatics Association , volume=

    Next-generation phenotyping of electronic health records: support for high-throughput clinical research , author=. Journal of the American Medical Informatics Association , volume=. 2013 , publisher=

    2013

  11. [11]

    Frontiers in Artificial Intelligence , volume=

    Medical Information Mart for Intensive Care: A foundation for the fusion of artificial intelligence and real-world data , author=. Frontiers in Artificial Intelligence , volume=. 2021 , publisher=

    2021

  12. [12]

    is” and “is documented

    The difference between “is” and “is documented”: EHR misclassification in the ICU , author=. Critical Care Medicine , volume=

  13. [13]

    2014 , publisher=

    Markov decision processes: discrete stochastic dynamic programming , author=. 2014 , publisher=

    2014

  14. [14]

    arXiv preprint arXiv:1707.06347 , year=

    Proximal policy optimization algorithms , author=. arXiv preprint arXiv:1707.06347 , year=

    Pith/arXiv arXiv

  15. [15]

    Econometrica , volume=

    Who should be treated? empirical welfare maximization methods for treatment choice , author=. Econometrica , volume=. 2018 , publisher=

    2018

  16. [16]

    International Conference on Machine Learning , pages=

    Minimax weight and q-function learning for off-policy evaluation , author=. International Conference on Machine Learning , pages=. 2020 , organization=

    2020

  17. [17]

    Journal of Machine Learning Research , volume=

    Tree-based batch mode reinforcement learning , author=. Journal of Machine Learning Research , volume=. 2005 , publisher=

    2005

  18. [18]

    Epidemiology , volume=

    Identifiability and exchangeability for direct and indirect effects , author=. Epidemiology , volume=. 1992 , publisher=

    1992

  19. [19]

    The Annals of statistics , pages=

    Bayesian inference for causal effects: The role of randomization , author=. The Annals of statistics , pages=. 1978 , publisher=

    1978

  20. [20]

    Journal of the American statistical Association , volume=

    Estimation of regression coefficients when some regressors are not always observed , author=. Journal of the American statistical Association , volume=. 1994 , publisher=

    1994

  21. [21]

    Journal of the American Statistical Association , volume=

    Estimating individualized treatment rules using outcome weighted learning , author=. Journal of the American Statistical Association , volume=. 2012 , publisher=

    2012

  22. [22]

    2018 , publisher=

    Reinforcement learning: An introduction , author=. 2018 , publisher=

    2018

  23. [23]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

  24. [24]

    International Conference on Machine Learning , pages=

    Combining parametric and nonparametric models for off-policy evaluation , author=. International Conference on Machine Learning , pages=. 2019 , organization=

    2019

  25. [25]

    Advances in Neural Information Processing Systems , volume=

    Breaking the curse of horizon: Infinite-horizon off-policy estimation , author=. Advances in Neural Information Processing Systems , volume=

  26. [26]

    Advances in neural information processing systems , volume=

    Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections , author=. Advances in neural information processing systems , volume=

  27. [27]

    International conference on machine learning , pages=

    Data-efficient off-policy policy evaluation for reinforcement learning , author=. International conference on machine learning , pages=. 2016 , organization=

    2016

  28. [28]

    Advances in Neural Information Processing Systems , volume=

    Minimax estimation of conditional moment models , author=. Advances in Neural Information Processing Systems , volume=

  29. [29]

    Biometrics , volume=

    A robust method for estimating optimal treatment regimes , author=. Biometrics , volume=. 2012 , publisher=

    2012

  30. [30]

    Statistics in Precision Health: Theory, Methods and Applications , pages=

    Statistical reinforcement learning and dynamic treatment regimes , author=. Statistics in Precision Health: Theory, Methods and Applications , pages=. 2024 , publisher=

    2024

  31. [31]

    arXiv preprint arXiv:2510.20404 , year =

    Shuyuan Chen and Peng Zhang and Yifan Cui , title =. arXiv preprint arXiv:2510.20404 , year =. 2510.20404 , archivePrefix=

    arXiv

  32. [32]

    2015 , publisher=

    Adaptive treatment strategies in practice: planning trials and analyzing data for personalized medicine , author=. 2015 , publisher=

    2015

  33. [33]

    Journal of the American Statistical Association , volume=

    Contextual dynamic pricing with strategic buyers , author=. Journal of the American Statistical Association , volume=. 2025 , publisher=

    2025

  34. [34]

    Journal of the American Statistical Association , volume=

    Estimating Dynamic Treatment Regimes in Mobile Health Using V-Learning , author=. Journal of the American Statistical Association , volume=. 2020 , publisher=

    2020

  35. [35]

    International Conference on Machine Learning , pages=

    Doubly robust off-policy value evaluation for reinforcement learning , author=. International Conference on Machine Learning , pages=. 2016 , organization=

    2016

  36. [36]

    Operations Research , year=

    Rate-Optimal Online Learning for Dynamic Assortment Selection with Positioning , author=. Operations Research , year=

  37. [37]

    Journal of the Royal Statistical Society Series B: Statistical Methodology , volume=

    Statistical inference of the value function for reinforcement learning in infinite-horizon settings , author=. Journal of the Royal Statistical Society Series B: Statistical Methodology , volume=. 2022 , publisher=

    2022

  38. [38]

    arXiv preprint arXiv:2212.06355 , year=

    A review of off-policy evaluation in reinforcement learning , author=. arXiv preprint arXiv:2212.06355 , year=

    Pith/arXiv arXiv

  39. [39]

    Computer Science Department Faculty Publication Series , pages=

    Eligibility traces for off-policy policy evaluation , author=. Computer Science Department Faculty Publication Series , pages=

  40. [40]

    International Conference on Learning Representations , year=

    Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization , author=. International Conference on Learning Representations , year=

  41. [41]

    Advances in neural information processing systems , volume=

    Direct preference optimization: Your language model is secretly a reward model , author=. Advances in neural information processing systems , volume=

  42. [42]

    Surveys in operations research and management science , volume=

    Dynamic pricing and learning: Historical origins, current research, and new directions , author=. Surveys in operations research and management science , volume=. 2015 , publisher=

    2015

  43. [43]

    Econometrica , volume=

    Policy learning with observational data , author=. Econometrica , volume=. 2021 , publisher=

    2021

  44. [44]

    2018 , publisher=

    Double/debiased machine learning for treatment and structural parameters , author=. 2018 , publisher=

    2018

  45. [45]

    Biometrika , volume=

    The central role of the propensity score in observational studies for causal effects , author=. Biometrika , volume=. 1983 , publisher=

    1983

  46. [46]

    arXiv preprint arXiv:2405.09080 , year=

    Causal Inference for a Hidden Treatment , author=. arXiv preprint arXiv:2405.09080 , year=

    Pith/arXiv arXiv

  47. [47]

    2006 , publisher=

    Measurement error in nonlinear models: a modern perspective , author=. 2006 , publisher=

    2006

  48. [48]

    2009 , publisher=

    Measurement error models , author=. 2009 , publisher=

    2009

  49. [49]

    Journal of Econometrics , volume=

    Identification and estimation of nonlinear models with misclassification error using instrumental variables: A general solution , author=. Journal of Econometrics , volume=. 2008 , publisher=

    2008

  50. [50]

    Biometrics , volume=

    Variable selection in regression-based estimation of dynamic treatment regimes , author=. Biometrics , volume=. 2023 , publisher=

    2023

  51. [51]

    International conference on machine learning , pages=

    Deeply-debiased off-policy interval estimation , author=. International conference on machine learning , pages=. 2021 , organization=

    2021

  52. [52]

    Journal of the American Statistical Association , volume=

    Multi-armed angle-based direct learning for estimating optimal individualized treatment rules with various outcomes , author=. Journal of the American Statistical Association , volume=. 2020 , publisher=

    2020

  53. [53]

    Biometrics , volume=

    Doubly-robust dynamic treatment regimen estimation via weighted least squares , author=. Biometrics , volume=. 2015 , publisher=

    2015

  54. [54]

    Proceedings of the Second Seattle Symposium in Biostatistics , address =

    Optimal structural nested models for optimal sequential decisions , author=. Proceedings of the Second Seattle Symposium in Biostatistics , address =. 2004 , organization=

    2004

  55. [55]

    Annals of statistics , volume=

    Performance guarantees for individualized treatment rules , author=. Annals of statistics , volume=

  56. [56]

    Journal of the American Statistical Association , volume=

    A semiparametric instrumental variable approach to optimal treatment regimes under endogeneity , author=. Journal of the American Statistical Association , volume=. 2021 , publisher=

    2021

  57. [57]

    Journal of the American Statistical Association , volume=

    Quantile-optimal treatment regimes , author=. Journal of the American Statistical Association , volume=. 2018 , publisher=

    2018

  58. [58]

    Annals of statistics , volume=

    High-dimensional A-learning for optimal dynamic treatment regimes , author=. Annals of statistics , volume=. 2018 , publisher=

    2018

  59. [59]

    Nature , volume=

    Mastering the game of Go with deep neural networks and tree search , author=. Nature , volume=. 2016 , publisher=

    2016

  60. [60]

    arXiv preprint arXiv:2005.01643 , year=

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems , author=. arXiv preprint arXiv:2005.01643 , year=

    Pith/arXiv arXiv 2005

  61. [61]

    The Annals of Statistics , volume=

    Batch policy learning in average reward markov decision processes , author=. The Annals of Statistics , volume=. 2022 , publisher=

    2022

  62. [62]

    Journal of the Royal Statistical Society: Series B , volume=

    Optimal dynamic treatment regimes , author=. Journal of the Royal Statistical Society: Series B , volume=. 2003 , publisher=

    2003

  63. [63]

    arXiv preprint arXiv:2411.08126 , year=

    A Tale of Two Cities: Pessimism and Opportunism in Offline Dynamic Pricing , author=. arXiv preprint arXiv:2411.08126 , year=

    Pith/arXiv arXiv

  64. [64]

    Journal of the American Statistical Association , volume=

    Off-policy evaluation in doubly inhomogeneous environments , author=. Journal of the American Statistical Association , volume=. 2025 , publisher=

    2025

  65. [65]

    International Conference on Machine Learning , pages=

    On well-posedness and minimax optimal rates of nonparametric q-function estimation in off-policy evaluation , author=. International Conference on Machine Learning , pages=. 2022 , organization=

    2022

  66. [66]

    arXiv preprint arXiv:2004.07219 , year=

    D4rl: Datasets for deep data-driven reinforcement learning , author=. arXiv preprint arXiv:2004.07219 , year=

    Pith/arXiv arXiv 2004

  67. [67]

    Journal of the American Statistical Association , volume=

    Reinforcement learning trees , author=. Journal of the American Statistical Association , volume=. 2015 , publisher=

    2015

  68. [68]

    Journal of the American Statistical Association , pages=

    Reinforcement learning with continuous actions under unmeasured confounding , author=. Journal of the American Statistical Association , pages=. 2026 , publisher=

    2026

  69. [69]

    Journal of the American Statistical Association , pages=

    Risk-sensitive deep rl: Variance-constrained actor-critic provably finds globally optimal policy , author=. Journal of the American Statistical Association , pages=. 2026 , publisher=

    2026

  70. [70]

    Journal of the American Statistical Association , volume=

    Federated offline reinforcement learning , author=. Journal of the American Statistical Association , volume=. 2024 , publisher=

    2024

  71. [71]

    Maximum likelihood from incomplete data via the

    Dempster, Arthur P and Laird, Nan M and Rubin, Donald B , journal=. Maximum likelihood from incomplete data via the. 1977 , publisher=

    1977

  72. [72]

    Statistical Science , volume=

    Offline reinforcement learning in large state spaces: Algorithms and guarantees , author=. Statistical Science , volume=. 2025 , publisher=

    2025

  73. [73]

    the Annals of Probability , volume=

    Dependent central limit theorems and invariance principles , author=. the Annals of Probability , volume=. 1974 , publisher=

    1974

  74. [74]

    International Conference on Machine Learning , pages=

    Batch policy learning under constraints , author=. International Conference on Machine Learning , pages=. 2019 , organization=

    2019

  75. [75]

    Jama , volume=

    The third international consensus definitions for sepsis and septic shock (Sepsis-3) , author=. Jama , volume=

  76. [76]

    Intensive care medicine , volume=

    Surviving sepsis campaign: international guidelines for management of sepsis and septic shock 2021 , author=. Intensive care medicine , volume=. 2021 , publisher=

    2021

  77. [77]

    Critical care medicine , volume=

    The Sequential Organ Failure Assessment score for predicting outcome in patients with severe sepsis and evidence of hypoperfusion at the time of emergency department presentation , author=. Critical care medicine , volume=. 2009 , publisher=

    2009

  78. [78]

    Current Epidemiology Reports , volume =

    Measurement error and misclassification in electronic medical records: methods to mitigate bias , author =. Current Epidemiology Reports , volume =. 2018 , doi =

    2018

  79. [79]

    Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence , series =

    Causal inference with treatment measurement error: a nonparametric instrumental variable approach , author =. Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence , series =. 2022 , editor =

    2022

  80. [80]

    The Annals of Statistics , pages=

    On asymptotically efficient estimation in semiparametric models , author=. The Annals of Statistics , pages=. 1986 , publisher=

    1986