Pith. sign in

REVIEW 5 major objections 5 minor 79 references

Reinforcement Learning with Continuous Actions Under Unmeasured Confounding

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Bridge functions recover policy values and near-optimal policies in continuous-action RL with hidden confounders.

desk verdict The identification and algorithm are a genuine extension to continuous-action infinite-horizon POMDPs, but the root-n rate claims are not supported by the displayed theorems. read the letter →

arxiv 2505.00304 v1 pith:LUHB2V4D submitted 2025-05-01 stat.ML cs.LGstat.ME

Yuhan Li , Eugene Han , Yifan Hu , Wenzhuo Zhou , Zhengling Qi , Yifan Cui , Ruoqing Zhu This is my paper · ORCID
classification stat.MLcs.LGstat.ME MSC 62D2062G0568T0590C40
keywords off-policyevaluationlearningconfoundedPOMDPunmeasuredconfoundingcontinuousactionspacebridgefunctionsminimaxestimationpolicygradient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tackles offline reinforcement learning when the behavior policy depends on an unobserved state variable, actions are continuous, and the horizon is infinite. It claims that even though the true state is hidden, proxy variables observed alongside the rewards allow the value of any target policy to be identified from batch data, without discretizing the action space. On top of that identification, it constructs a penalized minimax estimator for the value and a policy-gradient procedure that searches the best policy within a parametric class. The central promise is practical: consistent policy evaluation at $O(n^{-1/2})$ error and learned policies with $O(p^{1/2}n^{-1/2})$ regret, in a setting where existing methods either assume away confounding or restrict to discrete actions.

What carries the argument

The load-bearing object is the pair of bridge functions: $Q^\pi(O_t,W_t,A_t)$ and $V^\pi(O_t,W_t)$, defined so that their conditional expectations given the full state equal the state-action and state value of the target policy. The identification equation uses the previous observed state-action pair $(O_{t-1},A_{t-1})$ as an action-inducing proxy and the reward proxy $W_t$ to condition away the unobserved state, yielding a conditional moment restriction with the Bellman residual. The estimation machinery is a penalized minimax problem whose inner maximization, over critic functions in a bounded RKHS, collapses to a closed form; this reduces the procedure to a single-stage regularized minimization that supports stochastic-gradient optimization and a policy-gradient loop over the policy parameters.

What would settle it

Run the estimator on a synthetic confounded POMDP that satisfies Assumptions 1-3 but generates $S_t$ independently of $(O_{t-1},A_{t-1})$ given $(O_t,A_t)$, so completeness fails; if the estimated $J(\pi)$ does not converge to the true policy value as $n$ grows, identification is genuinely doing the work. A data-only probe is to fit the conditional moment equation from multiple random initializations and with different kernel bandwidths: if the resulting policy-value estimates differ by more than the claimed $n^{-1/2}$ scale, the equation is admitting multiple bridge functions and the central identification is not operational.

Watch

Extended reading notes

Core claim

The paper's central claim is Theorem 3.1: in an infinite-horizon confounded POMDP satisfying Assumptions 1-4, there exist Q-bridge and V-bridge functions, and a particular pair solves the conditional moment equation $E[Q^\pi(O_t,W_t,A_t)-R_t-\gamma V^\pi(O_{t+1},W_{t+1})\mid O_{t-1},A_{t-1},O_t,A_t]=0$. Consequently the policy value $J(\pi)=E[V^\pi(O_0,W_0)]$ is nonparametrically identified from observed trajectories even though the behavior policy depends on the unobserved state. The paper then shows that the associated minimax estimator is consistent, attains $O(n^{-1/2})$ finite-sample error for a fixed policy (Theorem 4.2), and that the policy-gradient learner has regret $O(p^{1/2}n^{-1/2})$ (Theorem 4.3). The same identification also gives a tractable algorithm because the inner maximization over critic functions decouples into a closed-form kernel expression when critics are modelled in a reproducing kernel Hilbert space.

Load-bearing premise

The argument rests on an unverifiable completeness condition: the observed proxies must be rich enough that any two different hidden-state configurations produce different predictions from the observed variables. If that fails, the bridge functions are not identifiable and the error and regret guarantees collapse.

Editorial extensions

If this is right

  • Policy evaluation in confounded POMDPs with continuous actions no longer requires discretizing the action space; the bridge-function estimator is consistent and achieves $O(n^{-1/2})$ error for fixed target policies.
  • The learned in-class policy attains regret $O(p^{1/2}n^{-1/2})$, so batch data with unmeasured confounders can support near-optimal treatment or intervention recommendations.
  • The RKHS decoupling of the minimax problem yields a closed-form inner maximization, making the method computationally feasible for continuous states and actions rather than a purely theoretical identification result.
  • Because the identification uses reward proxies $W_t$ that need not cause $R_t$, the method applies to settings such as survey panels where auxiliary variables are correlated with outcomes but not driven by the action.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One implication left implicit is that the completeness assumption is untestable from data, so a practical safeguard would be to estimate the bridge functions under several candidate proxy sets and report the spread of the resulting policy values; the paper does not develop such a sensitivity analysis.
  • The conditional-moment identification is not tied to the discounted infinite-horizon objective; the same Q/V-bridge argument should extend to average-reward and finite-horizon objectives with only the Bellman residual changed.
  • A direct testable extension would be to evaluate the learned Pairfam policy by comparing its estimated long-term satisfaction against couples whose actual intimacy frequency matches the policy's recommended distribution in a hold-out wave.
  • The policy-gradient formulation avoids the per-iteration argmax over a continuous action space; the same trick could be imported into proximal value-based methods for POMDPs that currently rely on discrete actions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper studies offline policy evaluation and learning in an infinite-horizon confounded POMDP with continuous actions. It proposes Q- and V-bridge functions using time-dependent proxy variables, derives the conditional moment equation (3.3), and constructs a minimax estimator with an RKHS-based critic together with a policy-gradient algorithm. The headline theoretical claims are an O(n^{-1/2}) policy-evaluation error (Theorem 4.2), a regret bound O(p^{1/2}n^{-1/2}) (Theorem 4.3), and a supporting Bellman-error bound (Theorem 4.1). The methodology is illustrated with simulations and an application to the Pairfam data.

Significance. If the stated rates were rigorously established, this would be a valuable extension of proximal causal inference to continuous-action infinite-horizon settings, with a computationally attractive kernel decoupling of the minimax estimator and a practically relevant policy-gradient scheme. The identification strategy is plausible and follows the proximal template, and the empirical section, including the Pairfam application, is a strength. However, the displayed theorems in Section 4 do not establish the advertised rates, so the theoretical contribution is currently incomplete.

major comments (5)
  1. [Section 4, Theorem 4.1] The Bellman-error bound in Theorem 4.1 has no sample-size term: the displayed right-hand side depends only on λ_n, κ, h_1(Qπ) and δ. Taken literally, sending λ_n→0 would drive the bound to zero at fixed n, which is impossible for an estimator constructed from n finite trajectories. The missing empirical-process/optimization error must appear in the bound; without it, the claimed rate O(n^{-1/(1+α)}) and all subsequent results that rely on Theorem 4.1 are not supported.
  2. [Section 4, Theorem 4.2] Theorem 4.2 is not a genuine finite-sample high-probability bound because it contains an o_P(n^{-1/2}) remainder inside a statement that holds with probability at least 1−δ. In addition, the condition λ_n = o(n^{-1/(1+α)}) does not imply a root-n error for the Q-bridge estimation, since Theorem 4.1 provides no n-dependent control of ∥Q̂π−Qπ∥. The O(n^{-1/2}) policy-evaluation claim is therefore not established.
  3. [Section 4, Theorem 4.3 and Proposition 4.1] Theorem 4.3's regret bound requires a uniform version of the root-n evaluation error over the policy class Π. Proposition 4.1 extends Theorem 4.1 uniformly over Π but inherits the same defect: its bound is λ_n times a policy-complexity factor with no n-dependent term. Consequently the advertised O(p^{1/2}n^{-1/2}) regret rate is unsupported.
  4. [Section 3.1, Theorem 3.1] The identification theorem is the foundation of the paper, but its proof is not given in the main text and the 'regularity conditions' are left unspecified. The authors should present a complete proof with explicit conditions, either in the main text or in a supplement that is actually included and verifiable; as written, the existence of the bridge functions and the validity of Eq. (3.3) cannot be checked from the manuscript.
  5. [Section 3.3, Eq. (3.11)] The policy-gradient formula treats θ as a function of ζ but the displayed derivative does not account for the dependence of Q̂π on ζ through the estimation procedure and through the Bellman equation defining the bridge. The authors should clarify whether Algorithm 2 is an exact gradient method for the objective in (3.10) or a heuristic alternating scheme, and if the latter, what guarantees apply.
minor comments (5)
  1. [Introduction] There is a typo: 'the the curse of dimensionality' should read 'the curse of dimensionality'.
  2. [Section 4] There is a typo: 'demnstrated' should read 'demonstrated'.
  3. [Assumption 4] The completeness assumption is unverifiable from data; the authors should more explicitly acknowledge that the identification and all subsequent results depend on it, and discuss its plausibility in the Pairfam application.
  4. [Figure 2] The y-axis label 'l g MSE' should be 'log MSE'.
  5. [Section 6] The sentence 'As shown in Figure 2, our proposed OPE method exhibits the smallest bias...' appears to reference the simulation figure rather than a figure in the application section; please check the cross-reference.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: identification and estimation are derived from stated assumptions; cited proximal-causal machinery is prior published work.

full rationale

The paper's core identification result (Theorem 3.1) is not circular: the bridge functions are defined by the conditional moment restrictions (3.1)-(3.2), and equation (3.3) is presented as a derived consequence under Assumptions 1-4, not as an input. The minimax estimator (3.5) is obtained by minimizing the empirical analog of the conditional moment equation; no parameter is fitted to a target value and then renamed a prediction. The proximal-causal framework is imported from published prior work (Tchetgen Tchetgen et al. 2020; Cui et al. 2023; Miao et al. 2018), including by co-author Cui, but the present identification theorem is not justified merely by citing that work; the paper extends the framework under explicitly stated assumptions. Assumption 4 (completeness) is an unverifiable injectivity condition, but an assumption is not circularity. The theoretical-rate concerns raised by the skeptic (Theorem 4.1 has no sample-size term; Theorem 4.2 mixes o_P remainders with finite-sample probabilities) are correctness/reproducibility flaws in the proof, not derivation-by-construction circularity. The simulations and Pairfam application benchmark against external baselines rather than predicting values that were fit as inputs. No circular step can be exhibited from the text.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical or model entities are introduced; V/Q bridge functions are inherited from proximal causal inference and off-policy evaluation literature. The free parameters are regularizers and kernel bandwidths that are selected by cross-validation or heuristics, not physically motivated constants.

free parameters (3)
  • λ_n = chosen via k-fold cross-validation in Section 5
    Regularizer for the Q-bridge function in the finite-sample minimax estimator (3.5); the value is data-dependent and affects the fitted bridge function.
  • μ_n = chosen via k-fold cross-validation in Section 5
    Regularizer for the critic function class in (3.5); selected together with λ_n on validation policy value.
  • Kernel bandwidth bw0 = median heuristic
    Bandwidth of the Gaussian RKHS used for the critic function; determines the Gram matrix K_{nT} in (3.8) and the estimator behavior.
assumptions (5)
  • domain assumption Assumption 1: Markovianity of (S_t,O_t,A_t)
    States that next states depend on the past only through the current full state and action; used throughout for Bellman-style equations.
  • domain assumption Assumption 2: W_t is a reward-inducing proxy, W_t ⟂ (A_t,S_{t-1},O_{t-1}) | S_t,O_t, and W_t not independent of S_t | O_t
    Ensures W_t carries information about the hidden confounder without being affected by the action; central to identification.
  • domain assumption Assumption 3: (O_{t-1},A_{t-1}) ⟂ (W_t,R_t) | S_t,O_t,A_t
    Treats the past state-action pair as an action-inducing proxy in the proximal causal inference framework.
  • domain assumption Assumption 4: completeness and injectivity of conditional expectation operators over S_t and over (O_{t-1},A_{t-1})
    Unverifiable mathematical condition that guarantees bridge functions exist; failure would invalidate Theorem 3.1 and all subsequent rates.
  • domain assumption Assumptions 5-10: compact Lipschitz policy class, bounded rewards and function classes, well-posedness κ>0, metric entropy bounds, smooth direction function and ξ, coverage lower bounds
    Regularity and overlap conditions needed for the rates in Theorems 4.1-4.3; these are standard but strong and are not verifiable from the data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reinforcement Learning with Continuous Actions Under Unmeasured Confounding." pith.science (2026). https://pith.science/paper/LUHB2V4D

@misc{pith2026250500304,
  author       = {Pith},
  title        = {Pith review of: Reinforcement Learning with Continuous Actions Under Unmeasured Confounding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LUHB2V4D}},
  note         = {Machine review of arXiv:2505.00304}
}
read the original abstract

This paper addresses the challenge of offline policy learning in reinforcement learning with continuous action spaces when unmeasured confounders are present. While most existing research focuses on policy evaluation within partially observable Markov decision processes (POMDPs) and assumes discrete action spaces, we advance this field by establishing a novel identification result to enable the nonparametric estimation of policy value for a given target policy under an infinite-horizon framework. Leveraging this identification, we develop a minimax estimator and introduce a policy-gradient-based algorithm to identify the in-class optimal policy that maximizes the estimated policy value. Furthermore, we provide theoretical results regarding the consistency, finite-sample error bound, and regret bound of the resulting optimal policy. Extensive simulations and a real-world application using the German Family Panel data demonstrate the effectiveness of our proposed methodology.

Figures

Figures reproduced from arXiv: 2505.00304 by the authors.

Figure 1
Figure 1. The observed data for the confounded POMDP then have the form of Dn = {Di } n i=1 = {O i 0 , Wi 0 , Ai 0 , Ri 0 , Oi 1 , . . . , Oi T , Wi T , Ai T , Ri T , Oi T +1, Wi T +1} n i=1. The left panel of [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Logarithms of relative MSEs of the proposed (blue sqaures), MDPW (orange circles), [PITH_FULL_IMAGE:figures/full_fig_p022_2.png] view at source ↗
Figure 4
Figure 4. The estimated optimal policy distribution under typical states. The corresponding states are defined in [PITH_FULL_IMAGE:figures/full_fig_p026_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

79 extracted references · 64 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence := #2 '...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in " " * FUNCTION format....

  3. [3]

    Algoe, S. B. (2012), Find, remind, and bind: The functions of gratitude in everyday relationships, Social and personality psychology compass, 6, 455--469

    work page 2012

  4. [4]

    (2008), Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path, Machine Learning, 71, 89--129

    Antos, A., Szepesv \'a ri, C., and Munos, R. (2008), Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path, Machine Learning, 71, 89--129

    work page 2008

  5. [5]

    (2017), Breaking the curse of dimensionality with convex neural networks, The Journal of Machine Learning Research, 18, 629--681

    Bach, F. (2017), Breaking the curse of dimensionality with convex neural networks, The Journal of Machine Learning Research, 18, 629--681

    work page 2017

  6. [6]

    Beckmeyer, J. J. and Jamison, T. B. (2021), Identifying a typology of emerging adult romantic relationships: Implications for relationship education, Family Relations, 70, 305--318

    work page 2021

  7. [7]

    and Kallus, N

    Bennett, A. and Kallus, N. (2023), Proximal reinforcement learning: Efficient off-policy evaluation in partially observed markov decision processes, Operations Research

    work page 2023

  8. [8]

    (2021), Off-policy evaluation in infinite-horizon reinforcement learning with latent confounders, in International Conference on Artificial Intelligence and Statistics, PMLR, pp

    Bennett, A., Kallus, N., Li, L., and Mousavi, A. (2021), Off-policy evaluation in infinite-horizon reinforcement learning with latent confounders, in International Conference on Artificial Intelligence and Statistics, PMLR, pp. 1999--2007

    work page 2021

Show all 79 references
  1. [9]

    and Kennedy, E

    Bonvini, M. and Kennedy, E. H. (2022), Sensitivity analysis via the proportion of unmeasured confounding, Journal of the American Statistical Association, 117, 1540--1550

    2022

  2. [10]

    u derl, J., Schmiedeberg, C., Castiglioni, L., Arr \'a nz Becker, O., Buhr, P., Fu , D., Ludwig, V., Schr \

    Br \"u derl, J., Schmiedeberg, C., Castiglioni, L., Arr \'a nz Becker, O., Buhr, P., Fu , D., Ludwig, V., Schr \"o der, J., and Schumann, N. (2023), The German Family Panel: Study Design and Cumulated Field Report (Waves 1 to 14),

    2023

  3. [11]

    W., Yuan, Z., Zhou, S., Panerati, J., and Schoellig, A

    Brunke, L., Greeff, M., Hall, A. W., Yuan, Z., Zhou, S., Panerati, J., and Schoellig, A. P. (2022), Safe learning in robotics: From learning-based control to safe reinforcement learning, Annual Review of Control, Robotics, and Autonomous Systems, 5, 411--444

    2022

  4. [12]

    Bruns-Smith, D. A. (2021), Model-free and model-based policy evaluation when causality is uncertain, in International Conference on Machine Learning, PMLR, pp. 1116--1126

    2021

  5. [13]

    (2021), Jump interval-learning for individualized decision making, arXiv preprint arXiv:2111.08885

    Cai, H., Shi, C., Song, R., and Lu, W. (2021), Jump interval-learning for individualized decision making, arXiv preprint arXiv:2111.08885

    2021 arXiv

  6. [14]

    (2022), Reinforcement learning from partial observation: Linear function approximation with provable sample efficiency, in International Conference on Machine Learning, PMLR, pp

    Cai, Q., Yang, Z., and Wang, Z. (2022), Reinforcement learning from partial observation: Linear function approximation with provable sample efficiency, in International Conference on Machine Learning, PMLR, pp. 2485--2522

    2022

  7. [15]

    and Qi, Z

    Chen, X. and Qi, Z. (2022), On well-posedness and minimax optimal rates of nonparametric q-function estimation in off-policy evaluation, in International Conference on Machine Learning, PMLR, pp. 3558--3582

    2022

  8. [16]

    Chou, P.-W., Maturana, D., and Scherer, S. (2017), Improving stochastic policy gradients in continuous control with deep reinforcement learning using the beta distribution, in International conference on machine learning, PMLR, pp. 834--843

    2017

  9. [17]

    (2023), Semiparametric proximal causal inference, Journal of the American Statistical Association, 1--12

    Cui, Y., Pu, H., Shi, X., Miao, W., and Tchetgen Tchetgen, E. (2023), Semiparametric proximal causal inference, Journal of the American Statistical Association, 1--12

    2023

  10. [18]

    and Tchetgen Tchetgen, E

    Cui, Y. and Tchetgen Tchetgen, E. (2021), A semiparametric instrumental variable approach to optimal treatment regimes under endogeneity, Journal of the American Statistical Association, 116, 162--173

    2021

  11. [19]

    (2020), Minimax estimation of conditional moment models, Advances in Neural Information Processing Systems, 33, 12248--12262

    Dikkala, N., Lewis, G., Mackey, L., and Syrgkanis, V. (2020), Minimax estimation of conditional moment models, Advances in Neural Information Processing Systems, 33, 12248--12262

    2020

  12. [20]

    (2011), On the completeness condition in nonparametric instrumental problems, Econometric Theory, 27, 460--471

    D’Haultfoeuille, X. (2011), On the completeness condition in nonparametric instrumental problems, Econometric Theory, 27, 460--471

    2011

  13. [21]

    (2016), Regularized policy iteration with nonparametric function spaces, Journal of Machine Learning Research, 17, 1--66

    Farahm, A.-m., Ghavamzadeh, M., Szepesv \'a ri, C., and Mannor, S. (2016), Regularized policy iteration with nonparametric function spaces, Journal of Machine Learning Research, 17, 1--66

    2016

  14. [22]

    H., Moreau, Y., Murphy, S

    Fr \"o hlich, H., Balling, R., Beerenwinkel, N., Kohlbacher, O., Kumar, S., Lengauer, T., Maathuis, M. H., Moreau, Y., Murphy, S. A., Przytycka, T. M., et al. (2018), From hype to reality: data science enabling personalized medicine, BMC medicine, 16, 1--15

    2018

  15. [23]

    Fu, Z., Qi, Z., Wang, Z., Yang, Z., Xu, Y., and Kosorok, M. R. (2022), Offline reinforcement learning with instrumental variables in confounded markov decision processes, arXiv preprint arXiv:2209.08666

    2022 arXiv

  16. [24]

    and Gu, S

    Fujimoto, S. and Gu, S. S. (2021), A minimalist approach to offline reinforcement learning, Advances in neural information processing systems, 34, 20132--20145

    2021

  17. [25]

    Fukumizu, K., Gretton, A., Lanckriet, G., Sch \"o lkopf, B., and Sriperumbudur, B. K. (2009), Kernel choice and classifiability for RKHS embeddings of probability distributions, Advances in neural information processing systems, 22

    2009

  18. [26]

    Geer, S. A. (2000), Empirical Processes in M-estimation, vol. 6, Cambridge university press

    2000

  19. [27]

    (2022), Provably efficient offline reinforcement learning for partially observable markov decision processes, in International Conference on Machine Learning, PMLR, pp

    Guo, H., Cai, Q., Zhang, Y., Yang, Z., and Wang, Z. (2022), Provably efficient offline reinforcement learning for partially observable markov decision processes, in International Conference on Machine Learning, PMLR, pp. 8016--8038

    2022

  20. [28]

    (2018), Soft actor-critic algorithms and applications, arXiv preprint arXiv:1812.05905

    Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018), Soft actor-critic algorithms and applications, arXiv preprint arXiv:1812.05905

    2018 arXiv

  21. [29]

    (2021), Bootstrapping fitted q-evaluation for off-policy inference, in International Conference on Machine Learning, PMLR, pp

    Hao, B., Ji, X., Duan, Y., Lu, H., Szepesvari, C., and Wang, M. (2021), Bootstrapping fitted q-evaluation for off-policy inference, in International Conference on Machine Learning, PMLR, pp. 4074--4084

    2021

  22. [30]

    W., Lazaric, A., Ghavamzadeh, M., and Munos, R

    Hoffman, M. W., Lazaric, A., Ghavamzadeh, M., and Munos, R. (2011), Regularized least squares temporal difference learning with nested L2 and L1 penalization, in European Workshop on Reinforcement Learning, Springer, pp. 102--114

    2011

  23. [31]

    (2023), A Policy Gradient Method for Confounded POMDPs, arXiv preprint arXiv:2305.17083

    Hong, M., Qi, Z., and Xu, Y. (2023), A Policy Gradient Method for Confounded POMDPs, arXiv preprint arXiv:2305.17083

    2023 arXiv

  24. [32]

    (2020), Sample-efficient reinforcement learning of undercomplete pomdps, Advances in Neural Information Processing Systems, 33, 18530--18539

    Jin, C., Kakade, S., Krishnamurthy, A., and Liu, Q. (2020), Sample-efficient reinforcement learning of undercomplete pomdps, Advances in Neural Information Processing Systems, 33, 18530--18539

    2020

  25. [33]

    (2022), Doubly robust distributionally robust off-policy evaluation and learning, in International Conference on Machine Learning, PMLR, pp

    Kallus, N., Mao, X., Wang, K., and Zhou, Z. (2022), Doubly robust distributionally robust off-policy evaluation and learning, in International Conference on Machine Learning, PMLR, pp. 10598--10632

    2022

  26. [34]

    and Uehara, M

    Kallus, N. and Uehara, M. (2020), Statistically efficient off-policy policy gradients, in International Conference on Machine Learning, PMLR, pp. 5089--5100

    2020

  27. [35]

    and Zhou, A

    Kallus, N. and Zhou, A. (2020), Confounding-robust policy evaluation in infinite-horizon reinforcement learning, Advances in neural information processing systems, 33, 22293--22304

    2020

  28. [36]

    --- (2021), Minimax-Optimal Policy Learning Under Unobserved Confounding, Management Science, 67, 2870--2890

    2021

  29. [37]

    B., Volkmann, J

    Kashdan, T. B., Volkmann, J. R., Breen, W. E., and Han, S. (2007), Social anxiety and romantic relationships: The costs and benefits of negative emotion expression are context-dependent, Journal of Anxiety Disorders, 21, 475--492

    2007

  30. [38]

    (2021), Offline reinforcement learning with implicit q-learning, arXiv preprint arXiv:2110.06169

    Kostrikov, I., Nair, A., and Levine, S. (2021), Offline reinforcement learning with implicit q-learning, arXiv preprint arXiv:2110.06169

    2021 arXiv

  31. [39]

    (1989), Linear integral equations, vol

    Kress, R., Maz'ya, V., and Kozlov, V. (1989), Linear integral equations, vol. 82, Springer

    1989

  32. [40]

    (2020), Conservative q-learning for offline reinforcement learning, Advances in Neural Information Processing Systems, 33, 1179--1191

    Kumar, A., Zhou, A., Tucker, G., and Levine, S. (2020), Conservative q-learning for offline reinforcement learning, Advances in Neural Information Processing Systems, 33, 1179--1191

    2020

  33. [41]

    Le, B. M. and Impett, E. A. (2013), When Holding Back Helps: Suppressing Negative Emotions During Sacrifice Feels Authentic and Is Beneficial for Highly Interdependent People, Psychological Science, 24, 1809--1815

    2013

  34. [42]

    (2019), Batch policy learning under constraints, in International Conference on Machine Learning, PMLR, pp

    Le, H., Voloshin, C., and Yue, Y. (2019), Batch policy learning under constraints, in International Conference on Machine Learning, PMLR, pp. 3703--3712

    2019

  35. [43]

    (2018), Deep reinforcement learning in continuous action spaces: a case study in the game of simulated curling, in International conference on machine learning, PMLR, pp

    Lee, K., Kim, S.-A., Choi, J., and Lee, S.-W. (2018), Deep reinforcement learning in continuous action spaces: a case study in the game of simulated curling, in International conference on machine learning, PMLR, pp. 2937--2946

    2018

  36. [44]

    (2020), Offline reinforcement learning: Tutorial, review, and perspectives on open problems, arXiv preprint arXiv:2005.01643

    Levine, S., Kumar, A., Tucker, G., and Fu, J. (2020), Offline reinforcement learning: Tutorial, review, and perspectives on open problems, arXiv preprint arXiv:2005.01643

    2020 arXiv

  37. [45]

    (2021), Causal reinforcement learning: An instrumental variable approach, arXiv preprint arXiv:2103.04021

    Li, J., Luo, Y., and Zhang, X. (2021), Causal reinforcement learning: An instrumental variable approach, arXiv preprint arXiv:2103.04021

    2021 arXiv

  38. [46]

    (2023), Quasi-optimal Reinforcement Learning with Continuous Actions, in The Eleventh International Conference on Learning Representations

    Li, Y., Zhou, W., and Zhu, R. (2023), Quasi-optimal Reinforcement Learning with Continuous Actions, in The Eleventh International Conference on Learning Representations

    2023

  39. [47]

    (2021), Off-policy estimation of long-term average outcomes with applications to mobile health, Journal of the American Statistical Association, 116, 382--391

    Liao, P., Klasnja, P., and Murphy, S. (2021), Off-policy estimation of long-term average outcomes with applications to mobile health, Journal of the American Statistical Association, 116, 382--391

    2021

  40. [48]

    Liao, P., Qi, Z., Wan, R., Klasnja, P., and Murphy, S. A. (2022), Batch policy learning in average reward markov decision processes, Annals of statistics, 50, 3364

    2022

  41. [49]

    P., Hunt, J

    Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015), Continuous control with deep reinforcement learning, arXiv preprint arXiv:1509.02971

    2015 arXiv

  42. [50]

    (2018), Breaking the curse of horizon: Infinite-horizon off-policy estimation, Advances in neural information processing systems, 31

    Liu, Q., Li, L., Tang, Z., and Zhou, D. (2018), Breaking the curse of horizon: Infinite-horizon off-policy estimation, Advances in neural information processing systems, 31

    2018

  43. [51]

    (2022), Pessimism in the face of confounders: Provably efficient offline reinforcement learning in partially observable markov decision processes, arXiv preprint arXiv:2205.13589

    Lu, M., Min, Y., Wang, Z., and Yang, Z. (2022), Pessimism in the face of confounders: Provably efficient offline reinforcement learning in partially observable markov decision processes, arXiv preprint arXiv:2205.13589

    2022 arXiv

  44. [52]

    J., Laber, E

    Luckett, D. J., Laber, E. B., Kahkoska, A. R., Maahs, D. M., Mayer-Davis, E., and Kosorok, M. R. (2019), Estimating dynamic treatment regimes in mobile health using v-learning, Journal of the American Statistical Association

    2019

  45. [53]

    Miao, R., Qi, Z., and Zhang, X. (2022), Off-policy evaluation for episodic partially observable markov decision processes under non-parametric models, Advances in Neural Information Processing Systems, 35, 593--606

    2022

  46. [54]

    Miao, W., Geng, Z., and Tchetgen Tchetgen, E. J. (2018), Identifying causal effects with proxy variables of an unmeasured confounder, Biometrika, 105, 987--993

    2018

  47. [55]

    A., Veness, J., Bellemare, M

    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015), Human-level control through deep reinforcement learning, nature, 518, 529--533

    2015

  48. [56]

    Newey, W. K. and Powell, J. L. (2003), Instrumental variable estimation of nonparametric models, Econometrica, 71, 1565--1578

    2003

  49. [57]

    Ogolsky, B. G. and Bowers, J. R. (2013), A meta-analytic review of relationship maintenance and its correlates, Journal of Social and Personal Relationships, 30, 343--367

    2013

  50. [58]

    (2000), Eligibility traces for off-policy policy evaluation, Computer Science Department Faculty Publication Series, 80

    Precup, D. (2000), Eligibility traces for off-policy policy evaluation, Computer Science Department Faculty Publication Series, 80

    2000

  51. [59]

    (2023), Proximal learning for individualized treatment regimes under unmeasured confounding, Journal of the American Statistical Association, 1--14

    Qi, Z., Miao, R., and Zhang, X. (2023), Proximal learning for individualized treatment regimes under unmeasured confounding, Journal of the American Statistical Association, 1--14

    2023

  52. [60]

    Shi, C., Uehara, M., Huang, J., and Jiang, N. (2022 a ), A minimax learning approach to off-policy evaluation in confounded partially observable markov decision processes, in International Conference on Machine Learning, PMLR, pp. 20057--20094

    2022

  53. [61]

    Shi, C., Zhang, S., Lu, W., and Song, R. (2022 b ), Statistical inference of the value function for reinforcement learning in infinite-horizon settings, Journal of the Royal Statistical Society Series B: Statistical Methodology, 84, 765--793

    2022

  54. [62]

    (2022 c ), Off-policy confidence interval estimation with confounded markov decision process, Journal of the American Statistical Association, 1--12

    Shi, C., Zhu, J., Ye, S., Luo, S., Zhu, H., and Song, R. (2022 c ), Off-policy confidence interval estimation with confounded markov decision process, Journal of the American Statistical Association, 1--12

    2022

  55. [63]

    Sutton, R. S. and Barto, A. G. (2018), Reinforcement learning: An introduction, MIT press

    2018

  56. [64]

    (2020), An introduction to proximal causal learning, arXiv preprint arXiv:2009.10982

    Tchetgen Tchetgen, E., Ying, A., Cui, Y., Shi, X., and Miao, W. (2020), An introduction to proximal causal learning, arXiv preprint arXiv:2009.10982

    2020 arXiv

  57. [65]

    and Brunskill, E

    Thomas, P. and Brunskill, E. (2016), Data-efficient off-policy policy evaluation for reinforcement learning, in International Conference on Machine Learning, PMLR, pp. 2139--2148

    2016

  58. [66]

    (2020), Minimax weight and q-function learning for off-policy evaluation, in International Conference on Machine Learning, PMLR, pp

    Uehara, M., Huang, J., and Jiang, N. (2020), Minimax weight and q-function learning for off-policy evaluation, in International Conference on Machine Learning, PMLR, pp. 9659--9668

    2020

  59. [67]

    (2024), Future-dependent value-based off-policy evaluation in pomdps, Advances in Neural Information Processing Systems, 36

    Uehara, M., Kiyohara, H., Bennett, A., Chernozhukov, V., Jiang, N., Kallus, N., Shi, C., and Sun, W. (2024), Future-dependent value-based off-policy evaluation in pomdps, Advances in Neural Information Processing Systems, 36

    2024

  60. [68]

    and Groothuis-Oudshoorn, K

    Van Buuren, S. and Groothuis-Oudshoorn, K. (2011), mice: Multivariate imputation by chained equations in R, Journal of statistical software, 45, 1--67

    2011

  61. [69]

    and Zou, S

    Wang, Y. and Zou, S. (2022), Policy gradient method for robust reinforcement learning, in International conference on machine learning, PMLR, pp. 23484--23526

    2022

  62. [70]

    (2019), Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling, Advances in neural information processing systems, 32

    Xie, T., Ma, Y., and Wang, Y.-X. (2019), Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling, Advances in neural information processing systems, 32

    2019

  63. [71]

    (2023), An instrumental variable approach to confounded off-policy evaluation, in International Conference on Machine Learning, PMLR, pp

    Xu, Y., Zhu, J., Shi, C., Luo, S., and Song, R. (2023), An instrumental variable approach to confounded off-policy evaluation, in International Conference on Machine Learning, PMLR, pp. 38848--38880

    2023

  64. [72]

    and Bareinboim, E

    Zhang, J. and Bareinboim, E. (2016), Markov decision processes with unobserved confounders: A causal approach, Purdue AI Lab, West Lafayette, IN, USA, Tech. Rep

    2016

  65. [73]

    (2020), Causal imitation learning with unobserved confounders, Advances in neural information processing systems, 33, 12263--12274

    Zhang, J., Kumor, D., and Bareinboim, E. (2020), Causal imitation learning with unobserved confounders, Advances in neural information processing systems, 33, 12263--12274

    2020

  66. [74]

    and Jiang, N

    Zhang, Y. and Jiang, N. (2024), On the Curses of Future and History in Future-dependent Value Functions for Off-policy Evaluation, arXiv preprint arXiv:2402.14703

    2024 arXiv

  67. [75]

    (2024), Bi-Level Offline Policy Optimization with Limited Exploration, Advances in Neural Information Processing Systems, 36

    Zhou, W. (2024), Bi-Level Offline Policy Optimization with Limited Exploration, Advances in Neural Information Processing Systems, 36

    2024

  68. [76]

    (2024 a ), Policy learning for individualized treatment regimes on infinite time horizon, in Statistics in Precision Health: Theory, Methods and Applications, Springer, pp

    Zhou, W., Li, Y., and Zhu, R. (2024 a ), Policy learning for individualized treatment regimes on infinite time horizon, in Statistics in Precision Health: Theory, Methods and Applications, Springer, pp. 65--100

    2024

  69. [77]

    (2023), Distributional shift-aware off-policy interval estimation: A unified error quantification framework, arXiv preprint arXiv:2309.13278

    Zhou, W., Li, Y., Zhu, R., and Qu, A. (2023), Distributional shift-aware off-policy interval estimation: A unified error quantification framework, arXiv preprint arXiv:2309.13278

    2023 arXiv

  70. [78]

    (2024 b ), Estimating optimal infinite horizon dynamic treatment regimes via pt-learning, Journal of the American Statistical Association, 119, 625--638

    Zhou, W., Zhu, R., and Qu, A. (2024 b ), Estimating optimal infinite horizon dynamic treatment regimes via pt-learning, Journal of the American Statistical Association, 119, 625--638

    2024

  71. [79]

    (2020), Safe, efficient, and comfortable velocity control based on reinforcement learning for autonomous driving, Transportation Research Part C: Emerging Technologies, 117, 102662

    Zhu, M., Wang, Y., Pu, Z., Hu, J., Wang, X., and Ke, R. (2020), Safe, efficient, and comfortable velocity control based on reinforcement learning for autonomous driving, Transportation Research Part C: Emerging Technologies, 117, 102662

    2020

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.