Semi-pessimistic Reinforcement Learning

Chengchun Shi; Gholamali Aminian; Jiaang Yao; Jin Zhu; Lexin Li; Omar Rivasplata; Simon Little; Xin Zhou

REVIEW 3 major objections 5 minor 61 references

Semi-pessimistic Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Semi-pessimistic pseudo labeling provably turns abundant unlabeled transitions into near-optimal offline RL policies under a mild semi-coverage condition.

desk verdict Nice semi-pessimistic RL idea, but the central guarantee is not actually derived: the regret bound leaves the reward error term unquantified, and the advertised improvement over labeled-only methods is not proven. read the letter →

arxiv 2505.19002 v1 pith:FNZM4IX3 submitted 2025-05-25 cs.LG cs.AIstat.ML

Jin Zhu , Xin Zhou , Jiaang Yao , Gholamali Aminian , Omar Rivasplata , Simon Little , Lexin Li , Chengchun Shi This is my paper

classification cs.LGcs.AIstat.ML

keywords offlinereinforcementlearningsemi-supervisedpessimisticprinciplepseudolabelinguncertaintyquantificationregretanalysiscoverageconditionsadaptivedeepbrainstimulation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a semi-supervised offline reinforcement learning method, semi-pessimistic pseudo labeling (SPL), that learns a one-sided lower confidence bound on the reward function, imputes pessimistic rewards for unlabeled transitions, and then runs a standard RL algorithm on the combined labeled and unlabeled data. The central claim is that abundant unlabeled data guarantees policy improvement under a new semi-coverage condition: the labeled data need only cover the optimal policy's state-action distribution, while the union of labeled and unlabeled data must cover every policy. The regret analysis shows that in the infinite-unlabeled-data limit, the regret is dominated by the reward estimation error over the labeled distribution, so a tighter reward lower bound directly yields a better policy. If correct, this matters for applications such as adaptive deep brain stimulation, where rewards are measured rarely but state-action observations are plentiful, because it shows unlabeled data alone can compensate for missing reward information.

What carries the argument

The load-bearing object is the semi-pessimistic pseudo-label reward $\hat R_{\mathrm{SPL}}(s,a) = \hat R_{\mathrm{SUQ}}(s,a) - z_{1-\alpha/2}\,\Delta_{\mathrm{SUQ}}(s,a)$, a one-sided lower confidence bound on the expected reward built by semi-supervised uncertainty quantification: an auxiliary model (a random forest in the implementation) is fit on the labeled data, and its projection onto a linear feature space is refined using the unlabeled data to tighten the variance estimate. The companion concept is semi-coverage: the labeled data cover the optimal policy's visitation distribution ($B_L^* < \infty$) while labeled plus unlabeled data cover every policy ($B_D < \infty$). The proofs use the performance difference lemma to turn regret into a reward estimation error term plus a supervised learning error term, with pointwise rather than uniform uncertainty quantification of the reward replacing the usual sequential Q-function uncertainty quantification.

What would settle it

On a small tabular MDP with known rewards, compute $\lVert R - \hat R_{\mathrm{SPL}}\rVert_{d_L}$ directly as the labeled sample size grows under fixed noise; if this quantity does not approach zero, Corollary 1 predicts a nonvanishing regret and the guaranteed-improvement claim fails. A sharper check is to construct a case where the one-sided lower bound is biased downward by a constant, in which case the regret bound is dominated by that constant and no amount of unlabeled data can remove it.

Watch

Extended reading notes

Core claim

The paper's central assertion is that a policy trained on pessimistically imputed rewards is provably near-optimal when the combined dataset satisfies semi-coverage. Formally, Corollary 1 states that with infinitely many unlabeled transitions, a sufficiently large number of fitted Q-iteration steps, and a vanishing significance level, the regret satisfies $\mathbb{E}[J(\pi^*) - J(\hat\pi)] = O\!\left(\sqrt{B_L^*}\,\lVert R - \hat R_{\mathrm{SPL}}\rVert_{d_L}\right)$, so the only remaining error is how well the pessimistic reward lower bound approximates the true reward in the labeled-data norm. Theorems 1 and 2 give finite-sample versions for the model-free (fitted Q iteration) and model-based (MOPO-style) instantiations, and the analytic comparisons argue that this regret is smaller than pseudo labeling without pessimism, which requires full coverage on the labeled data, and than unlabeled data sharing, whose minimal-reward imputation keeps a constant reward estimation error.

Load-bearing premise

The whole argument rests on the pessimistic reward estimate being close to the true reward over the labeled data; the paper only proves the estimate sits below the true reward with high probability, not that the gap shrinks as more data arrive.

Editorial extensions

If this is right

With infinitely many unlabeled transitions, the regret of both the model-free and model-based algorithms is dominated by $\lVert R - \hat R_{\mathrm{SPL}}\rVert_{d_L}$, so sharper reward lower bounds translate directly into better policies.
Only a pointwise, one-sided lower bound on the reward is needed, which avoids the Bonferroni-style inflation that sequential uniform Q-function uncertainty quantification incurs.
The pessimistic pseudo-labeling step is algorithm-agnostic: it can be coupled with fitted Q iteration or with model-based policy optimization such as the MOPO-style algorithm, reusing their existing machinery.
Under semi-coverage, the transition estimation error of the model-based variant shrinks at rate $n^{-1/2}$ as the unlabeled data grow, so the labeled-data reward error becomes the bottleneck.
In the adaptive deep brain stimulation study with a 120-to-1 ratio of unlabeled to labeled observations, SPL attains the highest cumulative reward among the compared methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the theorems deliver conditional regret bounds; the 'guaranteed improvement' reading requires $\lVert R - \hat R_{\mathrm{SPL}}\rVert_{d_L}$ to vanish with the labeled sample size, which is not proven, and could be tested directly by measuring the labeled-data bias of the pessimistic estimator on an MDP with known rewards.
Editorial inference: the same semi-pessimistic construction could be applied to impute other missing quantities, such as value targets or successor features, whenever unlabeled transitions are plentiful and a pointwise lower bound is available.
Editorial inference: the SPL-versus-PPL comparison suggests a data-driven spectrum between reward-only pessimism and full transition-aware pessimism; the better operating point likely depends on how close the unlabeled behavior policy is to the optimal policy.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper proposes a simple and appealing way to use unlabeled data in offline RL: build a pessimistic reward estimator, then run any standard RL algorithm. That is a genuinely simpler recipe than the usual pessimistic Q-function or transition penalty, and the semi-coverage condition (partial coverage on labeled data, full coverage on the union) is a reasonable middle ground. The analytic comparison with PL, UDS, and PDS is useful, and the adaptive DBS application is a good motivating example.

What is actually new: the semi-pessimistic principle, the notion of semi-coverage, and the use of a pointwise reward lower confidence bound from prediction-powered inference. The paper does a solid job explaining the method and situating it relative to existing work. The numerical results are plausible, and the empirical comparisons are broad.

The soft spots are real. The main theorems bound regret by, among other terms, ||R - bR_SPL||_{dL}, and Corollary 1 states that with infinite unlabeled data and small alpha, regret is O(sqrt(B_L^*) ||R - bR_SPL||_{dL}). No theorem shows that this reward error term is small or shrinks. Lemma 2 only compares the uncertainty width Delta_SUG to Delta_INI, not the error norm. The authors assert in Section 4.3 that the error is O_p(n_L^{-1/2}), but that does not follow from the stated assumptions unless the linear model is correctly specified, which they never assume. As n_U -> infinity, bR_SUG converges to the projection of R onto the feature space, not to R itself. So the 'guaranteed improvement' advertised in the abstract is not derived.

There is also a smaller technical mismatch: Assumption 2 gives a one-sided lower confidence bound, but the proofs use a two-sided event {|bR_l - R| <= Gamma}. This is fixable by replacing the event with the one-sided one, so I do not treat it as a core problem.

One more issue: in the synthetic experiments the appendix describes an uncertainty-quantile filter that retains only state-action pairs with Delta below the q-th quantile (q=0.9 full coverage, 0.3 partial). This filter is not mentioned in the main text and has no theoretical counterpart, so the experimental comparison is not as clean as the paper suggests.

Who is this for: researchers working on offline RL with missing rewards or semi-supervised RL. They will find the semi-coverage condition and the pointwise uncertainty idea useful, even if they do not accept the current guarantee. The paper deserves a serious referee, but the authors need to either prove the reward error vanishes (under appropriate assumptions) or sharply weaken the improvement claim. I would send it to review, with the expectation of major revision.

Referee Report

3 major / 5 minor

Summary. This paper proposes a semi-supervised offline reinforcement learning method called semi-pessimistic pseudo labeling (SPL). The method learns a pointwise pessimistic lower bound for the reward function using both labeled and unlabeled data (semi-supervised uncertainty quantification, SUQ), uses this bound as a pseudo-label for the unlabeled tuples, and then plugs the augmented dataset into standard RL algorithms: a model-free algorithm based on fitted Q-iteration (Algorithm 2) and a model-based algorithm based on MOPO (Algorithm 3). The theoretical core (Theorems 1 and 2) upper-bounds the regret of the learned policy in terms of the semi-coverage coefficients B_L^* and B_D, a pointwise uncertainty quantification assumption (Assumption 2), the reward estimation error ||R − bR_SPL||_{dL}, and standard boundedness, mixing, and completeness conditions. Corollary 1 claims that, with infinite unlabeled data and a sufficiently small significance level α, the regret reduces to O(√B_L^* ||R − bR_SPL||_{dL}). The manuscript also provides an analytic comparison with pseudo-labeling, UDS, and PDS, and reports experiments on a synthetic environment, MuJoCo benchmarks, and a semi-synthetic adaptive deep brain stimulation application, where SPL is competitive.

Significance. The paper addresses a practically important problem — offline RL with scarce reward labels — and the high-level idea of replacing sequential Q-function pessimism with a single reward-level pessimistic fit is attractive for the reasons the authors state (simplicity, flexibility, computational feasibility for device-constrained applications). The semi-coverage condition (Assumption 1) is a natural intermediate coverage notion, and the numerical evaluation is a genuine strength: the DBS application and the comparisons against PL, UDS, and PDS are informative, and the experiments use a reasonable number of replications. The proof machinery in Appendix A (Lemma 3's β-mixing empirical-process bound for FQI) is standard in structure and appears technically careful. The weakness is that the abstract's central promise — a guaranteed improvement from vast unlabeled data under mild conditions — is not delivered by the theorems: the dominant reward-error term is never proved to vanish, the key Assumption 2 is asserted rather than derived, and the uncertainty-reduction mechanism (Lemma 2) rests on an assumption that essentially contains its own conclusion.

major comments (3)

[§4.2, Corollary 1; Theorems 1–2 (Eqs. (6)–(7))] The dominant term in Corollary 1 is ∥R − bRSPL∥_{dL}, and no theorem in the paper shows that this term vanishes or even shrinks. As nU → ∞, g⊤bθU converges to the projection of the auxiliary model bRAUX onto the span of g, and the construction (4) subtracts z_{1−α/2}∆SUG(s, a); neither the projection bias nor the penalty term is bounded in any lemma or theorem. The paper's own explanation in §4.3 that 'the error ∥R − bRSPL∥dL can be upper bounded by Op(n−1/2L)' is an assertion without proof. Moreover, Corollary 1 requires α to be sufficiently small, but the penalty z_{1−α/2} grows as α → 0, so the reward-error norm can increase as the significance level is decreased; no trade-off analysis is given. Consequently, E[J(π∗) − J(bπ)] is not shown to converge to zero even with infinite unlabeled data, and the abstract's 'guaranteed improvement' claim is left without support. To substantiate the advertised conclusion the authors would need a theorem establishing ∥R − bRSPL∥_{dL} = o_p(1), e.g., under correct specification of the linear reward model or a vanishing approximation error of the feature class; no such theorem appears.
[§3.3, Assumption 2; §A.1–A.5] Assumption 2 is the load-bearing premise of both theorems, yet its validity is never established. The text following (4) asserts that 'Following Angelopoulos et al. (2023), we can show that bRSPL(s, a) satisfies Assumption 2, provided that OLS offers a reasonable approximation of the true reward,' but no theorem proves the pointwise lower-confidence-bound property for the specific construction (2)–(4) with a random-forest auxiliary model, sandwich variance estimators, and a possibly mis-specified linear model g⊤θ. The debiased estimator g⊤bθSUQ has a bias equal to the projection error of R (or of bRAUX) onto the span of g; until this bias is controlled, a Wald-type interval is not a valid lower confidence bound. In addition, the proofs in §A.1 and §A.5 define the success event E(s, a) = {|bRSPL(s, a) − R(s, a)| ≤ Γα(s, a)}, which is two-sided, whereas Assumption 2 only guarantees the one-sided event {bRSPL(s, a) ≤ R(s, a)}; the regret bounds in Theorems 1 and 2 depend on the absolute deviation ∥R − bRSPL∥_{dL}, which is not controlled by the one-sided property at any rate. The authors should either give a rigorous derivation of Assumption 2 from primitive conditions (including the bias terms), or state explicitly that the reward-error norm is an unquantified quantity and adjust the claims accordingly.
[§3.3, Lemma 2 and Assumption 3] Lemma 2's claim that SUQ yields a tighter uncertainty width than the labeled-only benchmark is essentially assumed rather than derived. The proof reduces the conclusion to inequality (9), which is exactly the variance-dominance condition (5) stated in Assumption 3; the lemma restates the assumption in population form, with the only additional steps being the residual decomposition and the limit nL/nU → 0 with true covariance matrices. It does not address the finite-sample behavior of the estimated sandwich covariances used in Algorithms 1–3. More importantly, even granting Lemma 2, it does not transfer to the regret bounds: Theorems 1 and 2 depend on ∥R − bRSPL∥_{dL}, not on ∆SUG, and no inequality relating these two quantities (e.g., ∥R − bRSPL∥_{dL} ≤ C·∆SUG) appears in the manuscript; the remark after Corollary 1 that the reward error 'is typically proportional to the uncertainty quantification ∆' is a heuristic. The paper's claimed mechanism — tighter uncertainty from unlabeled data leading to a provably smaller regret — is therefore not established.

minor comments (5)

[§3.3, Eqs. (2)–(4), Algorithm 1] The estimators are subscripted 'SUG' (bθSUG, bRSUG, ∆SUG) throughout, while the method is named SUQ ('semi-supervised uncertainty quantification'); please use one consistent acronym.
[§5.3 and author affiliations] 'University of California at San Francesco' should read 'San Francisco'.
[§6, Theorem 3] Theorem 3 uses the notation ∥R − bRTAℓ∥_{dL} without defining bRTAℓ; the surrounding text defines bRPPL and ∆TAW, so these notations should be reconciled.
[§4.1, Assumption 6] Assumption 6 requires bRSPL to belong to a finite hypothesis class F almost surely, but the construction (2)–(4) with OLS and random Fourier features produces a continuously parameterized, data-dependent function; the assumption as stated does not cover the implemented algorithm (the remark after Assumption 7 indicates a VC-class relaxation, but the stated theorems rely on the finite-class form).
[§5.1, Figure 3(b)] In Figure 3(b) the horizontal axis is labeled 'Ratio' but the caption says 'the size of the labeled data nL (horizontal axis) varies'; the caption appears to be copied from panel (a) and should be corrected, since the experiment varies nU/nL with nL fixed at 32.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the regret bounds are proven from stated assumptions; the unsupported reward-error shrinkage is a correctness gap, not a definitional reduction.

full rationale

The paper's derivation chain is non-circular. Theorems 1 and 2 are upper bounds obtained from Assumptions 1, 2, and 4-7 via the performance-difference lemma, blocking arguments for beta-mixing data, and empirical process bounds in Appendix A; the bounds are not equal by construction to any fitted quantity. Corollary 1 specializes these bounds to n_U -> infinity and states that the regret is O(sqrt(B_L^*) ||R - bR_SPL||_{dL}); this is a direct algebraic consequence of Theorems 1 and 2, not a substitution of the conclusion into the premise. The reward estimator bR_SPL = bR_SUG - z_{1-alpha/2} Delta_SUG is a lower-confidence-bound construction imported from Angelopoulos et al. (2023), an external source with no author overlap; Assumption 2 is a validity condition for that construction, not the target regret. Lemma 2 proves Delta*_SUG <= Delta*_INI under Assumption 3 as a standalone population inequality; even if Assumption 3 is tailored, it is an assumption, not a circular definition. The paper's claim that ||R - bR_SPL||_{dL} is typically proportional to Delta and is O_p(n_L^{-1/2}) is asserted in Section 4.3 but not proved, so the advertised guaranteed improvement is not fully supported; that is a correctness gap, not circularity. No load-bearing self-citation occurs: the cited Angelopoulos et al. result is external, and the authors' own prior work is cited only as background for standard pessimistic and uncertainty-quantification techniques. Therefore the circularity score is 0.

Assumptions & free parameters 2 free parameters · 8 assumptions · 0 invented entities

The central claim rests on a chain of assumptions: semi-coverage, pointwise one-sided UQ, the auxiliary-model condition (3), boundedness, mixing, finite classes, completeness, and the implicit correctness of the linear reward model. The most fragile is the implicit bound on ||R - bR_SPL||_{dL}, which the paper does not prove.

free parameters (2)

Significance level alpha = 0.05 in experiments
User-chosen level controlling the pessimism penalty; appears linearly in the regret bounds.
Uncertainty quantile q = 0.9 (full coverage) and 0.3 (partial coverage) in synthetic experiments
Post-hoc filter selecting unlabeled tuples with low uncertainty; not part of Algorithm 1 and not analyzed in the theory.

assumptions (8)

domain assumption Assumption 1 (semi-coverage): dL covers the optimal policy's visitation (B_L^* finite) and dL∪U covers all policies (BD finite).
Introduced as the paper's core coverage relaxation; weaker than full coverage on L but stronger than partial coverage, and argued to become plausible as U grows.
domain assumption Assumption 2 (pointwise lower-bound UQ): P(bR_SPL(s,a) <= R(s,a)) >= 1-alpha for every (s,a).
Controls overestimation. Note that the proofs require the two-sided event |bR_SPL - R| <= Gamma, which is not implied by this one-sided assumption.
ad hoc to paper Assumption 3 (auxiliary model representation): bRAUX = beta0 R + beta1^T g + e with 0<beta0<2 and variance dominance condition (5).
Tailored to prove Lemma 2 that SUQ gives tighter uncertainty than using L alone; not a standard assumption in the cited literature.
standard math Assumption 4 (boundedness): rewards and estimated rewards bounded by Rmax; Q-functions bounded by Vmax.
Standard regularity for regret analysis.
domain assumption Assumption 5 (exponential beta-mixing): tuples in L∪U satisfy beta(q) <= kappa rho^q.
Required for the dependent-data Bernstein inequality in Lemma 3.
standard math Assumption 6 (finite hypothesis classes): bR_SPL in F and Q is finite.
Simplifies concentration arguments; the paper notes extensions to VC classes.
standard math Assumption 7 (completeness): for all f in F, Q in Q, f + gamma P max_a Q in Q.
Standard for fitted Q iteration; removes approximation error in the Bellman residual.
ad hoc to paper Implicit reward model correctness: the reward is well approximated by g^T theta so that the OLS-based lower bound is close to R in dL norm.
The paper only states 'provided that OLS offers a reasonable approximation of the true reward' (Section 3.3) without formal conditions; the claimed vanishing regret depends on this.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Semi-pessimistic Reinforcement Learning." pith.science (2026). https://pith.science/paper/FNZM4IX3

@misc{pith2026250519002,
  author       = {Pith},
  title        = {Pith review of: Semi-pessimistic Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FNZM4IX3}},
  note         = {Machine review of arXiv:2505.19002}
}

read the original abstract

Offline reinforcement learning (RL) aims to learn an optimal policy from pre-collected data. However, it faces challenges of distributional shift, where the learned policy may encounter unseen scenarios not covered in the offline data. Additionally, numerous applications suffer from a scarcity of labeled reward data. Relying on labeled data alone often leads to a narrow state-action distribution, further amplifying the distributional shift, and resulting in suboptimal policy learning. To address these issues, we first recognize that the volume of unlabeled data is typically substantially larger than that of labeled data. We then propose a semi-pessimistic RL method to effectively leverage abundant unlabeled data. Our approach offers several advantages. It considerably simplifies the learning process, as it seeks a lower bound of the reward function, rather than that of the Q-function or state transition function. It is highly flexible, and can be integrated with a range of model-free and model-based RL algorithms. It enjoys the guaranteed improvement when utilizing vast unlabeled data, but requires much less restrictive conditions. We compare our method with a number of alternative solutions, both analytically and numerically, and demonstrate its clear competitiveness. We further illustrate with an application to adaptive deep brain stimulation for Parkinson's disease.

Figures

Figures reproduced from arXiv: 2505.19002 by the authors.

**Figure 2.** Visualization of the state-action distribution of [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

**Figure 3.** Average regret of the policy learned by numerous model-free RL algorithms in [PITH_FULL_IMAGE:figures/full_fig_p022_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Average cumulative reward of the policy learned by numerous model-based RL [PITH_FULL_IMAGE:figures/full_fig_p024_4.png]

**Figure 5.** Figure 5: The cumulative reward of SPL and PPL under different values of [PITH_FULL_IMAGE:figures/full_fig_p028_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 49 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION article output.bibitem format.authors "author" output.check author format.key output output.year.check new.block format.title "title" output.check new.block crossref missing format.jour.vol output format.article.crossref output.nonnull format.pages output if new.block note output fin.entry FUNCTION b...
[2]

Angelopoulos, A. N., S. Bates, C. Fannjiang, M. I. Jordan, and T. Zrnic (2023). Prediction-powered inference. Science\/ 382\/ (6671), 669--674

2023
[3]

Bai, C., L. Wang, Z. Yang, Z.-H. Deng, A. Garg, P. Liu, and Z. Wang (2022). Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In International Conference on Learning Representations

work page 2022
[4]

Bradley, R. C. (1986). Basic properties of strong mixing conditions. Dependence in Probability and Statistics: A Survey of Recent Results\/ , 165--192

work page 1986
[5]

Chakrabortty, A. and T. Cai (2018). Efficient and adaptive linear regression in semi-supervised settings . Annals of Statistics\/ 46\/ (4), 1541 -- 1572

work page 2018
[6]

Chen, E. Y., R. Song, and M. I. Jordan (2024). Reinforcement learning in latent heterogeneous environments. Journal of the American Statistical Association\/ 119\/ (548), 3113--3126

work page 2024
[7]

Chen, J. and N. Jiang (2019). Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning , pp.\ 1042--1051

work page 2019
[8]

Qi, and R

Chen, X., Z. Qi, and R. Wan (2023). Steel: Singularity-aware reinforcement learning. arXiv preprint arXiv:2301.13152\/

arXiv 2023

Show all 61 references

[9]

Dedecker, J. and S. Louhichi (2002). Maximal inequalities and empirical central limit theorems. In Empirical process techniques for dependent data , pp.\ 137--159. Springer

2002
[10]

Geurts, and L

Ernst, D., P. Geurts, and L. Wehenkel (2005). Tree-based batch mode reinforcement learning. Journal of Machine Learning Research\/ 6

2005
[11]

Kumar, O

Fu, J., A. Kumar, O. Nachum, G. Tucker, and S. Levine (2020). D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219\/

2020 arXiv
[12]

Haarnoja, T., A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al. (2018). Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905\/

2018 arXiv
[13]

Hao, B., Y. Duan, T. Lattimore, C. Szepesv \'a ri, and M. Wang (2021). Sparse feature selection makes batch reinforcement learning more sample efficient. In International Conference on Machine Learning , pp.\ 4063--4073. PMLR

2021
[14]

Hu, H., Y. Yang, Q. Zhao, and C. Zhang (2023). The provable benefit of unsupervised data sharing for offline reinforcement learning. In The Eleventh International Conference on Learning Representations

2023
[15]

without

Jin, Y., Z. Ren, Z. Yang, and Z. Wang (2022). Policy learning" without" overlap: Pessimism and generalized empirical bernstein's inequality. arXiv preprint arXiv:2212.09900\/

2022 arXiv
[16]

Yang, and Z

Jin, Y., Z. Yang, and Z. Wang (2021). Is pessimism provably efficient for offline RL ? In International Conference on Machine Learning , pp.\ 5084--5096. PMLR

2021
[17]

Kakade, S. and J. Langford (2002). Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning , pp.\ 267--274

2002
[18]

Kauermann, G. and R. J. Carroll (2001). A note on the efficiency of sandwich covariance matrix estimation. Journal of the American Statistical Association\/ 96\/ (456), 1387--1396

2001
[19]

Rajeswaran, P

Kidambi, R., A. Rajeswaran, P. Netrapalli, and T. Joachims (2020). Morel: Model-based offline reinforcement learning. Advances in neural information processing systems\/ 33 , 21810--21823

2020
[20]

Kingma, D. P. and J. Ba (2014). Adam: A method for stochastic optimization. In International Conference on Learning Representations

2014
[21]

Zolna, Y

Konyushkova, K., K. Zolna, Y. Aytar, A. Novikov, S. Reed, S. Cabi, and N. de Freitas (2020). Semi-supervised reward learning for offline reinforcement learning. Neural Information Processing Systems, Offline Reinforcement Learning Workshop\/

2020
[22]

Kou, Y., Z. Chen, Y. Cao, and Q. Gu (2023). How does semi-supervised learning with pseudo-labelers work? a case study. In The Eleventh International Conference on Learning Representations

2023
[23]

Lee, D.-H. et al. (2013). Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML , Volume 3, pp.\ 896. Atlanta

2013
[24]

Kumar, G

Levine, S., A. Kumar, G. Tucker, and J. Fu (2020). Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR\/ abs/2005.01643

2020 arXiv
[25]

Li, G., L. Shi, Y. Chen, Y. Chi, and Y. Wei (2024). Settling the sample complexity of model-based offline reinforcement learning. The Annals of Statistics\/ 52\/ (1), 233--260

2024
[26]

Zhou, and R

Li, Y., W. Zhou, and R. Zhu (2023). Quasi-optimal reinforcement learning with continuous actions. In The Eleventh International Conference on Learning Representations

2023
[27]

Liao, P., Z. Qi, R. Wan, P. Klasnja, and S. A. Murphy (2022). Batch policy learning in average reward markov decision processes. Annals of statistics\/ 50\/ (6), 3364--3387

2022
[28]

Pogosyan, S

Little, S., A. Pogosyan, S. Neal, B. Zavala, L. Zrinzo, M. Hariz, T. Foltynie, P. Limousin, K. Ashkan, J. FitzGerald, A. L. Green, T. Z. Aziz, and P. Brown (2013, September). Adaptive deep brain stimulation in advanced parkinson disease. Annals of Neurology\/ 74\/ (3), 449--457

2013
[29]

Liu, W., J. Tu, Y. Zhang, and X. Chen (2023). Online estimation and inference for robust policy evaluation in reinforcement learning. arXiv preprint arXiv:2310.02581\/

2023 arXiv
[30]

Swaminathan, A

Liu, Y., A. Swaminathan, A. Agarwal, and E. Brunskill (2020). Provably good batch off-policy reinforcement learning without great exploration. Advances in neural information processing systems\/ 33 , 1264--1274

2020
[31]

Luckett, D. J., E. B. Laber, A. R. Kahkoska, D. M. Maahs, E. Mayer-Davis, and M. R. Kosorok (2020). Estimating dynamic treatment regimes in mobile health using v-learning. Journal of the american statistical association\/

2020
[32]

Wen, and X

Mou, W., Z. Wen, and X. Chen (2020). On the sample complexity of reinforcement learning with policy space generalization. arXiv preprint arXiv:2008.07353\/

2020 arXiv
[33]

Gilron, S

Neumann, W.-J., R. Gilron, S. Little, and G. Tinkhauser (2023). Adaptive deep brain stimulation: From experimental evidence toward practical implementation. Movement Disorders\/ 38\/ (6), 937--948

2023
[34]

Oehrn, C. R., C. Palmisano, P. A. Starr, S. Little, et al. (2024). Chronic adaptive deep brain stimulation versus conventional stimulation in parkinson’s disease: a blinded randomized feasibility trial. Nature Medicine\/ 30\/ (8), 1234--1240

2024
[35]

Okun, M. S. (2012). Deep-brain stimulation for parkinson's disease. New England Journal of Medicine\/ 367\/ (16), 1529--1538

2012
[36]

Rahimi, A. and B. Recht (2007). Random features for large-scale kernel machines. Advances in neural information processing systems\/ 20

2007
[37]

Rahimi, A. and B. Recht (2008). Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. Advances in neural information processing systems\/ 21

2008
[38]

Ramprasad, P., Y. Li, Z. Yang, Z. Wang, W. W. Sun, and G. Cheng (2023). Online bootstrap inference for policy evaluation in reinforcement learning. Journal of the American Statistical Association\/ 118\/ (544), 2901--2914

2023
[39]

Riedmiller, M. (2005). Neural fitted Q iteration--first experiences with a data efficient neural reinforcement learning method. In Machine Learning: ECML 2005: 16th European Conference on Machine Learning, Porto, Portugal, October 3-7, 2005. Proceedings 16 , pp.\ 317--328

2005
[40]

Shi, C., Z. Qi, J. Wang, and F. Zhou (2024). Value enhancement of reinforcement learning via efficient and robust trust region optimization. Journal of the American Statistical Association\/ 119\/ (547), 2011--2025

2024
[41]

Zhang, W

Shi, C., S. Zhang, W. Lu, and R. Song (2022). Statistical inference of the value function for reinforcement learning in infinite-horizon settings. Journal of the Royal Statistical Society: Series B (Statistical Methodology)\/ 84\/ (3), 765--793

2022
[42]

Sonabend-W, A., N. Laha, A. N. Ananthakrishnan, T. Cai, and R. Mukherjee (2023). Semi-supervised off-policy reinforcement learning and value estimation for dynamic treatment regimes. Journal of Machine Learning Research\/ 24\/ (323), 1--86

2023
[43]

Sonabend-W, A., N. Laha, R. Mukherjee, and T. Cai (2020). Semi-supervised learning for doubly robust offline policy evaluation. In Offline Reinforcement Learning Workshop at Neural Information Processing Systems

2020
[44]

Sutton, R. S. and A. G. Barto (2018). Reinforcement learning: An introduction . MIT press

2018
[45]

Imaizumi, N

Uehara, M., M. Imaizumi, N. Jiang, N. Kallus, W. Sun, and T. Xie (2021). Finite sample analysis of minimax offline reinforcement learning: Completeness, fast rates and first-order efficiency. arXiv preprint arXiv:2102.02981\/

2021 arXiv
[46]

Uehara, M. and W. Sun (2022). Pessimistic model-based offline reinforcement learning under partial coverage. In International Conference on Learning Representations

2022
[47]

Wang, D., C. Shi, S. Luo, and W. W. Sun (2024). Pessimistic causal reinforcement learning with mediators for confounded offline data. arXiv preprint arXiv:2403.11841\/

2024 arXiv
[48]

Qi, and R

Wang, J., Z. Qi, and R. K. Wong (2023). Projected state-action balancing weights for offline reinforcement learning. The Annals of Statistics\/ 51\/ (4), 1639--1665

2023
[49]

Watkins, C. J. and P. Dayan (1992). Q -learning. Machine learning\/ 8 , 279--292

1992
[50]

Wu, D., Y. Jiao, L. Shen, H. Yang, and X. Lu (2024). Neural network approximation for pessimistic offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , Volume 38, pp.\ 15868--15877

2024
[51]

Levine, and O

Yang, M., S. Levine, and O. Nachum (2022). TRAIL : Near-optimal imitation learning with suboptimal data. In International Conference on Learning Representations

2022
[52]

Yang, M. and O. Nachum (2021). Representation matters: Offline pretraining for sequential decision making. In International Conference on Machine Learning , pp.\ 11784--11794. PMLR

2021
[53]

Yang, X., Z. Song, I. King, and Z. Xu (2023). A survey on deep semi-supervised learning. IEEE Transactions on Knowledge and Data Engineering\/ 35\/ (9), 8934--8954

2023
[54]

Yin, M., Y. Duan, M. Wang, and Y.-X. Wang (2022). Near-optimal offline reinforcement learning with linear representation: Leveraging variance information with pessimism. In International Conference on Learning Representation

2022
[55]

Kumar, Y

Yu, T., A. Kumar, Y. Chebotar, K. Hausman, C. Finn, and S. Levine (2022). How to leverage unlabeled data in offline reinforcement learning. In International Conference on Machine Learning , pp.\ 25611--25635. PMLR

2022
[56]

Thomas, L

Yu, T., G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma (2020). MOPO : Model-based offline policy optimization. Advances in Neural Information Processing Systems\/ 33 , 14129--14142

2020
[57]

Zhang, A., L. D. Brown, and T. T. Cai (2019). Semi-supervised inference: General theory and estimation of means. The Annals of Statistics\/ 47\/ (5), 2538--2566

2019
[58]

Zhang, L., Y. Peng, J. Liang, W. Yang, and Z. Zhang (2023). Estimation and inference in distributional reinforcement learning. arXiv preprint arXiv:2309.17262\/

2023
[59]

Zhou, A. (2024). Reward-relevance-filtered linear offline reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pp.\ 3025--3033. PMLR

2024
[60]

Zhu, and A

Zhou, W., R. Zhu, and A. Qu (2024). Estimating optimal infinite horizon dynamic treatment regimes via pt-learning. Journal of the American Statistical Association\/ 119\/ (545), 625--638

2024
[61]

Zhou, Y., Z. Qi, C. Shi, and L. Li (2023). Optimizing pessimism in dynamic treatment regimes: A bayesian learning approach. In International Conference on Artificial Intelligence and Statistics , pp.\ 6704--6721. PMLR

2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION article output.bibitem format.authors "author" output.check author format.key output output.year.check new.block format.title "title" output.check new.block crossref missing format.jour.vol output format.article.crossref output.nonnull format.pages output if new.block note output fin.entry FUNCTION b...

[2] [2]

Angelopoulos, A. N., S. Bates, C. Fannjiang, M. I. Jordan, and T. Zrnic (2023). Prediction-powered inference. Science\/ 382\/ (6671), 669--674

2023

[3] [3]

Bai, C., L. Wang, Z. Yang, Z.-H. Deng, A. Garg, P. Liu, and Z. Wang (2022). Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In International Conference on Learning Representations

work page 2022

[4] [4]

Bradley, R. C. (1986). Basic properties of strong mixing conditions. Dependence in Probability and Statistics: A Survey of Recent Results\/ , 165--192

work page 1986

[5] [5]

Chakrabortty, A. and T. Cai (2018). Efficient and adaptive linear regression in semi-supervised settings . Annals of Statistics\/ 46\/ (4), 1541 -- 1572

work page 2018

[6] [6]

Chen, E. Y., R. Song, and M. I. Jordan (2024). Reinforcement learning in latent heterogeneous environments. Journal of the American Statistical Association\/ 119\/ (548), 3113--3126

work page 2024

[7] [7]

Chen, J. and N. Jiang (2019). Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning , pp.\ 1042--1051

work page 2019

[8] [8]

Qi, and R

Chen, X., Z. Qi, and R. Wan (2023). Steel: Singularity-aware reinforcement learning. arXiv preprint arXiv:2301.13152\/

arXiv 2023