Pith. sign in

REVIEW 4 major objections 4 minor 61 references

A Provable Approach for End-to-End Safe Reinforcement Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A provably safe way to deploy return-conditioned RL policies: keep the policy fixed and tune only its target returns via Gaussian processes.

desk verdict Real asymptotic GP result, but the safety guarantee as stated does not follow from the paper's own assumptions; needs major revision. read the letter →

arxiv 2505.21852 v1 pith:YBQEDX4N submitted 2025-05-28 cs.LG cs.AIcs.ITcs.ROmath.IT

Akifumi Wachi , Kohei Miyaguchi , Takumi Tanabe , Rei Sato , Youhei Akimoto This is my paper
classification cs.LGcs.AIcs.ITcs.ROmath.IT
keywords safereinforcementlearningreturn-conditionedsupervisedconstraineddecisiontransformerGaussianprocessoptimizationexplorationofflinehigh-probabilitysafetyguaranteeBayesian
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PLS attacks the longstanding 'lifetime safety' problem in RL by splitting the process: learn a return-conditioned policy entirely offline, then fix it and tune only the two target numbers that condition it. The paper proves that the gap between target returns and what the policy actually achieves is asymptotically a Gaussian process, so tuning those numbers with GP-based safe Bayesian optimization is mathematically justified. The theorems promise that every deployed setting satisfies the safety constraint with probability at least $1 - \Delta$, and that the reward return closes to within $\varepsilon$ of the best reachable safe setting after a bounded number of online observations. A sympathetic reader would care because this is a route to deploy complex transformer policies in safety-critical settings while verifying only a two-dimensional knob, not the whole policy.

What carries the argument

The load-bearing object is the target-return vector $z = (R, G)$, the tiny conditioning interface of a return-conditioned policy; the load-bearing identity is Theorem 1's asymptotic decomposition of the target-actual return gap into an unbiased Gaussian-process term plus a bias. That identity converts policy analysis into a two-dimensional Bayesian optimization problem. Safety is carried by confidence intervals built with $\alpha_{\diamond,j} = \sqrt{2\log(|Z| j^2\pi^2/(6\Delta))}$: Lemma 7 guarantees that the true return values stay inside these intervals for every $z \in \mathcal{Z}$ and every $j$ with probability at least $1 - \Delta$, and the safe set is expanded pessimistically using the upper confidence bound $u_N(z)$ together with the Lipschitz radius $L \cdot d(z, z')$. This same mechanism bounds the exploration rounds and yields the finite-time near-optimality statement.

What would settle it

Over many independent runs of PLS on a benchmark task, count the fraction of iterations in which the realized safety cost $J_g(\pi_z)$ exceeds the GP upper confidence bound; if that fraction is consistently larger than the declared failure probability $\Delta$, the confidence-interval lemma behind the high-probability safety guarantee is not holding.

Watch

Extended reading notes

Core claim

The central claim is that the map from a target-return pair $z = (R, G)$ to the actual returns $J(\pi_z)$ of a pre-trained constrained decision transformer is, asymptotically in the offline dataset size $n$, a Gaussian process up to a small bias: $J(\pi_z) - z = \frac{H^2}{\sqrt{n}} F(z) + \varepsilon(z) + o_P(1/\sqrt{n})$, where $F$ is a GP sample path (Theorem 1). PLS then treats the two-dimensional target return as the only decision variable, models $J_r$ and $J_g$ with independent GPs, expands the set of provably safe $z$ through confidence-interval boundaries plus a Lipschitz radius, and afterwards picks the most promising safe $z$. The safety and near-optimality guarantees (Theorems 2 and 3) follow from these confidence intervals; the paper is explicit that the guarantee concerns near-optimal target returns, not directly a near-optimal policy. Empirically, on the Bullet-Safety-Gym and Safety-Gymnasium suites, PLS is reported to be the only method tested that never violates the safety constraint across all tasks.

Load-bearing premise

The safety guarantee requires that the true mapping from target returns to actual safety cost lies inside the Gaussian-process confidence intervals at every candidate setting; the paper derives this only as an asymptotic approximation and does not prove it for its fixed kernel choice with finite data.

Editorial extensions

If this is right

  • A single offline-trained return-conditioned policy can be used under different safety thresholds $b$ by simply selecting different target returns, with the same high-probability safety guarantee.
  • The online phase requires only a two-dimensional optimization, so sample complexity is on the order of GP iterations rather than full policy-gradient updates; every interaction is a roll-out of the fixed policy.
  • If the guarantees hold, the GP-based safe-optimization step can be attached to any return-conditioned architecture, not only CDT, as long as the target-to-actual return mapping is smooth enough.
  • The paper's theorems certify near-optimal target returns, not near-optimal policies; reaching a near-optimal policy under lifetime-safety constraints remains an open problem.
  • Since the safety guarantee is high-probability ($1 - \Delta$), rare violations are not excluded; a small number of unsafe deployments can still occur during exploration, as the paper's own experiments show.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because Theorem 1's Gaussian process has an unknown, kernel-dependent covariance, a practitioner would need to verify GP calibration on the actual task before trusting the $1 - \Delta$ safety statement; the fixed RBF kernels are a modelling choice, not a proven match to $F$.
  • The two-dimensional tuning interface suggests a natural extension: multiple safety constraints could be handled by enlarging $z$ to $(R, G_1, \dots, G_m)$, though the GP sample complexity in higher dimensions would grow.
  • The template of 'learn offline, verify online on a low-dimensional conditioning knob' could be applied to other generative or sequence models whose behaviour is conditioned on a scalar goal, giving a way to retrofit safety certificates to existing pretrained policies.
  • A testable prediction: the empirical violation rate during safe exploration should be close to $\Delta$ whenever the GP is well specified, and should exceed $\Delta$ on tasks where the target-return mapping has sharp discontinuities.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes PLS, a two-stage safe reinforcement learning algorithm: (1) offline training of a return-conditioned policy (a constrained decision transformer), and (2) online deployment in which the two-dimensional target return z=(R,G) is optimized with Gaussian processes, using a safe-exploration phase followed by reward maximization. The main theoretical claims are Theorem 1, an asymptotic characterization of the mismatch between target and actual returns as a Gaussian process, and Theorems 2-3, which respectively assert a high-probability safety guarantee and near-optimality of the selected target returns. Experiments on Bullet-Safety-Gym and Safety-Gymnasium show that PLS is the only method that satisfies the safety threshold in all tasks, while also achieving competitive reward returns.

Significance. If established, the paper would make a useful contribution: it addresses a real problem (safety during both learning and deployment), the idea of reducing online adaptation to a two-dimensional target-return optimization is elegant and computationally appealing, and Theorem 1's bias-variance decomposition (Eq. (16)) is a nontrivial analysis of return-conditioned policies. The empirical comparison is broad, covering eight tasks and six baselines. However, the central safety guarantee is not established: the proof rests on a GP calibration lemma whose conditions are neither stated nor satisfied, and the paper's own assumptions and experiments cast doubt on the theorem. The near-optimality result also has a gap between average regret and the point actually deployed.

major comments (4)
  1. [Section 6.3, Lemma 7, Appendix I.2] Theorem 2 is proved via Lemma 7, which is cited from Lemma 5.1 of Srinivas et al. That result is valid only when the true function has bounded norm in the RKHS of the kernel used for the GP and the observation noise is Gaussian with the assumed variance. The paper never states or verifies these conditions for the RBF kernels used in Section 7 and Appendix J. Moreover, the paper's own Theorem 1 / Theorem 4 (Eqs. (4) and (15)) implies that J_g(pi_z) is approximately G + H^2 / sqrt(n) F_g(z), and the term G is linear in the target safety-cost coordinate; such a linear function is not in the RKHS of a Gaussian RBF kernel, since its RKHS norm is infinite. Thus the premise of Lemma 7 is not met by the paper's own model. The text in Appendix I.2 explicitly states 'we cannot assume that the functions are endowed with reproducing kernel Hilbert space (RKHS)', which is inconsistent with the invocation of Lemma 7. Consequently, Theorem 2 does not follow as stated, even setting aside the fact that Eq. (5) contains no RKHS-norm term in the definition of alpha.
  2. [Section 7, Figure 3] The experimental results are in tension with Theorem 2. In several tasks (HopperVelocity, AntRun, AntVelocity, HalfCheetahVelocity), Figure 3 shows normalized safety cost above 1 during GP iterations. Since the chosen z_j are inside Y_j, the algorithm's upper confidence bound satisfies u_j(z_j) <= b; an observed J_g(z_j) > b then implies u_j(z_j) < J_g(z_j), a direct violation of the confidence inequality in Lemma 7. The text in Section 7 acknowledges that 'a small number of unsafe deployments may occur', which is precisely the event that Theorem 2 excludes. The authors should either reconcile these observations with Theorem 2 (for example, by showing they are Monte-Carlo evaluation noise rather than estimates of the true J_g) or revise the theorem to state a weaker guarantee.
  3. [Theorem 3, Appendix I.2] The proof of Theorem 3 establishes an average-regret bound over the N^# reward-maximization samples (Eq. (69)), from which the authors conclude that 'there exists z_hat in the samples' with J_r(pi_{z_hat}) >= J_r(pi_{z*}) - E. However, Algorithm 1 deploys the final sample z_{N^dagger + N^#} (line 25), and the statement of Theorem 3 says that PLS finds a near-optimal z after collecting N^# observations. Nothing in the proof shows that the final deployed point is the near-optimal one. The algorithm should keep and return the best observed point, and the theorem should be stated for that point, or the proof should be strengthened to cover the deployed point.
  4. [Section 6.1 and Theorem 1] The theoretical justification does not match the model used by the algorithm. Theorem 1 says the return mismatch J(pi_z) - z is asymptotically a Gaussian process with the kernel k in Eq. (26), whose hyperparameters depend on the unknown model and the MLE; Section 6 instead fits two independent GPs with RBF kernels and fixed lengthscales and output scales (Appendix J). The paper neither estimates k nor shows that the RBF posterior is calibrated for J_g. If the authors intend the safety guarantee in Theorem 2 to be conditional on a correctly specified GP, that assumption must be stated explicitly and separated from the asymptotic statement in Theorem 1; as written, the paper claims Theorem 1 justifies the GP model, but the model used is not the one appearing in Theorem 1.
minor comments (4)
  1. [Section 6.1, text after Eq. (5)] The phrase 'characterized by the the mean' contains a duplicated article, and 'the identify matrix' should be 'the identity matrix'.
  2. [Section 7, Main results] The sentence 'demonstrates its its superior overall performance' contains a duplicated 'its'.
  3. [Appendix J, Table 3] The hyperparameter name 'Droupout' should be spelled 'Dropout'.
  4. [Assumption 7 and Appendix J, Tables 4-5] Assumption 7 says the initial safe set Z0 is a singleton seed set, but the experimental tables list intervals such as Safety [0,5]; please clarify whether every point in the interval is assumed safe or whether only one seed point is used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central derivation is a genuine bias-variance analysis, and the GP safety guarantees rest on external confidence-bound lemmas; the unverified RKHS assumption is a correctness gap, not a circular reduction.

full rationale

Walking the derivation chain, Theorem 1 is obtained from an explicit bias-variance decomposition (Eq. 16) with the three terms bounded in Theorems 5-7 under stated assumptions (near-determinism, coverage, realizability, regularity). No quantity is fitted and then renamed as a prediction. The near-optimality claim in Theorem 3 is explicitly relative to z* defined in Eq. (63) as the best reachable safe target return, and the paper's own Limitations section admits this does not imply a near-optimal policy; this is a weakening of scope, not a circularity. The safety guarantee (Theorem 2) and near-optimality bound (Theorem 3) are derived from Lemma 7 and Lemma 8, which are cited to Srinivas et al. [45]; these are external mathematical results, not self-citations or inputs equivalent to the conclusions. The appendix does contain a serious unstated-assumption problem: Appendix I.2 says 'for the compatibility with Theorem 1, we cannot assume that the functions are endowed with reproducing kernel Hilbert space (RKHS), which leads to a different bound in terms of optimality,' yet the proof then invokes Lemma 8 whose proof 'directly follows from Lemma 5.4 in Srinivas et al. [45]', a lemma whose standard hypotheses include a bounded RKHS norm. This, together with using RBF kernels rather than the kernel k from Eq. (26), means the high-probability guarantees are conditional on an unverified model-correctness assumption. That is a correctness and misspecification risk, not a circular reduction: the cited lemmas are not defined in terms of the paper's conclusions, and the paper does not fit a parameter and then claim to predict that same parameter. Therefore, no load-bearing circular step is exhibited.

Assumptions & free parameters 9 free parameters · 6 assumptions · 0 invented entities

The central theoretical result rests on a chain of domain assumptions (near-determinism, coverage, boundedness, realizability, initial safe set) and on an implicit, unverified assumption that the true return functions are well modeled by the specific RBF GPs used in the experiments. The GP hyperparameters, the initial safe sets, and several algorithm constants are hand-chosen per task, and some are not reported.

free parameters (9)
  • GP lengthscale for reward = 50 (Bullet-Safety-Gym) or 100 (Safety-Gymnasium Velocity)
    Chosen per task family, not derived; the GP safety and reward bounds depend on these hyperparameters.
  • GP lengthscale for safety cost = 5.0 for all tasks
    Chosen by hand; affects the posterior covariance and hence the confidence intervals used in Theorem 2.
  • GP output variance for reward = 1.0 (Bullet-Safety-Gym) or 100 (Safety-Gymnasium Velocity)
    Chosen per task family; input to the GP prior and posterior.
  • GP output variance for safety cost = 1.0 for all tasks
    Chosen by hand.
  • GP observation noise variance nu^2 = not reported
    The model assumes i.i.d. Gaussian noise with variance nu^2, but the value used in experiments is not stated.
  • Allowed failure probability Delta = not reported
    Used to set alpha in Eq (5); no value given in experiments.
  • Tolerance zeta for safe exploration termination = not reported
    Termination condition for safe exploration is not specified.
  • Initial safe set Z0 per task = e.g., Ant-Circle reward [250,300], safety [0,5]
    Assumption 7 requires a known safe seed; Z0 is hand-specified per task, influencing safety and reachable set.
  • Lipschitz constant L = 0
    Set to 0 in experiments, effectively skipping Lipschitz-based expansion of the safe set.
assumptions (6)
  • domain assumption Assumption 1: Near-deterministic transitions.
    Reward and safety-cost densities and state transitions are nearly deterministic with small eps_r, eps_s, delta; needed for Theorem 5.
  • domain assumption Assumptions 2-4: initial coverage, boundedness, continuity of the return density f(z|x).
    Needed for the bias bound Theorem 5.
  • standard math Assumptions 5-6 (formal 14-15): soft realizability and regularity of the model family (Theta, Phi).
    Standard asymptotic MLE conditions for Theorems 6 and 7.
  • domain assumption Assumption 10 (in main paper): behavior policy beta is conditionally independent of past rewards and safety costs.
    Used in Lemma 5 to relate conditional return densities.
  • domain assumption Assumption 7: a known initial safe set Z0 exists.
    Required for Theorem 2; the experiments hand-provide safe seed ranges.
  • ad hoc to paper Implicit GP correctness: Jg(pi_z) and Jr(pi_z) are samples from the chosen RBF GPs with the specified hyperparameters and noise variance.
    Not stated as an assumption but required for Lemma 7 (GP confidence intervals) and hence for Theorems 2-3; Theorem 1 only gives an asymptotic GP with a different kernel.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Provable Approach for End-to-End Safe Reinforcement Learning." pith.science (2026). https://pith.science/paper/YBQEDX4N

@misc{pith2026250521852,
  author       = {Pith},
  title        = {Pith review of: A Provable Approach for End-to-End Safe Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YBQEDX4N}},
  note         = {Machine review of arXiv:2505.21852}
}
read the original abstract

A longstanding goal in safe reinforcement learning (RL) is a method to ensure the safety of a policy throughout the entire process, from learning to operation. However, existing safe RL paradigms inherently struggle to achieve this objective. We propose a method, called Provably Lifetime Safe RL (PLS), that integrates offline safe RL with safe policy deployment to address this challenge. Our proposed method learns a policy offline using return-conditioned supervised learning and then deploys the resulting policy while cautiously optimizing a limited set of parameters, known as target returns, using Gaussian processes (GPs). Theoretically, we justify the use of GPs by analyzing the mathematical relationship between target and actual returns. We then prove that PLS finds near-optimal target returns while guaranteeing safety with high probability. Empirically, we demonstrate that PLS outperforms baselines both in safety and reward performance, thereby achieving the longstanding goal to obtain high rewards while ensuring the safety of a policy throughout the lifetime from learning to operation.

Figures

Figures reproduced from arXiv: 2505.21852 by the authors.

Figure 1
Figure 1. A conceptual illustration of PLS. After learning a return-conditioned policy using offline safe RL, PLS optimizes target returns through safe online policy evaluation via Gaussian processes. A key advantage of PLS is that safety is guaranteed at least with high probability in the entire process. safe policy in a real environment due to distribution mismatch issues between the offline data and the actual environment,… view at source ↗
Figure 2
Figure 2. Relations between target safety cost return [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Experimental results on how our PLS ensures the satisfaction of the safety constraint while obtaining new GP observations. Black dotted lines represent the normalized safety threshold. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 50 canonical work pages

  1. [1]

    Achiam, D

    J. Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained policy optimization. In International Conference on Machine Learning (ICML), pages 22–31, 2017

    work page 2017

  2. [2]

    Alshiekh, R

    M. Alshiekh, R. Bloem, R. Ehlers, B. Könighofer, S. Niekum, and U. Topcu. Safe reinforcement learning via shielding. In AAAI Conference on Artificial Intelligence (AAAI), 2018

    work page 2018

  3. [3]

    E. Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999

    work page 1999

  4. [4]

    Amodei, C

    D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016

    arXiv 2016

  5. [5]

    Berkenkamp, M

    F. Berkenkamp, M. Turchetta, A. Schoellig, and A. Krause. Safe model-based reinforcement learning with stability guarantees. In Advances in Neural Information Processing Systems (NeurIPS), 2017

    work page 2017

  6. [6]

    Bhatnagar and K

    S. Bhatnagar and K. Lakshmanan. An online actor–critic algorithm with function approximation for constrained Markov decision processes. Journal of Optimization Theory and Applications, 153(3):688–708, 2012

    work page 2012

  7. [7]

    Black, M

    K. Black, M. Janner, Y . Du, I. Kostrikov, and S. Levine. Training diffusion models with reinforcement learning. In International Conference on Learning Representations (ICLR), 2024

    work page 2024

  8. [8]

    V . S. Borkar. An actor-critic algorithm for constrained markov decision processes.Systems & control letters, 54(3):207–213, 2005

    work page 2005

Show all 61 references
  1. [9]

    Brandfonbrener, A

    D. Brandfonbrener, A. Bietti, J. Buckman, R. Laroche, and J. Bruna. When does return- conditioned supervised learning work for offline reinforcement learning? Advances in Neural Information Processing Systems (NeurIPS), 35:1542–1553, 2022

    2022

  2. [10]

    L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in Neural Information Processing Systems (NeurIPS), 34:15084–15097, 2021

    2021

  3. [11]

    Cheng, G

    R. Cheng, G. Orosz, R. M. Murray, and J. W. Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the AAAI conference on artificial intelligence (AAAI), volume 33, pages 3387–3395, 2019

    2019

  4. [12]

    S. R. Chowdhury and A. Gopalan. On kernelized multi-armed bandits. In International Conference on Machine Learning (ICML), pages 844–853, 2017

    2017

  5. [13]

    Da Costa, M

    N. Da Costa, M. Pförtner, L. Da Costa, and P. Hennig. Sample path regularity of Gaussian processes from the covariance kernel. arXiv preprint arXiv:2312.14886, 2023

    2023

  6. [14]

    Emmons, B

    S. Emmons, B. Eysenbach, I. Kostrikov, and S. Levine. RvS: What is essential for offline RL via supervised learning? In International Conference on Learning Representations (ICLR), 2021

    2021

  7. [15]

    J. Fu, M. Norouzi, O. Nachum, G. Tucker, A. Novikov, M. Yang, M. R. Zhang, Y . Chen, A. Kumar, C. Paduraru, et al. Benchmarks for deep off-policy evaluation. In International Conference on Learning Representations (ICLR), 2021

    2021

  8. [16]

    Fujimoto, D

    S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In International Conference on Machine Learning (ICML), pages 2052–2062, 2019

    2019

  9. [17]

    Fulton and A

    N. Fulton and A. Platzer. Safe reinforcement learning via formal methods: Toward safe control through proof and learning. In AAAI Conference on Artificial Intelligence (AAAI), 2018. 10

    2018

  10. [18]

    Garcıa and F

    J. Garcıa and F. Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research (JMLR), 16(1):1437–1480, 2015

    2015

  11. [19]

    Gronauer

    S. Gronauer. Bullet-safety-gym: A framework for constrained reinforcement learning. Technical report, mediaTUM, 2022

    2022

  12. [20]

    S. Gu, L. Yang, Y . Du, G. Chen, F. Walter, J. Wang, and A. Knoll. A review of safe reinforcement learning: Methods, theory and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  13. [21]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  14. [22]

    Z. Guo, W. Zhou, S. Wang, and W. Li. Constraint-conditioned actor-critic for offline safe reinforcement learning. In International Conference on Learning Representations (ICLR), 2025

    2025

  15. [23]

    Hambly, R

    B. Hambly, R. Xu, and H. Yang. Recent advances in reinforcement learning in finance. Mathematical Finance, 33(3):437–503, 2023

    2023

  16. [24]

    K.-C. Hsu, A. Z. Ren, D. P. Nguyen, A. Majumdar, and J. F. Fisac. Sim-to-lab-to-real: Safe reinforcement learning with shielding and generalization guarantees. Artificial Intelligence, 314: 103811, 2023

    2023

  17. [25]

    N. Hunt, N. Fulton, S. Magliacane, T. N. Hoang, S. Das, and A. Solar-Lezama. Verifiably safe exploration for end-to-end reinforcement learning. In Proceedings of the 24th International Conference on Hybrid Systems: Computation and Control, pages 1–11, 2021

    2021

  18. [26]

    J. Ji, B. Zhang, J. Zhou, X. Pan, W. Huang, R. Sun, Y . Geng, Y . Zhong, J. Dai, and Y . Yang. Safety gymnasium: A unified safe reinforcement learning benchmark. In Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023

    2023

  19. [27]

    Krasowski, J

    H. Krasowski, J. Thumm, M. Müller, L. Schäfer, X. Wang, and M. Althoff. Provably safe reinforcement learning: Conceptual analysis, survey, and benchmarking. arXiv preprint arXiv:2205.06750, 2022

    2022 arXiv

  20. [28]

    Kumar, J

    A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019

    2019

  21. [29]

    Kumar, X

    A. Kumar, X. B. Peng, and S. Levine. Reward-conditioned policies. arXiv preprint arXiv:1912.13465, 2019

    1912 arXiv

  22. [30]

    H. Le, C. V oloshin, and Y . Yue. Batch policy learning under constraints. In International Conference on Machine Learning (ICML), pages 3703–3712, 2019

    2019

  23. [31]

    J. Lee, C. Paduraru, D. J. Mankowitz, N. Heess, D. Precup, K.-E. Kim, and A. Guez. COptiDICE: Offline constrained reinforcement learning via stationary distribution correction estimation. In International Conference on Learning Representations (ICLR), 2021

    2021

  24. [32]

    Levine, C

    S. Levine, C. Finn, T. Darrell, et al. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research (JMLR), 17(1):1334–1373, 2016

    2016

  25. [33]

    Levine, A

    S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020

    2005 arXiv

  26. [34]

    Y . Li, Y . Wen, D. Tao, and K. Guan. Transforming cooling optimization for green data center via deep reinforcement learning. IEEE transactions on cybernetics, 50(5):2002–2013, 2019

    2002

  27. [35]

    Q. Lin, B. Tang, Z. Wu, C. Yu, S. Mao, Q. Xie, X. Wang, and D. Wang. Safe offline rein- forcement learning with real-time budget constraints. In International Conference on Machine Learning (ICML), pages 21127–21152. PMLR, 2023

    2023

  28. [36]

    Z. Liu, Z. Guo, H. Lin, Y . Yao, J. Zhu, Z. Cen, H. Hu, W. Yu, T. Zhang, J. Tan, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint arXiv:2306.09303, 2023. 11

    2023 arXiv

  29. [37]

    Z. Liu, Z. Guo, Y . Yao, Z. Cen, W. Yu, T. Zhang, and D. Zhao. Constrained decision transformer for offline safe reinforcement learning. In International Conference on Machine Learning (ICML), 2023

    2023

  30. [38]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  31. [39]

    Paternain, M

    S. Paternain, M. Calvo-Fullana, L. F. Chamon, and A. Ribeiro. Safe policies for reinforcement learning via primal-dual methods. arXiv preprint arXiv:1911.09101, 2019

    1911 arXiv

  32. [40]

    R. F. Prudencio, M. R. Maximo, and E. L. Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023

    2023

  33. [41]

    A. Radford. Improving language understanding by generative pre-training. OpenAI, 2018

    2018

  34. [42]

    Satija, P

    H. Satija, P. S. Thomas, J. Pineau, and R. Laroche. Multi-objective SPIBB: Seldonian offline policy improvement with safety constraints in finite MDPs. In Advances in Neural Information Processing Systems, volume 34, 2021

    2021

  35. [43]

    Schmidhuber

    J. Schmidhuber. Reinforcement learning upside down: Don’t predict rewards–just map them to actions. arXiv preprint arXiv:1912.02875, 2019

    1912 arXiv

  36. [44]

    Sootla, A

    A. Sootla, A. Cowen-Rivers, J. Wang, and H. Bou Ammar. Enhancing safe exploration using safety state augmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  37. [45]

    Srinivas, A

    N. Srinivas, A. Krause, S. M. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In International Conference on Machine Learning (ICML), 2010

    2010

  38. [46]

    R. K. Srivastava, P. Shyam, F. Mutz, W. Ja´skowski, and J. Schmidhuber. Training agents using upside-down reinforcement learning. arXiv preprint arXiv:1912.02877, 2019

    1912 arXiv

  39. [47]

    Stooke, J

    A. Stooke, J. Achiam, and P. Abbeel. Responsive safety in reinforcement learning by PID Lagrangian methods. In International Conference on Machine Learning (ICML), 2020

    2020

  40. [48]

    Y . Sui, A. Gotovos, J. W. Burdick, and A. Krause. Safe exploration for optimization with Gaussian processes. In International Conference on Machine Learning (ICML), 2015

    2015

  41. [49]

    Y . Sui, V . Zhuang, J. W. Burdick, and Y . Yue. Stagewise safe Bayesian optimization with Gaussian processes. In International Conference on Machine Learning (ICML), 2018

    2018

  42. [50]

    Turchetta, F

    M. Turchetta, F. Berkenkamp, and A. Krause. Safe exploration in finite Markov decision processes with Gaussian processes. In Advances in Neural Information Processing Systems (NeurIPS), 2016

    2016

  43. [51]

    A. W. Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000

    2000

  44. [52]

    Wachi and Y

    A. Wachi and Y . Sui. Safe reinforcement learning in constrained Markov decision processes. In International Conference on Machine Learning (ICML), 2020

    2020

  45. [53]

    Wachi, W

    A. Wachi, W. Hashimoto, X. Shen, and K. Hashimoto. Safe exploration in reinforcement learning: A generalized formulation and algorithms. In Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  46. [54]

    Wachi, X

    A. Wachi, X. Shen, and Y . Sui. A survey of constraint formulations in safe reinforcement learning. In International Joint Conference on Artificial Intelligence (IJCAI), pages 8262–8271, 2024

    2024

  47. [55]

    R. Wu, Y . Zhang, Z. Yang, and Z. Wang. Offline constrained multi-objective reinforcement learning via pessimistic dual value iteration. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 12

    2021

  48. [56]

    H. Xu, X. Zhan, and X. Zhu. Constraints penalized Q-learning for safe offline reinforcement learning. In AAAI Conference on Artificial Intelligence (AAAI), 2022

    2022

  49. [57]

    Yang and M

    L. Yang and M. Wang. Reinforcement learning in feature space: Matrix bandit, kernels, and regret bound. In International Conference on Machine Learning (ICML), 2020

    2020

  50. [58]

    Y . Yao, Z. Liu, Z. Cen, J. Zhu, W. Yu, T. Zhang, and D. Zhao. Constraint-conditioned policy optimization for versatile safe reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 36:12555–12568, 2023

    2023

  51. [59]

    C. Yu, J. Liu, S. Nemati, and G. Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1–36, 2021. 13 Appendix A Broader Impacts We believe that our proposed approachPLS plays a significant role in enhancing the benefits associated with reinf...

    2021

  52. [60]

    the reward density pr(r′|s, a) := d dr′ PT {r ≤ r′|s, a}4 is well-defined and bounded by ϵr outside the δ-neighborhood of ˆr(s, a), i.e., supr:∥r−ˆr(s,a)∥∞>δ pr(r|s, a) ≤ ϵr, and

  53. [61]

    Let β : X → ∆(A) be a behavior policy and D := {Ξ(i)}n i=1 ∼ (Pβ)n be a collection of n i.i.d

    the successor state s′ coincides with ˆs′(s, a) with probability of at least 1 − ϵs, for all s ∈ Sand a ∈ A. Let β : X → ∆(A) be a behavior policy and D := {Ξ(i)}n i=1 ∼ (Pβ)n be a collection of n i.i.d. copies of episodes generated by β. Assumption 10 (Reward-independent beha...

    2000

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.