Skill-based Safe Reinforcement Learning with Risk Planning

Hanping Zhang; Yuhong Guo

REVIEW 5 major objections 6 minor 32 references

Skill-based Safe Reinforcement Learning with Risk Planning

T0 review · 5 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SSkP claims a demonstration-trained skill risk predictor, used as the objective of a cross-entropy planning loop, consistently outperforms prior safe RL methods on four robotic benchmarks with higher reward and fewer violations.

desk verdict Useful safe-RL-with-demos method, but the consistency claim is overstated and the risk predictor's online calibration is unvalidated; deserves a serious referee without a desk reject. read the letter →

arxiv 2505.01619 v1 pith:4B5OHG54 submitted 2025-05-02 cs.LG cs.AI

Hanping Zhang , Yuhong Guo This is my paper · ORCID

classification cs.LGcs.AI

keywords safereinforcementlearningskillpositive-unlabeledriskplanningcross-entropymethodfromdemonstrationsconstrainedMarkovdecisionprocessroboticcontrol

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Safe reinforcement learning usually pays for its education in safety violations: the agent must explore the real environment to discover what is dangerous. This paper claims there is a cheaper route: first learn a skill risk predictor from a small offline demonstration dataset, then use that predictor as a planning objective that screens candidate skills before they are ever executed online. The claimed result, from a method called SSkP, is that this two-stage scheme consistently beats prior state-of-the-art safe RL methods (Recovery RL, CPQ, SMBPO) on four robotic simulation environments, reaching higher average episode reward with fewer safety violations within a fixed budget of online timesteps. If the claim holds, offline demonstrations can be exploited for safety without pre-training the policy itself, and even rare violations inside demonstrations suffice to build a useful risk signal.

What carries the argument

The load-bearing object is the skill risk predictor $P_\zeta(c|s,z)$, a small MLP trained with a non-negative positive-unlabeled loss built on an unbiased estimator of the negative loss, with the positive-class prior $\lambda$ estimated from the data. Its training data are state-skill pairs produced by a skill model, whose encoder $q_\mu(z_t|a_t)$, decoder $p_\nu(a_t|z_t)$, and prior $q_\psi(z_t|s_t)$ are trained on the demonstrations and turn length-$H$ action sequences into latent skills; a pair is positive if a violation occurred within the next $H$ steps, and near trajectory ends the prior network supplies the skill for a state. The second mechanism is the risk planner, a cross-entropy-method loop that starts from a Gaussian fit to skills sampled from the current policy $\pi_\theta$, iteratively samples a batch, scores each candidate with the predictor, refits the Gaussian to the lowest-risk $k$ skills, and after $N_p$ iterations samples the skill to execute. The planner converts a static offline-trained classifier into an online safety filter, and online rollouts contribute fresh state-skill pairs that retrain the predictor each episode, keeping the planner's objective aligned with the encountered environment.

What would settle it

Collect the state-skill pairs visited by the online policy during SSkP training, group them by the predictor's output $P_\zeta(c|s,z)$, and compare against whether a violation actually occurred within the next $H$ steps when those skills were executed; if the predictor is miscalibrated on policy-visited states while remaining accurate on demonstration states, the planner's safety gains would be an artifact of the demonstrations rather than a property of the online process. A second direct test is to freeze the predictor instead of updating it each episode and measure whether the violation count rises.

Watch

Extended reading notes

Core claim

The paper's discovery is that a single trained object, a skill risk predictor $P_\zeta(c|s,z)$ estimating the probability that choosing latent skill $z$ in state $s$ produces a safety violation ($c>0$) within the next $H$ steps, can carry the safety burden of online RL. The predictor is learned with positive-unlabeled (PU) learning on offline demonstrations, where positive examples are the state-skill pairs that actually led to a violation within the next $H$ steps and all other pairs are unlabeled, a construction that avoids both the scarcity of violations in demonstration data and the need to label near-miss decisions as safe or unsafe. During online learning the predictor becomes the objective of a risk planning process: starting from a Gaussian fit over skills sampled from the current policy, the planner iteratively resamples candidate skills, keeps the top-$k$ with the lowest predicted risk, refits the Gaussian, and after $N_p$ iterations executes the winning skill. SSkP reports that this loop, together with periodic updates of the predictor from online rollouts, consistently outperforms Recovery RL, CPQ, and SMBPO across Ant, Cheetah, Hopper, and Humanoid, and its ablations show that both the risk predictor and the iterated planner are needed for the gain.

Load-bearing premise

The risk predictor is trained on offline demonstration state-skill pairs and then used online to score skills sampled from the evolving policy, so the method collapses if the predictor becomes miscalibrated on states the online exploration reaches, because the planner would then minimize a stale objective and stop preventing real violations.

Editorial extensions

If this is right

Removing the skill risk predictor and the planner (the SSkP-w/o-RP variant) causes a substantial decline in reward-versus-violation performance on Ant and Hopper, so the demonstration-derived risk signal, not just the skill prior, carries the safety gain.
Replacing the iterated cross-entropy planner with a single-batch naive planner (SSkP-NP) hurts performance, so the iterative Gaussian refinement adds value beyond one-shot risk screening.
Under the paper's cost-sensitive sample-efficiency metric (per-timestep reward divided by total violations), SSkP ranks first in three of four environments and second in Ant, where SMBPO leads.
The planning loop lowers the mean predicted risk of its sampled skills across all $N_p$ iterations in every environment, which the paper reads as evidence that the planner genuinely converges toward safer skills.
Because CPQ, which pre-trains its policy on the offline data, stalls at low reward in all environments, the paper concludes that demonstrations are better exploited for safety through a risk predictor than through policy pre-training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The PU construction suggests SSkP's advantage should grow as demonstrations get safer, since the unbiased negative-loss estimator matters most when confirmed violations are rare; a direct test is varying the demonstration violation rate and watching the margin over baselines.
The predictor is trained on demonstration states but deployed on states visited by the online policy, so its calibration under distribution shift is the main open risk; a natural follow-up is checking whether predicted risk tracks the empirical violation frequency on out-of-distribution states and whether the per-episode updates are what preserve validity.
Because the planner is a zeroth-order solver for $\arg\min_z P_\zeta(c|s,z)$, the planning loop is agnostic to how the risk signal is produced and could wrap any black-box safety score, which connects the method to constrained model-predictive control.
The fixed skill horizon $H$ and planner sizes ($N_s$, $k$, $N_p$) likely interact with how far ahead dangers can be foreseen in each environment, so ablating $H$ per environment is a concrete way to test where the method's benefit concentrates.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a legitimate engineering contribution combining SPiRL skill priors, PU learning, and CEM planning for safe RL with offline demos, but the headline claim of consistent SOTA is overstated and the risk predictor's reliability under online distribution shift is not established. It deserves a serious referee, not a desk reject.

What's new: the specific pipeline — a PU-trained skill risk predictor Pζ(c|s,z) used as the scoring function in a cross-entropy planner over skill latents, with simultaneous online adaptation — is not in the cited literature. The experiments span four MuJoCo environments and compare against Recovery RL, CPQ, and SMBPO, and the ablations (full SSkP vs. naive planning vs. no risk predictor) are well-designed. That is real work and worth engaging with.

The soft spots, in order of importance. First, the abstract's 'consistently outperforms' is contradicted by the paper's own Table 1, where SMBPO is best on Ant, and by Appendix Figure 6, where Cheetah sample efficiency is comparable to Recovery RL/SMBPO. This is a framing problem, not a fatal one. Second, Algorithm 1 initializes the planning distribution from the skill prior qψ, while Section 4.2.1 says it initializes from the current policy πθ. That is a concrete inconsistency that matters because the choice changes the distribution of skills being scored. Third — the load-bearing issue — the risk predictor is trained offline on demonstration state-skill pairs, then used to score skills proposed by the online policy, which may visit states far from the demonstrations. The paper provides no calibration check on the states/skills actually selected by the planner. Figure 5 only shows that CEM iterations reduce Pζ, which is the optimizer minimizing its own objective, not evidence that Pζ tracks true violation risk. Worse, the online adaptation loop in Eq. 7 adds randomly sampled skills z∼qψ from every intermediate state in a violated episode to the positive set, even though those skills were never executed. That injects false positives into the PU data and can bias the predictor further. This is a real flaw, not a nitpick.

The paper releases no code or data, so independent verification is limited. The citation pattern is appropriate; the PU and skill-learning references are the right ones.

Who this is for: people working on safe RL with offline demonstrations, or on learned risk models for planning. They will get a useful baseline and a clear problem formulation, but they should not take the safety improvement at face value.

Recommendation: send to peer review with the expectation of major revision. A good reviewer should probe the risk predictor's calibration on online data and ask the authors to fix or justify the Eq. 7 positive-set construction.

Referee Report

5 major / 6 minor

Summary. The paper proposes SSkP, a safe reinforcement learning method that learns a skill risk predictor P_ζ(c|s,z) from offline demonstration data using positive-unlabeled (PU) learning, and then uses this predictor as the scoring function in a cross-entropy-method (CEM) risk planning procedure during online interaction. The planned skill is decoded into an action sequence to interact with the environment, and the skill policy is trained with a skill-based SAC objective. The method is evaluated on Ant, Cheetah, Hopper, and Humanoid MuJoCo environments against CPQ, SMBPO, and Recovery RL, with additional ablations of the risk planning components.

Significance. If the risk predictor can be shown to remain calibrated under online distribution shift, the approach is a meaningful step toward leveraging offline demonstrations for safe exploration, and it could be useful in applications where safety violations are costly. The paper's strengths include the use of PU learning to avoid negative labels, the integration of a CEM planner with a learned risk model, and the inclusion of ablation studies. However, the empirical support for the central 'consistent' claim is incomplete, and several implementation details require clarification before the contribution can be fully assessed.

major comments (5)

[§4.2.1, Algorithm 1, Algorithm 2] The text and pseudocode disagree on the initialization of the risk planning distribution: Section 4.2.1 states that Ns skills are sampled from the current safe policy πθ(·|st) to compute the initial Gaussian, while Algorithm 1 initializes (µ0,σ0^2)←qψ(·|st), and Algorithm 2 line 4 calls Risk Planning with πθ as an argument that Algorithm 1 does not accept. This ambiguity determines whether the planner searches over the demonstration prior or the online policy, which is directly relevant to the distribution-shift concern for Pζ. Please reconcile the text, pseudocode, and the actual implementation.
[§4.2.2, Eq. (7), Algorithm 2 lines 9-12] When an episode ends in a violation, every intermediate state si receives a skill z∼qψ(·|si) sampled from the skill prior, and this state-skill pair is added to the positive set Dp. Since those skills were never executed, there is no evidence that they lead to violations; labeling them as positive unsafe examples injects false positives into the PU training data and can bias the risk predictor. The paper should either justify this adaptation rule with empirical evidence or modify it, for example by only labeling the actually executed skill zt as positive.
[§4.1.1, §4.2.1] The central safety improvement claims rest on Pζ(c|s,z) being an accurate risk measure for the skills considered by the planner, but the predictor is trained on demonstration state-skill pairs and applied to skills sampled from the prior or online policy that may be out-of-distribution. No calibration analysis is provided for the online setting. Figure 5 only demonstrates that the CEM iterations reduce Pζ; since planning explicitly minimizes Pζ, this is circular evidence and does not establish that lower Pζ corresponds to lower true violation risk. Please report a calibration check on held-out online rollouts, for instance comparing predicted risk with empirical violation frequency in bins.
[Abstract, §5.2, Table 1, Appendix Figure 6] The abstract's claim that the method 'consistently outperforms' previous state-of-the-art safe RL methods is contradicted by the paper's own results: Table 1 shows SSkP is second-best on Ant (PtR/#V 23.54 vs SMBPO 28.68), and Appendix Figure 6 shows comparable Cheetah sample efficiency to Recovery RL and SMBPO. Section 5.2 acknowledges the Ant exception, but the abstract and conclusion do not. Please soften the claims accordingly and discuss the conditions under which SSkP is expected to help.
[§5.2, Table 1] All results are averages over only three runs, and Table 1 reports no variance or significance testing for the PtR/#V ratio. The 'notable performance gains' asserted for Cheetah and Hopper would be more convincing with per-run values or confidence intervals on the ratio metric. Given the variability visible in Figure 3, the empirical superiority claim needs stronger statistical support.

minor comments (6)

[§5.4] The notation ∇p̄_i for the change in average predicted risk is confusing because ∇ usually denotes gradient; using Δp̄_i or δ_i would be clearer. The sentence 'with the increase of the risk planning iterations, −∇p̄_i becomes larger and hence p̄_i becomes smaller' is also awkwardly phrased.
[Eq. (4)] The PU loss in Eq. (4) would benefit from a more explicit connection to the nnPU formulation of Kiryo et al., since the slack variable ξ is set to 0 and the reader must infer how the relaxed constraint is derived.
[§2, Related Works] SAFER (Slack et al., 2022) is a closely related skill-based safe RL method, but it is only mentioned in passing and not included in the comparison; the authors should either compare against it or explain why it is excluded.
[§5.1, Implementation Details] The hyperparameters (H=10, skill dimension=10, Ns=512, k=64, Np=6, λ) are fixed without any sensitivity study; a brief sensitivity analysis or a paragraph justifying these choices would strengthen the paper.
[Figures 3 and 4] The figure legends are not consistent across plots, making it hard to compare methods; please use a uniform legend with the same colors and line styles in all figures.
[§3] The demonstration dataset Dd is introduced in the problem setting but never formally defined with an explicit set-builder notation; its notation is also close to the online dataset D, which could confuse readers. Please clarify the notation.

Circularity Check

1 steps flagged · score 2.0 of 10

One non-load-bearing validation step is self-definitional; the central empirical comparison against external baselines is independent.

self definitional [Section 5.4 ('Further Study of Risk Planning Process'), Figure 5, with Algorithm 1 and Eqs. (5)-(6)]
"We can see with the increase of the risk planning iterations, −∇¯pi becomes larger and hence ¯pi becomes smaller, indicating the sampled skills from each current Gaussian distribution are safer than previous iterations. Overall, the results validate that the risk planning process can effectively find safer skills z by minimizing Pζ(c = 1|st,z)."

Algorithm 1 explicitly updates the Gaussian using the top-k skills with the lowest Pζ values (Eqs. 5-6), so the average predicted risk p̄_i is forced to decrease by construction at every iteration. Section 5.4 then presents this enforced decrease as evidence that the planned skills are 'safer'. But 'safer' is operationalized as 'lower Pζ', which is precisely the objective being minimized. The result therefore only confirms that the cross-entropy optimizer descends on its own scoring function; it does not independently validate reduced true safety violations. This circular step is not load-bearing for the paper's main empirical claim, which is based on ground-truth reward and violation counts against external baselines.

full rationale

The paper's central claim is an empirical comparison of SSkP against Recovery RL, CPQ, and SMBPO over ground-truth episode rewards and cumulative safety violations in MuJoCo environments. Those results are not derived from the skill risk predictor by construction, so they are not circular. No load-bearing self-citation chain is present, and the PU-learning and CEM components are adopted from cited prior work rather than from the authors' own unverified claims. The only concrete circularity is in Section 5.4, where the risk-planning validation reduces to showing that minimizing Pζ lowers Pζ; this is true by definition of the algorithm and does not affect the external benchmark comparison. The offline-trained predictor's calibration under online distribution shift, and the potential false-positive injection in Eq. (7), are correctness and validity concerns rather than equivalence-by-construction circularity, so they do not raise the circularity score. Overall, one minor, non-load-bearing self-definitional validation step is present; the derivation is otherwise self-contained against independent empirical benchmarks.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central method rests on a small set of hand-set planning hyperparameters (H, skill dimension, Ns, k, Np, ξ) and on PU learning assumptions about the class prior and the meaning of positive examples. The most fragile element is the transfer assumption that a risk predictor trained on offline demonstrations remains calibrated for online exploration states. No new physical or conceptual entities are introduced; skills and the risk predictor are learned functions over existing representations.

free parameters (7)

skill horizon H = 10
Sets the length of action sequences compressed into a skill and defines the look-ahead window for labeling positive unsafe examples. Chosen by hand; no sensitivity analysis.
skill vector dimension = 10
Latent skill dimension; chosen without reported tuning or sensitivity.
risk planning sample count Ns = 512
Number of skills sampled per planning iteration; hand-set.
top-k safe skills = 64
Number of lowest-risk skills kept each planning iteration; hand-set.
planning iterations Np = 6
Number of CEM refinement iterations; hand-set.
PU slack variable ξ = 0
Relaxation in non-negative PU loss; set following Xu and Denil (2021).
positive class prior λ = estimated from data
Class prior estimated via methods of Jain et al. 2016 or Christoffel et al. 2016, but the estimate and procedure are not reported.

assumptions (5)

domain assumption The environment is a Constrained MDP with cost function C and strict termination on any positive cost (c_t > 0).
Adopted from Section 3 and Hans et al. 2008; all experiments assume termination on violation, which shapes the positive-label construction.
domain assumption The unlabeled decision pairs in the PU risk predictor are drawn from the same distribution as positives and negatives, and the estimated class prior λ is accurate.
Standard PU learning assumption invoked in Section 4.1.1 through Eq. (2)-(4). If violated, the risk predictor is biased.
domain assumption The pre-trained skill model (encoder, decoder, prior from SPiRL) transfers to online states and provides a prior q_ψ that is a useful behavior prior for the skill policy.
The online policy is initialized and regularized toward q_ψ in Section 4.2.2. If q_ψ is a poor prior, the KL regularizer hurts learning.
ad hoc to paper The skill risk predictor trained on demonstration data remains reliable for skills sampled from the online policy πθ during exploration.
Load-bearing for risk planning: Algorithm 1 minimizes Pζ(c|s,z). Distribution shift between offline demonstrations and online exploration is not quantified. This is the weakest assumption.
ad hoc to paper A state-skill pair is unsafe if and only if following the encoded skill from that state leads to a safety violation within the next H steps; near-terminal states can use the skill prior to synthesize skills.
This defines Dp and Du in Section 4.1.1. The near-terminal substitution with q_ψ(z|s) is not validated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Skill-based Safe Reinforcement Learning with Risk Planning." pith.science (2026). https://pith.science/paper/4B5OHG54

@misc{pith2026250501619,
  author       = {Pith},
  title        = {Pith review of: Skill-based Safe Reinforcement Learning with Risk Planning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4B5OHG54}},
  note         = {Machine review of arXiv:2505.01619}
}

read the original abstract

Safe Reinforcement Learning (Safe RL) aims to ensure safety when an RL agent conducts learning by interacting with real-world environments where improper actions can induce high costs or lead to severe consequences. In this paper, we propose a novel Safe Skill Planning (SSkP) approach to enhance effective safe RL by exploiting auxiliary offline demonstration data. SSkP involves a two-stage process. First, we employ PU learning to learn a skill risk predictor from the offline demonstration data. Then, based on the learned skill risk predictor, we develop a novel risk planning process to enhance online safe RL and learn a risk-averse safe policy efficiently through interactions with the online RL environment, while simultaneously adapting the skill risk predictor to the environment. We conduct experiments in several benchmark robotic simulation environments. The experimental results demonstrate that the proposed approach consistently outperforms previous state-of-the-art safe RL methods.

Figures

Figures reproduced from arXiv: 2505.01619 by the authors.

**Figure 1.** The framework of the proposed method, SSkP, which learns a skill risk predictor from the offline demonstration data and then deploys it to enhance online safe RL through risk planning. During the skill risk predictor learning stage, SSkP assembles PU data and trains a decision risk predictor Pζ (c|st, zt) based on a skill model, which produces skill prior qψ(·|st) and skill decoder pν(at|zt). In the online safe poli… view at source ↗

**Figure 2.** The four environments employed in the experiments are displayed from left to right: Ant, Cheetah, Hopper, Humanoid. The figures present instances of failure in each environment where safety constraints are violated. Algorithm 2 Online Safe Policy Learning Input: skill prior qψ(·|s), decoder pν(·|z), skill risk predictor Pζ (c|s, z), Dp and Du Initialize: data buffer D, skill policy network πθ(z|s) Procedure: 1: for … view at source ↗

**Figure 3.** This figure presents the performance of each comparison method in terms of the average episode reward vs. the total number of safety violations encountered during online training within a fixed total number of timesteps on all four environments: Ant, Cheetah, Hopper, and Humanoid. The results represent the averages over three runs, with the shadow indicating the standard deviations [PITH_FULL_IMAGE:figures/full_fig… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: The ablation study results in two environments: Ant and Hopper by comparing three methods: SSkP—the proposed approach; SSkP-NP—the variant that replaces risk planning with a naive planning process; and SSkP-w/o-RP—the variant that drops risk predictor and risk planning…

**Figure 5.** Figure 5: Risk prediction probability changes, ∇p¯i = ¯pi − p¯0, along the planning iteration number i from the initial average risk prediction probability p¯0. The results are the averages computed with the risk planning procedure on 100 randomly sampled states st. risk plannin…

**Figure 6.** Figure 6: The alternative evaluation of the safe RL results presents the episode rewards and the cumulative number of violations separately along the environment steps. Top: Sample efficiency curves illustrating episode rewards v.s. the total number of environmental steps across…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 28 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

Constrained policy optimization

Achiam, J., Held, D., Tamar, A., and Abbeel, P. Constrained policy optimization. In International conference on machine learning (ICML), 2017

work page 2017
[3]

Constrained Markov decision processes: stochastic modeling

Altman, E. Constrained Markov decision processes: stochastic modeling. Routledge, 1999

work page 1999
[4]

and Yarats, D

Amos, B. and Yarats, D. The differentiable cross-entropy method. In International Conference on Machine Learning (ICML), pp.\ 291--302. PMLR, 2020

work page 2020
[5]

D., Chernova, S., Veloso, M., and Browning, B

Argall, B. D., Chernova, S., Veloso, M., and Browning, B. A survey of robot learning from demonstration. Robotics and autonomous systems, 57 0 (5): 0 469--483, 2009

2009
[6]

I., Kroese, D

Botev, Z. I., Kroese, D. P., Rubinstein, R. Y., and L’Ecuyer, P. The cross-entropy method for optimization. In Handbook of statistics, volume 31, pp.\ 35--59. Elsevier, 2013

work page 2013
[7]

W., Yuan, Z., Zhou, S., Panerati, J., and Schoellig, A

Brunke, L., Greeff, M., Hall, A. W., Yuan, Z., Zhou, S., Panerati, J., and Schoellig, A. P. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 2021

work page 2021
[8]

B., Chernova, S., Taylor, M

Brys, T., Harutyunyan, A., Suay, H. B., Chernova, S., Taylor, M. E., and Now \'e , A. Reinforcement learning from demonstration through shaping. In International Joint Conference on Artificial Intelligence (IJCAI), 2015

work page 2015

Show all 32 references

[9]

Class-prior estimation for learning from positive and unlabeled data

Christoffel, M., Niu, G., and Sugiyama, M. Class-prior estimation for learning from positive and unlabeled data. In Asian Conference on Machine Learning (ACML), 2016

2016
[10]

Convex formulation for learning from positive and unlabeled data

Du Plessis, M., Niu, G., and Sugiyama, M. Convex formulation for learning from positive and unlabeled data. In International Conference on Machine Learning (ICML), 2015

2015
[11]

C., Niu, G., and Sugiyama, M

Du Plessis, M. C., Niu, G., and Sugiyama, M. Analysis of learning from positive and unlabeled data. Advances in Neural Information Processing Systems (NeurIPS), 2014

2014
[12]

and Fern \'a ndez, F

Garc a, J. and Fern \'a ndez, F. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research (JMLR), pp.\ 1437--1480, 2015

2015
[13]

Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning (ICML), 2018

2018
[14]

M., and Udluft, S

Hans, A., Schneega , D., Sch \"a fer, A. M., and Udluft, S. Safe exploration for reinforcement learning. In ESANN, 2008

2008
[15]

and Ermon, S

Ho, J. and Ermon, S. Generative adversarial imitation learning. Advances in Neural Information Processing Systems (NeurIPS), 2016

2016
[16]

Estimating the class prior and posterior from noisy positives and unlabeled data

Jain, S., White, M., and Radivojac, P. Estimating the class prior and posterior from noisy positives and unlabeled data. Advances in neural information processing systems (NeurIPS), 29, 2016

2016
[17]

C., and Sugiyama, M

Kiryo, R., Niu, G., Du Plessis, M. C., and Sugiyama, M. Positive-unlabeled learning with non-negative risk estimator. Advances in Neural Information Processing Systems (NeurIPS), 2017

2017
[18]

and Whiteson, S

Koppejan, R. and Whiteson, S. Neuroevolutionary reinforcement learning for generalized control of simulated helicopters. Evolutionary intelligence, 2011

2011
[19]

Datasets and benchmarks for offline safe reinforcement learning

Liu, Z., Guo, Z., Lin, H., Yao, Y., Zhu, J., Cen, Z., Hu, H., Yu, W., Zhang, T., Tan, J., et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint arXiv:2306.09303, 2023

2023 arXiv
[20]

Accelerating reinforcement learning with learned skill priors

Pertsch, K., Lee, Y., and Lim, J. Accelerating reinforcement learning with learned skill priors. In Conference on Robot Learning (CoRL), 2021

2021
[21]

Rubinstein, R. Y. Optimization of computer simulation models with rare events. European Journal of Operational Research (EJOR), 99 0 (1): 0 89--112, 1997

1997
[22]

Z., Chow, Y., Dai, B., and Wichers, N

Slack, D. Z., Chow, Y., Dai, B., and Wichers, N. SAFER : Data-efficient and safe reinforcement learning via skill acquisition. In Decision Awareness in Reinforcement Learning Workshop at ICML 2022, 2022

2022
[23]

Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018

2018
[24]

J., and Mannor, S

Tessler, C., Mankowitz, D. J., and Mannor, S. Reward constrained policy optimization. In International Conference on Learning Representations (ICLR), 2019

2019
[25]

E., Levine, S., Borrelli, F., and Goldberg, K

Thananjeyan, B., Balakrishna, A., Rosolia, U., Li, F., McAllister, R., Gonzalez, J. E., Levine, S., Borrelli, F., and Goldberg, K. Safety augmented value estimation from demonstrations (saved): Safe deep model-based rl for sparse cost robotic tasks. IEEE Robotics and Automatio...

2020
[26]

E., Ibarz, J., Finn, C., and Goldberg, K

Thananjeyan, B., Balakrishna, A., Nair, S., Luo, M., Srinivasan, K., Hwang, M., Gonzalez, J. E., Ibarz, J., Finn, C., and Goldberg, K. Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 2021

2021
[27]

Safe reinforcement learning by imagining the near future

Thomas, G., Luo, Y., and Ma, T. Safe reinforcement learning by imagining the near future. In Advances in Neural Information Processing Systems (NeurIPS), 2021

2021
[28]

and Schwartz, A

Thrun, S. and Schwartz, A. Finding structure in reinforcement learning. Advances in neural information processing systems (NeurIPS), 7, 1994

1994
[29]

Mujoco: A physics engine for model-based control

Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In IEEE/RSJ International Conference on intelligent Robots and Systems (IROS), 2012

2012
[30]

E., Xu, S., and Peng, H

Wen, L., Duan, J., Li, S. E., Xu, S., and Peng, H. Safe reinforcement learning for autonomous vehicles through parallel constrained policy optimization. In IEEE International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2020

2020
[31]

and Denil, M

Xu, D. and Denil, M. Positive-unlabeled reward learning. In Conference on Robot Learning (CoRL), 2021

2021
[32]

Constraints penalized q-learning for safe offline reinforcement learning

Xu, H., Zhan, X., and Zhu, X. Constraints penalized q-learning for safe offline reinforcement learning. In AAAI Conference on Artificial Intelligence (AAAI), 2022

2022

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

Constrained policy optimization

Achiam, J., Held, D., Tamar, A., and Abbeel, P. Constrained policy optimization. In International conference on machine learning (ICML), 2017

work page 2017

[3] [3]

Constrained Markov decision processes: stochastic modeling

Altman, E. Constrained Markov decision processes: stochastic modeling. Routledge, 1999

work page 1999

[4] [4]

and Yarats, D

Amos, B. and Yarats, D. The differentiable cross-entropy method. In International Conference on Machine Learning (ICML), pp.\ 291--302. PMLR, 2020

work page 2020

[5] [5]

D., Chernova, S., Veloso, M., and Browning, B

Argall, B. D., Chernova, S., Veloso, M., and Browning, B. A survey of robot learning from demonstration. Robotics and autonomous systems, 57 0 (5): 0 469--483, 2009

2009

[6] [6]

I., Kroese, D

Botev, Z. I., Kroese, D. P., Rubinstein, R. Y., and L’Ecuyer, P. The cross-entropy method for optimization. In Handbook of statistics, volume 31, pp.\ 35--59. Elsevier, 2013

work page 2013

[7] [7]

W., Yuan, Z., Zhou, S., Panerati, J., and Schoellig, A

Brunke, L., Greeff, M., Hall, A. W., Yuan, Z., Zhou, S., Panerati, J., and Schoellig, A. P. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 2021

work page 2021

[8] [8]

B., Chernova, S., Taylor, M

Brys, T., Harutyunyan, A., Suay, H. B., Chernova, S., Taylor, M. E., and Now \'e , A. Reinforcement learning from demonstration through shaping. In International Joint Conference on Artificial Intelligence (IJCAI), 2015

work page 2015