{"id":"10332138-26c1-4e60-ae1c-abcc79f10451","arxiv_id":"2505.01619","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"SSkP uses PU learning to build a skill risk predictor from demonstrations and a cross-entropy risk planner to select safe skills during online RL, outperforming prior methods on most tested MuJoCo tasks.","lead":"Researchers introduce SSkP, a safe reinforcement learning method that first learns which skills are risky from offline demonstrations, then uses that risk information to plan safer actions during online training. The method reports higher rewards with fewer safety violations than three prior safe RL methods on four robot simulation tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The risk predictor used as the planning objective is trained offline but applied to online policy skills; its calibration is unvalidated, and the online adaptation in Eq. 7 can inject false positives, so the claimed safety improvement is not yet established.","rationale":"The reader's weakest assumption correctly identifies distribution shift between the offline demonstration distribution and the online policy's exploration distribution as the core risk to the central claim. I agree with that identification, and my stress-test expands it: the paper's own remedy for distribution shift, online PU adaptation in Eq. 7, is internally problematic because it labels randomly sampled, never-executed skills as positive whenever an episode later violates. This makes the concern more specific than a generic extrapolation worry. The reader's conditional verdict is appropriate: the method is plausible and the experiments show a generally positive trend, but the central safety mechanism is not independently validated. The paper also overstates its results: Table 1 places SSkP second on Ant and Appendix Figure 6 shows comparable Cheetah performance, and no code or data are released to disambiguate the Algorithm 1 versus Section 4.2.1 initialization mismatch. None of this forces a rejection; a calibration study and a corrected adaptation procedure would settle the main concern, so the verdict remains conditional.","tokens_in":11966,"tokens_out":6960,"duration_ms":77212,"concrete_test":"Run SSkP online with a frozen Pζ, logging each planned state-skill pair (s,z) and whether a violation occurs within H steps. Partition the logged pairs by predicted risk into deciles and compute calibration error between mean predicted risk and empirical violation frequency, focusing on the low-risk deciles selected by the planner. If calibration error exceeds 0.1 or the empirical violation rate is not monotonically increasing in predicted risk, the planning objective is miscalibrated on the online distribution. A second arm should ablate the Eq. 7 online adaptation entirely; if violations do not increase, the adaptation is not load-bearing, and if they do, the false-positive labeling in Eq. 7 must be corrected before the claimed safety benefit can be accepted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires Pζ(c|s,z) to be a reliable risk objective for the skills considered by the planner. Section 4.1.1 trains Pζ only on state-skill pairs from the offline demonstration manifold, while Section 4.2.1 and Algorithm 1 use it to select skills sampled from an iteratively refined Gaussian around qψ(·|st) (or πθ(·|st), depending on whether one follows the text or the algorithm). These online exploration skills can be out-of-distribution relative to the demonstrations, and nothing in the paper measures Pζ's calibration on the states and skills the planner actually selects. Figure 5 only shows that the CEM iterations reduce Pζ, which is circular evidence that the optimizer minimizes its own objective, not that Pζ corresponds to true violation risk. The intended correction for distribution shift, online adaptation via Eq. 7 and Algorithm 2 lines 9-12, is itself flawed: whenever an episode ends in a violation, all intermediate states receive (si, z∼qψ(·|si)) with z sampled from the skill prior, and these randomly sampled, never-executed skills are added to the positive set Dp. This injects false positives into Pζ's training data and can bias the predictor, so the mechanism meant to restore calibration may actually degrade it. Thus, the safety improvement attributed to risk planning currently rests on an unvalidated and potentially corrupted risk predictor.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SSkP, a safe reinforcement learning method that learns a skill risk predictor P_ζ(c|s,z) from offline demonstration data using positive-unlabeled (PU) learning, and then uses this predictor as the scoring function in a cross-entropy-method (CEM) risk planning procedure during online interaction. The planned skill is decoded into an action sequence to interact with the environment, and the skill policy is trained with a skill-based SAC objective. The method is evaluated on Ant, Cheetah, Hopper, and Humanoid MuJoCo environments against CPQ, SMBPO, and Recovery RL, with additional ablations of the risk planning components.","tokens_in":12297,"tokens_out":8007,"duration_ms":76306,"significance":"If the risk predictor can be shown to remain calibrated under online distribution shift, the approach is a meaningful step toward leveraging offline demonstrations for safe exploration, and it could be useful in applications where safety violations are costly. The paper's strengths include the use of PU learning to avoid negative labels, the integration of a CEM planner with a learned risk model, and the inclusion of ablation studies. However, the empirical support for the central 'consistent' claim is incomplete, and several implementation details require clarification before the contribution can be fully assessed.","major_comments":[{"comment":"The text and pseudocode disagree on the initialization of the risk planning distribution: Section 4.2.1 states that Ns skills are sampled from the current safe policy πθ(·|st) to compute the initial Gaussian, while Algorithm 1 initializes (µ0,σ0^2)←qψ(·|st), and Algorithm 2 line 4 calls Risk Planning with πθ as an argument that Algorithm 1 does not accept. This ambiguity determines whether the planner searches over the demonstration prior or the online policy, which is directly relevant to the distribution-shift concern for Pζ. Please reconcile the text, pseudocode, and the actual implementation.","section":"§4.2.1, Algorithm 1, Algorithm 2"},{"comment":"When an episode ends in a violation, every intermediate state si receives a skill z∼qψ(·|si) sampled from the skill prior, and this state-skill pair is added to the positive set Dp. Since those skills were never executed, there is no evidence that they lead to violations; labeling them as positive unsafe examples injects false positives into the PU training data and can bias the risk predictor. The paper should either justify this adaptation rule with empirical evidence or modify it, for example by only labeling the actually executed skill zt as positive.","section":"§4.2.2, Eq. (7), Algorithm 2 lines 9-12"},{"comment":"The central safety improvement claims rest on Pζ(c|s,z) being an accurate risk measure for the skills considered by the planner, but the predictor is trained on demonstration state-skill pairs and applied to skills sampled from the prior or online policy that may be out-of-distribution. No calibration analysis is provided for the online setting. Figure 5 only demonstrates that the CEM iterations reduce Pζ; since planning explicitly minimizes Pζ, this is circular evidence and does not establish that lower Pζ corresponds to lower true violation risk. Please report a calibration check on held-out online rollouts, for instance comparing predicted risk with empirical violation frequency in bins.","section":"§4.1.1, §4.2.1"},{"comment":"The abstract's claim that the method 'consistently outperforms' previous state-of-the-art safe RL methods is contradicted by the paper's own results: Table 1 shows SSkP is second-best on Ant (PtR/#V 23.54 vs SMBPO 28.68), and Appendix Figure 6 shows comparable Cheetah sample efficiency to Recovery RL and SMBPO. Section 5.2 acknowledges the Ant exception, but the abstract and conclusion do not. Please soften the claims accordingly and discuss the conditions under which SSkP is expected to help.","section":"Abstract, §5.2, Table 1, Appendix Figure 6"},{"comment":"All results are averages over only three runs, and Table 1 reports no variance or significance testing for the PtR/#V ratio. The 'notable performance gains' asserted for Cheetah and Hopper would be more convincing with per-run values or confidence intervals on the ratio metric. Given the variability visible in Figure 3, the empirical superiority claim needs stronger statistical support.","section":"§5.2, Table 1"}],"minor_comments":[{"comment":"The notation ∇p̄_i for the change in average predicted risk is confusing because ∇ usually denotes gradient; using Δp̄_i or δ_i would be clearer. The sentence 'with the increase of the risk planning iterations, −∇p̄_i becomes larger and hence p̄_i becomes smaller' is also awkwardly phrased.","section":"§5.4"},{"comment":"The PU loss in Eq. (4) would benefit from a more explicit connection to the nnPU formulation of Kiryo et al., since the slack variable ξ is set to 0 and the reader must infer how the relaxed constraint is derived.","section":"Eq. (4)"},{"comment":"SAFER (Slack et al., 2022) is a closely related skill-based safe RL method, but it is only mentioned in passing and not included in the comparison; the authors should either compare against it or explain why it is excluded.","section":"§2, Related Works"},{"comment":"The hyperparameters (H=10, skill dimension=10, Ns=512, k=64, Np=6, λ) are fixed without any sensitivity study; a brief sensitivity analysis or a paragraph justifying these choices would strengthen the paper.","section":"§5.1, Implementation Details"},{"comment":"The figure legends are not consistent across plots, making it hard to compare methods; please use a uniform legend with the same colors and line styles in all figures.","section":"Figures 3 and 4"},{"comment":"The demonstration dataset Dd is introduced in the problem setting but never formally defined with an explicit set-builder notation; its notation is also close to the online dataset D, which could confuse readers. Please clarify the notation.","section":"§3"}],"recommendation":"major_revision","confidential_remarks":"The paper's headline claim of 'consistently outperforming' is not supported by the data in Table 1 and Appendix Figure 6, and the internal inconsistency between Section 4.2.1 and Algorithm 1 needs to be resolved. The most important scientific concern is the unvalidated calibration of the risk predictor under online distribution shift, compounded by the false-positive injection in Eq. (7). The paper would become publishable if these points are addressed with additional analyses or careful rewording. I also note that the closely related SAFER method is omitted from the comparison, which the authors should at least discuss."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate engineering contribution combining SPiRL skill priors, PU learning, and CEM planning for safe RL with offline demos, but the headline claim of consistent SOTA is overstated and the risk predictor's reliability under online distribution shift is not established. It deserves a serious referee, not a desk reject.\n\nWhat's new: the specific pipeline — a PU-trained skill risk predictor Pζ(c|s,z) used as the scoring function in a cross-entropy planner over skill latents, with simultaneous online adaptation — is not in the cited literature. The experiments span four MuJoCo environments and compare against Recovery RL, CPQ, and SMBPO, and the ablations (full SSkP vs. naive planning vs. no risk predictor) are well-designed. That is real work and worth engaging with.\n\nThe soft spots, in order of importance. First, the abstract's 'consistently outperforms' is contradicted by the paper's own Table 1, where SMBPO is best on Ant, and by Appendix Figure 6, where Cheetah sample efficiency is comparable to Recovery RL/SMBPO. This is a framing problem, not a fatal one. Second, Algorithm 1 initializes the planning distribution from the skill prior qψ, while Section 4.2.1 says it initializes from the current policy πθ. That is a concrete inconsistency that matters because the choice changes the distribution of skills being scored. Third — the load-bearing issue — the risk predictor is trained offline on demonstration state-skill pairs, then used to score skills proposed by the online policy, which may visit states far from the demonstrations. The paper provides no calibration check on the states/skills actually selected by the planner. Figure 5 only shows that CEM iterations reduce Pζ, which is the optimizer minimizing its own objective, not evidence that Pζ tracks true violation risk. Worse, the online adaptation loop in Eq. 7 adds randomly sampled skills z∼qψ from every intermediate state in a violated episode to the positive set, even though those skills were never executed. That injects false positives into the PU data and can bias the predictor further. This is a real flaw, not a nitpick.\n\nThe paper releases no code or data, so independent verification is limited. The citation pattern is appropriate; the PU and skill-learning references are the right ones.\n\nWho this is for: people working on safe RL with offline demonstrations, or on learned risk models for planning. They will get a useful baseline and a clear problem formulation, but they should not take the safety improvement at face value.\n\nRecommendation: send to peer review with the expectation of major revision. A good reviewer should probe the risk predictor's calibration on online data and ask the authors to fix or justify the Eq. 7 positive-set construction.","headline":"Useful safe-RL-with-demos method, but the consistency claim is overstated and the risk predictor's online calibration is unvalidated; deserves a serious referee without a desk reject.","tokens_in":12811,"tokens_out":4223,"would_cite":false,"duration_ms":39844,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SSkP claims a demonstration-trained skill risk predictor, used as the objective of a cross-entropy planning loop, consistently outperforms prior safe RL methods on four robotic benchmarks with higher reward and fewer violations.","keywords":["safe reinforcement learning","skill learning","positive-unlabeled learning","risk planning","cross-entropy method","learning from demonstrations","constrained Markov decision process","robotic control"],"falsifier":"Collect the state-skill pairs visited by the online policy during SSkP training, group them by the predictor's output $P_\\zeta(c|s,z)$, and compare against whether a violation actually occurred within the next $H$ steps when those skills were executed; if the predictor is miscalibrated on policy-visited states while remaining accurate on demonstration states, the planner's safety gains would be an artifact of the demonstrations rather than a property of the online process. A second direct test is to freeze the predictor instead of updating it each episode and measure whether the violation count rises.","tokens_in":11710,"feed_emoji":"🦾","tokens_out":11738,"duration_ms":96374,"temperature":0.7,"pith_summary":"Safe reinforcement learning usually pays for its education in safety violations: the agent must explore the real environment to discover what is dangerous. This paper claims there is a cheaper route: first learn a skill risk predictor from a small offline demonstration dataset, then use that predictor as a planning objective that screens candidate skills before they are ever executed online. The claimed result, from a method called SSkP, is that this two-stage scheme consistently beats prior state-of-the-art safe RL methods (Recovery RL, CPQ, SMBPO) on four robotic simulation environments, reaching higher average episode reward with fewer safety violations within a fixed budget of online timesteps. If the claim holds, offline demonstrations can be exploited for safety without pre-training the policy itself, and even rare violations inside demonstrations suffice to build a useful risk signal.","feed_headline":"Skill-risk planning outperforms safe RL baselines on four robots","feed_subtitle":"A demonstration-trained predictor screens candidate skills before execution, lifting reward while cutting safety violations.","key_machinery":"The load-bearing object is the skill risk predictor $P_\\zeta(c|s,z)$, a small MLP trained with a non-negative positive-unlabeled loss built on an unbiased estimator of the negative loss, with the positive-class prior $\\lambda$ estimated from the data. Its training data are state-skill pairs produced by a skill model, whose encoder $q_\\mu(z_t|a_t)$, decoder $p_\\nu(a_t|z_t)$, and prior $q_\\psi(z_t|s_t)$ are trained on the demonstrations and turn length-$H$ action sequences into latent skills; a pair is positive if a violation occurred within the next $H$ steps, and near trajectory ends the prior network supplies the skill for a state. The second mechanism is the risk planner, a cross-entropy-method loop that starts from a Gaussian fit to skills sampled from the current policy $\\pi_\\theta$, iteratively samples a batch, scores each candidate with the predictor, refits the Gaussian to the lowest-risk $k$ skills, and after $N_p$ iterations samples the skill to execute. The planner converts a static offline-trained classifier into an online safety filter, and online rollouts contribute fresh state-skill pairs that retrain the predictor each episode, keeping the planner's objective aligned with the encountered environment.","core_discovery":"The paper's discovery is that a single trained object, a skill risk predictor $P_\\zeta(c|s,z)$ estimating the probability that choosing latent skill $z$ in state $s$ produces a safety violation ($c>0$) within the next $H$ steps, can carry the safety burden of online RL. The predictor is learned with positive-unlabeled (PU) learning on offline demonstrations, where positive examples are the state-skill pairs that actually led to a violation within the next $H$ steps and all other pairs are unlabeled, a construction that avoids both the scarcity of violations in demonstration data and the need to label near-miss decisions as safe or unsafe. During online learning the predictor becomes the objective of a risk planning process: starting from a Gaussian fit over skills sampled from the current policy, the planner iteratively resamples candidate skills, keeps the top-$k$ with the lowest predicted risk, refits the Gaussian, and after $N_p$ iterations executes the winning skill. SSkP reports that this loop, together with periodic updates of the predictor from online rollouts, consistently outperforms Recovery RL, CPQ, and SMBPO across Ant, Cheetah, Hopper, and Humanoid, and its ablations show that both the risk predictor and the iterated planner are needed for the gain.","pith_inferences":["The PU construction suggests SSkP's advantage should grow as demonstrations get safer, since the unbiased negative-loss estimator matters most when confirmed violations are rare; a direct test is varying the demonstration violation rate and watching the margin over baselines.","The predictor is trained on demonstration states but deployed on states visited by the online policy, so its calibration under distribution shift is the main open risk; a natural follow-up is checking whether predicted risk tracks the empirical violation frequency on out-of-distribution states and whether the per-episode updates are what preserve validity.","Because the planner is a zeroth-order solver for $\\arg\\min_z P_\\zeta(c|s,z)$, the planning loop is agnostic to how the risk signal is produced and could wrap any black-box safety score, which connects the method to constrained model-predictive control.","The fixed skill horizon $H$ and planner sizes ($N_s$, $k$, $N_p$) likely interact with how far ahead dangers can be foreseen in each environment, so ablating $H$ per environment is a concrete way to test where the method's benefit concentrates."],"forward_implications":["Removing the skill risk predictor and the planner (the SSkP-w/o-RP variant) causes a substantial decline in reward-versus-violation performance on Ant and Hopper, so the demonstration-derived risk signal, not just the skill prior, carries the safety gain.","Replacing the iterated cross-entropy planner with a single-batch naive planner (SSkP-NP) hurts performance, so the iterative Gaussian refinement adds value beyond one-shot risk screening.","Under the paper's cost-sensitive sample-efficiency metric (per-timestep reward divided by total violations), SSkP ranks first in three of four environments and second in Ant, where SMBPO leads.","The planning loop lowers the mean predicted risk of its sampled skills across all $N_p$ iterations in every environment, which the paper reads as evidence that the planner genuinely converges toward safer skills.","Because CPQ, which pre-trains its policy on the offline data, stalls at low reward in all environments, the paper concludes that demonstrations are better exploited for safety through a risk predictor than through policy pre-training."],"supporting_citations":[{"why":"Supplies the deep skill model (encoder, decoder, prior) that SSkP uses to encode length-H action sequences as latent skills and to generate skill decisions for states.","marker":"(Pertsch et al., 2021)"},{"why":"Provides the non-negative PU loss with slack that SSkP minimizes to train the skill risk predictor.","marker":"(Kiryo et al., 2017)"},{"why":"Gives the unbiased estimation of the true negative loss that makes the PU training objective feasible without confirmed negative examples.","marker":"(Du Plessis et al., 2015)"},{"why":"Defines the cross-entropy method that the risk planning loop instantiates as a zeroth-order solver for the risk minimization.","marker":"(Botev et al., 2013)"},{"why":"Recovery RL is the main demonstration-based safe RL baseline that SSkP is compared against.","marker":"(Thananjeyan et al., 2021)"},{"why":"CPQ is the offline-pretrained safe RL baseline that SSkP compares against and outperforms.","marker":"(Xu et al., 2022)"},{"why":"Provides SMBPO, a model-based safe RL baseline, and the customized MuJoCo environments and evaluation scheme used in the experiments.","marker":"(Thomas et al., 2021)"},{"why":"Soft actor-critic is the base RL algorithm the skill policy is trained with, using a KL replacement of the entropy term.","marker":"(Haarnoja et al., 2018)"},{"why":"Supplies the class-prior estimation used to set the lambda parameter in the PU risk-predictor loss.","marker":"(Jain et al., 2016)"}],"fun_headline_variants":["PU-learned skill risk drives safer RL","Skill risk predictor and planning improve safe RL","Offline skill risk planning outperforms safe RL baselines","Risk planning with skill predictor cuts safety violations","Skill risk prediction lifts safe RL on four robots"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The risk predictor is trained on offline demonstration state-skill pairs and then used online to score skills sampled from the evolving policy, so the method collapses if the predictor becomes miscalibrated on states the online exploration reaches, because the planner would then minimize a stale objective and stop preventing real violations.","fun_headline_variants_meta":{"raw":{"variants":["PU-learned skill risk drives safer RL","Skill risk predictor and planning improve safe RL","Offline skill risk planning outperforms safe RL baselines","Risk planning with skill predictor cuts safety violations","Skill risk prediction lifts safe RL on four robots"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1440,"prompt_tokens":943,"completion_tokens":497,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":427}},"tokens_in":559,"tokens_out":497,"duration_ms":5277,"temperature":1.0,"reasoning_tokens":427,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:14:42.218928+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect the state-skill pairs visited by the online policy during SSkP training, group them by the predictor's output $P_\\zeta(c|s,z)$, and compare against whether a violation actually occurred within the next $H$ steps when those skills were executed; if the predictor is miscalibrated on policy-visited states while remaining accurate on demonstration states, the planner's safety gains would be an artifact of the demonstrations rather than a property of the online process. A second direct test is to freeze the predictor instead of updating it each episode and measure whether the violation count rises.","supporting_citations":[{"cited_title":"Accelerating reinforcement learning with learned skill priors","cited_arxiv_id":null,"evidence_quote":"Supplies the deep skill model (encoder, decoder, prior) that SSkP uses to encode length-H action sequences as latent skills and to generate skill decisions for states."},{"cited_title":"C., and Sugiyama, M","cited_arxiv_id":null,"evidence_quote":"Provides the non-negative PU loss with slack that SSkP minimizes to train the skill risk predictor."},{"cited_title":"Convex formulation for learning from positive and unlabeled data","cited_arxiv_id":null,"evidence_quote":"Gives the unbiased estimation of the true negative loss that makes the PU training objective feasible without confirmed negative examples."},{"cited_title":"I., Kroese, D","cited_arxiv_id":null,"evidence_quote":"Defines the cross-entropy method that the risk planning loop instantiates as a zeroth-order solver for the risk minimization."},{"cited_title":"E., Ibarz, J., Finn, C., and Goldberg, K","cited_arxiv_id":null,"evidence_quote":"Recovery RL is the main demonstration-based safe RL baseline that SSkP is compared against."},{"cited_title":"Safe reinforcement learning by imagining the near future","cited_arxiv_id":null,"evidence_quote":"Provides SMBPO, a model-based safe RL baseline, and the customized MuJoCo environments and evaluation scheme used in the experiments."},{"cited_title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","cited_arxiv_id":null,"evidence_quote":"Soft actor-critic is the base RL algorithm the skill policy is trained with, using a KL replacement of the entropy term."},{"cited_title":"Estimating the class prior and posterior from noisy positives and unlabeled data","cited_arxiv_id":null,"evidence_quote":"Supplies the class-prior estimation used to set the lambda parameter in the PU risk-predictor loss."}],"review_version":1}