Pith. sign in

REVIEW 4 major objections 34 references

SafeExplorer: An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventions

T0 review · 4 major / 0 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A masked policy gradient makes recovery interventions usable in on-policy RL without biasing updates, cutting training falls by tens to hundreds of times while matching PPO reward.

desk verdict Clean density-free masked gradient for recovery-mixed PPO, solid sim ablations, but the abstract fall factors omit recovery pre-training cost and partly self-set the success bar. read the letter →

arxiv 2607.08925 v1 pith:T4MMAUXI submitted 2026-07-09 cs.LG

Elham Daneshmand , Majid Khadiv , Glen Berseth , Hsiu-Chin Lin This is my paper · ORCID
classification cs.LG
keywords safereinforcementlearningpolicygradientrecoveryproximaloptimizationmixedleggedlocomotionimportancesamplingregion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

When you train a robot with reinforcement learning, every fall can damage hardware, so the practical goal is to minimize falls during training rather than merely trade them against reward. A common fix hands control to a separate recovery controller whenever the agent leaves a safe region, but those mixed rollouts silently corrupt every on-policy update, and importance sampling cannot fix the bias when the recovery controller is deterministic. SafeExplorer is a drop-in change to PPO whose core is an unbiased gradient that scores only the safe steps and never needs the recovery policy’s density, so it works for both deterministic and stochastic recoveries. Two extra pieces speed credit assignment at the boundary: a closed-form value for recovery-triggering states under deterministic dynamics, and an imitation loss that copies recovery actions only after successful recoveries. On HalfCheetah, Ant, and Unitree Go1 it reduces training falls by factors of 233×, 48×, and 26× versus plain PPO while matching or beating final reward, and on Ant—where recovery itself is unreliable—it is the only method that reaches the success bar.

What carries the argument

Unbiased partition-policy gradient (Theorem 1 / masked safe-step PPO surrogate): the mixed-policy return’s gradient factors so only main-policy actions inside the main-policy set contribute score terms; recovery actions drop out of differentiation because the recovery measure is θ-independent.

What would settle it

Train SafeExplorer and standard PPO (and an importance-sampling variant with stochastic recovery) on the same three locomotion tasks with the same frozen recovery controllers; if the masked estimator does not match or beat IS when recovery is stochastic, or if total training falls do not drop by large factors while final unmodified task reward stays at or above PPO, the central empirical claim fails.

Watch

Extended reading notes

Core claim

The central claim is that mixed-policy rollouts from a main policy plus a θ-independent recovery policy admit an unbiased policy-gradient estimator that uses the main-policy score function only on safe timesteps and never evaluates the recovery density. That estimator stays valid for deterministic recoveries where importance sampling is undefined, and with analytic recovery values and hard outcome-gated imitation it turns recovery from a source of bias into a source of signal, drastically cutting training falls without sacrificing asymptotic reward.

Load-bearing premise

The closed-form recovery value that densifies learning at the safe-region boundary assumes deterministic dynamics and a deterministic recovery path; under real process noise that value becomes a noisy single-sample estimate the paper does not bound.

Editorial extensions

If this is right

  • On-policy methods can keep an external recovery or shield controller active during training without corrupting the gradient, including when that controller is deterministic (MPC, greedy actor).
  • Safe-region curricula plus outcome-gated imitation can break the feedback loop in which unreliable recovery keeps re-entering failure states.
  • Jump-start RL and state-triggered shielding become special cases of the same factorization, so the same masked gradient applies there.
  • Training-time fall counts, not only final return, become a first-class metric for real-robot RL benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same masking idea should transfer to human-in-the-loop intervention and emergency-stop stacks in driving or manipulation once a fixed recovery or teacher is available.
  • Excluding recovery pre-training cost from reported budgets means hardware fall savings will look smaller until recovery acquisition is amortized or learned online.
  • If process noise is large, multi-sample Monte-Carlo recovery targets (sketched in the appendix) may be required before the analytic-value gains hold outside simulation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 0 minor

Summary. SafeExplorer addresses on-policy bias when a recovery policy intervenes outside a designer-specified safe region. The core contribution is a masked policy-gradient estimator (Theorem 1 / Corollary 2–3) that uses the main-policy score only on safe steps and never evaluates the recovery density, remaining valid for deterministic recoveries where importance sampling is undefined. Supporting pieces include an objective-gap bound between mixed and main returns (Theorem 4), a closed-form recovery value under deterministic dynamics (Theorem 6), and an outcome-gated compatibility regularizer that imitates only successful recovery segments. Empirically, on HalfCheetah, Ant, and Unitree Go1 (5 seeds), the method reports large reductions in training falls versus PPO while matching or exceeding final reward, and is the only method to reach the 80% success bar on Ant under an unreliable recovery.

Significance. If the results hold under a fully costed safety ledger, this is a useful and well-scoped contribution to hard-safety on-policy RL. The factorization proof for the mixed-policy gradient is standard but correctly specialized to density-free external controllers, and the paper is unusually thorough: full proofs, rliable IQM with bootstrap CIs, a controlled ablation ladder, IS variance diagnostics, CMDP feasibility audits, and extensive sensitivity tables. The Ant unreliable-recovery regime is a genuine stress test that prior recovery-using methods fail. The work is simulation-only and locomotion-only, but the estimator itself is not robot-specific and could transfer to other intervention settings (JSRL, shields) as the corollaries claim.

major comments (4)
  1. Abstract and Table 1 advertise 233× / 48× / 26× fall reductions over PPO, but Section D.3 states recovery pre-training costs 5M / 3M / 10M environment steps (and untracked falls) that are excluded from Table 5 budgets and from all fall counts. Without a free recovery the method cannot run; for a hardware-motivated safety claim this incomplete ledger is load-bearing. Either fold pre-training falls/steps into the reported totals (or a total-cost column), or reframe the abstract factors as conditional on an already-available recovery and report the pre-training cost prominently next to the headline numbers.
  2. Section 6 defines success as 80% of the best final reward attained by any method; on Ant and Go1 that best method is SafeExplorer, so Figure 1’s falls-to-success metric partly uses a bar the method itself sets. This does not invalidate Table 1’s within-protocol ordering, but it weakens the abstract claim that SafeExplorer is “the only method that reaches 80% of the best final reward” as an independent success criterion. Redefine success against a fixed external bar (e.g., 80% of unconstrained PPO’s final reward, or a pre-specified absolute threshold) and recompute Figure 1, or report both bars.
  3. Theorem 6 / Proposition 6 and the largest HalfCheetah single-ingredient gain rely on deterministic dynamics and recovery so the post-trigger path is a single fixed trajectory. Section A.5 correctly notes that under process noise the target becomes a single-sample MC estimate, but the paper never bounds that variance or measures degradation under controlled noise. Given that the analytic-V overwrite is presented as a core algorithmic piece (Algorithm 1, Section 4.3), either add a noise-robustness experiment (or multi-sample MC ablation) or clearly demote the analytic value to an optional, determinism-dependent acceleration rather than a general component of the method.
  4. Theorem 4 bounds |J − J_mix| by the unobservable out-of-region rate β(θ,d); Section E.12 uses the observable recovery rate α only as a diagnostic, and Section A.7 admits that a non-vacuous observable bound is open under deterministic recovery. The empirical gap-tracking in Table 15 / Figure 5 is supportive but not a substitute. Either strengthen the claim language around Theorem 4 to “structural proportionality tracked by α” rather than a deployable guarantee, or provide at least one of the sketched routes (smoothed recovery, offline π_θ evaluation) as a concrete bound on the body runs.

Circularity Check

1 steps flagged · score 1.0 of 10

No derivation circularity: Theorem 1 is a standard score-function factorization; only a mild protocol self-reference in the Ant/Go1 success bar.

  1. other [Section 6 (Experimental Setup); Abstract; Figure 1]
    "A single metric adjudicates all three: the number of training falls a method incurs before first reaching task success, where success is 80% of the best final reward attained by any method in that environment. On Ant and Go1 that best method is SafeExplorer, so the reward winner partly sets its own bar"

    The falls-to-success claim and the abstract’s “only method that reaches 80% of the best final reward” on Ant use a threshold defined from the best observed reward, which is SafeExplorer’s. That is a mild self-referential metric design, not a fitted-parameter-as-prediction or definitional collapse of Theorem 1. Raw Table 1 rewards and total falls remain independently measured.

full rationale

The load-bearing theoretical claim is Theorem 1 (unbiased partition-policy gradient). Its proof (A.1) factors the mixed trajectory measure into θ-dependent main-policy factors on M and θ-independent recovery factors outside M, then applies the product rule and score identity only to the former. That is a self-contained derivation from the definition of π_mix; it does not fit parameters to fall counts, import uniqueness from the authors, or rename a known empirical pattern as a prediction. Theorems 4–6 and the compatibility regularizer are likewise consequences of stated assumptions (Performance Difference Lemma; deterministic path collapse; outcome-gated NLL), not tautologies of the reported metrics. Empirical quantities—total falls, reward, α, J_mix−J—are measured on held-out evaluation protocols and are not forced by the loss. The sole mild self-reference is the falls-to-success bar (Section 6): success = 80% of the best final reward of any method, which on Ant and Go1 is SafeExplorer itself, so the “only method to reach success on Ant” claim partly uses a bar the method sets. That affects Figure 1’s ranking narrative, not the abstract’s 233×/48×/26× factors (those are total-training-falls ratios from Table 1 vs PPO) and not any equation. Recovery pre-training cost exclusion (D.3) is a ledger incompleteness, not circular derivation. Score 1 for that single non-load-bearing protocol self-reference; core math is independent.

Assumptions & free parameters 5 free parameters · 6 assumptions · 3 invented entities

The central unbiasedness claim rests on standard score-function regularity and θ-independence of the recovery measure—not on fitted constants. Empirical fall reductions further depend on hand-chosen safe-region geometry, curriculum endpoints d_max, compatibility weight λ_compat, and pre-trained SAC recoveries whose training cost is outside the reported budget. Analytic-value gains add a determinism assumption. No new physical entity is postulated; the ‘invented’ pieces are algorithmic constructs.

free parameters (5)
  • d_max (safe-region curriculum endpoint) = HC:2.0; Ant:0.4; Go1:0.15 (task-space units)
    Per-env hyperparameter (HalfCheetah 2.0, Ant 0.4, Go1 0.15) set by qualitative intervention-rate criterion without systematic search under the curriculum; shapes gap bound and fall counts via Theorem 4.
  • λ_compat (compatibility regularizer weight) = 1e-3 (default)
    Default 1e-3 chosen on falls not reward; sweep shows environment-specific optima and 8× fall sensitivity within a decade on HalfCheetah.
  • Safe-region distance D and scaling (esp. Ant tilt)
    Hand-designed (z,tilt) Euclidean mixes; Ant uses a rescaled axis-angle that strongly downweights orientation—an implementation choice kept for fidelity.
  • SAC recovery pre-training budgets and reward shaping = HC 5M / Ant 3M / Go1 10M steps
    Separate 3–10M-step SAC trainings with env-specific costs; Ant’s control-magnitude penalty makes recovery unreliable—the regime the method claims to handle.
  • CMDP cost limit 0.05 = 0.05 falls/episode
    Constraint budget for CPO/PPO-Lagrangian comparison; feasibility fails on Ant/Go1, which is part of the argument but is a free solver setting.
assumptions (6)
  • standard math Assumption 1 (R1–R3): main-policy log-density differentiable on support in M; bounded rewards/horizon; differentiation under the integral.
    Standard policy-gradient regularity; holds for diagonal-Gaussian PPO with bounded mean/log-std.
  • domain assumption Recovery measure μ (or π_rec) is independent of main-policy parameters θ.
    Load-bearing for Theorem 1 factorization; recovery is frozen SAC/MPC, never co-trained.
  • domain assumption Assumption 2: approximate one-step π_θ-invariance of R with slack η (used to bound β and the objective gap).
    Not forced to zero under reading (R-B); paper tracks observable α as diagnostic instead of a certified bound on β.
  • domain assumption Deterministic dynamics P and deterministic recovery for closed-form recovery value (Theorem 6).
    Exact analytic V; under noise becomes single-sample MC (A.5). Experiments use greedy SAC mean.
  • domain assumption Episodes start inside R (supp p(s1) ⊆ R) for the β ≤ ηγ/(1−γ) derivation.
    Stated in A.2; all three envs reset upright inside the region.
  • ad hoc to paper Hard outcome gate uses realized segment success (re-entry) as the imitation filter.
    Design choice that dominates soft gates empirically; without it the method fails on unreliable Ant recovery.
invented entities (3)
  • Partition-masked mixed-policy gradient (SafeExplorer core estimator) independent evidence
    purpose: Unbiased ∇θ J_mix without evaluating recovery density; drop-in PPO surrogate over safe steps only.
    Algorithmic construct derived from trajectory factorization; not a physical entity. Independent evidence is the theorem plus IS comparison experiments.
  • Outcome-gated compatibility score C_θ(τ) / L_compat independent evidence
    purpose: Pull main policy toward successful recovery actions only, without needing recovery density.
    New regularizer relative to ungated DAgger/JSRL; falsifiable via ablations (hard gate required on Ant).
  • Analytic recovery-value overwrite at trigger states independent evidence
    purpose: Dense, low-variance critic targets at the safe-region boundary under determinism.
    Direct consequence of deterministic unrolling; largest single-ingredient reward gain on HalfCheetah in ablations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SafeExplorer: An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventions." pith.science (2026). https://pith.science/paper/T4MMAUXI

@misc{pith2026260708925,
  author       = {Pith},
  title        = {Pith review of: SafeExplorer: An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T4MMAUXI}},
  note         = {Machine review of arXiv:2607.08925}
}
read the original abstract

Training reinforcement-learning agents directly on physical robots makes every fall costly, since a fall can damage the platform and cannot be undone like a simulator reset; the goal is therefore to minimize falls during training rather than trade them off against return, as constrained Markov decision process (MDP) formulations do. A standard mitigation hands control to a separate recovery policy whenever the agent leaves a designer-specified safe region (a subset of state space it should stay within), but the resulting mixed-policy rollouts silently bias every on-policy update, and the importance-sampling correction that would remove this bias is ill-defined whenever the recovery policy is deterministic. We address this bias with a drop-in modification of proximal policy optimization (PPO). Its core is an unbiased policy-gradient estimator that uses the score function only at safe timesteps and never evaluates the recovery policy's density, so it stays valid even when the recovery policy is deterministic, exactly where importance sampling breaks, and it empirically dominates importance sampling even when the recovery policy is stochastic. Because the recovery policy still makes credit assignment slow near the safe-region boundary, two further components accelerate learning: a closed-form value for recovery-triggering states when dynamics and recovery are deterministic, and an imitation loss that copies recovery actions only when recovery succeeds. On a three-environment, five-seed benchmark, the resulting algorithm reduces training-time falls by factors of 233x, 48x, and 26x on HalfCheetah, Ant, and Unitree Go1 over standard PPO, while matching or exceeding PPO's final reward, and on Ant, where the recovery policy is unreliable, it is the only method that reaches 80% of the best final reward.

Figures

Figures reproduced from arXiv: 2607.08925 by the authors.

Figure 1
Figure 1. Training falls to reach task success (success [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. rliable interquartile-mean intervals of reward and total falls for the six methods in Table 1, with [PITH_FULL_IMAGE:figures/full_fig_p030_2.png] view at source ↗
Figure 3
Figure 3. Per-environment learning curves: rows are episodic reward and cumulative training-time falls; [PITH_FULL_IMAGE:figures/full_fig_p031_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Ablation ladder on the reward-vs-falls plane: PPO and the four rungs, Unmasked PG [PITH_FULL_IMAGE:figures/full_fig_p031_4.png]
Figure 5
Figure 5. Figure 5: Objective gap over training for the SafeEx [PITH_FULL_IMAGE:figures/full_fig_p036_5.png]
Figure 6
Figure 6. Figure 6: Recovery rate α (fraction of rollout steps on which recovery fired) over training, per environment, for the recovery-using methods (the CMDP solvers act with a single policy and never invoke recovery, so they have no α). Solid = mean across seeds, shaded = [min, max] e…
Figure 7
Figure 7. Figure 7: Masked gradient versus IS on a stochastic recovery, SafeExplorer (mean [PITH_FULL_IMAGE:figures/full_fig_p038_7.png]
Figure 8
Figure 8. Figure 8: Training SafeExplorer with the masked gradient versus the truncated-IS gradient (mean [PITH_FULL_IMAGE:figures/full_fig_p038_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 5 canonical work pages

  1. [1]

    Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada

    Aaron D. Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada. Control barrier functions: Theory and applications. InProceedings of the 2019 European Control Conference (ECC), pp. 3420–3431,

    2019

  2. [2]

    ISBN 9781605585161

    Association for Computing Machinery. ISBN 9781605585161. doi: 10.1145/1553374.1553380. URLhttps://doi.org/10.1145/1553374.1553380. Miroslav Bogdanovic, Majid Khadiv, and Ludovic Righetti. Model-free reinforcement learning for robust locomotion using demonstrations from trajectory optimization.Frontiers in Robotics and AI, 9,

    doi:10.1145/1553374.1553380

  3. [3]

    Safe exploration in continuous action spaces.CoRR, abs/1801.08757,

    Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerík, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces.CoRR, abs/1801.08757,

    arXiv

  4. [4]

    Mohammadhosein Hasanbeig, Alessandro Abate, and Daniel Kroening

    doi: 10.15607/RSS.2019.XV.011. Mohammadhosein Hasanbeig, Alessandro Abate, and Daniel Kroening. Cautious reinforcement learning with logical constraints. In Amal El Fallah Seghrouchni, Gita Sukthankar, Bo An, and Neil Yorke- Smith (eds.),Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’20, Auckland, New ...

    work page doi:10.15607/rss.2019.xv.011 2019

  5. [5]

    Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and Jo ao G

    doi: 10.5555/3398761.3398821. Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and Jo ao G. M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms.Journal of Machine Learning Research, 23(274):1–18,

    doi:10.5555/3398761.3398821

  6. [6]

    Choi, Michael Janner, Claire J

    Katie Kang, Paula Gradu, Jason J. Choi, Michael Janner, Claire J. Tomlin, and Sergey Levine. Lyapunov density models: Constraining distribution shift in learning-based control. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.),International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Balt...

    2022

  7. [7]

    Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik

    doi: 10.1109/ICRA.2019.8793698. Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. RMA: rapid motor adaptation for legged robots. In Dylan A. Shell, Marc Toussaint, and M. Ani Hsieh (eds.),Robotics: Science and Systems XVII, Virtual Event, July 12-16, 2021,

    doi:10.1109/icra.2019.8793698 2019

  8. [8]

    Robust Recovery Controller for a Quadrupedal Robot using Deep Reinforcement Learning

    doi: 10.15607/RSS.2021.XVII.011. Joonho Lee, Jemin Hwangbo, and Marco Hutter. Robust recovery controller for a quadrupedal robot using deep reinforcement learning.CoRR, abs/1901.07517,

    work page Pith review arXiv doi:10.15607/rss.2021.xvii.011 2021

Show all 34 references
  1. [9]

    Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E

    doi: 10.1109/ICRA.2018.8463162. Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, and Peter Stone. Curricu- lum learning for reinforcement learning domains: A framework and survey.Journal of Machine Learning Research, 21(181):1–50,

    2018 doi

  2. [10]

    Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang-Wei Edward Lee, Jie Tan, and Sergey Levine

    doi: 10.1109/ICRA.2018.8460528. Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang-Wei Edward Lee, Jie Tan, and Sergey Levine. Learning agile robotic locomotion skills by imitating animals. In Marc Toussaint, Antonio Bicchi, and Tucker Hermans (eds.),Robotics: Science and Syste...

    2018 doi

  3. [11]

    Xun Pua and Majid Khadiv

    doi: 10.15607/RSS.2020.XVI.064. Xun Pua and Majid Khadiv. Safe learning of locomotion skills from mpc. In2024 IEEE-RAS 23rd Interna- tional Conference on Humanoid Robots (Humanoids), pp. 459–466,

    2020 doi

  4. [12]

    2024.10769799

    doi: 10.1109/Humanoids58906. 2024.10769799. Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. Technical report, OpenAI,

    2024 doi

  5. [13]

    Learning to walk in minutes using massively parallel deep reinforcement learning

    Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Aleksandra Faust, David Hsu, and Gerhard Neumann (eds.), Conference on Robot Learning, 8-11 November 2021, London, UK, volume 164 ...

    2021

  6. [14]

    Trial without error: Towards safe reinforcement learning via human intervention

    15 William Saunders, Girish Sastry, Andreas Stuhlmüller, and Owain Evans. Trial without error: Towards safe reinforcement learning via human intervention. InProceedings of the 17th International Conference on Au- tonomous Agents and MultiAgent Systems, AAMAS ’18, pp. 2067–2069...

  7. [15]

    Proximal policy optimiza- tion algorithms.CoRR, abs/1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms.CoRR, abs/1707.06347,

    arXiv

  8. [16]

    Smith, J

    Laura M. Smith, J. Chase Kew, Tianyu Li, Linda Luu, Xue Bin Peng, Sehoon Ha, Jie Tan, and Sergey Levine. Learning and adapting agile locomotion skills by transferring experience. In Kostas E. Bekris, KrisHauser, Sylvia L.Herbert, andJingjinYu(eds.),Robotics: Science and System...

    2023 doi

  9. [17]

    Learning to be safe: Deep RL with a safety critic.CoRR, abs/2010.14603,

    Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, and Chelsea Finn. Learning to be safe: Deep RL with a safety critic.CoRR, abs/2010.14603,

    2010 arXiv

  10. [18]

    Sim-to-real: Learning agile locomotion for quadruped robots

    Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. In Hadas Kress- Gazit, Siddhartha S. Srinivasa, Tom Howard, and Nikolay Atanasov (eds.),Robotics: Sc...

    2018

  11. [19]

    Chen Tessler, Daniel J

    doi: 10.15607/RSS.2018.XIV.010. Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization.CoRR, abs/1805.11074,

    2018 doi arXiv

  12. [20]

    Emanuel Todorov, Tom Erez, and Yuval Tassa

    doi: 10.1109/LRA.2021.3070252. Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026–5033. IEEE,

    2021 doi

  13. [21]

    Mark Towers, Jordan K

    doi: 10.1109/IROS.2012.6386109. Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez-Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Sh...

    2012 doi

  14. [22]

    URLhttps://doi.org/10.24963/ijcai.2024/913

    doi: 10.24963/ijcai.2024/913. URLhttps://doi.org/10.24963/ijcai.2024/913. Survey Track. Nolan C. Wagener, Byron Boots, and Ching-An Cheng. Safe reinforcement learning using advantage-based intervention. InProceedings of the 38th International Conference on Machine Learning (IC...

    2024 doi

  15. [23]

    Kevin Zakka, Yuval Tassa, and MuJoCo Menagerie Contributors

    doi: 10.1109/IROS47612.2022.9982038. Kevin Zakka, Yuval Tassa, and MuJoCo Menagerie Contributors. MuJoCo Menagerie: A collection of high-quality models for the MuJoCo physics engine,

    2022 doi

  16. [24]

    URLhttps://doi.org/10.24963/ijcai.2023/763

    doi: 10.24963/ijcai.2023/763. URLhttps://doi.org/10.24963/ijcai.2023/763. Survey Track. A Full Proofs A.1 Proof of Theorem 1 (unbiased gradient ofJmix(θ)) Proof.Starting from the definition,J mix(θ) = ∫ pmix θ r(τ)dτ.Under Assumption 1, differentiation and in- tegration commut...

    2023 doi

  17. [25]

    Clipping bounds variance downstream of the singularity rather than removing it

    reach the mixed-policy gradient by clipping the importance ratiopθ/pmix θ , but that ratio is still undefined at the deterministic-recovery steps whereµis a point mass. Clipping bounds variance downstream of the singularity rather than removing it. The factor-level proof above...

    2002

  18. [26]

    or of demonstrations (Nair et al., 2018). Proof.(a)From Equation (9) withwt =σseg(t), whereseg(t)is the index of the recovery segment containing t(Section 4.4), and the assumptionσk≡1, Lcompat σ≡1(θ) =−λcompat Nrec ∑ t:st/∈R 1·logπθ(arec t |st) =−λcompat Nrec ∑ t:st/∈R logπθ(π...

    2018

  19. [27]

    or of demonstrations (Nair et al., 2018). A.7 Sketch: why an observable upper bound onβis hard in continuous actions The bound onβ(θ,d)above is stated in terms of the per-step invariance slackη, which is not directly measurable; a bound in terms of an observable quantity would...

    2018

  20. [28]

    is in use: •Analytic V on:train the critic onalltimesteps. Recovery-state targets under Theorem 6 carry no Monte-Carlo variance and are exact over the segment; the only residual error is the critic’s own error at the single re-entry state (failure-segment targets are exact), s...

    2021

  21. [29]

    CPO and PPO-Lagrangian additionally carry the constraint hyperparameters their objective requires (Section C.3)

    Symbol in Algorithm 1 Default value N(total updates)total_timesteps/(num_envs×num_steps) E(parallel envs) 4 Tro (rollout length) 2048 K(PPO epochs) 10 cv (value coef) 0.5 ce (entropy coef) 0 λcompat (compat coef)10 −3 ϵ(PPO clip) 0.2 γ,GAEλ0.99, 0.95 rterm (terminal reward) en...

  22. [30]

    Per-environmentdmax values are in env-specific task-space units, not comparable across environments. Parameter V alue Optimizer Adam (eps = 10 −5) Learning rate 3×10−4 Anneal LR linearly to 0 yes Number of parallel envs 4 Rollout length per env 2048 Batch size (num_envs×num_st...

  23. [31]

    with a custom 12-DoF position-actuator configuration and a velocity-tracking reward adapted fromlegged_gym(Rudin et al., 2021). E Additional Results Run-batch provenance.The sensitivity studies in this appendix come from dedicated sweep batches run independently of the Table 1...

    2021

  24. [32]

    global”). An alternative “per-segment

    analyzes the clean ablation ladder Unmasked PG→masked learned V→analyticV→hardC θ(τ)(SafeExplorer); Table 8 tabulates its full numbers. The two soft-gate variants, a soft re-entry-advantage gatewsoft t (Equation (14)) applied on the analytic-Vmodel, and the same soft gate appl...

  25. [33]

    decomposes as ˆgIS = ∑ t:st∈R ∇θlogπθ(at|st)At    ˆgmasked + ∑ t:st/∈R wt∇θlogπθ(at|st)At,(15) withA t the PPO advantage estimate. The second sum would be mean zero by the score-function identity only ifAt were replaced by an action-independent baselineb(st); with the actu...

    2018

  26. [34]

    confirms the cost (Figure 8). These runs use the sampled-recovery setup of this appendix, so their absolute rewards are not comparable to Table 1’s greedy-recovery numbers; only the within-setup masked-vs-IS differences are meaningful. OnHalfCheetah the weights die immediately...

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.