Penalized Proximal Policy Optimization for Safe Reinforcement Learning
read the original abstract
Safe reinforcement learning aims to learn the optimal policy while satisfying safety constraints, which is essential in real-world applications. However, current algorithms still struggle for efficient policy updates with hard constraint satisfaction. In this paper, we propose Penalized Proximal Policy Optimization (P3O), which solves the cumbersome constrained policy iteration via a single minimization of an equivalent unconstrained problem. Specifically, P3O utilizes a simple-yet-effective penalty function to eliminate cost constraints and removes the trust-region constraint by the clipped surrogate objective. We theoretically prove the exactness of the proposed method with a finite penalty factor and provide a worst-case analysis for approximate error when evaluated on sample trajectories. Moreover, we extend P3O to more challenging multi-constraint and multi-agent scenarios which are less studied in previous work. Extensive experiments show that P3O outperforms state-of-the-art algorithms with respect to both reward improvement and constraint satisfaction on a set of constrained locomotive tasks.
This paper has not been read by Pith yet.
Forward citations
Cited by 4 Pith papers
-
Seeing Before Colliding: Anticipatory Safe RL with Frozen Vision-Language Models
VLM-Safe-RL adds frozen VLM signals as anticipatory costs to the CMDP Lagrangian update via dual-path CLIP, VLM-Lagrange, and confidence gating, outperforming baselines on Safety-Gymnasium FormulaOne while showing par...
-
CRAX: Fast Safe Reinforcement Learning Benchmarking
CRAX is a new fast benchmark suite for constrained RL built on MJX, with six environment suites and tasks across difficulty levels, showing no single safe RL method dominates and benefits from curriculum learning.
-
PPO-EAL: Exact Augmented Lagrangian Proximal Policy Optimization for Safe Robotic Control
PPO-EAL integrates exact augmented Lagrangian optimization into PPO for safe robotic control, with claimed theoretical guarantees and better empirical safety-performance tradeoffs on several robot benchmarks including...
-
MUJICA: Multi-skill Unified Joint Integration of Control Architecture for Wheeled-Legged Robots
A single reinforcement learning policy jointly trains multiple locomotion skills for wheeled-legged robots with DC-motor constraints and learns a proprioceptive skill selector for adaptive behavior.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.