Pith. sign in

REVIEW 2 major objections 5 minor 39 references

This paper claims that a smoothly saturating penalty on constraint cost, applied inside the safe region as an incentive, lets a PPO-style constrained RL algorithm find exact constrained optima and lower violation rates.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 19:29 UTC pith:5P23W4EU

load-bearing objection The main optimality theorem is false (invalid inequality in Lemma 4); the algorithm is a plausible empirical variant but the paper needs major revision. the 2 major comments →

arxiv 2509.09208 v1 pith:5P23W4EU submitted 2025-09-11 cs.LG cs.AI

Incentivizing Safer Actions in Policy Optimization for Constrained Reinforcement Learning

Somnath Hazra , Pallab Dasgupta , Soumyajit Dey This is my paper
classification cs.LG cs.AI
keywords constrained reinforcement learningCMDPsafetypolicy optimizationpenalty functionCELUproximal policy optimizationLagrangian relaxation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to show that constrained reinforcement learning can be made both safer and no less reward-optimal by replacing hard constraint handling with a smooth, saturating penalty built from the constraint cost. The authors propose IP3O, a PPO-style algorithm whose loss subtracts a CELU-shaped incentive when the cost constraint is satisfied and gradually converts it into a penalty as the constraint boundary is crossed. They argue this smooth transition stabilizes training near the boundary and avoids both risky shortcuts and overly conservative detours. Their central theoretical claim is that, when the penalty weight is at least the infinity norm of the optimal Lagrange multiplier, the limit of policies minimizing the penalized loss is also an exact solution to the original constrained problem.

Core claim

The paper's central discovery is that a particular activation function, CELU, when used as a penalty on the cost constraint's surrogate loss, creates an incentive zone inside the safe region and a smooth penalty zone outside it. The authors prove (Theorem 1) that under Slater's condition and with η ≥ ||λ*||∞, the minimizers of the combined loss L_R + η Σ CELU(L_Ci) coincide with the solutions of the original CMDP, so the incentive does not distort the optimum. Theorem 2 bounds the worst-case suboptimality from using the current policy's advantage estimates and the truncated penalty, giving a finite error in terms of KL divergence and advantage magnitudes. Empirically, IP3O is compared agains

What carries the argument

The CELU function—CELU(x,α)=x for x≥0 and α(exp(x/α)−1) for x<0—is the load-bearing component. Its negative branch saturates at −α, creating a bounded incentive for staying strictly inside the feasible region and a smooth, continuously differentiable transition into penalty outside. This saturating shape is what the authors claim prevents overly conservative policies while still guiding the policy away from the boundary. The other key piece is the importance-weighted surrogate loss from PPO, extended to cost advantages, with the penalty applied per-constraint and summed into the total loss.

Load-bearing premise

The load-bearing premise is that the penalty term η·CELU(x) dominates λ_i·x for every possible constraint slack x; the supplied proof only shows CELU(x) ≥ −α, which is too weak to guarantee this domination, so the optimality-equivalence theorem is not proven by the argument given.

What would settle it

Compute the pointwise gap: for a constraint with Lagrange multiplier λ_i and penalty weight η, find any slack value x where η·CELU(x) < λ_i·x. For instance, α=1, η=3, λ_i=1, x=−0.21 yields 3·CELU(−0.21) ≈ −0.57 < −0.21, violating the inequality. Such a counterexample would show the proof of Lemma 4 fails, and if the corresponding policy exists it would contradict Theorem 1's equivalence claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If Theorem 1 holds, safe RL algorithms can use a single scalar penalty weight η without a learned Lagrange multiplier and still reach the exact constrained optimum, removing dual oscillation.
  • The smooth incentive region means policies are proactively steered away from constraint boundaries rather than only reacting after violation, which should reduce boundary noise during training.
  • The worst-case error bound in Theorem 2 gives a concrete guarantee on suboptimality when the cost surrogate is approximated, enabling safe deployment with known performance degradation.
  • The method's extension to multi-agent cooperative driving shows the incentive principle is not limited to single-agent control and could apply to other multi-agent safety tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A stronger version of the theorem might hold for any penalty whose negative branch lies above the line λ_i·x, not just CELU; this would make the design principle transferable to other saturating activations like GELU or softplus.
  • The incentive region could be tuned per constraint rather than globally, weighting safety of the most critical constraints more heavily; the paper's η is a single scalar, but a per-constraint η_i would be a natural extension.
  • The empirical claim that IP3O reduces violations without sacrificing return could be tested on safety benchmarks with tight budgets, where a conservative bias would be most visible.
  • If the proof gap for Lemma 4 is filled, the algorithm's theoretical guarantee would be genuinely parameter-free in η; otherwise, the practical method still works but lacks the advertised optimality certificate.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes IP3O, a constrained RL algorithm that augments the PPO loss with a CELU penalty on the constraint slack (Eq. 6). CELU provides a bounded incentive for strictly feasible policies and turns into a linear penalty after violation. The authors claim that for η ≥ ||λ*||∞ the penalized loss shares the same optimum as the original constrained problem (Theorem 1), and derive a worst-case optimality error bound (Theorem 2). They report experiments on MuJoCo Safety Velocity, Safety Gymnasium, Bullet Safety Gymnasium, and MetaDrive.

Significance. If the theoretical claims held, IP3O would offer a simple smooth penalty with an equivalence guarantee and broad empirical support. The empirical component is genuinely broad: multiple single-agent and multi-agent benchmark suites, baselines adapted from OmniSafe, and ablations on α and d, with code and supplementary material provided. However, the central equivalence is false: Lemma 4's key inequality fails for negative slack, and a one-dimensional example contradicts Theorem 1. Theorem 2 is proved for an h-clipped loss that the authors state they did not use. Consequently, the paper's advertised theoretical guarantees are not established; the remaining contribution is an empirical study of a heuristic penalty.

major comments (2)
  1. [Supplementary A.1, Lemma 4] Lemma 4's proof asserts for L_Ci<0 that η CELU(L_Ci) ≥ λ_i^* L_Ci, citing CELU(x)≥−α. This implication is invalid. From CELU(x)≥−α one only gets η CELU(x)≥−ηα; since λ_i x can be arbitrarily close to 0 from below, the claimed inequality fails. Example: α=1, η=3, λ_i=1, x=−0.21: CELU(x)=e^{−0.21}−1≈−0.189, so η CELU(x)≈−0.568<−0.21=λ_i x. More generally, for x→0−, CELU(x)=x+O(x^2), so the inequality becomes η x≥λ_i x, false for every η>λ_i. Hence the negative-slack branch of Lemma 4 collapses and Theorem 1 is not proved. The failure is not merely a proof gap: for scalar problem max x subject to x≤0 (so λ*=1), Eq. (6) with α=1, η=3 is minimized at x=ln(1/3)≈−1.099, a strictly feasible point, not at the constrained optimum x=0. Thus the claimed equivalence in Theorem 1 is false as stated.
  2. [Theorem 2 and Section 4.2] Theorem 2's error bound contains |α log(h)|, but h appears only in the clipped variant max(CELU(·),−α(1−h)). The authors explicitly state in Section 4.2 that this clipping was not required in practice, and Algorithm 1 uses the unclipped Eq. (6). Therefore Theorem 2 is a bound for a different algorithm; for the actual loss the derivation of the |α log(h)| term does not apply, and the residual CELU gradient continues below −α(1−h). The proof also refers to 'Lemma 2' (the intended reference is Lemma 6). This leaves the paper without a worst-case guarantee for the evaluated algorithm.
minor comments (5)
  1. [Figure 2 caption] Typo: 'Comaprison' should be 'Comparison'.
  2. [Supplementary A] Typo: 'dicuss' should be 'discuss'.
  3. [Equation (3) and Algorithm 1] The state distribution is written as d^π in Eq. (3) but as d^πk in the surrounding text and algorithm; the notation should be made consistent.
  4. [Theorem 1 statement] The condition η ≥ ||λ*||∞ involves the unknown optimal Lagrange multiplier; the paper should state explicitly whether this is a non-constructive existence guarantee and how the chosen η=20 relates to λ*.
  5. [Lemma 5] The proof silently assumes ¯π is feasible for Eq. (10) at the start; this assumption should be stated explicitly.

Circularity Check

0 steps flagged

No significant circularity: The theoretical claims are conditional equivalence and error-bound statements, not definitions; the main concern is a proof gap in Lemma 4, which is a correctness issue rather than circular reasoning.

full rationale

The paper's derivation is not circular. Theorem 1 is a conditional equivalence claim: it asserts that if the penalty coefficient η dominates the optimal dual multiplier λ* of the original constrained problem, then the minimizer of the penalized loss (Eq. 6/11) is also a minimizer of the surrogate constrained problem (Eq. 4/10). The condition references the dual solution but is an external hypothesis, not a quantity fitted from the algorithm's own outputs; the proof attempts to compare solution sets via Lemmas 4 and 5 rather than defining the solution into existence. Theorem 2's error bound is derived from the performance-difference lemma (Lemma 6, citing Achiam et al.) and from the Lipschitz property of CELU; the appearance of the algorithm's own hyperparameters η, α, h in the bound is ordinary for an optimization error bound and does not amount to renaming a fitted quantity as a prediction. There are no self-citations, no imported uniqueness theorems, and no ansatz smuggled in from the authors' prior work; the CELU choice is attributed to Barron (2017) and Clevert (2015). The main substantive concern is a correctness gap in the supplementary proof of Lemma 4: the negative-slack branch claims η·CELU(x) ≥ λ_i·x from CELU(x) ≥ −α, which does not follow (for η > λ_i and x < 0 near 0, the inequality is false). That is an invalid proof step, not a circular definition, so it does not raise the circularity score.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The method's core novelty is the CELU penalty, whose two free parameters (α, η) must be tuned per environment; the claimed theoretical guarantee additionally depends on the unknown Lagrange multiplier λ* and the unused clipping threshold h. No new entities are proposed. The proof of Theorem 1 relies on an inequality that is false, which is captured as an axiom failure.

free parameters (3)
  • η (penalty factor) = 20.0
    Hyper-parameter in loss (6). Theorem 1 requires η ≥ ||λ*||_∞, but λ* is unknown, so η is set by hand; no sensitivity analysis beyond the fixed value.
  • α (CELU slope) = 0.5 (MuJoCo Velocity); 0.1 (Safety Gym); 1.0 (Bullet Safety Gym)
    Controls the incentive floor and safety-reward trade-off. Tuned per environment; ablations in §5.5 show α strongly affects violation rate and reward.
  • h (stagnation clipping) = not used in practice
    Mentioned in §4.2 and used in Theorem 2's bound (|α log h|), but authors state practical runs did not require it; the theorem's bound therefore applies to a modified variant, not the evaluated algorithm.
axioms (4)
  • standard math Performance difference lemma (Kakade-Langford)
    Used to derive surrogate objectives (Eq. 2-4) and Lemma 3 in the supplementary material.
  • domain assumption Slater's condition and strong duality for the CMDP
    Invoked in Theorem 1; may not hold for all continuous-control CMDPs with learned approximations, and no verification is provided.
  • standard math CELU is 1-Lipschitz and CELU(x) ≥ x
    Used in the proof of Theorem 2 to bound |CELU(x)-CELU(y)| by |x-y|; the property CELU(x) ≥ x is not sufficient to prove Lemma 4, which is a key logical gap.
  • domain assumption Convergence of the iterative policy update to a limit π*
    Theorems 1 and 2 assume the sequence {π_k} from Algorithm 1 converges; no proof of convergence is given.

pith-pipeline@v1.3.0-alltime-deepseek · 13260 in / 17031 out tokens · 166280 ms · 2026-08-04T19:29:58.152406+00:00 · methodology

0 comments
read the original abstract

Constrained Reinforcement Learning (RL) aims to maximize the return while adhering to predefined constraint limits, which represent domain-specific safety requirements. In continuous control settings, where learning agents govern system actions, balancing the trade-off between reward maximization and constraint satisfaction remains a significant challenge. Policy optimization methods often exhibit instability near constraint boundaries, resulting in suboptimal training performance. To address this issue, we introduce a novel approach that integrates an adaptive incentive mechanism in addition to the reward structure to stay within the constraint bound before approaching the constraint boundary. Building on this insight, we propose Incrementally Penalized Proximal Policy Optimization (IP3O), a practical algorithm that enforces a progressively increasing penalty to stabilize training dynamics. Through empirical evaluation on benchmark environments, we demonstrate the efficacy of IP3O compared to the performance of state-of-the-art Safe RL algorithms. Furthermore, we provide theoretical guarantees by deriving a bound on the worst-case error of the optimality achieved by our algorithm.

Figures

Figures reproduced from arXiv: 2509.09208 by Pallab Dasgupta, Somnath Hazra, Soumyajit Dey.

Figure 1
Figure 1. Figure 1: Illustration of a risky trajectory and a conservation trajec [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comaprison of activation functions and their gradients. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The loss function for the reward is derived [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 3
Figure 3. Figure 3: The overall algorithm describing the loss function for [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison with the baselines using the MuJoCo safety velocity scenarios. Our method is marked as IP3O (blue). The dashed line [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison with the baselines using the Safety Gymnasium scenarios. Our method is marked as IP3O (blue). The dashed line [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison with the baselines using the Bullet Safety Gymnasium scenarios. Our method is marked as IP3O (blue). The dashed [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Evaluation results on the MetaDrive simulator [PITH_FULL_IMAGE:figures/full_fig_p006_8.png] view at source ↗
Figure 7
Figure 7. Figure 7: Summary of constraint violations across all environments. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 10
Figure 10. Figure 10: Ablation experiments using the cost limit [PITH_FULL_IMAGE:figures/full_fig_p007_10.png] view at source ↗
Figure 9
Figure 9. Figure 9: Ablation experiments using α on the Half-Cheetah and the Humanoid environments. The effect of α is studied in the Half-Cheetah and Hu￾manoid tasks from the MuJoCo Safety Velocity benchmark. For these experiments, the cost limit d is fixed at 25. As shown in [PITH_FULL_IMAGE:figures/full_fig_p007_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 6 linked inside Pith

  1. [1]

    Constrained policy optimiza- tion

    [Achiamet al., 2017 ] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimiza- tion. InInternational Conference on Machine Learning, pages 22–31. PMLR,

    2017

  2. [2]

    We also conducted experi- ments using the MetaDrive simulator [Liet al., 2022 ]

    B Empirical Details We showed evaluation results across three widely-used safe RL environments: MuJoCo Safety Velocity [Jiet al., 2023 ], Safety Gymnasium [Rayet al., 2019 ], and Bullet Safety Gymnasium [Gronauer, 2022 ]. We also conducted experi- ments using the MetaDrive simulator [Liet al., 2022 ]. All the environments can be defined using a CMDP. B.1 ...

    2023

  3. [4]

    Risk-constrained reinforcement learning with percentile risk criteria.Journal of Machine Learning Research, 18(167):1–51,

    [Chowet al., 2018 ] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria.Journal of Machine Learning Research, 18(167):1–51,

    2018

  4. [9]

    A general safety framework for learning-based control in uncertain robotic systems.IEEE Transactions on Automatic Control, 64(7):2737–2752,

    [Fisacet al., 2018 ] Jaime F Fisac, Anayo K Akametalu, Melanie N Zeilinger, Shahab Kaynama, Jeremy Gillula, and Claire J Tomlin. A general safety framework for learning-based control in uncertain robotic systems.IEEE Transactions on Automatic Control, 64(7):2737–2752,

    2018

  5. [10]

    Exterior penalty pol- icy optimization with penalty metric network under con- straints

    [Gaoet al., 2024 ] Shiqing Gao, Jiaxin Ding, Luoyi Fu, Xin- bing Wang, and Chenghu Zhou. Exterior penalty pol- icy optimization with penalty metric network under con- straints. InProceedings of the Thirty-Third Interna- tional Joint Conference on Artificial Intelligence, IJCAI- 24,

    2024

  6. [13]

    Omnisafe: An infrastructure for accelerating safe reinforcement learn- ing research.Journal of Machine Learning Research, 25(285):1–6,

    [Jiet al., 2024 ] Jiaming Ji, Jiayi Zhou, Borong Zhang, Jun- tao Dai, Xuehai Pan, Ruiyang Sun, Weidong Huang, Yi- ran Geng, Mickel Liu, and Yaodong Yang. Omnisafe: An infrastructure for accelerating safe reinforcement learn- ing research.Journal of Machine Learning Research, 25(285):1–6,

    2024

  7. [14]

    Doubly ro- bust off-policy value evaluation for reinforcement learn- ing

    [Jiang and Li, 2016] Nan Jiang and Lihong Li. Doubly ro- bust off-policy value evaluation for reinforcement learn- ing. InInternational Conference on Machine Learning, pages 652–661. PMLR,

    2016

  8. [17]

    End-to-end training of deep visuomotor policies.Journal of Machine Learning Re- search, 17(39):1–40,

    [Levineet al., 2016 ] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies.Journal of Machine Learning Re- search, 17(39):1–40,

    2016

  9. [18]

    Metadrive: Composing diverse driving scenarios for generalizable re- inforcement learning.IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 45(3):3461–3475,

    [Liet al., 2022 ] Quanyi Li, Zhenghao Peng, Lan Feng, Qi- hang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable re- inforcement learning.IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 45(3):3461–3475,

    2022

  10. [19]

    Ipo: Interior-point policy optimization under constraints

    [Liuet al., 2020 ] Yongshuai Liu, Jiaxin Ding, and Xin Liu. Ipo: Interior-point policy optimization under constraints. InProceedings of the AAAI Conference on Artificial Intel- ligence, volume 34, pages 4940–4947,

    2020

  11. [20]

    The law and ethics of high-frequency trading.Minn

    [McNamara, 2016] Steven McNamara. The law and ethics of high-frequency trading.Minn. JL Sci. & Tech., 17:71,

    2016

  12. [21]

    Chance-constrained dynamic pro- gramming with application to risk-aware robotic space ex- ploration.Autonomous Robots, 39:555–571,

    [Onoet al., 2015 ] Masahiro Ono, Marco Pavone, Yoshiaki Kuwata, and J Balaram. Chance-constrained dynamic pro- gramming with application to risk-aware robotic space ex- ploration.Autonomous Robots, 39:555–571,

    2015

  13. [22]

    Benchmarking safe exploration in deep rein- forcement learning.arXiv preprint arXiv:1910.01708, 7(1):2,

    [Rayet al., 2019 ] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep rein- forcement learning.arXiv preprint arXiv:1910.01708, 7(1):2,

    Pith/arXiv arXiv 2019

  14. [24]

    Trust region policy opti- mization.arXiv preprint arXiv:1502.05477,

    [Schulman, 2015] John Schulman. Trust region policy opti- mization.arXiv preprint arXiv:1502.05477,

    Pith/arXiv arXiv 2015

  15. [25]

    Responsive safety in reinforcement learn- ing by pid lagrangian methods

    [Stookeet al., 2020 ] Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learn- ing by pid lagrangian methods. InInternational Con- ference on Machine Learning, pages 9133–9143. PMLR,

    2020

  16. [26]

    Value-decomposition networks for cooperative multi-agent learning.arXiv preprint arXiv:1706.05296,

    [Sunehaget al., 2017 ] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning.arXiv preprint arXiv:1706.05296,

    Pith/arXiv arXiv 2017

  17. [27]

    MIT press,

    [Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto.Reinforcement learning: An introduction. MIT press,

    2018

  18. [28]

    Reward constrained policy optimization

    [Tessleret al., 2018 ] Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. International Conference on Learning Representations,

    2018

  19. [29]

    Projection- based constrained policy optimization.International Conference on Learning Representations,

    [Yanget al., 2020 ] Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection- based constrained policy optimization.International Conference on Learning Representations,

    2020

  20. [30]

    Constrained update projection approach to safe policy optimization.Advances in Neural Information Pro- cessing Systems, 35:9111–9124,

    [Yanget al., 2022 ] Long Yang, Jiaming Ji, Juntao Dai, Lin- rui Zhang, Binbin Zhou, Pengfei Li, Yaodong Yang, and Gang Pan. Constrained update projection approach to safe policy optimization.Advances in Neural Information Pro- cessing Systems, 35:9111–9124,

    2022

  21. [31]

    Convergent policy optimization for safe reinforcement learning.Advances in Neural Information Processing Systems, 32,

    [Yuet al., 2019 ] Ming Yu, Zhuoran Yang, Mladen Kolar, and Zhaoran Wang. Convergent policy optimization for safe reinforcement learning.Advances in Neural Information Processing Systems, 32,

    2019

  22. [32]

    Reinforcement learning in healthcare: A survey.ACM Computing Surveys (CSUR), 55(1):1–36,

    [Yuet al., 2021 ] Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey.ACM Computing Surveys (CSUR), 55(1):1–36,

    2021

  23. [33]

    The surprising effectiveness of ppo in cooperative multi-agent games.Advances in Neural Information Processing Sys- tems, 35:24611–24624,

    [Yuet al., 2022 ] Chao Yu, Akash Velu, Eugene Vinitsky, Ji- axuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games.Advances in Neural Information Processing Sys- tems, 35:24611–24624,

    2022

  24. [34]

    First order constrained optimization in policy space.Advances in Neural Information Processing Sys- tems, 33:15338–15349,

    [Zhanget al., 2020 ] Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space.Advances in Neural Information Processing Sys- tems, 33:15338–15349,

    2020

  25. [35]

    Penalized proximal policy optimization for safe rein- forcement learning

    [Zhanget al., 2022 ] Linrui Zhang, Li Shen, Long Yang, Shixiang Chen, Bo Yuan, Xueqian Wang, and Dacheng Tao. Penalized proximal policy optimization for safe rein- forcement learning. InInternational Joint Conference on Artificial Intelligence,

    2022

  26. [36]

    Evaluating model-free reinforcement learning toward safety-critical tasks

    [Zhanget al., 2023 ] Linrui Zhang, Qin Zhang, Li Shen, Bo Yuan, Xueqian Wang, and Dacheng Tao. Evaluating model-free reinforcement learning toward safety-critical tasks. InProceedings of the AAAI Conference on Artifi- cial Intelligence, volume 37, pages 15313–15321,

    2023

  27. [37]

    A.1 Proof of Theorem 1 Before discussing the proof of the Theorem, we mention the performance difference lemma for expressing the perfor- mance bound over policy improvement. Lemma 3( [Kakade and Langford, 2002 ]).Given a reward functionR, for any two policiesπandπ ′ and any start state distributionρ, J π′ R − Jπ R = 1 1−γ Es∼dπ′ a∼π′ [Aπ R(s, a)](8) In g...

    2002

  28. [38]

    A.2 Proof of Theorem 2 Before discussing the proof for Theorem 2 we discuss the following lemma for defining the limits on performance dif- ference given that the trajectories for policy optimization are sampled from the current policy,π k. Lemma 6( [Achiamet al., 2017 ]).For any reward function, R, and policiesπandπ ′, letε π′ R = maxs |Ea∼π′[Aπ R(s, a)]...

    2017

  29. [2002]

    Deep reinforcement learning for autonomous driving: A survey.IEEE Transactions on In- telligent Transportation Systems, 23(6):4909–4926,

    [Kiranet al., 2021 ] B Ravi Kiran, Ibrahim Sobh, Victor Tal- paert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yoga- mani, and Patrick P´erez. Deep reinforcement learning for autonomous driving: A survey.IEEE Transactions on In- telligent Transportation Systems, 23(6):4909–4926,

    2021

  30. [2015]

    Augmented proximal policy op- timization for safe reinforcement learning

    [Daiet al., 2023 ] Juntao Dai, Jiaming Ji, Long Yang, Qian Zheng, and Gang Pan. Augmented proximal policy op- timization for safe reinforcement learning. InProceed- ings of the AAAI Conference on Artificial Intelligence, vol- ume 37, pages 7288–7295,

    2023

  31. [2016]

    Approximately optimal approximate reinforcement learning

    [Kakade and Langford, 2002] Sham Kakade and John Lang- ford. Approximately optimal approximate reinforcement learning. InProceedings of the Nineteenth International Conference on Machine Learning, pages 267–274,

    2002

  32. [2017]

    Routledge,

    [Altman, 2021] Eitan Altman.Constrained Markov decision processes. Routledge,

    2021

  33. [2018]

    Fast and accurate deep network learning by exponential linear units (elus).arXiv preprint arXiv:1511.07289,

    [Clevert, 2015] Djork-Arn´e Clevert. Fast and accurate deep network learning by exponential linear units (elus).arXiv preprint arXiv:1511.07289,

    Pith/arXiv arXiv 2015

  34. [2019]

    Prox- imal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    [Schulmanet al., 2017 ] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv 2017

  35. [2020]

    Trustworthy artificial intelligence requirements in the autonomous driving domain.Computer, 56(2):29–39,

    [Fernandez-Llorca and G´omez, 2023] David Fernandez- Llorca and Emilia G ´omez. Trustworthy artificial intelligence requirements in the autonomous driving domain.Computer, 56(2):29–39,

    2023

  36. [2021]

    Continuously dif- ferentiable exponential linear units.arXiv preprint arXiv:1704.07483,

    [Barron, 2017] Jonathan T Barron. Continuously dif- ferentiable exponential linear units.arXiv preprint arXiv:1704.07483,

    Pith/arXiv arXiv 2017

  37. [2022]

    Safety gymna- sium: A unified safe reinforcement learning benchmark

    [Jiet al., 2023 ] Jiaming Ji, Borong Zhang, Jiayi Zhou, Xue- hai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yi- fan Zhong, Josef Dai, and Yaodong Yang. Safety gymna- sium: A unified safe reinforcement learning benchmark. InThirty-seventh Conference on Neural Information Pro- cessing Systems Datasets and Benchmarks Track,

    2023

  38. [2023]

    Natural policy gradient primal-dual method for constrained markov decision pro- cesses.Advances in Neural Information Processing Sys- tems, 33:8378–8390,

    [Dinget al., 2020 ] Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovic. Natural policy gradient primal-dual method for constrained markov decision pro- cesses.Advances in Neural Information Processing Sys- tems, 33:8378–8390,

    2020

  39. [2024]

    Bullet-safety-gym: A framework for constrained reinforcement learning

    [Gronauer, 2022] Sven Gronauer. Bullet-safety-gym: A framework for constrained reinforcement learning

    2022