Pith. sign in

REVIEW 5 major objections 6 minor 33 references

Proactive Constrained Policy Optimization with Preemptive Penalty

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PCPO claims that embedding a log-barrier penalty into the safe-RL objective, instead of penalizing violations after they happen, provably reduces constraint violations and stabilizes training, with a duality-gap bound that shrinks as the…

desk verdict Promising algorithm, broken theory: the empirical stability of the barrier-plus-intrinsic-reward recipe deserves a look, but the 'provably fewer violations' claim does not follow from the proofs. read the letter →

arxiv 2508.01883 v2 pith:FQSDQZFN submitted 2025-08-03 cs.LG

Ning Yang , Pengyu Wang , Guoqing Liu , Haifeng Zhang , Pin Lv , Jun Wang This is my paper · ORCID
classification cs.LG
keywords safereinforcementlearningconstrainedpolicyoptimizationpreemptivepenaltylog-barrierfunctiondualitygapintrinsicrewardMarkovdecisionprocess
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Safe reinforcement learning usually enforces constraints by punishing violations after they occur, which can make the policy oscillate and overshoot. This paper proposes PCPO, which instead embeds an extended log-barrier term into the optimization objective so the policy feels a penalty before it reaches the constraint boundary, and adds a constraint-aware intrinsic reward that activates only near the boundary. The paper proves that the barrier formulation has a duality gap bounded by $m/\tau + \eta \sum_i I_i^{\max}$, that each PCPO update improves the objective by at least an explicit lower bound, and that cumulative constraint violations stay below those of Lagrangian methods by a gap that grows without bound as training continues. In the paper's experiments on velocity and navigation tasks, PCPO reports higher reward with lower and more stable cost than the compared safe-RL baselines.

What carries the argument

The load-bearing object is the extended log-barrier function $\varphi_\tau(g) = -\frac{1}{\tau}\log(-g)$ for $g \leq -1/\tau^2$, continued linearly as $\tau g - \frac{1}{\tau}\log(1/\tau^2)+\frac{1}{\tau}$ for $g > -1/\tau^2$. Its derivative is strictly positive as the constraint slack approaches zero from below, so the policy is pushed away from the boundary before violating; the same derivative yields the implicit dual variable $\lambda^*_i=\varphi'_\tau(g_{C_i}(\pi^*))$. The constraint-aware intrinsic reward $I^{\pi}_{C_i}$ gates a normalized cost advantage by a sigmoid activated near the boundary, and the practical update is a natural-gradient step under a KL trust region using a regularized Fisher information matrix.

What would settle it

Run PCPO on a small constrained MDP where the exact argmax of Eq. (9) under the KL constraint can be computed, and compare the implemented update of Eq. (21) with that argmax; if on any such instance the empirical cumulative constraint violation over a long horizon fails to stay below a Lagrangian baseline, then Proposition 4.4's unbounded gap does not hold for the implemented algorithm.

Watch

Extended reading notes

Core claim

The central claim is that a preemptive log-barrier penalty can enforce safety in constrained policy optimization without sacrificing performance, where Lagrangian methods cannot. Concretely, PCPO maximizes $G(\pi_\theta)=f(\pi_\theta)-\sum_i \varphi_\tau(g_{C_i}(\pi_\theta))+\eta \sum_i I^{\pi_\theta}_{C_i}$, with $\varphi_\tau$ an extended log-barrier that is active inside the feasible region as $g_{C_i}$ approaches zero, and $I^{\pi_\theta}_{C_i}$ a gated intrinsic reward. Because the barrier derivative is strictly positive near the boundary, the update feels a repulsive force before violation; that derivative acts as an implicit dual variable, and Theorem 4.2 bounds the duality gap by $m/\tau + \eta \sum_i I_i^{\max}$. Theorem 4.3 gives a lower bound on the per-iteration improvement of $G$, and Proposition 4.4 asserts the cumulative-violation gap over Lagrangian methods grows to infinity with the number of iterations.

Load-bearing premise

The proofs analyze the exact argmax of the barrier objective under the KL constraint, while the implemented algorithm takes a single natural-gradient step on a linear-quadratic approximation; the paper does not bound the error of that approximation, so the theorems apply to the deployed method only if the approximation is accurate.

Editorial extensions

If this is right

  • The nonzero barrier gradient inside the feasible region means PCPO does not stall at the constraint boundary, so training should show fewer oscillations than Lagrangian or post-hoc penalty methods.
  • Larger $\tau$ shrinks the duality-gap upper bound toward $m/\tau + \eta \sum_i I_i^{\max}$, giving a principled trade-off between constraint tightness and closeness to the true constrained optimum.
  • Theorem 4.3's explicit lower bound on per-iteration improvement provides an update acceptance test: an observed improvement below the bound signals that the trust-region or barrier parameter needs adjustment.
  • Proposition 4.4 implies that the cumulative-violation advantage over Lagrangian methods grows with training length, so long-horizon runs should show PCPO costs stabilizing below the threshold while Lagrangian costs keep oscillating.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The missing bridge between theory and practice is an approximation-error bound for the natural-gradient step; adding one would turn the exact-argmax theorems into guarantees for the executed algorithm.
  • Reading the barrier derivative as an implicit dual variable makes PCPO a primal-dual method whose multiplier is updated by the geometry of the constraint violation, a perspective that could simplify or replace standard multiplier schedules.
  • The gated intrinsic reward is defined per constraint, so it could be applied to multi-constraint problems by letting each constraint drive exploration only when it is the nearest active boundary; the paper does not test this.
  • Proposition 4.4 predicts that Lagrangian methods' average violation does not vanish in the tail of training; a direct test is to run the Lagrangian baseline and PCPO well beyond the paper's 10-30 million sample budgets and compare violation rates.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes Proactive Constrained Policy Optimization (PCPO), a safe RL method that augments the constrained objective with an extended log-barrier term and a constraint-aware intrinsic reward, and updates the policy by a natural-gradient step under a KL trust-region constraint. The authors claim three theoretical results: an upper bound on the duality gap (Theorem 4.2), a lower bound on per-iteration improvement of the augmented objective (Theorem 4.3), and a guarantee that PCPO has asymptotically smaller cumulative constraint violations than Lagrangian safe RL methods (Proposition 4.4). Experiments on Safety Gymnasium velocity and navigation tasks compare PCPO with CUP, EPO, FOCOPS, and TRPOLag, reporting higher reward and more stable cost behavior, along with ablations, sensitivity analyses, and generalization tests.

Significance. If the theoretical claims were valid, the paper would make a useful contribution: the preemptive barrier formulation is a reasonable alternative to Lagrangian methods, and the constraint-aware intrinsic reward is a sensible mechanism for boundary-aware exploration. The experimental study is fairly extensive, including multiple environments, six seeds with bootstrap confidence intervals, ablation studies, sensitivity analyses, and a generalization test. However, the central stated contribution is the theoretical framework, and the proofs contain load-bearing gaps: Proposition 4.1 is circular, Theorem 4.2 bounds a quantity that is not the true duality gap, Theorem 4.3 has algebraic inconsistencies, and Proposition 4.4 does not follow from the preceding lower-bound comparison. Because the headline claim of provably fewer constraint violations is unsupported, the paper's significance as a theoretical contribution is not established, and the empirical results alone are not sufficient to justify publication in its current form.

major comments (5)
  1. [§4.3, Appendix E, Eq. (14) and Eqs. (74)–(80)] Proposition 4.4, the formal statement of the paper's main claim that PCPO provably incurs fewer cumulative constraint violations than Lagrangian methods, is not proved. The proof defines LPCPO and LLagrange as lower bounds on the change of the augmented objective G(π)=f(π)−Σφτ(gCi(π))+ηΣI, then shows LLagrange<LPCPO in Eq. (74), and asserts in Eq. (76) that JC(πP_{k+1})≤JC(πL_{k+1})−δ_k. This implication is a non-sequitur: a comparison of lower bounds on ΔG says nothing about the ordering of raw cumulative costs JC between two different algorithms, since G also contains reward and intrinsic-reward terms. The quantity δ_k is defined as the difference of lower bounds of ΔG, not as a per-step cost gap, and no argument is given that δ_k>0 at every iteration. The asymptotic statement Δ(T)→∞ additionally relies on Eq. (80), liminf_{T→∞}(1/T)VL(T)>0, which is asserted without proof for the compared Lagrangian baselines. Thus the central guarantee of provably fewer violations rests on an unproved, and currently false-looking, chain of inference.
  2. [§4.2, Appendix B, Eq. (34)] Proposition 4.1 is circular. The proof needs to establish that the policy updated with the intrinsic reward has better constraint satisfaction than the policy updated without it, but it assumes this conclusion directly: 'Additionally, the intrinsic reward pushes the policy inward, meaning gCi(πk+1)≤gCi(π̄k+1).' The earlier assumption f(πk+1)≈f(π̄k+1) is also asserted without a bound. These are exactly the facts that would need to be derived from the definitions of the two updates, rather than stipulated. Consequently Eq. (10) of Proposition 4.1 is not proven.
  3. [§4.3, Appendix C, Eqs. (36)–(40)] Theorem 4.2 does not establish an upper bound on the duality gap of the original problem. The proof defines G(λ*) in Eq. (38) using π*, the optimal policy of the primal problem, and sets λ*i=φ'τ(gCi(π*)) in Eq. (36) without showing that this λ* is a feasible dual solution or that G(λ*) is the value of the Lagrangian dual of problem (1). The quantity −Σλ*i gCi(π*)+ηΣE[Iπ*] is then bounded by m/τ+ηmImax, but this is a bound on a specially constructed expression, not on the true Lagrangian dual gap. Moreover, since η=ω·Gmax/Imax+ε, the term ηmImax is essentially fixed by the definition of η, so the claimed dependence of the gap on τ is not the only parametric dependence. The result as stated is therefore not a duality-gap theorem for the PCPO update.
  4. [§5.1, Eqs. (9), (17)–(21)] The theoretical results analyze the exact argmax of G(πθ) under the KL constraint in Eq. (9), but the implemented algorithm updates θ by a single natural-gradient step on a linear objective with a quadratic KL approximation, Eq. (21). No bound is given on the approximation error between the exact solution of Eq. (9) and the one-step update, nor on the effect of the Fisher-matrix regularization Ĥ=H+λI. Theorems 4.2 and 4.3 and Proposition 4.4 therefore do not apply to the evaluated algorithm unless an additional approximation-error bound is supplied. This is a load-bearing gap between theory and implementation, not merely a presentation issue.
  5. [Appendix D, Eqs. (61)–(62) and Theorem 4.3, Eq. (13)] Theorem 4.3's proof contains algebraic inconsistencies that affect the stated lower bound. In Eq. (61), the second case is −τmdi − (√(2δγ)/(1−γ)²) Σ(επk+1 + τ επk_Ci + τ επk+1_Ci), while Eq. (62) and the theorem's second case claim −mdi + η(πk+1) + Σ(ηCi(πk)+ηCi(πk+1)) − ηΣImax. The factor τ on mdi and on the ε-terms disappears without explanation, and the signs of the ηCi terms are inconsistent with their definition as negative quantities. In the first case, Eq. (54) has a log argument of the form 1+..., while the theorem uses log((2−di)/(2ηCi(πk+1))), a different expression. Because the proof does not connect these expressions, the lower bound in Eq. (13) is not established as stated.
minor comments (6)
  1. [§4.3] The text after Proposition 4.4 refers to 'Proposition 4' instead of 'Proposition 4.4'; the appendix heading has the same mismatch.
  2. [Table A3] In the CarCircle1-v0 row, the EPO return is written as '16 , 43 ± 0.58', which appears to be a typo for '16.43 ± 0.58'.
  3. [§4.2, Eq. (8)] The gating function σ(α(δ+gCi(πθ))) and the subsequent text 'Change the triggering condition ... to δ+gCi(πθ)≥0' are unclear: it is not stated how the gating condition in the equation is reconciled with the textual activation condition.
  4. [Algorithm 2, line 10] Algorithm 2 includes a backtracking line search to enforce the KL constraint, while Algorithm 1 and Section 5.1 describe the update as the direct closed-form natural-gradient step in Eq. (21). The relationship between these two update procedures should be clarified.
  5. [Appendix D, Eq. (53)] The Holder-inequality step is written imprecisely: the expectation E[A^{πk+1}_{Ci}] is bounded as if it were a norm of a vector, and the intermediate notation dπk+1−dπk is not defined as a vector here. The intended appeal to Lemma/Corollary 3 of [14] should be stated explicitly.
  6. [Appendix F.4 and Figure A5] The sensitivity analysis for ω reports that performance is robust, but the figure axis labels appear to show only the values 0.1, 0.3, 0.5; the caption and text should state the full range of tested values.

Circularity Check

3 steps flagged · score 6.0 of 10

The central claim of provably fewer violations is constructed rather than derived: Proposition 4.4 defines a lower-bound gap and then asserts it as a per-step cost gap, and Theorem 4.2 names a self-defined barrier derivative a duality gap.

  1. self definitional [Section 4.3, Theorem 4.2; Appendix C, Eqs. (36), (39)-(40).]
    "For convenience, to analyze the optimal implicit Lagrangian dual variables, we only rewrite G (π∗) as G (π∗, λ∗) in Theorem 4.2. ... λ∗ i = φ′ τ (gCi (π∗)) . (36) ... G (λ∗) − J (π∗) = − ... λ∗ i gCi (π∗) + η ... E[I π∗ ]. (39)"

    The quantity called the duality gap is not the gap between the maximum of primal problem (1) and the maximum of the barrier problem (9); it is G(π∗, λ∗) − J(π∗), where λ∗ is defined to be the derivative of the barrier φτ at π∗. With that definition, −λ∗i gCi(π∗) = 1/τ in the logarithmic region and ≤ 1/τ in the linear region by the piecewise definition of φτ, so the m/τ term in Eq. (11) is true by construction. The theorem therefore proves an upper bound on a self-defined expression, not on a duality gap between the two stated optimization problems.

  2. self definitional [Appendix E, Eqs. (72)-(76), Proposition 4.4.]
    "Define the difference between the lower bounds of the two in each update as δk, a positive number: δk = LPCPO − LLagrange > 0. ... Since LPCPO and LLagrange measure the minimum improvement in the objective function ... a larger LPCPO implies that PCPO achieves better control over constraint violations for equivalent reward improvements. Consequently, for any iteration k, the cumulative constraint cost under PCPO’s updated policy πP k+1 must be bounded above by that of the Lagrangian-updated policy πL k+1 minus the gap δk, leading to the relation: JC(πP k+1) ≤ JC(πL k+1) − δk, (76)."

    The proof establishes at most LLagrange < LPCPO, where both quantities are lower bounds on the change of the augmented objective G = f − Σφτ(g) + ηΣI. The proposition to be proved is a per-iteration ordering of the raw cumulative cost JC between PCPO and Lagrangian updates. The gap δk is defined as a difference of lower bounds on G, and the same symbol δk is then carried into Eq. (77) as if it were a per-step violation gap between JC(πP) and JC(πL). No equation connects a larger lower bound on ΔG to a smaller cumulative cost JC, so the conclusion (76) is not derived; it is asserted by identifying a self-defined lower-bound gap with the desired cost gap. Summing (77) to obtain (79) and then claiming Δ(T) → ∞ via the asserted Eq. (80) inherits the same construction.

1 more flagged steps
  1. other [Appendix B, Proof of Proposition 4.1, after Eq. (33).]
    "Since πk+1 is optimized under a richer objective (includes I π Ci), and the trust region ensures both updates are close, we can assume f (πk+1) ≈ f (¯πk+1). Additionally, the intrinsic reward pushes the policy inward, meaning gCi (πk+1) ≤ gCi (¯πk+1)."

    Proposition 4.1 is supposed to show that adding the constraint-aware intrinsic reward provides an additional positive boost to policy optimization. The proof assumes precisely the property to be established: that the intrinsic reward 'pushes the policy inward,' i.e., gCi(πk+1) ≤ gCi(¯πk+1). The preceding algebra does not yield this inequality; it is the intended benefit of the intrinsic reward restated as an assumption. Hence the proposition's conclusion is contained in its proof's assumption, making this an auxiliary circular step.

full rationale

The paper's experimental section is self-contained and compares against external baselines, so the empirical claims are not circular. The circularity is in the theoretical derivation chain. Theorem 4.2 renames the barrier objective evaluated with λ∗ = φ′τ(g(π∗)) as 'the dual,' so the m/τ bound is fixed by the piecewise definition of λ∗ rather than by any primal-dual optimization argument. Proposition 4.4, which is the formal statement of the headline guarantee of provably fewer violations than Lagrangian methods, constructs its per-iteration gap δk as the difference of lower bounds on the augmented objective G and then uses that same δk as if it were a gap in raw cumulative cost JC. That identification is not derived and is the central load-bearing step; the non-sequitur is visible in Eqs. (74)-(76). Proposition 4.1's proof is also circular in assuming the intrinsic reward improves constraint satisfaction in order to prove the intrinsic reward improves optimization. These issues are substantive, but they are not driven by self-citation: the CPO-style bounds imported from [14] are external, so no self-citation chain is load-bearing. The central theoretical guarantee therefore reduces by construction, giving a partial circularity score of 6 rather than a higher score.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The central claims require several hand-set parameters (barrier steepness, gating sharpness, intrinsic reward weight) and assumptions (initial feasibility, bounded intrinsic reward, exactness of the approximate update). The paper's Proposition 4.1 depends on assuming the intrinsic reward improves constraint satisfaction, and Proposition 4.4 depends on unproved comparisons of lower bounds.

free parameters (6)
  • τ (barrier steepness) = 20 (default)
    Controls the slope of the extended log-barrier in Eq (5); sensitivity analysis over {2,5,10,20} shows performance varies with τ, so the choice is not neutral.
  • α (gating sharpness for intrinsic reward) = 0.3 (default, tuned per environment)
    Sharpness of the sigmoid gate in Eq (8); the appendix states the optimal value varies across scenarios and seeds and is tuned individually.
  • β (softmax scaling) = 1 (default)
    Scales the cost-advantage ratio before the softmax in Eq (8); set by hand.
  • ω (intrinsic reward weight) = 0.1, 0.3, 0.5 (tested)
    Defines η = ω·Gmax/Imax + ε in Eq (9); sensitivity is shown in Figure A5 and no principled selection rule is given.
  • Gmax and Imax in the definition of η = Unspecified
    Used to define the intrinsic reward coefficient η but never concretely defined, leaving the method underspecified.
  • δ (KL trust region bound) = 0.01
    Standard trust-region parameter, not fitted to the target result, but a hand-set hyperparameter used in the theoretical bounds.
assumptions (5)
  • domain assumption The initial policy π0 is feasible and the iterates used in the theoretical analysis satisfy the constraint-slack conditions required by the log-barrier (g_Ci(πk) ≤ -1/τ^2 in the regular case).
    Invoked in the proof of Theorem 4.3 via CPO bounds from reference [14] and stated in Section 6.1 as 'the assumption that the initial policy is theoretically feasible'.
  • domain assumption The extended log-barrier φτ is convex, continuous, and its derivative can be treated as the implicit dual variable λ* (Eq 12).
    Used in Theorem 4.2 and Eq (6); the paper does not verify these properties for the non-convex policy parameterization.
  • ad hoc to paper The natural-gradient step in Eq (21) accurately approximates the exact solution of Eq (9), so the theoretical bounds apply to the implemented algorithm.
    The theorems analyze the exact argmax of G under the KL constraint, while Algorithm 1/2 uses a linear objective and quadratic KL approximation; no error bound is provided.
  • ad hoc to paper In Proposition 4.1, f(πk+1) ≈ f(π̄k+1) and g_Ci(πk+1) ≤ g_Ci(π̄k+1).
    Stated in Appendix B around Eq (34); this assumes the intrinsic reward improves constraint satisfaction, which is the content of the proposition.
  • domain assumption The intrinsic reward I_Ci is non-negative and bounded above by Imax.
    Used in Theorems 4.2 and 4.3; the paper argues it follows from the softmax and sigmoid construction in Eq (8), but Imax is never quantified.
invented entities (1)
  • Constraint-aware intrinsic reward I_Ci
    purpose: Added to the policy optimization objective to guide boundary-aware exploration and reduce constraint violations (Eq 7-9).
    This is an algorithmic construct with no external measurable signature. Its claimed benefit rests on Proposition 4.1, whose proof assumes the reward pushes the policy toward feasibility, so the entity has no independent support.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Proactive Constrained Policy Optimization with Preemptive Penalty." pith.science (2026). https://pith.science/paper/FQSDQZFN

@misc{pith2026250801883,
  author       = {Pith},
  title        = {Pith review of: Proactive Constrained Policy Optimization with Preemptive Penalty},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FQSDQZFN}},
  note         = {Machine review of arXiv:2508.01883}
}
read the original abstract

Safe Reinforcement Learning (RL) often faces significant issues such as constraint violations and instability, necessitating the use of constrained policy optimization, which seeks optimal policies while ensuring adherence to specific constraints like safety. Typically, constrained optimization problems are addressed by the Lagrangian method, a post-violation remedial approach that may result in oscillations and overshoots. Motivated by this, we propose a novel method named Proactive Constrained Policy Optimization (PCPO) that incorporates a preemptive penalty mechanism. This mechanism integrates barrier items into the objective function as the policy nears the boundary, imposing a cost. Meanwhile, we introduce a constraint-aware intrinsic reward to guide boundary-aware exploration, which is activated only when the policy approaches the constraint boundary. We establish theoretical upper and lower bounds for the duality gap and the performance of the PCPO update, shedding light on the method's convergence characteristics. Additionally, to enhance the optimization performance, we adopt a policy iteration approach. An interesting finding is that PCPO demonstrates significant stability in experiments. Experimental results indicate that the PCPO framework provides a robust solution for policy optimization under constraints, with important implications for future research and practical applications.

Figures

Figures reproduced from arXiv: 2508.01883 by the authors.

Figure 1
Figure 1. Lagrange and barrier terms vs. constraint violations [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The average performance for CUP, EPO, FOCOPS, TRPOLag and PCPO over 6 seeds [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 23 canonical work pages

  1. [1]

    Safe exploration in continuous action spaces, 2018

    Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces, 2018

    work page 2018

  2. [2]

    A lyapunov-based approach to safe reinforcement learning

    Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. A lyapunov-based approach to safe reinforcement learning. Advances in neural information processing systems, 31, 2018

    2018

  3. [3]

    Lyapunov-based safe reinforcement learning for microgrid energy management.IEEE transactions on neural networks and learning systems, 2024

    Guokai Hao, Yuanzheng Li, Yang Li, Lin Jiang, and Zhigang Zeng. Lyapunov-based safe reinforcement learning for microgrid energy management.IEEE transactions on neural networks and learning systems, 2024

    work page 2024

  4. [4]

    Reward constrained policy optimization

    Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv:1805.11074, 2018

    arXiv 2018

  5. [5]

    A review of safe reinforcement learning: Methods, theories and applications

    Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. A review of safe reinforcement learning: Methods, theories and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  6. [6]

    Cvar-constrained policy optimization for safe reinforcement learning

    Qiyuan Zhang, Shu Leng, Xiaoteng Ma, Qihan Liu, Xueqian Wang, Bin Liang, Yu Liu, and Jun Yang. Cvar-constrained policy optimization for safe reinforcement learning. IEEE transactions on neural networks and learning systems, 36(1):830–841, 2024

    work page 2024

  7. [7]

    Safe reinforcement learning for multi-agent systems with risk constraints

    Ramen Ghosh. Safe reinforcement learning for multi-agent systems with risk constraints. 2025

    work page 2025

  8. [8]

    Lyapunov-based safe policy optimization for continuous control, 2019

    Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. Lyapunov-based safe policy optimization for continuous control, 2019

    work page 2019

Show all 33 references
  1. [9]

    Responsive safety in reinforcement learning by pid lagrangian methods

    Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. InInternational Conference on Machine Learning, pages 9133–9143. PMLR, 2020

    2020

  2. [10]

    Soufi Enayati, Mehran Ghafarian Tamizi, and Homayoun Najjaran

    Homayoun Honari, Amir M. Soufi Enayati, Mehran Ghafarian Tamizi, and Homayoun Najjaran. Meta sac-lag: Towards deployable safe reinforcement learning via metagradient-based hyperpa- rameter tuning. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IRO...

    2024

  3. [11]

    A survey of constraint formulations in safe reinforcement learning, 2024

    Akifumi Wachi, Xun Shen, and Yanan Sui. A survey of constraint formulations in safe reinforcement learning, 2024

    2024

  4. [12]

    Projection-based constrained policy optimization

    Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-based constrained policy optimization. arXiv preprint arXiv:2010.03152, 2020

    2010 arXiv

  5. [13]

    Risk-constrained reinforcement learning with percentile risk criteria

    Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18(167):1–51, 2018

    2018

  6. [14]

    Constrained policy optimization

    Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22–31. PMLR, 2017

    2017

  7. [15]

    Safe reinforcement learning using advantage-based intervention

    Nolan C Wagener, Byron Boots, and Ching-An Cheng. Safe reinforcement learning using advantage-based intervention. In International Conference on Machine Learning, pages 10630– 10640. PMLR, 2021

    2021

  8. [16]

    Provably efficient primal-dual reinforcement learning for cmdps with non-stationary objectives and constraints

    Yuhao Ding and Javad Lavaei. Provably efficient primal-dual reinforcement learning for cmdps with non-stationary objectives and constraints. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 7396–7404, 2023

    2023

  9. [17]

    Scalable primal- dual actor-critic method for safe multi-agent rl with general utilities

    Donghao Ying, Yunkai Zhang, Yuhao Ding, Alec Koppel, and Javad Lavaei. Scalable primal- dual actor-critic method for safe multi-agent rl with general utilities. Advances in Neural Information Processing Systems, 36, 2024

    2024

  10. [18]

    Adaptive primal-dual method for safe reinforce- ment learning, 2024

    Weiqin Chen, James Onyejizu, Long Vu, Lan Hoang, Dharmashankar Subramanian, Koushik Kar, Sandipan Mishra, and Santiago Paternain. Adaptive primal-dual method for safe reinforce- ment learning, 2024

    2024

  11. [19]

    Balance reward and safety optimization for safe reinforcement learning: A perspective of gradient manipulation

    Shangding Gu, Bilgehan Sel, Yuhao Ding, Lu Wang, Qingwei Lin, Ming Jin, and Alois Knoll. Balance reward and safety optimization for safe reinforcement learning: A perspective of gradient manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38...

    2024

  12. [20]

    Safe cor: A dual-expert approach to integrating imitation learning and safe reinforcement learning using constraint rewards

    Hyeokjin Kwon, Gunmin Lee, Junseo Lee, and Songhwai Oh. Safe cor: A dual-expert approach to integrating imitation learning and safe reinforcement learning using constraint rewards. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 2893–...

    2024

  13. [21]

    Safe reinforcement learning via episodic control

    Zhuo Li, Derui Zhu, and Jens Grossklags. Safe reinforcement learning via episodic control. IEEE Access, 2025

    2025

  14. [22]

    Comprehensive overview of reward engineering and shaping in advancing reinforcement learning applications

    Sinan Ibrahim, Mostafa Mostafa, Ali Jnadi, Hadi Salloum, and Pavel Osinenko. Comprehensive overview of reward engineering and shaping in advancing reinforcement learning applications. IEEE Access, 12:175473–175500, 2024

    2024

  15. [23]

    A review of safe reinforcement learning methods for modern power systems

    Tong Su, Tong Wu, Junbo Zhao, Anna Scaglione, and Le Xie. A review of safe reinforcement learning methods for modern power systems. Proceedings of the IEEE, 2025

    2025

  16. [24]

    Constrained Markov decision processes

    Eitan Altman. Constrained Markov decision processes. Routledge, 2021

    2021

  17. [25]

    Reinforcement learning, 2015

    David Silver. Reinforcement learning, 2015

    2015

  18. [26]

    Asynchronous methods for deep reinforce- ment learning

    V olodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference on machine learning, pages 1928–1937. PMLR, 2016

    1928

  19. [27]

    Constrained deep networks: Lagrangian optimization via log-barrier extensions

    Hoel Kervadec, Jose Dolz, Jing Yuan, Christian Desrosiers, Eric Granger, and Ismail Ben Ayed. Constrained deep networks: Lagrangian optimization via log-barrier extensions. In 2022 30th European Signal Processing Conference (EUSIPCO), pages 962–966. IEEE, 2022

    2022

  20. [28]

    A tutorial on mm algorithms

    David R Hunter and Kenneth Lange. A tutorial on mm algorithms. The American Statistician, 58(1):30–37, 2004

    2004

  21. [29]

    Safety gymnasium: A unified safe reinforcement learning benchmark

    Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: A unified safe reinforcement learning benchmark. Advances in Neural Information Processing Systems, 36, 2023

    2023

  22. [30]

    Constrained update projection approach to safe policy optimization

    Long Yang, Jiaming Ji, Juntao Dai, Linrui Zhang, Binbin Zhou, Pengfei Li, Yaodong Yang, and Gang Pan. Constrained update projection approach to safe policy optimization. Advances in Neural Information Processing Systems, 35:9111–9124, 2022

    2022

  23. [31]

    First order constrained optimization in policy space

    Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Systems, 33:15338–15349, 2020

    2020

  24. [32]

    Trust region policy optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889–1897. PMLR, 2015

    2015

  25. [33]

    repulsive force

    Shiqing Gao, Jiaxin Ding, Luoyi Fu, Xinbing Wang, and Cheng Zhou. Exterior penalty policy optimization with penalty metric network under constraints. In International Joint Conference on Artificial Intelligence, 2024. 12 A Preliminaries Recalling the definitions for pt π(s) an...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.