Pith. sign in

REVIEW 1 major objections 5 minor 27 references

Boundary-Seeking Policy Gradient for Safe Reinforcement Learning

T0 review · 1 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proposes and analyzes BSPG, a policy-gradient update that explicitly drives constrained policies to the safety boundary, proving the constraint residual converges to zero from either side and that convergent limits are…

desk verdict A genuinely two-sided boundary-seeking update with sound exact-gradient theorems, whose main caveats are a load-bearing regularity assumption that fails at a natural class of boundary optima and an evaluation too thin to support the comparative claims. read the letter →

arxiv 2608.10204 v1 pith:DIUNESJX submitted 2026-08-10 cs.LG

Chenhua Fan , Jiahui Zhu , Yuhang Zhang , Honghao Wei This is my paper · ORCID
classification cs.LG
keywords safereinforcementlearningconstrainedMarkovdecisionprocesspolicygradientconstraintboundarydecompositionKKTconditionsoccupancymeasureprimal-dualmethods
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Safe reinforcement learning usually treats "feasible" as the goal, but when the optimal constrained policy spends the entire safety budget, stopping in the interior wastes reward that could be converted from unused cost slack. This paper introduces Boundary-Seeking Policy Gradient (BSPG), a first-order update that splits the gradient into a tangential part (improve reward while preserving cost to first order) and a signed normal part (push the policy toward the cost limit, from either side, with strength proportional to $J_c(\pi_\theta) - d$). For the exact-gradient idealization the paper proves the residual shrinks to zero at rate $O(1/\sqrt{T})$, that any convergent iterate lies on the active boundary and is first-order stationary there, and that the limit satisfies KKT conditions when it is locally optimal over the feasible set. On a standard navigation task, BSPG reports higher reward and tighter boundary adherence than the compared baselines.

What carries the argument

The load-bearing object is the orthogonal decomposition of the reward gradient with respect to the cost gradient: the tangential piece $g_r^\perp$ moves reward uphill while leaving $J_c$ unchanged to first order, and the boundary-attraction piece $g_b$ is a signed residual times the normalized cost gradient. Together they define the BSPG direction $g_{\mathrm{up}}=g_r^\perp+\eta g_b$, equivalently a gradient step on the Lagrangian with an induced multiplier. The theoretical analysis is carried by the Lyapunov function $V(\theta)=\tfrac12\delta(\theta)^2$: the normal term gives a negative drift proportional to $\delta^2$, which yields the summability $\sum_t\alpha_t\delta_t^2<\infty$ and the residual bound $O(1/\sqrt{T})$, while the tangential term is exactly the component that vanishes at first-order stationarity on the active set.

What would settle it

Run exact-gradient BSPG on a CMDP whose cost function is constant over a region containing the constraint boundary (so $g_c=0$ there) and start with a nonzero residual. If the residual still converges to zero, Assumption 2 is unnecessary; if the update stalls or becomes undefined at the flat region, the theorem is limited to cost gradients bounded away from zero.

Watch

Extended reading notes

Core claim

The central claim is that the constraint residual $\delta(\theta)=J_c(\pi_\theta)-d$ should be used as a signed control signal, not merely as a feasibility alarm. The paper proves (Theorem 4.1) that whenever no unconstrained reward maximizer is feasible, every constrained-optimal stationary policy of the CMDP satisfies $J_c=d$, so the constrained optimum lies on the boundary $M$. The BSPG update combines the tangentially projected reward gradient $g_r^\perp = g_r - \frac{\langle g_r,g_c\rangle}{\|g_c\|^2}g_c$ with a signed normal attraction $g_b=-\delta\, g_c/(\|g_c\|+\varepsilon)$; under exact gradients the paper shows $\sum_t \alpha_t \delta_t^2 < \infty$ and $\delta_t\to 0$ (Theorem 6.1), $\langle g_r,g_r^\perp\rangle=\|g_r^\perp\|^2\ge 0$ (Theorem 6.3), and that any convergent limit $\theta_*$ lies on $M$ with $g_r(\theta_*)=\lambda_* g_c(\theta_*)$, $\lambda_*=\langle g_r,g_c\rangle/\|g_c\|^2$, satisfying KKT when it is also a local maximizer over the feasible set (Theorem 6.4). The same direction equals $\nabla_\theta[J_r(\pi_\theta)-\lambda_t J_c(\pi_\theta)]$ with an algebraic multiplier $\lambda_t=\langle g_r,g_c\rangle/\|g_c\|^2 + \eta\delta/(\|g_c\|+\varepsilon)$ and no learned dual variable.

Load-bearing premise

The convergence proof assumes the cost gradient never vanishes during training: there is a fixed positive lower bound on $\|g_c(\theta)\|$ over all iterates, so the tangential/normal splitting and the Lyapunov argument stay well defined; if the cost landscape flattens out, the claim does not apply.

Editorial extensions

If this is right

  • A safe RL policy trained with BSPG would use the full safety budget rather than settling in the feasible interior, converting unused slack into reward when reward and cost gradients align.
  • The induced algebraic multiplier removes the need to learn or tune a dual variable, simplifying constrained policy optimization pipelines.
  • Because the residual converges to zero from both sides, BSPG certifies an active constraint at the limit, a property that feasibility-only analyses such as CRPO's do not provide.
  • A convergent run ends at a first-order stationary point on the active boundary, satisfying KKT if it is locally optimal, so the final policy is locally maximal for reward under the equality constraint.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension not made in the paper: the signed normal term could be summed across multiple constraints, so residuals that remain nonzero after training would identify which constraints are active.
  • Speculative extension: the algebraic multiplier $\lambda_t$ could serve as a cheaper, oscillation-free proxy for a learned dual multiplier, but the paper does not test this.
  • The empirical claim rests on one navigation task; a multi-environment sweep would test whether boundary-tightness persists under different cost-gradient geometries.
  • The theory covers the exact-residual idealization; extending it to the estimated residual (19) under a critic-error bound is a natural open step the paper leaves unaddressed.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper studies constrained Markov decision processes and proposes Boundary-Seeking Policy Gradient (BSPG), a first-order update that splits the search direction into a tangential component that improves reward while preserving cost to first order and a signed, residual-driven normal component that attracts the policy toward the constraint boundary from either side. The update admits an algebraic Lagrangian form with an induced multiplier and no learned dual variable. For the idealized exact-gradient update with exact residual, the paper proves that the constraint residual satisfies a summable-square bound and converges to zero from either side with an O(1/sqrt(T)) finite-horizon rate (Theorem 6.1 and Corollary 6.2), that the tangential component is a reward-ascent direction (Theorem 6.3), and that any convergent parameter sequence converges to a stationary point on the active constraint set, satisfying KKT conditions under local optimality (Theorem 6.4). A boundary-optimality principle for occupancy-measure CMDPs is given in Theorem 4.1. Experiments on SafetyPointGoal1-v0 compare BSPG with CRPO and ESPO and report higher reward with tighter boundary proximity. The paper explicitly disclaims convergence guarantees for the stochastic PPO-based implementation and labels the residual estimator in Eq. (19) a heuristic.

Significance. If the results hold, the main contribution is a simple, dual-free first-order update with genuinely two-sided regulation of the constraint residual, complementing analyses that only certify feasibility. The proofs in Appendices B and D are internally consistent: the Lyapunov argument leading to Eq. (20) is valid, and the stationarity argument in Theorem 6.4 is sound. The paper is also honest about scope: it disclaims convergence for the practical PPO implementation, labels Eq. (19) an implementation heuristic, and avoids claiming a general empirical ordering. The boundary-proximity metric is a useful diagnostic that is rarely reported in safe RL. However, the theory rests on a strong uniform lower bound on the cost-gradient norm (Assumption 2), which is not verified empirically and can fail at natural boundary optima; this materially narrows the practical reach of the headline residual-convergence guarantee. Overall, the paper makes a credible theoretical contribution, but the gap between the stated assumptions and the settings of interest needs to be addressed before publication.

major comments (1)
  1. [Section VI, Assumption 2 and Eq. (20)] The contraction in Theorem 6.1 depends critically on Assumption 2's uniform lower bound ||g_c(theta)|| >= mu > 0. It enters the coefficient mu' = eta*mu^2/(G+epsilon) through Eq. (27), where the negative drift is -delta_t^2 * ||g_c||^2 / (||g_c||+epsilon). If ||g_c|| approaches zero, this drift vanishes and the Lyapunov recursion (20) loses its contraction, so delta_t -> 0 is no longer established. This is not a remote corner case: in a tabular softmax parameterization, a deterministic cost-minimizing policy has g_c = 0; if the threshold d equals the minimum achievable cost, the constrained optimum highlighted by Theorem 4.1 lies exactly at such a point and violates Assumption 2. The paper calls Assumption 2 a nondegeneracy condition but does not verify it on SafetyPointGoal1-v0 or analyze this failure mode. Since Theorem 6.1 is the main support for the abstract's two-sided boundary-convergence claim, the guarantee is materially narrower than the prose suggests. Please either weaken the assumption (for example, by adding a perturbation or regularization argument that handles g_c = 0), restrict the headline claims to the nondegenerate regime, or provide evidence that the regime is relevant to the experiments.
minor comments (5)
  1. [Section VII, Fig. 3] The empirical evaluation reports only one environment and two baselines, with no seed count, no confidence intervals or standard deviations, and no final numerical table. Please add multiple seeds with error bars and a table of final mean and variance values; the qualitative curves alone are difficult to assess.
  2. [Section IV and Appendix A] The proof of Theorem 4.1 appears twice, once in the main text and again in Appendix A. Please delete one copy or make the appendix version a cross-reference.
  3. [Section IV, after proof of Theorem 4.1] The sentence 'The proof, which relies only on the convexity of the occupancy-measure polytope and the linearity of Jr, Jc in the occupancy measure [4].' is a fragment; it should be completed or merged with the preceding paragraph.
  4. [Section V-D, Eq. (19)] The residual estimator is explicitly labeled a heuristic, but the two terms in Eq. (19) are averaged under different state distributions. It would help to state more precisely what conditions (for example, unbiasedness of the critic and on-policy sampling) would be needed for the estimator to be consistent, or to remove the second term if it is purely ad hoc.
  5. [Algorithm 1] The algorithm does not specify the values of eta, delta_max, epsilon, the PPO clip range, or the learning-rate schedule used in the experiments. Please include these details to make the experiments reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the convergence and stationarity theorems are proved from stated regularity assumptions and the update's own geometry, not from the conclusions they establish.

full rationale

The paper's derivation chain is self-contained and non-circular. Theorem 4.1 is proved from the standard linear-programming view over occupancy measures (external citation [4]) and a stated non-feasibility hypothesis on unconstrained maximizers; it does not assume the boundary conclusion. The BSPG update in Eqs. (13)-(15) is an explicit geometric construction: the tangential term is the projection of the reward gradient onto the cost level set, and the normal term is proportional to the signed residual. The implicit Lagrangian form in Eqs. (16)-(18) is an algebraic identity obtained by collecting coefficients of g_c; the 'implicit multiplier' is an induced coefficient, not a fitted or learned parameter. Theorem 6.1's residual convergence follows from the Lyapunov function V = (1/2)δ^2 and the direct computation that the normal term contributes a negative drift proportional to δ^2, under the stated smoothness, boundedness, and nondegeneracy assumptions. This is a theorem about an algorithm designed to reduce |δ|, not a prediction smuggled from its inputs. The same holds for Theorem 6.3, where the ascent identity is Cauchy-Schwarz applied to the projection, and Theorem 6.4, whose proof is elementary and self-contained. The authors' self-citations ([7]-[9], [11]) appear only as related-work pointers and are not load-bearing. Assumption 2's uniform lower bound ||g_c|| >= μ is a genuine regularity condition; its possible failure at cost-minimizing boundary optima is a correctness/robustness concern, not circularity, and the paper explicitly states that its guarantees are conditional on Assumptions 1-3. The empirical boundary-proximity metric tracks the algorithm's own objective, but the reported reward comparison against baselines is an independent evaluation, so it does not constitute a fitted input being relabeled as a prediction. Overall, no step reduces by construction or by self-citation to its own inputs.

Assumptions & free parameters 3 free parameters · 7 assumptions · 0 invented entities

The central claims rest on three regularity assumptions and the structural CMDP assumptions. No new physical or mathematical entities are introduced; the 'implicit multiplier' λ_t is an algebraic combination of gradients and residual, not a learned or fitted parameter. The main concern is Assumption 2, which is strong and unverified.

free parameters (3)
  • boundary-attraction coefficient η = not reported
    Scales the signed residual normal term in Eq. (15); convergence theory only needs η > 0, but the induced multiplier (16) and empirical performance depend on its value. Value used in experiments is not stated.
  • residual clipping threshold δ_max = not reported
    Clips the estimated residual in Algorithm 1 line 7; needed for stability of the practical update. Value not reported.
  • gradient stabilizer ε = not reported
    Used in denominators of Eqs. (14)-(16) and in normalized gradients in Algorithm 1 to avoid division by zero. Value not reported.
assumptions (7)
  • standard math Finite state and action spaces, discounted CMDP, stationary policy class (Theorem 4.1 and Appendix A).
    Ensures the occupancy-measure set is a compact convex polytope and Jr, Jc are linear functionals, giving the boundary-optimality proof.
  • domain assumption No unconstrained reward maximizer is feasible (Theorem 4.1 hypothesis).
    Condition under which every constrained optimum uses the full cost budget; the algorithm does not verify it and may behave suboptimally when it fails.
  • domain assumption Realizability condition (9): max over parameterized feasible policies equals the exact constrained optimum (Corollary 4.2).
    Transfers the boundary target from occupancy-measure space to the parameter space used by the algorithm.
  • domain assumption Assumption 1: L-Lipschitz gradients and bounded gradient norms (Section VI).
    Used in the descent lemma and Lyapunov bounds in Appendix B.
  • domain assumption Assumption 2: uniform lower bound μ on ||g_c|| and bounded residual |δ| (Section VI).
    Makes the gradient decomposition (13) well-defined and provides the negative drift in Eq. (20); the most fragile assumption.
  • standard math Assumption 3: step sizes with Σα_t = ∞ and Σα_t^2 < ∞ (Section VI).
    Standard Robbins-Monro condition for residual convergence and limit characterization.
  • domain assumption In Theorem 6.4, the limit θ* is additionally a local maximizer of Jr over the feasible set (Section VI).
    Needed to conclude λ* ≥ 0 and full KKT conditions; stationarity on M holds without it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Boundary-Seeking Policy Gradient for Safe Reinforcement Learning." pith.science (2026). https://pith.science/paper/DIUNESJX

@misc{pith2026260810204,
  author       = {Pith},
  title        = {Pith review of: Boundary-Seeking Policy Gradient for Safe Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DIUNESJX}},
  note         = {Machine review of arXiv:2608.10204}
}
abstract

Safe reinforcement learning maximizes reward subject to safety constraints. For Constrained Markov Decision Processes, the linear-programming view over occupancy measures implies that whenever the constraint is active at optimality, the optimal policy lies exactly on the constraint boundary, yet standard gradient-based methods do not exploit this structure and often settle in the feasible interior. We introduce Boundary-Seeking Policy Gradient (BSPG), a first-order method whose update combines a tangential component that improves reward while preserving cost to first order with a signed, residual-driven normal component that regulates the policy toward the active boundary from either side; the combined direction admits an algebraic Lagrangian form with an induced coefficient and no learned dual variable. Under exact gradients and stated regularity conditions, the constraint residual converges to zero from either side with a finite-horizon $O(1/\sqrt{T})$ bound, the tangential component is a reward-ascent direction on the boundary, and any convergent parameter sequence is stationary on the active constraint set, satisfying the KKT conditions when the limit is also a local maximizer over the feasible set. This complements existing analyses, which certify feasibility but do not characterize the constraint value at convergence. On a standard Safety-Gymnasium navigation task, BSPG attains higher reward while tracking the boundary more tightly than the compared baselines.

Figures

Figures reproduced from arXiv: 2608.10204 by the authors.

Figure 1
Figure 1. Local update geometry of ESPO [18] and BSPG. ESPO shapes its direction from the reward–cost gradient relation; BSPG adds a normal component set by the signed constraint residual. For the ideal update (ˆδ = δ), this component points toward M from both the feasible (ˆδ < 0) and infeasible (ˆδ > 0) sides. Primal methods restrict or correct updates by feasibility. CPO [12] solves a constrained local subproblem with guar… view at source ↗
Figure 2
Figure 2. Update trajectories of BSPG and ESPO on a synthetic [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Results on SafetyPointGoal1-v0. baselines remain more conservative and settle well inside the feasible region. This behavior is most clearly illustrated in the boundary-proximity plot, where BSPG remains consistently closest to zero, indicating tighter tracking of the active constraint boundary. Taken together, these results support the proposed boundary-regulation mechanism in the evaluated task; they do not by the… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 17 canonical work pages

  1. [1]

    A general reinforcement learning algorithm that masters chess, shogi, and go through self-play,

    D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis, “A general reinforcement learning algorithm that masters chess, shogi, and go through self-play,”Science, vol. 362, no. 6419, pp. 1140–1144, 2018

    2018

  2. [2]

    Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning,

    Q. Li, Z. Peng, L. Feng, Q. Zhang, Z. Xue, and B. Zhou, “Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning,”IEEE transactions on pattern analysis and machine intelli- gence, vol. 45, no. 3, pp. 3461–3475, 2022

    2022

  3. [3]

    Reinforcement learning in robotics: A survey,

    J. Kober, J. A. Bagnell, and J. Peters, “Reinforcement learning in robotics: A survey,”The International Journal of Robotics Research, vol. 32, no. 11, pp. 1238–1274, 2013

    2013

  4. [4]

    Altman,Constrained Markov decision processes

    E. Altman,Constrained Markov decision processes. CRC Press, 1999, vol. 7

    1999

  5. [5]

    Responsive safety in rein- forcement learning by pid lagrangian methods,

    A. Stooke, J. Achiam, and P. Abbeel, “Responsive safety in rein- forcement learning by pid lagrangian methods,” inInt. Conf. Machine Learning (ICML). PMLR, 2020, pp. 9133–9143

    work page 2020

  6. [6]

    Safe policies for reinforcement learning via primal-dual methods,

    S. Paternain, M. Calvo-Fullana, L. F. O. Chamon, and A. Ribeiro, “Safe policies for reinforcement learning via primal-dual methods,”IEEE Trans. Autom. Control, pp. 1–1, 2022

    work page 2022

  7. [7]

    Triple-Q: a model-free algorithm for constrained reinforcement learning with sublinear regret and zero constraint violation,

    H. Wei, X. Liu, and L. Ying, “Triple-Q: a model-free algorithm for constrained reinforcement learning with sublinear regret and zero constraint violation,” inInt. Conf. Artificial Intelligence and Statistics (AISTATS), 2022

    work page 2022

  8. [8]

    Provably efficient model-free algorithms for non-stationary CMDPs,

    H. Wei, A. Ghosh, N. Shroff, L. Ying, and X. Zhou, “Provably efficient model-free algorithms for non-stationary CMDPs,” inInt. Conf. Artificial Intelligence and Statistics (AISTATS). PMLR, 2023, pp. 6527–6570

    work page 2023

Show all 27 references
  1. [9]

    Safe and efficient: A primal- dual method for offline convex cmdps under partial data coverage,

    H. Zhang, X. Peng, H. Wei, and X. Liu, “Safe and efficient: A primal- dual method for offline convex cmdps under partial data coverage,” in Advances Neural Information Processing Systems (NeurIPS), 2024

    2024

  2. [10]

    Provably efficient safe exploration via primal-dual policy optimization,

    D. Ding, X. Wei, Z. Yang, Z. Wang, and M. Jovanovic, “Provably efficient safe exploration via primal-dual policy optimization,” inInt. Conf. Artificial Intelligence and Statistics (AISTATS), vol. 130. PMLR, 2021, pp. 3304–3312

    2021

  3. [11]

    An optimistic algorithm for online cmdps with anytime adversarial constraints,

    J. Zhu, K. Yu, D. Lee, X. Liu, and H. Wei, “An optimistic algorithm for online cmdps with anytime adversarial constraints,” inInt. Conf. Machine Learning (ICML), ser. ICML’25. JMLR.org, 2025

    2025

  4. [12]

    Constrained policy optimization,

    J. Achiam, D. Held, A. Tamar, and P. Abbeel, “Constrained policy optimization,” inInt. Conf. Machine Learning (ICML), vol. 70. JMLR, 2017, pp. 22–31

    2017

  5. [13]

    Projection- based constrained policy optimization,

    T.-Y . Yang, J. Rosca, K. Narasimhan, and P. J. Ramadge, “Projection- based constrained policy optimization,” inInt. Conf. on Learning Representations (ICLR), 2020

    2020

  6. [14]

    First order constrained optimization in policy space,

    Y . Zhang, Q. Vuong, and K. Ross, “First order constrained optimization in policy space,”Advances in Neural Information Processing Systems, vol. 33, pp. 15 338–15 349, 2020

    2020

  7. [15]

    Constrained update projection approach to safe policy optimization,

    L. Yang, J. Ji, J. Dai, L. Zhang, B. Zhou, P. Li, Y . Yang, and G. Pan, “Constrained update projection approach to safe policy optimization,” Advances in Neural Information Processing Systems, vol. 35, pp. 9111– 9124, 2022

    2022

  8. [16]

    Crpo: A new approach for safe reinforcement learning with convergence guarantee,

    T. Xu, Y . Liang, and G. Lan, “Crpo: A new approach for safe reinforcement learning with convergence guarantee,” inInt. Conf. Machine Learning (ICML), M. Meila and T. Z. 0001, Eds., vol. 139. PMLR, 2021, pp. 11 480–11 491

    2021

  9. [17]

    Balance reward and safety optimization for safe reinforcement learning: A perspective of gradient manipulation,

    S. Gu, B. Sel, Y . Ding, L. Wang, Q. Lin, M. Jin, and A. Knoll, “Balance reward and safety optimization for safe reinforcement learning: A perspective of gradient manipulation,” inAAAI Conf. Artificial Intelligence, vol. 38, no. 19, 2024, pp. 21 099–21 106

    2024

  10. [18]

    Enhancing efficiency of safe reinforcement learning via sample manipulation,

    S. Gu, L. Shi, Y . Ding, A. Knoll, C. Spanos, A. Wierman, and M. Jin, “Enhancing efficiency of safe reinforcement learning via sample manipulation,”Advances in Neural Information Processing Systems, vol. 37, pp. 17 247–17 285, 2024

    2024

  11. [19]

    Gradient shaping for multi-constraint safe reinforcement learning,

    Y . Yao, Z. Liu, Z. Cen, P. Huang, T. Zhang, W. Yu, and D. Zhao, “Gradient shaping for multi-constraint safe reinforcement learning,” in 6th annual learning for dynamics & control conference. PMLR, 2024, pp. 25–39

    2024

  12. [20]

    Benchmarking safe exploration in deep reinforcement learning,

    A. Ray, J. Achiam, and D. Amodei, “Benchmarking safe exploration in deep reinforcement learning,”arXiv preprint arXiv:1910.01708, vol. 7, no. 1, p. 2, 2019

    1910 arXiv

  13. [21]

    Reward constrained policy optimization,

    C. Tessler, D. J. Mankowitz, and S. Mannor, “Reward constrained policy optimization,”arXiv preprint arXiv:1805.11074, 2018

    2018 arXiv

  14. [22]

    Off-policy primal-dual safe reinforcement learning,

    Z. Wu, B. Tang, Q. Lin, C. Yu, S. Mao, Q. Xie, X. Wang, and D. Wang, “Off-policy primal-dual safe reinforcement learning,”arXiv preprint arXiv:2401.14758, 2024

    2024 arXiv

  15. [23]

    Constrained proximal policy optimization,

    C. Xuan, F. Zhang, F. Yin, and H.-K. Lam, “Constrained proximal policy optimization,”arXiv preprint arXiv:2305.14216, 2023

    2023 arXiv

  16. [24]

    Policy gradient methods for reinforcement learning with function approximation,

    R. S. Sutton, D. McAllester, S. Singh, and Y . Mansour, “Policy gradient methods for reinforcement learning with function approximation,” inAdvances Neural Information Processing Systems (NeurIPS), ser. NIPS’99, 1999

    1999

  17. [25]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017

    2017

  18. [26]

    V . S. Borkar,Stochastic approximation: a dynamical systems viewpoint. Springer, 2009, vol. 48

    2009

  19. [27]

    Omnisafe: An infrastructure for accelerating safe reinforcement learning research,

    J. Ji, J. Zhou, B. Zhang, J. Dai, X. Pan, R. Sun, W. Huang, Y . Geng, M. Liu, and Y . Yang, “Omnisafe: An infrastructure for accelerating safe reinforcement learning research,”Journal of Machine Learning Research, vol. 25, no. 285, pp. 1–6, 2024. APPENDIX A. Proof of Theorem 4...

    2024

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.