Pith. sign in

REVIEW 3 major objections 4 minor 64 references

Safe Reinforcement Learning using Finite-Horizon Gradient-based Estimation

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A finite-horizon safety constraint can be estimated by its first-order policy-parameter gradient, and the paper proves the error is bounded by the squared update size.

desk verdict A real mismatch identified and a clever fix, but the safety guarantee is proved for exact gradients while the implementation uses a biased critic-bootstrapped gradient. read the letter →

arxiv 2412.11138 v1 pith:GWDVDW6Y submitted 2024-12-15 cs.LG cs.AI

Juntao Dai , Yaodong Yang , Qian Zheng , Gang Pan This is my paper · ORCID
classification cs.LGcs.AI
keywords safereinforcementlearningconstrainedMarkovdecisionprocessfinite-horizonconstraintsgradient-basedestimationtrust-regionoptimizationdifferentiablesimulationpolicyconstraintsatisfaction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that finite-horizon, non-discounted safety constraints in reinforcement learning can be estimated accurately from first-order gradients, and that this estimate is enough to keep every policy update feasible inside a trust region. It argues that the standard advantage-based estimation, which relies on an infinite-horizon discounted advantage function, can have relative errors above 1.0 on finite-horizon constraints and therefore misleads safe policy updates. The proposed Gradient-based Estimation (GBE) approximates the next constraint value as $J_C(\theta_0)+\delta^\top\nabla_\theta J_C(\theta_0)$ and bounds the error by $\frac{1}{2}\epsilon\|\delta\|_2^2$, so update size, not the discount factor, becomes the lever for safety. On top of GBE the paper builds Constrained Gradient-based Policy Optimization (CGPO), whose experiments on differentiable control tasks report faster convergence and fewer constraint violations than prior safe RL algorithms.

What carries the argument

The machinery is the Gradient-based Estimation map $\hat J_f(\theta_0+\delta)=J_f(\theta_0)+\delta^\top\nabla_\theta J_f(\theta_0)$ applied to both reward and cost, with gradients obtained by back-propagation through a differentiable simulator or a learned world model. Its error is $O(\|\delta\|_2^2)$, which is what converts safety into a trust-region condition: the optimizer solves the small quadratic program $\max_\delta g_k^\top\delta$ subject to $c_k+q_k^\top\delta\leq0$ and $\|\delta\|_2^2\leq\hat\delta$, using closed-form KKT dual variables $\lambda^*$, $\nu^*$ and update $\theta_{k+1}=\theta_k+(g_k-\nu^*q_k)/\lambda^*$, with an adaptive radius that shrinks when observed reward or constraint changes disagree with the estimates.

What would settle it

Measure the true constraint value $J_C(\theta_{k+1})$ after an update in a differentiable environment where the exact full-horizon gradient is computable, and compare the prediction $J_C(\theta_k)+\delta^\top q_k$ with it; if the error exceeds $\frac{1}{2}\hat\delta\max\|\nabla^2 J_C\|$ at small $\hat\delta$, or if replacing the exact gradient by the SHAC-style critic gradient pushes the relative error above 1.0, the claimed guarantee is false.

Watch

Extended reading notes

Core claim

The central claim is that a small parameter update $\delta$ from $\theta_0$ changes the finite-horizon constraint function by $\delta^\top\nabla_\theta J_C(\theta_0)$ up to an error of at most $\frac{1}{2}\epsilon\|\delta\|_2^2$, where $\epsilon$ bounds the Hessian of $J_C$ along the line segment between $\theta_0$ and $\theta_0+\delta$; the same first-order estimate is used for the reward objective. This is contrasted with the infinite-horizon advantage identity behind ABE, which requires $\gamma\neq1$ and a discounted state distribution, and which the paper shows can mispredict finite-horizon constraint changes by more than the true change itself. Using GBE, CGPO defines each update step as the solution of a convex trust-region sub-problem: maximize $\delta^\top g_k$ subject to $c_k+\delta^\top q_k\leq0$ and $\|\delta\|_2^2\leq\hat\delta$, with three explicit cases depending on whether the whole trust region is infeasible, feasible, or split by the constraint boundary. The worst-case theorem then bounds reward loss by $\frac{1}{2}\epsilon_R\hat\delta$ and constraint violation by $b+\frac{1}{2}\epsilon_C\hat\delta$ for feasible starting policies.

Load-bearing premise

The whole argument presumes that the gradient used in the Taylor estimate is the exact gradient of the true finite-horizon constraint function, while the implementation computes it through a short-horizon loss with a learned critic bootstrap that can carry bias this analysis does not charge.

Editorial extensions

If this is right

  • If GBE is right, safety violation at each update is controlled by the trust-region radius: the next policy satisfies $J_C(\theta_{k+1})\leq b+\frac{1}{2}\epsilon_C\hat\delta$ instead of inheriting an infinite-horizon approximation error.
  • Finite-horizon undiscounted constraints, the common form in practice, can be handled directly by a deep safe RL algorithm without being recast as discounted infinite-horizon sums.
  • CGPO always has a well-defined update direction: steepest descent on the constraint when the trust region is entirely infeasible, steepest ascent on reward when entirely feasible, and a KKT-based convex update in the split case.
  • Empirically, the paper reports that CGPO converges in fewer environment steps and with lower violation ratios than primal-dual, CPO/CUP, and Lagrangian differentiable-RL baselines across four differentiable control tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This suggests a direct ablation: compare GBE with full-trajectory backpropagation against the SHAC-style short-horizon critic gradient; if the critic bootstrap biases the gradient, the stated worst-case bounds will be optimistic even when GBE itself is exact.
  • The estimator's only structural requirement is twice-differentiability of the constraint in policy parameters, so the same trust-region scheme transfers to deterministic trajectory optimization and model-predictive control, where no policy network or reward sampling is needed.
  • Because the paper's relative-error metric divides by the true constraint change, GBE looks bad on nearly flat constraints even when its absolute error is tiny; an absolute-violation metric would make the comparison with ABE more interpretable.
  • The adaptive-radius rule embodies a general principle: when an estimator has quadratic error, the optimizer can treat its radius as a learnable safety margin and use the ratio of predicted to observed reward and constraint changes to expand or shrink it, without estimating Hessians.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper addresses safe reinforcement learning with finite-horizon, non-discounted constraints, which are common in benchmarks but poorly matched to the infinite-horizon advantage-based estimation (ABE) used by prior deep Safe RL algorithms. The authors propose Gradient-based Estimation (GBE), a first-order Taylor expansion of the constraint and reward functions along differentiable trajectories, and build a trust-region algorithm, Constrained Gradient-based Policy Optimization (CGPO), that repeatedly solves a linearly constrained quadratic subproblem. They prove solvability conditions for the subproblem, a worst-case Taylor remainder bound, and report experiments on four differentiable Brax-based tasks showing faster convergence and lower constraint-violation ratios than several baselines. The paper also includes ablations comparing GBE with ABE, a world-model extension, and an adaptive trust-region radius.

Significance. The core idea is simple and potentially valuable: use analytic finite-horizon gradients instead of infinite-horizon advantages to estimate constraint changes, then keep updates small in parameter space so that first-order estimates are credible. The mathematical statements for exact gradients are standard Taylor remainder/KKT arguments and are essentially correct. The empirical study is reasonably thorough for a conference paper: multiple environments, several baselines including differentiable-RL baselines, five seeds, an estimation-error ablation, and a world-model ablation. However, the paper's most important claims—that each CGPO update is feasible and that the empirical gains are explained by accurate GBE estimates—depend on a mismatch between the analyzed object (exact gradients) and the implemented object (short-horizon critic-bootstrapped gradients). Until this gap is closed with an error analysis or by changing the implementation, the safety guarantees are not established for the algorithm that is actually run. The paper is therefore worth revising rather than accepting in its current form.

major comments (3)
  1. [Sections 4.1, 5.2, and 5.3, Eqs. (7), (17), and (20)] The theoretical guarantees assume that q_k and g_k in the trust-region subproblem are the exact gradients of the true finite-horizon functions J_C and J_R. In the actual algorithm, q_k is computed from the SHAC-style loss in Eq. (20), whose last term is the learned critic V_C^psi(s_{t0+h}). The back-propagated gradient therefore contains a term dV_C^psi/ds_{t0+h} * ds_{t0+h}/dtheta, which is a biased estimate of the true constraint gradient unless the critic is exactly the remaining finite-horizon cost. Writing q_k = grad J_C(theta_k) + e, the linearized constraint error in the subproblem becomes delta^T e, which is O(||delta||), while Lemma 4.1's bound only covers a term of order ||delta||^2. Consequently, Theorem 5.3's bound JC(theta_{k+1}) <= b + (1/2) epsilon_C delta_hat is not valid for the implemented algorithm, and a systematically biased q_k can make the subproblem select an update that the linearized constraint declares feasible while the true constraint moves beyond b. Section 7 discusses environmental differentiability but does not mention this critic-induced gradient error. This is load-bearing because the low violation ratios in Table 1 are attributed precisely to the accuracy of GBE estimates.
  2. [Section 5.2 and abstract/conclusion, Eq. (17)] The paper repeatedly claims that CGPO 'ensures the feasibility of each update,' but Theorem 5.3 only proves JC(theta_{k+1}) <= b + (1/2) epsilon_C delta_hat, i.e., the actual constraint can exceed the threshold by an O(delta_hat) amount. This is a worst-case violation bound, not a feasibility guarantee. If 'feasible' means JC <= b, the theorem does not establish it; if 'feasible' means 'within a tolerance,' that tolerance is never defined or used in Algorithm 1. The abstract and conclusion should be reworded, or the theorem strengthened by adding a constraint-backoff mechanism that enforces JC(theta_k) + delta^T q_k <= b - margin with a margin derived from the Hessian bound.
  3. [Algorithm 1 and Corollary 5.2] The corollary states that the entire trust region is feasible when c_k^2/q_k^T q_k - delta_hat > 0 and c_k <= 0, but Algorithm 1 uses the strict condition c_k < 0 in the second branch. At exactly c_k = 0 with a fully feasible trust region, the algorithm falls into the third branch and solves a KKT system that may not be the intended update. This boundary mismatch is minor in practice but should be corrected for logical consistency with the stated theory.
minor comments (4)
  1. [Lemma 4.1 and Appendix A.1] The statement of Lemma 4.1 defines epsilon using ||nabla^2_theta J_R|| but the lemma is about J_f; it should be ||nabla^2_theta J_f||. The proof also calls the remainder a 'Peano' remainder, but Eq. (25) is the Lagrange form of the remainder.
  2. [Section 5.1, notation paragraph] The notation line defines 'gk = ∇θJR(θk), gk = ∇θJC(θk)', using the same symbol for both the reward and constraint gradients; the second should be q_k.
  3. [Eq. (18) and its definition of zeta_k] The toleration ratio zeta_k has a denominator |JC(theta_{k+1}) - hat JC(theta_{k+1})| that can be zero in practice; the paper does not specify a fallback value when this occurs.
  4. [Figure 5 and Section 6.3] The relative error metric divides by the true constraint change; if that change is near zero, the ratio is unstable. The caption says errors are averaged over 100 repetitions but does not report error bars or confidence intervals, making it hard to assess the significance of the GBE versus ABE difference.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GBE is a genuine Taylor approximation validated against simulated ground truth, and the cited prior work is background rather than load-bearing.

full rationale

The derivation chain is self-contained. Section 4.1 defines GBE as a first-order Taylor expansion of the true finite-horizon constraint function (Eq. 7), and Lemma 4.1 bounds the truncation error by the standard Taylor remainder 1/2 epsilon ||delta||^2, assuming only twice differentiability of J_f; the bound does not presuppose the target feasibility result. The empirical validation in Figure 5 and Appendix C compares the GBE and ABE estimates against the actually simulated constraint after the update using the relative-error metric (Eq. 74), so the predicted quantity is not fitted or renamed from the measured quantity. The trust-region subproblem (Eq. 9) and Theorem 5.3 are analyzed with exact gradients, while the practical SHAC-style gradient in Eq. (20) introduces a learned critic bootstrap; this is a theory-implementation bias/gap concern, not a circular reduction, and Section 7 does acknowledge environmental differentiability as a limitation. No fitted parameter is later called a prediction, no load-bearing uniqueness theorem or ansatz is imported from the authors' own prior work, and self-citations such as APPO, CUP, and Safety-Gymnasium serve only as background comparisons. The paper's 'first' claims are positional novelty statements rather than conclusions derived from its own outputs. Overall, the estimation is a standard mathematical expansion checked against independent simulator rollouts, so no circular step is present.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The derivation is a first-order Taylor expansion with an exact-gradient assumption. The main unstated assumption is the fidelity of the short-horizon, critic-bootstrapped gradient used in practice; this gap is not analyzed. There are no target-fitted free parameters, only hand-chosen hyperparameters.

free parameters (1)
  • Trust region radius and adaptation hyperparameters = δ init 1e-3 to 1e-2, δ lower 1e-4, δ upper 1e-2 to 1e-1, β1=0.8, β2=1.25, η1, η2
    Chosen by hand per task (Table 4); they control update size and error tolerance and affect performance, but are not fitted to the target reward or constraint result.
assumptions (4)
  • domain assumption Environment transition F(s,a) is differentiable and the policy is differentiable in θ, so backpropagation yields gradients of reward and cost
    Section 3.3 and Section 4.1; the entire GBE mechanism and CGPO rely on this.
  • standard math J_R and J_C are twice differentiable near θ0, so the Taylor remainder with Peano form holds
    Lemma 4.1; needed for the error bound O(||δ||^2).
  • domain assumption The SHAC-style gradient computed from Eq. (19)-(20) with a learned critic is a faithful approximation of the true gradient of J_C
    Section 5.3; theory assumes exact gradient but the algorithm uses the bootstrapped loss gradient, and no error bound covers this approximation.
  • domain assumption Slater's condition holds when the trust region partially intersects the feasible region, giving strong duality
    Section 5.1; the KKT-based closed-form solver requires strong duality for the convex sub-problem.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Safe Reinforcement Learning using Finite-Horizon Gradient-based Estimation." pith.science (2026). https://pith.science/paper/GWDVDW6Y

@misc{pith2026241211138,
  author       = {Pith},
  title        = {Pith review of: Safe Reinforcement Learning using Finite-Horizon Gradient-based Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GWDVDW6Y}},
  note         = {Machine review of arXiv:2412.11138}
}
read the original abstract

A key aspect of Safe Reinforcement Learning (Safe RL) involves estimating the constraint condition for the next policy, which is crucial for guiding the optimization of safe policy updates. However, the existing Advantage-based Estimation (ABE) method relies on the infinite-horizon discounted advantage function. This dependence leads to catastrophic errors in finite-horizon scenarios with non-discounted constraints, resulting in safety-violation updates. In response, we propose the first estimation method for finite-horizon non-discounted constraints in deep Safe RL, termed Gradient-based Estimation (GBE), which relies on the analytic gradient derived along trajectories. Our theoretical and empirical analyses demonstrate that GBE can effectively estimate constraint changes over a finite horizon. Constructing a surrogate optimization problem with GBE, we developed a novel Safe RL algorithm called Constrained Gradient-based Policy Optimization (CGPO). CGPO identifies feasible optimal policies by iteratively resolving sub-problems within trust regions. Our empirical results reveal that CGPO, unlike baseline algorithms, successfully estimates the constraint functions of subsequent policies, thereby ensuring the efficiency and feasibility of each update.

Figures

Figures reproduced from arXiv: 2412.11138 by the authors.

Figure 1
Figure 1. Advantage-based Estimation fails even in simple envi￾ronments under finite-horizon constraints. (a) The cost obtained by the agent while traversing along the x-axis, namely, ct = c(xt). (b) Relative errors in the estimation of changes in the finite-horizon cumulative constraint (i.e., P⊤ t=1 ct ≤ b). The ABE method gen￾erates relative errors even greater than 1.0, showing completely incorrect estimations. Refer to A… view at source ↗
Figure 2
Figure 2. The computational relationship between the policy up￾date δ, the gradient of the objective function g, and the gradient of the constraint function g varies in three scenarios. 5. Constrained Gradient-based Policy Optimization Based on the constrained surrogate sub-problem in Equa￾tion (9), we develop a novel Safe RL algorithm named Constrained Gradient-based Policy Optimization (CGPO). 5.1. Solution to Surrogate Sub… view at source ↗
Figure 3
Figure 3. Gradient computation graph for the short-horizon ap￾proach. Here, F(s) represents the differentiable dynamics of the environment, R is the reward signal, C is the cost signal, πθ is the parameterized policy, and V R ϕ and V C ϕ are the value functions for the return and constraint. Zero-order Batch Gradient (ZoBG) method and the First￾order Batch Gradient (FoBG) method (Suh et al., 2022). In this section, we present… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Training curves of certain algorithms on different tasks, showing episodic return and constraint for 5 random seeds. Solid lines represent the mean, while the shaded areas indicate variance, without any smoothing to the curves. CGPO demonstrates superior efficiency in …
Figure 7
Figure 7. Figure 7: Adaptive trust region radius ˆδ enables CGPO to achieve better performance improvements and constraint satisfaction. primal Safe RL problem, preventing traditional algorithms from precisely meeting safety constraints. In the HalfCheetah task, errors are higher than in …
Figure 6
Figure 6. Figure 6: CGPO uses gradients from the World Models in non￾differentiable tasks. The grey shading represents predictions from the World Model. Compared to SHAC-Lag and BPTT-Lag, CGPO exhibits more stable convergence. As [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: The computational relationship between the policy update δ, the gradient of the objective function g, and the gradient of the constraint function g varies in three scenarios. A.2.2. THE SOLUTION TO THE SUB-PROBLEM Based on Theorem 5.1 and Corollary 5.2, solving the sub…
Figure 9
Figure 9. Figure 9: Computation graph of CGPO. A.4. Gradient Calculation Given our focus on employing gradients to solve Safe RL problems, our algorithm can incorporate any differentiable method for computing gradients gk and qk, such as BPTT (Mozer, 2013) and SHAC (Jie Xu et al., 2022). …
Figure 10
Figure 10. Figure 10: Training curves of CGPO using ZoBG method and FoBG method to calculate the gradients, showing episodic return and constraint for 5 random seeds. Solid lines represent the mean, while the shaded areas indicate variance, without any smoothing to the curves. In summary, …
Figure 11
Figure 11. Figure 11: Cost values at different x-axis positions for the agent. To clearly demonstrate the shortcomings of the Advantage-based Estimation (ABE) method, we designed a simple task. The goal of this task is to train an agent to move along the x-axis, receiving rewards and incur…
Figure 12
Figure 12. Figure 12: Training curves compared to the Traditional Safe RL Algorithms on different tasks, showing episodic return and constraint for 5 random seeds. Solid lines represent the mean, while the shaded areas indicate variance, without any smoothing to the curves. (a) CartPole (b…
Figure 13
Figure 13. Figure 13: Training curves compared to the Lagrangian-revised differentiable Safe RL algorithms on different tasks, showing episodic return and constraint for 5 random seeds. Solid lines represent the mean, while the shaded areas indicate variance, without any smoothing to the c…
Figure 14
Figure 14. Figure 14: Ablation studies on the hyper-parameters of the adaptive method of trust-region radius [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]
Figure 15
Figure 15. Figure 15: Four distinct agents used in our experiments originate from Brax, a fully differentiable physics engine. E.1. Position-constrained CartPole This environment features a cart that moves linearly with one end of a pole fixed to it and the other end free. The goal is to p…
Figure 16
Figure 16. Figure 16: Cost values at different x-axis positions for the agent. Non-differentiable Function Environment. We design a function environment similar to the simple environment in [PITH_FULL_IMAGE:figures/full_fig_p030_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 38 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Constrained policy optimization, 2017

    Achiam, J., Held, D., Tamar, A., and Abbeel, P. Constrained policy optimization, 2017

    work page 2017

  3. [3]

    Constrained Markov decision processes, volume 7

    Altman, E. Constrained Markov decision processes, volume 7. CRC press, 1999

    work page 1999

  4. [4]

    Constrained policy optimization via bayesian world models

    As, Y., Usmanova, I., Curi, S., and Krause, A. Constrained policy optimization via bayesian world models. arXiv preprint arXiv:2201.09802, 2022

    arXiv 2022

  5. [5]

    Robots that interact with humans: a review of safety technologies and standards

    Bogue, R. Robots that interact with humans: a review of safety technologies and standards. Industrial Robot: An International Journal, 44 0 (4): 0 395--400, 2017

    work page 2017

  6. [6]

    Risk-constrained reinforcement learning with percentile risk criteria

    Chow, Y., Ghavamzadeh, M., Janson, L., and Pavone, M. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18 0 (167): 0 1--51, 2018

    2018

  7. [7]

    Model-augmented actor-critic: Backpropagating through paths

    Clavera, I., Fu, V., and Abbeel, P. Model-augmented actor-critic: Backpropagating through paths. arXiv preprint arXiv:2005.08068, 2020

    arXiv 2005

  8. [8]

    Augmented proximal policy optimization for safe reinforcement learning

    Dai, J., Ji, J., Yang, L., Zheng, Q., and Pan, G. Augmented proximal policy optimization for safe reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.\ 7288--7295, 2023

    work page 2023

Show all 64 references
  1. [9]

    Safe RLHF : Safe reinforcement learning from human feedback

    Dai, J., Pan, X., Sun, R., Ji, J., Xu, X., Liu, M., Wang, Y., and Yang, Y. Safe RLHF : Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=TyFrPOKYXw

    2024

  2. [10]

    A differentiable physics engine for deep learning in robotics

    Degrave, J., Hermans, M., Dambre, J., et al. A differentiable physics engine for deep learning in robotics. Frontiers in neurorobotics, pp.\ 6, 2019

    2019

  3. [11]

    A., Farouk, H., and Mofreh, E

    ElDahshan, K. A., Farouk, H., and Mofreh, E. Deep reinforcement learning based video games: A review. In 2022 2nd International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC), pp.\ 302--309. IEEE, 2022

    2022

  4. [12]

    D., Frey, E., Raichuk, A., Girgin, S., Mordatch, I., and Bachem, O

    Freeman, C. D., Frey, E., Raichuk, A., Girgin, S., Mordatch, I., and Bachem, O. Brax - a differentiable physics engine for large scale rigid body simulation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021

    2021

  5. [13]

    A Review and Outlook on Predictive Cruise Control of Vehicles and Typical Applications Under Cloud Control System

    Gao, B., Wan, K., Chen, Q., Wang, Z., Li, R., Jiang, Y., Mei, R., Luo, Y., and Li, K. A Review and Outlook on Predictive Cruise Control of Vehicles and Typical Applications Under Cloud Control System . Machine Intelligence Research, 20 0 (5): 0 614--639, October 2023. ISSN 273...

    2023 doi

  6. [14]

    and Fern \'a ndez, F

    Garc a, J. and Fern \'a ndez, F. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16 0 (1): 0 1437--1480, 2015

    2015

  7. [15]

    Bullet-safety-gym: A framework for constrained reinforcement learning

    Gronauer, S. Bullet-safety-gym: A framework for constrained reinforcement learning. 2022

    2022

  8. [16]

    and Bhatnagar, S

    Guin, S. and Bhatnagar, S. A policy gradient approach for finite horizon constrained markov decision processes. In 2023 62nd IEEE Conference on Decision and Control (CDC), pp.\ 3353--3359. IEEE, 2023

    2023

  9. [17]

    Personalized robotic control via constrained multi-objective reinforcement learning

    He, X., Hu, Z., Yang, H., and Lv, C. Personalized robotic control via constrained multi-objective reinforcement learning. Neurocomputing, 565: 0 126986, 2024

    2024

  10. [18]

    A., Cleac'h, S

    Howell, T. A., Cleac'h, S. L., Br \"u digam, J., Kolter, J. Z., Schwager, M., and Manchester, Z. Dojo: A differentiable physics engine for robotics. arXiv preprint arXiv:2203.00806, 2022

    2022 arXiv

  11. [19]

    Deep differentiable reinforcement learning and optimal trading

    Jaisson, T. Deep differentiable reinforcement learning and optimal trading. Quantitative Finance, 22 0 (8): 0 1429--1443, 2022

    2022

  12. [20]

    Ai alignment: A comprehensive survey

    Ji, J., Qiu, T., Chen, B., Zhang, B., Lou, H., Wang, K., Duan, Y., He, Z., Zhou, J., Zhang, Z., et al. Ai alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852, 2023 a

    2023 arXiv

  13. [21]

    Safety-gymnasium: A unified safe reinforcement learning benchmark

    Ji, J., Zhang, B., Zhou, J., Pan, X., Huang, W., Sun, R., Geng, Y., Zhong, Y., Dai, J., and Yang, Y. Safety-gymnasium: A unified safe reinforcement learning benchmark. arXiv preprint arXiv:2310.12567, 2023 b

    2023 arXiv

  14. [22]

    Omnisafe: An infrastructure for accelerating safe reinforcement learning research

    Ji, J., Zhou, J., Zhang, B., Dai, J., Pan, X., Sun, R., Huang, W., Geng, Y., Liu, M., and Yang, Y. Omnisafe: An infrastructure for accelerating safe reinforcement learning research. arXiv preprint arXiv:2305.09304, 2023 c

    2023 arXiv

  15. [23]

    Aligner: Achieving efficient alignment through weak-to-strong correction

    Ji, J., Chen, B., Lou, H., Hong, D., Zhang, B., Pan, X., Dai, J., and Yang, Y. Aligner: Achieving efficient alignment through weak-to-strong correction. arXiv preprint arXiv:2402.02416, 2024 a

    2024 arXiv

  16. [24]

    Beavertails: Towards improved safety alignment of llm via a human-preference dataset

    Ji, J., Liu, M., Dai, J., Pan, X., Zhang, C., Bian, C., Chen, B., Sun, R., Wang, Y., and Yang, Y. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36, 2024 b

    2024

  17. [25]

    Jie Xu, V., Makoviychuk, Y., and Narang, F. R. Accelerated policy learning with parallel differentiable simulation. In ICLR, 2022

    2022

  18. [26]

    and Langford, J

    Kakade, S. and Langford, J. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, ICML '02, pp.\ 267–274, San Francisco, CA, USA, 2002. Morgan Kaufmann Publishers Inc. ISBN 1558608737

    2002

  19. [27]

    C., Jain, R., and Nuzzo, P

    Kalagarla, K. C., Jain, R., and Nuzzo, P. A sample-efficient algorithm for episodic finite-horizon mdp with constraints. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.\ 8030--8037, 2021

    2021

  20. [28]

    Reparameterization gradient for non-differentiable models

    Lee, W., Yu, H., and Yang, H. Reparameterization gradient for non-differentiable models. Advances in Neural Information Processing Systems, 31, 2018

    2018

  21. [29]

    Constrained variational policy optimization for safe reinforcement learning

    Liu, Z., Cen, Z., Isenbaev, V., Liu, W., Wu, S., Li, B., and Zhao, D. Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning, pp.\ 13644--13668. PMLR, 2022

    2022

  22. [30]

    An off-policy trust region policy optimization method with monotonic improvement guarantee for deep reinforcement learning

    Meng, W., Zheng, Q., Shi, Y., and Pan, G. An off-policy trust region policy optimization method with monotonic improvement guarantee for deep reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 33 0 (5): 0 2223--2235, 2022. doi:10.1109/TNNLS.2020.3044196

    2022

  23. [31]

    D., Schoenholz, S

    Metz, L., Freeman, C. D., Schoenholz, S. S., and Kachman, T. Gradients are not all you need. arXiv preprint arXiv:2111.05803, 2021

    2021 arXiv

  24. [32]

    Monte carlo gradient estimation in machine learning

    Mohamed, S., Rosca, M., Figurnov, M., and Mnih, A. Monte carlo gradient estimation in machine learning. The Journal of Machine Learning Research, 21 0 (1): 0 5183--5244, 2020

    2020

  25. [33]

    Mora, M. A. Z., Peychev, M., Ha, S., Vechev, M., and Coros, S. Pods: Policy optimization via differentiable simulation. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Resear...

    2021

  26. [34]

    A focused backpropagation algorithm for temporal pattern recognition

    Mozer, M. A focused backpropagation algorithm for temporal pattern recognition. Complex Systems, 3, 01 1995

    1995

  27. [35]

    Mozer, M. C. A focused backpropagation algorithm for temporal pattern recognition. In Backpropagation, pp.\ 137--169. Psychology Press, 2013

    2013

  28. [36]

    Muhammad, K., Ullah, A., Lloret, J., Del Ser, J., and de Albuquerque, V. H. C. Deep learning for safe autonomous driving: Current challenges and future directions. IEEE Transactions on Intelligent Transportation Systems, 22 0 (7): 0 4316--4336, 2020

    2020

  29. [37]

    Trajectory planning with miscellaneous safety critical zones**this work was supported by ffi - strategic vehicle research and innovation

    Nilsson, J., Fredriksson, J., and Coelingh, E. Trajectory planning with miscellaneous safety critical zones**this work was supported by ffi - strategic vehicle research and innovation. IFAC-PapersOnLine, 50 0 (1): 0 9083--9088, 2017. ISSN 2405-8963. doi:https://doi.org/10.1016...

    2017 doi

  30. [38]

    M., Smaby, N., and Cutkosky, M

    Okamura, A. M., Smaby, N., and Cutkosky, M. R. An overview of dexterous manipulation. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), volume 1, pp.\ 255--262. IEEE, 2000

    2000

  31. [39]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 0 27730--27744, 2022

    2022

  32. [40]

    Model-based reinforcement learning with scalable composite policy gradient estimators

    Parmas, P., Seno, T., and Aoki, Y. Model-based reinforcement learning with scalable composite policy gradient estimators. In International Conference on Machine Learning, pp.\ 27346--27377. PMLR, 2023 a

    2023

  33. [41]

    Model-based reinforcement learning with scalable composite policy gradient estimators

    Parmas, P., Seno, T., and Aoki, Y. Model-based reinforcement learning with scalable composite policy gradient estimators. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Lea...

    2023

  34. [42]

    and Barr, A

    Platt, J. and Barr, A. Constrained differential optimization. In Neural Information Processing Systems, 1987

    1987

  35. [43]

    E., Perescu-Popescu, L., and Mastorakis, N

    Popescu, M.-C., Balas, V. E., Perescu-Popescu, L., and Mastorakis, N. Multilayer perceptron and neural networks. WSEAS Transactions on Circuits and Systems, 8 0 (7): 0 579--588, 2009

    2009

  36. [44]

    Puterman, M. L. Markov decision processes. Handbooks in operations research and management science, 2: 0 331--434, 1990

    1990

  37. [45]

    Puterman, M. L. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014

    2014

  38. [46]

    D., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023

    2023 arXiv

  39. [47]

    Benchmarking safe exploration in deep reinforcement learning

    Ray, A., Achiam, J., and Amodei, D. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708, 7 0 (1): 0 2, 2019

    1910 arXiv

  40. [48]

    Trust region policy optimization

    Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International conference on machine learning, pp.\ 1889--1897. PMLR, 2015

    2015

  41. [49]

    Tbq ( ): Improving efficiency of trace utilization for off-policy reinforcement learning

    Shi, L., Li, S., Cao, L., Yang, L., and Pan, G. Tbq ( ): Improving efficiency of trace utilization for off-policy reinforcement learning. arXiv preprint arXiv:1905.07237, 2019

    1905 arXiv

  42. [50]

    J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al

    Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. nature, 529 0 (7587): 0 484--489, 2016

    2016

  43. [51]

    Mastering the game of go without human knowledge

    Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge. nature, 550 0 (7676): 0 354--359, 2017

    2017

  44. [52]

    Singh, B., Kumar, R., and Singh, V. P. Reinforcement learning in robotic applications: a comprehensive survey. Artificial Intelligence Review, pp.\ 1--46, 2022

    2022

  45. [53]

    Responsive safety in reinforcement learning by pid lagrangian methods

    Stooke, A., Achiam, J., and Abbeel, P. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pp.\ 9133--9143. PMLR, 2020

    2020

  46. [54]

    J., Simchowitz, M., Zhang, K., and Tedrake, R

    Suh, H. J., Simchowitz, M., Zhang, K., and Tedrake, R. Do differentiable simulators give better policy gradients? In International Conference on Machine Learning, pp.\ 20668--20696. PMLR, 2022

    2022

  47. [55]

    Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018

    2018

  48. [56]

    W., Wang, T., Shang, Y., and Wu, Z

    Wah, B. W., Wang, T., Shang, Y., and Wu, Z. Improving the performance of weighted lagrange-multiplier methods for nonlinear constrained optimization. Information Sciences, 124 0 (1-4): 0 241--272, 2000

    2000

  49. [57]

    Development of a humanoid robot control system based on ar-bci and slam navigation

    Wang, Y., Zhang, M., Li, M., Cui, H., and Chen, X. Development of a humanoid robot control system based on ar-bci and slam navigation. Cognitive Neurodynamics, pp.\ 1--14, 2024

    2024

  50. [58]

    Werling, K., Omens, D., Lee, J., Exarchos, I., and Liu, C. K. Fast and feature-complete differentiable physics engine for articulated rigid bodies with contact constraints. In Robotics: Science and Systems, 2021

    2021

  51. [59]

    Fluidlab: A differentiable environment for benchmarking complex fluid manipulation

    Xian, Z., Zhu, B., Xu, Z., Tung, H.-Y., Torralba, A., Fragkiadaki, K., and Gan, C. Fluidlab: A differentiable environment for benchmarking complex fluid manipulation. arXiv preprint arXiv:2303.02346, 2023

    2023 arXiv

  52. [60]

    Trustworthy reinforcement learning against intrinsic vulnerabilities: Robustness, safety, and generalizability

    Xu, M., Liu, Z., Huang, P., Ding, W., Cen, Z., Li, B., and Zhao, D. Trustworthy reinforcement learning against intrinsic vulnerabilities: Robustness, safety, and generalizability. arXiv preprint arXiv:2209.08025, 2022

    2022 arXiv

  53. [61]

    A unified approach for multi-step temporal-difference learning with eligibility traces in reinforcement learning

    Yang, L., Shi, M., Zheng, Q., Meng, W., and Pan, G. A unified approach for multi-step temporal-difference learning with eligibility traces in reinforcement learning. arXiv preprint arXiv:1802.03171, 2018

    2018 arXiv

  54. [62]

    Constrained update projection approach to safe policy optimization

    Yang, L., Ji, J., Dai, J., Zhang, L., Zhou, B., Li, P., Yang, Y., and Pan, G. Constrained update projection approach to safe policy optimization. Advances in Neural Information Processing Systems, 35: 0 9111--9124, 2022

    2022

  55. [63]

    Yang, T.-Y., Rosca, J., Narasimhan, K., and Ramadge, P. J. Projection-based constrained policy optimization. arXiv preprint arXiv:2010.03152, 2020

    2010 arXiv

  56. [64]

    First order constrained optimization in policy space

    Zhang, Y., Vuong, Q., and Ross, K. First order constrained optimization in policy space. Advances in Neural Information Processing Systems, 33: 0 15338--15349, 2020

    2020

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.