Tilted Quantile Gradient Updates for Quantile-Constrained Reinforcement Learning

Chenglin Li; Guangchun Ruan; Hua Geng

REVIEW 5 major objections 5 minor 30 references

Tilted Quantile Gradient Updates for Quantile-Constrained Reinforcement Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes Tilted Quantile Policy Optimization (TQPO), which estimates the gradient of a cumulative-cost quantile constraint directly from samples and uses a tilted Lagrange-multiplier update to avoid over-conservative policies…

desk verdict The quantile gradient derivation has a sign error that inverts the safety objective, and the convergence proof analyzes a different algorithm, so TQPO's central claims fail as written despite a reasonable experimental setup. read the letter →

arxiv 2412.13184 v1 pith:UVBHDFTP submitted 2024-12-17 cs.LG cs.AI

Chenglin Li , Guangchun Ruan , Hua Geng This is my paper · ORCID

classification cs.LGcs.AI

keywords quantile-constrainedreinforcementlearningsafevalue-at-risklikelihoodratiogradientestimationLagrangianmethodtiltedupdatechanceconstraintspolicyoptimization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Tilted Quantile Policy Optimization (TQPO), an algorithm for safe reinforcement learning that enforces safety as a quantile constraint on cumulative cost rather than the usual expectation constraint. It shows that the gradient of the quantile can be estimated directly from samples via a likelihood-ratio identity, avoiding the biased expectation-form approximations used by prior work. A tilted update rate for the Lagrange multiplier compensates for the asymmetric distribution of the quantile around the threshold, preventing over-conservative policies. The authors prove convergence to the optimal solution under Lipschitz and step-size assumptions, and demonstrate in Safety Gym tasks that TQPO meets its quantile constraints with safety probability closer to the target and higher return than baselines.

What carries the argument

The load-bearing objects are (i) the sampling-based quantile gradient estimator ∇θ q̂_{1−ε} = −(1/N) Σ_i I(C(s_i,π_θ) ≤ q_k) Σ_t ∇_θ log π_θ(a_t|s_t), which replaces the intractable true quantile gradient; (ii) the tilted multiplier update η = (F_q(d)+δ)/(1+δ) for q ≥ d and η = (1−F_q(d)+δ)/(1+δ) for q < d, which makes the Lagrange multiplier recover quickly from overshoot; and (iii) the three-time-scale stochastic approximation recursion (q fastest, θ medium, λ slowest) that the convergence proof is built on. The proof replaces the implemented clipped PPO loss with the smooth Lagrangian L(θ,λ,q), so the theoretical convergence applies to this idealized surrogate.

What would settle it

Run TQPO on the same Safety Gym tasks with the tilted update replaced by a fixed, square-summable step-size schedule for λ that satisfies Assumption 3, while keeping everything else identical. If the algorithm still achieves the same return and safety probability, the tilted term is not needed for the claimed gains; if performance drops sharply, the implemented update is load-bearing and the convergence proof's idealization misses it.

Watch

Extended reading notes

Core claim

The paper's central claim is that the quantile-constrained RL problem max_θ V(s,π_θ) subject to q_{1−ε}(π_θ) ≤ d can be solved efficiently by estimating the quantile gradient directly. Using the inverse function theorem, the paper writes ∇_θ q_{1−ε} = −∇_θ F(q;π_θ)/f(q;π_θ), estimates the numerator with a likelihood-ratio gradient over indicators I(C(s,π_θ) ≤ q), and drops the positive denominator as a harmless scaling. To keep the Lagrange multiplier from overshooting, it tilts the multiplier step size so that λ decreases roughly nine times faster than it increases, using the CDF of the quantile distribution to set the tilt adaptively. The paper proves that the idealized recursions converge almost surely to the optimal solution under two-time-scale stochastic approximation assumptions, and experiments on Safety Gym tasks show TQPO's safety probability tracks the chosen 1−ε level while its return exceeds the QCPO baseline.

Load-bearing premise

The proof of convergence applies to an idealized version of TQPO in which the adaptive tilted update and clipped PPO loss are replaced by a smooth Lagrangian objective with fixed step sizes; the implemented algorithm may not satisfy those assumptions.

Editorial extensions

If this is right

Safety constraints in RL can be enforced at a chosen violation probability rather than in expectation, giving a safety probability that matches the target level.
Quantile-constrained policies trained with TQPO achieve higher return than the prior QCPO baseline while satisfying the same constraints.
The sampling-based quantile gradient removes the need for distributional assumptions or additive expectation corrections, reducing bias and training time.
The tilted update can be applied to any Lagrangian-based constrained RL algorithm that suffers from asymmetric constraint slack.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: The tilted update is essentially an asymmetric, data-dependent learning rate for the Lagrange multiplier; similar adaptive schemes could improve convergence in other primal-dual algorithms where the constraint slack is skewed.
Editorial inference: The proof's replacement of the implemented clipped PPO loss with the smooth Lagrangian objective means the convergence result is for an idealized variant; establishing convergence for the exact implemented update (or for the adaptive tilted η) is an open question.
Editorial inference: The likelihood-ratio quantile gradient estimator might be transferable to other risk-sensitive objectives such as CVaR or spectral risk measures, where similar density-ratio issues arise.
Editorial inference: A testable extension is to apply the tilted update to QCPO or other quantile methods and measure whether the safety probability tracks the target more tightly, controlling for the gradient estimator.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

You should know that the central derivation of this paper is wrong. In Eq. (8) they define F(q) as the CDF P(C≤q), then compute its gradient as ∇E[I(C≥q)] = -∇E[I(C≤q)]. That is the negative of the correct score-function gradient. The mistake propagates through Eq. (11) and into the implemented advantage in Eq. (16), which subtracts λI(C≤q) from the reward advantage. The correct Lagrangian gradient should add λI(C≤q) (up to the positive density factor). So the algorithm as written pushes the policy toward higher cost, not lower; it is not optimizing the quantile-constrained problem in Eq. (6). This is not a minor typo—it flips the sign of the constraint gradient.

The density term 1/f(q) is also dropped with only the observation that f>0. But f depends on θ, so this changes the gradient direction and magnitude, not just a scaling constant that λ could absorb. The convergence section then explicitly replaces the implemented PPO-style loss with the Lagrangian objective, and Assumption 3 requires step sizes that rule out the adaptive tilted update in Eq. (14). So Theorems 1–2 do not cover the algorithm that actually runs.

In fairness, the paper is not without merit. Quantile-constrained RL is a relevant problem, and the idea of a tilted update to speed up recovery from Lagrange multiplier overshoot is a reasonable heuristic. The experiments are clearly described, with code, ablations, and comparisons against QCPO and PPO-Lag. The ablation showing that the tilted term helps is genuinely useful if the rest were sound.

But the sign error is load-bearing. If the implemented code has the correct sign, this is a fixable write-up error, but the paper as submitted is internally inconsistent: Eq. (8) contradicts its own definition of F(q). A referee would need to check the code to see if the performance results are real. As it stands, the theoretical claims do not hold and the algorithm's actual objective is unclear.

My take: this deserves a serious referee's time because the problem is worthwhile and the errors are concrete, but it should be rejected or sent back for major revision. I would not cite it in its current form.

Referee Report

5 major / 5 minor

Summary. The paper proposes Tilted Quantile Policy Optimization (TQPO), a reinforcement learning method that replaces expectation-form safety constraints with a quantile (value-at-risk) constraint on cumulative cost. The method estimates the quantile gradient via a likelihood-ratio sampling technique, applies a tilted adaptive update to the Lagrange multiplier to compensate for asymmetric quantile distributions, and is evaluated on three Safety Gym tasks against QCPO and PPO-Lag. The paper also claims convergence guarantees via two-timescale stochastic approximation. The central claims are that TQPO directly estimates quantile gradients without expectation approximations, achieves higher return while satisfying quantile constraints, and provably converges to the optimal solution.

Significance. The problem of enforcing high-probability safety constraints in RL is important, and the paper addresses a meaningful gap by avoiding expectation-form approximations. The experimental study is reasonably extensive, includes ablations, and the code is released. However, the central theoretical claims are undermined by a sign error in the quantile-gradient derivation, by the unjustified dropping of the density term, and by a convergence analysis that analyzes a different update rule than the one implemented. As a result, the paper's main claims about direct quantile-gradient estimation and provable convergence are not supported by the current manuscript.

major comments (5)

[Estimating Quantile Gradients Through Sampling, Eq. (8)] Equation (8) states that grad_theta F(q; pi_theta) = grad_theta E[I(C >= q)] = -grad_theta E[I(C <= q)]. Since F(q; pi_theta) is defined in Eq. (5) as E[I(C <= q)], the correct score-function identity is grad_theta F = +E[I(C <= q) grad_theta log pi], not its negative. This sign error flips the quantile gradient in Eq. (7) and propagates to Eq. (11), producing an advantage function in Eq. (16) that subtracts lambda * I(C <= q). The implemented algorithm therefore penalizes trajectories whose cost is below the quantile, which pushes the policy toward higher cost and is the opposite of the intended safety gradient. This is a load-bearing error because the algorithm is not optimizing the quantile-constrained objective in Eq. (6).
[Estimating Quantile Gradients Through Sampling, after Eq. (7)] The paper drops the density f(q) from the denominator of Eq. (7) with the justification that the PDF is always positive. Positivity only preserves the directional sign of the gradient, not its magnitude. In the Lagrangian objective (10), the relative weight between the return gradient and the constraint gradient matters, and f(q) depends on the policy parameter theta. The approximation therefore changes the effective objective in a way that a Lagrange multiplier rescaling cannot absorb. This is a second load-bearing issue that invalidates the claimed equivalence between the sampled update and the true quantile-constrained gradient.
[Convergence Analysis, first paragraph] The paper explicitly states that, for the convenience of theoretical analysis, it replaces the implemented loss L_theta in Eq. (15) with the Lagrangian objective L(theta, lambda, q) in Eq. (10). Consequently, Theorems 1 and 2 analyze a different algorithm than the one whose empirical performance is reported. No argument is given that the PPO clipped surrogate objective in Eq. (15) behaves like the Lagrangian gradient in the limit, so the convergence proofs do not cover the actual TQPO updates.
[Tilted Quantile Gradient Update, Eq. (14), and Assumption 3] The tilted update rate eta in Eq. (14) is computed from F_q(d), the empirical CDF of the quantile being optimized, making the rate data-dependent and self-referential. Assumption 3 requires deterministic positive, nonsummable, square-summable step-size sequences with eta_k = o(beta_k) and beta_k = o(alpha_k). The adaptive eta produced by Eq. (14) does not satisfy these conditions, and the paper provides no alternative analysis for the adaptive scheme. The convergence theorems therefore do not cover the tilted updates that are central to the reported performance gains.
[Convergence Analysis, Lemma 3 and Theorem 2] Lemma 3 assumes that L(theta, q) is strictly concave on Theta, which is not satisfied by typical neural policy parameterizations and is not justified in the paper. Theorem 2 is asserted by invoking Borkar (2008) without a formal statement or proof; since the appendix is omitted from the manuscript, the reader cannot verify the claimed convergence to the optimal solution of the constrained problem. These are additional load-bearing gaps in the theoretical contribution.

minor comments (5)

[Theorem 1] In the statement of Theorem 1, the update is written as theta_{k+1} = theta + beta_k(...), which should presumably be theta_{k+1} = theta_k + beta_k(...).
[Theorem 1, condition (iii)] The condition "P_infinity alpha_k m_k, P_infinity beta_k n_k < infinity" is ambiguous: it is not clear whether the sums are meant to converge absolutely, almost surely, or in probability, and no normalization is specified.
[Tilted Quantile Gradient Update, Eq. (14)] The definition of F_q(d) is not precise: it is described as the CDF of the distribution of the quantile q_{1-epsilon}(pi_theta) at d, but the manuscript does not specify how this distribution is formed or how the per-epoch sampling estimate is computed.
[Abstract and Conclusion] The abstract and conclusion state that the proposed model 'fully guarantees safety' or 'fully satisfies the quantile constraints', but the experimental safety probabilities reported in Table 1 are in the range 89-97%, which are close to but not always above the specified levels.
[Introduction, related work on QCPO] The statement that QCPO 'required the cumulative cost distribution' is vague; a precise comparison of the distributional assumptions made by QCPO and TQPO would help the reader evaluate the claimed advantage.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TQPO's derivation chain is self-contained; the proof/algorithm mismatch and sign issue are correctness concerns, not circular reductions.

full rationale

The paper's central derivation does not reduce to its own inputs. The quantile-gradient estimator is built from the inverse function theorem applied to the CDF and a likelihood-ratio (score-function) estimate of the CDF gradient, which is a standard external technique (Glynn et al. 2021; Hong and Liu 2009) rather than a restatement of TQPO's conclusion. The quantile q is updated by a Robbins-Monro-style recursion (Eqs. 9 and 17a), the policy by a Lagrangian-style gradient update, and the convergence argument invokes standard two-timescale stochastic approximation results (Borkar 1997, 2008; Gattami et al. 2021) with no load-bearing self-citations. The tilted rate eta in Eq. (14) is data-dependent, and the convergence section explicitly replaces the implemented PPO loss L_theta with the Lagrangian objective L(theta, lambda, q) while assuming step-size conditions in Assumption 3 that the adaptive eta does not satisfy; this is a rigor gap between the proof and the implemented algorithm, not a case where a prediction is equivalent to an input by construction. There is also no renaming of a known result as a new derivation: the likelihood-ratio quantile sensitivity is cited to prior external literature, and the empirical comparisons are against external benchmarks. Accordingly, no specific circular step can be quoted, and the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 8 assumptions · 0 invented entities

The central convergence claim relies on Assumptions 1-4 and Slater's condition, none of which are verified in the experiments. The tilted update introduces a data-dependent learning rate that violates Assumption 3. The gradient estimator drops the density f(q) and contains a sign inconsistency in Eq. (8).

free parameters (4)

delta (smoothing factor in tilted update) = 0.1 (example in text)
Used in Eq. (14) to define eta+ and eta-; chosen by hand, not derived.
update rates alpha_k, beta_k, eta_k = Not specified in paper
Learning rates for quantile q, policy theta, and Lagrange multiplier lambda; treated as hyperparameters.
quantile update rate alpha (Eq. 9) = Not specified
Smoothing rate for the moving average estimate of the quantile.
Lagrange multiplier update rate eta (Eq. 13) = Adaptive, depends on F_q(d)
In the tilted update, eta becomes a data-dependent schedule rather than a fixed constant.

assumptions (8)

domain assumption The cumulative cost distribution F(q; pi_theta) is continuous and differentiable.
Stated as 'a minor assumption' before Eq. (7); needed for the inverse function theorem.
ad hoc to paper Assumption 1: L(theta, lambda, q) is continuous and differentiable with respect to theta.
Imposed in the Convergence Analysis section without verification.
ad hoc to paper Assumption 2: grad_theta L is Lipschitz continuous w.r.t. theta, lambda, and q.
Standard stochastic approximation condition, but not verified for the neural network policy.
ad hoc to paper Assumption 3: update rates are positive, nonsummable, square summable, with eta_k = o(beta_k) and beta_k = o(alpha_k).
Required for the three-timescale proof, but the tilted eta is adaptive and does not satisfy these conditions.
ad hoc to paper Assumption 4: the log gradient of the policy is bounded on the state space.
Used in the proof of Theorem 1; not guaranteed for common policy parameterizations.
ad hoc to paper Slater's condition holds for the quantile-constrained problem.
Invoked in Theorem 2 to ensure convergence of the Lagrangian method.
domain assumption The probability density f(q; pi_theta) is positive so that the gradient direction is preserved after dropping it.
The paper drops f(q) in Eq. (11) with only a directional justification.
domain assumption Finite-horizon episode cumulative cost approximates the infinite-horizon discounted cumulative cost.
Experiments use episodes of 1000 steps; the quantile constraint is defined on the infinite-horizon cost.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Tilted Quantile Gradient Updates for Quantile-Constrained Reinforcement Learning." pith.science (2026). https://pith.science/paper/UVBHDFTP

@misc{pith2026241213184,
  author       = {Pith},
  title        = {Pith review of: Tilted Quantile Gradient Updates for Quantile-Constrained Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UVBHDFTP}},
  note         = {Machine review of arXiv:2412.13184}
}

read the original abstract

Safe reinforcement learning (RL) is a popular and versatile paradigm to learn reward-maximizing policies with safety guarantees. Previous works tend to express the safety constraints in an expectation form due to the ease of implementation, but this turns out to be ineffective in maintaining safety constraints with high probability. To this end, we move to the quantile-constrained RL that enables a higher level of safety without any expectation-form approximations. We directly estimate the quantile gradients through sampling and provide the theoretical proofs of convergence. Then a tilted update strategy for quantile gradients is implemented to compensate the asymmetric distributional density, with a direct benefit of return performance. Experiments demonstrate that the proposed model fully meets safety requirements (quantile constraints) while outperforming the state-of-the-art benchmarks with higher return.

Figures

Figures reproduced from arXiv: 2412.13184 by the authors.

**Figure 1.** Safety Gym simulation environments randomness. In SimpleEnv (Fig. 1a), the obstacles include fixed hazards (blue) and none-goal buttons (orange). When the robot reach the goal (orange covered by grey shadow), the environment swaps the goal and the none-goal button, therefore the new goal is generated deterministically. In DynamicEnv (Fig. 1b), when the robot reaches the goal (green), a new goal is generated randoml… view at source ↗

**Figure 2.** Average Cost (Row 1) and Cost Quantile (Row 2) of three algorithms on SimpleEnv (Column 1), DynamicEnv [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Return (Row 1) and Safety Probability (Row 2) of three algorithms on SimpleEnv (Column 1), DynamicEnv (Column [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Distributions of quantile q1−ε w.o. (top) and w. (bottom) tilted term. The black vertical dashed line is the threshold d, ∆λ + is the increase of λ when q1−ε ≥ d, ∆λ − represent the decrease of λ when q1−ε < d. Algorithms SimpleEnv 95% DynamicEnv 95% R Pr R Pr QCPO 2.1…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 22 canonical work pages

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Achiam, J.; Held, D.; Tamar, A.; and Abbeel, P. 2017. Constrained policy optimization. In International conference on machine learning, 22--31. PMLR

2017
[4]

Alshiekh, M.; Bloem, R.; Ehlers, R.; K \"o nighofer, B.; Niekum, S.; and Topcu, U. 2018. Safe reinforcement learning via shielding. In Proceedings of the AAAI conference on artificial intelligence, volume 32

work page 2018
[5]

Altman, E. 2021. Constrained Markov decision processes. Routledge

work page 2021
[6]

Borkar, V. S. 1997. Stochastic approximation with two time scales. Systems & Control Letters, 29(5): 291--294

work page 1997
[7]

Borkar, V. S. 2008. Stochastic approximation: a dynamical systems viewpoint, volume 9. Springer

work page 2008
[8]

Carr, S.; Jansen, N.; Junges, S.; and Topcu, U. 2023. Safe reinforcement learning via shielding under partial observability. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 14748--14756

work page 2023

Show all 30 references

[9]

Chen, W.; Subramanian, D.; and Paternain, S. 2024. Probabilistic constraint for safety-critical reinforcement learning. IEEE Transactions on Automatic Control

2024
[10]

M.; and Burdick, J

Cheng, R.; Orosz, G.; Murray, R. M.; and Burdick, J. W. 2019. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, 3387--3395

2019
[11]

Chow, Y.; Ghavamzadeh, M.; Janson, L.; and Pavone, M. 2018. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18(167): 1--51

2018
[12]

Gattami, A.; Bai, Q.; and Aggarwal, V. 2021. Reinforcement learning for constrained markov decision processes. In International Conference on Artificial Intelligence and Statistics, 2656--2664. PMLR

2021
[13]

W.; Peng, Y.; Fu, M

Glynn, P. W.; Peng, Y.; Fu, M. C.; and Hu, J.-Q. 2021. Computing sensitivities for distortion risk measures. INFORMS Journal on Computing, 33(4): 1520--1532

2021
[14]

J.; and Liu, G

Hong, L. J.; and Liu, G. 2009. Simulating sensitivities of conditional value at risk. Management Science, 55(2): 281--293

2009
[15]

Jiang, G.; and Fu, M. C. 2015. On estimating quantile sensitivities via infinitesimal perturbation analysis. Operations Research, 63(2): 435--441

2015
[16]

Jiang, J.; Peng, Y.; and Hu, J. 2022. Quantile-based policy optimization for reinforcement learning. In 2022 Winter Simulation Conference (WSC), 2712--2723. IEEE

2022
[17]

Jung, W.; Cho, M.; Park, J.; and Sung, Y. 2022. Quantile constrained reinforcement learning: A reinforcement learning framework constraining outage probability. Advances in Neural Information Processing Systems, 35: 6437--6449

2022
[18]

Liang, Q.; Que, F.; and Modiano, E. 2018. Accelerated primal-dual policy optimization for safe reinforcement learning. arXiv preprint arXiv:1802.06480

2018 arXiv
[19]

Liu, Y.; Ding, J.; and Liu, X. 2020. Ipo: Interior-point policy optimization under constraints. In Proceedings of the AAAI conference on artificial intelligence, volume 34, 4940--4947

2020
[20]

Marvi, Z.; and Kiumarsi, B. 2021. Safe reinforcement learning: A control barrier function optimization approach. International Journal of Robust and Nonlinear Control, 31(6): 1923--1940

2021
[21]

Paternain, S.; Chamon, L.; Calvo-Fullana, M.; and Ribeiro, A. 2019. Constrained reinforcement learning has zero duality gap. Advances in Neural Information Processing Systems, 32

2019
[22]

E.; Xie, G.; Zhang, C.; Guan, Y.; Mu, Y.; and Sun, E

Peng, B.; Duan, J.; Chen, J.; Li, S. E.; Xie, G.; Zhang, C.; Guan, Y.; Mu, Y.; and Sun, E. 2022. Model-based chance-constrained reinforcement learning via separated proportional-integral lagrangian. IEEE Transactions on Neural Networks and Learning Systems, 35(1): 466--478

2022
[23]

Pfrommer, S.; Gautam, T.; Zhou, A.; and Sojoudi, S. 2022. Safe reinforcement learning with chance-constrained model predictive control. In Learning for Dynamics and Control Conference, 291--303. PMLR

2022
[24]

Ray, A.; Achiam, J.; and Amodei, D. 2019. Benchmarking Safe Exploration in Deep Reinforcement Learning

2019
[25]

Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

2017 arXiv
[26]

Steinwart, I.; and Christmann, A. 2011. Estimating conditional quantiles with the help of the pinball loss

2011
[27]

Stooke, A.; and Abbeel, P. 2019. rlpyt: A research code base for deep reinforcement learning in pytorch. arXiv preprint arXiv:1909.01500

2019 arXiv
[28]

J.; and Mannor, S

Tessler, C.; Mankowitz, D. J.; and Mannor, S. 2018. Reward constrained policy optimization. arXiv preprint arXiv:1805.11074

2018 arXiv
[29]

Todorov, E.; Erez, T.; and Tassa, Y. 2012. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, 5026--5033. IEEE

2012
[30]

Yang, T.-Y.; Rosca, J.; Narasimhan, K.; and Ramadge, P. J. 2020. Projection-based constrained policy optimization. arXiv preprint arXiv:2010.03152

2020 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Achiam, J.; Held, D.; Tamar, A.; and Abbeel, P. 2017. Constrained policy optimization. In International conference on machine learning, 22--31. PMLR

2017

[4] [4]

Alshiekh, M.; Bloem, R.; Ehlers, R.; K \"o nighofer, B.; Niekum, S.; and Topcu, U. 2018. Safe reinforcement learning via shielding. In Proceedings of the AAAI conference on artificial intelligence, volume 32

work page 2018

[5] [5]

Altman, E. 2021. Constrained Markov decision processes. Routledge

work page 2021

[6] [6]

Borkar, V. S. 1997. Stochastic approximation with two time scales. Systems & Control Letters, 29(5): 291--294

work page 1997

[7] [7]

Borkar, V. S. 2008. Stochastic approximation: a dynamical systems viewpoint, volume 9. Springer

work page 2008

[8] [8]

Carr, S.; Jansen, N.; Junges, S.; and Topcu, U. 2023. Safe reinforcement learning via shielding under partial observability. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 14748--14756

work page 2023