Constrained Policy Optimization with Cantelli-Bounded Value-at-Risk

Jan-Peter Calliess; Rohan Tangri

REVIEW 3 major objections 5 minor 4 references

Constrained Policy Optimization with Cantelli-Bounded Value-at-Risk

T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VaR-CPO replaces a non-differentiable Value-at-Risk constraint with a conservative moment-based surrogate, proves a worst-case training-time violation bound, and reports zero constraint violations in feasible environments.

desk verdict VaR-CPO has a real contribution in the state-augmentation trick and the CPO extension, but the central safety guarantee is proven for an idealized update that Algorithm 1 does not actually solve, and the empirical evidence is too thin to carry the zero-violation claim. read the letter →

arxiv 2601.22993 v4 pith:CLKGG7GH submitted 2026-01-30 cs.LG stat.ML

Rohan Tangri , Jan-Peter Calliess This is my paper · ORCID

classification cs.LGstat.ML

keywords risk-constrainedreinforcementlearningValue-at-Riskone-sidedChebyshevinequalityCantelliconstrainedpolicyoptimizationtrustregionsafeexplorationmoment-basedconstraint

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces VaR-CPO, a reinforcement-learning method that optimizes policies under Value-at-Risk constraints, which cap the probability that cumulative cost exceeds a threshold. Because the VaR objective is non-differentiable and produces sparse failure signals, the authors replace it with a conservative surrogate derived from the one-sided Chebyshev inequality, using only the mean and variance of the cost return. They prove a worst-case bound on constraint violation during the trust-region update, extending the safety guarantees of constrained policy optimization to the tail-risk setting. Empirically, on two continuous-control benchmarks, VaR-CPO satisfies its constraint with zero training-time violations in feasible environments, without ever experiencing a failure.

What carries the argument

The load-bearing object is the one-sided Chebyshev (Cantelli) inequality: for a random variable with mean μ and variance σ², P(C − μ ≥ λ) ≤ σ²/(σ² + λ²). The method enforces the stronger condition that this bound is at most ε, which rearranges to a quadratic constraint on the first two moments of the cost return. To make the second moment tractable, the state is augmented with the discounted accumulated cost y_t, allowing C(τ)² to be written as a discounted sum of local terms; the original constraint becomes an expected-return inequality on an augmented cost. The trust-region update then mirrors constrained policy optimization, with the augmented cost and a dynamic bound, and Theorem 4.1 giv

What would settle it

Run the practical VaR-CPO algorithm on a feasible environment (one where a safe policy is known to exist) over many seeds, recording the 95th percentile of the cost return at every timestep. If any seed shows a violation after the point where the paper reports 'earliest permanent satisfaction', or if the worst observed violation exceeds the bound from Theorem 4.1 computed from the algorithm's own advantage estimates, the paper's central safety claim fails.

Watch

Extended reading notes

Core claim

The central claim is that the intractable VaR constraint P(C(τ) ≥ ρ) ≤ ε can be replaced by the moment condition (1/ε − 1)σ²(π) − (ρ − μ(π))² ≤ 0, which is conservative for any cost distribution with finite mean and variance. The paper shows that optimizing this surrogate inside a trust-region update is safe: Theorem 4.1 bounds the worst-case Chebyshev constraint violation by K(α^{C̃} + (2α^C/ε)(μ(π_k)+α^C/(1−γ_c))), with K = √(2δγ_c)/(1−γ_c)². It also reports that VaR-CPO achieves the strongest practical safety of the compared methods, including zero constraint violations in feasible settings, by learning a conservative mean-variance tradeoff rather than testing the VaR boundary directly.

Load-bearing premise

The proof of the worst-case violation bound assumes the update solves the trust-region subproblem with true expectations, but the practical algorithm uses estimated advantages and moments from sampled trajectories, and the paper does not show the bound survives that estimation; additionally, the Chebyshev surrogate is only valid when μ(π) < ρ, with no proof that the recovery mode returns the policy to that regime.

Editorial extensions

If this is right

Training-time safety: the worst-case bound means each policy update has a controlled constraint violation, extending CPO's guarantee to tail-risk constraints.
Zero-failure learning: in feasible environments, the agent can satisfy the VaR constraint without ever observing a catastrophic failure, because the surrogate is based on moments rather than exceedance events.
Dense gradients: the moment-based surrogate avoids the sparse Bernoulli cost signal of indicator-based VaR methods, making the constraint differentiable and easier to optimize.
Distribution-free conservatism: the Chebyshev bound holds for any cost distribution with finite first two moments, so the method is safe even when the cost tail is unknown or heavy.
Recovery mode: if the expected cost enters the regime μ ≥ ρ where the surrogate is invalid, the algorithm switches to a standard expected-cost constraint to return to the valid region.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The moment-based surrogate and its trust-region bound could be plugged into other constrained RL algorithms as a generic tail-risk constraint module, not just within CPO's update rule.
The state-augmentation trick for decomposing second moments generalizes to higher moments of the discounted return, potentially enabling constraints on skewness or kurtosis, though Chebyshev-type bounds become looser.
The zero-violation result likely depends on the safe region being reachable without crossing the boundary; in environments where the safe set is disconnected or requires transient violation, the method may need a stronger exploration mechanism.
A practical open question is the gap between Theorem 4.1, which assumes true expectations, and the algorithm's use of estimated advantages and moments; measuring this gap empirically would tell whether the guarantee holds for the implemented method.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know first that this is not a paper to dismiss on sight. The state-augmentation decomposition in Eq. 21—turning the squared discounted cost into a local per-step reward—is clean and genuinely new as far as I can tell from the cited work. Extending CPO's worst-case analysis to a Chebyshev surrogate constraint is a sensible theoretical exercise, and the proof of Theorem 4.1 is internally consistent. The method addresses a real gap: dense, differentiable gradients for VaR-like constraints without requiring failure experience.

The soft spots are mostly concentrated in the gap between the theory and the implementation. Theorem 4.1 is proven for the exact solution of the trust-region subproblem under true expectations. The proof relies on the inequality L_C(π_{k+1}) - \hat{d}(π_{k+1}) ≤ 0, which is exactly the constraint in the idealized update. Algorithm 1 instead solves a first-order Taylor approximation (Eqs. 39-41) with GAE-estimated advantages and TD/MC-estimated moments. Satisfying that linearized constraint does not imply the exact nonlinear constraint is satisfied, because the Z^2 term in \hat{d} has zero gradient at θ_k. The paper gives no argument that the bound survives this approximation. This is not a minor omission; it means the headline safety guarantee is unproven for the actual algorithm. The bound also applies to the Chebyshev surrogate J_C, not to the VaR exceedance itself, and the recovery mode for μ(π) ≥ ρ is outside the theorem entirely.

The empirical section is suggestive but weak: two custom environments, five seeds, no code or hyperparameters released. The "zero violations" claim is interesting but it is an empirical observation on a narrow benchmark, not a demonstrated property. There is also a distracting naming inconsistency—the abstract says "Canary" while the body says "VaR-CPO"—which suggests the manuscript is not fully polished.

Who gets value from this paper: researchers working on risk-constrained RL, especially those who want a practical alternative to CVaR with dense gradients. The core idea is worth pursuing, and the state-augmentation trick alone is a useful building block. I would send it to peer review with the expectation of major revision: the theoretical claim needs to be either proven for the linearized update or honestly weakened, and the experiments need more environments, more seeds, and code.

Bottom line: worth a serious referee, not a desk reject.

Referee Report

3 major / 5 minor

Summary. The paper introduces VaR-CPO, a constrained policy optimization method for Value-at-Risk constraints on the discounted cost return in CMDPs. It replaces the non-smooth VaR constraint with a conservative, differentiable surrogate derived from the one-sided Chebyshev (Cantelli) inequality, using the first two moments of the cost return. A state-augmentation scheme makes the second-moment term amenable to a per-step discounted representation. The method is integrated into CPO's trust-region framework, with a claimed worst-case bound on surrogate-constraint violation during training (Theorem 4.1). Empirical evaluations on modified Brax Ant and FrozenLake-style environments compare VaR-CPO with PPO, CPO, and CPPO, reporting zero training-time constraint violations in feasible settings and competitive reward.

Significance. If the theoretical claims held for the implemented algorithm, this would be a useful contribution to risk-constrained RL: it offers a dense-gradient surrogate for VaR constraints, a clean state-augmentation construction, and CPO-style worst-case analysis. The algebraic derivation of the Chebyshev surrogate and the moment decomposition is largely consistent, and the empirical study is clearly motivated and compares meaningful baselines. However, the central safety guarantee is not actually established for Algorithm 1: Theorem 4.1 is proved for an idealized exact trust-region update with true expectations, while the implemented method uses estimated first-order linearizations. In addition, the theorem bounds the Chebyshev surrogate value, not the original VaR exceedance, so its interpretation as a safety guarantee is overstated. These issues are load-bearing, though in principle fixable by adding estimation/linearization error bounds or by substantially narrowing the claimed guarantees.

major comments (3)

[§4.4 and §4.6, Theorem 4.1 vs. Algorithm 1] Theorem 4.1 is proved under the condition L_{\tilde C}(\pi_{k+1}) - \hat d(\pi_{k+1}) \le 0 (Eq. 29), for a policy that exactly solves Eqs. 28–30 with true expectations. Algorithm 1 instead solves the first-order QP (39–41) with GAE/TD/MC estimates. The condition c + b^T(\theta-\theta_k) \le 0 does not imply Eq. 29: the quadratic Z^2 term in \hat d (Eq. 27) has zero gradient at \theta_k and is therefore invisible to the linearized constraint, and no error-propagation or concentration argument for the estimated advantages/moments is supplied. The central claim that the implemented VaR-CPO update is safe is therefore unsupported.
[§4.4, Eq. (31)] The theorem bounds the Chebyshev surrogate J_C = J_{\tilde C} - d, not the original VaR exceedance P(C \ge \rho) - \epsilon. A positive bound on J_C does not translate into a uniform bound on the VaR exceedance without additional control on the variance; for example, when \mu approaches \rho from below, the Cantelli ratio can be close to 1 while J_C is O(1). The abstract and conclusion refer to 'worst-case bounds for constraint violation' and 'safety guarantee' without this caveat, overstating what Eq. (31) establishes.
[§4.5, Recovery mode] The recovery updates (33–35), used when \mu(\pi_k) \ge \rho, are outside the assumptions of Theorem 4.1. No argument is given that the recovery mode reaches the valid regime \mu(\pi_k) < \rho, or that it does so within any bounded number of steps. Since the Chebyshev surrogate (16)–(18) is only valid for \mu < \rho, the safety of the full algorithm during recovery is not guaranteed.

minor comments (5)

[Notation, Eqs. (12) and (18)] The symbol J_C is used for the expected cost return in Eq. (12) and then redefined as the Chebyshev surrogate in Eq. (18). This overloading is confusing, especially in Theorem 4.1 where J_C denotes the surrogate. Please use distinct notation, e.g., J_{\mathrm{Cheb}} or J_{\mathrm{VaR}}^{\mathrm{sur}}.
[Theorem 4.1 statement] The theorem writes \bar D_{KL}(\pi_k,\pi_{k+1}) while the update (30) uses \bar D_{KL}(\pi_{k+1},\pi_k). Since KL is asymmetric, the notation should be made consistent.
[Fig. 2 caption] The legend label 'Var-CPO' is a typo; it should be 'VaR-CPO'.
[§4.4, sentence after Eq. (31)] The phrase 'this approximation is safe' is too strong. Equation (31) gives an upper bound on the surrogate violation, but that upper bound is positive and does not imply zero violation or safety in the original VaR sense. Consider rewording to 'bounded surrogate violation.'
[Algorithm 1, Step 2] The algorithm leaves open the choice between TD and MC estimators for \mu(\theta_k) and J_{\tilde C}(\theta_k). The bias/variance properties of these choices interact with the GAE advantages and with the theoretical bound; a brief discussion or sensitivity check would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivation uses external classical inequalities and CPO/TRPO bounds, with no fitted-to-prediction loop and no load-bearing self-citation.

full rationale

The central derivation chain is not circular. The VaR condition P(C>=rho)<=epsilon (Eq. 13) is relaxed via the classical one-sided Chebyshev/Cantelli inequality into a conservative sufficient condition (Eqs. 16-18), which is an external mathematical result rather than an assumption equivalent to the conclusion. The augmented-state representation (Eqs. 19-23) is derived by exact algebra in Appendices A.2-A.3, and the surrogate boundary dhat(pi) in Eq. 27 is an explicit first-order expansion, not a hidden restatement of the target. Theorem 4.1 (Eq. 31) is proven for the exact trust-region subproblem (Eqs. 28-30) using CPO/TRPO-style advantage, visitation-frequency, and Pinsker bounds; the proof does not use the paper's empirical zero-violation results or fit any constants to observed constraint satisfaction. The algorithm's Taylor/GAE implementation (Eqs. 36-41) is not covered by the theorem, and the recovery mode (Eqs. 33-35) is not analyzed, but these are soundness gaps between theorem and practical algorithm rather than cases where a prediction reduces by construction to an input. There are no self-citations by the authors in the load-bearing argument, and no fitted parameter is relabeled as a prediction. Therefore the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central derivation uses only a classical inequality and CPO's external bounds. The only hand-tuned inputs are hyperparameters such as δ and γ_c, plus the user-chosen VaR threshold ρ and confidence level ϵ. No new physical or model entities are introduced.

free parameters (2)

KL trust-region size δ
Hand-chosen hyperparameter that enters the theoretical bound K = sqrt(2δ)γc/(1-γc)^2 and controls the update step size. Its value is not reported.
Cost discount factor γ_c
Discount applied to the cost return; appears in the worst-case bound and in the state augmentation. Chosen by hand, but no value is stated.

assumptions (4)

standard math One-sided Chebyshev (Cantelli) inequality holds for the discounted cost return C(τ), which has finite first and second moments
Used in Eq. 16 to replace the VaR constraint. Requires finite variance and λ = ρ - μ > 0; the latter is handled by a separate recovery mode.
domain assumption CPO trust-region performance and violation bounds (Achiam et al. 2017) are valid in the augmented MDP
Theorem 4.1 and the inherited reward-degradation bound rely on CPO's bounds on L(π) and L_C(π). The paper does not re-derive these and assumes they extend to the augmented state space.
domain assumption The augmented state x_t = (s_t, y_t, γ_c^t) makes the process Markovian and J_\tilde{C} a standard cumulative cost
Needed for Eqs. 21-23. y_t is a deterministic function of the cost history and γ_c^t injects time, so the augmented state carries the information needed for the second-moment decomposition.
ad hoc to paper Estimated advantages and moment returns are sufficiently accurate for both the update and the worst-case bound
Theorem 4.1 is stated for exact expectations, but Algorithm 1 uses GAE and TD/MC estimates. The paper does not quantify this approximation error.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Constrained Policy Optimization with Cantelli-Bounded Value-at-Risk." pith.science (2026). https://pith.science/paper/CLKGG7GH

@misc{pith2026260122993,
  author       = {Pith},
  title        = {Pith review of: Constrained Policy Optimization with Cantelli-Bounded Value-at-Risk},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CLKGG7GH}},
  note         = {Machine review of arXiv:2601.22993}
}

read the original abstract

We introduce Canary, a risk-averse method designed to optimize Value-at-Risk (VaR) constrained reinforcement learning (RL) problems. We employ Cantelli's inequality to obtain a tractable, conservative and smooth bound on the VaR constraint based on the first two moments of the cost return. This yields a constraint estimator that remains stable with tight violation thresholds in dense cost regimes. Extending the trust-region framework of the Constrained Policy Optimization (CPO) method, we further provide worst-case bounds for both policy improvement and constraint violation during the training process. Empirically, across continuous-control safety benchmarks, Canary most reliably satisfies its constraint, with the fewest violations and the earliest permanent satisfaction, while remaining reward-competitive with other baselines that also satisfy.

Figures

Figures reproduced from arXiv: 2601.22993 by the authors.

**Figure 1.** Conservative Chebyshev Surrogate: The feasible VaR regions for cost threshold ρ = 100 and violation probability ϵ = 0.05. The Chebyshev surrogate is valid for any distribution with finite first and second moments, which requires it to be overly conservative compared to a scenario where the underlying cost distribution is known to be Gaussian for example. ensure that the probability of the discounted cumulative cost,… view at source ↗

**Figure 2.** IcyLake Performance Analysis: Comparison of VaR-CPO (blue), PPO (orange), CPO (green) and CPPO (red) over 1m simulation timesteps. Shaded areas represent one standard deviation across 5 seeds. Figure 2a shows the first 10k timesteps to highlight reward return convergence. ment (Freeman et al., 2021). In this scenario, the agent must maximize forward velocity while managing a limited battery budget and navigating ad… view at source ↗

**Figure 3.** Probability mass function of the IcyLake environment state costs The IcyLake environment is specifically designed to evaluate an agent’s ability to manage tail risk in scenarios where minimizing expected cost leads to unsafe behavior. This environment consists of a 4 × 4 grid based on the classic FrozenLake layout. The agent receives a reward of 1 upon reaching the target state and 0 otherwise. The primary distinct… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: EcoAnt Performance Analysis: Comparison of VaR-CPO (blue), PPO (orange), CPO (green), and CPPO (red) across battery sizes 50 (left - agents start unsafe) and 500 (right - agents start safe). Charts (a-b) show results over 5m timesteps, while (c-d) show 10m timesteps to…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

4 extracted references · 3 linked inside Pith

[2000]

org/CorpusID:854622

URL https://api.semanticscholar. org/CorpusID:854622. Schulman, J., Levine, S., Moritz, P., Jordan, M. I., and Abbeel, P. Trust region policy optimization, 2017a. URL https://arxiv.org/abs/1502.05477. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms, 2017b. URL https://arxiv.org/abs/1707. 06347. S...

arXiv 2018
[2017]

doi: 10.1080/00031305.2016

ISSN 1537-2731. doi: 10.1080/00031305.2016. 1186559. URL http://dx.doi.org/10.1080/ 00031305.2016.1186559. Tagawa, K. Chebyshev inequality based approach to chance constrained portfolio optimization, 2017. URL https: //www.iaras.org/iaras/filedownloads/ ijmcm/2017/001-0009(2017).pdf. Tessler, C., Mankowitz, D. J., and Mannor, S. Reward constrained policy ...

arXiv 2016
[2018]

cc/paper_files/paper/2018/file/ 4fe5149039b52765bde64beb9f674940-Paper

URL https://proceedings.neurips. cc/paper_files/paper/2018/file/ 4fe5149039b52765bde64beb9f674940-Paper. pdf. Freeman, C. D., Frey, E., Raichuk, A., Girgin, S., Mordatch, I., and Bachem, O. Brax - a differentiable physics engine for large scale rigid body simulation, 2021. URL http: //github.com/google/brax. Hambly, B., Xu, R., and Yang, H. Recent advance...

arXiv 2018
[2022]

cc/paper_files/paper/2022/file/ 8 VaR Constrained Policy Optimization c88a2bd0e793550d0e885aa6e31ca277-Paper-Conference

URL https://proceedings.neurips. cc/paper_files/paper/2022/file/ 8 VaR Constrained Policy Optimization c88a2bd0e793550d0e885aa6e31ca277-Paper-Conference. pdf. Ray, A., Achiam, J., and Amodei, D. Benchmarking Safe Exploration in Deep Reinforcement Learning. 2019. Rockafellar, R. and Uryasev, S. Conditional value-at-risk for general loss distributions.Journ...

doi:10.1016/s0378-4266(02)00271-6 2022

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [2000]

org/CorpusID:854622

URL https://api.semanticscholar. org/CorpusID:854622. Schulman, J., Levine, S., Moritz, P., Jordan, M. I., and Abbeel, P. Trust region policy optimization, 2017a. URL https://arxiv.org/abs/1502.05477. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms, 2017b. URL https://arxiv.org/abs/1707. 06347. S...

arXiv 2018

[2] [2017]

doi: 10.1080/00031305.2016

ISSN 1537-2731. doi: 10.1080/00031305.2016. 1186559. URL http://dx.doi.org/10.1080/ 00031305.2016.1186559. Tagawa, K. Chebyshev inequality based approach to chance constrained portfolio optimization, 2017. URL https: //www.iaras.org/iaras/filedownloads/ ijmcm/2017/001-0009(2017).pdf. Tessler, C., Mankowitz, D. J., and Mannor, S. Reward constrained policy ...

arXiv 2016

[3] [2018]

cc/paper_files/paper/2018/file/ 4fe5149039b52765bde64beb9f674940-Paper

URL https://proceedings.neurips. cc/paper_files/paper/2018/file/ 4fe5149039b52765bde64beb9f674940-Paper. pdf. Freeman, C. D., Frey, E., Raichuk, A., Girgin, S., Mordatch, I., and Bachem, O. Brax - a differentiable physics engine for large scale rigid body simulation, 2021. URL http: //github.com/google/brax. Hambly, B., Xu, R., and Yang, H. Recent advance...

arXiv 2018

[4] [2022]

cc/paper_files/paper/2022/file/ 8 VaR Constrained Policy Optimization c88a2bd0e793550d0e885aa6e31ca277-Paper-Conference

URL https://proceedings.neurips. cc/paper_files/paper/2022/file/ 8 VaR Constrained Policy Optimization c88a2bd0e793550d0e885aa6e31ca277-Paper-Conference. pdf. Ray, A., Achiam, J., and Amodei, D. Benchmarking Safe Exploration in Deep Reinforcement Learning. 2019. Rockafellar, R. and Uryasev, S. Conditional value-at-risk for general loss distributions.Journ...

doi:10.1016/s0378-4266(02)00271-6 2022