Boosting CVaR Policy Optimization with Quantile Gradients

Erick Delage; Yudong Luo

REVIEW 3 major objections 5 minor 5 references

Boosting CVaR Policy Optimization with Quantile Gradients

T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Augmenting CVaR policy gradients with an expected-quantile term lets risk-averse reinforcement learning use all sampled trajectories instead of only the worst tail.

desk verdict The empirical claim is credible, but the VaR-gradient theory is much weaker than the paper's presentation suggests. read the letter →

arxiv 2601.22100 v3 pith:QXY2NOMK submitted 2026-01-29 cs.LG

Yudong Luo , Erick Delage This is my paper · ORCID

classification cs.LG

keywords CVaRvalue-at-riskpolicygradientsampleefficiencyrisk-aversereinforcementlearningquantileregressionBellmanoperatorMarkovianpolicies

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that CVaR policy gradients are sample-inefficient because they discard all but the worst α-fraction of trajectories, and that adding an expected quantile (VaR) term to the objective fixes this. Quantile optimization has a Bellman-style dynamic programming form, so every sampled transition can contribute through a learned quantile value function. The paper derives a VaR Bellman operator, adapts it to Markovian policies via cumulative-reward tracking, and combines the resulting VaR gradient with the standard CVaR gradient. In three domains with identifiable risk-averse behavior, the combined algorithm learns risk-averse policies faster and more reliably than CVaR-PG and prior sample-efficient CVaR methods. A sympathetic reader would care because the method offers a practical, on-policy way to optimize a static tail-risk measure without throwing away most of the data.

What carries the argument

The central object is the augmented objective max_{π∈Π_M} ω CVaR_α[g0] + (1−ω) E_{β∼U[0,α]}[VaR_β[g0]], where CVaR is the average of the worst α tail and VaR is the β-quantile. The argument runs through a VaR Bellman operator on the state-quantile value function v(s,α), built from quantile-regression losses; a cumulative-reward tracking rule that updates the quantile level as α_t = min{β | v(s_t,β) ≥ (v(s_{t-1},α_{t-1}) − r_{t-1})/γ}; and a Markovian advantage term Ā(s_t, α_t, a_t) = E[∂l_{α_t}(δ_v)] that supplies the VaR policy gradient. The VaR component is what lets all trajectories—including successful ones—contribute to policy updates, while the CVaR component keeps the worst-case tail

What would settle it

In a tabular MDP with known optimal Markovian VaR policy, compute the quantile value function of that policy and run the cumulative-reward tracking procedure; if it ever selects an action different from the optimal one, or if two actions tie at the tracked quantile level, the recovery claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that optimizing CVaR at risk level α together with E_{β∼U[0,α]}[VaR_β] preserves the CVaR objective while converting the problem from a tail-only gradient estimator to one that can use all sampled data. The authors prove that a new VaR Bellman operator v(s,α) ← v(s,α) + η max_a E[∂l_{α}(δ_v)] has the optimal state-quantile value as its fixed point, and they assert that a unique stationary Markovian α0-VaR optimal policy can be executed by tracking cumulative rewards against this value function. Combining the resulting VaR policy gradient with the CVaR-PG estimator yields their CVaR-VaR algorithm, which the experiments show attains both high expe

Load-bearing premise

The load-bearing premise is that a unique stationary Markovian VaR-optimal policy can be recovered by tracking cumulative rewards against the learned quantile value function—asserted by definition in Proposition 3.3—while the paper itself states that the convergence of Markovian VaR updates from an arbitrary value function remains unknown.

Editorial extensions

If this is right

Risk-averse policies for static CVaR can be learned with on-policy gradient methods that use every sampled trajectory instead of only the worst αN, reducing the sample cost of tail-risk optimization.
The approach directly addresses the blindness-to-success problem: high-return trajectories now shape the quantile value function and the policy, rather than being discarded.
The VaR Bellman operator gives a dynamic-programming template for quantile value learning in actor-critic settings, with contraction guarantees under a soft quantile loss.
In the tested domains, the combined algorithm reaches high expected return and high risk-averse rates where CVaR-PG fails or converges slowly, and it outperforms or matches existing sample-efficient CVaR baselines.
The VaR-PG component can be combined with other sample-efficiency techniques such as multi-step advantage estimation, and can be annealed as training proceeds.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the open convergence question is resolved, the same augmentation could make static CVaR optimization practical in much larger MDPs; until then, the gains rest on empirical support rather than a proven convergence guarantee.
A similar augmentation may transfer to other tail-dependent risk measures that are integrals of quantiles, because those also admit quantile-regression-style formulations.
The practical bottleneck is likely the accuracy of the learned quantile value function: quantile crossing and coarse discretization of α could hurt cumulative-reward tracking more than the original tail-selection issue.
One testable extension is to replace the uniform draw of initial α_0 with a learned distribution over risk levels, which could adapt the trade-off between CVaR and VaR per episode.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take. The paper's empirical story is believable: adding a quantile-based advantage term to CVaR-PG does make learning faster and more stable in the three toy domains, and the authors are honest that the convergence theory is open. But the theoretical grounding for that quantile term does not survive close reading. Eq. 7 is a rewrite of the same CVaR objective—E_{β}[VaR_β]=CVaR_α—so this isn't a new objective. The real claim is that g2 is a useful gradient estimator. That may be true in practice, but the proof of Prop 3.3 rests on an equality that is asserted rather than shown: q^{π̄*}(s_t,α_t,π̄*(s_t))=v*(s_t,α_t). This is exactly the kind of fact that needs proof. The paper's response that it holds 'by definition of stationary and Markovian' conflates local optimality within Π_M with global optimality. The proof reads as circular, and the appendix's open-convergence note makes clear the authors know this.

What's actually new: the state-only VaR Bellman operator (Eq. 8), the cumulative-reward tracking adaptation to Markovian policies, and the CVaR+VaR gradient combination. These are real ideas and worth testing. The experiments are cleanly run for what they are: three domains with verifiable risk-averse behavior, 10 seeds, and standard comparison baselines. The improvement in risk-averse rate and CVaR is consistent.

Two smaller problems. First, Algorithm 4's update uses g1+g2 unweighted, while the appendix says ω starts at 0.5 and decays to 0. That mismatch makes the actual method underdetermined. Second, there is no code, and the hyperparameter selection is reported as a set, not the specific values used—so replication is harder than it should be.

Bottom line: this deserves a serious referee, but the paper should not be accepted as is. The authors need to either prove Prop 3.3 under stated assumptions (or weaken its claim), clarify the ω schedule in the algorithm text, and ideally release code. If the theory remains speculative, reframing the contribution as an empirical heuristic would be the honest move. I'd take it to a reading group; there's plenty to discuss in the gap between what it claims and what it shows.

Referee Report

3 major / 5 minor

Summary. The paper proposes a method to improve the sample efficiency of CVaR policy gradient (CVaR-PG) by adding a quantile/VaR policy-gradient component. It defines the objective in Eq. (7) as a weighted combination of CVaR and an expected VaR term, derives a soft VaR Bellman operator (Propositions 3.1 and 3.2), attempts to reduce VaR optimization to the Markovian policy class (Propositions 3.3 and 3.4), and combines the standard CVaR-PG estimator with a quantile-value-based advantage estimator. Experiments on Maze, LunarLander, and InvertedPendulum report that the combined method learns faster and achieves higher risk-averse rates and CVaR values than CVaR-PG, PCVaR-PG, RET-CAP, and MIX.

Significance. If the empirical results hold, the method is practically attractive: it suggests a way to use all sampled trajectories in risk-averse policy optimization and to overcome the 'blindness to success' of CVaR-PG. The experimental domains are chosen to have verifiable risk-averse behavior, and the reported curves are consistent across seeds. The paper also provides contraction and fixed-point results for a soft VaR Bellman operator, which is a useful technical step. However, the theoretical foundation for the VaR component that drives the claimed improvement is not established: Eq. (7) is mathematically identical to CVaR, and Proposition 3.3, the key result connecting the Markovian VaR policy to the quantile value function, is asserted rather than proven. The paper explicitly leaves convergence open. The current contribution is therefore best read as an empirical heuristic, but the manuscript presents it as a grounded optimization method.

major comments (3)

[§3, Eq. (7)] The proposed 'augmented' objective does not augment CVaR. Since E_{β~U[0,α]} VaR_β[g0] = (1/α)∫_0^α VaR_β dβ = CVaR_α[g0] by Eq. (1), Eq. (7) equals CVaR_α[g0] for every ω∈(0,1). Thus the second term is not a separate objective component; the abstract's statement that 'this does not alter the CVaR objective' is true, but it also means the method's theoretical claim must be recast: g2 is an additional gradient estimator for the same CVaR objective. No unbiasedness or consistency proof for g2 is given. As written, the theoretical motivation for the algorithm is incomplete: if g2 is biased, the combined update need not optimize CVaR.
[§3.2, Prop. 3.3] The load-bearing equality q^{π̄*}(s_t,α_t,π̄*(s_t)) = v*(s_t,α_t) is not proved. Optimality of π̄* within the Markovian class Π_M only concerns the initial risk level α0; it does not imply that π̄* realizes the unrestricted optimal quantile value v* at every (s,α) on the Algorithm 2 trajectory. The assertion 'by definition of stationary and Markovian' is insufficient. The contradiction step is also invalid: q^{π̄*} < v* does not imply that some alternative action b at s_t achieves v*(s_t,α_t), since v* may be attained only by a history-dependent policy. The proof further assumes monotonicity of max_b q^{π̄*}(s_1,β,b) in β and uses the equality at s_1 in the alpha-selection step, which is the very property being proved. Because Proposition 3.4 and the update rule in Eqs. (12)–(13) inherit this result, the VaR-PG component is not shown to be a valid descent direction. The limitation statem
[§3.3 and Algorithm 4] The trade-off parameter ω does not appear in Algorithm 4. The text says the two gradients are 'combined via the trade-off parameter ω', but the displayed update is θ ← θ + lr(g1 + g2). In Appendices C.3 and C.4, ω is set to 0 after 40%/25% of iterations; under the displayed update this schedule has no effect. The reader cannot determine whether the evaluated method is g1+g2, ωg1+(1−ω)g2, or another combination. This must be corrected, and the actual ω schedule used in the experiments should be reported in the algorithm pseudocode.

minor comments (5)

[Throughout] There are several typos and inconsistencies: 'Markovain' (§2.2), 'future rerturn' (§2.2), 'deep neutral networks' (Appendix C.1), and inconsistent table labels 'CVAR-VAR' versus 'CVaR-VaR' (Tables 1–3).
[Prop. 3.2] The proposition statement does not mention the density assumption on r(s,a)+γv*(s',u) that is used in the proof (Appendix A.4). Please state the assumption explicitly in the proposition.
[Appendix A.2] The definition of p_i is malformed: 'P(˜r(s, a) = ri|s, a, si)' appears inside the definition of P(˜y=i|s,a) and should be written more carefully.
[Algorithm 4] The projection α ← proj(α, Λ) is not defined. Specify whether this is nearest-neighbor, floor, or some other mapping to the discretized quantile levels.
[Experiments] No sensitivity analysis is reported for the number of quantiles I or the trade-off ω. Since the method's name and motivation depend on these choices, a small ablation would strengthen the empirical claims.

Circularity Check

3 steps flagged · score 6.0 of 10

The 'augmented' objective in Eq. 7 is CVaR by Eq. 1, and Prop. 3.3's key equality is assumed 'by definition of stationary and Markovian,' so the VaR-PG grounding is definitionally circular; empirical comparisons remain independent.

self definitional [Sec. 3, Eq. (7); cf. Sec. 2.1, Eq. (1)]
"we consider the following objective max_{π∈Π_M} ω CVaR_α[g0] + (1−ω) E_{β∼U[0,α]} VaR_β[g0] ... This does not alter the CVaR objective since CVaR corresponds to the expectation of quantile over the tail."

By the paper's own Eq. (1), CVaR_α[g0] = (1/α)∫_0^α VaR_β[g0] dβ = E_{β∼U[0,α]} VaR_β[g0]. Therefore Eq. (7) equals ω CVaR_α[g0] + (1−ω) CVaR_α[g0] = CVaR_α[g0] for every ω ∈ (0,1). The 'expected quantile term' is not an independent objective but is CVaR by definition. Any gradient estimator for that term is, by construction, a gradient estimator for the same CVaR objective, so the claimed augmentation reduces to the original objective.
self definitional [Sec. 3.2, Proposition 3.3 and its proof]
"The key to proof is to show that q^{¯π∗}(s_t, α_t, ¯π∗(s_t)) = v∗(s_t, α_t). ... Recall that by definition ¯π∗(s_t) = ˆπ∗(s_t, α_t)."

The proposition's central equality says that the quantile value of the stationary Markovian α0-VaR optimal policy equals the global optimal quantile value along the Algo. 2 tracking path. The proof introduces this as 'by definition of stationary and Markovian' — i.e., it assumes that the Markovian optimal policy coincides with the global quantile-optimal action at every encountered (s_t, α_t). That coincidence is the substantive content to be established, not a definitional consequence. The subsequent contradiction uses this assumed equality to conclude the equality, so Prop. 3.3 reduces to its own assumption rather than proving it.

1 more flagged steps

self definitional [Sec. 3.2, Proposition 3.4 proof]
"From the proof of Proposition 3.3, we know max_a q^{¯π∗}(s_t, α_t, a) = v^{¯π∗}(s_t, α_t) = v∗(s_t, α_t)."

Proposition 3.4's objective in Eq. (11) is justified entirely through the equality inherited from Proposition 3.3. Since that equality was only assumed 'by definition' (and not independently derived), the claimed optimality of ¯π∗ for the Markovian VaR objective in Eq. (11) is not an independent result; it is the same circular assumption restated as an optimization objective. Thus the VaR-PG component is not established as a grounded gradient of Eq. (7).

full rationale

The empirical section is not circular: CVaR-VaR is compared against external baselines (CVaR-PG, PCVaR-PG, RET-CAP, MIX, REINFORCE) in domains with verifiable risk-averse behavior, and those results could in principle falsify the sample-efficiency claim. The score is elevated not because of the experiments but because the theoretical derivation of the VaR-PG component rests on two reduction-by-definition steps. First, the 'augmented' objective in Eq. (7) is exactly CVaR_α by the paper's own Eq. (1), so the proposed second term is the same objective renamed as an expected quantile. Second, Proposition 3.3's key equality q^{¯π∗} = v∗ is asserted 'by definition of stationary and Markovian,' and Proposition 3.4 then inherits that equality; the proof gives no independent argument that Markovian α0-VaR optimality implies per-(s,α) quantile optimality. The paper itself states in the Limitations that convergence of the Markovian VaR updates 'remain unknown,' which confirms that this circular/assumed equality is load-bearing rather than a side remark. Self-citations to Hau et al. (2025) are present and are used for the VaR Bellman framework, but the more direct circularity is the definitional equivalence of Eq. (7) and the assumed equality in Prop. 3.3. Overall, the central theoretical grounding of the added gradient is partially circular, while the empirical contribution retains independent content, so a score of 6 is appropriate.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central algorithm depends on standard quantile/CVaR mathematics plus several hand-set hyperparameters and two unproved structural assumptions: the Prop 3.3 equality and the convergence-of-iteration assumption that the authors themselves flag as open. No new physical or mathematical entities are introduced.

free parameters (5)

omega (trade-off) = 0.5 in Maze; 0.5 decaying to 0 in LunarLander and InvertedPendulum
Chosen by hand to weight CVaR and VaR gradient components. The objective Eq. 7 is mathematically independent of omega, but the gradient combination in experiments depends on it, and Algorithm 4 omits it entirely.
Number of quantiles I = 10
Discretization of quantile levels for the value network; chosen without a sensitivity study.
Multi-step lambda = 0.95
Used for advantage estimation and weighted quantile regression; standard but hand-set.
Policy learning rate = 5e-4 or 3e-4 per domain
Selected from {7e-4, 5e-4, 3e-4, 1e-4, 7e-5} per domain; no fixed rule or sensitivity analysis reported.
Baseline q*_alpha values = 270 (LunarLander), 230 (InvertedPendulum), 0 (Maze)
Hand-set for RET-CAP/PCVaR baselines from the range {230,250,270,290}; important for fairness of the empirical comparison.

assumptions (6)

standard math Quantile elicits the argmin of quantile regression loss l_alpha (Eq. 5).
Used to convert the VaR Bellman equation into a gradient-based update; standard property of quantiles.
standard math CVaR variational representation and VaR-as-integral representation (Eq. 2).
Basis for both the CVaR-PG estimator and the claimed equivalence between CVaR and the expected VaR term.
domain assumption Returns are continuous and quantiles are unique (Sec. 2.1).
The paper assumes q_-_alpha = q+_alpha; in discrete or flat-tail environments this fails and gradient vanishing can occur, which the paper acknowledges but does not resolve in theory.
ad hoc to paper A unique stationary Markovian alpha0-VaR optimal policy bar-pi* exists and satisfies bar-pi*(s_t)=pi-hat*(s_t,alpha_t) along the Algo. 2 path (Prop. 3.3).
Load-bearing premise of the Markovian VaR policy-gradient derivation; asserted 'by definition' without proof.
ad hoc to paper The learned quantile value function v(s,alpha) is accurate and monotone enough for the alpha-selection rule in Eq. 12 to track true quantile levels.
Monotonicity is enforced architecturally with softplus, but approximation error and off-distribution states can break the min{beta | v(s,beta) >= z} computation.
standard math Lemma B.4 of Hau et al. (2025) on VaR decomposition with non-decreasing functions.
Used in Appendix A.2 to rederive the random-reward VaR Bellman equation; prior published result, but from a paper co-authored by the second author.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Boosting CVaR Policy Optimization with Quantile Gradients." pith.science (2026). https://pith.science/paper/QXY2NOMK

@misc{pith2026260122100,
  author       = {Pith},
  title        = {Pith review of: Boosting CVaR Policy Optimization with Quantile Gradients},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QXY2NOMK}},
  note         = {Machine review of arXiv:2601.22100}
}

read the original abstract

Optimizing Conditional Value-at-risk (CVaR) using policy gradient (a.k.a CVaR-PG) faces significant challenges of sample inefficiency. This inefficiency stems from the fact that it focuses on tail-end performance and overlooks many sampled trajectories. We address this problem by augmenting CVaR with an expected quantile term. Quantile optimization admits a dynamic programming formulation that leverages all sampled data, thus improves sample efficiency. This does not alter the CVaR objective since CVaR corresponds to the expectation of quantile over the tail. Empirical results in domains with verifiable risk-averse behavior show that our algorithm within the Markovian policy class substantially improves upon CVaR-PG and consistently outperforms other existing methods.

Figures

Figures reproduced from arXiv: 2601.22100 by the authors.

**Figure 2.** (a) Expected return (b) Long (risk-averse) path rate in Maze. Curves are averaged over 10 seeds with shaded [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

**Figure 3.** Expected return, risk-averse rate, and CVaR 0.2 of return in LunarLander and Inverted Pendulum. Curves [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

5 extracted references · 1 linked inside Pith

[1]

and Ott, J

B¨auerle, N. and Ott, J. (2011). Markov decision processes with average-value-at-risk criteria.Mathematical Methods of Operations Research, 74:361–379. Brockman, G., Cheung, V ., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). Openai gym. arXiv preprint arXiv:1606.01540. Dabney, W., Ostrovski, G., Silver, D., and Munos, R. (...

arXiv 2011
[4]

20 Tamar, A., Di Castro, D., and Mannor, S

MIT press Cambridge. 20 Tamar, A., Di Castro, D., and Mannor, S. (2012). Policy gradients with variance related risk criteria. InProceedings of the International Conference on Machine Learning (ICML), pages 387–396. Tamar, A., Glassner, Y ., and Mannor, S. (2015). Optimizing the cvar via sampling. InProceedings of the AAAI Conference on Artificial Intelli...

2012
[29]

Todorov, E., Erez, T., and Tassa, Y . (2012). Mujoco: A physics engine for model-based control. In2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026–5033. IEEE. Zhou, F., Wang, J., and Feng, X. (2020). Non-crossing quantile regression for distributional reinforcement learning. Advances in neural information processing sys...

2012
[32]

Greenberg, I., Chow, Y ., Ghavamzadeh, M., and Mannor, S. (2022). Efficient risk-averse reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 35:32639–32652. Hau, J. L., Delage, E., Derman, E., Ghavamzadeh, M., and Petrik, M. (2025). Q-learning for quantile mdps: A decomposition, performance, and convergence analysis. InProc...

2022
[1447]

Lim, S. H. and Malik, I. (2022). Distributional reinforcement learning for risk-sensitive policies.Advances in Neural Information Processing Systems (NeurIPS), 35:30977–30989. Luo, Y ., Liu, G., Duan, H., Schulte, O., and Poupart, P. (2022). Distributional reinforcement learning with monotonic splines. InInternational Conference on Learning Representation...

2022

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

and Ott, J

B¨auerle, N. and Ott, J. (2011). Markov decision processes with average-value-at-risk criteria.Mathematical Methods of Operations Research, 74:361–379. Brockman, G., Cheung, V ., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). Openai gym. arXiv preprint arXiv:1606.01540. Dabney, W., Ostrovski, G., Silver, D., and Munos, R. (...

arXiv 2011

[2] [4]

20 Tamar, A., Di Castro, D., and Mannor, S

MIT press Cambridge. 20 Tamar, A., Di Castro, D., and Mannor, S. (2012). Policy gradients with variance related risk criteria. InProceedings of the International Conference on Machine Learning (ICML), pages 387–396. Tamar, A., Glassner, Y ., and Mannor, S. (2015). Optimizing the cvar via sampling. InProceedings of the AAAI Conference on Artificial Intelli...

2012

[3] [29]

Todorov, E., Erez, T., and Tassa, Y . (2012). Mujoco: A physics engine for model-based control. In2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026–5033. IEEE. Zhou, F., Wang, J., and Feng, X. (2020). Non-crossing quantile regression for distributional reinforcement learning. Advances in neural information processing sys...

2012

[4] [32]

Greenberg, I., Chow, Y ., Ghavamzadeh, M., and Mannor, S. (2022). Efficient risk-averse reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 35:32639–32652. Hau, J. L., Delage, E., Derman, E., Ghavamzadeh, M., and Petrik, M. (2025). Q-learning for quantile mdps: A decomposition, performance, and convergence analysis. InProc...

2022

[5] [1447]

Lim, S. H. and Malik, I. (2022). Distributional reinforcement learning for risk-sensitive policies.Advances in Neural Information Processing Systems (NeurIPS), 35:30977–30989. Luo, Y ., Liu, G., Duan, H., Schulte, O., and Poupart, P. (2022). Distributional reinforcement learning with monotonic splines. InInternational Conference on Learning Representation...

2022