{"id":"b62e295c-491e-4d3a-8599-61f0821ab8b2","arxiv_id":"2601.22100","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Augmenting CVaR policy gradient with a Value-at-risk/quantile gradient term improves sample efficiency in risk-averse RL on modified maze, lander, and pendulum benchmarks.","lead":"This paper combines CVaR policy-gradient with quantile (Value-at-risk) gradients to make risk-averse reinforcement learning use its samples more efficiently. If the empirical results hold, it gives a simple way to train policies that avoid rare catastrophic outcomes where existing CVaR methods fail.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The VaR-PG term is not a grounded gradient of Eq. 7: the expected-quantile term equals CVaR_alpha, so the objective is unchanged, and Prop. 3.3's key equality is asserted rather than proven, with convergence explicitly left open.","rationale":"The reader's weakest assumption correctly identifies Prop. 3.3 as the load-bearing theoretical step. My analysis agrees and sharpens it: (i) the objective in Eq. (7) is literally CVaR_alpha because the expected quantile over [0,alpha] equals CVaR, so the only possible benefit of the VaR term must come from the estimator g2, not the objective; (ii) the proof of Prop. 3.3 is circular and conflates Markovian optimality with global optimality, making the equality q^{pi*}=v* unestablished; (iii) the paper itself concedes convergence is open. Since the reader already issued CONDITIONAL based on this concern, my read does not change the verdict. A direct brute-force test of Prop. 3.3 on a finite tabular MDP is the cleanest way to settle whether the theoretical foundation holds.","tokens_in":20945,"tokens_out":10129,"duration_ms":111212,"concrete_test":"Enumerate a small finite MDP where the unique Markovian alpha0-VaR optimal policy pi* is found by brute force; compute q^{pi*} exactly by solving the quantile Bellman equation for that policy; then run Algo. 3 with q^{pi*} from all initial states and alpha0 in [0,alpha]. If any reachable (s,alpha) yields an action different from pi*(s), Proposition 3.3 fails, and the VaR-PG component loses its theoretical grounding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (7) is mathematically redundant: E_{beta~U[0,alpha]} VaR_beta[g0] = (1/alpha)∫_0^alpha VaR_beta dbeta = CVaR_alpha[g0], so the objective is exactly CVaR_alpha for every omega in (0,1). The proposed 'augmentation' is therefore not a new objective; it is a second gradient estimator g2. The paper's only theoretical justification for g2 is Prop. 3.3/3.4. Prop. 3.3 asserts that for a stationary Markovian alpha0-VaR optimal policy pi*, q^{pi*}(s_t,alpha_t,pi*(s_t)) = v*(s_t,alpha_t) along the Algo. 2 path, 'by definition of stationary and Markovian'. This conflates Markovian optimality with global optimality: if pi* is optimal only within Pi_M, its quantile value can be strictly below the global v*, and the proof's contradiction fails. The proof also uses the equality to establish the alpha-tracking identity it is meant to prove (circularity). The paper explicitly states that convergence of the Markovian VaR updates from arbitrary v is 'an open question'. Consequently g2 is not demonstrated to be an unbiased gradient or even a consistent descent direction for Eq. (7); the algorithm reduces to CVaR-PG plus an ungrounded heuristic. If g2 is biased, the combined update may not optimize CVaR, and the empirical gains may reflect a different (unexamined) objective.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method to improve the sample efficiency of CVaR policy gradient (CVaR-PG) by adding a quantile/VaR policy-gradient component. It defines the objective in Eq. (7) as a weighted combination of CVaR and an expected VaR term, derives a soft VaR Bellman operator (Propositions 3.1 and 3.2), attempts to reduce VaR optimization to the Markovian policy class (Propositions 3.3 and 3.4), and combines the standard CVaR-PG estimator with a quantile-value-based advantage estimator. Experiments on Maze, LunarLander, and InvertedPendulum report that the combined method learns faster and achieves higher risk-averse rates and CVaR values than CVaR-PG, PCVaR-PG, RET-CAP, and MIX.","tokens_in":21481,"tokens_out":8308,"duration_ms":84182,"significance":"If the empirical results hold, the method is practically attractive: it suggests a way to use all sampled trajectories in risk-averse policy optimization and to overcome the 'blindness to success' of CVaR-PG. The experimental domains are chosen to have verifiable risk-averse behavior, and the reported curves are consistent across seeds. The paper also provides contraction and fixed-point results for a soft VaR Bellman operator, which is a useful technical step. However, the theoretical foundation for the VaR component that drives the claimed improvement is not established: Eq. (7) is mathematically identical to CVaR, and Proposition 3.3, the key result connecting the Markovian VaR policy to the quantile value function, is asserted rather than proven. The paper explicitly leaves convergence open. The current contribution is therefore best read as an empirical heuristic, but the manuscript presents it as a grounded optimization method.","major_comments":[{"comment":"The proposed 'augmented' objective does not augment CVaR. Since E_{β~U[0,α]} VaR_β[g0] = (1/α)∫_0^α VaR_β dβ = CVaR_α[g0] by Eq. (1), Eq. (7) equals CVaR_α[g0] for every ω∈(0,1). Thus the second term is not a separate objective component; the abstract's statement that 'this does not alter the CVaR objective' is true, but it also means the method's theoretical claim must be recast: g2 is an additional gradient estimator for the same CVaR objective. No unbiasedness or consistency proof for g2 is given. As written, the theoretical motivation for the algorithm is incomplete: if g2 is biased, the combined update need not optimize CVaR.","section":"§3, Eq. (7)"},{"comment":"The load-bearing equality q^{π̄*}(s_t,α_t,π̄*(s_t)) = v*(s_t,α_t) is not proved. Optimality of π̄* within the Markovian class Π_M only concerns the initial risk level α0; it does not imply that π̄* realizes the unrestricted optimal quantile value v* at every (s,α) on the Algorithm 2 trajectory. The assertion 'by definition of stationary and Markovian' is insufficient. The contradiction step is also invalid: q^{π̄*} < v* does not imply that some alternative action b at s_t achieves v*(s_t,α_t), since v* may be attained only by a history-dependent policy. The proof further assumes monotonicity of max_b q^{π̄*}(s_1,β,b) in β and uses the equality at s_1 in the alpha-selection step, which is the very property being proved. Because Proposition 3.4 and the update rule in Eqs. (12)–(13) inherit this result, the VaR-PG component is not shown to be a valid descent direction. The limitation statem","section":"§3.2, Prop. 3.3"},{"comment":"The trade-off parameter ω does not appear in Algorithm 4. The text says the two gradients are 'combined via the trade-off parameter ω', but the displayed update is θ ← θ + lr(g1 + g2). In Appendices C.3 and C.4, ω is set to 0 after 40%/25% of iterations; under the displayed update this schedule has no effect. The reader cannot determine whether the evaluated method is g1+g2, ωg1+(1−ω)g2, or another combination. This must be corrected, and the actual ω schedule used in the experiments should be reported in the algorithm pseudocode.","section":"§3.3 and Algorithm 4"}],"minor_comments":[{"comment":"There are several typos and inconsistencies: 'Markovain' (§2.2), 'future rerturn' (§2.2), 'deep neutral networks' (Appendix C.1), and inconsistent table labels 'CVAR-VAR' versus 'CVaR-VaR' (Tables 1–3).","section":"Throughout"},{"comment":"The proposition statement does not mention the density assumption on r(s,a)+γv*(s',u) that is used in the proof (Appendix A.4). Please state the assumption explicitly in the proposition.","section":"Prop. 3.2"},{"comment":"The definition of p_i is malformed: 'P(˜r(s, a) = ri|s, a, si)' appears inside the definition of P(˜y=i|s,a) and should be written more carefully.","section":"Appendix A.2"},{"comment":"The projection α ← proj(α, Λ) is not defined. Specify whether this is nearest-neighbor, floor, or some other mapping to the discretized quantile levels.","section":"Algorithm 4"},{"comment":"No sensitivity analysis is reported for the number of quantiles I or the trade-off ω. Since the method's name and motivation depend on these choices, a small ablation would strengthen the empirical claims.","section":"Experiments"}],"recommendation":"major_revision","confidential_remarks":"I agree with the stress-test concern: Eq. (7) is a rewrite of CVaR, and Proposition 3.3 is not proven. The central empirical direction is useful, but the manuscript currently presents an ungrounded gradient component as a principled method. The revision must either prove the needed properties of g2 (unbiasedness or at least a consistent descent direction) or clearly reposition the method as an empirically motivated heuristic and provide additional validation (ablations, statistical tests, and the actual ω schedule). If Proposition 3.3 cannot be repaired, the paper should not be accepted in its current theoretical framing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper's empirical story is believable: adding a quantile-based advantage term to CVaR-PG does make learning faster and more stable in the three toy domains, and the authors are honest that the convergence theory is open. But the theoretical grounding for that quantile term does not survive close reading. Eq. 7 is a rewrite of the same CVaR objective—E_{β}[VaR_β]=CVaR_α—so this isn't a new objective. The real claim is that g2 is a useful gradient estimator. That may be true in practice, but the proof of Prop 3.3 rests on an equality that is asserted rather than shown: q^{π̄*}(s_t,α_t,π̄*(s_t))=v*(s_t,α_t). This is exactly the kind of fact that needs proof. The paper's response that it holds 'by definition of stationary and Markovian' conflates local optimality within Π_M with global optimality. The proof reads as circular, and the appendix's open-convergence note makes clear the authors know this.\n\nWhat's actually new: the state-only VaR Bellman operator (Eq. 8), the cumulative-reward tracking adaptation to Markovian policies, and the CVaR+VaR gradient combination. These are real ideas and worth testing. The experiments are cleanly run for what they are: three domains with verifiable risk-averse behavior, 10 seeds, and standard comparison baselines. The improvement in risk-averse rate and CVaR is consistent.\n\nTwo smaller problems. First, Algorithm 4's update uses g1+g2 unweighted, while the appendix says ω starts at 0.5 and decays to 0. That mismatch makes the actual method underdetermined. Second, there is no code, and the hyperparameter selection is reported as a set, not the specific values used—so replication is harder than it should be.\n\nBottom line: this deserves a serious referee, but the paper should not be accepted as is. The authors need to either prove Prop 3.3 under stated assumptions (or weaken its claim), clarify the ω schedule in the algorithm text, and ideally release code. If the theory remains speculative, reframing the contribution as an empirical heuristic would be the honest move. I'd take it to a reading group; there's plenty to discuss in the gap between what it claims and what it shows.","headline":"The empirical claim is credible, but the VaR-gradient theory is much weaker than the paper's presentation suggests.","tokens_in":21852,"tokens_out":5688,"would_cite":true,"duration_ms":56255,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Augmenting CVaR policy gradients with an expected-quantile term lets risk-averse reinforcement learning use all sampled trajectories instead of only the worst tail.","keywords":["CVaR","value-at-risk","policy gradient","sample efficiency","risk-averse reinforcement learning","quantile regression","Bellman operator","Markovian policies"],"falsifier":"In a tabular MDP with known optimal Markovian VaR policy, compute the quantile value function of that policy and run the cumulative-reward tracking procedure; if it ever selects an action different from the optimal one, or if two actions tie at the tracked quantile level, the recovery claim fails.","tokens_in":20855,"feed_emoji":"🛡️","tokens_out":7874,"duration_ms":73772,"temperature":0.7,"pith_summary":"This paper claims that CVaR policy gradients are sample-inefficient because they discard all but the worst α-fraction of trajectories, and that adding an expected quantile (VaR) term to the objective fixes this. Quantile optimization has a Bellman-style dynamic programming form, so every sampled transition can contribute through a learned quantile value function. The paper derives a VaR Bellman operator, adapts it to Markovian policies via cumulative-reward tracking, and combines the resulting VaR gradient with the standard CVaR gradient. In three domains with identifiable risk-averse behavior, the combined algorithm learns risk-averse policies faster and more reliably than CVaR-PG and prior sample-efficient CVaR methods. A sympathetic reader would care because the method offers a practical, on-policy way to optimize a static tail-risk measure without throwing away most of the data.","feed_headline":"Quantile gradients boost CVaR policy learning with all trajectories","feed_subtitle":"CVaR gradients only see worst trajectories; a quantile term keeps every sample useful for risk-averse policies.","key_machinery":"The central object is the augmented objective max_{π∈Π_M} ω CVaR_α[g0] + (1−ω) E_{β∼U[0,α]}[VaR_β[g0]], where CVaR is the average of the worst α tail and VaR is the β-quantile. The argument runs through a VaR Bellman operator on the state-quantile value function v(s,α), built from quantile-regression losses; a cumulative-reward tracking rule that updates the quantile level as α_t = min{β | v(s_t,β) ≥ (v(s_{t-1},α_{t-1}) − r_{t-1})/γ}; and a Markovian advantage term Ā(s_t, α_t, a_t) = E[∂l_{α_t}(δ_v)] that supplies the VaR policy gradient. The VaR component is what lets all trajectories—including successful ones—contribute to policy updates, while the CVaR component keeps the worst-case tail","core_discovery":"On the paper's own terms, the central claim is that optimizing CVaR at risk level α together with E_{β∼U[0,α]}[VaR_β] preserves the CVaR objective while converting the problem from a tail-only gradient estimator to one that can use all sampled data. The authors prove that a new VaR Bellman operator v(s,α) ← v(s,α) + η max_a E[∂l_{α}(δ_v)] has the optimal state-quantile value as its fixed point, and they assert that a unique stationary Markovian α0-VaR optimal policy can be executed by tracking cumulative rewards against this value function. Combining the resulting VaR policy gradient with the CVaR-PG estimator yields their CVaR-VaR algorithm, which the experiments show attains both high expe","pith_inferences":["If the open convergence question is resolved, the same augmentation could make static CVaR optimization practical in much larger MDPs; until then, the gains rest on empirical support rather than a proven convergence guarantee.","A similar augmentation may transfer to other tail-dependent risk measures that are integrals of quantiles, because those also admit quantile-regression-style formulations.","The practical bottleneck is likely the accuracy of the learned quantile value function: quantile crossing and coarse discretization of α could hurt cumulative-reward tracking more than the original tail-selection issue.","One testable extension is to replace the uniform draw of initial α_0 with a learned distribution over risk levels, which could adapt the trade-off between CVaR and VaR per episode."],"forward_implications":["Risk-averse policies for static CVaR can be learned with on-policy gradient methods that use every sampled trajectory instead of only the worst αN, reducing the sample cost of tail-risk optimization.","The approach directly addresses the blindness-to-success problem: high-return trajectories now shape the quantile value function and the policy, rather than being discarded.","The VaR Bellman operator gives a dynamic-programming template for quantile value learning in actor-critic settings, with contraction guarantees under a soft quantile loss.","In the tested domains, the combined algorithm reaches high expected return and high risk-averse rates where CVaR-PG fails or converges slowly, and it outperforms or matches existing sample-efficient CVaR baselines.","The VaR-PG component can be combined with other sample-efficiency techniques such as multi-step advantage estimation, and can be annealed as training proceeds."],"fun_headline_variants":["Quantile gradients make CVaR policy learning sample-efficient","All trajectories, not just tails: quantile boost for CVaR","CVaR meets quantile gradients: risk-averse with less data","Quantile term recycles samples for CVaR policy gradients","Boost CVaR optimization by leveraging every trajectory"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a unique stationary Markovian VaR-optimal policy can be recovered by tracking cumulative rewards against the learned quantile value function—asserted by definition in Proposition 3.3—while the paper itself states that the convergence of Markovian VaR updates from an arbitrary value function remains unknown.","fun_headline_variants_meta":{"raw":{"variants":["Quantile gradients make CVaR policy learning sample-efficient","All trajectories, not just tails: quantile boost for CVaR","CVaR meets quantile gradients: risk-averse with less data","Quantile term recycles samples for CVaR policy gradients","Boost CVaR optimization by leveraging every trajectory"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000455,"raw_usage":{"total_tokens":2093,"prompt_tokens":683,"completion_tokens":1410,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":427,"completion_tokens_details":{"reasoning_tokens":1324}},"tokens_in":427,"tokens_out":1410,"duration_ms":10486,"temperature":1.0,"reasoning_tokens":1324,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T06:43:27.656401+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a tabular MDP with known optimal Markovian VaR policy, compute the quantile value function of that policy and run the cumulative-reward tracking procedure; if it ever selects an action different from the optimal one, or if two actions tie at the tracked quantile level, the recovery claim fails.","supporting_citations":[],"review_version":1}