{"id":"676a9163-e912-4a4d-88e8-781e7e132e75","arxiv_id":"2412.13184","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"TQPO estimates gradients of quantile safety constraints directly through sampling and adds a tilted update to the Lagrange multiplier, improving return while satisfying the constraints.","lead":"This paper introduces TQPO, a safe reinforcement learning algorithm that enforces safety by constraining the quantile (for example, the 95th percentile) of cumulative cost instead of its average. The authors report that TQPO satisfies these quantile constraints while achieving higher returns than existing baselines on three Safety Gym tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (8) misidentifies the CDF gradient and flips its sign; the implemented update in Eq. (16) penalizes safe trajectories, so TQPO does not actually descend the quantile-constrained objective.","rationale":"The reader's verdict is REJECT, and I agree with that verdict, so no verdict change is needed. The reader's weakest_assumption highlighted the mismatch between the convergence proof and the implemented tilted update; that concern is real and remains valid. However, I find an even more fundamental problem in the quantile-gradient derivation itself. The sign error in Eq. (8) and the unjustified removal of the density term in Eq. (7) mean that the policy update in Eq. (16) does not compute or descend the true quantile gradient. If the sign is wrong, the algorithm is, at best, solving a different optimization problem, and the theoretical convergence results, even if internally correct, do not apply to the actual method. The reader's rationale mentions the sign inconsistency and the density drop, so there is partial agreement, but the core load-bearing issue is the derivation error rather than the proof-algorithm gap. A rigorous re-derivation or a simple finite-difference check on a toy MDP would settle this; the analytical error is already transparent from comparing Eq. (8) with the definition of F(q).","tokens_in":12014,"tokens_out":5658,"duration_ms":56668,"concrete_test":"Set up a single-state MDP with a Gaussian policy parameterized by its mean, where the cumulative cost distribution and its quantile gradient can be computed exactly by finite differences. Compute the true ∇θq_{1-ε} and compare its sign with the TQPO update direction induced by Eq. (16) (i.e., the sign of -λI(C≤q) in the policy gradient). If the TQPO direction increases the quantile when the true gradient decreases it, the estimator is invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that TQPO directly estimates quantile gradients fails at the first step of the derivation. In Eq. (8), the paper states ∇θF(q;πθ)=∇θE[I(C≥q)]=-∇θE[I(C≤q)]. But F is the CDF: F(q;πθ)=E[I(C≤q)], not E[I(C≥q)]. Correct score-function differentiation gives ∇θF=+E[I(C≤q)Σ∇logπ]. The erroneous minus sign propagates through Eq. (7): ∇θq=-∇θF/f becomes +E[I(C≤q)Σ]/f, the opposite of the true quantile gradient. Substituted into the Lagrangian (Eq. 10), this turns ∇θL=∇θV-λ∇θq into ∇θV-λE[I(C≤q)Σ] (after dropping 1/f), exactly the form in Eq. (11). Consequently the advantage in Eq. (16) subtracts λI(C≤q), penalizing trajectories whose cumulative cost is below the quantile and pushing the policy toward higher cost. The correct Lagrangian gradient should add λI(C≤q). Separately, the density term 1/f(q;πθ) is dropped with only the justification that f>0; f depends on θ, so the gradient magnitude and even the effective balance between return and constraint are altered in a way the Lagrange multiplier cannot absorb. These errors mean the implemented algorithm is not optimizing the quantile-constrained problem stated in Eq. (6), and the convergence theorems analyze a different objective.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Tilted Quantile Policy Optimization (TQPO), a reinforcement learning method that replaces expectation-form safety constraints with a quantile (value-at-risk) constraint on cumulative cost. The method estimates the quantile gradient via a likelihood-ratio sampling technique, applies a tilted adaptive update to the Lagrange multiplier to compensate for asymmetric quantile distributions, and is evaluated on three Safety Gym tasks against QCPO and PPO-Lag. The paper also claims convergence guarantees via two-timescale stochastic approximation. The central claims are that TQPO directly estimates quantile gradients without expectation approximations, achieves higher return while satisfying quantile constraints, and provably converges to the optimal solution.","tokens_in":12472,"tokens_out":5835,"duration_ms":52811,"significance":"The problem of enforcing high-probability safety constraints in RL is important, and the paper addresses a meaningful gap by avoiding expectation-form approximations. The experimental study is reasonably extensive, includes ablations, and the code is released. However, the central theoretical claims are undermined by a sign error in the quantile-gradient derivation, by the unjustified dropping of the density term, and by a convergence analysis that analyzes a different update rule than the one implemented. As a result, the paper's main claims about direct quantile-gradient estimation and provable convergence are not supported by the current manuscript.","major_comments":[{"comment":"Equation (8) states that grad_theta F(q; pi_theta) = grad_theta E[I(C >= q)] = -grad_theta E[I(C <= q)]. Since F(q; pi_theta) is defined in Eq. (5) as E[I(C <= q)], the correct score-function identity is grad_theta F = +E[I(C <= q) grad_theta log pi], not its negative. This sign error flips the quantile gradient in Eq. (7) and propagates to Eq. (11), producing an advantage function in Eq. (16) that subtracts lambda * I(C <= q). The implemented algorithm therefore penalizes trajectories whose cost is below the quantile, which pushes the policy toward higher cost and is the opposite of the intended safety gradient. This is a load-bearing error because the algorithm is not optimizing the quantile-constrained objective in Eq. (6).","section":"Estimating Quantile Gradients Through Sampling, Eq. (8)"},{"comment":"The paper drops the density f(q) from the denominator of Eq. (7) with the justification that the PDF is always positive. Positivity only preserves the directional sign of the gradient, not its magnitude. In the Lagrangian objective (10), the relative weight between the return gradient and the constraint gradient matters, and f(q) depends on the policy parameter theta. The approximation therefore changes the effective objective in a way that a Lagrange multiplier rescaling cannot absorb. This is a second load-bearing issue that invalidates the claimed equivalence between the sampled update and the true quantile-constrained gradient.","section":"Estimating Quantile Gradients Through Sampling, after Eq. (7)"},{"comment":"The paper explicitly states that, for the convenience of theoretical analysis, it replaces the implemented loss L_theta in Eq. (15) with the Lagrangian objective L(theta, lambda, q) in Eq. (10). Consequently, Theorems 1 and 2 analyze a different algorithm than the one whose empirical performance is reported. No argument is given that the PPO clipped surrogate objective in Eq. (15) behaves like the Lagrangian gradient in the limit, so the convergence proofs do not cover the actual TQPO updates.","section":"Convergence Analysis, first paragraph"},{"comment":"The tilted update rate eta in Eq. (14) is computed from F_q(d), the empirical CDF of the quantile being optimized, making the rate data-dependent and self-referential. Assumption 3 requires deterministic positive, nonsummable, square-summable step-size sequences with eta_k = o(beta_k) and beta_k = o(alpha_k). The adaptive eta produced by Eq. (14) does not satisfy these conditions, and the paper provides no alternative analysis for the adaptive scheme. The convergence theorems therefore do not cover the tilted updates that are central to the reported performance gains.","section":"Tilted Quantile Gradient Update, Eq. (14), and Assumption 3"},{"comment":"Lemma 3 assumes that L(theta, q) is strictly concave on Theta, which is not satisfied by typical neural policy parameterizations and is not justified in the paper. Theorem 2 is asserted by invoking Borkar (2008) without a formal statement or proof; since the appendix is omitted from the manuscript, the reader cannot verify the claimed convergence to the optimal solution of the constrained problem. These are additional load-bearing gaps in the theoretical contribution.","section":"Convergence Analysis, Lemma 3 and Theorem 2"}],"minor_comments":[{"comment":"In the statement of Theorem 1, the update is written as theta_{k+1} = theta + beta_k(...), which should presumably be theta_{k+1} = theta_k + beta_k(...).","section":"Theorem 1"},{"comment":"The condition \"P_infinity alpha_k m_k, P_infinity beta_k n_k < infinity\" is ambiguous: it is not clear whether the sums are meant to converge absolutely, almost surely, or in probability, and no normalization is specified.","section":"Theorem 1, condition (iii)"},{"comment":"The definition of F_q(d) is not precise: it is described as the CDF of the distribution of the quantile q_{1-epsilon}(pi_theta) at d, but the manuscript does not specify how this distribution is formed or how the per-epoch sampling estimate is computed.","section":"Tilted Quantile Gradient Update, Eq. (14)"},{"comment":"The abstract and conclusion state that the proposed model 'fully guarantees safety' or 'fully satisfies the quantile constraints', but the experimental safety probabilities reported in Table 1 are in the range 89-97%, which are close to but not always above the specified levels.","section":"Abstract and Conclusion"},{"comment":"The statement that QCPO 'required the cumulative cost distribution' is vague; a precise comparison of the distributional assumptions made by QCPO and TQPO would help the reader evaluate the claimed advantage.","section":"Introduction, related work on QCPO"}],"recommendation":"reject","confidential_remarks":"The sign error in Eq. (8) is not a minor typo: it changes the direction of the constraint gradient and therefore invalidates the core algorithmic claim. Even if that error were corrected, the convergence analysis explicitly analyzes a modified update rule and does not cover the adaptive tilted rate. These are load-bearing problems that require a substantial reworking of the derivation, the algorithm, and the theory, beyond what a standard revision would permit. The experimental results may still be of interest, but they cannot compensate for the unsupported central claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know that the central derivation of this paper is wrong. In Eq. (8) they define F(q) as the CDF P(C≤q), then compute its gradient as ∇E[I(C≥q)] = -∇E[I(C≤q)]. That is the negative of the correct score-function gradient. The mistake propagates through Eq. (11) and into the implemented advantage in Eq. (16), which subtracts λI(C≤q) from the reward advantage. The correct Lagrangian gradient should add λI(C≤q) (up to the positive density factor). So the algorithm as written pushes the policy toward higher cost, not lower; it is not optimizing the quantile-constrained problem in Eq. (6). This is not a minor typo—it flips the sign of the constraint gradient.\n\nThe density term 1/f(q) is also dropped with only the observation that f>0. But f depends on θ, so this changes the gradient direction and magnitude, not just a scaling constant that λ could absorb. The convergence section then explicitly replaces the implemented PPO-style loss with the Lagrangian objective, and Assumption 3 requires step sizes that rule out the adaptive tilted update in Eq. (14). So Theorems 1–2 do not cover the algorithm that actually runs.\n\nIn fairness, the paper is not without merit. Quantile-constrained RL is a relevant problem, and the idea of a tilted update to speed up recovery from Lagrange multiplier overshoot is a reasonable heuristic. The experiments are clearly described, with code, ablations, and comparisons against QCPO and PPO-Lag. The ablation showing that the tilted term helps is genuinely useful if the rest were sound.\n\nBut the sign error is load-bearing. If the implemented code has the correct sign, this is a fixable write-up error, but the paper as submitted is internally inconsistent: Eq. (8) contradicts its own definition of F(q). A referee would need to check the code to see if the performance results are real. As it stands, the theoretical claims do not hold and the algorithm's actual objective is unclear.\n\nMy take: this deserves a serious referee's time because the problem is worthwhile and the errors are concrete, but it should be rejected or sent back for major revision. I would not cite it in its current form.","headline":"The quantile gradient derivation has a sign error that inverts the safety objective, and the convergence proof analyzes a different algorithm, so TQPO's central claims fail as written despite a reasonable experimental setup.","tokens_in":12868,"tokens_out":5250,"would_cite":false,"duration_ms":49686,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes Tilted Quantile Policy Optimization (TQPO), which estimates the gradient of a cumulative-cost quantile constraint directly from samples and uses a tilted Lagrange-multiplier update to avoid over-conservative policies…","keywords":["quantile-constrained reinforcement learning","safe reinforcement learning","value-at-risk","likelihood ratio gradient estimation","Lagrangian method","tilted update","chance constraints","policy optimization"],"falsifier":"Run TQPO on the same Safety Gym tasks with the tilted update replaced by a fixed, square-summable step-size schedule for λ that satisfies Assumption 3, while keeping everything else identical. If the algorithm still achieves the same return and safety probability, the tilted term is not needed for the claimed gains; if performance drops sharply, the implemented update is load-bearing and the convergence proof's idealization misses it.","tokens_in":11803,"feed_emoji":"🎯","tokens_out":5937,"duration_ms":49610,"temperature":0.7,"pith_summary":"The paper proposes Tilted Quantile Policy Optimization (TQPO), an algorithm for safe reinforcement learning that enforces safety as a quantile constraint on cumulative cost rather than the usual expectation constraint. It shows that the gradient of the quantile can be estimated directly from samples via a likelihood-ratio identity, avoiding the biased expectation-form approximations used by prior work. A tilted update rate for the Lagrange multiplier compensates for the asymmetric distribution of the quantile around the threshold, preventing over-conservative policies. The authors prove convergence to the optimal solution under Lipschitz and step-size assumptions, and demonstrate in Safety Gym tasks that TQPO meets its quantile constraints with safety probability closer to the target and higher return than baselines.","feed_headline":"Direct quantile gradients let safe RL hit its safety target","feed_subtitle":"TQPO samples the quantile gradient, tilts the Lagrange update, and beats expectation-based and prior quantile methods.","key_machinery":"The load-bearing objects are (i) the sampling-based quantile gradient estimator ∇θ q̂_{1−ε} = −(1/N) Σ_i I(C(s_i,π_θ) ≤ q_k) Σ_t ∇_θ log π_θ(a_t|s_t), which replaces the intractable true quantile gradient; (ii) the tilted multiplier update η = (F_q(d)+δ)/(1+δ) for q ≥ d and η = (1−F_q(d)+δ)/(1+δ) for q < d, which makes the Lagrange multiplier recover quickly from overshoot; and (iii) the three-time-scale stochastic approximation recursion (q fastest, θ medium, λ slowest) that the convergence proof is built on. The proof replaces the implemented clipped PPO loss with the smooth Lagrangian L(θ,λ,q), so the theoretical convergence applies to this idealized surrogate.","core_discovery":"The paper's central claim is that the quantile-constrained RL problem max_θ V(s,π_θ) subject to q_{1−ε}(π_θ) ≤ d can be solved efficiently by estimating the quantile gradient directly. Using the inverse function theorem, the paper writes ∇_θ q_{1−ε} = −∇_θ F(q;π_θ)/f(q;π_θ), estimates the numerator with a likelihood-ratio gradient over indicators I(C(s,π_θ) ≤ q), and drops the positive denominator as a harmless scaling. To keep the Lagrange multiplier from overshooting, it tilts the multiplier step size so that λ decreases roughly nine times faster than it increases, using the CDF of the quantile distribution to set the tilt adaptively. The paper proves that the idealized recursions converge almost surely to the optimal solution under two-time-scale stochastic approximation assumptions, and experiments on Safety Gym tasks show TQPO's safety probability tracks the chosen 1−ε level while its return exceeds the QCPO baseline.","pith_inferences":["Editorial inference: The tilted update is essentially an asymmetric, data-dependent learning rate for the Lagrange multiplier; similar adaptive schemes could improve convergence in other primal-dual algorithms where the constraint slack is skewed.","Editorial inference: The proof's replacement of the implemented clipped PPO loss with the smooth Lagrangian objective means the convergence result is for an idealized variant; establishing convergence for the exact implemented update (or for the adaptive tilted η) is an open question.","Editorial inference: The likelihood-ratio quantile gradient estimator might be transferable to other risk-sensitive objectives such as CVaR or spectral risk measures, where similar density-ratio issues arise.","Editorial inference: A testable extension is to apply the tilted update to QCPO or other quantile methods and measure whether the safety probability tracks the target more tightly, controlling for the gradient estimator."],"forward_implications":["Safety constraints in RL can be enforced at a chosen violation probability rather than in expectation, giving a safety probability that matches the target level.","Quantile-constrained policies trained with TQPO achieve higher return than the prior QCPO baseline while satisfying the same constraints.","The sampling-based quantile gradient removes the need for distributional assumptions or additive expectation corrections, reducing bias and training time.","The tilted update can be applied to any Lagrangian-based constrained RL algorithm that suffers from asymmetric constraint slack."],"supporting_citations":[{"why":"Defines the QCPO baseline, whose additive expectation approximation TQPO replaces with direct quantile gradient estimation.","marker":"Jung et al. 2022"},{"why":"Supplies the likelihood-ratio technique for computing sensitivities of risk measures, which underlies the sampling-based quantile gradient estimator.","marker":"Glynn et al. 2021"},{"why":"Provides the PPO algorithm that TQPO extends with quantile constraints and clipped surrogate objectives.","marker":"Schulman et al. 2017"},{"why":"Provides the two-time-scale stochastic approximation theory used in the convergence proofs.","marker":"Borkar 2008"},{"why":"Defines the constrained Markov decision process framework that the quantile-constrained problem builds on.","marker":"Altman 2021"},{"why":"Offers perturbation-analysis methods for quantile sensitivity that motivate the direct gradient estimation approach.","marker":"Jiang and Fu 2015"},{"why":"Supplies the Safety Gym benchmark environments used for the empirical evaluation.","marker":"Ray, Achiam, and Amodei 2019"}],"fun_headline_variants":["Direct quantile gradient sampling for safe RL constraints","Tilted quantile gradients keep safe RL within its budget","Quantile-constrained RL without expectation shortcuts","Sample the quantile gradient, tilt the Lagrange step","New RL method hits quantile safety targets directly"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof of convergence applies to an idealized version of TQPO in which the adaptive tilted update and clipped PPO loss are replaced by a smooth Lagrangian objective with fixed step sizes; the implemented algorithm may not satisfy those assumptions.","fun_headline_variants_meta":{"raw":{"variants":["Direct quantile gradient sampling for safe RL constraints","Tilted quantile gradients keep safe RL within its budget","Quantile-constrained RL without expectation shortcuts","Sample the quantile gradient, tilt the Lagrange step","New RL method hits quantile safety targets directly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000256,"raw_usage":{"total_tokens":1545,"prompt_tokens":887,"completion_tokens":658,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":584}},"tokens_in":503,"tokens_out":658,"duration_ms":7409,"temperature":1.0,"reasoning_tokens":584,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:21:02.607038+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TQPO on the same Safety Gym tasks with the tilted update replaced by a fixed, square-summable step-size schedule for λ that satisfies Assumption 3, while keeping everything else identical. If the algorithm still achieves the same return and safety probability, the tilted term is not needed for the claimed gains; if performance drops sharply, the implemented update is load-bearing and the convergence proof's idealization misses it.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the two-time-scale stochastic approximation theory used in the convergence proofs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the QCPO baseline, whose additive expectation approximation TQPO replaces with direct quantile gradient estimation."},{"cited_title":"W.; Peng, Y.; Fu, M","cited_arxiv_id":null,"evidence_quote":"Supplies the likelihood-ratio technique for computing sensitivities of risk measures, which underlies the sampling-based quantile gradient estimator."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the constrained Markov decision process framework that the quantile-constrained problem builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Offers perturbation-analysis methods for quantile sensitivity that motivate the direct gradient estimation approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Safety Gym benchmark environments used for the empirical evaluation."}],"review_version":1}