{"id":"8888f6b8-5006-417d-b8f4-6bd407aafea9","arxiv_id":"2412.11138","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper introduces Gradient-based Estimation (GBE), a Taylor-expansion method using analytic trajectory gradients, and the trust-region algorithm CGPO for safe RL with finite-horizon constraints.","lead":"Safe reinforcement learning algorithms usually estimate future constraint violations with an infinite-horizon formula, which goes badly wrong when the safety rule is a finite-horizon budget. The authors replace that estimate with a trajectory-gradient Taylor approximation, and build a trust-region policy optimizer (CGPO) that satisfies finite-horizon constraints more reliably in differentiable simulators.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The safety guarantee in Theorem 5.3 is proved for exact gradients, but the implemented CGPO uses the SHAC-style critic-bootstrapped gradient from Eq. (20); no error term for this bias appears in Lemma 4.1 or Theorem 5.3, so the feasibility claim is not established for the algorithm actually run.","rationale":"The reader's weakest assumption identifies exactly the gap I consider load-bearing: the theory assumes the exact finite-horizon constraint gradient, while the implementation uses a short-horizon, critic-bootstrapped gradient from Eqs. (19)-(20). My analysis sharpens the point: the critic bias enters at first order in delta, not at the second-order level bounded by Lemma 4.1. This affects both the GBE accuracy claim and the safety bound in Theorem 5.3, which together underpin the reported empirical feasibility. I do not see this as a fatal flaw; the paper's empirical comparison, GBE-versus-ABE ablation, and adaptive-radius analysis are useful and the proposed method may well work in practice. But the central safety guarantee is presently stated for an idealized gradient, while Algorithm 1 uses a biased one, and no theorem or experiment directly quantifies that gap. This is an addressable issue rather than a reason to reject; hence the reader's CONDITIONAL verdict remains appropriate, and my stress-test does not move it.","tokens_in":30491,"tokens_out":3724,"duration_ms":39031,"concrete_test":"On the Appendix C FunctionEnv and one Brax task, at several policy checkpoints compute q_SHAC from Eq. (20) and q_exact from full-trajectory analytic backpropagation through the differentiable simulator (or central finite differences of J_C with respect to theta). Report the relative gradient error ||q_SHAC - q_exact|| / ||q_exact|| and the actual linearized-constraint bias delta^T (q_SHAC - q_exact) for the delta produced by Algorithm 1. Then run CGPO twice, once with Eq. (20) gradients and once with exact gradients, and check per iteration whether Eq. (17) holds. If exact-gradient CGPO respects b + 1/2 * epsilon_C * delta_hat while SHAC-gradient CGPO violates it, the theoretical guarantee does not cover the implemented algorithm.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central feasibility argument has a gap between the object analyzed and the object implemented. Lemma 4.1 and Theorem 5.3 bound first-order Taylor estimation error by 1/2 * epsilon * ||delta||^2, assuming that the gradient q_k in the trust-region subproblem is the exact gradient of the true finite-horizon constraint function J_C(theta_0). In Algorithm 1, however, q_k is computed from Eq. (20), whose short-horizon loss ends with the learned critic bootstrap V_C_psi(s_{t0+h}). The backpropagated gradient therefore contains a term dV_C_psi/ds_{t0+h} * ds_{t0+h}/dtheta, and unless the critic is exactly the true future-cost function, q_k is a biased estimate of the true gradient. Writing q_k = grad J_C(theta_0) + e, the linearized constraint error becomes delta^T e plus the quadratic remainder; the bias term is O(||delta||), not O(||delta||^2), so Lemma 4.1's bound does not cover it, and Theorem 5.3's J_C(theta_{k+1}) <= b + 1/2 * epsilon_C * delta_hat inherits an unaccounted additive term of order sqrt(delta_hat) * ||e||. A systematically biased q_k can make the subproblem choose a delta that the linearized constraint declares feasible while the true constraint moves beyond b. Section 7 acknowledges environmental differentiability as a limitation but does not discuss this critic-induced gradient error. Since the low violation ratios in Table 1 are attributed to accurate GBE estimates, this mismatch between the theory and the practical gradient estimator is the most load-bearing unresolved concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses safe reinforcement learning with finite-horizon, non-discounted constraints, which are common in benchmarks but poorly matched to the infinite-horizon advantage-based estimation (ABE) used by prior deep Safe RL algorithms. The authors propose Gradient-based Estimation (GBE), a first-order Taylor expansion of the constraint and reward functions along differentiable trajectories, and build a trust-region algorithm, Constrained Gradient-based Policy Optimization (CGPO), that repeatedly solves a linearly constrained quadratic subproblem. They prove solvability conditions for the subproblem, a worst-case Taylor remainder bound, and report experiments on four differentiable Brax-based tasks showing faster convergence and lower constraint-violation ratios than several baselines. The paper also includes ablations comparing GBE with ABE, a world-model extension, and an adaptive trust-region radius.","tokens_in":30814,"tokens_out":4694,"duration_ms":44862,"significance":"The core idea is simple and potentially valuable: use analytic finite-horizon gradients instead of infinite-horizon advantages to estimate constraint changes, then keep updates small in parameter space so that first-order estimates are credible. The mathematical statements for exact gradients are standard Taylor remainder/KKT arguments and are essentially correct. The empirical study is reasonably thorough for a conference paper: multiple environments, several baselines including differentiable-RL baselines, five seeds, an estimation-error ablation, and a world-model ablation. However, the paper's most important claims—that each CGPO update is feasible and that the empirical gains are explained by accurate GBE estimates—depend on a mismatch between the analyzed object (exact gradients) and the implemented object (short-horizon critic-bootstrapped gradients). Until this gap is closed with an error analysis or by changing the implementation, the safety guarantees are not established for the algorithm that is actually run. The paper is therefore worth revising rather than accepting in its current form.","major_comments":[{"comment":"The theoretical guarantees assume that q_k and g_k in the trust-region subproblem are the exact gradients of the true finite-horizon functions J_C and J_R. In the actual algorithm, q_k is computed from the SHAC-style loss in Eq. (20), whose last term is the learned critic V_C^psi(s_{t0+h}). The back-propagated gradient therefore contains a term dV_C^psi/ds_{t0+h} * ds_{t0+h}/dtheta, which is a biased estimate of the true constraint gradient unless the critic is exactly the remaining finite-horizon cost. Writing q_k = grad J_C(theta_k) + e, the linearized constraint error in the subproblem becomes delta^T e, which is O(||delta||), while Lemma 4.1's bound only covers a term of order ||delta||^2. Consequently, Theorem 5.3's bound JC(theta_{k+1}) <= b + (1/2) epsilon_C delta_hat is not valid for the implemented algorithm, and a systematically biased q_k can make the subproblem select an update that the linearized constraint declares feasible while the true constraint moves beyond b. Section 7 discusses environmental differentiability but does not mention this critic-induced gradient error. This is load-bearing because the low violation ratios in Table 1 are attributed precisely to the accuracy of GBE estimates.","section":"Sections 4.1, 5.2, and 5.3, Eqs. (7), (17), and (20)"},{"comment":"The paper repeatedly claims that CGPO 'ensures the feasibility of each update,' but Theorem 5.3 only proves JC(theta_{k+1}) <= b + (1/2) epsilon_C delta_hat, i.e., the actual constraint can exceed the threshold by an O(delta_hat) amount. This is a worst-case violation bound, not a feasibility guarantee. If 'feasible' means JC <= b, the theorem does not establish it; if 'feasible' means 'within a tolerance,' that tolerance is never defined or used in Algorithm 1. The abstract and conclusion should be reworded, or the theorem strengthened by adding a constraint-backoff mechanism that enforces JC(theta_k) + delta^T q_k <= b - margin with a margin derived from the Hessian bound.","section":"Section 5.2 and abstract/conclusion, Eq. (17)"},{"comment":"The corollary states that the entire trust region is feasible when c_k^2/q_k^T q_k - delta_hat > 0 and c_k <= 0, but Algorithm 1 uses the strict condition c_k < 0 in the second branch. At exactly c_k = 0 with a fully feasible trust region, the algorithm falls into the third branch and solves a KKT system that may not be the intended update. This boundary mismatch is minor in practice but should be corrected for logical consistency with the stated theory.","section":"Algorithm 1 and Corollary 5.2"}],"minor_comments":[{"comment":"The statement of Lemma 4.1 defines epsilon using ||nabla^2_theta J_R|| but the lemma is about J_f; it should be ||nabla^2_theta J_f||. The proof also calls the remainder a 'Peano' remainder, but Eq. (25) is the Lagrange form of the remainder.","section":"Lemma 4.1 and Appendix A.1"},{"comment":"The notation line defines 'gk = ∇θJR(θk), gk = ∇θJC(θk)', using the same symbol for both the reward and constraint gradients; the second should be q_k.","section":"Section 5.1, notation paragraph"},{"comment":"The toleration ratio zeta_k has a denominator |JC(theta_{k+1}) - hat JC(theta_{k+1})| that can be zero in practice; the paper does not specify a fallback value when this occurs.","section":"Eq. (18) and its definition of zeta_k"},{"comment":"The relative error metric divides by the true constraint change; if that change is near zero, the ratio is unstable. The caption says errors are averaged over 100 repetitions but does not report error bars or confidence intervals, making it hard to assess the significance of the GBE versus ABE difference.","section":"Figure 5 and Section 6.3"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the exact-gradient versus implemented-gradient mismatch. This is fixable: the authors could either analyze the SHAC-style gradient error and add it to Lemma 4.1/Theorem 5.3, or run the main experiments with full-trajectory exact gradients (e.g., BPTT without a value-function bootstrap) as the primary implementation. If they choose the former, the error term will likely depend on critic approximation error and the horizon h, and the safety bound may become substantially weaker. The paper's empirical claims are believable as engineering results, but the title and abstract promise a safety guarantee that the current theory does not support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read this. The problem it targets is real: most deep Safe RL algorithms estimate finite-horizon non-discounted constraints with infinite-horizon discounted advantage estimators, and the paper shows that this can produce relative errors above 1.0. GBE replaces that with a first-order Taylor estimate using analytic simulator gradients, and the Figure 5 check shows it tracks actual constraint changes far better. The core observation is useful and the experiments match the claim better than most Safe RL papers.\n\nWhat is new: not the Taylor expansion itself, but using it as a constraint estimator for finite-horizon CMDPs, plus a trust-region subproblem solver and an adaptive radius. The ablations are informative: GBE vs ABE, FoBG vs ZoBG in Appendix B, radius adaptation, and world-model augmentation. The paper also introduces differentiable Safe RL tasks on Brax, which should be useful to the community. Lemma 4.1 and Theorem 5.3 are standard Taylor and KKT arguments, correctly presented for exact gradients.\n\nThe soft spot is the gap between theory and implementation. The guarantees assume q_k is the exact gradient of the true finite-horizon constraint. Algorithm 1 computes q_k from the SHAC-style loss in Eq. (20), which truncates after h steps and replaces the tail with a learned critic V_C_psi. The backpropagated gradient therefore includes dV_C_psi/ds_{t0+h} * ds_{t0+h}/dtheta. Unless the critic is exact, q_k is biased. The bias term is O(||delta||), not O(||delta||^2), so Lemma 4.1 does not cover it and Theorem 5.3's constraint bound inherits an additive ||delta|| ||e|| term. Section 7 acknowledges differentiability as a limitation but does not analyze this critic-induced gradient error. This is the load-bearing unresolved issue, and the stress-test note is correct: the feasibility guarantee is not established for the algorithm as actually run.\n\nMinor issues: Table 1 lacks error bars even though Figure 4 shows variance, and no code is released. These are fixable. The empirical scope is four self-authored Brax tasks, so the generality claim is modest. None of this makes the paper a reject; it makes it a paper that needs the authors to either bound the gradient bias or restrict the theoretical claims.\n\nBottom line: this deserves a serious referee. The idea is clearly motivated, the empirical comparison is suggestive, and the theory-implementation gap is addressable. Send it to review; expect heavy but productive revision.","headline":"A real mismatch identified and a clever fix, but the safety guarantee is proved for exact gradients while the implementation uses a biased critic-bootstrapped gradient.","tokens_in":31378,"tokens_out":2581,"would_cite":true,"duration_ms":24620,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A finite-horizon safety constraint can be estimated by its first-order policy-parameter gradient, and the paper proves the error is bounded by the squared update size.","keywords":["safe reinforcement learning","constrained Markov decision process","finite-horizon constraints","gradient-based estimation","trust-region optimization","differentiable simulation","policy optimization","constraint satisfaction"],"falsifier":"Measure the true constraint value $J_C(\\theta_{k+1})$ after an update in a differentiable environment where the exact full-horizon gradient is computable, and compare the prediction $J_C(\\theta_k)+\\delta^\\top q_k$ with it; if the error exceeds $\\frac{1}{2}\\hat\\delta\\max\\|\\nabla^2 J_C\\|$ at small $\\hat\\delta$, or if replacing the exact gradient by the SHAC-style critic gradient pushes the relative error above 1.0, the claimed guarantee is false.","tokens_in":30249,"feed_emoji":"🛡️","tokens_out":12346,"duration_ms":103776,"temperature":0.7,"pith_summary":"The paper is trying to establish that finite-horizon, non-discounted safety constraints in reinforcement learning can be estimated accurately from first-order gradients, and that this estimate is enough to keep every policy update feasible inside a trust region. It argues that the standard advantage-based estimation, which relies on an infinite-horizon discounted advantage function, can have relative errors above 1.0 on finite-horizon constraints and therefore misleads safe policy updates. The proposed Gradient-based Estimation (GBE) approximates the next constraint value as $J_C(\\theta_0)+\\delta^\\top\\nabla_\\theta J_C(\\theta_0)$ and bounds the error by $\\frac{1}{2}\\epsilon\\|\\delta\\|_2^2$, so update size, not the discount factor, becomes the lever for safety. On top of GBE the paper builds Constrained Gradient-based Policy Optimization (CGPO), whose experiments on differentiable control tasks report faster convergence and fewer constraint violations than prior safe RL algorithms.","feed_headline":"Gradient estimates keep finite-horizon safe RL updates within limits","feed_subtitle":"Advantage estimates mispredict finite-horizon constraints; trajectory gradients inside a trust region keep updates safe.","key_machinery":"The machinery is the Gradient-based Estimation map $\\hat J_f(\\theta_0+\\delta)=J_f(\\theta_0)+\\delta^\\top\\nabla_\\theta J_f(\\theta_0)$ applied to both reward and cost, with gradients obtained by back-propagation through a differentiable simulator or a learned world model. Its error is $O(\\|\\delta\\|_2^2)$, which is what converts safety into a trust-region condition: the optimizer solves the small quadratic program $\\max_\\delta g_k^\\top\\delta$ subject to $c_k+q_k^\\top\\delta\\leq0$ and $\\|\\delta\\|_2^2\\leq\\hat\\delta$, using closed-form KKT dual variables $\\lambda^*$, $\\nu^*$ and update $\\theta_{k+1}=\\theta_k+(g_k-\\nu^*q_k)/\\lambda^*$, with an adaptive radius that shrinks when observed reward or constraint changes disagree with the estimates.","core_discovery":"The central claim is that a small parameter update $\\delta$ from $\\theta_0$ changes the finite-horizon constraint function by $\\delta^\\top\\nabla_\\theta J_C(\\theta_0)$ up to an error of at most $\\frac{1}{2}\\epsilon\\|\\delta\\|_2^2$, where $\\epsilon$ bounds the Hessian of $J_C$ along the line segment between $\\theta_0$ and $\\theta_0+\\delta$; the same first-order estimate is used for the reward objective. This is contrasted with the infinite-horizon advantage identity behind ABE, which requires $\\gamma\\neq1$ and a discounted state distribution, and which the paper shows can mispredict finite-horizon constraint changes by more than the true change itself. Using GBE, CGPO defines each update step as the solution of a convex trust-region sub-problem: maximize $\\delta^\\top g_k$ subject to $c_k+\\delta^\\top q_k\\leq0$ and $\\|\\delta\\|_2^2\\leq\\hat\\delta$, with three explicit cases depending on whether the whole trust region is infeasible, feasible, or split by the constraint boundary. The worst-case theorem then bounds reward loss by $\\frac{1}{2}\\epsilon_R\\hat\\delta$ and constraint violation by $b+\\frac{1}{2}\\epsilon_C\\hat\\delta$ for feasible starting policies.","pith_inferences":["This suggests a direct ablation: compare GBE with full-trajectory backpropagation against the SHAC-style short-horizon critic gradient; if the critic bootstrap biases the gradient, the stated worst-case bounds will be optimistic even when GBE itself is exact.","The estimator's only structural requirement is twice-differentiability of the constraint in policy parameters, so the same trust-region scheme transfers to deterministic trajectory optimization and model-predictive control, where no policy network or reward sampling is needed.","Because the paper's relative-error metric divides by the true constraint change, GBE looks bad on nearly flat constraints even when its absolute error is tiny; an absolute-violation metric would make the comparison with ABE more interpretable.","The adaptive-radius rule embodies a general principle: when an estimator has quadratic error, the optimizer can treat its radius as a learnable safety margin and use the ratio of predicted to observed reward and constraint changes to expand or shrink it, without estimating Hessians."],"forward_implications":["If GBE is right, safety violation at each update is controlled by the trust-region radius: the next policy satisfies $J_C(\\theta_{k+1})\\leq b+\\frac{1}{2}\\epsilon_C\\hat\\delta$ instead of inheriting an infinite-horizon approximation error.","Finite-horizon undiscounted constraints, the common form in practice, can be handled directly by a deep safe RL algorithm without being recast as discounted infinite-horizon sums.","CGPO always has a well-defined update direction: steepest descent on the constraint when the trust region is entirely infeasible, steepest ascent on reward when entirely feasible, and a KKT-based convex update in the split case.","Empirically, the paper reports that CGPO converges in fewer environment steps and with lower violation ratios than primal-dual, CPO/CUP, and Lagrangian differentiable-RL baselines across four differentiable control tasks."],"supporting_citations":[{"why":"Supplies the Advantage-based Estimation formula (Equation 2) that the paper argues fails for finite-horizon constraints, and provides the CPO baseline.","marker":"Achiam et al., 2017"},{"why":"Gives the infinite-horizon advantage identity (Equation 1) that ABE approximates, which is the source of the mismatch with finite-horizon constraints.","marker":"Kakade & Langford, 2002"},{"why":"Provides the short-horizon actor-critic gradient (SHAC) used to compute objective and constraint gradients in the implementation.","marker":"Jie Xu et al., 2022"},{"why":"Brax is the differentiable physics engine on which the four test tasks are built.","marker":"Freeman et al., 2021"},{"why":"Classifies zero-order and first-order batch gradient methods and supports the choice of first-order gradients for accurate constraint estimation.","marker":"Suh et al., 2022"},{"why":"Defines the CMDP framework with a constraint set and feasible policy set, the formal setting of the paper.","marker":"Altman, 1999"}],"fun_headline_variants":["GBE: First estimator for finite-horizon non-discounted safe RL constraints","Finite-horizon safe RL: gradient estimation outperforms advantage","Trust-region CGPO uses gradients to keep finite-horizon constraints","Gradient beats advantage in finite-horizon safe RL constraints"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument presumes that the gradient used in the Taylor estimate is the exact gradient of the true finite-horizon constraint function, while the implementation computes it through a short-horizon loss with a learned critic bootstrap that can carry bias this analysis does not charge.","fun_headline_variants_meta":{"raw":{"variants":["GBE: First estimator for finite-horizon non-discounted safe RL constraints","Finite-horizon safe RL: gradient estimation outperforms advantage","Trust-region CGPO uses gradients to keep finite-horizon constraints","Gradient beats advantage in finite-horizon safe RL constraints"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001102,"raw_usage":{"total_tokens":4630,"prompt_tokens":1014,"completion_tokens":3616,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":3538}},"tokens_in":630,"tokens_out":3616,"duration_ms":24112,"temperature":1.0,"reasoning_tokens":3538,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:16:32.596303+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the true constraint value $J_C(\\theta_{k+1})$ after an update in a differentiable environment where the exact full-horizon gradient is computable, and compare the prediction $J_C(\\theta_k)+\\delta^\\top q_k$ with it; if the error exceeds $\\frac{1}{2}\\hat\\delta\\max\\|\\nabla^2 J_C\\|$ at small $\\hat\\delta$, or if replacing the exact gradient by the SHAC-style critic gradient pushes the relative error above 1.0, the claimed guarantee is false.","supporting_citations":[{"cited_title":"Constrained policy optimization, 2017","cited_arxiv_id":null,"evidence_quote":"Supplies the Advantage-based Estimation formula (Equation 2) that the paper argues fails for finite-horizon constraints, and provides the CPO baseline."},{"cited_title":"and Langford, J","cited_arxiv_id":null,"evidence_quote":"Gives the infinite-horizon advantage identity (Equation 1) that ABE approximates, which is the source of the mismatch with finite-horizon constraints."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the short-horizon actor-critic gradient (SHAC) used to compute objective and constraint gradients in the implementation."},{"cited_title":"D., Frey, E., Raichuk, A., Girgin, S., Mordatch, I., and Bachem, O","cited_arxiv_id":null,"evidence_quote":"Brax is the differentiable physics engine on which the four test tasks are built."},{"cited_title":"J., Simchowitz, M., Zhang, K., and Tedrake, R","cited_arxiv_id":null,"evidence_quote":"Classifies zero-order and first-order batch gradient methods and supports the choice of first-order gradients for accurate constraint estimation."},{"cited_title":"Constrained Markov decision processes, volume 7","cited_arxiv_id":null,"evidence_quote":"Defines the CMDP framework with a constraint set and feasible policy set, the formal setting of the paper."}],"review_version":1}