{"id":"8d5bf74a-99e1-4527-a542-efc4e242a8d3","arxiv_id":"2608.10204","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A policy-gradient algorithm, BSPG, explicitly drives the safety-cost residual to zero from either side of the feasibility boundary and proves residual convergence and stationarity on the active constraint set for the exact-gradient update.","lead":"This paper proposes a reinforcement learning method that guides a policy to operate exactly at the boundary of what safety allows, rather than staying conservatively inside the safe zone. The method aims to turn unused safety margin into extra reward, with convergence guarantees for an idealized version of the algorithm.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Assumption 2's lower bound on cost-gradient norm is load-bearing; it can fail exactly at cost-minimizing boundary optima, narrowing the residual-convergence claim.","rationale":"The reader's weakest assumption is Assumption 2, and my stress-test identifies the same load-bearing point. The exact-gradient theory is internally consistent: the proofs of Theorems 6.1-6.4 follow from the stated assumptions, and the paper is transparent that the practical PPO implementation is not covered. My objection is not that the proofs are wrong, but that Assumption 2 is not a harmless regularity condition. It is needed to make the decomposition (13) well-defined and to produce the negative drift in Eq. (20). Cost gradients vanish at deterministic cost-minimizing policies in standard softmax parameterizations, and when the cost threshold is set at the minimum feasible cost, the active boundary optimum is exactly such a point. The theorem then proves boundary convergence only in cases where the boundary is nondegenerate, which may exclude the regime that motivates boundary seeking. This does not contradict the stated theorem, but it weakens the practical force of the central claim, reinforcing the reader's conditional verdict rather than overturning it. The experiments are too thin to verify Assumption 2 or to establish the empirical superiority claim: no seeds, no variance bars, one environment. If the authors added the proposed toy experiment, it would either exhibit the failure mode or show that the degeneracy does not arise in practice; either way, the scope of the central claim would be clarified.","tokens_in":13913,"tokens_out":13277,"duration_ms":142470,"concrete_test":"Run exact-gradient BSPG on a one-state, two-action tabular CMDP with cost threshold d equal to the minimum achievable cost, so the constrained optimum is the cost-minimizing policy and g_c=0 there; track the residual δ_t. If δ_t fails to converge to zero while ∥g_c∥ approaches zero, Assumption 2 is violated at a boundary optimum and the theorem's scope excludes an active-boundary regime central to the method's motivation. As a supplementary check, record min_t∥g_c(θ_t)∥ during the reported SafetyPointGoal runs to assess whether any positive μ is plausible in that setting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The residual-convergence theorem (Thm 6.1) depends on the negative drift term μ'α_tδ_t^2 in Eq. (20), with μ' = ημ²/(G+ε), where μ>0 is Assumption 2's uniform lower bound ∥g_c(θ)∥≥μ on the set containing all iterates and segments. If ∥g_c∥ approaches zero, the normal attraction term g_b = -δ g_c/(∥g_c∥+ε) has projection -δ²∥g_c∥²/(∥g_c∥+ε) on g_c, which vanishes; the Lyapunov recursion loses its δ² contraction, and δ_t→0 is not established. This is not a remote technicality. In a softmax/tabular parameterization, g_c vanishes at a deterministic cost-minimizing policy. If d equals the minimum achievable cost, the active set M contains exactly that point: a boundary optimum in the sense of Theorem 4.1 that violates Assumption 2. Thus BSPG cannot be guaranteed to reach a natural class of active optima. The paper labels Assumption 2 a nondegeneracy condition but never verifies it on SafetyPointGoal or analyzes the failure mode where the constrained optimum has g_c=0. Since Theorem 6.1 is the core support for the abstract's boundary-convergence claim, the guarantee is materially narrower than the prose suggests.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies constrained Markov decision processes and proposes Boundary-Seeking Policy Gradient (BSPG), a first-order update that splits the search direction into a tangential component that improves reward while preserving cost to first order and a signed, residual-driven normal component that attracts the policy toward the constraint boundary from either side. The update admits an algebraic Lagrangian form with an induced multiplier and no learned dual variable. For the idealized exact-gradient update with exact residual, the paper proves that the constraint residual satisfies a summable-square bound and converges to zero from either side with an O(1/sqrt(T)) finite-horizon rate (Theorem 6.1 and Corollary 6.2), that the tangential component is a reward-ascent direction (Theorem 6.3), and that any convergent parameter sequence converges to a stationary point on the active constraint set, satisfying KKT conditions under local optimality (Theorem 6.4). A boundary-optimality principle for occupancy-measure CMDPs is given in Theorem 4.1. Experiments on SafetyPointGoal1-v0 compare BSPG with CRPO and ESPO and report higher reward with tighter boundary proximity. The paper explicitly disclaims convergence guarantees for the stochastic PPO-based implementation and labels the residual estimator in Eq. (19) a heuristic.","tokens_in":14132,"tokens_out":11832,"duration_ms":118275,"significance":"If the results hold, the main contribution is a simple, dual-free first-order update with genuinely two-sided regulation of the constraint residual, complementing analyses that only certify feasibility. The proofs in Appendices B and D are internally consistent: the Lyapunov argument leading to Eq. (20) is valid, and the stationarity argument in Theorem 6.4 is sound. The paper is also honest about scope: it disclaims convergence for the practical PPO implementation, labels Eq. (19) an implementation heuristic, and avoids claiming a general empirical ordering. The boundary-proximity metric is a useful diagnostic that is rarely reported in safe RL. However, the theory rests on a strong uniform lower bound on the cost-gradient norm (Assumption 2), which is not verified empirically and can fail at natural boundary optima; this materially narrows the practical reach of the headline residual-convergence guarantee. Overall, the paper makes a credible theoretical contribution, but the gap between the stated assumptions and the settings of interest needs to be addressed before publication.","major_comments":[{"comment":"The contraction in Theorem 6.1 depends critically on Assumption 2's uniform lower bound ||g_c(theta)|| >= mu > 0. It enters the coefficient mu' = eta*mu^2/(G+epsilon) through Eq. (27), where the negative drift is -delta_t^2 * ||g_c||^2 / (||g_c||+epsilon). If ||g_c|| approaches zero, this drift vanishes and the Lyapunov recursion (20) loses its contraction, so delta_t -> 0 is no longer established. This is not a remote corner case: in a tabular softmax parameterization, a deterministic cost-minimizing policy has g_c = 0; if the threshold d equals the minimum achievable cost, the constrained optimum highlighted by Theorem 4.1 lies exactly at such a point and violates Assumption 2. The paper calls Assumption 2 a nondegeneracy condition but does not verify it on SafetyPointGoal1-v0 or analyze this failure mode. Since Theorem 6.1 is the main support for the abstract's two-sided boundary-convergence claim, the guarantee is materially narrower than the prose suggests. Please either weaken the assumption (for example, by adding a perturbation or regularization argument that handles g_c = 0), restrict the headline claims to the nondegenerate regime, or provide evidence that the regime is relevant to the experiments.","section":"Section VI, Assumption 2 and Eq. (20)"}],"minor_comments":[{"comment":"The empirical evaluation reports only one environment and two baselines, with no seed count, no confidence intervals or standard deviations, and no final numerical table. Please add multiple seeds with error bars and a table of final mean and variance values; the qualitative curves alone are difficult to assess.","section":"Section VII, Fig. 3"},{"comment":"The proof of Theorem 4.1 appears twice, once in the main text and again in Appendix A. Please delete one copy or make the appendix version a cross-reference.","section":"Section IV and Appendix A"},{"comment":"The sentence 'The proof, which relies only on the convexity of the occupancy-measure polytope and the linearity of Jr, Jc in the occupancy measure [4].' is a fragment; it should be completed or merged with the preceding paragraph.","section":"Section IV, after proof of Theorem 4.1"},{"comment":"The residual estimator is explicitly labeled a heuristic, but the two terms in Eq. (19) are averaged under different state distributions. It would help to state more precisely what conditions (for example, unbiasedness of the critic and on-policy sampling) would be needed for the estimator to be consistent, or to remove the second term if it is purely ad hoc.","section":"Section V-D, Eq. (19)"},{"comment":"The algorithm does not specify the values of eta, delta_max, epsilon, the PPO clip range, or the learning-rate schedule used in the experiments. Please include these details to make the experiments reproducible.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The theoretical core appears sound, and the paper's explicit scope disclaimers are a strength rather than a weakness. The main concern is that Assumption 2, which is load-bearing for Theorem 6.1, can fail at natural boundary optima; this should be addressed by either relaxing the assumption, restricting the claims, or providing empirical support for the nondegenerate regime. The empirical section is thin for a paper whose abstract makes an empirical claim; I would recommend requesting multiple seeds and at least one additional environment during revision. I saw no evidence of circular reasoning or unsupported 'parameter-free' claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Know this about the paper: BSPG splits the safe-RL update into a tangential reward-ascent term and a signed residual-driven normal term, and it is the first in this literature to push the policy toward the constraint boundary explicitly from both sides. The exact-gradient theory is internally consistent; I checked the Lyapunov argument in Appendix B and the limit characterization in Appendix D, and they hold under the stated assumptions. The paper is also honest about scope: it disclaims convergence for the PPO implementation, flags the residual estimator in Eq. (19) as a heuristic, and does not oversell the experiments.\n\nThe genuinely new things: the two-sided signed normal component, absent from ESPO, PCRPO, and the primal-dual methods; the implicit-multiplier identity (Eqs. 16-18), which is an algebraic consequence of the decomposition rather than a learned dual variable, so there is no circularity; and the boundary-proximity diagnostic metric, which is useful and, to my knowledge, new to the safe-RL evaluation toolkit.\n\nSoft spots, in proportion. First, Assumption 2 is load-bearing and the stress-test concern lands. The contraction in Theorem 6.1 comes from the negative drift ημ²α_tδ_t²/(G+ε); when the cost gradient vanishes, the normal component's projection on g_c vanishes and the Lyapunov recursion loses its contraction. Because g_c is zero at a deterministic cost-minimizing policy under softmax/tabular parameterization, the assumption fails exactly at a natural class of boundary optima, namely when the threshold d sits at the minimum achievable cost. Theorem 6.1 is then not applicable, so the abstract's 'from either side' convergence claim is materially narrower than the prose suggests. This is a real caveat, not a fatal flaw: the theorem is valid under its stated condition, but the degenerate case deserves an explicit discussion or a separate analysis.\n\nSecond, the experiments are one environment, two baselines, with no seeds, variance, or hyperparameter values reported. That is enough to illustrate the mechanism, not enough to support the comparative claims. The paper says as much in its own caveat, which I appreciate, but the evaluation is an outline.\n\nThird, the interior-optimum regime is unexamined. When an unconstrained reward maximizer is feasible, Theorem 4.1 does not apply, and BSPG's boundary term actively pushes cost upward, away from the unconstrained optimum. If the gradient alignment is unfavorable there, the normal term can reduce reward, and there is no analysis or diagnostic for that situation.\n\nWho this is for: people working on gradient-manipulation and primal safe-RL methods. The theorem statements and proofs are worth a referee's time. My recommendation: send it to peer review, and let the authors earn acceptance by verifying or addressing Assumption 2, expanding the evaluation, and covering the interior case.","headline":"A genuinely two-sided boundary-seeking update with sound exact-gradient theorems, whose main caveats are a load-bearing regularity assumption that fails at a natural class of boundary optima and an evaluation too thin to support the comparative claims.","tokens_in":14725,"tokens_out":5834,"would_cite":true,"duration_ms":52694,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes and analyzes BSPG, a policy-gradient update that explicitly drives constrained policies to the safety boundary, proving the constraint residual converges to zero from either side and that convergent limits are…","keywords":["safe reinforcement learning","constrained Markov decision process","policy gradient","constraint boundary","gradient decomposition","KKT conditions","occupancy measure","primal-dual methods"],"falsifier":"Run exact-gradient BSPG on a CMDP whose cost function is constant over a region containing the constraint boundary (so $g_c=0$ there) and start with a nonzero residual. If the residual still converges to zero, Assumption 2 is unnecessary; if the update stalls or becomes undefined at the flat region, the theorem is limited to cost gradients bounded away from zero.","tokens_in":13654,"feed_emoji":"🛡️","tokens_out":9068,"duration_ms":78123,"temperature":0.7,"pith_summary":"Safe reinforcement learning usually treats \"feasible\" as the goal, but when the optimal constrained policy spends the entire safety budget, stopping in the interior wastes reward that could be converted from unused cost slack. This paper introduces Boundary-Seeking Policy Gradient (BSPG), a first-order update that splits the gradient into a tangential part (improve reward while preserving cost to first order) and a signed normal part (push the policy toward the cost limit, from either side, with strength proportional to $J_c(\\pi_\\theta) - d$). For the exact-gradient idealization the paper proves the residual shrinks to zero at rate $O(1/\\sqrt{T})$, that any convergent iterate lies on the active boundary and is first-order stationary there, and that the limit satisfies KKT conditions when it is locally optimal over the feasible set. On a standard navigation task, BSPG reports higher reward and tighter boundary adherence than the compared baselines.","feed_headline":"Policy update that seeks the safety limit beats interior-safe RL","feed_subtitle":"A tangential-plus-normal gradient step converts unused safety budget into reward while driving the constraint residual to zero.","key_machinery":"The load-bearing object is the orthogonal decomposition of the reward gradient with respect to the cost gradient: the tangential piece $g_r^\\perp$ moves reward uphill while leaving $J_c$ unchanged to first order, and the boundary-attraction piece $g_b$ is a signed residual times the normalized cost gradient. Together they define the BSPG direction $g_{\\mathrm{up}}=g_r^\\perp+\\eta g_b$, equivalently a gradient step on the Lagrangian with an induced multiplier. The theoretical analysis is carried by the Lyapunov function $V(\\theta)=\\tfrac12\\delta(\\theta)^2$: the normal term gives a negative drift proportional to $\\delta^2$, which yields the summability $\\sum_t\\alpha_t\\delta_t^2<\\infty$ and the residual bound $O(1/\\sqrt{T})$, while the tangential term is exactly the component that vanishes at first-order stationarity on the active set.","core_discovery":"The central claim is that the constraint residual $\\delta(\\theta)=J_c(\\pi_\\theta)-d$ should be used as a signed control signal, not merely as a feasibility alarm. The paper proves (Theorem 4.1) that whenever no unconstrained reward maximizer is feasible, every constrained-optimal stationary policy of the CMDP satisfies $J_c=d$, so the constrained optimum lies on the boundary $M$. The BSPG update combines the tangentially projected reward gradient $g_r^\\perp = g_r - \\frac{\\langle g_r,g_c\\rangle}{\\|g_c\\|^2}g_c$ with a signed normal attraction $g_b=-\\delta\\, g_c/(\\|g_c\\|+\\varepsilon)$; under exact gradients the paper shows $\\sum_t \\alpha_t \\delta_t^2 < \\infty$ and $\\delta_t\\to 0$ (Theorem 6.1), $\\langle g_r,g_r^\\perp\\rangle=\\|g_r^\\perp\\|^2\\ge 0$ (Theorem 6.3), and that any convergent limit $\\theta_*$ lies on $M$ with $g_r(\\theta_*)=\\lambda_* g_c(\\theta_*)$, $\\lambda_*=\\langle g_r,g_c\\rangle/\\|g_c\\|^2$, satisfying KKT when it is also a local maximizer over the feasible set (Theorem 6.4). The same direction equals $\\nabla_\\theta[J_r(\\pi_\\theta)-\\lambda_t J_c(\\pi_\\theta)]$ with an algebraic multiplier $\\lambda_t=\\langle g_r,g_c\\rangle/\\|g_c\\|^2 + \\eta\\delta/(\\|g_c\\|+\\varepsilon)$ and no learned dual variable.","pith_inferences":["Extension not made in the paper: the signed normal term could be summed across multiple constraints, so residuals that remain nonzero after training would identify which constraints are active.","Speculative extension: the algebraic multiplier $\\lambda_t$ could serve as a cheaper, oscillation-free proxy for a learned dual multiplier, but the paper does not test this.","The empirical claim rests on one navigation task; a multi-environment sweep would test whether boundary-tightness persists under different cost-gradient geometries.","The theory covers the exact-residual idealization; extending it to the estimated residual (19) under a critic-error bound is a natural open step the paper leaves unaddressed."],"forward_implications":["A safe RL policy trained with BSPG would use the full safety budget rather than settling in the feasible interior, converting unused slack into reward when reward and cost gradients align.","The induced algebraic multiplier removes the need to learn or tune a dual variable, simplifying constrained policy optimization pipelines.","Because the residual converges to zero from both sides, BSPG certifies an active constraint at the limit, a property that feasibility-only analyses such as CRPO's do not provide.","A convergent run ends at a first-order stationary point on the active boundary, satisfying KKT if it is locally optimal, so the final policy is locally maximal for reward under the equality constraint."],"supporting_citations":[{"why":"This reference supplies the occupancy-measure linear-programming view of CMDPs that Theorem 4.1 uses to prove the boundary-optimality result.","marker":"[4]"},{"why":"This is the CRPO baseline, whose feasibility-only convergence guarantee is contrasted in Remark 6.6 and which BSPG is compared against in the experiments.","marker":"[16]"},{"why":"This is PCRPO, the gradient-manipulation method contrasted in Remark 6.6 because its KKT limit need not have an active constraint.","marker":"[17]"},{"why":"This is ESPO, the gradient-geometry baseline whose one-sided update geometry is contrasted with BSPG's two-sided normal term and compared experimentally.","marker":"[18]"},{"why":"This is the policy gradient theorem that defines the reward and cost gradients on which the BSPG decomposition is built.","marker":"[24]"},{"why":"This is PPO, whose clipped surrogate objectives form the practical implementation of BSPG in Algorithm 1.","marker":"[25]"},{"why":"This reference provides the evaluation infrastructure used for the navigation experiment that supports the empirical claim.","marker":"[27]"}],"fun_headline_variants":["Boundary-seeking gradient rides the safety limit for extra reward","Signed constraint residual steers safe RL to the active boundary","Regulate to the safety edge: residual-driven policy update","Turn safety slack into reward with residual-signed descent","BSPG: tangential reward ascent plus boundary attraction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The convergence proof assumes the cost gradient never vanishes during training: there is a fixed positive lower bound on $\\|g_c(\\theta)\\|$ over all iterates, so the tangential/normal splitting and the Lyapunov argument stay well defined; if the cost landscape flattens out, the claim does not apply.","fun_headline_variants_meta":{"raw":{"variants":["Boundary-seeking gradient rides the safety limit for extra reward","Signed constraint residual steers safe RL to the active boundary","Regulate to the safety edge: residual-driven policy update","Turn safety slack into reward with residual-signed descent","BSPG: tangential reward ascent plus boundary attraction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000258,"raw_usage":{"total_tokens":1666,"prompt_tokens":1116,"completion_tokens":550,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":732,"completion_tokens_details":{"reasoning_tokens":471}},"tokens_in":732,"tokens_out":550,"duration_ms":5547,"temperature":1.0,"reasoning_tokens":471,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:11:11.304170+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run exact-gradient BSPG on a CMDP whose cost function is constant over a region containing the constraint boundary (so $g_c=0$ there) and start with a nonzero residual. If the residual still converges to zero, Assumption 2 is unnecessary; if the update stalls or becomes undefined at the flat region, the theorem is limited to cost gradients bounded away from zero.","supporting_citations":[{"cited_title":"Crpo: A new approach for safe reinforcement learning with convergence guarantee,","cited_arxiv_id":null,"evidence_quote":"This is the CRPO baseline, whose feasibility-only convergence guarantee is contrasted in Remark 6.6 and which BSPG is compared against in the experiments."},{"cited_title":"Balance reward and safety optimization for safe reinforcement learning: A perspective of gradient manipulation,","cited_arxiv_id":null,"evidence_quote":"This is PCRPO, the gradient-manipulation method contrasted in Remark 6.6 because its KKT limit need not have an active constraint."},{"cited_title":"Enhancing efficiency of safe reinforcement learning via sample manipulation,","cited_arxiv_id":null,"evidence_quote":"This is ESPO, the gradient-geometry baseline whose one-sided update geometry is contrasted with BSPG's two-sided normal term and compared experimentally."},{"cited_title":"Policy gradient methods for reinforcement learning with function approximation,","cited_arxiv_id":null,"evidence_quote":"This is the policy gradient theorem that defines the reward and cost gradients on which the BSPG decomposition is built."},{"cited_title":"Omnisafe: An infrastructure for accelerating safe reinforcement learning research,","cited_arxiv_id":null,"evidence_quote":"This reference provides the evaluation infrastructure used for the navigation experiment that supports the empirical claim."}],"review_version":1}