{"id":"14009478-57be-4a9d-9ac0-a9736e0e6313","arxiv_id":"2504.19473","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"SAC-CLF augments Soft Actor-Critic with an adaptive CLF-based safety filter and control smoothing, reporting improved stability in two simulated nonlinear control tasks.","lead":"This paper adds a safety filter to the Soft Actor-Critic reinforcement learning algorithm by solving a quadratic program with an adaptive control Lyapunov function constraint. It reports lower costs and smoother controls on a nonlinear benchmark and a satellite attitude control simulation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The safety guarantee breaks because Eq. (30) puts the slack variable only in the objective, not in the CLF constraint, so the QP can be infeasible under exactly the unmodeled disturbances the paper claims to handle; the adaptive-gain analysis does not supply the missing feasibility argument.","rationale":"The reader's weakest_assumption correctly identifies that D⊆B is unverified for the LQR-derived CLF, and that without this inclusion the QP constraint in Eq. (30) can be infeasible. I agree that this is a central gap. However, I would sharpen the load-bearing concern further: the QP's own formulation makes the gap irreparable as stated, because the slack variable ε does not appear in the constraint at all, so even the intended 'small, controlled violation' mechanism cannot activate. This is an internal inconsistency, not merely a missing numerical check, and it directly undermines the safety guarantee during learning, which is the paper's headline contribution. The adaptive-constraint analysis is also load-bearing: the paper explicitly claims robustness to unmodeled dynamics, but the adaptation law in Eq. (22) and its Laplace 'stability proof' in Eqs. (23)–(27) provide no rigorous bound on the actual V̇ and no feasibility recovery. The reader's verdict of REJECT is therefore supported by my independent concern, and I would keep it unchanged. I do not see a way that the existing experiments, even if their hyperparameter tables were corrected, could establish the central safety claim without either (a) adding ε to the QP constraint and proving feasibility or (b) verifying D⊆B on the true nonlinear model including disturbances. The lack of external baselines and the swapped hyperparameter tables (Tables II and III) further compound the evidence problem, but the infeasibility issue is logically prior: if the QP can have no solution, the method has no defined behavior at the moment it is needed most.","tokens_in":14196,"tokens_out":3285,"duration_ms":37960,"concrete_test":"Reproduce the exact QP in Eq. (30) on the NCT system with the reported LQR-derived P and with the same bounded model bias used in the adaptive-constraint experiments, starting from the initial condition used in Fig. 6/7. Count the number of timesteps over 500 steps on which the QP is infeasible (i.e., no u satisfies the CLF inequality and actuator limits). If the infeasibility count is nonzero, the safety guarantee fails because the algorithm has no fallback. As an analytical companion check, inspect the KKT conditions of Eq. (30): since ε does not appear in any constraint, the optimal ε is always 0, so the QP is a hard-constrained problem and feasibility is equivalent to the unverified condition e(t)∈B at every step.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SAC-CLF guarantees safety during learning. The operative mechanism is the QP in Eq. (30), whose constraint is ∂V/∂e [f(x)+g(x)u] ≤ −η(t)V(e), with u bounds. The slack variable ε appears only in the objective (the term Kεε), not in this inequality. Therefore the QP is a hard-constrained projection: whenever the CLF decrease condition cannot be met at the current state, there is no feasible solution and no defined fallback in Algorithm 1. Safety is then not merely degraded; the controller has no output.\n\nThis infeasibility is not a rare edge case. The CLF V(e)=e^T P e is obtained from a local linearization and is only verified in the paper for the nominal model around the equilibrium. Definition 3 requires D={e: V(e)<V0} with D⊆B, where B is the set of states admitting some input satisfying the CLF decrease condition, but the paper never verifies D⊆B for the NCT or satellite systems, nor gives V0. Theorem 2 proves nominal forward invariance assuming a CLF already exists; it does not establish that the LQR-derived V satisfies the condition over the simulated region. Under the model bias used in the adaptive-constraint experiments, the actual V̇ includes ∂V/∂e d(x), which is absent from Eq. (30)'s constraint, so even the nominal condition can fail.\n\nThe adaptive η(t) does not fix feasibility. Equations (15)–(22) attempt to compensate by choosing kη(t) such that −∂V/∂e d(x) = η0 kη(t) V(e), but d(x) is unknown and Eq. (22) only reacts after a discrepancy is observed, with no proof that the reaction keeps the constraint feasible during transients. The Laplace-domain argument in Eqs. (23)–(27) is not a valid stability proof: it treats a nonlinear, state-dependent adaptation law as a linear time-invariant system, and the 'pole' s2 = −ε/(η0 V(E(s))) mixes a function of the state with a Laplace variable in a way that is not mathematically defined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SAC-CLF, an integration of Soft Actor-Critic with a quadratic Control Lyapunov Function designed through linearization and LQR. The method solves a QP at each control step to project the RL-recommended action onto a set defined by a CLF decrease condition, with an adaptively tuned decay rate eta(t) and a smoothing term that penalizes changes from the previous control input. The authors claim three contributions: a task-specific CLF design, adaptive constraint strength for robustness to unmodeled dynamics, and safety-prioritized input smoothing. The paper includes a formal forward-invariance theorem, an adaptive-gain analysis based on a transfer function, Algorithm 1, and simulations on a nonlinear control benchmark and a satellite attitude control problem.","tokens_in":14648,"tokens_out":6259,"duration_ms":63635,"significance":"If the safety guarantee and robustness results were established, the paper would be a useful practical contribution: the LQR-based construction gives a concrete, non-circular way to obtain a CLF candidate for nonlinear systems, and the integration with SAC addresses a real deployment concern in safe reinforcement learning. The experiments cover two relevant aerospace-style benchmarks, and the idea of adapting the CLF constraint severity is interesting. However, the central theoretical claims are not supported: the implemented QP does not contain the claimed slack relaxation, the forward-invariance proof is circular, and the adaptive analysis is not a valid stability argument. The paper also does not provide code or a full hyperparameter set, which limits reproducibility. The significance of the contribution therefore cannot be assessed until these load-bearing issues are resolved.","major_comments":[{"comment":"In Eq. (30), the slack variable ε appears only in the objective term Kεε and not in the CLF inequality, so the stated relaxation is not implemented: any feasible point is solved with ε=0 and the QP is hard-constrained. The text on page 6 says ε allows \"a small, controlled violation\", but the constraint in Eq. (30) is the hard condition ∂V/∂e [f(x)+g(x)u] ≤ −η(t)V(e). Consequently, whenever the CLF decrease condition cannot be met at the current state, the QP is infeasible and Algorithm 1 has no defined control output; this includes states reached under the unmodeled dynamics d(x) that the adaptive constraint is supposed to handle. The claimed safety during learning is therefore not delivered by the implemented mechanism.","section":"III-D, Eq. (30)"},{"comment":"The proof of Theorem 2 assumes the property it is supposed to establish: step 4 states that \"If the CLF decrease condition is satisfied at e(0), and the system dynamics preserve this property, then it is satisfied for all e(t)\", which is exactly forward invariance. The theorem also assumes a control law u=k(e) with V̇≤0 exists along the trajectory, but never shows that the QP in Eq. (30) produces such a control, nor that the LQR-derived V(e)=eᵀ P e satisfies the condition in Definition 2 over the operating region. Definition 3 requires D⊆B, but the paper neither computes V0 nor verifies D⊆B for either the NCT system or the satellite model; without that, Theorem 2 does not apply to the experiments.","section":"III-A, Theorem 2"},{"comment":"The adaptive-gain analysis does not establish robustness. Eq. (19) asks that −∂V/∂e d(x)=η0 kη(t) V(x), which requires exact knowledge of the unknown disturbance d(x) to choose kη(t); Eq. (22) is a first-order heuristic whose convergence to that value is not proven. The transfer function in Eq. (23) treats the time-domain quantity V(e(t)) as a Laplace variable V(E(s)), and Eq. (25) gives a \"pole\" s2 that depends on s through V(E(s)); the left-half-plane conclusion is therefore not a valid stability statement. These gaps matter because Eq. (30) omits d(x), so the adaptation is the only mechanism claimed to compensate for model bias, but no bound on ∂V/∂e d(x) or feasibility certificate is provided.","section":"III-B, Eqs. (15)-(27)"},{"comment":"The hyperparameter tables appear to invert the experimental conditions. In Table II, the satellite rows labeled \"adaptive constraints\" have ωη=0.0 while the corresponding \"constant constraints\" rows have ωη=0.01; in Table III, the rows labeled \"with vibration-dampening term\" set β=0.0 and the rows labeled \"without\" set β=1.0, whereas Eq. (30) defines β as the smoothing weight. As written, the figures labeled as showing adaptive or smoothed versions are actually the non-adaptive or unsmoothed configurations, so the empirical support for contributions (2) and (3) is not reproducible from the reported settings.","section":"IV-C and IV-D, Tables II and III"}],"minor_comments":[{"comment":"Theorem 1 is a restatement of the standard CLF definition, not a theorem; consider calling it Definition.","section":"III-A, Theorem 1"},{"comment":"The notation switches between V(e(t)) and V(x(t)) in the same derivation; unify the argument of the Lyapunov function to avoid ambiguity.","section":"III-B, Eqs. (16)-(17)"},{"comment":"The simulations compare only variants of the proposed method; the conclusion that SAC-CLF \"outperforms other leading algorithms\" is not supported by any baseline such as unconstrained SAC, CPO, or Lagrangian methods.","section":"IV"},{"comment":"Reproducibility would benefit from reporting the LQR weights Q and R, learning rates, network sizes, and the actual values of Kε (whether 10^8 or 108 in the tables).","section":"IV, Tables I-III"},{"comment":"Minor typos and notation issues include \"adpative\" in Table II, and the inconsistent use of V(x(t)) versus V(e(t)) in Eqs. (19)-(22).","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The paper is aimed at a control and aerospace audience, and the theoretical safety guarantee is central to its claims. Because the QP in Eq. (30) does not implement the stated slack relaxation, the forward-invariance proof is circular, and the adaptive gain analysis is mathematically invalid, the main contribution is not established. The additional inconsistency in the hyperparameter tables for the adaptive and smoothing experiments means the empirical claims are also not reproducible as reported. These are not merely presentation issues; they would require reworking the core algorithm, proving a genuine forward-invariance theorem with verified D⊆B, and re-running the experiments with corrected settings. For these reasons I recommend rejection rather than minor or major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the SAC-CLF paper. The engineering intuition is fine, but the central safety claim doesn't hold up. In the main QP, Eq. (30), the slack variable ε appears only in the objective; the CLF constraint is hard. So when the LQR-derived CLF cannot be decreased fast enough—which is exactly the situation under the unmodeled disturbance the paper aims to handle—the QP becomes infeasible and the controller has no defined output. That voids the promised safe-exploration guarantee.\n\nCredit where due: the paper openly identifies real limitations of CLF-based safe RL, and the combination of an LQR-designed CLF, an adaptive constraint strength, and a command-smoothing term is a plausible engineering recipe. The simulations on the NCT system and satellite attitude control show qualitative benefits—the LQR CLF beats a unit-matrix CLF on the satellite task, and the smoothing term clearly reduces command chatter.\n\nThe soft spots are load-bearing, not cosmetic. The Laplace-domain analysis in Eqs. (23)–(27) is not a valid stability proof; it treats V(E(s)), a state-dependent quantity, as if it were a constant in the Laplace domain, and the \"pole\" s2 = −ε/(η0 V(E(s))) is not a pole in any standard sense. Theorem 2's forward-invariance proof assumes the CLF decrease condition holds at every future state, which is precisely what it is meant to prove. The hyperparameter tables for the two ablation studies are swapped: Table III gives β=0 for \"with dampening\" and β=1 for \"without,\" which is backwards. And the conclusion claims superiority over other algorithms, but there are no external baselines and no code or data.\n\nThe adaptive constraint law is an interesting heuristic. With a correct feasibility argument and a real Lyapunov analysis, it could become a contribution. As written, the theory doesn't support the method, and the experiments don't support the claims.\n\nThis paper is for practitioners who want a drop-in stability filter for SAC; they'll find useful ideas here, but they should not rely on the guarantees. I would not send it to peer review in its current form. I'd ask the authors to fix the QP, replace the Laplace analysis with a sound adaptive-stability proof, correct the tables, add baselines, and resubmit. A revised version could be a decent paper; this one isn't there yet.","headline":"The paper's engineering instinct is fine but the safety guarantee collapses: the QP slack is in the wrong place, the adaptive-stability analysis isn't a proof, and the ablation tables are swapped.","tokens_in":15212,"tokens_out":4641,"would_cite":false,"duration_ms":44006,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SAC-CLF filters a soft actor-critic policy through a quadratic-programming controller built on an LQR-derived control Lyapunov function, and claims this keeps learning safe and stable on nonlinear and satellite attitude tasks.","keywords":["Reinforcement learning","Safe exploration","Control Lyapunov function","Quadratic programming","Soft Actor-Critic","Satellite attitude control","Adaptive constraints","Control input smoothing"],"falsifier":"Run the NCT or satellite model from an initial error inside the claimed safe ball D with the LQR-derived P and check whether the QP in Eq. (30) ever becomes infeasible or allows V(e) to rise above V0 during an episode; one such trajectory would show the D subset of B assumption, on which the safety guarantee rests, fails for that system.","tokens_in":13887,"feed_emoji":"🛰️","tokens_out":4037,"duration_ms":44201,"temperature":0.7,"pith_summary":"The paper tries to establish that a reinforcement learning agent can explore and learn in safety-critical control tasks without leaving a certified stable region. It introduces SAC-CLF, which takes each action proposed by Soft Actor-Critic and projects it through a quadratic program that enforces a control Lyapunov function decrease condition, so the closed-loop state stays inside a safe energy ball. Three mechanisms carry the argument: an LQR-based procedure to design the Lyapunov function for nonlinear systems, an adaptive tightening or relaxing of the stability constraint to handle unmodeled dynamics, and a smoothing term that penalizes abrupt control changes while still prioritizing safety. If this works as claimed, model-free RL could be applied to real physical systems like spacecraft during training rather than only after extensive simulation.","feed_headline":"Lyapunov filter lets RL agents learn without leaving safety","feed_subtitle":"A quadratic-programming Lyapunov filter adds stability guarantees and smoother commands to soft actor-critic training.","key_machinery":"The central object is the quadratic program in Eq. (30): given the RL action u_RL(t), it solves for the control input u(t) closest to that action while satisfying the CLF condition partial V/partial e [f(x)+g(x)u] <= -eta(t)V(e), actuator limits, and a smoothing penalty against the previous input. The Lyapunov function is built from the positive definite matrix P obtained by solving the algebraic Riccati equation for the linearized nominal system, so V(e)=e^T P e. The adaptive strength eta(t)=eta0(1+k_eta(t)) is updated from the discrepancy between desired and actual Lyapunov derivatives, and the $\\beta$-weighting term makes the input a compromise between the RL recommendation and the previously applied control. This QP layer is what converts an unconstrained RL policy into one that, by construction, respects a stability certificate at every step.","core_discovery":"The central claim is that solving the quadratic program in Eq. (30) at each control step, with a Lyapunov function V(e)=e^T P e obtained by linearizing the system and solving the algebraic Riccati equation, keeps the state inside a safe energy ball D={e: V(e)<V0} while the RL policy keeps learning. The QP minimizes the deviation from the RL-suggested action plus a vibration-dampening term, subject to the CLF decrease constraint and actuator limits. An adaptive parameter eta(t) adjusts the constraint's conservatism by comparing the desired and actual Lyapunov derivative, so the method claims robustness to model uncertainty without permanently sacrificing performance. Simulations on the NCT system and satellite attitude control are presented as evidence that the customized LQR-designed CLF outperforms a unit-matrix CLF, that adaptive constraints reduce cost and variance under model bias, and that the smoothing term reduces command oscillation.","pith_inferences":["An implication the authors leave implicit is that the same QP safety filter could be attached to other actor-critic algorithms beyond SAC, since the filter only needs a proposed action and a Lyapunov certificate to project onto.","The paper's proof of the safe energy ball assumes D is contained in the safe state set B, but it never verifies this containment computationally; a natural testable extension is to estimate the largest V0 for which D subset of B holds, which would turn the guarantee from formal to practically certified.","The adaptive eta(t) rule is a heuristic driven by the derivative discrepancy; one could instead estimate the disturbance bound d(x) online and set eta(t) from that bound, making the robustness margin explicit rather than emergent.","The smoothing rule is equivalent to a constrained low-pass filter on the RL action; its real payoff would be on hardware with actuator limits and wear, so an experiment comparing actuator usage and tracking error on a physical testbed would sharpen the claim."],"forward_implications":["If the central claim is correct, RL agents can be trained on physical systems from the first episode while the controlled state remains inside a prescribed safe energy ball.","The LQR-based CLF design would replace hand-crafted Lyapunov functions for nonlinear regulator problems, giving a systematic recipe that starts from a local linear model and yields a global safety filter.","Adaptive constraint strength would let the method tighten safety margins when unmodeled dynamics push the Lyapunov derivative off target and relax them when the system is behaving well, so robustness does not demand permanently conservative control.","The vibration-dampening term would reduce actuator wear and command chatter without resorting to naive low-pass filtering that ignores the safety constraint.","On the demonstrated tasks, the method claims lower cost and lower variance than fixed-constraint baselines, meaning safer exploration also leads to better final performance rather than just constrained behavior."],"supporting_citations":[{"why":"Supplies the Soft Actor-Critic base algorithm that SAC-CLF extends with the Lyapunov safety filter.","marker":"[24]"},{"why":"Provides the control-barrier-function quadratic programming framework that motivates enforcing safety constraints through a QP layer.","marker":"[13]"},{"why":"Shows CLF-based quadratic programs used in real-time bipedal walking with torque saturation, the direct precedent for Eq. (30).","marker":"[25]"},{"why":"Establishes the control Lyapunov function background and rapidly exponentially stabilizing CLF concepts that the paper builds on.","marker":"[27]"},{"why":"Defines the NCT system used as one of the two simulation testbeds for validating the method.","marker":"[28]"}],"fun_headline_variants":["Adaptive Lyapunov filter keeps RL safe while learning","SAC-CLF: RL with built-in stability guarantees","Safe RL via adaptive Lyapunov constraints","Lyapunov-based RL smooths commands and guarantees stability"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantee rests on assuming that the quadratic function V(e)=e^TPe from the linearized LQR problem is a valid control Lyapunov function for the true nonlinear system over an energy ball D that lies entirely inside the set of states where some admissible input makes V decrease, and that training starts inside that ball; the paper states but never verifies this containment for either test system.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive Lyapunov filter keeps RL safe while learning","SAC-CLF: RL with built-in stability guarantees","Safe RL via adaptive Lyapunov constraints","Lyapunov-based RL smooths commands and guarantees stability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000605,"raw_usage":{"total_tokens":2808,"prompt_tokens":915,"completion_tokens":1893,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":1829}},"tokens_in":531,"tokens_out":1893,"duration_ms":13600,"temperature":1.0,"reasoning_tokens":1829,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:02:36.827303+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the NCT or satellite model from an initial error inside the claimed safe ball D with the LQR-derived P and check whether the QP in Eq. (30) ever becomes infeasible or allows V(e) to rise above V0 during an episode; one such trajectory would show the D subset of B assumption, on which the safety guarantee rests, fails for that system.","supporting_citations":[{"cited_title":"Torque saturation in bipedal robotic walking through control lyapunov function-based quadratic programs,","cited_arxiv_id":null,"evidence_quote":"Shows CLF-based quadratic programs used in real-time bipedal walking with torque saturation, the direct precedent for Eq. (30)."},{"cited_title":"Rapidly exponentially stabilizing control lyapunov functions and hybrid zero dynamics,","cited_arxiv_id":null,"evidence_quote":"Establishes the control Lyapunov function background and rapidly exponentially stabilizing CLF concepts that the paper builds on."},{"cited_title":"Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem,","cited_arxiv_id":null,"evidence_quote":"Defines the NCT system used as one of the two simulation testbeds for validating the method."}],"review_version":1}