{"id":"7a3b3b95-d1f4-4429-9e55-7e3966d59e15","arxiv_id":"2607.22201","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"low","formal_verification":"none","parameter_count":1,"one_line_summary":"Trajectory KL regularization in stochastic optimal control reduces to a quadratic drift penalty, reducing TRSOC to standard SOC (and LQR with modified control weight).","lead":"A control paper shows that adding a KL penalty between the controlled and a reference trajectory distribution is equivalent to adding a quadratic drift-mismatch term to the running cost, so the regularized problem stays a standard stochastic optimal control problem. The authors derive the corresponding HJB equation and a modified Riccati equation, and illustrate a performance-vs-reference trade-off in simulations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 5's explicit feedback law is derived as an unconstrained minimizer, but the paper's admissible set U is only assumed closed convex; for constrained U the formula is not the minimizer of (27).","rationale":"This is load-bearing because Proposition 5 is a headline contribution, not a side remark. The proof's convexity argument only establishes uniqueness over R^m, not over a constrained closed convex set. The control-affine quadratic specialization is meant to characterize the optimal policy; without an interior-point or U = R^m assumption, that characterization is invalid. The issue does not affect Lemma 1 or the HJB identity itself, so the reader's CONDITIONAL verdict remains appropriate; however, the scope of the closed-form results must be narrowed. I partially agree with the reader's emphasis on Assumption 1, but that limitation is explicitly acknowledged in Remark 1, whereas the U = R^m gap is unstated and directly contradicts the proof of Proposition 5.","tokens_in":11912,"tokens_out":16085,"duration_ms":170035,"concrete_test":"Take a scalar instance of (27) with U = [-1,1], eR(x)=1, B(x)=1, and ∇V(x)=2. Formula (28) gives u* = -2, which is not in U; the true minimizer of the quadratic over U is u* = -1. Recomputing Proposition 5 for this U shows the proposition is false as stated. A fix is to add 'assume U = R^m' before Proposition 5, or to replace (28) by the projection of -eR^{-1}B^T∇V onto U.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper defines the admissible set U in Section II as a closed convex subset of R^m. Proposition 5, however, computes the minimizer of (27) by differentiating in u and solving eR(x)u + B(x)^T∇V(x) = 0, then asserts this is the unique minimizer. This is valid only if U = R^m (or if the stationary point is interior to U). If U is a proper closed convex set, the argmin of a strictly convex quadratic over U is the projection of the unconstrained point onto U, with KKT normal-cone conditions; formula (28) is generally infeasible. The subsequent closed-loop dynamics (30), the Riccati equation (50), and the LQ solution inherit this gap. Since U = R^m is not stated in Section V, the central 'explicit optimal feedback law' claim is not proven under the paper's own admissibility assumptions. This is distinct from the shared-diffusion issue, which is explicitly acknowledged in Remark 1.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces trajectory-regularized stochastic optimal control (TRSOC), which augments a standard stochastic optimal control objective with the KL divergence between the controlled and reference trajectory distributions. Under a common diffusion coefficient and via Girsanov's theorem, the trajectory KL is reduced to a quadratic drift-mismatch penalty, producing a modified running cost that preserves dynamic programming structure. The authors state finite- and infinite-horizon DP and HJB equations, give an explicit feedback formula for control-affine quadratic-cost problems, analyze closed-loop stability through Foster-Lyapunov conditions, specialize to LQ systems with a modified Riccati equation, and present numerical experiments showing a performance-versus-deviation trade-off, including with a learned reference drift.","tokens_in":12161,"tokens_out":9629,"duration_ms":98903,"significance":"The core identity in Lemma 1 is standard and correctly applied, and the resulting interpretation of TRSOC as standard SOC with an augmented running cost is elegant and potentially useful for behavior-regularized control. The paper is also honest about the shared-diffusion limitation in Remark 1. The LQ specialization gives a clean noise-aware control-weight modification, and the experiments illustrate the intended trade-off. However, two technical gaps affect the central formulas: the control-set issue in Proposition 5 and a notational collision around f0 that drops a baseline controller from the drift penalty. The proof of Proposition 4 also uses an unstated nonnegativity assumption. These are fixable, but until they are addressed the claimed explicit feedback law and LQ solution are not established in the stated generality.","major_comments":[{"comment":"The symbol f0 is used both for the reference drift in Eq. (2) and for the zero-control drift in the control-affine decomposition Eq. (23). If the reference is generated by a baseline controller u0, as described in Section II and used in Section VIII, then the reference drift is f(x,u0(x)), not the zero-control drift. The drift mismatch is then B(x)(u-u0(x)), so the KL penalty in Eq. (24) should contain (u-u0)^T B^T(σσ^T)^{-1}B(u-u0), which introduces linear and constant terms in u. Formula (28), the closed-loop dynamics (30), and the Riccati equation (50) are the solution only for u0=0. The deviation metric (57) uses u_t-u_ref_t, confirming the mismatch between the theory and the experiments. Please introduce separate notation for the zero-control drift and re-derive the HJB minimizer for a general baseline reference.","section":"Section V, Eqs. (23)-(24) and (28); Section VIII, Eq. (57)"},{"comment":"The admissible set U is introduced in Section II as a closed convex subset of R^m. The proof of Proposition 5 differentiates the quadratic expression and solves eR(x)u + B(x)^T ∇V(x)=0, then invokes convexity to claim the minimizer. Convexity gives uniqueness of the unconstrained minimizer only when U=R^m. If U is a proper closed convex set, the minimizer is the projection of the unconstrained point onto U, characterized by KKT/normal-cone conditions, and (28) may not be admissible. Since Eq. (30), the optimal feedback (49), and the ARE (50) all inherit this step, the paper should either assume U=R^m explicitly or state the constrained minimizer.","section":"Section V, Proposition 5, Eq. (28)"},{"comment":"In the proof of the reference-preserving limit, the inequality chain drops the term J0_x(uλ) with the comment 'Dropping J0_x(uλ)≥0'. Nonnegativity of the unregularized cost is not assumed in the problem formulation; only continuity and polynomial growth are stated in Section II. If ℓ can take negative values, the inequality λ/2 e^{-ρT} E[...] ≤ J0_x(u0) is not justified. The result needs an explicit assumption that ℓ (and hence J0) is nonnegative or bounded below, or a modified argument that does not require dropping a possibly negative term.","section":"Section IV, Proposition 4"}],"minor_comments":[{"comment":"The proposition numbering is duplicated: Proposition 6 appears both for the exponential moment bound and for the value-function Lyapunov result, and Proposition 7 appears both for the invariant measure and for the discounted ARE result.","section":"Throughout; Section VI and VII"},{"comment":"There is a typographical error in the definition of x_ref(t): the bracket expression reads [pref(t), vref(t))] and should be [pref(t), vref(t)].","section":"Section VIII, Eq. (56)"},{"comment":"The text 'The authors are withHybrid Robotics' is missing a space; it should read 'with Hybrid Robotics'.","section":"Section I, author footnote"},{"comment":"The DP theorem and verification theorem are stated under 'standard assumptions' without proof. This is acceptable as a citation to standard SOC theory, but the manuscript would be clearer if it explicitly listed the regularity conditions (including existence of an admissible minimizer in Theorem 2) rather than referencing them only implicitly.","section":"Section III, Theorems 1 and 2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a correct and useful core reduction, but the two main technical issues are substantive enough that the advertised explicit feedback law does not follow under the paper's own assumptions. Both are repairable in a revision: clarify the reference-drift notation and re-derive the controlled-affine/LQ formulas for a u0-induced reference, and either assume U=R^m or derive the projected minimizer. I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a serviceable paper that re-derives a known connection—trajectory KL between two diffusions with the same noise reduces to a quadratic drift penalty—and wraps it in a control-theoretic package. The derivation is standard, and the LQ case just gives discounted LQR with a modified control weight. There's a real gap in Proposition 5: the paper defines U as a closed convex subset of R^m, but the proof of (28) differentiates an unconstrained quadratic and ignores the constraint. That needs fixing before publication.\n\nWhat's good: Lemma 1 is correct and clearly stated, and the paper is upfront (Remark 1) that the quadratic representation relies on shared diffusion and drift-only control. The DP/HJB discussion is standard but sound. The LQ section cleanly shows the effective weight R+λB^T(ΣΣ^T)^{-1}B. The numerical experiments illustrate the trade-off, though they're light on detail.\n\nWhere it's soft: novelty. The identity is the classical Girsanov/KL-control result, and the paper itself cites [14], [15], [16]. Calling TRSOC a new framework oversells it; it's a useful reformulation. The constrained-control issue is more serious: if U is not all of R^m, the minimizer of (27) is the projection of the unconstrained point onto U, not (28). The paper does not state U=R^m anywhere in Section V. This affects the closed-loop dynamics and the Riccati equation. Theorem 1 and 2 rely on 'standard assumptions' without proof, which is fine for a short paper but should be stated precisely. Stability results depend on Assumptions 3 and 4, which may hold but are not verified.\n\nWho's this for: someone wanting a tidy statement of KL-regularized SOC in the HJB framework, or a baseline for behavior regularization in control/RL. It's not a breakthrough but it's a reasonable reference.\n\nRecommendation: send it to peer review. The core derivation is correct and the constrained-control gap is fixable (either restrict U=R^m or derive the projection form). It deserves a referee's time, but expect revisions to tighten the novelty claim and the admissible-set assumptions.","headline":"Clean reformulation of KL control as a drift penalty; the math is mostly right but the novelty is limited and Proposition 5's unconstrained minimizer ignores the paper's own constrained admissible set.","tokens_in":12657,"tokens_out":2132,"would_cite":false,"duration_ms":22158,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["49L20","93E20","60H10","49N10"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that KL-based trajectory regularization in stochastic optimal control is equivalent to adding a quadratic drift-mismatch cost, preserving the standard dynamic programming and HJB structure.","keywords":["stochastic optimal control","trajectory regularization","KL divergence","Girsanov theorem","HJB equation","dynamic programming","linear-quadratic regulator","reference dynamics"],"falsifier":"Take two scalar SDEs with different diffusion coefficients (e.g., dX = u dt + dW and dX = 0 dt + 2 dW), simulate paths, and estimate KL(P^u||P^0) by Monte Carlo; the result will diverge or will not match (1/2) E∫||σ^{-1}(f-f0)||^2 dt computed with a single σ, showing that the identity (5) requires the shared-diffusion condition. A more direct falsifier: for a control-affine quadratic problem with a known HJB solution, solve the HJB with the original R and with eR = R + λB^T(σσ^T)^{-1}B and compare the resulting value functions; any mismatch beyond numerical error would disprove Proposition 5.","tokens_in":11781,"feed_emoji":"🎯","tokens_out":4179,"duration_ms":37913,"temperature":0.7,"pith_summary":"This paper establishes that penalizing the KL divergence between the trajectory distribution induced by a controller and that of a reference dynamics is equivalent to adding a quadratic penalty on the drift mismatch to the running cost, provided both share the same diffusion and control enters only through the drift. Because the KL term becomes local and additive, the regularized problem keeps the dynamic programming structure and HJB characterization of standard stochastic optimal control. The paper derives an explicit optimal feedback law for control-affine quadratic systems, which turns out to be standard LQR with an inflated control weight; in the linear-quadratic case the solution is discounted LQR with a modified Riccati equation. Numerical examples show the regularization parameter interpolates between performance-driven and reference-preserving behavior, including references learned from offline data.","feed_headline":"Trajectory KL regularization reduces to a quadratic drift penalty","feed_subtitle":"Adding a KL term to the control objective preserves dynamic programming and yields explicit LQR-style feedback.","key_machinery":"The load-bearing object is the trajectory KL identity (Equation 5) and its time-shifted version (6), which follow from Girsanov's theorem under a shared, invertible diffusion and drift-only control. This identity converts the KL divergence between path measures into an additive running cost, so the regularized objective remains time-consistent and fits the standard SDE/Dynamic-Programming framework; all subsequent results—HJB, verification, explicit feedback, LQ/Riccati—are consequences of that reduction.","core_discovery":"The central discovery is the identity KL(P^u || P^0) = (1/2) E^{P^u} ∫_0^T ||σ^{-1}(f(X,u)-f0(X))||^2 dt (Lemma 1), obtained via Girsanov's theorem. This turns a trajectory-level, seemingly non-local KL regularization into an instantaneous quadratic cost on the difference between the controlled and reference drifts, scaled by the inverse diffusion. As a consequence, TRSOC is just standard SOC with the running cost ℓ + (λ/2)||σ^{-1}(f-f0)||^2, so the value function satisfies the usual HJB equation, and for control-affine dynamics with quadratic control cost the optimal feedback is u* = -(R + λB^T(σσ^T)^{-1}B)^{-1} B^T ∇V. In the linear-quadratic case this reduces to discounted LQR with contro","pith_inferences":["One could test the identity directly: for a simple scalar SDE with a known analytic KL, compare the Monte Carlo estimate of KL between controlled and reference paths with the quadratic drift integral to see how quickly finite-time errors vanish.","The framework suggests a practical recipe for offline-to-online RL: learn a reference drift from offline data, then solve TRSOC online; the paper's data-driven experiment illustrates this but leaves the theoretical guarantees for learned references open.","Because the regularization only penalizes drift mismatch, any uncertainty about the diffusion coefficient would corrupt the fidelity of the KL penalty; a natural extension is to make the penalty robust to misspecified σ.","The connection to KL control (optimizing over path measures) is structural: TRSOC stays within Markov controls, whereas the path-measure formulation can yield risk-sensitive objectives; exploring when the two coincide could transfer techniques between the literatures."],"forward_implications":["In control-affine systems with quadratic control cost, the optimal policy is computed from the unmodified HJB equation by replacing the control weight R with R + λB^T(σσ^T)^{-1}B; directions with small noise are penalized more heavily.","In the linear-quadratic specialization, TRSOC is exactly discounted LQR with an augmented control weight, so a stabilizing solution exists under standard stabilizability/detectability and the closed loop admits a Gaussian invariant measure.","As λ increases, the optimal controller's drift converges to the reference drift (Proposition 4), so λ traces a Pareto curve between task performance and deviation from the reference.","The performance loss from regularization is bounded by the reference-deviation of the unregularized optimal control (inequality (21)).","Because the KL penalty is local, no state augmentation or risk-sensitive expectation is needed; the regularized problem retains the Markov structure and the DP principle."],"fun_headline_variants":["Trajectory KL becomes a quadratic drift penalty in SOC","KL between trajectories reduces to a drift mismatch cost","Girsanov turns trajectory KL into a simple quadratic term","Stochastic control with KL: trajectory divergence is just a drift penalty","TRSOC: KL regularization collapses to a quadratic cost"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire reduction rests on the assumption that the controlled and reference SDEs share the same invertible diffusion coefficient and that control acts only on the drift, so that Girsanov's theorem makes the trajectory KL equal to the quadratic drift-mismatch integral; if the noise level or structure differs, or σ is control-dependent, the equivalence fails and the framework no longer reduces to standard SOC.","fun_headline_variants_meta":{"raw":{"variants":["Trajectory KL becomes a quadratic drift penalty in SOC","KL between trajectories reduces to a drift mismatch cost","Girsanov turns trajectory KL into a simple quadratic term","Stochastic control with KL: trajectory divergence is just a drift penalty","TRSOC: KL regularization collapses to a quadratic cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1087,"prompt_tokens":703,"completion_tokens":384,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":447,"completion_tokens_details":{"reasoning_tokens":304}},"tokens_in":447,"tokens_out":384,"duration_ms":4131,"temperature":1.0,"reasoning_tokens":304,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T05:27:59.213726+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take two scalar SDEs with different diffusion coefficients (e.g., dX = u dt + dW and dX = 0 dt + 2 dW), simulate paths, and estimate KL(P^u||P^0) by Monte Carlo; the result will diverge or will not match (1/2) E∫||σ^{-1}(f-f0)||^2 dt computed with a single σ, showing that the identity (5) requires the shared-diffusion condition. A more direct falsifier: for a control-affine quadratic problem with a known HJB solution, solve the HJB with the original R and with eR = R + λB^T(σσ^T)^{-1}B and compare the resulting value functions; any mismatch beyond numerical error would disprove Proposition 5.","supporting_citations":[],"review_version":1}