REVIEW 3 major objections 4 minor 21 references
Trajectory-Regularized Stochastic Optimal Control via KL Divergence
T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read This paper shows that KL-based trajectory regularization in stochastic optimal control is equivalent to adding a quadratic drift-mismatch cost, preserving the standard dynamic programming and HJB structure.
desk verdict Clean reformulation of KL control as a drift penalty; the math is mostly right but the novelty is limited and Proposition 5's unconstrained minimizer ignores the paper's own constrained admissible set. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the trajectory KL identity (Equation 5) and its time-shifted version (6), which follow from Girsanov's theorem under a shared, invertible diffusion and drift-only control. This identity converts the KL divergence between path measures into an additive running cost, so the regularized objective remains time-consistent and fits the standard SDE/Dynamic-Programming framework; all subsequent results—HJB, verification, explicit feedback, LQ/Riccati—are consequences of that reduction.
What would settle it
Take two scalar SDEs with different diffusion coefficients (e.g., dX = u dt + dW and dX = 0 dt + 2 dW), simulate paths, and estimate KL(P^u||P^0) by Monte Carlo; the result will diverge or will not match (1/2) E∫||σ^{-1}(f-f0)||^2 dt computed with a single σ, showing that the identity (5) requires the shared-diffusion condition. A more direct falsifier: for a control-affine quadratic problem with a known HJB solution, solve the HJB with the original R and with eR = R + λB^T(σσ^T)^{-1}B and compare the resulting value functions; any mismatch beyond numerical error would disprove Proposition 5.
Extended reading notes
Core claim
The central discovery is the identity KL(P^u || P^0) = (1/2) E^{P^u} ∫_0^T ||σ^{-1}(f(X,u)-f0(X))||^2 dt (Lemma 1), obtained via Girsanov's theorem. This turns a trajectory-level, seemingly non-local KL regularization into an instantaneous quadratic cost on the difference between the controlled and reference drifts, scaled by the inverse diffusion. As a consequence, TRSOC is just standard SOC with the running cost ℓ + (λ/2)||σ^{-1}(f-f0)||^2, so the value function satisfies the usual HJB equation, and for control-affine dynamics with quadratic control cost the optimal feedback is u* = -(R + λB^T(σσ^T)^{-1}B)^{-1} B^T ∇V. In the linear-quadratic case this reduces to discounted LQR with contro
Load-bearing premise
The entire reduction rests on the assumption that the controlled and reference SDEs share the same invertible diffusion coefficient and that control acts only on the drift, so that Girsanov's theorem makes the trajectory KL equal to the quadratic drift-mismatch integral; if the noise level or structure differs, or σ is control-dependent, the equivalence fails and the framework no longer reduces to standard SOC.
Editorial extensions
If this is right
- In control-affine systems with quadratic control cost, the optimal policy is computed from the unmodified HJB equation by replacing the control weight R with R + λB^T(σσ^T)^{-1}B; directions with small noise are penalized more heavily.
- In the linear-quadratic specialization, TRSOC is exactly discounted LQR with an augmented control weight, so a stabilizing solution exists under standard stabilizability/detectability and the closed loop admits a Gaussian invariant measure.
- As λ increases, the optimal controller's drift converges to the reference drift (Proposition 4), so λ traces a Pareto curve between task performance and deviation from the reference.
- The performance loss from regularization is bounded by the reference-deviation of the unregularized optimal control (inequality (21)).
- Because the KL penalty is local, no state augmentation or risk-sensitive expectation is needed; the regularized problem retains the Markov structure and the DP principle.
Reading between the lines
- One could test the identity directly: for a simple scalar SDE with a known analytic KL, compare the Monte Carlo estimate of KL between controlled and reference paths with the quadratic drift integral to see how quickly finite-time errors vanish.
- The framework suggests a practical recipe for offline-to-online RL: learn a reference drift from offline data, then solve TRSOC online; the paper's data-driven experiment illustrates this but leaves the theoretical guarantees for learned references open.
- Because the regularization only penalizes drift mismatch, any uncertainty about the diffusion coefficient would corrupt the fidelity of the KL penalty; a natural extension is to make the penalty robust to misspecified σ.
- The connection to KL control (optimizing over path measures) is structural: TRSOC stays within Markov controls, whereas the path-measure formulation can yield risk-sensitive objectives; exploring when the two coincide could transfer techniques between the literatures.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces trajectory-regularized stochastic optimal control (TRSOC), which augments a standard stochastic optimal control objective with the KL divergence between the controlled and reference trajectory distributions. Under a common diffusion coefficient and via Girsanov's theorem, the trajectory KL is reduced to a quadratic drift-mismatch penalty, producing a modified running cost that preserves dynamic programming structure. The authors state finite- and infinite-horizon DP and HJB equations, give an explicit feedback formula for control-affine quadratic-cost problems, analyze closed-loop stability through Foster-Lyapunov conditions, specialize to LQ systems with a modified Riccati equation, and present numerical experiments showing a performance-versus-deviation trade-off, including with a learned reference drift.
Significance. The core identity in Lemma 1 is standard and correctly applied, and the resulting interpretation of TRSOC as standard SOC with an augmented running cost is elegant and potentially useful for behavior-regularized control. The paper is also honest about the shared-diffusion limitation in Remark 1. The LQ specialization gives a clean noise-aware control-weight modification, and the experiments illustrate the intended trade-off. However, two technical gaps affect the central formulas: the control-set issue in Proposition 5 and a notational collision around f0 that drops a baseline controller from the drift penalty. The proof of Proposition 4 also uses an unstated nonnegativity assumption. These are fixable, but until they are addressed the claimed explicit feedback law and LQ solution are not established in the stated generality.
major comments (3)
- [Section V, Eqs. (23)-(24) and (28); Section VIII, Eq. (57)] The symbol f0 is used both for the reference drift in Eq. (2) and for the zero-control drift in the control-affine decomposition Eq. (23). If the reference is generated by a baseline controller u0, as described in Section II and used in Section VIII, then the reference drift is f(x,u0(x)), not the zero-control drift. The drift mismatch is then B(x)(u-u0(x)), so the KL penalty in Eq. (24) should contain (u-u0)^T B^T(σσ^T)^{-1}B(u-u0), which introduces linear and constant terms in u. Formula (28), the closed-loop dynamics (30), and the Riccati equation (50) are the solution only for u0=0. The deviation metric (57) uses u_t-u_ref_t, confirming the mismatch between the theory and the experiments. Please introduce separate notation for the zero-control drift and re-derive the HJB minimizer for a general baseline reference.
- [Section V, Proposition 5, Eq. (28)] The admissible set U is introduced in Section II as a closed convex subset of R^m. The proof of Proposition 5 differentiates the quadratic expression and solves eR(x)u + B(x)^T ∇V(x)=0, then invokes convexity to claim the minimizer. Convexity gives uniqueness of the unconstrained minimizer only when U=R^m. If U is a proper closed convex set, the minimizer is the projection of the unconstrained point onto U, characterized by KKT/normal-cone conditions, and (28) may not be admissible. Since Eq. (30), the optimal feedback (49), and the ARE (50) all inherit this step, the paper should either assume U=R^m explicitly or state the constrained minimizer.
- [Section IV, Proposition 4] In the proof of the reference-preserving limit, the inequality chain drops the term J0_x(uλ) with the comment 'Dropping J0_x(uλ)≥0'. Nonnegativity of the unregularized cost is not assumed in the problem formulation; only continuity and polynomial growth are stated in Section II. If ℓ can take negative values, the inequality λ/2 e^{-ρT} E[...] ≤ J0_x(u0) is not justified. The result needs an explicit assumption that ℓ (and hence J0) is nonnegative or bounded below, or a modified argument that does not require dropping a possibly negative term.
minor comments (4)
- [Throughout; Section VI and VII] The proposition numbering is duplicated: Proposition 6 appears both for the exponential moment bound and for the value-function Lyapunov result, and Proposition 7 appears both for the invariant measure and for the discounted ARE result.
- [Section VIII, Eq. (56)] There is a typographical error in the definition of x_ref(t): the bracket expression reads [pref(t), vref(t))] and should be [pref(t), vref(t)].
- [Section I, author footnote] The text 'The authors are withHybrid Robotics' is missing a space; it should read 'with Hybrid Robotics'.
- [Section III, Theorems 1 and 2] The DP theorem and verification theorem are stated under 'standard assumptions' without proof. This is acceptable as a citation to standard SOC theory, but the manuscript would be clearer if it explicitly listed the regularity conditions (including existence of an admissible minimizer in Theorem 2) rather than referencing them only implicitly.
Circularity Check
No significant circularity: the central KL-to-quadratic-cost identity rests on Girsanov's theorem, and the DP/HJB/Riccati results follow from standard SOC and LQR arguments.
full rationale
The paper's central derivation is self-contained modulo standard external results. Lemma 1 derives the trajectory-KL identity (5) from Girsanov's theorem under Assumption 1, rather than assuming the conclusion; this is an external mathematical input, not a self-citation or definitional equivalence. The reduction of TRSOC to an SOC problem with augmented running cost (7) is a direct substitution, and the DP/HJB characterizations are standard consequences. Proposition 5's feedback law (28) is an exact minimizer of the unconstrained quadratic in (27); the proof correctly invokes convexity after differentiation. The skeptical issue that U is only assumed closed convex, so the unconstrained minimizer may not be feasible, is a genuine correctness gap but is not circularity. The performance–deviation inequality (21) is indeed just the decomposition J_lambda = J_0 + (lambda/2)D plus optimality, but the paper presents it as a quantifying property of the objective, not as an empirical prediction, so it does not constitute circular reasoning. The self-citations ([3], [7], [12]) appear in related-work and motivation contexts and are not load-bearing for Lemma 1, Proposition 5, or the LQ Riccati result. Remark 1 explicitly acknowledges the shared-diffusion limitation, further indicating that the claimed reduction is conditional on stated assumptions rather than disguised. Overall, no circular step is exhibited.
Assumptions & free parameters
free parameters (1)
- λ (regularization weight) =
varied (e.g., λ=0.1 in trajectory visualization)
assumptions (5)
- standard math Girsanov theorem and Novikov condition (Assumption 1) give absolute continuity P^u ≪ P^0 and the density formula used in Lemma 1.
- domain assumption The controlled and reference dynamics share the same diffusion coefficient σ, and control affects only the drift.
- domain assumption Standard well-posedness of the SDEs and DP/HJB theory under polynomial-growth costs.
- domain assumption Assumption 3 (Foster-Lyapunov condition) for the stability and invariant-measure results.
- ad hoc to paper Assumption 4: L(x,u*(x)) ≥ c1 V(x) - c2 for the value function to serve as a Lyapunov function.
Cite this review
Pith. "Pith review of Trajectory-Regularized Stochastic Optimal Control via KL Divergence." pith.science (2026). https://pith.science/paper/ZBCOK4FI
@misc{pith2026260722201,
author = {Pith},
title = {Pith review of: Trajectory-Regularized Stochastic Optimal Control via KL Divergence},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZBCOK4FI}},
note = {Machine review of arXiv:2607.22201}
}
read the original abstract
We introduce trajectory-regularized stochastic optimal control (TRSOC), which augments standard stochastic optimal control (SOC) with a Kullback--Leibler (KL) divergence between controlled and reference trajectory distributions. Using Girsanov's theorem, the trajectory KL reduces to a quadratic drift mismatch penalty, yielding a modified running cost that preserves the dynamic programming (DP) structure. We derive the corresponding Hamilton--Jacobi--Bellman (HJB) equation and characterize the optimal policy. In the linear-quadratic (LQ) setting, the formulation admits a closed-form solution with an augmented control cost. Experiments show that the regularization parameter induces a trade-off between performance-driven and reference-preserving behavior, including cases with reference dynamics learned from offline data.
Figures
Reference graph
Works this paper leans on
-
[14]
A variational representation for certain functionals of brownian motion,
M. Bou ´e and P. Dupuis, “A variational representation for certain functionals of brownian motion,”The Annals of Probability, 1998
1998
-
[15]
Representation formula for the entropy and functional inequalities,
J. Lehec, “Representation formula for the entropy and functional inequalities,” inProbabilit ´es et statistiques, 2013, pp. 885–899
2013
-
[16]
Explicit solution of relative entropy weighted control,
J. Bierkens and H. J. Kappen, “Explicit solution of relative entropy weighted control,”Systems & Control Letters, vol. 72, 2014
2014
-
[1]
W. H. Fleming and H. M. Soner,Controlled Markov processes and viscosity solutions. Springer, 2006
2006
-
[2]
Yong and X
J. Yong and X. Y . Zhou,Stochastic controls: Hamiltonian systems and HJB equations. Springer Science & Business Media, 1999
1999
-
[3]
M. Kim, “Finite memory belief approximation for optimal control in partially observable markov decision processes,”arXiv preprint arXiv:2601.03132, 2026
arXiv 2026
-
[4]
Bertsekas,Dynamic programming and optimal control: Volume I
D. Bertsekas,Dynamic programming and optimal control: Volume I. Athena scientific, 2012, vol. 4
2012
-
[5]
A survey of the schr ¨odinger problem and some of its connections with optimal transport,
C. L ´eonard, “A survey of the schr ¨odinger problem and some of its connections with optimal transport,”arXiv preprint arXiv:1308.0215,
Show all 21 references
-
[6]
Trust region policy optimization,
J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” inInternational conference on machine learning, PMLR, 2015, pp. 1889–1897
2015
-
[7]
Robust adversarial policy optimization un- der dynamics uncertainty,
M. Kim and K. Sreenath, “Robust adversarial policy optimization un- der dynamics uncertainty,”arXiv preprint arXiv:2604.10974, 2026
2026 arXiv
-
[8]
Karatzas and S
I. Karatzas and S. Shreve,Brownian motion and stochastic calculus. springer, 2014
2014
-
[9]
An introduction to stochastic control theory, path integrals and reinforcement learning,
H. J. Kappen, “An introduction to stochastic control theory, path integrals and reinforcement learning,” inAIP conference proceedings, American Institute of Physics, vol. 887, 2007, pp. 149–181
2007
-
[10]
S. A. Thijssen,Path integral control. Sl: sn, 2016
2016
-
[11]
Model predictive path integral control: From theory to parallel computation,
G. Williams, A. Aldrich, and E. A. Theodorou, “Model predictive path integral control: From theory to parallel computation,”Journal of Guidance, Control, and Dynamics, vol. 40, no. 2, 2017
2017
-
[12]
Wombet: World model-based experience transfer for robust and sample-efficient reinforcement learning,
M. Kim and K. Sreenath, “Wombet: World model-based experience transfer for robust and sample-efficient reinforcement learning,” in Annual Learning for Dynamics and Control Conference, PMLR, 2026, pp. 1121–1133
2026
-
[13]
Linear theory for control of nonlinear stochastic systems,
H. J. Kappen, “Linear theory for control of nonlinear stochastic systems,”Physical review letters, vol. 95, no. 20, p. 200 201, 2005
2005
-
[17]
Stochastic differential equations,
B. Øksendal, “Stochastic differential equations,” inStochastic differ- ential equations: an introduction with applications, Springer, 2003
2003
-
[18]
I. V . Girsanov,Lectures on mathematical theory of extremum prob- lems. Springer Science & Business Media, 2012
2012
-
[19]
Linearly-solvable markov decision problems,
E. Todorov, “Linearly-solvable markov decision problems,”Ad- vances in neural information processing systems, vol. 19, 2006
2006
-
[20]
S. P. Meyn and R. L. Tweedie,Markov chains and stochastic stability. Springer Science & Business Media, 2012
2012
-
[21]
La th ´eorie g ´en´erale de la mesure dans son application `a l’ ´etude des syst `emes dynamiques de la m´ecanique non lin´eaire,
N. Kryloff and N. Bogoliouboff, “La th ´eorie g ´en´erale de la mesure dans son application `a l’ ´etude des syst `emes dynamiques de la m´ecanique non lin´eaire,”Annals of mathematics, vol. 38, no. 1, 1937
1937
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.