Data-Driven Critic-Free Policy Iteration for Continuous-Time Linear Quadratic Regulation

Hongye Su; Jiacheng Wu; Yang Zhu

Data-Driven Critic-Free Policy Iteration for Continuous-Time Linear Quadratic Regulation

T0 review · 0 major / 5 minor · reviewed 2026-07-10 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Critic-free off-policy policy iteration recovers Kleinman updates for continuous-time LQR from input-state data alone.

desk verdict Clean, exact critic-free off-policy PI for CT-LQR that recovers Kleinman via endpoint null-space projection; solid theory, honest scope. read the letter →

arxiv 2607.08204 v1 pith:T75O3J3Q submitted 2026-07-09 eess.SY cs.SY

Jiacheng Wu , Yang Zhu , Hongye Su This is my paper · ORCID

classification eess.SYcs.SY

keywords policyiterationlinearquadraticregulationdata-drivencontrolcritic-freereinforcementlearningendpointnull-spaceprojectionKleinmanoff-policy

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard data-driven policy iteration for continuous-time linear quadratic regulation with unknown dynamics jointly estimates a value matrix (the critic) and the next feedback gain (the actor) in one large least-squares problem. This paper shows the critic is unnecessary for the improvement step. By anchoring the Riccati equation at a known stabilizing gain and projecting the integral data identities onto the null space of the endpoint state increments, the unknown value matrix drops out exactly. What remains is an actor-only regression whose unique solution, under a checkable projected rank condition, is precisely the classical Kleinman gain. The method therefore inherits Kleinman’s stabilizing and convergence guarantees while cutting the repeated least-squares dimension from n(n+1)/2 + mn down to mn and removing the rank burden that would have been needed only for critic identification.

What carries the argument

Endpoint null-space projection of the anchored integral identity: an orthonormal basis of the kernel of the stacked endpoint increments eliminates every term that involves the unknown value matrix, leaving a residual factorization Φ_F(K) = 2Z vec(G_F(K)) that is solvable for the feedback gain alone.

What would settle it

Collect finite input-state batches on a stabilizable LQR plant so that the projected matrix Z has full column rank mn, run the actor-only update, and check whether the resulting gains match the sequence obtained by classical Kleinman iteration on the true A and B matrices to machine precision; any systematic deviation falsifies the claimed equivalence.

Watch

Extended reading notes

Core claim

Under a known stabilizing anchor and a verifiable projected full-column-rank condition on the data matrix Z, the critic-free projected residual equation is equivalent to the anchored policy-space residual, and each actor-only least-squares update coincides exactly with one Kleinman policy-improvement step. Consequently the algorithm generates the Kleinman sequence and retains its stabilizing and monotonic convergence properties without ever regressing on a value matrix.

Load-bearing premise

The projected data matrix built from the exploration trajectory and the endpoint null-space basis must have full column rank equal to the number of free gain entries; ordinary persistent excitation of the probe signal does not guarantee this by itself.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The real move here is simple and useful: they anchor the Riccati residual at a known stabilizing F, then project the integral identities onto the null space of the endpoint increments so the unknown P term vanishes exactly. Under a checkable projected rank condition on Z, the actor-only least-squares step is identical to Kleinman, not an approximation. That is new relative to Jiang–Jiang off-policy PI and to the model-based policy-space Riccati work they cite (Sassano, Possieri–Sassano). The rank decomposition (Theorem 3) is clean: conventional full rank splits into endpoint critic rank plus projected actor rank, and they only keep the latter. Dimension drops from n(n+1)/2 + mn to mn, and the proofs (Lemmas 1–2, Theorems 1–2, Corollary 1) are explicit and checkable.

What they do well: the math is self-contained, the projection identity (16) really does cancel for any symmetric matrix, and they are honest that Assumption 3 is not automatic PE. Simulations match the classical method on the diesel-engine example and show the expected runtime scaling. No circularity; they target Kleinman as an external benchmark and prove equivalence.

Soft spots are real but proportionate. You still need a stabilizing seed F (standard for PI). Assumption 3 depends on the state response and the projection, so data collection is not free. Everything is noise-free; no code. Those limit impact to continuous-time LQR with a known seed, but they do not break the central claim.

This is for people who already work on data-driven LQR / ADP and care about regression size and rank burden. It deserves a serious referee. I would cite the rank split and the projection trick if I am writing in this lane. Send it out.

Referee Report

0 major / 5 minor

Summary. The paper develops a data-driven critic-free off-policy policy iteration method for continuous-time LQR with unknown A and B. Starting from a known stabilizing gain F, it anchors the Riccati equation in policy space via the residual G_F(K) ≜ RK − B^T P_F(K), where P_F(K) solves a fixed stable Lyapunov equation. An endpoint null-space projection onto ker(D) then eliminates the value-matrix endpoint terms from the integral identity, producing an actor-only projected equation Φ_F(K)=0. Under the verifiable rank condition rank(Z)=mn, this equation is equivalent to G_F(K)=0 and therefore to the ARE; a Newton step on the projected residual yields a least-squares update whose unique solution is exactly the Kleinman gain. The conventional joint full-rank condition is decomposed into an endpoint critic rank condition and a projected actor rank condition, so the method removes the critic rank requirement and reduces the repeated regression dimension from n(n+1)/2+mn to mn while retaining Kleinman stability and convergence. Comparative simulations on a diesel-engine example and a scalability study support the claims.

Significance. If the equivalence holds, the result cleanly separates the data-informativity burden of value-matrix identification from that of policy improvement in continuous-time off-policy PI. The reduction of the repeated least-squares dimension to mn, together with the ability to remain well-posed when the conventional joint regression is rank-deficient (Corollary 2), is practically relevant for high-dimensional systems. The proofs are self-contained and target classical Kleinman iteration as an external benchmark rather than assuming the answer; the rank decomposition (Theorem 3) and the explicit residual factorization (20) are particularly clear. The contribution is incremental relative to model-based policy-space characterizations, but the data-driven elimination of the critic via endpoint projection is new and useful for the ADP/RL community.

minor comments (5)

Remark 2 correctly notes that rank(Z)=mn is not implied by PE of v alone; a short numerical illustration of a PE signal that fails the projected rank (or a brief discussion of how to choose sampling intervals to enlarge q) would help practitioners.
Section IV reports only noise-free simulations. A single noisy-data trial (or a short remark on the effect of measurement noise on the projected least-squares solve) would strengthen the experimental section without changing the theoretical claims.
Notation: the distinction between vecs and vech is standard but could be recalled once more when δxx and D are introduced, to avoid momentary confusion for readers less familiar with half-vectorization conventions.
In the complexity paragraph, the ratio N/q · (1+(n+1)/(2m))^2 is informative; stating the typical range of N/q observed in the examples would make the saving more concrete.
A few minor typos appear (e.g., spacing around “policy-space”, “endpoint null-space”). A light copy-edit pass would polish the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: actor-only update is proved equivalent to classical Kleinman under an independent projected rank condition, not defined into the answer.

full rationale

The paper’s load-bearing chain is a self-contained equivalence proof, not a fit or a definitional tautology. Lemma 1 reparameterizes the ARE as the anchored residual G_F(K)=0 via a fixed Hurwitz Lyapunov map at a known stabilizing F; that is a change of unknowns, not circularity. The endpoint null-space projection (16)–(20) cancels the critic endpoint term for any symmetric matrix by construction of ker(D), then residual factorization Φ_F(K)=2Z vec(G_F(K)) plus Assumption 3 (rank(Z)=mn) yields Theorem 1’s equivalence. Theorem 2 shows the Newton step (25) has unique solution equal to the Kleinman gain by verifying that R^{-1}B^T P_k satisfies the projected identity and that ker(Γ_k)={0} via nonsingularity of the Fréchet derivative of G_F at a stabilizing K_k. Corollary 1 then imports the classical Kleinman stabilizing/monotonicity properties from Kleinman (1968) [19], an external classical result, not a self-citation uniqueness theorem. Conventional off-policy PI [9] is used only as a baseline to decompose rank(Θ_k)=rank(D)+rank(Z). No parameter is fitted and then “predicted”; no ansatz is smuggled via overlapping-author citation; the reduced regression dimension mn is a consequence of removing the critic block, not a renaming of an empirical pattern. Assumption 3 is stated as a verifiable finite-data condition (Remark 2), not assumed away. Score 0 is therefore appropriate.

Assumptions & free parameters 0 free parameters · 5 assumptions · 2 invented entities

The result rests on standard LQR/Lyapunov theory, the classical Kleinman iteration, and two domain assumptions standard in policy iteration (stabilizability/detectability and a known stabilizing seed). The only paper-specific constructions are the anchored residual G_F and the endpoint null-space projection; both are defined from known F and data, not postulated as new physical entities. No free parameters are fitted to produce the optimal gain.

assumptions (5)

domain assumption Pair (A,B) stabilizable, (A,Q^{1/2}) detectable; unique stabilizing ARE solution exists.
Assumption 1 and standard LQR theory; used to guarantee K* and P* exist.
domain assumption A known gain F exists such that A−BF is Hurwitz.
Assumption 1; anchors the Lyapunov operator L_F and the behavior policy u=−Fx+v.
standard math For Hurwitz A_F, the Lyapunov map P ↦ A_F^T P + P A_F is invertible on symmetric matrices.
Used to define P_F(K) uniquely in Definition 1.
standard math Kleinman iteration from a stabilizing seed produces stabilizing gains, monotone P_k ↓ P*, and K_k → K*.
Cited [19]; Corollary 1 inherits convergence from this classical result once each step matches Kleinman.
ad hoc to paper Projected data matrix Z has full column rank mn (Assumption 3).
Paper-specific finite-data informativity condition; necessary and sufficient for uniqueness of the actor-only update.

invented entities (2)

Anchored policy-space residual G_F(K) ≜ RK − B^T P_F(K) independent evidence
purpose: Rewrite ARE optimality as an equation only in the gain K, relative to fixed anchor F.
Defined in Definition 1; equivalent to ARE by Lemma 1. Algebraic reparameterization, not a new physical object.
Endpoint null-space projection via orthonormal basis W of ker(D) independent evidence
purpose: Exactly cancel critic-dependent endpoint terms from integral identities, yielding actor-only data equations.
Constructed from measured endpoint products D_j; Remark 1 stresses exact cancellation for any symmetric P.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Data-Driven Critic-Free Policy Iteration for Continuous-Time Linear Quadratic Regulation." pith.science (2026). https://pith.science/paper/T75O3J3Q

@misc{pith2026260708204,
  author       = {Pith},
  title        = {Pith review of: Data-Driven Critic-Free Policy Iteration for Continuous-Time Linear Quadratic Regulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T75O3J3Q}},
  note         = {Machine review of arXiv:2607.08204}
}

abstract

For continuous-time linear quadratic regulation with unknown system matrices, data-driven off-policy policy iteration typically estimates the value matrix and the improved feedback gain through a joint critic--actor regression. We show that the critic is not needed in the policy-improvement step. The key is to anchor the Riccati equation at a known stabilizing gain and express optimality as a policy-space residual. An endpoint null-space projection then removes the value-matrix term from the integral data equation. This yields a critic-free, actor-only least-squares update computed directly from input-state data. Under a verifiable projected rank condition, the resulting data equation is equivalent to the policy-space residual equation, and each update coincides with the Kleinman iteration. Thus, the stabilizing and convergence properties of Kleinman iteration are retained without a critic regression. We further show that the conventional off-policy full-rank condition decomposes into an endpoint critic rank condition and a projected actor rank condition. The proposed method removes the rank requirement needed for critic identification while retaining the one needed for policy improvement. The repeated least-squares dimension is reduced from $n(n+1)/2+mn$ to $mn$. Finally, comparative simulations validate the effectiveness of the proposed algorithm.

Figures

Figures reproduced from arXiv: 2607.08204 by the authors.

**Figure 2.** Per-iteration runtime comparison between Algorith [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 21 canonical work pages

[1]

F. L. Lewis, D. Vrabie, and V . L. Syrmos, Optimal control. John Wiley & Sons, 2012

work page 2012
[2]

Data-enabled policy optimization for direct adaptive learning of the LQR,

F. Zhao, F. D¨ orﬂer, A. Chiuso, and K. Y ou, “Data-enabled policy optimization for direct adaptive learning of the LQR,” IEEE Trans. Autom. Control, vol. 70, no. 11, pp. 7217–7232, Nov. 2025

work page 2025
[3]

Learning-based adaptive op- timal control of linear time-delay systems: A value iterati on approach,

L. Cui, B. Pang, M. Krsti´ c, and Z.-P . Jiang, “Learning-based adaptive op- timal control of linear time-delay systems: A value iterati on approach,” Automatica, vol. 171, p. 111944, 2025

work page 2025
[4]

Adaptive dynamic pr ogramming- regulated extremum seeking for distributed feedback optim ization,

T. Liu, M. Krsti´ c, and Z.-P . Jiang, “Adaptive dynamic pr ogramming- regulated extremum seeking for distributed feedback optim ization,” IEEE Trans. Autom. Control, vol. 70, no. 11, pp. 7675–7682, Nov. 2025

work page 2025
[5]

Online ada ptive algorithm for optimal control with integral reinforcement learning,

K. G. V amvoudakis, D. Vrabie, and F. L. Lewis, “Online ada ptive algorithm for optimal control with integral reinforcement learning,” Int. J. Robust Nonlinear Control , vol. 24, no. 17, pp. 2686–2710, 2014

work page 2014
[6]

Data-driven st ructured policy iteration for homogeneous distributed systems,

S. Alemzadeh, S. Talebi, and M. Mesbahi, “Data-driven st ructured policy iteration for homogeneous distributed systems,” IEEE Trans. Autom. Control, vol. 69, no. 9, pp. 5979–5994, Sep. 2024

work page 2024
[7]

Reinforcement learning-based adaptive optimal exponent ial tracking control of linear systems with unknown dynamics,

C. Chen, H. Modares, K. Xie, F. L. Lewis, Y . Wan, and S. Xie, “Reinforcement learning-based adaptive optimal exponent ial tracking control of linear systems with unknown dynamics,” IEEE Trans. Autom. Control, vol. 64, no. 11, pp. 4423–4438, Nov. 2019

work page 2019
[8]

Adaptive optimal control for continuous-time linear systems based o n policy iteration,

D. Vrabie, O. Pastravanu, M. Abu-Khalaf, and F. L. Lewis, “Adaptive optimal control for continuous-time linear systems based o n policy iteration,” Automatica, vol. 45, no. 2, pp. 477–484, 2009

work page 2009

Show all 21 references

[9]

Computational adaptive optim al control for continuous-time linear systems with completely unknown dy namics,

Y . Jiang and Z.-P . Jiang, “Computational adaptive optim al control for continuous-time linear systems with completely unknown dy namics,” Automatica, vol. 48, no. 10, pp. 2699–2704, 2012

2012
[10]

B. Lian, W. Xue, F. L. Lewis, H. Modares, and B. Kiumarsi, Integral and Inverse Reinforcement Learning for Optimal Control Sys tems and Games, Switzerland, Cham: Springer, 2024

2024
[11]

Reinforcement le arning and singular perturbation-based optimal speed synchronou s control of a ﬂexible coupling Dual-PMSM system,

J. Zhao, T. Cai, M. Xiong, and C. Y ang, “Reinforcement le arning and singular perturbation-based optimal speed synchronou s control of a ﬂexible coupling Dual-PMSM system,” IEEE Trans. Ind. Informat. , vol. 21, no. 12, pp. 9757–9766, Dec. 2025

2025
[12]

Data-driven near opt imization for fast sampling singularly perturbed systems,

H. Shen, C. Peng, H. Y an, and S. Xu, “Data-driven near opt imization for fast sampling singularly perturbed systems,” IEEE Trans. Autom. Control, vol. 69, no. 7, pp. 4689–4694, Jul. 2024

2024
[13]

Off-policy reinforcement learning for H∞ control of linear discrete-time systems with network induced dropouts,

Y . Jiang, T. Y ang, W. Gao, J. Wu, T. Chai, and F. L. Lewis, “ Off-policy reinforcement learning for H∞ control of linear discrete-time systems with network induced dropouts,” IEEE Trans. Autom. Control , vol. 70, no. 12, pp. 8000–8015, Dec. 2025

2025
[14]

Distributed optimal con trol using interval-excited relative-output reinforcement learnin g algorithm,

Y . Xu, W.-W. Che, and Z.-G. Wu, “Distributed optimal con trol using interval-excited relative-output reinforcement learnin g algorithm,” IEEE Trans. Autom. Control , in press, doi:10.1109/TAC.2026.3697997

2026 doi
[15]

Multia gent graphical games with inverse reinforcement learning,

V . S. Donge, B. Lian, F. L. Lewis, and A. Davoudi, “Multia gent graphical games with inverse reinforcement learning,” IEEE Trans. Control Netw. Syst., vol. 10, no. 2, pp. 841–852, Jun. 2023

2023
[16]

Policy gradient-bas ed algo- rithms for continuous-time linear quadratic control,

J. Bu, A. Mesbahi, and M. Mesbahi, “Policy gradient-bas ed algo- rithms for continuous-time linear quadratic control,” arXiv preprint arXiv:2006.09178, 2020. 8

2006 arXiv
[17]

Policy algebraic equation for the LQR and t he H∞ control problems,

M. Sassano, “Policy algebraic equation for the LQR and t he H∞ control problems,” IEEE Control Syst. Lett. , vol. 8, pp. 370–375, 2024

2024
[18]

Solving the linear quadrat ic regulator problem in the policy space: The policy algebraic Riccati eq uation,

C. Possieri and M. Sassano, “Solving the linear quadrat ic regulator problem in the policy space: The policy algebraic Riccati eq uation,” Automatica, vol. 185, p. 112738, 2026

2026
[19]

On an iterative technique for Riccati equ ation computa- tions,

D. Kleinman, “On an iterative technique for Riccati equ ation computa- tions,” IEEE Trans. Autom. Control , vol. 13, no. 1, pp. 114–115, Feb. 1968

1968
[20]

Fast LSTD using st ochastic approximation: Finite time analysis and application to tra fﬁc control,

L. Prashanth, N. Korda, and R. Munos, “Fast LSTD using st ochastic approximation: Finite time analysis and application to tra fﬁc control,” in Joint European conference on machine learning and knowledg e discovery in databases . Springer, 2014, pp. 66–81

2014
[21]

Policy iteration ba sed Q-learning for linear nonzero-sum quadratic differential games,

X. Li, Z. Peng, L. Liang, and W. Zha, “Policy iteration ba sed Q-learning for linear nonzero-sum quadratic differential games,” Sci. China Inf. Sci. , vol. 62, no. 5, p. 52204, 2019

2019

Pith tools

Reviewed July 10, 2026 · model on record in the stance chip above.

[1] [1]

F. L. Lewis, D. Vrabie, and V . L. Syrmos, Optimal control. John Wiley & Sons, 2012

work page 2012

[2] [2]

Data-enabled policy optimization for direct adaptive learning of the LQR,

F. Zhao, F. D¨ orﬂer, A. Chiuso, and K. Y ou, “Data-enabled policy optimization for direct adaptive learning of the LQR,” IEEE Trans. Autom. Control, vol. 70, no. 11, pp. 7217–7232, Nov. 2025

work page 2025

[3] [3]

Learning-based adaptive op- timal control of linear time-delay systems: A value iterati on approach,

L. Cui, B. Pang, M. Krsti´ c, and Z.-P . Jiang, “Learning-based adaptive op- timal control of linear time-delay systems: A value iterati on approach,” Automatica, vol. 171, p. 111944, 2025

work page 2025

[4] [4]

Adaptive dynamic pr ogramming- regulated extremum seeking for distributed feedback optim ization,

T. Liu, M. Krsti´ c, and Z.-P . Jiang, “Adaptive dynamic pr ogramming- regulated extremum seeking for distributed feedback optim ization,” IEEE Trans. Autom. Control, vol. 70, no. 11, pp. 7675–7682, Nov. 2025

work page 2025

[5] [5]

Online ada ptive algorithm for optimal control with integral reinforcement learning,

K. G. V amvoudakis, D. Vrabie, and F. L. Lewis, “Online ada ptive algorithm for optimal control with integral reinforcement learning,” Int. J. Robust Nonlinear Control , vol. 24, no. 17, pp. 2686–2710, 2014

work page 2014

[6] [6]

Data-driven st ructured policy iteration for homogeneous distributed systems,

S. Alemzadeh, S. Talebi, and M. Mesbahi, “Data-driven st ructured policy iteration for homogeneous distributed systems,” IEEE Trans. Autom. Control, vol. 69, no. 9, pp. 5979–5994, Sep. 2024

work page 2024

[7] [7]

Reinforcement learning-based adaptive optimal exponent ial tracking control of linear systems with unknown dynamics,

C. Chen, H. Modares, K. Xie, F. L. Lewis, Y . Wan, and S. Xie, “Reinforcement learning-based adaptive optimal exponent ial tracking control of linear systems with unknown dynamics,” IEEE Trans. Autom. Control, vol. 64, no. 11, pp. 4423–4438, Nov. 2019

work page 2019

[8] [8]

Adaptive optimal control for continuous-time linear systems based o n policy iteration,

D. Vrabie, O. Pastravanu, M. Abu-Khalaf, and F. L. Lewis, “Adaptive optimal control for continuous-time linear systems based o n policy iteration,” Automatica, vol. 45, no. 2, pp. 477–484, 2009

work page 2009