{"id":"f2417f6e-ccc2-4318-bea2-8eeb7c91c52b","arxiv_id":"2607.08204","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Endpoint null-space projection yields a critic-free off-policy update for continuous-time LQR that recovers the Kleinman gain under a projected actor rank condition.","lead":"A data-driven controller for linear systems can learn the optimal feedback gain without ever estimating the value function. By projecting out endpoint terms, the method keeps Kleinman-style convergence while cutting the regression size from quadratic in the state dimension to linear in the input-state product.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified","rationale":"The manuscript’s strongest claim is precisely the equivalence of the critic-free projected residual to the Kleinman update under the stated assumptions. The proofs are explicit, use only standard Lyapunov uniqueness for Hurwitz closed-loop matrices, and correctly separate the endpoint-critic rank from the projected-actor rank (Theorem 3). Assumption 3 is the sole nontrivial data condition and is already flagged by the authors and the reader as non-automatic; it does not undermine the logical claim that, when it holds, the actor-only step is exact. Limitations (need for a stabilizing seed F, noise-free experiments, no code) are real but outside the central theoretical claim. Consequently the reader’s ACCEPT / low correctness-risk assessment stands; no adjustment is warranted.","tokens_in":13499,"tokens_out":526,"duration_ms":6708,"concrete_test":"Independently re-derive the uniqueness half of Theorem 2: start from Γ_k vec(E)=0, construct Π=DP_F(K_k)[E] from the Lyapunov equation (28), obtain RE=B^T Π from full rank of Z, substitute into (28) to reach the closed-loop Lyapunov equation for Π, conclude Π=0 and E=0. If this chain holds without additional assumptions, the load-bearing uniqueness claim is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that, under Assumptions 1 and 3, the endpoint-nullspace-projected actor equation recovers exactly the Kleinman gain without identifying the critic (Theorems 1–2, Corollary 1). The proofs are self-contained: Lemma 1 equates G_F(K)=0 to the ARE; the projection identity (16) cancels the PF endpoint term for any symmetric matrix; residual factorization (20) plus full-column-rank of Z yields equivalence; the Newton step (25) is shown to be uniquely the Kleinman update by showing ker(Γ_k)={0} via the nonsingularity of the Fréchet derivative of G_F at a stabilizing K_k. The reader’s weakest assumption (Assumption 3) is correctly identified and is already stated as a verifiable finite-data condition rather than an automatic PE consequence (Remark 2). No internal inconsistency, hidden model dependence, or gap that would break the equivalence appears in the argument.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper develops a data-driven critic-free off-policy policy iteration method for continuous-time LQR with unknown A and B. Starting from a known stabilizing gain F, it anchors the Riccati equation in policy space via the residual G_F(K) ≜ RK − B^T P_F(K), where P_F(K) solves a fixed stable Lyapunov equation. An endpoint null-space projection onto ker(D) then eliminates the value-matrix endpoint terms from the integral identity, producing an actor-only projected equation Φ_F(K)=0. Under the verifiable rank condition rank(Z)=mn, this equation is equivalent to G_F(K)=0 and therefore to the ARE; a Newton step on the projected residual yields a least-squares update whose unique solution is exactly the Kleinman gain. The conventional joint full-rank condition is decomposed into an endpoint critic rank condition and a projected actor rank condition, so the method removes the critic rank requirement and reduces the repeated regression dimension from n(n+1)/2+mn to mn while retaining Kleinman stability and convergence. Comparative simulations on a diesel-engine example and a scalability study support the claims.","tokens_in":13774,"tokens_out":786,"duration_ms":7354,"significance":"If the equivalence holds, the result cleanly separates the data-informativity burden of value-matrix identification from that of policy improvement in continuous-time off-policy PI. The reduction of the repeated least-squares dimension to mn, together with the ability to remain well-posed when the conventional joint regression is rank-deficient (Corollary 2), is practically relevant for high-dimensional systems. The proofs are self-contained and target classical Kleinman iteration as an external benchmark rather than assuming the answer; the rank decomposition (Theorem 3) and the explicit residual factorization (20) are particularly clear. The contribution is incremental relative to model-based policy-space characterizations, but the data-driven elimination of the critic via endpoint projection is new and useful for the ADP/RL community.","major_comments":[],"minor_comments":[{"comment":"Remark 2 correctly notes that rank(Z)=mn is not implied by PE of v alone; a short numerical illustration of a PE signal that fails the projected rank (or a brief discussion of how to choose sampling intervals to enlarge q) would help practitioners.","section":null},{"comment":"Section IV reports only noise-free simulations. A single noisy-data trial (or a short remark on the effect of measurement noise on the projected least-squares solve) would strengthen the experimental section without changing the theoretical claims.","section":null},{"comment":"Notation: the distinction between vecs and vech is standard but could be recalled once more when δxx and D are introduced, to avoid momentary confusion for readers less familiar with half-vectorization conventions.","section":null},{"comment":"In the complexity paragraph, the ratio N/q · (1+(n+1)/(2m))^2 is informative; stating the typical range of N/q observed in the examples would make the saving more concrete.","section":null},{"comment":"A few minor typos appear (e.g., spacing around “policy-space”, “endpoint null-space”). A light copy-edit pass would polish the text.","section":null}],"recommendation":"accept","confidential_remarks":"The manuscript is technically solid and the central equivalence is carefully proved. I see no load-bearing gaps. Fit for a control/systems journal is good; the contribution is more incremental than transformative, but the clarity of the rank decomposition and the practical dimension reduction justify acceptance. No concerns about citation pattern or novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The real move here is simple and useful: they anchor the Riccati residual at a known stabilizing F, then project the integral identities onto the null space of the endpoint increments so the unknown P term vanishes exactly. Under a checkable projected rank condition on Z, the actor-only least-squares step is identical to Kleinman, not an approximation. That is new relative to Jiang–Jiang off-policy PI and to the model-based policy-space Riccati work they cite (Sassano, Possieri–Sassano). The rank decomposition (Theorem 3) is clean: conventional full rank splits into endpoint critic rank plus projected actor rank, and they only keep the latter. Dimension drops from n(n+1)/2 + mn to mn, and the proofs (Lemmas 1–2, Theorems 1–2, Corollary 1) are explicit and checkable.\n\nWhat they do well: the math is self-contained, the projection identity (16) really does cancel for any symmetric matrix, and they are honest that Assumption 3 is not automatic PE. Simulations match the classical method on the diesel-engine example and show the expected runtime scaling. No circularity; they target Kleinman as an external benchmark and prove equivalence.\n\nSoft spots are real but proportionate. You still need a stabilizing seed F (standard for PI). Assumption 3 depends on the state response and the projection, so data collection is not free. Everything is noise-free; no code. Those limit impact to continuous-time LQR with a known seed, but they do not break the central claim.\n\nThis is for people who already work on data-driven LQR / ADP and care about regression size and rank burden. It deserves a serious referee. I would cite the rank split and the projection trick if I am writing in this lane. Send it out.","headline":"Clean, exact critic-free off-policy PI for CT-LQR that recovers Kleinman via endpoint null-space projection; solid theory, honest scope.","tokens_in":14359,"tokens_out":459,"would_cite":true,"duration_ms":4812,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Critic-free off-policy policy iteration recovers Kleinman updates for continuous-time LQR from input-state data alone.","keywords":["policy iteration","linear quadratic regulation","data-driven control","critic-free reinforcement learning","endpoint null-space projection","Kleinman iteration","off-policy learning"],"falsifier":"Collect finite input-state batches on a stabilizable LQR plant so that the projected matrix Z has full column rank mn, run the actor-only update, and check whether the resulting gains match the sequence obtained by classical Kleinman iteration on the true A and B matrices to machine precision; any systematic deviation falsifies the claimed equivalence.","tokens_in":14398,"feed_emoji":"⚙️","tokens_out":625,"duration_ms":5975,"temperature":0.7,"pith_summary":"Standard data-driven policy iteration for continuous-time linear quadratic regulation with unknown dynamics jointly estimates a value matrix (the critic) and the next feedback gain (the actor) in one large least-squares problem. This paper shows the critic is unnecessary for the improvement step. By anchoring the Riccati equation at a known stabilizing gain and projecting the integral data identities onto the null space of the endpoint state increments, the unknown value matrix drops out exactly. What remains is an actor-only regression whose unique solution, under a checkable projected rank condition, is precisely the classical Kleinman gain. The method therefore inherits Kleinman’s stabilizing and convergence guarantees while cutting the repeated least-squares dimension from n(n+1)/2 + mn down to mn and removing the rank burden that would have been needed only for critic identification.","feed_headline":"LQR policy iteration without the critic","feed_subtitle":"Endpoint projection drops the value matrix; actor-only least squares still recovers exact Kleinman gains","key_machinery":"Endpoint null-space projection of the anchored integral identity: an orthonormal basis of the kernel of the stacked endpoint increments eliminates every term that involves the unknown value matrix, leaving a residual factorization Φ_F(K) = 2Z vec(G_F(K)) that is solvable for the feedback gain alone.","core_discovery":"Under a known stabilizing anchor and a verifiable projected full-column-rank condition on the data matrix Z, the critic-free projected residual equation is equivalent to the anchored policy-space residual, and each actor-only least-squares update coincides exactly with one Kleinman policy-improvement step. Consequently the algorithm generates the Kleinman sequence and retains its stabilizing and monotonic convergence properties without ever regressing on a value matrix.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Actor-only LS recovers exact Kleinman gains for continuous LQR","Endpoint projection drops value matrix in data-driven policy iteration","Critic-free update coincides with Kleinman from input-state data","Projected residual equation yields critic-free LQR policy iteration","Anchor-and-project removes critic rank need while keeping Kleinman sequence"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The projected data matrix built from the exploration trajectory and the endpoint null-space basis must have full column rank equal to the number of free gain entries; ordinary persistent excitation of the probe signal does not guarantee this by itself.","fun_headline_variants_meta":{"raw":{"variants":["Actor-only LS recovers exact Kleinman gains for continuous LQR","Endpoint projection drops value matrix in data-driven policy iteration","Critic-free update coincides with Kleinman from input-state data","Projected residual equation yields critic-free LQR policy iteration","Anchor-and-project removes critic rank need while keeping Kleinman sequence"]},"model":"grok-4.5","effort":"low","cost_usd":0.004298,"raw_usage":{"total_tokens":1301,"prompt_tokens":781,"num_sources_used":0,"completion_tokens":91,"cost_in_usd_ticks":42980000,"prompt_tokens_details":{"text_tokens":781,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":429,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":781,"tokens_out":91,"duration_ms":6142,"temperature":1.0,"reasoning_tokens":429,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-10T11:22:05.050991+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Collect finite input-state batches on a stabilizable LQR plant so that the projected matrix Z has full column rank mn, run the actor-only update, and check whether the resulting gains match the sequence obtained by classical Kleinman iteration on the true A and B matrices to machine precision; any systematic deviation falsifies the claimed equivalence.","supporting_citations":[],"review_version":1}