{"id":"3eafde47-c757-4f8c-965a-85f0ce9c5ecb","arxiv_id":"2412.01085","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A data-driven policy iteration framework using Yosida-based Koopman generator identification with control-affine observables yields stabilizing optimal controllers for unknown systems up to 9 dimensions, with small cost error versus a known-dynamics oracle.","lead":"This paper combines a Koopman-operator method for learning unknown control-affine dynamics from data with a policy iteration scheme that solves the Hamilton-Jacobi-Bellman equation, and tests it on systems up to 9 state dimensions. It reports that the learned controllers stabilize the true systems and that accumulated costs match those of a model-based oracle to within 1e-5 to 1e-3, but it ships no code and omits key tuning parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3 certifies exact GHJB solutions, but the algorithm solves a random-feature least-squares problem; with no residual or representability bound, the proof does not cover the computed controller.","rationale":"The reader's weakest assumption precisely identifies the gap between the exact GHJB solution used in Theorem 3 and the random-feature least-squares solution used by the algorithm. My reading confirms this is the most load-bearing concern: the paper's headline theoretical claim is that the learned value function and control laws converge to those of the true system, but the proof chain (Assumption 1 plus Theorem 3) never reaches the actually computed β. The empirical results are credible and the method may well work, but the central proof does not certify the central claim. Assumption 1 is a second, related gap—sampling density is asserted rather than verified—but it is less fundamental because Table 3 at least provides indirect evidence of identification accuracy. The GHJB approximation gap has no corresponding evidence, since no residual or projection error is reported. The appropriate verdict remains conditional, matching the reader's assessment; I do not see grounds to reject the empirical contribution, but the theoretical guarantee should be tightened or explicitly downgraded before the convergence claim is accepted.","tokens_in":13790,"tokens_out":3490,"duration_ms":37410,"concrete_test":"For the inverted-pendulum experiment (s=200), after identifying f̂ and ĝ, compute the least-squares β* from Section 4.4 and evaluate the GHJB residual r(x) = Q(x) + κ_β*(x)^T R κ_β*(x) + DV_β*(x)·(f̂(x) + ĝ(x)κ_β*(x)) on a dense grid over x ∈ [-1,1]^2. Recompute with s=400, 800, and 1600. If max|r| does not decay toward the θ scale required by Theorem 3, or if the difference between V_β* and a high-order Galerkin solution of the same GHJB equation is not small relative to the claimed 10^-3 cost error, then the computed controller is not certified by Theorem 3.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4 replaces the GHJB policy evaluation (16) with a random-feature ansatz V(x) = β^T σ(Wx+b) and solves a linear least-squares problem for β. Theorem 3 in Section 5, and its Appendix A proof, apply only to the exact solutions V_h^(i) of the GHJB equations (19)–(20). The paper provides no residual bound, no representability condition, and no consistency estimate relating the least-squares solution β to V_h^(i). Consequently, even under Assumption 1 and even with perfect identification, the computed value function and policy could be arbitrarily far from the exact GHJB solution. The claimed guarantee that the learned value function and control laws converge to those of the true system therefore does not formally apply to the quantities actually computed in the experiments. This is a correctness gap in the central theorem, not merely a missing numerical detail.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a data-driven optimal control framework for unknown control-affine nonlinear systems. The method first identifies the dynamics by estimating the Koopman generator via a Yosida approximation with polynomial observables that include control-affine coupling terms, then solves the optimal control problem by policy iteration in which the Generalized Hamilton-Jacobi-Bellman (GHJB) equation is approximately solved with random-feature neural networks. The authors claim that the identified vector field converges to the true vector field and that the value function and control policy converge at each iteration to those of the true system. Numerical experiments on inverted pendulum, cartpole, 2D quadrotor, and 3D quadrotor systems report low identification errors and accumulated-cost errors between 10^-5 and 10^-3 relative to a model-based oracle. The central theoretical result, Theorem 3, is meant to certify the convergence of the value function and policy, but the proof only covers exact solutions of the GHJB equation, not the random-feature least-squares solution actually computed by the algorithm.","tokens_in":13961,"tokens_out":5904,"duration_ms":56241,"significance":"If the theoretical gaps were closed, the paper would be a valuable contribution. The empirical results are strong: the resolvent-based control-affine identification improves identification errors by one to two orders of magnitude over the logarithm-based and lifted-linear baselines, and the learned controllers stabilize true 9-dimensional systems with cost errors on the order of 1e-5 to 1e-3. The paper also explicitly avoids the operator-logarithm step, which is a technically useful idea. However, the claimed convergence guarantee does not currently cover the actual algorithm, and the proof of Theorem 3 contains sign and regularity issues. These are load-bearing problems for the paper's main theoretical claim, so the manuscript needs substantial revision before it can be accepted.","major_comments":[{"comment":"Theorem 3 in Section 5, proved in Appendix A, applies only to the exact solutions V_h^{(i)} of the GHJB equations (19)-(20). The algorithm in Section 4.4, however, does not solve these equations; it fits a random-feature ansatz V(x) = beta^T sigma(Wx+b) by linear least squares. The paper provides no residual bound, no representability condition, and no consistency estimate relating the least-squares solution to the exact GHJB solution. Therefore the abstract's claim that 'the learned value function and control laws are proven to converge' is not justified for the quantities actually computed in the experiments. This is the main gap in the theoretical certificate.","section":"Section 4.4 and Theorem 3"},{"comment":"In the proof of Theorem 3, L_h^{(i)} is defined as -Q(x) - kappa_{h}^{(i-1)}(x)^T R kappa_{h}^{(i-1)}(x), which is negative definite. The tail integrals in (23)-(24) are therefore non-positive, so the inequalities 0 <= integral_{T1}^infinity L_h^{(i)} dt <= theta/2 cannot hold as written. The proof of the value-function bound |V_h^{(i)} - V^{(i)}| <= theta is thus not established. The sign convention in the GHJB equations and the subsequent integral representation must be reconciled, and absolute values should be used for the tail estimates.","section":"Appendix A, Eqs. (23)-(24)"},{"comment":"The proof of the control-law convergence uses the bound ||D L^{(i)} - D L_h^{(i)}||_infinity <= C_L ||L^{(i)} - L_h^{(i)}||_infinity in Eq. (50) and assumes in Eq. (52) that D L_h^{(i)} is Lipschitz. Neither statement follows from the theorem's assumption that L^{(i)}, L_h^{(i)} in C^1(Omega). These estimates require at least C^2 regularity or Lipschitz gradients. The assumptions of Theorem 3 must be strengthened or the proof revised accordingly.","section":"Appendix A, Eqs. (50) and (52)"},{"comment":"Assumption 1 simply postulates that sufficiently dense sampling makes the identified vector field uniformly close to the truncated Yosida vector field. No sample-complexity bound, noise model, or relation between the number of samples M, the truncation parameters lambda and Tmax, and the finite-dimensional subspace N is given. Table 3 reports average identification errors for the specific test systems, but this does not verify the uniform closeness over the whole compact set M required for the subsequent conclusion (18). This assumption is load-bearing for the identification part of the theory and should either be proved for the proposed least-squares estimator or explicitly tested and reported.","section":"Section 5, Assumption 1"},{"comment":"The paper does not explain how an initial admissible (stabilizing) policy kappa_h^{(0)} is obtained from data, even though Theorem 3 assumes ||kappa_h^{(0)} - kappa^{(0)}||_infinity < delta and policy iteration requires a stabilizing initial controller. Without a data-driven procedure for computing such an initial policy, the practical applicability of the method is not fully demonstrated, and this requirement should be discussed explicitly.","section":"Section 4.4 and Section 6"}],"minor_comments":[{"comment":"Several parameter values needed for reproducibility are missing: the Yosida parameter lambda, the truncation horizon Tmax, the numerical quadrature scheme for (13), and the activation function and distribution for the random features W and b in Section 4.4 are not reported in Tables 1-2 or in the text.","section":"Section 6.1"},{"comment":"The left panels of Figure 1 plot 'Mean error' with scale factors (1e-4, 1e-3, 1e-5) but do not state whether the displayed quantity is the absolute cost difference |C_hat(t)-C(t)| or a relative error, and no error bars are provided.","section":"Section 6.2, Figure 1"},{"comment":"There are typos in the table and text: '2D quatroter' and '2D quadroter' should be '2D quadrotor', and 'Hidden unites' should be 'Hidden units'.","section":"Section 6.1"},{"comment":"The sentence 'The proof is provided in the arXiv version (Zeng et al., 2024c)' is unnecessary because the appendix contains a full proof of Theorem 3; this reference to a preprint of the same paper should be removed or corrected.","section":"Section 5"},{"comment":"The notation [L_hat_N]_{k,i} uses the index i, which also denotes the policy-iteration step; this dual use is confusing and should be disambiguated.","section":"Section 4.3, Eq. (15)"}],"recommendation":"major_revision","confidential_remarks":"The paper relies heavily on the authors' own prior work (Meng et al. 2024b and Zeng et al. 2024c) for key convergence results, and at the time of review those are arXiv preprints rather than peer-reviewed publications. The editor may wish to verify their availability and technical correctness. The main gap between Theorem 3 and the algorithm in Section 4.4 is serious and should be a condition for acceptance, as should the sign inconsistency in the proof of Theorem 3."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a solid empirical paper with an overreaching Theorem 3. The identification step is a genuine improvement — adding x-u coupling terms to the polynomial lift and using the Yosida approximation directly recovers f and g an order of magnitude better than the log-based and lifted-linear baselines, and the cost error versus the model-based oracle is genuinely small (1e-5 to 1e-3) on 2D through 9D systems. That part I believe.\n\nThe load-bearing soft spot is exactly what the stress-test says. The paper claims convergence of the learned value function and control laws to those of the true system, but Theorem 3 proves convergence for the exact solutions V_h^(i) of the GHJB equation, not for the random-feature least-squares solution β^T σ(Wx+b) that the algorithm actually computes. There is no residual bound, no representability condition, no consistency estimate linking the least-squares β to the exact V_h^(i). So the main theoretical certificate does not cover the computed controller. That is a real gap, not a nitpick. It does not make the empirical claim false, but it makes the advertised 'theoretically certifiable framework' considerably weaker. The proof in Appendix A also assumes V_h and V are C^1 and uses Lipschitz constants in a Gronwall argument, but it never establishes that the random-feature policy iteration actually produces a C^1 value function; the exact GHJB solution may be C^1, but the random-feature fit has the smoothness of its basis, and no smoothness is shown.\n\nAssumption 1 (dense sampling makes the identified vector field close to the truncated Yosida generator) is asserted, not checked. The small identification errors in Table 3 are supportive, but the paper never reports the actual ∥F_{λ,T,N} − F̂∥ values. Some experimental parameters are also missing — the random feature weight/bias distributions, the activation function, and the specific λ, Tmax, psum values are only partially listed.\n\nOn balance: the empirical contribution is real and relevant; the theory needs substantial work before it matches the 'certifiable' claim. I would send this to review, not desk-reject, but I would push hard on the GHJB-versus-random-feature gap and on reporting the missing implementation details. The paper is a serious piece of work from a group that clearly knows the area; it is just not yet as certified as it claims to be.","headline":"Solid empirical paper with an overreaching theorem: the random-feature GHJB fit is not covered by the convergence proof, but the identification improvement is real and the experiments are convincing.","tokens_in":14514,"tokens_out":2670,"would_cite":true,"duration_ms":24675,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A data-driven controller trained on sampled trajectories can match the value function, control law, and closed-loop cost of a controller designed with full knowledge of the underlying nonlinear dynamics.","keywords":["Koopman operator","system identification","optimal control","policy iteration","Hamilton-Jacobi-Bellman equation","Yosida approximation","random feature networks","high-dimensional nonlinear systems"],"falsifier":"Run the pipeline on a control-affine system with a known closed-form GHJB solution, record the residual of the random-feature least-squares GHJB solve at each policy-iteration step, and compare the realized value-function error to the error Theorem 3 would bound from the identified dynamics alone; if the residual is not small, the computed controller can violate the claimed guarantee. Alternatively, run the 3D quadrotor test on a domain substantially larger than $[-0.1, 0.1]^9$ and check whether the $10^{-5}$ to $10^{-3}$ accumulated-cost error persists.","tokens_in":13526,"feed_emoji":"🎛️","tokens_out":6366,"duration_ms":55541,"temperature":0.7,"pith_summary":"This paper asks whether a near-optimal stabilizing controller for an unknown nonlinear system can be learned from sampled trajectories alone, with the same guarantees as a model-based design. It claims yes: identify the drift $f$ and actuation $g$ from data through a modified Koopman-generator estimate, then run policy iteration where each step solves the generalized Hamilton-Jacobi-Bellman equation by a random-feature least-squares fit. The authors prove that, at each iteration, the learned value function and control law converge uniformly to those of the true system as identification resolution and function-approximation capacity grow. In experiments on systems with 2 to 9 states, the accumulated cost of the learned control matches a controller trained with full model knowledge to within $10^{-5}$ to $10^{-3}$.","feed_headline":"Data-only control matches full-model optimal cost to 1e-5","feed_subtitle":"A Koopman-generator pipeline identifies drift and actuation from samples, then solves the HJB equation with a random-feature least-squares…","key_machinery":"The load-bearing object is the Yosida approximation of the Koopman generator, $L_\\lambda \\varphi = \\lambda^2 \\int_0^\\infty e^{-\\lambda t} U^t \\varphi \\, dt - \\lambda \\varphi$, truncated to a finite horizon and evaluated on sampled data by quadrature, then fitted by least squares to give a matrix representation of the generator. Using observables that are polynomial in $x$ and linear in $u$ lets the identified generator directly recover the control-affine terms $f(x) + g(x)u$. The value-function step is carried by a random-feature network $\\hat{V}(x) = \\beta^T \\sigma(Wx + b)$ with random $W$ and $b$, which turns each GHJB policy-evaluation equation into a linear least-squares problem for $\\beta$ and makes the iteration scalable to state dimension 9.","core_discovery":"The central claim is that a fully data-driven procedure can reproduce the result of exact policy iteration for a control-affine nonlinear system. The procedure first samples trajectories of the unknown dynamics, uses a Yosida-type approximation to estimate the Koopman generator on an augmented state-input space, and recovers estimates of $f(x)$ and $g(x)$. It then solves the GHJB equation at each policy-evaluation step by fitting a random-feature neural network through linear least squares, which keeps the iteration tractable. The paper's Theorem 3 states that if the identified vector field and the initial policy are close enough to the true ones, the value function and policy obtained at every iteration are uniformly close to the exact ones; Theorem 2 provides the convergence of the identified vector field as $\\lambda \\to \\infty$, $T_{\\max} \\to \\infty$, and basis size $N \\to \\infty$. The numerical section demonstrates stabilizing control and cost errors from $10^{-5}$ to $10^{-3}$ on inverted pendulum, cartpole, 2D quadrotor, and 3D quadrotor examples.","pith_inferences":["A natural next step would be to turn the least-squares residual of the random-feature GHJB solve into a data-dependent certificate, since the paper's convergence proof assumes the exact GHJB solution is available.","If the same identification step is combined with a different PDE solver (for example a physics-informed network), the policy-iteration guarantee would likely transfer, because the theorem only requires closeness of the value function and its gradient.","The current polynomial observables are linear in each control input; extending to non-affine systems would require observables with higher powers of $u$, which would change the recovery step and is not covered by the analysis.","The cost-error range $10^{-5}$ to $10^{-3}$ is reported on small compact domains; testing the method on larger or less regular operating regions would reveal how quickly the guarantee degrades with domain size."],"forward_implications":["A controller can be synthesized for a 9-state, 4-input nonlinear plant without a model, and still stabilize the true system in simulation.","Each policy-iteration step inherits a uniform convergence guarantee: if the identified dynamics are close to the true dynamics, the value function and policy are close to the exact ones at that step.","The closed-loop accumulated cost of the learned controller tracks the cost of a full-model controller to within $10^{-5}$ to $10^{-3}$ on the tested domains.","The improved identification accuracy (one to two orders of magnitude lower $f$ and $g$ error than the log-based and lifted-linear baselines) is what makes the HJB solution usable in higher dimensions."],"supporting_citations":[{"why":"Introduces the Koopman operator whose generator the paper estimates for system identification.","marker":"Koopman (1931)"},{"why":"Supplies the Yosida-approximation convergence results (Theorems 3.3 and 4.2) on which Theorem 2's vector-field convergence is built.","marker":"Meng et al. (2024b)"},{"why":"Provides the logarithm-based lifting baseline for identification accuracy and the standard Koopman-lifting identification approach.","marker":"Mauroy and Gonçalves (2019)"},{"why":"Is the lifted linear-model baseline used for control-oriented Koopman identification, often neglecting coupling terms.","marker":"Korda and Mezić (2018b)"},{"why":"Is the source of the exact policy-iteration convergence theory and the ADP baseline compared in experiments.","marker":"Jiang and Jiang (2017)"},{"why":"Establishes the Galerkin GHJB policy-evaluation approach that the random-feature least-squares solver replaces.","marker":"Beard et al. (1997)"},{"why":"Provides the continuous-dependence-on-parameters lemma used in the proof of Theorem 3 to bound trajectory differences.","marker":"Khalil (2002)"},{"why":"Provides the Lyapunov characterization and derivative formulas for value functions used in the proof of Theorem 3.","marker":"Liu et al. (2023)"}],"fun_headline_variants":["Data-driven control hits optimal cost to 1e-5","Model-free Koopman control matches optimal to 1e-5","Koopman-based RL stabilizes unknown 9D systems","Data-only policy iteration reaches 1e-5 optimal cost","Unknown dynamics tamed via Koopman generator learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that dense enough sampling makes the identified vector field close to the truncated Yosida generator, and that the random-feature least-squares fit is close enough to the exact GHJB solution for Theorem 3 to apply; the paper does not bound the second gap.","fun_headline_variants_meta":{"raw":{"variants":["Data-driven control hits optimal cost to 1e-5","Model-free Koopman control matches optimal to 1e-5","Koopman-based RL stabilizes unknown 9D systems","Data-only policy iteration reaches 1e-5 optimal cost","Unknown dynamics tamed via Koopman generator learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00054,"raw_usage":{"total_tokens":2583,"prompt_tokens":933,"completion_tokens":1650,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":1564}},"tokens_in":549,"tokens_out":1650,"duration_ms":11617,"temperature":1.0,"reasoning_tokens":1564,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:42:05.671835+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pipeline on a control-affine system with a known closed-form GHJB solution, record the residual of the random-feature least-squares GHJB solve at each policy-iteration step, and compare the realized value-function error to the error Theorem 3 would bound from the identified dynamics alone; if the residual is not small, the computed controller can violate the claimed guarantee. Alternatively, run the 3D quadrotor test on a domain substantially larger than $[-0.1, 0.1]^9$ and check whether the $10^{-5}$ to $10^{-3}$ accumulated-cost error persists.","supporting_citations":[{"cited_title":"Hamiltonian systems and transformation in Hilbert space","cited_arxiv_id":null,"evidence_quote":"Introduces the Koopman operator whose generator the paper estimates for system identification."},{"cited_title":"Koopman-based lifting techniques for nonlinear systems identification","cited_arxiv_id":null,"evidence_quote":"Provides the logarithm-based lifting baseline for identification accuracy and the standard Koopman-lifting identification approach."},{"cited_title":"Robust Adaptive Dynamic Programming","cited_arxiv_id":null,"evidence_quote":"Is the source of the exact policy-iteration convergence theory and the ADP baseline compared in experiments."},{"cited_title":"Galerkin approximations of the generalized H amilton- J acobi- B ellman equation","cited_arxiv_id":null,"evidence_quote":"Establishes the Galerkin GHJB policy-evaluation approach that the random-feature least-squares solver replaces."},{"cited_title":"Physics-Informed Neural Network Lyapunov Functions: PDE Characterization, Learning, and Verification","cited_arxiv_id":"2312.09131","evidence_quote":"Provides the Lyapunov characterization and derivative formulas for value functions used in the proof of Theorem 3."}],"review_version":1}