REVIEW 5 major objections 5 minor 28 references
Data-driven optimal control of unknown nonlinear dynamical systems using the Koopman operator
T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A data-driven controller trained on sampled trajectories can match the value function, control law, and closed-loop cost of a controller designed with full knowledge of the underlying nonlinear dynamics.
desk verdict Solid empirical paper with an overreaching theorem: the random-feature GHJB fit is not covered by the convergence proof, but the identification improvement is real and the experiments are convincing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Yosida approximation of the Koopman generator, $L_\lambda \varphi = \lambda^2 \int_0^\infty e^{-\lambda t} U^t \varphi \, dt - \lambda \varphi$, truncated to a finite horizon and evaluated on sampled data by quadrature, then fitted by least squares to give a matrix representation of the generator. Using observables that are polynomial in $x$ and linear in $u$ lets the identified generator directly recover the control-affine terms $f(x) + g(x)u$. The value-function step is carried by a random-feature network $\hat{V}(x) = \beta^T \sigma(Wx + b)$ with random $W$ and $b$, which turns each GHJB policy-evaluation equation into a linear least-squares problem for $\beta$ and makes the iteration scalable to state dimension 9.
What would settle it
Run the pipeline on a control-affine system with a known closed-form GHJB solution, record the residual of the random-feature least-squares GHJB solve at each policy-iteration step, and compare the realized value-function error to the error Theorem 3 would bound from the identified dynamics alone; if the residual is not small, the computed controller can violate the claimed guarantee. Alternatively, run the 3D quadrotor test on a domain substantially larger than $[-0.1, 0.1]^9$ and check whether the $10^{-5}$ to $10^{-3}$ accumulated-cost error persists.
Extended reading notes
Core claim
The central claim is that a fully data-driven procedure can reproduce the result of exact policy iteration for a control-affine nonlinear system. The procedure first samples trajectories of the unknown dynamics, uses a Yosida-type approximation to estimate the Koopman generator on an augmented state-input space, and recovers estimates of $f(x)$ and $g(x)$. It then solves the GHJB equation at each policy-evaluation step by fitting a random-feature neural network through linear least squares, which keeps the iteration tractable. The paper's Theorem 3 states that if the identified vector field and the initial policy are close enough to the true ones, the value function and policy obtained at every iteration are uniformly close to the exact ones; Theorem 2 provides the convergence of the identified vector field as $\lambda \to \infty$, $T_{\max} \to \infty$, and basis size $N \to \infty$. The numerical section demonstrates stabilizing control and cost errors from $10^{-5}$ to $10^{-3}$ on inverted pendulum, cartpole, 2D quadrotor, and 3D quadrotor examples.
Load-bearing premise
The argument assumes that dense enough sampling makes the identified vector field close to the truncated Yosida generator, and that the random-feature least-squares fit is close enough to the exact GHJB solution for Theorem 3 to apply; the paper does not bound the second gap.
Editorial extensions
If this is right
- A controller can be synthesized for a 9-state, 4-input nonlinear plant without a model, and still stabilize the true system in simulation.
- Each policy-iteration step inherits a uniform convergence guarantee: if the identified dynamics are close to the true dynamics, the value function and policy are close to the exact ones at that step.
- The closed-loop accumulated cost of the learned controller tracks the cost of a full-model controller to within $10^{-5}$ to $10^{-3}$ on the tested domains.
- The improved identification accuracy (one to two orders of magnitude lower $f$ and $g$ error than the log-based and lifted-linear baselines) is what makes the HJB solution usable in higher dimensions.
Reading between the lines
- A natural next step would be to turn the least-squares residual of the random-feature GHJB solve into a data-dependent certificate, since the paper's convergence proof assumes the exact GHJB solution is available.
- If the same identification step is combined with a different PDE solver (for example a physics-informed network), the policy-iteration guarantee would likely transfer, because the theorem only requires closeness of the value function and its gradient.
- The current polynomial observables are linear in each control input; extending to non-affine systems would require observables with higher powers of $u$, which would change the recovery step and is not covered by the analysis.
- The cost-error range $10^{-5}$ to $10^{-3}$ is reported on small compact domains; testing the method on larger or less regular operating regions would reveal how quickly the guarantee degrades with domain size.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a data-driven optimal control framework for unknown control-affine nonlinear systems. The method first identifies the dynamics by estimating the Koopman generator via a Yosida approximation with polynomial observables that include control-affine coupling terms, then solves the optimal control problem by policy iteration in which the Generalized Hamilton-Jacobi-Bellman (GHJB) equation is approximately solved with random-feature neural networks. The authors claim that the identified vector field converges to the true vector field and that the value function and control policy converge at each iteration to those of the true system. Numerical experiments on inverted pendulum, cartpole, 2D quadrotor, and 3D quadrotor systems report low identification errors and accumulated-cost errors between 10^-5 and 10^-3 relative to a model-based oracle. The central theoretical result, Theorem 3, is meant to certify the convergence of the value function and policy, but the proof only covers exact solutions of the GHJB equation, not the random-feature least-squares solution actually computed by the algorithm.
Significance. If the theoretical gaps were closed, the paper would be a valuable contribution. The empirical results are strong: the resolvent-based control-affine identification improves identification errors by one to two orders of magnitude over the logarithm-based and lifted-linear baselines, and the learned controllers stabilize true 9-dimensional systems with cost errors on the order of 1e-5 to 1e-3. The paper also explicitly avoids the operator-logarithm step, which is a technically useful idea. However, the claimed convergence guarantee does not currently cover the actual algorithm, and the proof of Theorem 3 contains sign and regularity issues. These are load-bearing problems for the paper's main theoretical claim, so the manuscript needs substantial revision before it can be accepted.
major comments (5)
- [Section 4.4 and Theorem 3] Theorem 3 in Section 5, proved in Appendix A, applies only to the exact solutions V_h^{(i)} of the GHJB equations (19)-(20). The algorithm in Section 4.4, however, does not solve these equations; it fits a random-feature ansatz V(x) = beta^T sigma(Wx+b) by linear least squares. The paper provides no residual bound, no representability condition, and no consistency estimate relating the least-squares solution to the exact GHJB solution. Therefore the abstract's claim that 'the learned value function and control laws are proven to converge' is not justified for the quantities actually computed in the experiments. This is the main gap in the theoretical certificate.
- [Appendix A, Eqs. (23)-(24)] In the proof of Theorem 3, L_h^{(i)} is defined as -Q(x) - kappa_{h}^{(i-1)}(x)^T R kappa_{h}^{(i-1)}(x), which is negative definite. The tail integrals in (23)-(24) are therefore non-positive, so the inequalities 0 <= integral_{T1}^infinity L_h^{(i)} dt <= theta/2 cannot hold as written. The proof of the value-function bound |V_h^{(i)} - V^{(i)}| <= theta is thus not established. The sign convention in the GHJB equations and the subsequent integral representation must be reconciled, and absolute values should be used for the tail estimates.
- [Appendix A, Eqs. (50) and (52)] The proof of the control-law convergence uses the bound ||D L^{(i)} - D L_h^{(i)}||_infinity <= C_L ||L^{(i)} - L_h^{(i)}||_infinity in Eq. (50) and assumes in Eq. (52) that D L_h^{(i)} is Lipschitz. Neither statement follows from the theorem's assumption that L^{(i)}, L_h^{(i)} in C^1(Omega). These estimates require at least C^2 regularity or Lipschitz gradients. The assumptions of Theorem 3 must be strengthened or the proof revised accordingly.
- [Section 5, Assumption 1] Assumption 1 simply postulates that sufficiently dense sampling makes the identified vector field uniformly close to the truncated Yosida vector field. No sample-complexity bound, noise model, or relation between the number of samples M, the truncation parameters lambda and Tmax, and the finite-dimensional subspace N is given. Table 3 reports average identification errors for the specific test systems, but this does not verify the uniform closeness over the whole compact set M required for the subsequent conclusion (18). This assumption is load-bearing for the identification part of the theory and should either be proved for the proposed least-squares estimator or explicitly tested and reported.
- [Section 4.4 and Section 6] The paper does not explain how an initial admissible (stabilizing) policy kappa_h^{(0)} is obtained from data, even though Theorem 3 assumes ||kappa_h^{(0)} - kappa^{(0)}||_infinity < delta and policy iteration requires a stabilizing initial controller. Without a data-driven procedure for computing such an initial policy, the practical applicability of the method is not fully demonstrated, and this requirement should be discussed explicitly.
minor comments (5)
- [Section 6.1] Several parameter values needed for reproducibility are missing: the Yosida parameter lambda, the truncation horizon Tmax, the numerical quadrature scheme for (13), and the activation function and distribution for the random features W and b in Section 4.4 are not reported in Tables 1-2 or in the text.
- [Section 6.2, Figure 1] The left panels of Figure 1 plot 'Mean error' with scale factors (1e-4, 1e-3, 1e-5) but do not state whether the displayed quantity is the absolute cost difference |C_hat(t)-C(t)| or a relative error, and no error bars are provided.
- [Section 6.1] There are typos in the table and text: '2D quatroter' and '2D quadroter' should be '2D quadrotor', and 'Hidden unites' should be 'Hidden units'.
- [Section 5] The sentence 'The proof is provided in the arXiv version (Zeng et al., 2024c)' is unnecessary because the appendix contains a full proof of Theorem 3; this reference to a preprint of the same paper should be removed or corrected.
- [Section 4.3, Eq. (15)] The notation [L_hat_N]_{k,i} uses the index i, which also denotes the policy-iteration step; this dual use is confusing and should be disambiguated.
Circularity Check
No circularity: the derivation is self-contained; the known approximation gap is a correctness concern, not a definitional reduction.
full rationale
The claimed convergence results relate quantities that are defined independently rather than in terms of one another. The identified vector field is obtained from the Yosida approximation of the Koopman generator, the value functions are defined as integral costs, and the control laws are defined through the Hamiltonians of the identified model. Theorem 3 and its Appendix A proof concern the exact GHJB solutions of the identified and true systems, and the proof is a standard continuity argument using Gronwall-type estimates; it does not assume the computed random-feature least-squares coefficient beta equals the exact GHJB solution. That leaves a real representability gap between the algorithm of Section 4.4 and the theorem, but the gap is a missing residual bound rather than a circular step, since the theorem's hypotheses do not include the conclusion. The reliance on prior work (Meng et al. 2024b for Yosida convergence and error bounds, Liu et al. 2023 for derivative formulas) is self-citation, but those are stated mathematical results with hypotheses that do not themselves assert the present paper's convergence claims, so they are independent support rather than load-bearing circularity. The reported cost errors are computed by forward simulation of the true system under the learned controller, giving genuine comparisons against a model-based oracle rather than fitted inputs renamed as predictions. No step reduces to its own inputs by construction, so the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (6)
- lambda (Yosida parameter) =
not reported
- Tmax (truncation horizon) =
1 (implied by data horizon t in [0,1])
- Polynomial order pmax or psum =
5 (2D), 3 (others)
- Number of random features s =
200 (2D), 3200 (others)
- Cost weighting matrices Q and R =
not reported
- Random feature weights W and b distribution =
not reported
assumptions (4)
- domain assumption Sampled initial conditions for the extended system (11) are sufficiently dense in M so that the identified vector field \hat F from (15) satisfies ||F_{lambda,Tmax,N} - \hat F||_infty <= theta (Assumption 1).
- domain assumption The GHJB solution V^{(i)} and V_h^{(i)} are C^1 and the value functions are finite Lyapunov functions on Omega.
- ad hoc to paper L_h^{(i)} is C^1 on Omega, and in the proof of Theorem 3 the derivative DL_h is assumed Lipschitz, which requires C^2 smoothness.
- standard math The system is control-affine with f in C^1, g smooth, f(0)=0, and 0 is an equilibrium.
Cite this review
Pith. "Pith review of Data-driven optimal control of unknown nonlinear dynamical systems using the Koopman operator." pith.science (2026). https://pith.science/paper/4VOPFHFN
@misc{pith2026241201085,
author = {Pith},
title = {Pith review of: Data-driven optimal control of unknown nonlinear dynamical systems using the Koopman operator},
year = {2026},
howpublished = {\url{https://pith.science/paper/4VOPFHFN}},
note = {Machine review of arXiv:2412.01085}
}
abstract
Nonlinear optimal control is vital for numerous applications but remains challenging for unknown systems due to the difficulties in accurately modelling dynamics and handling computational demands, particularly in high-dimensional settings. This work develops a theoretically certifiable framework that integrates a modified Koopman operator approach with model-based reinforcement learning to address these challenges. By relaxing the requirements on observable functions, our method incorporates nonlinear terms involving both states and control inputs, significantly enhancing system identification accuracy. Moreover, by leveraging the power of neural networks to solve partial differential equations (PDEs), our approach is able to achieving stabilizing control for high-dimensional dynamical systems, up to 9-dimensional. The learned value function and control laws are proven to converge to those of the true system at each iteration. Additionally, the accumulated cost of the learned control closely approximates that of the true system, with errors ranging from $10^{-5}$ to $10^{-3}$.
Figures
Reference graph
Works this paper leans on
-
[1]
Galerkin approximations of the generalized H amilton- J acobi- B ellman equation
Randal W Beard, George N Saridis, and John T Wen. Galerkin approximations of the generalized H amilton- J acobi- B ellman equation. Automatica, 33 0 (12): 0 2159--2177, 1997
work page 1997
-
[2]
Approximate solutions to the time-invariant H amilton-- J acobi-- B ellman equation
Randal W Beard, George N Saridis, and John T Wen. Approximate solutions to the time-invariant H amilton-- J acobi-- B ellman equation. Journal of Optimization theory and Applications, 96: 0 589--626, 1998
work page 1998
-
[3]
Improving the Closed-Loop Performance of Nonlinear Systems
Randal Winston Beard. Improving the Closed-Loop Performance of Nonlinear Systems. Rensselaer Polytechnic Institute, 1995
work page 1995
-
[4]
Bellman, R.E
R. Bellman, R.E. Bellman, and Rand Corporation. Dynamic Programming. Rand Corporation research study. Princeton University Press, 1957
1957
-
[5]
A piecewise learning framework for control of unknown nonlinear systems with stability guarantees
Milad Farsi, Yinan Li, Ye Yuan, and Jun Liu. A piecewise learning framework for control of unknown nonlinear systems with stability guarantees. In Learning for Dynamics and Control Conference, pages 830--843. PMLR, 2022
work page 2022
-
[6]
R.A. Howard. Dynamic Programming and Markov Processes. Technology Press of Massachusetts Institute of Technology, 1960
work page 1960
-
[7]
Yu Jiang and Zhong-Ping Jiang. Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics. Automatica, 48 0 (10): 0 2699--2704, 2012
work page 2012
-
[8]
Robust adaptive dynamic programming and feedback stabilization of nonlinear systems
Yu Jiang and Zhong-Ping Jiang. Robust adaptive dynamic programming and feedback stabilization of nonlinear systems. IEEE Transactions on Neural Networks and Learning Systems, 25 0 (5): 0 882--893, 2014
work page 2014
Show all 28 references
-
[9]
Robust Adaptive Dynamic Programming
Yu Jiang and Zhong-Ping Jiang. Robust Adaptive Dynamic Programming. John Wiley & Sons, 2017
2017
-
[10]
H.K. Khalil. Nonlinear Systems. Pearson Education. Prentice Hall, 2002. ISBN 9780130673893
2002
-
[11]
Hamiltonian systems and transformation in Hilbert space
Bernard O Koopman. Hamiltonian systems and transformation in Hilbert space . Proceedings of the National Academy of Sciences of the United States of America, 17 0 (5): 0 315, 1931
1931
-
[12]
On convergence of extended dynamic mode decomposition to the K oopman operator
Milan Korda and Igor Mezi \'c . On convergence of extended dynamic mode decomposition to the K oopman operator. Journal of Nonlinear Science, 28: 0 687--710, 2018 a
2018
-
[13]
Linear predictors for nonlinear dynamical systems: K oopman operator meets model predictive control
Milan Korda and Igor Mezi \'c . Linear predictors for nonlinear dynamical systems: K oopman operator meets model predictive control. Automatica, 93: 0 149--160, 2018 b
2018
-
[14]
Construction of suboptimal control sequences
RJ Leake and Ruey-Wen Liu. Construction of suboptimal control sequences. SIAM Journal on Control, 5 0 (1): 0 54--63, 1967
1967
-
[15]
Physics-informed neural network L yapunov functions: PDE characterization, learning, and verification
Jun Liu, Yiming Meng, Maxwell Fitzsimmons, and Ruikun Zhou. Physics-informed neural network L yapunov functions: PDE characterization, learning, and verification. arXiv preprint arXiv:2312.09131, 2023
2023 arXiv
-
[16]
Koopman-based lifting techniques for nonlinear systems identification
Alexandre Mauroy and Jorge Gon c alves. Koopman-based lifting techniques for nonlinear systems identification. IEEE Transactions on Automatic Control, 65 0 (6): 0 2550--2565, 2019
2019
-
[17]
The Koopman Operator in Systems and Control: Concepts, Methodologies, and Applications, volume 484
Alexandre Mauroy, Igor Mezi \'c , and Yoshihiko Susuki. The Koopman Operator in Systems and Control: Concepts, Methodologies, and Applications, volume 484. Springer Nature, 2020
2020
-
[18]
Physics-informed neural network policy iteration: Algorithms, convergence, and verification
Yiming Meng, Ruikun Zhou, Amartya Mukherjee, Maxwell Fitzsimmons, Christopher Song, and Jun Liu. Physics-informed neural network policy iteration: Algorithms, convergence, and verification. In Forty-first International Conference on Machine Learning (ICML), pages 35378--35403....
2024
-
[19]
Koopman-based learning of infinitesimal generators without operator logarithm
Yiming Meng, Ruikun Zhou, Melkior Ornik, and Jun Liu. Koopman-based learning of infinitesimal generators without operator logarithm. arXiv preprint arXiv:2403.15688, 2024 b
2024 arXiv
-
[20]
An approximation theory of optimal control for trainable manipulators
George N Saridis and Chun-Sing G Lee. An approximation theory of optimal control for trainable manipulators. IEEE Transactions on systems, Man, and Cybernetics, 9 0 (3): 0 152--159, 1979
1979
-
[21]
Dynamic mode decomposition of numerical and experimental data
Peter J Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of fluid mechanics, 656: 0 5--28, 2010
2010
-
[22]
Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems
Draguna Vrabie and Frank Lewis. Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems. Neural Networks, 22 0 (3): 0 237--246, 2009
2009
-
[23]
A data--driven approximation of the K oopman operator: Extending dynamic mode decomposition
Matthew O Williams, Ioannis G Kevrekidis, and Clarence W Rowley. A data--driven approximation of the K oopman operator: Extending dynamic mode decomposition. Journal of Nonlinear Science, 25: 0 1307--1346, 2015
2015
-
[24]
A sampling theorem for exact identification of continuous-time nonlinear dynamical systems
Zhexuan Zeng, Zuogong Yue, Alexandre Mauroy, Jorge Gon c alves, and Ye Yuan. A sampling theorem for exact identification of continuous-time nonlinear dynamical systems. In 2022 IEEE 61st Conference on Decision and Control (CDC), pages 6686--6692. IEEE, 2022
2022
-
[25]
A generalized N yquist- S hannon sampling theorem using the K oopman operator
Zhexuan Zeng, Jun Liu, and Ye Yuan. A generalized N yquist- S hannon sampling theorem using the K oopman operator. IEEE Transactions on Signal Processing, 2024 a
2024
-
[26]
A sampling theorem for exact identification of continuous-time nonlinear dynamical systems
Zhexuan Zeng, Zuogong Yue, Alexandre Mauroy, Jorge Gon c alves, and Ye Yuan. A sampling theorem for exact identification of continuous-time nonlinear dynamical systems. IEEE Transactions on Automatic Control, 2024 b
2024
-
[27]
Data-driven optimal control of unknown nonlinear dynamical systems using the K oopman operator
Zhexuan Zeng, Ruikun Zhou, Yiming Meng, and Jun Liu. Data-driven optimal control of unknown nonlinear dynamical systems using the K oopman operator. arXiv preprint, 2024 c
2024
-
[28]
Physics-informed extreme learning machine L yapunov functions
Ruikun Zhou, Maxwell Fitzsimmons, Yiming Meng, and Jun Liu. Physics-informed extreme learning machine L yapunov functions. IEEE Control Systems Letters, 2024
2024
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.