A gradient flow on a continuous-time Bellman error parametrized by feedback gain converges to the optimal LQR controller and stays inside the stabilizing region.
Policy Gradient-based Algorithms for Continuous-time Linear Quadratic Control
6 Pith papers cite this work. Polarity classification is still indexing.
6
Pith papers citing it
abstract
We consider the continuous-time Linear-Quadratic-Regulator (LQR) problem in terms of optimizing a real-valued matrix function over the set of feedback gains. The results developed are in parallel to those in Bu et al. [1] for discrete-time LTI systems. In this direction, we characterize several analytical properties (smoothness, coerciveness, quadratic growth) that are crucial in the analysis of gradient-based algorithms. We also point out similarities and distinctive features of the continuous time setup in comparison with its discrete time analogue. First, we examine three types of well-posed flows direct policy update for LQR: gradient flow, natural gradient flow and the quasi-Newton flow. The coercive property of the corresponding cost function suggests that these flows admit unique solutions while the gradient dominated property indicates that the underling Lyapunov functionals decay at an exponential rate; quadratic growth on the other hand guarantees that the trajectories of these flows are exponentially stable in the sense of Lyapunov. We then discuss the forward Euler discretization of these flows, realized as gradient descent, natural gradient descent and quasi-Newton iteration. We present stepsize criteria for gradient descent and natural gradient descent, guaranteeing that both algorithms converge linearly to the global optima. An optimal stepsize for the quasi-Newton iteration is also proposed, guaranteeing a $Q$-quadratic convergence rate--and in the meantime--recovering the Kleinman-Newton iteration. Lastly, we examine LQR state feedback synthesis with a sparsity pattern. In this case, we develop the necessary formalism and insights for projected gradient descent, allowing us to guarantee a sublinear rate of convergence to a first-order stationary point.
representative citing papers
Endpoint null-space projection yields a critic-free off-policy update for continuous-time LQR that recovers the Kleinman gain under a projected actor rank condition.
An indirect optimization method inner-approximates the set of passivating state-feedback gains for continuous-time LTI systems by a convex polytope and uses projected gradient flow to minimize the LQR cost inside that polytope.
Riemannian regularization reshapes the policy optimization landscape to enable learning of Kalman gains from data under unknown and rank-deficient covariances with non-asymptotic convergence guarantees.
A convex data-driven formulation yields the optimal LQI feedback gain for continuous-time systems directly from measured data without system matrices.
The authors adapt closed-loop and IRL parameterizations to continuous time, deriving policy iteration schemes, a data-driven CARE, convex reformulations, and a policy gradient flow while unifying the two approaches.
citing papers explorer
-
Bridging Continuous-time LQR and Reinforcement Learning via Gradient Flow of the Bellman Error
A gradient flow on a continuous-time Bellman error parametrized by feedback gain converges to the optimal LQR controller and stays inside the stabilizing region.
-
Data-Driven Critic-Free Policy Iteration for Continuous-Time Linear Quadratic Regulation
Endpoint null-space projection yields a critic-free off-policy update for continuous-time LQR that recovers the Kleinman gain under a projected actor rank condition.
-
Towards Optimal Passive Feedback Control of LTI Systems under LQR Performance
An indirect optimization method inner-approximates the set of passivating state-feedback gains for continuous-time LTI systems by a convex polytope and uses projected gradient flow to minimize the LQR cost inside that polytope.
-
Learning Kalman Policy for Singular Unknown Covariances via Riemannian Regularization
Riemannian regularization reshapes the policy optimization landscape to enable learning of Kalman gains from data under unknown and rank-deficient covariances with non-asymptotic convergence guarantees.
-
Data-driven Linear Quadratic Integral Control: A Convex Formulation and Policy Gradient Approach
A convex data-driven formulation yields the optimal LQI feedback gain for continuous-time systems directly from measured data without system matrices.
-
Data-Driven Continuous-Time Linear Quadratic Regulator via Closed-Loop and Reinforcement Learning Parameterizations
The authors adapt closed-loop and IRL parameterizations to continuous time, deriving policy iteration schemes, a data-driven CARE, convex reformulations, and a policy gradient flow while unifying the two approaches.