RECORD
Combining Model-Based and Model-Free Methods for Nonlinear Control: A Provably Convergent Policy Gradient Approach
Optimization and Control · Submitted 12 Jun 2020
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim We show this hybrid approach outperforms the model-based controller while avoiding the convergence issues associated with model-free approaches via both numerical experiments and theoretical analyses, in which we derive sufficient conditions on the non-linear component such that our approach is guaranteed to converge to the (nearly) global optimal controller.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Model-free learning-based control methods have seen great success recently. However, such methods typically suffer from poor sample complexity and limited convergence guarantees. This is in sharp contrast to classical model-based control, which has a rich theory but typically requires strong modeling assumptions. In this paper, we combine the two approaches to achieve the best of both worlds. We consider a dynamical system with both linear and non-linear components and develop a novel approach to use the linear model to define a warm start for a model-free, policy gradient method. We show this hybrid approach outperforms the model-based controller while avoiding the convergence issues associated with model-free approaches via both numerical experiments and theoretical analyses, in which we derive sufficient conditions on the non-linear component such that our approach is guaranteed to converge to the (nearly) global optimal controller.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Model-free learning-based control methods have seen great success recently. | background 1.00 | 0.04 | 0.01 | asserted 1.00 |
| s1 | However, such methods typically suffer from poor sample complexity and limited convergence guarantees. | background 0.69 | 0.06 | 0.02 | asserted 0.90 |
| s2 | This is in sharp contrast to classical model-based control, which has a rich theory but typically requires strong modeling assumptions. | background 0.94 | 0.04 | 0.02 | asserted 0.98 |
| s3 | In this paper, we combine the two approaches to achieve the best of both worlds. | contribution 0.97 | 0.54 | 0.12 | asserted 0.92 |
| s4 | We consider a dynamical system with both linear and non-linear components and develop a novel approach to use the linear model to define a warm start for a model-free, policy gradient method. | method 0.88 | 0.96 | 0.06 | asserted 0.98 |
| s5 | We show this hybrid approach outperforms the model-based controller while avoiding the convergence issues associated with model-free approaches via both numerical experiments and theoretical analyses, in which we derive sufficient conditions on the non-linear component such that our approach is guaranteed to converge to the (nearly) global optimal controller. | result 0.92 | 0.97 | 0.93 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Model-free learning-based control methods have seen great success recently. However, such methods typically suffer from poor sample complexity and limited convergence guarantees. This is in sharp contrast to classical model-based control, which has a rich theory but typically requires strong modeling assumptions. In this paper, we combine the two approaches to achieve the best of both worlds. We consider a dynamical system with both linear and non-linear components and develop a novel approach to use the linear model to define a warm start for a model-free, policy gradient method. We show this hybrid approach outperforms the model-based controller while avoiding the convergence issues associated with model-free approaches via both numerical experiments and theoretical analyses, in which we derive sufficient conditions on the non-linear component such that our approach is guaranteed to converge to the (nearly) global optimal controller.
Human review
References
How the literature uses it
14 arXiv papers cite this in 30 sentences
Papers citing it · 19 sentences without a confident label
Compares with it 1 paper
-
In fact, for $= 0.5$, the GC algorithm did not enter the model-based regime at all; the algorithm only stayed in the model-free regime and behaved identically to the model-free method in [ref] shown in Fig. [ref].
Accelerating Model-Free Policy Optimization Using Model-Based Gradient: A Composite Optimization Perspective · Results · 0.97
Cites it as background 5 papers
-
We consider a modified discrete-time LQR problem, which appeared in [ref] .
Accelerating Model-Free Policy Optimization Using Model-Based Gradient: A Composite Optimization Perspective · Background: Nonlinear optimal control with quadratic cost · 0.88
-
Lastly, our work is relevant to the literature on PO for control [ref] and the theoretical foundation underlying this approach [ref] .
Policy Optimization for PDE Control with a Warm Start · Related Literature · 0.93
-
For the analysis of PG methods for nonlinear systems, we omit here and refer interested readers to the following literature [ref] .
Convergence and Sample Complexity of Policy Gradient Methods for Stabilizing Linear Systems · Nonlinear systems · 0.90
What it cites · 13 sentences without a confident label
Compares with 1 paper
-
This is different than existing landscape analysis for non-convex optimization in other contexts like deep learning, where only local convexity is shown without showing how to enter its attraction basin [ref] .
Stochastic Mirror Descent on Overparameterized Nonlinear Models: Convergence, Implicit Regularization, and Generalization · Main Results · 0.96
Cites as background 19 papers
-
A common theme in this line of work is that the underlying dynamical system is assumed to be linear, under which the cost function is shown to satisfy a “gradient dominance” property [ref] , which implies the model-free policy search method will converge to the global optimal controller.
Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator · Introduction · 0.81
-
Our work is mostly related to the class of model-free policy search methods for the Linear Quadratic Regulator (LQR), e.g. zeroth order policy search in [ref] and actor-critic methods in [ref] .
Distributed Reinforcement Learning for Decentralized Linear Quadratic Control: A Derivative-Free Policy Optimization Approach · Introduction · 0.89
-
Our work is mostly related to the class of model-free policy search methods for the Linear Quadratic Regulator (LQR), e.g. zeroth order policy search in [ref] and actor-critic methods in [ref] .
Convergence and sample complexity of gradient methods for the model-free linear quadratic regulator problem · Introduction · 0.89
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.