How are policy gradient methods affected by the limits of control?

Anastasios Tsiamis; Henrik Sandberg; Ingvar Ziemann; Nikolai Matni

RECORD

How are policy gradient methods affected by the limits of control?

Ingvar Ziemann, Anastasios Tsiamis, Henrik Sandberg, Nikolai Matni

Optimization and Control · Submitted 14 Jun 2022

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim Our main result is that ill-conditioned linear systems in the sense of Doyle inevitably lead to noisy gradient estimates.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2206.06863 (version 1)

Pith IDD3MMDWRM, Pith's permanent record of this paper and its review

CitationAnastasios Tsiamis, Henrik Sandberg, Ingvar Ziemann, Nikolai Matni. "How are policy gradient methods affected by the limits of control?." arXiv:2206.06863 (2022). https://pith.science/paper/D3MMDWRM

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 3,459 input tokens.

What the abstract itself states, as probabilities: claims first 0.03 · states limitation 0.46 · human participants 0.02 · controlled comparison 0.03 · randomized assignment 0.02 · states quantitative result 0.09 · compares with prior results 0.07 · mentions code or data availability 0.02.

The abstract, sentence by sentence

arXiv listing abstract v1. Results and the main claim are underlined; show every role.

We study stochastic policy gradient methods from the perspective of control-theoretic limitations. Our main result is that ill-conditioned linear systems in the sense of Doyle inevitably lead to noisy gradient estimates. We also give an example of a class of stable systems in which policy gradient methods suffer from the curse of dimensionality. Our results apply to both state feedback and partially observed systems.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	We study stochastic policy gradient methods from the perspective of control-theoretic limitations.	contribution 0.96	0.84	0.03	asserted 0.66
s1	Our main result is that ill-conditioned linear systems in the sense of Doyle inevitably lead to noisy gradient estimates.	result 1.00	0.88	0.98	asserted 1.00
s2	We also give an example of a class of stable systems in which policy gradient methods suffer from the curse of dimensionality.	result 0.97	0.92	0.04	asserted 1.00
s3	Our results apply to both state feedback and partially observed systems.	result 0.50	0.69	0.03	asserted 1.00

What kind of paper the classifier reports this is: theoretical result 1.00 · evidence from proof or derivation 0.99.

What was examined, and what was not

sources read

arXiv listing abstract v1

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

We study stochastic policy gradient methods from the perspective of control-theoretic limitations. Our main result is that ill-conditioned linear systems in the sense of Doyle inevitably lead to noisy gradient estimates. We also give an example of a class of stable systems in which policy gradient methods suffer from the curse of dimensionality. Our results apply to both state feedback and partially observed systems.

Human review

How Pith checks papers → · Full audit of this paper →

How the literature uses it

3 arXiv papers cite this in 3 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites · 7 sentences without a confident label

Uses the method or tool of 2 papers

The proof of this fact relies on a construction due to [ref] .

Learning to Control Linear Systems can be Hard · Consequences of Theorem [ref] · 0.95
This perturbation is convenient since $A'+B'K_⋆= A+BK_⋆$ for any $Δ$ and has previously been used in [ref] .

Regret Lower Bounds for Learning Linear Quadratic Gaussian Systems · Local Minimax Complexities · 0.91

Compares with 2 papers

It so happens that this complexity measure is very similar to that dictating regret lower bounds in adaptive LQR [ref] .

Regret Lower Bounds for Learning Linear Quadratic Gaussian Systems · Consequences of Theorem [ref] · 1.00
While we give lower bounds valid for any estimator in this work, [ref] analyzes the variance of a particular gradient estimator known as REINFORCE.

Analyzing the Variance of Policy Gradient Estimators for the Linear-Quadratic Regulator · Related Work · 1.00

Cites as background 6 papers

Our work also relates to [ref] , which also study fundamental limits in learning-enabled control.

Learning to Control Linear Systems can be Hard · Related Work · 0.85
Our work also relates to [ref] , which also study fundamental limits in learning-enabled control.

Linear Systems can be Hard to Learn · Related Work · 0.85
Our work also relates to [ref] , which also study fundamental limits in learning-enabled control.

Regret Lower Bounds for Learning Linear Quadratic Gaussian Systems · Related Work · 0.85

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.