Pith. sign in

REVIEW 2 major objections 6 minor 24 references

Some remarks on gradient dominance and LQR policy optimization

T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper establishes that continuous-time LQR losses satisfy a saturated gradient dominance condition, yielding linear-exponential convergence and strong input-to-state stability of perturbed policy gradient flow.

desk verdict A clean, honest survey that repackages already-published LQR results into a useful sat-PŁI/ISS framework; the central claims are sound, but it is not a new-results paper and it contains one wrong motivating remark. read the letter →

arxiv 2507.10452 v2 pith:QFWJRI4G submitted 2025-07-14 cs.LG

Eduardo D. Sontag This is my paper · ORCID
classification cs.LG
keywords gradientdominancePolyak-Lojasiewiczinequalityinput-to-statestabilityLQRpolicyoptimizationcontinuous-timelinearquadraticregulatorflowoverparametrizedneuralnetworkslinear-exponentialconvergence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Policy optimization for the continuous-time linear quadratic regulator (LQR) has no global Polyak–Łojasiewicz gradient dominance bound, so exponential convergence rates can vanish for large initial conditions. This paper articulates and applies a generalized 'saturated' PŁI condition, $\alpha(r)=\sqrt{ar/(b+r)}$, which is global yet only guarantees linear decrease for large losses and exponential decrease near the optimum. The paper argues that every continuous-time LQR loss obeys this condition, and that the resulting perturbed gradient flow is strongly input-to-state stable: bounded errors in gradient estimates — from adversarial attacks, oracle mistakes, early stopping, or approximate simulators — cause only bounded degradation of the loss. The payoff is a principled robustness analysis for model-free LQR learning. The paper further shows that overparametrized linear neural network feedback can recover a genuine global PŁI estimate on uniformly imbalanced invariant sets, restoring global exponential convergence.

What carries the argument

The central object is the saturated Polyak–Łojasiewicz estimate (sat-PŁI): a comparison function $\alpha(r)=\sqrt{ar/(b+r)}$ that behaves like $\sqrt{(a/b)r}$ for small excess loss $r=L(k)-L_*$ and saturates at $\sqrt{a}$ for large $r$. This function carries the argument by converting gradient dominance $\|\nabla L(k)\|\ge\alpha(L(k)-L_*)$ into the differential inequality $\dot\ell\le-\alpha(\ell)^2$ for the excess loss, which integrates to exponential decay near the optimum and at worst linear decay globally. The paper's second mechanism is the dissipation-inequality view of ISS: read $\|\nabla L(k)\|\ge\alpha(L(k)-L_*)$ with $\alpha\in K_\infty$, $\alpha\in K$, or $\alpha$ merely positive definite, and one obtains ISS, siISS, or iISS respectively for the perturbed flow; sat-PŁI lands in the intersection that gives both iISS and siISS. A 'size function' — a proper, coercive, positive-definite distance to the target set — is what lets these properties be formulated on the stabilizing domain $D$ rather than all of $\mathbb{R}^n$.

What would settle it

Take any continuous-time LQR problem and run the gradient flow from stabilizing gains approaching the boundary of the stabilizing set; if $\|\nabla L(k_j)\|$ tends to zero while $L(k_j)-L_*$ tends to infinity, the claimed sat-PŁI estimate fails. For the strong-iISS claim, exhibit a bounded additive disturbance of arbitrarily small $L^\infty$ norm that drives the excess loss to infinity; such an example would falsify the robustness statement.

Watch

Extended reading notes

Core claim

The central claim is that the loss $L(k)$ of any continuous-time LQR problem satisfies a sat-PŁI gradient dominance estimate on the open stabilizing domain $D$: there exist $a,b>0$ such that $\|\nabla L(k)\|^2 \ge a(L(k)-L_*)/(b + L(k)-L_*)$ for all stabilizing gains $k$, where $L_*$ is the optimal cost. Equivalently, the comparison function $\alpha(r)=\sqrt{ar/(b+r)}$ interpolates between a $\sqrt{\lambda r}$-type local exponential rate and a constant global linear rate. The paper states that the perturbed gradient flow $\dot k = -\nabla L(k)^\top + B(k)u$ is strongly iISS — both integral ISS and small-input ISS — so bounded additive gradient errors yield bounded asymptotic suboptimality with controlled overshoot. For linear feedforward neural network controllers, the paper claims that on forward-invariant sets where the layer imbalance is bounded below, a genuine global PŁI estimate holds, giving global exponential convergence and faster convergence for larger imbalance.

Load-bearing premise

The robustness conclusions depend on the assumption that the gap between a candidate loss and the optimal loss grows without bound as the candidate approaches the boundary of the stabilizing region, and that gradient errors are bounded and additive rather than multiplicative; this coercivity is stated as a structural assumption and is not proved here for general LQR problems.

Editorial extensions

If this is right

  • Continuous-time LQR policy gradient flow converges to the optimal feedback with a rate that is exponential near the optimum and only linear far from it, which is the observed 'linear-exponential' behavior.
  • Bounded additive disturbances in the gradient estimate cause only bounded asymptotic suboptimality, with the transient overshoot controlled by a $KL$ estimate.
  • The gradient-dominance hierarchy gl-PŁI, sat-PŁI, sgl-PŁI, loc-PŁI maps onto the robustness hierarchy ISS, siISS, iISS, so any loss satisfying sat-PŁI is automatically strongly iISS under the stated structural assumptions.
  • For linear feedforward neural network controllers, overparametrization can restore global exponential convergence on uniformly imbalanced forward-invariant sets, and larger imbalance makes the loss decrease faster.
  • Except for a measure-zero set of initializations, single-hidden-layer linear network gradient flow converges to the global optimum; the non-optimal critical points are strict saddles whose products give low-rank approximations of the optimal feedback.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension of the same template: any policy-optimization loss that satisfies sat-PŁI and whose excess loss is coercive on its feasible domain should inherit strong iISS, so the coercivity check becomes the main bottleneck to carry these guarantees beyond LQR.
  • The Euler-discretization example shows the discrete-time PŁI constant $\lambda_h$ degrading as the step $h\to0$; a reader might infer that discrete-time gradient analyses become arbitrarily pessimistic for small learning rates, making the continuous-time bound the more honest rate estimate in that regime.
  • The imbalance speed-up theorem suggests a testable design principle for overparametrized control problems beyond LQR: initialize factorized controllers with deliberately unbalanced layer norms to accelerate gradient training.
  • Because the stated robustness results require bounded additive errors, multiplicative or unbounded estimation noise would need a separate analysis; the paper does not make claims for that regime.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This short paper, based on a keynote talk, reviews and connects several generalized Polyak-Łojasiewicz (PŁ) gradient-dominance conditions (gl-PŁI, sgl-PŁI, sat-PŁI) with input-to-state stability (ISS) properties of perturbed gradient flows. It applies these ideas to continuous-time linear quadratic regulator (LQR) policy optimization, asserting that the CT LQR loss satisfies a sat-PŁI estimate and that the associated perturbed gradient flow is strongly iISS (both iISS and siISS), in contrast to the discrete-time LQR problem which satisfies gl-PŁI and yields global exponential convergence. The paper also summarizes results on overparametrized linear feedforward neural network controllers for LQR, where an 'imbalance' measure speeds up convergence and can restore gl-PŁI on forward-invariant sets. Most of the main theorems are attributed to prior publications, especially [6] and [8].

Significance. If the stated results hold, the paper provides a valuable conceptual bridge between gradient-dominance inequalities and robustness concepts such as ISS, shedding light on the qualitative difference between continuous- and discrete-time LQR. The sat-PŁI class is a useful intermediate notion between global and semiglobal PŁI, and the connection to ISS could have applications beyond LQR. The paper is honest in flagging which results are proved in the scalar/narrow setting and which are supported mainly by simulation. The worked examples (1D integrator, Euler discretization) are clear and instructive. The main limitation is that the paper is a summary: several central theorems are quoted without proof, and the overparametrized section relies on an in-preparation reference for the key rigorous claims.

major comments (2)
  1. [Section 3, Theorem (three bullets after 'When applied using L as a Lyapunov function')] This theorem is the principal bridge between gradient-dominance classes and ISS properties, yet it is stated without proof or a specific citation. The rest of the paper's robustness discussion for LQR depends on it. Please add a proof sketch, for example the derivation ˙ℓ = -‖∇L‖² + ∇L·Bu ≤ -½‖∇L‖² + (‖B‖²/2)|u|² ≤ -½α(ℓ)² + c|u|², and indicate which of the standard ISS Lyapunov theorems (e.g., in [19]) the three conclusions follow from. If this theorem is already in [6] or [20], cite the precise statement.
  2. [Section 4, 'Imbalance Speed-up Theorem'] The theorem is stated in full generality ('Suppose that two solutions have... the respective imbalances satisfy...') without specifying the dimension of the system or the number of hidden layers. Later in the same section the text explains that theoretical results are only for n=m=1, N=2, and that for larger dimensions only simulations are provided. As written, the theorem overclaims. Please restate it under the exact hypotheses for which it is proved, and clearly separate the rigorous scalar/narrow case from numerical evidence for the general case.
minor comments (6)
  1. [Section 2, paragraph after the definition of gl-PŁI] The statement 'It can be shown that all strictly convex functions satisfy gl-PŁI' is false. For example, L(k)=k^4 on R is strictly convex but the ratio ‖∇L‖²/(L-L*) = 16k² tends to 0 as k→0, so no global λ exists. Please replace 'strictly convex' with 'strongly convex' or add a qualifying condition.
  2. [Section 3, Theorem] Even if the theorem is a standard corollary of known Lyapunov characterizations, the paper would be more self-contained with a brief justification or a pointer to the exact result. The current presentation leaves the reader to reconstruct the proof.
  3. [Section 4, reference [22]] The 'Imbalance Speed-up Theorem' and the gl-PŁI recovery result are attributed to [22], which is marked 'In preparation'. For a journal submission, please either update the reference if it has appeared, or explicitly state that these are preliminary results.
  4. [References, [12]] The author name 'T. C. I. Kolmanovsky' appears to be a typo; the intended author is likely I. (Ilya) Kolmanovsky. Please verify and correct.
  5. [Section 2, sat-PŁI discussion] The phrase 'global linear convergence' is used to describe the linear-in-time decrease of the loss under sat-PŁI, which differs from the optimization literature's 'linear convergence' meaning exponential decrease per iteration. A clarifying remark would prevent confusion.
  6. [Section 2, summary diagram] The diagram with vertical arrows from PŁ classes to K∞/K/PD is not fully labeled; it would be clearer to explicitly state that these arrows mean 'the gradient dominance estimate holds with an α of this class.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central sat-PŁI and strong-iISS claims rest on a published journal result [6] and a standard dissipation-inequality bridge theorem; Section 4's in-preparation citations are a support gap, not a circular reduction.

full rationale

Walking the claimed derivation chain, no load-bearing step is equivalent to its own input. Section 2's 'Theorem. [6] L admits a class sat-PŁI ... gradient dominance estimate' is explicitly attributed to Cui-Jiang-Sontag, Systems & Control Letters 188 (2024), a published, externally checkable article; the manuscript's footnote says it is a short summary of slides, so this is delegation to published work, not definitional circularity. Section 3's bridge theorem (K∞⇒ISS, K⇒siISS, PD⇒iISS when L−L* is a size function) is a direct dissipation-inequality argument: along kdot=−∇L^T+B(k)u one has ell_dot≤−1/2 α(ell)^2+c|u|^2, which is exactly the standard ISS/siISS/iISS Lyapunov dissipation form; the implication is not an assumption disguised as a conclusion. The LQR 'strongly iISS' theorem then follows from the cited sat-PŁI estimate plus this bridge, so the central content does not reduce to a fit or a renamed known result. The only flagged weakness is Section 4, where gl-PŁI on imbalance sets and the ISS gain property are stated with support in [8] (to appear) and [22] (in preparation, same research group); these are explicit sketches and are not used in the central CT-LQR/ISS derivation, so they are a missing-proof/support gap rather than circularity. The false motivating remark that all strictly convex functions satisfy gl-PŁI is incorrect without coercivity but is isolated and not used in any theorem, so it is a correctness risk, not circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 2 invented entities

The paper's central conclusions depend on a large body of prior work, most of it authored by the same group. The listed axioms are the unproved-in-this-paper properties of the LQR loss and the overparametrized dynamics that the survey takes as given, including coercivity of the loss on the stabilizing domain and the specific gradient lower bounds.

assumptions (5)
  • standard math Standard nonlinear convergence theorems apply: Łojasiewicz Gradient Theorem, Krasovskii-LaSalle invariance, and Center-Stable Manifold theorem.
    Used in Section 1 to conclude generic convergence to minimizers; no proofs are given in this paper.
  • domain assumption L(k) - L* is a proper, positive-definite size function on the stabilizing domain D.
    Required by the ISS Lyapunov theorem in Section 3; the paper asserts L is a size function but does not prove coercivity for general LQR instances.
  • domain assumption The CT LQR loss L admits a global sat-PŁI estimate with constants a,b > 0.
    Stated as Theorem [6] in Section 2; not derived in this paper, and it is load-bearing for the strongly-iISS corollary.
  • ad hoc to paper For the overparametrized LQR problem, the gradient norm lower bound ∥∇L∥² ≥ ρ min{∥k1∥²+∥k2∥²,1}(L-L*) holds.
    Introduced in Section 4 with 'One can show'; no proof or stable reference is provided, and it supports the recovery of gl-PŁI on imbalanced sets.
  • ad hoc to paper Solutions of the overparametrized gradient flow remain in forward-invariant uniformly imbalanced sets and satisfy the Imbalance Speed-up Theorem.
    Stated in Section 4; proofs are deferred to the in-preparation manuscript [22].
invented entities (2)
  • sat-PŁI (saturated Polyak-Łojasiewicz Inequality) class
    purpose: Characterize gradient dominance for CT LQR loss with linear-exponential convergence; used to derive ISS and iISS properties for perturbed flows.
    A new mathematical definition introduced in Section 2; no external falsifiable benchmark is given, and the existence for LQR is asserted via self-cited reference [6].
  • Imbalance measure c_i (eigenvalue concentration measure)
    purpose: Quantify overparametrization asymmetry in linear feedforward neural network feedback; predicts faster convergence and robustness for larger imbalance.
    Defined in Section 4 from constant matrices C_i; the speed-up and robustness predictions are the paper's own claims, with no independent empirical handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Some remarks on gradient dominance and LQR policy optimization." pith.science (2026). https://pith.science/paper/QFWJRI4G

@misc{pith2026250710452,
  author       = {Pith},
  title        = {Pith review of: Some remarks on gradient dominance and LQR policy optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QFWJRI4G}},
  note         = {Machine review of arXiv:2507.10452}
}
read the original abstract

Solutions of optimization problems, including policy optimization in reinforcement learning, typically rely upon some variant of gradient descent. There has been much recent work in the machine learning, control, and optimization communities applying the Polyak-{\L}ojasiewicz Inequality (PLI) to such problems in order to establish an exponential rate of convergence (a.k.a. ``linear convergence'' in the local-iteration language of numerical analysis) of loss functions to their minima under the gradient flow. Often, as is the case of policy iteration for the continuous-time LQR problem, this rate vanishes for large initial conditions, resulting in a mixed globally linear / locally exponential behavior. This is in sharp contrast with the discrete-time LQR problem, where there is global exponential convergence. That gap between CT and DT behaviors motivates the search for various generalized PLI-like conditions, and this talk will address that topic. Moreover, these generalizations are key to understanding the transient and asymptotic effects of errors in the estimation of the gradient, errors which might arise from adversarial attacks, wrong evaluation by an oracle, early stopping of a simulation, inaccurate and very approximate digital twins, stochastic computations (algorithm ``reproducibility''), or learning by sampling from limited data. We describe an ``input to state stability'' (ISS) analysis of this issue. The second part discusses convergence and PLI-like properties of ``linear feedforward neural networks'' in feedback control. Much of the work described here was done in collaboration with Arthur Castello B. de Oliveira, Leilei Cui, Zhong-Ping Jiang, and Milad Siami.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 19 canonical work pages

  1. [6]

    L. Cui, Z. Jiang, and E. D. Sontag. Small-disturbance input-to-state stability of perturbed gradient flows: Applications to LQR problem. Systems and Control Letters , 188:105804, 2024. ISSN 0167-6911. doi: https://doi.org/10.1016/j.sysconle.2024.105804

    arXiv 2024

  2. [8]

    de Olivera, M

    A. de Olivera, M. Siami, and E. Sontag. Convergence analysis of overparametrized LQR formula- tions. Automatica, 2025. To appear

    work page 2025

  3. [22]

    M. Wafi, A. de Olivera, and E. Sontag. On the (almost) global exponential convergence of overpa- rameterized policy optimization for the LQR problem, 2025. In preparation. 18

    work page 2025

  4. [19]

    E. Sontag. Input to state stability: Basic concepts and results. In P. Nistri and G. Stefani, editors, Nonlinear and Optimal Control Theory, pages 163–220. Springer-Verlag, Berlin, 2007

    work page 2007

  5. [20]

    E. Sontag. Remarks on input to state stability of perturbed gradient flows, motivated by model-free feedback control learning. Systems and Control Letters, 161:105138, 2022

    work page 2022

  6. [1]

    D. P. Bertsekas. Nonlinear Programming. Athena Scientific, Belmont, Massachusetts, second edition edition, 1999

    work page 1999

  7. [2]

    D. P. Bertsekas and J. N. Tsitsiklis. Gradient convergence in gradient methods with errors. SIAM Journal on Optimization, 10(3):627–642, 2000

    work page 2000

  8. [3]

    Online Optimization of Switched LTI Systems Using Continuous-Time and Hybrid Accelerated Gradient Flows

    G. Bianchin, J. Poveda, and E. Dall’Anese. Online optimization of switched LTI systems using continuous-time and hybrid accelerated gradient flows. arXiv 2008.03903, 2020

    work page Pith review arXiv 2008

Show all 24 references
  1. [4]

    J. Bu, A. Mesbahi, and M. Mesbahi. Policy gradient-based algorithms for continuous-time linear quadratic control. arXiv preprint arXiv:2006.09178, 2020

    2006 arXiv

  2. [5]

    Cherukuri, E

    A. Cherukuri, E. Mallada, S. Low, and J. Cortés. The role of convexity in saddle-point dynamics: Lyapunov function and robustness. IEEE Transactions on Automatic Control, 63(8):2449–2464,

  3. [7]

    de Olivera, M

    A. de Olivera, M. Siami, and E. Sontag. Remarks on the gradient training of linear neural network based feedback for the LQR problem. In Proc. 2024 63rd IEEE Conference on Decision and Control (CDC), pages 7846–7852, 2024

    2024

  4. [9]

    Fatkhullin and B

    I. Fatkhullin and B. Polyak. Optimizing static linear feedback: Gradient method. SIAM Journal on Control and Optimization , 59(5):3887–3911, Jan. 2021. ISSN 0363-0129, 1095-7138. doi: 10.1137/20M1329858

    2021 doi

  5. [10]

    Fatkhullin, J

    I. Fatkhullin, J. Etesami, N. He, and N. Kiyavash. Sharp analysis of stochastic optimization under global kurdyka-łojasiewicz inequality. Advances in Neural Information Processing Systems , 36, 2022

    2022

  6. [11]

    Fazel, R

    M. Fazel, R. Ge, S. Kakade, and M. Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. In International conference on machine learning, pages 1467–1476. PMLR, 2018

    2018

  7. [12]

    T. C. I. Kolmanovsky. Input-to-state stability of a bilevel proximal gradient descent algorithm, 2022

    2022

  8. [13]

    Levine and M

    W. Levine and M. Athans. On the determination of the optimal constant output feedback gains for linear multivariable systems. IEEE Transactions on Automatic Control, 15(1):44–48, 1970. doi: 10.1109/TAC.1970.1099363

    1970

  9. [14]

    Mohammadi, A

    H. Mohammadi, A. Zare, M. Soltanolkotabi, and M. R. Jovanovic. Convergence and sample com- plexity of gradient methods for the model-free linear-quadratic regulator problem. IEEE Trans- actions on Automatic Control, 67(5):2435–2450, May 2022. ISSN 0018-9286, 1558-2523, 2334-

    2022

  10. [15]

    Panageas and G

    I. Panageas and G. Piliouras. Gradient descent only converges to minimizers: Non-isolated critical points and invariant regions. In C. H. Papadimitriou, editor, 8th Innovations in Theoretical Com- puter Science Conference (ITCS 2017), volume 67 of Leibniz International Proceed...

    2017 doi

  11. [16]

    B. Pang, T. Bian, and Z. P. Jiang. Robust policy iteration for continuous-time linear quadratic regulation. IEEE Transactions on Automatic Control, 67(1):504–511, 2022

    2022

  12. [17]

    B. T. Polyak. Introduction to Optimization. Optimization Software, Inc., Publications Division, New York, 1987

    1987

  13. [18]

    J. I. Poveda and M. Krsti ´c. Nonsmooth extremum seeking control with user-prescribed fixed- time convergence. IEEE Transactions on Automatic Control , 66(12):6156–6163, 2021. doi: 10.1109/TAC.2021.3063700

    2021

  14. [21]

    Suttner and S

    R. Suttner and S. Dashkovskiy. Robustness properties of a large-amplitude, high-frequency ex- tremum seeking control scheme. arXiv 2009.14676, 2021

    2009 arXiv

  15. [2018]

    doi: 10.1109/TAC.2017.2778689

    2017

  16. [3303]

    doi: 10.1109/TAC.2021.3087455. 17

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.