Contraction-Aware Reinforcement Learning for Nonlinear Control with Statistical Robustness

Hiroyasu Tsukamoto; Huy T. Tran; Minjae Cho

Contraction-Aware Reinforcement Learning for Nonlinear Control with Statistical Robustness

T0 review · 5 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training a contraction metric alongside an actor-critic policy yields a path tracker that minimizes cumulative error and asymptotically converges to the reference, provided some contracting policy exists.

desk verdict Promising empirical RL+contraction method, but Theorem 1's proof is broken and the advertised certificate is unestablished. read the letter →

arxiv 2506.15700 v2 pith:GPAZI223 submitted 2025-05-28 cs.LG cs.AI

Minjae Cho , Hiroyasu Tsukamoto , Huy T. Tran This is my paper · ORCID

classification cs.LGcs.AI

keywords ContractiontheoryControlmetricsReinforcementlearningActor-criticPathtrackingIncrementalexponentialstabilityLearneddynamicsmodelsUncertainsystems

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Contraction actor-critic (CAC) claims to give reinforcement-learned path-tracking policies a contraction certificate: a guarantee that trajectories initialized close to a reference converge to it. The route is to train, alongside the policy, a contraction metric generator (CMG) that outputs a positive-definite Riemannian metric, then use that metric to define the reward the policy maximizes. The paper proves that if at least one contracting policy exists, the optimal policy for this reward converges asymptotically, $\lim_{t\to\infty} \|\delta x(t)\|_M^2 = 0$. In simulation, CAC has the lowest cumulative tracking error on three of four benchmarks and near-best on the fourth, and a TurtleBot3 deployment transfers the learned behavior to hardware. The contribution is a way to fold a stability certificate from contraction theory into model-free RL without solving a convex feasibility problem.

What carries the argument

The central object is the contraction metric $M(x) \succ 0$, a positive-definite Riemannian metric under which the differential Lyapunov function $V = \delta x^\top M \delta x$ decays at a prescribed rate. The contraction metric generator is a neural network trained by the loss in Equation (9) to output a distribution over such metrics whose samples approximately satisfy the contraction and CCM conditions in Equations (3)–(5), evaluated with a learned dynamics model from Equation (8). The reward $R(x) = 1/(1 + \delta x^\top M \delta x) + \beta_\pi H(\pi_\theta(x))$ converts the metric into an RL signal, so maximizing it is equivalent to minimizing a bounded monotone function of the metric-weighted tracking error. A freeze-and-learn loop alternates CMG updates with $n$ policy updates, and the load-bearing estimate is the geometric-series bound of Lemma 1: $\|\delta x(t_k)\|_M^2 \le \|\delta x(t_0)\|_M^2 e^{-2\lambda k \Delta t}$ under a contracting policy.

What would settle it

Run CAC on a linear system whose optimal LQR controller is provably contracting, then measure the closed-loop squared metric distance $\|\delta x(t)\|_M^2$ along a reference over many initial offsets. If the trained policy's error does not decay exponentially at a positive rate for all tested references, the theorem's conclusion is not realized by the implemented algorithm.

Watch

Extended reading notes

Core claim

The central claim is that simultaneously learning a CMG and an actor-critic policy yields a policy that minimizes cumulative tracking error while inheriting a contraction certificate. Formally, Theorem 1 states that under Assumption 1, which postulates at least one contracting policy $\pi_c$ with rate $\lambda > 0$, the optimal policy $\pi^*$ obtained by maximizing the discounted reward $\tilde{R}(x) = 1/(1 + \|\delta x\|_M^2)$ satisfies $\lim_{t\to\infty} \|\delta x(t)\|_M^2 = 0$ for every initial condition. The proof chain is: Lemma 1 bounds the assumed contracting policy's cumulative metric distance by a geometric series, Lemma 2 equates reward maximization with cost minimization, and the resulting finite discounted cost implies convergence by Cauchy's criterion. Empirically, the paper reports that CAC achieves the lowest modified area-under-the-curve tracking error on PVTOL, NeuralLander, and Quadrotor, with Car comparable to the LQR baseline, and that a simulated-trained policy tracks a reference path on a real TurtleBot3 where C3M fails and PPO diverges.

Load-bearing premise

The proof assumes that the policy that maximizes the metric-weighted reward is itself one of the contracting policies for which the exponential-decay inequality was assumed, and the paper does not prove that the optimal policy inherits contraction.

Editorial extensions

If this is right

End-to-end training can replace the infinite-dimensional convex feasibility search in CCM synthesis, provided the learned dynamics model is accurate enough for the CMG loss to certify contraction.
Policies trained this way retain the per-step cost of a neural network, so the certificate does not impose the real-time optimization overhead of LQR-based certified controllers.
The reward-conditioned entropy term in the CMG loss is part of the method: ablating it degrades Car and NeuralLander tracking, suggesting the alternating freeze-and-learn schedule is needed for stable co-training.
Because the dynamics model is used only to evaluate the CCM conditions, the policy itself can be trained model-free, which is what lets CAC transfer from simulation to a TurtleBot3 where C3M fails and PPO diverges.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the theorem's proof transfers a contraction bound from an assumed contracting policy to the optimal policy, the cleanest test of the theory is to measure the trained policy's own contraction rate under the learned metric; if that rate is non-positive, the practical gains should be attributed to reward shaping rather than to the certificate.
The paper defines the tracking displacement in practice as a Euclidean difference, while the theoretical argument is stated along geodesics; replacing the Euclidean term with a geodesic or neural approximation of the metric distance would align the implementation with the proven statement.
The freeze-and-learn alternation is a generic recipe: any certificate that can be turned into a bounded reward, such as a control barrier function for safety, could be co-trained with a policy in the same way.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, here's my read. The interesting part of this paper is the combination: use a learned contraction metric generator to define a Lyapunov-style reward inside an actor-critic loop, with freeze-and-learn updates and reward-conditioned entropy. That specific design is not in [7] or [8], and it leads to a real empirical study: four simulated platforms plus a TurtleBot deployment, with baselines that are meaningfully compared. The performance story is plausible: CAC is best on PVTOL, NeuralLander, and Quadrotor, and comparable on Car, with far lower inference cost than SD-LQR. The authors are also honest that the learned dynamics are only used to evaluate the conditions and that real-world satisfaction is not guaranteed.

The paper's central formal claim, Theorem 1, does not hold as written. Lemma 1 has a sign error: for λ > 0 the denominator 1 - e^{2λΔt} is negative, so the claimed upper bound is negative. The correct geometric-series factor is 1 - e^{-2λΔt}. That invalidates the lemma as stated. More seriously, the proof of Theorem 1 applies Lemma 1 to an arbitrary policy π in Equation (23), but Lemma 1 is proved only for the assumed contracting policy πc. No argument connects the RL-optimal policy π* to that bound. The conclusion that a finite discounted sum implies lim_{t→∞} ||δx(t)||²_M = 0 also skips from sampled discrete terms to continuous time. The Limitations section concedes that conditions may not hold in practice, but the proof itself is the load-bearing issue and the paper doesn't flag it.

That said, the empirical method can still stand as a heuristic. The problem is that the abstract and conclusion sell the theorem as a guarantee, and the guarantee is not established. I would not rely on the theory in its current form, and I wouldn't cite it as a certificate.

Who should read this? Researchers working on RL with stability certificates will find the algorithmic idea worth discussing, and the empirical results are useful evidence that contraction-motivated rewards can help. The paper is also a good teaching example of a proof gap. A serious editor should send this to review—the empirical and algorithmic contributions deserve referee time—but the reviewers should insist on either a corrected proof of the transfer from πc to π*, or a re-framing of the theoretical claim as a heuristic rationale.

Referee Report

5 major / 7 minor

Summary. The manuscript proposes contraction actor-critic (CAC), an actor-critic variant in which a learned contraction metric generator (CMG) provides a reward signal based on a control contraction metric. The paper claims a theoretical guarantee (Theorem 1) that if at least one contracting policy exists, then the RL-optimal policy under the CAC reward converges asymptotically to the reference trajectory. Empirically, CAC is evaluated on four simulated platforms (Car, PVTOL, NeuralLander, Quadrotor) and on a real TurtleBot3, reporting lower modified AUC than C3M, PPO, SD-LQR, and LQR in several settings.

Significance. The empirical study is substantial and the paper is transparent about its limitations, including the online-interaction requirement and the gap between theoretical guarantees and practical condition satisfaction. If the convergence theorem were established, the paper would make a useful conceptual contribution by connecting CCM certificates to long-horizon RL objectives. The real-world robot experiments and the ablation of entropy regularization are valuable. However, the formal claim rests on a proof with a sign error, an unjustified transfer of a contraction bound to the RL-optimal policy, and a missing continuity argument; these issues are load-bearing rather than cosmetic. As it stands, the manuscript is best read as an empirical heuristic paper, not as a paper with a rigorous contraction certificate.

major comments (5)

[Appendix A.3, Eq. (23)] In Eq. (23), the proof applies Lemma 1's upper bound to Eπ[Σ γ^k C_k] for an arbitrary policy π, but Lemma 1's premise, Eq. (11), is stated in Assumption 1 only for the contracting policy π_c. No argument shows that the policy optimized by Eq. (14) is itself contracting or satisfies Eq. (11). The theorem's conclusion—that π* converges—is therefore imported from the assumption rather than derived; this is a circularity in the central claim.
[Lemma 1, Eq. (16)] The geometric-series evaluation in Eq. (16) is wrong in sign: since e^{-2λkΔt} with λ>0 and Δt>0, the sum is 1/(1-e^{-2λΔt}), not 1/(1-e^{2λΔt}). As written, the denominator is negative and Lemma 1 asserts a negative upper bound for a nonnegative quantity, which invalidates the bound used in Eq. (23).
[Appendix A.3, Eq. (25)] Eq. (24) only establishes that the infinite discounted cost sum is finite; the Cauchy convergence argument in Eq. (25) gives C_k → 0 along the sampled times t_k, and since C(x)=∥δx∥²_M/(1+∥δx∥²_M), it gives ∥δx(t_k)∥²_M → 0 along the sampling grid. To conclude lim_{t→∞}∥δx(t)∥²_M=0 in continuous time, the proof needs an additional argument (e.g., continuity of trajectories and a bound on inter-sample growth); none is provided.
[Section 3.2, Eq. (9)] The CMG loss in Eq. (9) evaluates the contraction and CCM conditions using the learned dynamics model f̂_ξ, B̂_ζ, and the Limitations section acknowledges that the guarantees may not hold during training or execution. The paper does not quantify how dynamics-model error propagates to the claimed contraction certificate, so the abstract's statement that CAC provides a contraction certificate is stronger than what is actually shown.
[Section 3.2 vs Section 4] Theorem 1 and Lemma 1 assume a fixed contraction metric M and reward R(x)=1/(1+δx^T M δx). In the actual algorithm, M is sampled from the CMG Mχ(x) and the CMG parameters χ are updated jointly with the policy (Algorithm 1, Eq. (9)). The objective in Eq. (14) is therefore non-stationary, and the theorem does not establish guarantees for the simultaneous-learning procedure described in the abstract.

minor comments (7)

[Eq. (10) and Table 2] Replace 'scaler' with 'scalar' in Eq. (10) and Table 2.
[Theorem 1] Theorem 1 states the contraction rate as α > 0, but Lemma 1 and the proof of Theorem 1 use λ; the notation should be unified.
[References] References [6] and [17] duplicate the same Singh et al. paper; the duplicate should be consolidated.
[Lemma 1] Lemma 1's statement uses J^{π_c}_T but its proof writes J^π_T; make the policy under which the expectation is taken explicit.
[Eq. (11)] The norm in Eq. (11) is defined through a geodesic integral, while Lemma 1 applies the bound at sampled instants; clarify how the continuous-time inequality transfers to the sampled norm.
[Table 1] The inference-time column does not state whether the CMG forward pass is included; specify this for reproducibility.
[Section 4 vs Algorithm 1] The theorem analyzes the reward with βπ=0, while Algorithm 1 and Table 2 use βπ=1e-2; the implications for the theoretical guarantee are not discussed.

Circularity Check

1 steps flagged · score 7.0 of 10

Theorem 1's proof invokes Lemma 1 for the RL-optimal policy without establishing the contraction premise (Eq. 11) for that policy, so the advertised convergence certificate is assumed rather than derived.

other [Section 4, Theorem 1 proof / Appendix A.3, Eq. (23), with Lemma 1 and Assumption 1]
"Eπ[Σ γ^k Ck] ≤ Eπ[Σ Ck] ≤ Eπ[Σ ∥δx(tk)∥^2_M] ≤ ∥δx(t0)∥^2_M / (1 − e^{2λΔt}), where the last inequality follows from Lemma 1. Assumption 1: 'There exists at least one contracting policy πc that ... satisfies ... ∥δx(t)∥^2_M ≤ ∥δx(t0)∥^2_M e^{−2λ(t−t0)}.'"

Lemma 1's bound is conditional on Eq. (11), which Assumption 1 postulates only for πc. Eq. (23) applies that bound to the policy π appearing in the cost/reward objective. If that π is π*, the proof is assuming the exponential-contraction inequality for π*—exactly the asymptotic-convergence conclusion of Theorem 1—so the theorem reduces to its premise. If the π in Eq. (23) is instead πc, the argument merely restates Lemma 1 and never transfers the finite-cost bound to π* via optimality; the subsequent Cauchy step then concludes convergence for a policy whose undiscounted error sum was never bounded. In either reading, the advertised certificate is assumed, not derived by the RL/CMG training.

full rationale

The only circularity that rises to the level of a load-bearing defect is in Theorem 1. Assumption 1 assumes a contracting policy πc satisfying Eq. (11). Lemma 1 then bounds J^{πc}_T using that inequality. In the proof of Theorem 1, Eq. (23) writes Eπ[Σ γ^k C_k] ≤ ... ≤ Eπ[Σ ||δx||^2_M] ≤ C/(1 − e^{2λΔt}) 'where the last inequality follows from Lemma 1.' This step silently transfers a premise that holds only for πc to the policy being optimized (π*), which is precisely the theorem's conclusion; alternatively it only re-establishes finiteness for πc and never transfers to π*. Either way the formal claim that the RL-optimal policy inherits a contraction certificate is not proved from the stated assumptions. Separately, Lemma 1's denominator 1 − e^{2λΔt} is negative for λ > 0 and Δt > 0; the correct geometric-series denominator is 1 − e^{−2λΔt}, an additional non-circular mathematical error. No self-citation is load-bearing here: refs [4, 5, 24] are background/tutorial citations by co-author Tsukamoto, but the theorem's defect is internal. The empirical comparisons (Table 1, real-robot tests) are independent evidence and could stand as heuristic claims. Score 7: the central theoretical result reduces to its own assumption; the experiments are not circular.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The algorithm depends on several fitted hyperparameters and on a learned dynamics model whose error is not quantified. The theoretical claim rests on an assumption (existence of a contracting policy) that duplicates the desired convergence property, and on unproven transfer of that property to the RL-optimized policy. No new physical entities are introduced; the CMG is an algorithmic construct without independent evidence.

free parameters (4)

Entropy scaler beta_M for CMG = 1e-2
Table 2; tunes the reward-conditioned entropy strength in Eq. (9); chosen empirically and affects exploration of the metric space.
Entropy scaler beta_pi for actor = 1e-2
Table 2; entropy weight in the reward Eq. (10); chosen empirically and shapes stochastic policy exploration.
CMG metric bounds = 0.1 to 10.0
Table 2; bounds learned metric eigenvalues; chosen by hand and constrains the feasible metric set.
Dynamics model weights (xi, zeta) = trained, no public checkpoints
Eq. (8); the CMG contraction conditions are evaluated on this learned model, so the certificate's validity depends on these fitted weights.

assumptions (5)

ad hoc to paper There exists at least one contracting policy pi_c satisfying Eq. (11) with rate lambda > 0.
Section 4, Assumption 1. This is the theorem's premise and essentially contains the target convergence property; it is not proved or checked in the experiments.
domain assumption System dynamics are control-affine with smooth drift f and actuation matrix B.
Section 2, Eq. (1). All theoretical and algorithmic components rely on this structure.
domain assumption The learned dynamics model accurately represents the true system so CCM inequalities (3)-(5) evaluated on it transfer to the real system.
Sections 3.1 and 3.2. The CMG loss and reward use approximated dynamics; the Limitations section admits guarantees may not hold during execution.
standard math The discount factor gamma is strictly less than 1 so infinite discounted sums are finite.
Lemma 2 and Theorem 1 use gamma in (0,1] but Eq. (20) requires finiteness; gamma=1 invalidates the separation of the sum.
domain assumption The null-space basis B_perp computed by SVD of the learned B has the correct rank and span.
Section 3.1. If the learned B has the wrong rank, the CCM conditions in Eqs. (4)-(5) are computed with the wrong constraint space.

invented entities (1)

Learned contraction metric distribution M_chi(x), the CMG
purpose: Generates the metric used in the reward 1/(1 + delta_x^T M delta_x) and in the CCM condition loss; couples RL optimization to the contraction certificate.
Introduced purely as an algorithmic construct. It makes no falsifiable prediction outside the paper's own simulated and robot evaluations, and the theoretical guarantee depends on Assumption 1 rather than on properties of this generator.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Contraction-Aware Reinforcement Learning for Nonlinear Control with Statistical Robustness." pith.science (2026). https://pith.science/paper/GPAZI223

@misc{pith2026250615700,
  author       = {Pith},
  title        = {Pith review of: Contraction-Aware Reinforcement Learning for Nonlinear Control with Statistical Robustness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GPAZI223}},
  note         = {Machine review of arXiv:2506.15700}
}

read the original abstract

Control contraction metrics (CCMs)-defined by Riemannian metrics under which a closed-loop system is incrementally exponentially stable-offer a constructive framework for synthesizing contracting policies in nonlinear path-tracking problems. However, while the synthesized policies ensure pointwise satisfaction of the CCM conditions, they may not ensure long-term optimality (i.e., minimizing cumulative trajectory-level tracking error) over both transient and steady-state regimes. Furthermore, the myopic nature of these policies could also make them more susceptible to learning biases when approximate dynamics are used to formulate CCMs. To address these issues, we propose to integrate CCMs into reinforcement learning (RL). CCMs provide dynamics-informed feedback for learning a policy that has a stability guarantee-i.e., is contraction-aware-while RL provides a framework for minimizing cumulative tracking error under approximate dynamics. Given a pretrained dynamics model, our algorithm, contraction-aware RL (CARL), simultaneously learns to generate CCMs and optimize a policy for rewards defined by those CCMs. We demonstrate that CARL enhances path-tracking performance and is robust to errors in approximated dynamics compared to relevant baselines in both simulated and real-world robot experiments. We also provide theoretical rationale for integrating CCMs into RL. Our code is available at https://github.com/Mgineer117/CARL, and a video of our real-world robot experiments can be found at https://youtu.be/sOJ4hulbop0.