Dynamical System Optimization

Emo Todorov

REVIEW 2 major objections 6 minor 25 references

Dynamical System Optimization

T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper derives a unified policy-gradient formula for parametric Markov chains and shows that the known policy gradients for stochastic, deterministic, and linearly-solvable MDPs are all special cases of it.

desk verdict A sound unification of policy gradient theorems with genuinely new MDP equivalences, but the DSO Chain Iteration is an unsupported leap from first-order gradient matching. read the letter →

arxiv 2506.08340 v1 pith:ED2AA4LL submitted 2025-06-10 cs.LG

Emo Todorov This is my paper · ORCID

classification cs.LG MSC 68T0590C4093E35

keywords dynamicalsystemoptimizationMarkovchainpolicygradientlinearly-solvableMDPnaturalproximalZ-learningidentification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Once a policy is fixed, the paper argues, actions are no longer decision variables: they are intermediate quantities generated by the policy, so the controlled system is just a Markov chain $P(x'|x,\theta)$ with a step cost $L(x,\theta)$ sharing the same parameter vector. The paper defines a cumulative-cost objective $J(\theta)$ for this chain, proves an exact gradient formula for it, and shows that the classical policy-gradient theorems for stochastic, deterministic, and linearly-solvable MDPs are the same formula in disguise. It then builds surrogate objectives, natural gradients, Hessian estimates, proximal and off-chain learning methods on top of that gradient, and applies the identical machinery to behavioral cloning, system identification, mechanism design, and state-estimator learning. A sympathetic reader would care because the paper is trying to show that the core of reinforcement-learning policy optimization can be recovered from a simpler object—an autonomous dynamical system—without any action-value function.

What carries the argument

The central object is a parametric Markov chain $P(x'|x,\theta)$ with step cost $L(x,\theta)$ sharing parameters, governed by the action-free Bellman equation $V(x,\theta)=L(x,\theta)+\gamma\int P(x'|x,\theta)V(x',\theta)dx'$. The load-bearing identity is Theorem 1's gradient formula, obtained by differentiating the Bellman equation, unfolding the recursion, and averaging over the discounted visitation density $\rho(x,\theta)$; the transition gradient is rewritten as $P\nabla \ln P$ so the result becomes a sampleable expectation. Around that identity the paper constructs the surrogate $S(\theta,\alpha)$, a function over the tangent bundle whose $\alpha$-gradient at $\alpha=0$ equals the DSO gradient but which requires no derivatives of $V$, $\rho$, or $d$, and which enables chain iteration, proximal clipping, Hessian approximation, and a tangent-space interpretation of the optimization step.

What would settle it

Run DSO Chain Iteration on a small tabular chain with non-negative costs and a parameterization where $S(\theta,0)\gg J(\theta)$, computing exact $J$ by enumeration; if for some initial $\theta$ the true cost $J(\theta_{k+1})$ exceeds $J(\theta_k)$ on a non-negligible fraction of steps across several step sizes $\kappa$, then the unproven policy-iteration premise fails and the central algorithmic claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that for any parametric Markov chain $P(x'|x,\theta)$ with cost $L(x,\theta)$, the gradient of the cumulative-cost objective is $\nabla_\theta J(\theta) = \mathbb{E}_{x\sim\rho(\cdot,\theta)}[\nabla_\theta L(x,\theta) + \gamma\int \nabla_\theta P(x'|x,\theta) V(x',\theta) dx']$ in the episodic setting, with an analogous average-cost formula using the stationary density and $\gamma=1$, and that this unified quantity recovers the policy gradients of stochastic, deterministic, and linearly-solvable MDPs. The paper further claims that the surrogate $S(\theta,\alpha)=\mathbb{E}_{x\sim\rho}[L(x,\theta+\alpha)+\gamma\int P(x'|x,\theta+\alpha)V(x',\theta)dx']$ has the same gradient at $\alpha=0$, so optimizing $S$ over $\alpha$ yields a policy-iteration analog, and that the same machinery supports likelihood-ratio proximal updates, Fisher-metric natural gradients, Hessian estimation, and off-chain Z-learning. Finally, it claims that all of these tools transfer unchanged to problems beyond policy optimization—behavioral cloning, mechanism design, system identification, state-estimator learning, and tuning of generative models—because policy parameters and system parameters are treated uniformly.

Load-bearing premise

The load-bearing premise is that minimizing the surrogate $S(\theta,\alpha)$ over $\alpha$ produces a meaningful improvement step for the true objective $J$, even though the paper only proves first-order gradient matching and notes that $S$ is a very poor global approximation of $J$.

Editorial extensions

If this is right

Because stochastic, deterministic, and linearly-solvable MDP policy gradients are special cases of one identity, algorithms developed for any one of these families carry over directly to the others.
Since $\theta$ is shared between the chain and the cost, a single gradient estimator can jointly optimize a policy, a physics model, a measurement model, and a state estimator without changing the algorithm.
The surrogate $S$ gives a policy-iteration analog—DSO Chain Iteration—and a PPO-style clipped update in a setting that has no actions.
The LMDP construction transfers into DSO as off-chain Z-learning, giving a learning rule that samples under the baseline chain or the current chain rather than relying on action-value functions.
Data-driven costs for behavioral cloning, system identification, and sensor calibration can be added directly to the DSO objective, enabling data-augmented policy optimization in the same framework.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the gradient estimate works as claimed in practice, it could serve as a drop-in estimator for policy-gradient methods in settings where actions are high-dimensional but transitions are cheap, because the estimator needs only transition samples and per-state costs.
The S-MDP, D-MDP, and L-MDP equivalences suggest that entropy-regularized, KL-regularized, and deterministic-policy objectives may be interchangeable design choices rather than separate algorithmic families; one testable consequence is that an L-MDP-style KL cost added to a deterministic-policy DSO reproduces the behavior of a stochastic-policy S-MDP.
For generative-AI tuning, the paper formalizes an idea that is usually treated as a heuristic: treat model sampling as the chain and feedback as the cost, then optimize by the chain gradient; comparing this estimator's variance and step-size behavior to policy-gradient baselines on a small toy would test the paper's claim that it is conceptually closer than RL.
The Hessian symmetry relation in Section 4.6 hints at a way to obtain unbiased episodic second-order estimates through the surrogate, but the paper does not prove error bounds; a natural next step is to check surrogate-Hessian updates against exact Newton steps on small tabular chains.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the core math is in good shape. Theorems 1–3 give a clean, self-contained derivation of the policy gradient for parametric Markov chains, and the fact that it reduces to the known stochastic, deterministic, and linearly-solvable policy gradients is a genuine unification, not a repackaging. The L-MDP:D-MDP and the general S-MDP:D-MDP equivalences beyond the Gaussian case are new and interesting, and the proofs are straightforward enough to check. I think the paper deserves a serious referee.

Where it wobbles is the algorithmic section. DSO chain iteration (Eq. 63) is pitched as an analog of policy iteration, but Lemma 2 only proves that the surrogate has the same first-order gradient at α=0. The paper itself notes that S is a very poor global approximation of J and that S(θ,0) ≫ J(θ). That means Eq. (62) is only a local expansion; controlling the remainder when α* is chosen by global minimization of S is not established. There is no monotonicity or boundedness argument. So the chain iteration and anything built on it—proximal methods, second-order steps—rest on an unproven premise. The Hessian approximation in Eq. (75) is explicitly stated without error bounds. Those are not fatal to the paper's theoretical contributions, but they are load-bearing for the advertised algorithmic applicability, and they should be flagged.

The application sections (mechanism design, brain-body dynamics, generative AI tuning) are speculative and untested. The paper is entirely theoretical, and that's fine, but those sections read as a research agenda rather than results. I'd want the author to separate the solid derivations from the wish-list.

One more note: Lemma 4 is delegated to [7]. That's acceptable since it's published work, but it means the off-chain Z-learning part depends on prior results rather than being self-contained here.

Bottom line: it's a serious theory paper with novel equivalences and clean derivations. The chain iteration overreach needs either proof or explicit retreat to 'suggested heuristic'. I'd send it to peer review with a request for a major revision that disentangles the proven parts from the conjectural ones. Reading group? Probably worth one session.

Referee Report

2 major / 6 minor

Summary. The paper introduces Dynamical System Optimization (DSO), a formalism in which a parametric Markov chain P(x'|x,θ) and a parametric step cost L(x,θ) define a cumulative-cost objective J(θ). It maps several MDP families—stochastic, deterministic, maximum-entropy, proximal-regularized, and linearly-solvable—into DSO problems, derives gradient formulas for episodic, average-cost, and time-varying settings (Theorems 1–3), recovers known policy-gradient theorems as special cases, and proposes a surrogate objective S(θ,α) whose α-gradient at α=0 matches ∇θJ(θ). It then sketches analogs of variance reduction, proximal optimization, natural gradients, Hessian estimation, chain/policy iteration, and off-chain Z-learning, and lists broader applications such as mechanism design, system identification, behavioral cloning, and state-estimator learning. The paper is entirely theoretical and contains no experiments.

Significance. If the derivations are correct, the paper offers a clean unifying viewpoint: policy optimization in several MDP families reduces to differentiating a Markov-chain objective, and known policy-gradient theorems follow from one formula. Theorems 1–3 and Lemma 2 are derived explicitly and are straightforward to verify; the equivalence mappings in Section 2 are also explicit. A strength is the use of known policy-gradient theorems as sanity checks for the DSO gradient, and the honest acknowledgment that the surrogate S is a poor global approximation. The paper does not provide code, experiments, or convergence guarantees, so its value is conceptual and pedagogical; the algorithmic promises in Sections 4.2 and 4.6 go beyond what is proved.

major comments (2)

[§4.2, Eq. (63)] The DSO Chain Iteration α*k = argmin_α S(θ_k, α), θ_{k+1} = θ_k + α*k is presented as an analog of policy iteration, but no improvement or convergence guarantee is provided. Lemma 2 only establishes first-order gradient matching at α=0, and Eq. (62)'s o(∥α∥) remainder is uncontrolled when α* is chosen as a global minimizer of S. The paper itself notes (p. 11) that S(θ,0) ≫ J(θ) and that S is a very poor global approximation; because V(x′,θ) and ρ(·,θ) are held fixed while θ+α changes P and L, minimizing S can favor states with low old-parameter value without accounting for the change in V, so a decrease in S need not imply a decrease in J. This should be fixed by adding a monotonicity or convergence theorem under explicit conditions, or by demoting the method to a heuristic and removing the policy-iteration-analog claim.
[§4.6, Eq. (75)] The approximation ∇²θJ(θ) ≈ ∇²αS(θ,α)|α=0 is stated without any error bound, and the text explicitly says 'we are not able to provide error bounds.' The symmetry identity in Eq. (77) relates cross-derivatives of S to the Hessian of J, but it does not justify replacing the θθ second derivative of J with the αα second derivative of S. Given that the abstract claims DSO computes the same quantities as Hessians, the paper should either prove an error bound, identify a limiting regime in which the approximation becomes exact, or clearly state that Eq. (75) is only an ad hoc curvature heuristic.

minor comments (6)

[Corollary 1 and Eq. (4)] There are typographical slips: 'desnsity' should be 'density', and 'terminal stares' should be 'terminal states'.
[Reference [17]] The reference is listed as 'Sutton and Brato'; the correct name is Barto.
[Corollary 1] The phrase 'When γ <0' should be 'When γ <1' or similar, since the paper assumes γ ∈ (0,1).
[Equation (23)] The notation \bar p(·,x) in Eq. (23) is inconsistent with \bar p(·|x) used elsewhere; please unify.
[Algorithm 1] The pseudocode states 'while rollout not ended' but does not specify the termination rule; the prose describes terminal states, random termination with probability 1−γ, or predefined times, and this should be reflected in the pseudocode.
[Section 4.7] The Z-learning updates in Eq. (92) are given as targets but the sentence 'We omit the algorithmic details' leaves the supervised-learning loss and optimizer unspecified; please provide at least one concrete instantiation or explicitly mark this as future work.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the DSO gradient derivation is self-contained, the policy-gradient recoveries are sanity checks, and the few self-citations are not load-bearing for the central claim.

full rationale

The central derivation is self-contained. Theorem 1 is obtained by differentiating the Bellman equation (3) and unfolding the discounted recursion; it never assumes an MDP policy-gradient theorem. Section 3.2 then substitutes the definitions (12), (18), and (21) into Theorem 1 and explicitly labels the comparison a 'sanity check', so the known policy gradients are consistency checks rather than inputs. Lemma 1 is an embedding built by construction: PG and LG are defined as the action-averaged transition and cost, so the equality of value functions is the intended definition of the mapping, not a prediction derived from a target result. The only self-citations are to the author's earlier LMDP work: Lemma 4 delegates uniqueness proofs to [7], and the S-MDP/D-MDP equivalence references [25]; neither supports the paper's central gradient or equivalence-of-formula claims, and the [25] construction is re-derived in Section 2.2. Section 4.2's DSO Chain Iteration lacks a convergence or improvement guarantee and the paper itself warns that S is a 'very poor approximation' of J globally; that is a correctness/rigor gap, not circularity, because Lemma 2 only establishes gradient matching and no step assumes the conclusion it purports to derive. Overall, no claimed prediction reduces by construction to its input.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The framework adds no fitted constants. Its main inputs are the Markov chain and cost, which are problem definitions, not free parameters. The derivation relies on standard regularity and ergodicity assumptions plus one non-trivial factorization assumption for the S-MDP:D-MDP equivalence. Lemma 4 is imported from prior self-cited work.

assumptions (5)

domain assumption P(x'|x,θ) and L(x,θ) are sufficiently smooth in θ to interchange differentiation and integration.
Stated in Section 2: 'We will be changing the order of gradients and integration, requiring mild regularity conditions (which we assume.)'
standard math The Bellman equations (3), (8), (11) have unique solutions V(x,θ).
Invoked in Section 2.1 after Lemma 1 to conclude v=V from equality of Bellman equations; uniqueness is standard in discounted and ergodic settings.
domain assumption The Markov chain P(x'|x,θ) is ergodic in the average-cost setting, so the stationary density d(x,θ) exists and is unique.
Stated in Section 2: 'This setting is only applicable when the Markov chain is ergodic.'
ad hoc to paper Every stochastic policy π(a|x,θ) can be written as π̃(a|μ(x,θ)) for some deterministic bottleneck μ.
Used in Section 2.2 for the S-MDP:D-MDP equivalence. Trivially satisfiable with μ(x,θ)=(x,θ), but the useful cases assume a minimal sufficient statistic η.
domain assumption The optimal non-parametric chain characterization (Lemma 4) is accepted from prior LMDP literature.
Section 4.7 Lemma 4 proof states: 'Detailed proofs of this and related results (including uniqueness) are given in [7] and subsequent papers in the LMDP series.' Not reproven here.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Dynamical System Optimization." pith.science (2026). https://pith.science/paper/ED2AA4LL

@misc{pith2026250608340,
  author       = {Pith},
  title        = {Pith review of: Dynamical System Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ED2AA4LL}},
  note         = {Machine review of arXiv:2506.08340}
}

read the original abstract

We develop an optimization framework centered around a core idea: once a (parametric) policy is specified, control authority is transferred to the policy, resulting in an autonomous dynamical system. Thus we should be able to optimize policy parameters without further reference to controls or actions, and without directly using the machinery of approximate Dynamic Programming and Reinforcement Learning. Here we derive simpler algorithms at the autonomous system level, and show that they compute the same quantities as policy gradients and Hessians, natural gradients, proximal methods. Analogs to approximate policy iteration and off-policy learning are also available. Since policy parameters and other system parameters are treated uniformly, the same algorithms apply to behavioral cloning, mechanism design, system identification, learning of state estimators. Tuning of generative AI models is not only possible, but is conceptually closer to the present framework than to Reinforcement Learning.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 24 canonical work pages

[7]

Todorov, Linearly-solvable Markov decision problems

E. Todorov, Linearly-solvable Markov decision problems. NIPS, 2006

work page 2006
[1]

Williams, Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learn- ing

R. Williams, Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learn- ing. Machine Learning, 1992

work page 1992
[2]

Bertsekas and J

D. Bertsekas and J. Tsitsiklis, Neuro-Dynamic Programming. Athena Scientific, 1996

1996
[3]

Amari, Natural Gradient Works Efficiently in Learning

S. Amari, Natural Gradient Works Efficiently in Learning. Neural Computation, 1998

work page 1998
[4]

Sutton, D

R. Sutton, D. McAllester, S. Singh and Y. Mansour, Policy Gradient Methods for Reinforcement Learning and Function Approximation. NIPS, 1999

work page 1999
[5]

Kakade, A Natural Policy Gradient

S. Kakade, A Natural Policy Gradient. NIPS, 2001

work page 2001
[6]

Kakade and J

S. Kakade and J. Langford, Approximately optimal approximate reinforcement learning. ICML, 2002

work page 2002
[8]

Peters and S

J. Peters and S. Schaal, Natural Actor-Critic. Neurocomputing, 2008

work page 2008

Show all 25 references

[9]

Todorov, Policy gradients in linearly-solvable MDPs

E. Todorov, Policy gradients in linearly-solvable MDPs. NIPS, 2010

2010
[10]

Ziebart, Modeling purposeful adaptive behavior with the principle of maximum causal entropy

B. Ziebart, Modeling purposeful adaptive behavior with the principle of maximum causal entropy. PhD thesis, 2010

2010
[11]

Silver, G

D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra and M. Riedmiller, Deterministic Policy Gradient Algorithms. ICML, 2014

2014
[12]

Schulman, S

J. Schulman, S. Levine, P. Abbeel, M. Jordan and P. Moritz, Trust region policy optimization. ICML, 2015. 20

2015
[13]

Schulman, P

J. Schulman, P. Moritz, S. Levine, M. Jordan and P. Abbeel, High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR, 2016

2016
[14]

Schulman, F

J. Schulman, F. Wolski, P. Dhariwal, A. Radford and O. Klimov, Proximal Policy Optimization Algo- rithms. arXiv, 2017

2017
[15]

Haarnoja, H

T. Haarnoja, H. Tang, P. Abbeel and S. Levine. Reinforcement learning with deep energy-based policies. ICML, 2017

2017
[16]

Haarnoja, A

T. Haarnoja, A. Zhou, P. Abbeel and S. Levine, Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. arXiv, 2018

2018
[17]

Sutton and A

R. Sutton and A. Brato, Reinforcement Learning: An Introduction (2nd ed.) MIT Press, 2018

2018
[18]

Abdolmaleki, J

A. Abdolmaleki, J. Springenberg, Y. Tassa, R. Munos, N. Heess and M. Riedmiller, Maximum a posteriori policy optimisation. ICLR, 2018

2018
[19]

Rajeswaran, V

A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov and S. Levine, Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. RSS, 2018

2018
[20]

Ha and J

D. Ha and J. Schmidhuber, World Models. arXiv, 2018

2018
[21]

Z. Shen, H. Hassani, C. Mi, H. Qian and A. Ribeiro, Hessian Aided Policy Gradient. ICML, 2019

2019
[22]

Geist, B

M. Geist, B. Scherrer and O. Pietquin, A Theory of Regularized Markov Decision Processes. ICML, 2019

2019
[23]

Martens, New Insights and Perspectives on the Natural Gradient Method

J. Martens, New Insights and Perspectives on the Natural Gradient Method. JMLR, 2020

2020
[24]

Bhandari and D

J. Bhandari and D. Russo, Global Optimality Guarantees for Policy Gradient Methods. Operations Research, 2024

2024
[25]

Todorov, Equivalence of stochastic and deterministic policy gradients

E. Todorov, Equivalence of stochastic and deterministic policy gradients. arXiv, 2025. 21

2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [7]

Todorov, Linearly-solvable Markov decision problems

E. Todorov, Linearly-solvable Markov decision problems. NIPS, 2006

work page 2006

[2] [1]

Williams, Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learn- ing

R. Williams, Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learn- ing. Machine Learning, 1992

work page 1992

[3] [2]

Bertsekas and J

D. Bertsekas and J. Tsitsiklis, Neuro-Dynamic Programming. Athena Scientific, 1996

1996

[4] [3]

Amari, Natural Gradient Works Efficiently in Learning

S. Amari, Natural Gradient Works Efficiently in Learning. Neural Computation, 1998

work page 1998

[5] [4]

Sutton, D

R. Sutton, D. McAllester, S. Singh and Y. Mansour, Policy Gradient Methods for Reinforcement Learning and Function Approximation. NIPS, 1999

work page 1999

[6] [5]

Kakade, A Natural Policy Gradient

S. Kakade, A Natural Policy Gradient. NIPS, 2001

work page 2001

[7] [6]

Kakade and J

S. Kakade and J. Langford, Approximately optimal approximate reinforcement learning. ICML, 2002

work page 2002

[8] [8]

Peters and S

J. Peters and S. Schaal, Natural Actor-Critic. Neurocomputing, 2008

work page 2008