Equivalence of stochastic and deterministic policy gradients

Emo Todorov

REVIEW 3 major objections 5 minor 14 references

Equivalence of stochastic and deterministic policy gradients

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proves that stochastic and deterministic policy gradients are identical in Gaussian-quadratic MDPs and gives a general construction turning any stochastic-policy MDP into an equivalent deterministic-policy MDP.

desk verdict The stochastic/deterministic policy gradient equivalence is real and cleanly proven for the quadratic-Gaussian family, but the abstract and natural-gradient claim are a bit stronger than the proven statements. read the letter →

arxiv 2505.23244 v2 pith:2M4SWOJV submitted 2025-05-29 cs.LG

Emo Todorov This is my paper

classification cs.LG

keywords policygradientsdeterministicgradientstochasticnaturalMarkovdecisionprocessescontinuouscontrolGaussiannoisesufficientstatistics

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that two commonly contrasted families of policy-gradient algorithms are, in a standard continuous-control setting, mathematically equivalent. In Markov decision processes with Gaussian control noise and quadratic control cost, the stochastic and deterministic policy gradients are the same, their natural gradients coincide after a natural redefinition of the deterministic metric, and the state value functions differ only by a constant; the state-control value functions are the only quantities that differ. The paper also proves a general conversion: any stochastic policy of the form $\pi(u|x,\theta)=\tilde{\pi}(u|\mu(x,\theta))$ can be replaced by a deterministic policy in a new MDP whose controls are $\eta=\mu(x,\theta)$, with identical value functions, performance, and gradients. If correct, this unifies the two algorithm families and supports approximating state value functions rather than state-control value functions.

What carries the argument

Two mathematical devices carry the argument. Stein's lemma—the integration-by-parts identity $\mathbb{E}_{y\sim\mathcal{N}(\mu,\Sigma)}[(y-\mu)g(y)] = \Sigma\,\mathbb{E}[\nabla g(y)]$—converts the stochastic policy gradient's score term into the derivative of the state-control value function, giving the stochastic gradient the same structural form as the deterministic gradient. The quadratic-cost identity $\mathbb{E}_{d\sim p_\Sigma}[\ell(x,\mu+d)] = \ell(x,\mu) + \operatorname{tr}(R\Sigma)$ makes the noise contribution to the cost a state-independent constant, so the state value functions differ only by a constant and the residual gradient terms cancel. For the general theorem, the mechanism is the factorization $\pi(u|x,\theta)=\tilde{\pi}(u|\mu(x,\theta))$: marginalizing the original cost and transition kernel over the noise density produces a new MDP whose deterministic control is exactly the sufficient statistic $\eta=\mu(x,\theta)$.

What would settle it

Take a one-dimensional nonlinear MDP satisfying the paper's conditions, e.g. $f(x,u)=x+u+\epsilon\sin(x+u)$, $q(x)=x^2$, $R=1$, $\Sigma=1$, and a parameterized $\mu(x,\theta)$; solve the Bellman equations exactly on a fine grid and compute the two gradients in (28). The theorem predicts $\nabla_\theta J_S=\nabla_\theta J_D$ exactly, so any discrepancy refutes the equivalence. To test the scope condition, repeat with $R(x)=x$ and constant $\Sigma$, where the paper predicts the equivalence can fail.

Watch

Extended reading notes

Core claim

The central claim is that the distinction between stochastic and deterministic policies in the quadratic-Gaussian MDP family is a distinction without a difference for policy-gradient purposes. Writing the dynamics as $x' = f(x, \mu(x,\theta)+d)$ with $d\sim\mathcal{N}(0,\Sigma)$, the paper shows that whether the noise is injected in the policy (stochastic case) or in the dynamics (deterministic case) leaves the discounted visitation density $\rho$, the policy gradient, and the natural-gradient metric unchanged, provided the deterministic metric is taken as $\tilde{F}_D = \mathbb{E}[\nabla_\theta\mu\,\Sigma^{-1}\nabla_\theta\mu^\top]$ instead of the unmodified metric. The state value functions satisfy $v_S = v_D + (1-\gamma)^{-1}\operatorname{tr}(R\Sigma)$, and the policy gradients satisfy $\nabla_\theta J_S = \nabla_\theta J_D$; the difference between $Q_S$ and $Q_D$ is exactly canceled by the difference in how the two gradient formulas evaluate the noise. The proof needs $\operatorname{tr}(R\Sigma)$ to be independent of state, or $R(x)=\alpha\Sigma(x)^{-1}$ in the state-dependent case. A separate, fully general theorem constructs, for any stochastic policy of the form $\pi(u|x,\theta)=\tilde{\pi}(u|\mu(x,\theta))$, a deterministic-policy MDP with control $\eta=\mu(x,\theta)$, cost $\ell_D(x,\eta)=\mathbb{E}_{u\sim\tilde{\pi}(\cdot|\eta)}[\ell_S(x,u)]$, and transition $p_D(x'|x,\eta)=\mathbb{E}_{u\sim\tilde{\pi}(\cdot|\eta)}[p_S(x'|x,u)]$; the two MDPs then have identical controlled transitions, state value functions, performance, and policy gradients.

Load-bearing premise

In the quadratic-Gaussian equivalence, everything hinges on the noise-to-cost constant $\operatorname{tr}(R\Sigma)$ being independent of state; if the control cost's curvature varies with state and the noise covariance does not scale as its inverse, the value-function offset becomes an unknown function of state and the equality of gradients is no longer proven.

Editorial extensions

If this is right

Exact policy gradients in the quadratic-Gaussian setting can be evaluated from the state value function alone, using either $K_S(x,u)=\Sigma^{-1}(u-\mu(x,\theta))v(f(x,u))$ or $K_D(x,u)=\nabla_u v(f(x,u))$, which agree in expectation.
Stochastic and deterministic natural policy-gradient methods can share a single corrected Fisher metric, so their update directions coincide.
The general conversion theorem transfers any algorithm designed for deterministic policies to stochastic-policy MDPs, and vice versa, without changing the objective.
Because the discounted visitation densities coincide, state-dependent exploration statistics and baselines built on them are identical across the two formulations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The exact equality of gradients in expectation suggests that practical differences between stochastic and deterministic policy-gradient algorithms come from estimator variance, exploration behavior, and function-approximation error rather than from the gradient direction itself; this is an inference, not a claim the paper tests.
The sufficient-statistics construction is likely to extend to other noise families with tractable score expectations, such as exponential-family policies, though the paper only works out the Gaussian case and mentions mixture-of-Gaussians extensions.
A concrete algorithmic prediction follows: a model-based policy-gradient learner that approximates only $v(x)$ should reproduce the common gradient with less approximation error than one that approximates $Q(x,u)$, since the state value function has lower dimension; benchmark comparisons would test this.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This paper delivers a genuine result: for Gaussian control noise with fixed covariance and quadratic control costs, the stochastic and deterministic policy gradients are exactly equal, with state value functions matching up to a constant. The proof is straightforward and checks out; the Stein's lemma step and the cancellation argument are sound. The delta-function derivation of the deterministic policy gradient in Section 2.1 is a nice pedagogical touch and not something I've seen laid out that cleanly. The authors also cite the relevant prior work (Expected Policy Gradients, Wasserstein Policy Optimization) and correctly note that earlier results needed extra assumptions like a quadratic Q or state-independent policies. So the claimed novelty is real: the fixed-noise, general-dynamics equivalence appears to be new.

The paper is also honest about its own limits. In Section 3.2 it explicitly says that if tr(RΣ) is not state-independent, the value functions differ by an unknown state-dependent function and equivalence no longer holds. The general construction in Section 4 is largely definitional — the D-MDP cost and transition are just the marginalized versions of the S-MDP, so equivalence holds by construction. That section is more of a re-parameterization than a deep theorem, though it does usefully unify the notation.

The soft spots are mostly presentation. The abstract advertises equivalence for 'Gaussian control noise and quadratic control costs' without stating the constant-trace condition, which is a real omission for anyone trying to apply the result with state-dependent covariance or cost Hessians. The natural gradient claim also needs the catch: it holds only for the modified Fisher metric F̃_D, not for the standard deterministic Fisher matrix. Both are fixable by a more careful abstract and a clarifying sentence. The algorithmic suggestion — approximate v instead of Q — is left untested, which is fine for a theory paper but should not be read as an empirical contribution.

Overall, this is a solid theory paper with a correct central proof and a clear, honest discussion of its boundaries. It deserves serious refereeing. I would recommend acceptance after minor revisions, primarily to align the abstract with the proven conditions and to state the regularity assumptions for Stein's lemma explicitly.

Referee Report

3 major / 5 minor

Summary. The paper studies the relationship between stochastic and deterministic policy gradients in continuous MDPs. In Section 3 it specializes to MDPs with Gaussian control noise and quadratic control cost, and proves that the two policy gradients are equal (Eq. 36), that the state value functions differ by a constant (Eq. 31), and that the natural-gradient correction matrices coincide after redefining the deterministic Fisher metric with the inverse noise covariance as a control-space metric (Eqs. 21-23). Section 4 presents a general construction that converts any stochastic-policy MDP into a deterministic-policy MDP by marginalizing the cost and transition probability over the policy noise (Theorem 1). The paper also sketches learning algorithms based on state-value approximation rather than state-action-value approximation.

Significance. If the special-case equivalence holds under the stated assumptions, it is an interesting and potentially impactful unification: it suggests that in a common class of continuous-control MDPs, stochastic and deterministic policy gradients coincide, and that algorithms may be built on state-value rather than state-action-value approximation. The derivation using Stein's lemma and the constant-offset argument is elegant and mostly rigorous. The general construction in Theorem 1 is formally correct but is essentially a reparameterization rather than a substantive equivalence between independently defined MDP families. The main weakness is that the abstract and parts of the summary overstate the proven scope of the equivalence.

major comments (3)

[Abstract and Section 3.2] The abstract claims that 'in a widely-used family of MDPs involving Gaussian control noise and quadratic control costs, the stochastic and deterministic policy gradients, natural gradients, and state value functions are identical' without stating the necessary condition that tr(RΣ) be state-independent. The proof of Eq. (36) in Section 3.2 requires v_S and v_D to differ by a constant, which is shown only when tr(RΣ) is constant (Eqs. 30-31); for state-dependent cost Hessians the paper itself notes that one must impose R(x)=αΣ(x)^{-1} with constant α, otherwise 'we can no longer show equivalence.' This condition should appear in the abstract and in the Section 3.3 summary, because as written a reader using, say, constant Σ and state-dependent R would be applying the theorem outside its proven regime.
[Section 3.1, Eqs. (21)-(23)] The claimed equality of natural gradients is only established for the modified deterministic metric \tilde F_D defined in Eq. (22), not for the standard deterministic Fisher-type matrix F_D in Eq. (10). The abstract's unqualified phrase 'natural gradients ... are identical' should be qualified to indicate that the deterministic natural gradient uses a control-space metric Σ^{-1}; otherwise the statement falsely suggests that the conventional DPG natural gradient equals the stochastic one. The distinction between F_D and \tilde F_D should be made prominent in the summary of results.
[Section 4, Theorem 1, Eq. (47)] Theorem 1 constructs the equivalent deterministic MDP by defining ℓ_D and p_D as the marginalizations of ℓ_S and p_S under the policy noise (Eq. 47). With these definitions, the equality of controlled transition probabilities and value functions follows directly from the Bellman equation, making the result a reparameterization rather than a substantive equivalence between two independently given MDP families. The terminology 'equivalent MDPs' is acceptable, but the theorem statement should acknowledge that the D-MDP is defined from the S-MDP, so the equivalence is by construction; this would prevent overinterpretation of the generality claim.

minor comments (5)

[Section 2.1, Eq. (11)] The interchange of differentiation and integration in Eq. (11) is not explicitly justified; state the regularity conditions on π(u|x,θ) and Q(x,u,θ) that make the dominated convergence theorem applicable.
[Section 3.1, Eq. (27)] Stein's lemma is invoked without stating the required conditions on the function g(y). To make the application to QS(x,u) fully rigorous, specify that g must be weakly differentiable and that g and its gradient satisfy appropriate growth conditions for the integration-by-parts step.
[Section 3.1, Eq. (18)] The notation |∇_y f(x,y)| is ambiguous when f is vector-valued; clarify that it denotes the absolute value of the determinant of the Jacobian matrix of f with respect to y.
[Section 3.4, Eq. (44)] The baseline terms involve \hat v(f(x,μ)) and its derivatives; the notation ∇_{μμ}\hat v should be defined explicitly, as it appears to be the Hessian of \hat v with respect to the control argument, not with respect to the function parameters.
[References] Reference [5] contains a typo: 'Lineary-solvable' should be 'Linearly-solvable.'

Circularity Check

2 steps flagged · score 4.0 of 10

QG policy-gradient equality is genuinely derived, but the general MDP equivalence and natural-gradient equality are definitional re-labelings.

self definitional [Section 4, Theorem 1, Eqs. (46)-(48)]
"Let D-MDP denote the MDP with state x, control η, and step cost ℓ_D(x, η) and transition probability p_D(x′|x, η) defined as: ℓ_D(x, η)≡∫ π̃(u|η)ℓ_S(x,u)du, p_D(x′|x,η)≡∫ π̃(u|η)p_S(x′|x,u)du ... Then for all x′,x,θ, P_S(x′|x,θ)=P_D(x′|x,θ), v_S(x,θ)=v_D(x,θ)."

The D-MDP is not an independent object: its cost and transition are defined as the π̃-marginals of the S-MDP cost and transition. Substituting the factorized policy π_S(u|x,θ)=π̃(u|µ(x,θ)) into the S-MDP Bellman equation yields exactly the D-MDP Bellman equation with the deterministic policy η=µ(x,θ); uniqueness of the Bellman solution then returns v_S=v_D. Thus the equivalence is true by construction, a re-parameterization of the stochastic policy's sufficient statistics, rather than a derived property of a pre-existing D-MDP. The paper is explicit that it is constructing the D-MDP, so this is not a hidden fit, but the theorem should not be read as an independent derivation of equivalence.
self definitional [Section 3.1, Eqs. (21)-(23)]
"F_S(θ)=E_{x∼ρ,d∼pΣ}[∇θµ Σ^{-1} d d^T Σ^{-1} ∇θµ^T] = E_{x∼ρ}[∇θµ Σ^{-1} ∇θµ^T] ... Comparing the two expressions suggests that F_D should be modified, using Σ^{-1} as a metric in control space: ilde F_D(θ)≡E_{x∼ρ}[∇θµ Σ^{-1} ∇θµ^T]. With this modification, the natural gradient correction matrices are the same: F_S(θ)= ilde F_D(θ)."

The modified deterministic metric ilde F_D is introduced with an expression identical to the simplified form of F_S, so Eq. (23) is an identity by definition rather than a demonstrated equality between F_S and the original deterministic Fisher metric F_D of Eq. (10). The abstract's statement that stochastic and deterministic 'natural gradients ... are identical' holds only after this redefinition; the standard F_D is not shown to equal F_S. This is a definitional relabeling, not a substantive theorem, though it is a legitimate choice if one accepts the modified metric.

full rationale

The central Section 3.2 result, ∇_θ J_S(θ)=∇_θ J_D(θ), is genuinely derived: it follows from the quadratic-cost identity (30), the constant offset v_S=v_D+c (31), Stein's lemma, and the shift identity (35). This chain does not reduce to a fit or to the paper's own definitions. The general construction in Theorem 1 is openly a construction, and its equivalence is definitional, which lowers the evidentiary weight of that generality but does not undermine the QG gradient equality. The natural-gradient equality is likewise definitional because ilde F_D is chosen to equal F_S. There is no load-bearing self-citation: the reference to the author's Dynamical System Optimization [14] appears only as a contextual remark and does not support the proofs. The main caveat is scope, not circularity: the equivalence in Eq. (36) requires tr(RΣ) to be state-independent, or R(x)=αΣ(x)^{-1} when both are state-dependent, a condition that is stated in the paper but omitted from the abstract's unqualified claim.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the Gaussian/quadratic structure and on smoothness of the dynamics; no parameters are fitted to data. The general construction uses the Bellman equation's uniqueness and defines the D-MDP to match the S-MDP's marginals.

assumptions (4)

domain assumption The dynamics f and cost ℓ are smooth enough for Stein's lemma and for interchanging differentiation and integration.
Invoked in Section 2.1 Eq. (11) and Section 3.1 Eqs. (26)-(28); the paper says only 'mild regularity assumptions'.
domain assumption Control noise is Gaussian with covariance Σ and the control cost is quadratic with symmetric R.
This is the defining special case of Section 3, stated in Eqs. (14) and (29).
domain assumption In the deterministic formulation the transition density p_D exists and has the form of Eq. (18), requiring f to be smooth and one-to-one on feasible states.
Section 3.1 Eq. (18); this justifies applying the deterministic policy gradient theorem to the (D) problem.
standard math The Bellman equation has a unique solution for the value function.
Used in the proof of Theorem 1, Section 4, to conclude v_S = v_D from identical Bellman equations.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Equivalence of stochastic and deterministic policy gradients." pith.science (2026). https://pith.science/paper/2M4SWOJV

@misc{pith2026250523244,
  author       = {Pith},
  title        = {Pith review of: Equivalence of stochastic and deterministic policy gradients},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2M4SWOJV}},
  note         = {Machine review of arXiv:2505.23244}
}

read the original abstract

Policy gradients in continuous control have been derived for both stochastic and deterministic policies. Here we study the relationship between the two. In a widely-used family of MDPs involving Gaussian control noise and quadratic control costs, we show that the stochastic and deterministic policy gradients, natural gradients, and state value functions are identical; while the state-control value functions are different. We then develop a general procedure for constructing an MDP with deterministic policy that is equivalent to a given MDP with stochastic policy. The controls of this new MDP are the sufficient statistics of the stochastic policy in the original MDP. Our results suggest that policy gradient methods can be unified by approximating state value functions rather than state-control value functions.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 6 canonical work pages

[1]

Williams, Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learn- ing

R. Williams, Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learn- ing. Machine Learning, 1992

1992
[2]

Amari, Natural Gradient Works Efficiently in Learning

S. Amari, Natural Gradient Works Efficiently in Learning. Neural Computation, 1998

1998
[3]

Sutton, D

R. Sutton, D. McAllester, S. Singh and Y. Mansour, Policy Gradient Methods for Reinforcement Learning and Function Approximation. NIPS, 1999

1999
[4]

Kakade, A Natural Policy Gradient

S. Kakade, A Natural Policy Gradient. NIPS, 2001

2001
[5]

Todorov, Lineary-solvable Markov decision problems

E. Todorov, Lineary-solvable Markov decision problems. NIPS, 2006

work page 2006
[6]

Peters and S

J. Peters and S. Schaal, Natural Actor-Critic. Neurocomputing, 2008

2008
[7]

Silver, G

D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra and M. Riedmiller, Deterministic Policy Gradient Algorithms. ICML, 2014

2014
[8]

Todorov, Convex and analytically-invertible dynamics with contacts and constraints: Theory and implementation in MuJoCo

E. Todorov, Convex and analytically-invertible dynamics with contacts and constraints: Theory and implementation in MuJoCo. ICRA, 2014

work page 2014

Show all 14 references

[9]

Sutton and A

R. Sutton and A. Brato, Reinforcement Learning: An Introduction (2nd ed.) MIT Press, 2018

2018
[10]

Ciosek and S

K. Ciosek and S. Whiteson, Expected Policy Gradients for Reinforcement Learning. JMLR, 2020

2020
[11]

Szepesv´ ari, Policy gradients

C. Szepesv´ ari, Policy gradients. In Planning in MDPs (ch. 16) rltheory.github.io, 2022

2022
[12]

Bhandari and D

J. Bhandari and D. Russo, Global Optimality Guarantees for Policy Gradient Methods. Operations Research, 2024

2024
[13]

D. Pfau, I. Davies, D. Borsa, J. Araujo, B. Tracey and H. van Hasselt, Wasserstein Policy Optimization. ICML 2025

2025
[14]

Todorov, Dynamical System Optimization

E. Todorov, Dynamical System Optimization. arXiv, 2025. 10

2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Williams, Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learn- ing

R. Williams, Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learn- ing. Machine Learning, 1992

1992

[2] [2]

Amari, Natural Gradient Works Efficiently in Learning

S. Amari, Natural Gradient Works Efficiently in Learning. Neural Computation, 1998

1998

[3] [3]

Sutton, D

R. Sutton, D. McAllester, S. Singh and Y. Mansour, Policy Gradient Methods for Reinforcement Learning and Function Approximation. NIPS, 1999

1999

[4] [4]

Kakade, A Natural Policy Gradient

S. Kakade, A Natural Policy Gradient. NIPS, 2001

2001

[5] [5]

Todorov, Lineary-solvable Markov decision problems

E. Todorov, Lineary-solvable Markov decision problems. NIPS, 2006

work page 2006

[6] [6]

Peters and S

J. Peters and S. Schaal, Natural Actor-Critic. Neurocomputing, 2008

2008

[7] [7]

Silver, G

D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra and M. Riedmiller, Deterministic Policy Gradient Algorithms. ICML, 2014

2014

[8] [8]

Todorov, Convex and analytically-invertible dynamics with contacts and constraints: Theory and implementation in MuJoCo

E. Todorov, Convex and analytically-invertible dynamics with contacts and constraints: Theory and implementation in MuJoCo. ICRA, 2014

work page 2014