Pith. sign in

REVIEW 2 major objections 3 minor 31 references

Mirror descent for constrained stochastic control problems

T0 review · 2 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proves that continuous-time mirror descent over Markov controls converges with linear rate when the Hamiltonian is uniformly convex in the action variable, and exponentially when it is strongly convex relative to a Bregman…

desk verdict The core mirror-descent convergence framework is solid and novel, but the worked examples, especially the unregularized ball case, do not satisfy the paper's own assumptions, so the headline linear-rate claim is not justified without a fix. read the letter →

arxiv 2506.02564 v1 pith:DIHY66IJ submitted 2025-06-03 math.OC

Deven Sethi , David Šiška This is my paper
classification math.OC MSC 49K4593E2090C25
keywords mirrordescentstochasticcontrolMarkovcontrolspolicygradientmethodsBregmandivergenceHamiltonianconvexitySobolevregularityperformancedifferencelemma
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Mirror descent is a gradient method that performs updates in a dual space and maps back to the constraint set through a mirror map. This paper brings that method to finite-horizon stochastic control with bounded convex action spaces, where the map from Markov controls to the value function is generally non-convex. The authors prove that the continuous-time mirror flow is well posed, that the value function decreases along it, and that the value gap to optimality decays linearly if the Hamiltonian is uniformly convex in the action variable and exponentially if it is strongly convex relative to an appropriate Bregman divergence. The pointwise bounds are explicit: $(V^{u_S}-V^{u_*})(t,x)\le D(Z_0,Z_*)(t,x)/S$ in the linear case, and exponential decay with rate $\lambda/2$ in the strongly convex case. The two mechanisms that make this work are the performance difference lemma, which quantifies how convexity is lost, and Sobolev regularity estimates for the value functions that justify differentiating along the flow.

What carries the argument

The key objects are the continuous-time mirror flow $\frac{d}{ds}Z_s=-\nabla_a H(\cdot,\nabla V^{u_s},u_s)$ with $u_s=\nabla\psi^*(Z_s)$, the Lyapunov function $D(Z,Z')(t,x)=\mathbb{E}^{\nabla\psi^*(Z')}_{t,x}\left[\int_t^{T_{\mathcal O}}\left(\psi^*(Z)-\psi^*(Z')-\nabla\psi^*(Z')\cdot(Z-Z')\right)(r,X_r)\,dr\right]$, and the performance difference lemma (Lemma 4.2), which expresses the value gap between two controls as an expected integral of their Hamiltonian difference. The mirror map $\nabla\psi^*$ enforces the constraint by construction, and the chain rule for the value function is justified by Hadamard differentiability together with $W^{2,1}_q$ Sobolev estimates. These estimates make the differentiation along the flow legitimate.

What would settle it

In the $\tau=0$ ball example (Example 3.1), take initial data with $|N^\top z|>R$ so the optimal control is $a^*(t,x,z)=R N^\top z/|N^\top z|$, which lies on the boundary of the ball. Then no bounded $Z_*$ satisfies $\nabla\psi^*(Z_*)=u_*$, so the bound (17) cannot even be written down; computing the actual value gap along the mirror flow would show whether the linear decay persists or degenerates, thereby delineating the theorem's true scope.

Watch

Extended reading notes

Core claim

The central result, Theorem 2.11, states that along the mirror flow $\frac{d}{ds}Z_s=-\nabla_a H(\cdot,\nabla V^{u_s},u_s)$ with $u_s=\nabla\psi^*(Z_s)$, the value gap obeys $(V^{u_S}-V^{u_*})(t,x)\le D(Z_0,Z_*)(t,x)S^{-1}$ when Assumption 2.10 holds with $\lambda=0$, and $(V^{u_S}-V^{u_*})(t,x)\le \frac{\lambda}{2}D(Z_0,Z_*)(t,x)(e^{\lambda S/2}-1)^{-1}$ together with $D(Z_S,Z_*)(t,x)\le e^{-\lambda S/2}D(Z_0,Z_*)(t,x)$ when $\lambda>0$ and the Bregman duality condition $D_\psi(\nabla\psi^*(y),\nabla\psi^*(y'))=D_{\psi^*}(y,y')$ holds. Here $D$ is a path-dependent Bregman Lyapunov function evaluated under the optimal control's process, so the convergence is pointwise in $(t,x)$. The proof overcomes the non-convexity of the control-to-value map by applying the performance difference lemma to pass from a local differential inequality to an integrated value-gap bound.

Load-bearing premise

The proofs assume the optimal control can be represented as $u_*=\nabla\psi^*(Z_*)$ for a bounded dual variable $Z_*$, meaning the optimal policy must stay in the interior of the action set; whenever the optimal control touches the boundary, the Lyapunov argument has no $Z_*$ to converge to.

Editorial extensions

If this is right

  • For $\lambda=0$, the value gap at any fixed $(t,x)$ is bounded by $D(Z_0,Z_*)(t,x)/S$, so after $S$ units of mirror-flow time the suboptimality is $O(1/S)$ pointwise.
  • For $\lambda>0$ with the Bregman duality condition, the value gap decays as $\frac{\lambda}{2}D(Z_0,Z_*)/(e^{\lambda S/2}-1)$ and the Lyapunov distance $D(Z_S,Z_*)$ decays at rate $e^{-\lambda S/2}$, both exponential with rate $\lambda/2$.
  • Along any solution the value function $V^{u_s}(t,x)$ is non-increasing in $s$, so mirror descent never worsens the policy in value.
  • The flow admits a unique solution for any bounded measurable initial dual variable $Z_0$, so the method is well posed without imposing Lipschitz structure on the policy class.
  • The worked examples show the assumptions are satisfied for a quadratic control problem on a ball and for entropy-regularized relaxed control on a finite action set, where the mirror map is softmax.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mirror map's interiority restriction could be removed, a similar Lyapunov inequality would likely yield convergence rates depending on how close the flow stays to the boundary; the paper does not address boundary-touching optimal controls.
  • Discretizing the mirror flow would likely yield $O(1/\varepsilon)$ and $O(\log(1/\varepsilon))$ complexity statements, but the paper's analysis is continuous-time only.
  • The performance difference lemma argument is not tied to mirror descent specifically and may transfer to other policy-gradient-type updates; this is an editorial inference, not a claim the paper makes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper introduces a continuous-time mirror descent flow for approximating optimal Markov controls in finite-horizon stochastic control problems with a bounded, convex action space. The flow evolves a dual variable Z_s by the negative action-gradient of the Hamiltonian, with primal controls recovered as u_s = ∇ψ*(Z_s). The main results are well-posedness of the flow (Theorem 2.8), monotonicity of the value function along the flow (Theorem 2.9), and linear (λ=0) or exponential (λ>0) convergence of the value gap to the optimum under a uniform convexity/strong-convexity assumption on the Hamiltonian (Theorem 2.11). The proofs use a performance difference lemma, Hadamard differentiability of the map Z↦V^{∇ψ*(Z)}, Lyapunov arguments with a Bregman divergence, and Sobolev-space regularity estimates for the associated linear parabolic PDEs. Two examples are provided: a linear-quadratic problem with controls in a ball and an entropy-regularized relaxed control problem with controls in a simplex.

Significance. If the stated results hold in their claimed generality, the paper provides a genuinely useful policy-gradient-style method for continuous-time stochastic control with convex action constraints, going beyond open-loop controls and addressing the nonconvexity of the objective via the performance difference lemma. The technical machinery—Hadamard differentiability of the value function, careful W^{2,1}_q estimates, and a Lyapunov function based on Bregman divergence—is substantial and appears carefully derived under the structural assumptions. The linear and exponential rates are parameter-free in the sense that they do not rely on fitting or on quantities that contain the target result; the Lyapunov analysis is structural. However, the examples as stated do not satisfy the standing assumptions, and the main theorem does not cover the boundary-optimal-control cases presented in the paper's headline example, which materially narrows the paper's applicability as written.

major comments (2)
  1. [Assumptions 2.1, 2.6 and Examples 3.1–3.2] The examples claimed to fit the framework do not satisfy the standing assumptions. Assumption 2.1 requires int(dom(ψ)) = A. For the ball example, ψ(a) = -log(R^2 - |a|^2) has domain the open ball, so int(dom(ψ)) is the open ball, not the closed ball A. For the simplex example, dom(ψ) is the closed simplex, which has empty interior in R^p, so int(dom(ψ)) cannot equal A. Assumption 2.6 requires ∇ψ*(R^p) = A, but formula (20) gives |∇ψ*(y)| = R^2|y|/(1+√(1+R^2|y|^2)) < R for every y, so the image is the open ball, and the softmax map in Example 3.2 takes values in the interior of the simplex. The statement in Section 3.1 that "Assumptions 2.1 and 2.7 are immediate" is therefore incorrect, and the validation of Assumption 2.6 in Appendix B.1 does not establish the required equality.
  2. [Theorem 2.11 and Example 3.1 (τ=0)] The proof of Theorem 2.11 requires a dual variable Z* ∈ B_b(O_T;R^p) satisfying u* = ∇ψ*(Z*), used in the Lyapunov function (16) and in the chain-rule computation displayed after (36). In the unregularized ball example with τ=0, the explicit minimizer for |N^T z| > R is a*(t,x,z) = R N^T z / |N^T z|, which lies on the boundary of the closed ball. Since ∇ψ* maps into the open ball, no bounded measurable Z* can satisfy u* = ∇ψ*(Z*) on the set where |N^T ∇V*| > R. Consequently D(Z0,Z*) is not defined and the linear rate (17) is not justified for this example, even though the τ=0 case is precisely the unregularized problem emphasized in the abstract and Section 3. The paper neither states an interiority condition on optimal controls nor provides an approximation argument for boundary optima, so the main convergence theorem does not cover the paper's headline example as stated. One cannot simply reinterpret A as the open ball, because for τ=0 the Hamiltonian infimum for |N^T z| > R would then not be attained, violating Assumption 2.4.
minor comments (3)
  1. [Lemma 4.1] The statement of the Feynman-Kac formula appears to have a typo: for a solution h of ∂h/∂t + L^u h + F = 0 with h=0 on the parabolic boundary, the representation should read h(t,x) = E^u_{t,x}[∫_t^{T_O} F(r,X_r) dr], not h(r,X_r) in the integrand. The proof of Lemma 4.2 uses the correct version with F, so this is a presentational error, but it should be fixed.
  2. [Theorem 2.11] The duality condition stated for the exponential-convergence case is written as D_ψ(∇ψ*(y),∇ψ*(y')) = D_{ψ*}(y,y'), but the proof in (37) requires D_ψ(∇ψ*(Z*),∇ψ*(Z_s)) = D_{ψ*}(Z_s,Z*), i.e. the arguments on the right-hand side are swapped. Appendix B.2 proves the identity with swapped arguments. The theorem statement should be corrected to D_ψ(∇ψ*(y),∇ψ*(y')) = D_{ψ*}(y',y).
  3. [Theorem 2.9] The notation X_{s,t'} in equation (14) is ambiguous; it should be made explicit that for each s this is the solution of (6) with control u_s = ∇ψ*(Z_s) starting from the fixed (t,x) in the statement of the theorem.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the convergence rates are derived from the stated convexity assumptions, the performance difference lemma, and the constructed Lyapunov function, not from fitted quantities or load-bearing self-citations.

full rationale

The paper's central derivation chain is self-contained. Theorem 2.11 is proved from Assumption 2.10 (the pointwise convexity/strong-convexity condition on the Hamiltonian), Lemma 4.2 (performance difference), and the Lyapunov function D defined in (16). The proof of Theorem 2.11 obtains the linear rate (17) by integrating the differential inequality (37) and using monotonicity from Theorem 2.9; the exponential rates (18)-(19) follow by applying Grönwall's inequality to (38). These rates are consequences of the assumptions and the mirror flow dynamics (1), and they are not re-imported from any fitted parameter or from a quantity that already contains the target bound. The static mirror-descent analogy in the Introduction is used only as motivation, not as a proof ingredient. The paper references its own prior work [26] only in the Introduction and in the related-works discussion, as complementary context; no proof step invokes [26] or any other self-citation as a load-bearing premise. There is no 'uniqueness theorem' imported from the authors' prior work and no ansatz smuggled in via citation. The reader's concern about the mirror maps in the examples mapping onto the open ball or the relative interior of the simplex, and the resulting issue for boundary optimal controls in Example 3.1, is a correctness/scope concern about whether Assumption 2.6's surjectivity condition ∇ψ*(R^p)=A is satisfied in the examples; it is not an instance of circular reasoning, because even if the assumption is violated, the proof is not assuming the convergence result it claims to prove. Accordingly, the honest finding is that there is no significant circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central result rests on standard PDE and SDE regularity theory plus the assumed strong convexity of the Hamiltonian and the surjectivity of the mirror map. No free parameters are fitted to data. The interiority constraint on the action set is the main hidden assumption.

assumptions (5)
  • domain assumption Action set A is bounded and convex and int(dom ψ)=A, forcing A open.
    Assumption 2.1, Section 2. This underlies the mirror map surjectivity and the flow construction.
  • domain assumption There exists a measurable minimizer a*(t,x,z) of the Hamiltonian.
    Assumption 2.4, used in Theorem 2.5 for existence of optimal control and in the HJB solution.
  • ad hoc to paper Uniform convexity or strong convexity of the Hamiltonian in the action variable.
    Assumption 2.10 is the key structural condition for the convergence rates. It is verified in the examples but is an assumption, not derived.
  • standard math Sobolev regularity and embedding results for parabolic PDEs.
    Lemma 4.3 and embedding (41), taken from literature [30,6]. These estimates are load-bearing for the regularity of value functions along the flow.
  • ad hoc to paper Bregman duality condition D_ψ(∇ψ*(y),∇ψ*(y'))=D_{ψ*}(y',y) when λ>0.
    Required in Theorem 2.11 for the exponential rate; verified for the ball and simplex examples but not general.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mirror descent for constrained stochastic control problems." pith.science (2026). https://pith.science/paper/DIHY66IJ

@misc{pith2026250602564,
  author       = {Pith},
  title        = {Pith review of: Mirror descent for constrained stochastic control problems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DIHY66IJ}},
  note         = {Machine review of arXiv:2506.02564}
}
read the original abstract

Mirror descent is a well established tool for solving convex optimization problems with convex constraints. This article introduces continuous-time mirror descent dynamics for approximating optimal Markov controls for stochastic control problems with the action space being bounded and convex. We show that if the Hamiltonian is uniformly convex in its action variable then mirror descent converges linearly while if it is uniformly strongly convex relative to an appropriate Bregman divergence, then the mirror flow converges exponentially. The two fundamental difficulties that must be overcome to prove such results are: first, the inherent lack of convexity of the map from Markov controls to the corresponding value function. Second, maintaining sufficient regularity of the value function and the Markov controls along the mirror descent updates. The first issue is handled using the performance difference lemma, while the second using careful Sobolev space estimates for the solutions of the associated linear PDEs.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 23 canonical work pages

  1. [26]

    Sethi, D

    D. Sethi, D. S i s ka, and Y. Zhang , Entropy annealing for policy mirror descent in continuous time and space , arXiv preprint arXiv:2405.20250, (2024)

    arXiv 2024

  2. [1]

    Agarwal, S

    A. Agarwal, S. M. Kakade, J. D. Lee, and G. Mahajan , On the theory of policy gradient methods: Optimality, approximation, and distribution shift , Journal of Machine Learning Research, 22 (2021), pp. 1--76

    work page 2021

  3. [2]

    Ahmed, N

    Z. Ahmed, N. Le Roux, M. Norouzi, and D. Schuurmans , Understanding the impact of entropy on policy optimization , in Proceedings of the 36th International Conference on Machine Learning, K. Chaudhuri and R. Salakhutdinov, eds., vol. 97 of Proceedings of Machine Learning Research, PMLR, 09--15 Jun 2019, pp. 151--160

    work page 2019

  4. [3]

    J. F. Bonnans and A. Shapiro , Perturbation analysis of optimization problems , Springer Science & Business Media, 2013

    2013

  5. [4]

    Cayci, N

    S. Cayci, N. He, and R. Srikant , Convergence of entropy-regularized natural policy gradient with linear function approximation , SIAM Journal on Optimization, 34 (2024), pp. 2729--2755

    work page 2024

  6. [5]

    S. Cen, C. Cheng, Y. Chen, Y. Wei, and Y. Chi , Fast global convergence of natural policy gradient methods with entropy regularization , Operations Research, 70 (2022), pp. 2563--2578

    work page 2022

  7. [6]

    W. H. Fleming and R. W. Rishel , Deterministic and stochastic optimal control , vol. 1, Springer Science & Business Media, 2012

    2012

  8. [7]

    Giegrich, C

    M. Giegrich, C. Reisinger, and Y. Zhang , Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems , SIAM Journal on Control and Optimization, 62 (2024), pp. 1060--1092

    work page 2024

Show all 31 references
  1. [8]

    Gilbarg, N

    D. Gilbarg, N. S. Trudinger, D. Gilbarg, and N. Trudinger , Elliptic partial differential equations of second order , vol. 224, Springer, 1977

    1977

  2. [9]

    Han et al

    J. Han et al. , Deep learning approximation for stochastic control problems , arXiv preprint arXiv:1611.07422, (2016)

    2016 arXiv

  3. [10]

    Huang, Z

    Y.-J. Huang, Z. Wang, and Z. Zhou , Convergence of policy iteration for entropy-regularized stochastic control problems , SIAM Journal on Control and Optimization, 63 (2025), pp. 752--777

    2025

  4. [11]

    Jia and X

    Y. Jia and X. Y. Zhou , Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms , Journal of Machine Learning Research, 23 (2022), pp. 1--50

    2022

  5. [12]

    Jia and X

    Y. Jia and X. Y. Zhou , q-learning in continuous time , Journal of Machine Learning Research, 24 (2023), pp. 1--61

    2023

  6. [13]

    Ju and G

    C. Ju and G. Lan , Policy optimization over general state and action spaces , arXiv preprint arXiv:2211.16715, (2022)

    2022

  7. [14]

    Kerimkulov, J.-M

    B. Kerimkulov, J.-M. Leahy, D. S i s ka, L. Szpruch, and Y. Zhang , A F isher-- R ao gradient flow for entropy-regularised markov decision processes in P olish spaces , arXiv preprint arXiv:2310.02951, (2023)

    2023 arXiv

  8. [15]

    Kerimkulov, D

    B. Kerimkulov, D. S i s ka, . Szpruch, and Y. Zhang , Mirror descent for stochastic control problems with measure-valued controls , arXiv preprint arXiv:2401.01198, (2024)

    2024 arXiv

  9. [16]

    N. V. Krylov , Controlled diffusion processes , vol. 14, Springer, 2008

    2008

  10. [17]

    Lan , Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes , Mathematical programming, 198 (2023), pp

    G. Lan , Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes , Mathematical programming, 198 (2023), pp. 1059--1106

    2023

  11. [18]

    Leahy, B

    J.-M. Leahy, B. Kerimkulov, D. Siska, and L. Szpruch , Convergence of policy gradient for entropy regularized MDP s with neural network approximation in the mean-field regime , in Proceedings of the 39th International Conference on Machine Learning, K. Chaudhuri, S. Jegelka, L...

    2022

  12. [19]

    J. Mei, C. Xiao, C. Szepesvari, and D. Schuurmans , On the global convergence rates of softmax policy gradient methods , in International conference on machine learning, PMLR, 2020, pp. 6820--6829

    2020

  13. [20]

    Munos , Policy gradient in continuous time , Journal of Machine Learning Research, 7 (2006), pp

    R. Munos , Policy gradient in continuous time , Journal of Machine Learning Research, 7 (2006), pp. 771--791

    2006

  14. [21]

    A. S. Nemirovskij and D. B. Yudin , Problem complexity and method efficiency in optimization , Wiley-Interscience, 1983

    1983

  15. [22]

    Pham , Continuous-time stochastic control and optimization with financial applications , vol

    H. Pham , Continuous-time stochastic control and optimization with financial applications , vol. 61, Springer Science & Business Media, 2009

    2009

  16. [23]

    Reisinger, W

    C. Reisinger, W. Stockinger, and Y. Zhang , Linear convergence of a policy gradient method for some finite horizon continuous time control problems , SIAM Journal on Control and Optimization, 61 (2023), pp. 3526--3558

    2023

  17. [24]

    Rockafellar , Convex analysis , Princeton Mathematical Series, 28 (1970)

    R. Rockafellar , Convex analysis , Princeton Mathematical Series, 28 (1970)

    1970

  18. [25]

    Sethi and D

    D. Sethi and D. S i s ka , The modified MSA , a gradient flow and convergence , The Annals of Applied Probability, 34 (2024), pp. 4455--4492

    2024

  19. [27]

    S i s ka and

    D. S i s ka and . Szpruch , Gradient flows for regularized stochastic control problems , SIAM Journal on Control and Optimization, 62 (2024), pp. 2036--2070

    2024

  20. [28]

    D. W. Stroock and S. S. Varadhan , Multidimensional diffusion processes , Springer, 1997

    1997

  21. [29]

    R. S. Sutton, A. G. Barto, et al. , Reinforcement learning: An introduction , vol. 1, MIT press Cambridge, 1998

    1998

  22. [30]

    Z. Wu, J. Yin, and C. Wang , Elliptic and parabolic equations , World Scientific Publishing Company, 2006

    2006

  23. [31]

    W. Zhan, S. Cen, B. Huang, Y. Chen, J. D. Lee, and Y. Chi , Policy mirror descent for regularized reinforcement learning: A generalized framework with linear convergence , SIAM Journal on Optimization, 33 (2023), pp. 1061--1091

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.