Convergence of Proximal Policy Gradient Method for Problems with Control Dependent Diffusion Coefficients

Ashley Davey; Harry Zheng

REVIEW 3 major objections 4 minor 15 references

Convergence of Proximal Policy Gradient Method for Problems with Control Dependent Diffusion Coefficients

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The proximal policy gradient method converges linearly for stochastic control problems with control in both drift and diffusion, provided either the running or terminal cost is strongly convex.

desk verdict Genuinely new adjoint-operator approach to PPGM with control-dependent diffusion, but the central contraction proof has a missing martingale integrability condition that needs fixing. read the letter →

arxiv 2505.18379 v1 pith:NMNULTSF submitted 2025-05-23 math.OC

Ashley Davey , Harry Zheng This is my paper

classification math.OC MSC 68Q2593E2049M0535C0565C30

keywords proximalpolicygradientstochasticcontrolcontrol-dependentdiffusionbackwarddifferentialequationlinearconvergencedeepneuralnetworkconstraintsH2

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proves that the proximal policy gradient method (PPGM) converges linearly for a class of constrained stochastic control problems where the control enters both the drift and the diffusion of the state process. The convergence is measured in the $H^2$ norm of square-integrable processes, and it holds when either the running cost or the terminal cost is strongly convex; the other terms may be non-convex and may have unbounded derivatives. The control iterates contract geometrically to a stationary point of the objective that respects the constraint set, with contraction factor $\hat{c} = 1 - \tau(\rho\mu - K)$. This matters because prior PPGM convergence results excluded control-dependent diffusion, where the extra $Z$ component of the adjoint backward stochastic differential equation is hard to control.

What carries the argument

The load-bearing object is the adjoint-operator representation of the BSDE terms. Starting from the fundamental matrix process $\Phi$ solving $d\Phi_s = A_s\Phi_s\,ds + C_s\Phi_s\,dW_s$, the paper defines operators $L_t$ and $L_{t,T}$ mapping controls to forward-state increments, and uses their adjoints $L^*_t$, $L^*_{t,T}$ to write $B^\top Y + D^\top Z$ without ever estimating the regularity of $Z$. The contraction argument then reduces to Lipschitz bounds on the adjoints plus strong-convexity inequalities from convex optimization, with the coercivity of the terminal map $L_{0,T}$ providing the extra contraction in the singular case where only the terminal cost is strongly convex.

What would settle it

Choose coefficients that satisfy Assumption 1 and simulate the PPGM updates in (9) while checking the lower bound $\|L_{0,T}u\|^2_{L^2} \ge \lambda\|u\|^2_{H^2}$ numerically; if successive iterates fail to contract in $H^2$ at the predicted rate $\hat{c} = 1 - \tau(\rho\mu - K)$ for any admissible $\tau$, the martingale assertion used to prove (29) is violated.

Watch

Extended reading notes

Core claim

The paper's central claim is Theorem 2.4: under the paper's Assumption 1, for sufficiently large convexity parameter $\mu>0$ and sufficiently small step size $\tau>0$ there exists $u^*\in U$ and $\hat{c}\in[0,1)$ such that $\|u^k-u^*\|_{H^2} \le \hat{c}^k \|u^0-u^*\|_{H^2}$, and $u^*$ is a stationary point of the objective subject to the control constraint. The proof works by representing the Hamiltonian derivative, which normally depends on the irregular process $Z^k$, through adjoint operators built from the linear state dynamics: $\partial_u H_t(X^k_t,u^k_t,Y^k_t,Z^k_t) = (L^*_{0,T}\nabla g(X^k_T))_t + (L^*_0 \partial_x f(X^k,u^k))_t + \partial_u f_t(X^k_t,u^k_t)$. With this representation the difference between successive iterates is a sum of Lipschitz terms plus a strong-convexity term, giving the contraction bound of Theorem 2.3 and hence the fixed-point convergence of Theorem 2.4.

Load-bearing premise

The proof relies on the stochastic integrals in the adjoint representation having zero expectation, a property that requires a product-integrability condition the paper does not verify for its own iterates; if that condition fails, the key formula and the lower bound that give the contraction break down.

Editorial extensions

If this is right

If Theorem 2.4 holds, PPGM attains a linear convergence rate in $H^2$ for a problem class previously open, giving an $\mathcal{O}(\log(1/\varepsilon))$ iteration count to reach accuracy $\varepsilon$.
Control-dependent diffusion is not merely tolerated: in the singular case it can supply convexity that removes the need for a strongly convex running cost.
The convergence guarantee applies to the control iterates themselves, not only to objective values, so the method identifies the optimal policy in a meaningful process norm.
The two numerical implementations, an ODE-based scheme for linear-quadratic problems and a deep-neural-network scheme for general problems, are presented as practical vehicles that inherit the convergence guarantee in the ideal setting, with approximation errors left for future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Extension: if the missing product-integrability check is supplied, the same adjoint representation could push the argument beyond linear state dynamics, since the representation, not the regularity of $Z$, is the heart of the proof.
Extension: the contraction factor $\hat{c}$ predicts a concrete stability trade-off: larger strong-convexity $\mu$ widens the admissible step-size range, so the numerical blow-up for small $r$ in the sensitivity experiment is consistent with the theory's sharp threshold behavior.
Extension: the singular-case coercivity condition $D^\top D - B^\top A^{-1}B \succeq \delta I$ resembles a convexity condition in $u$ for the variance of the state increment; testing whether the same condition appears as a sufficient condition in non-LQ problems would be a natural next step.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

The short version: this paper gives the first convergence guarantee for proximal policy gradient in continuous-time stochastic control with control in the diffusion, outside LQ. The main trick is an adjoint-operator representation of the BSDE gradient that avoids proving Z-regularity. That is a real idea, and the paper is worth engaging. But there is a technical gap in the martingale justification that the authors need to fix: the key identities in Lemma A.2 and the coercivity bound (29) drop Brownian stochastic integrals after claiming they have zero expectation, citing Zhang Problem 2.10.7. The required condition is square-integrability of the integrand, and here the integrand contains products like X^u Z^{η,ξ}. Knowing X∈S² and Z∈H² does not imply E∫|X|²|Z|²ds<∞, and the paper never proves it for the iterates. The same issue appears in the proof of (29), where d|L0u|² is integrated. Since these steps feed directly into the contraction estimate in Theorem 2.3 and hence Theorem 2.4, the main theorem is not established under the stated assumptions. This is a proof gap, not a counterexample; the LQ examples are Gaussian and don't expose it. It may be patchable by adding an explicit integrability condition on the iterates or by a localization argument with extra estimates.

What the paper does well: the problem choice is timely, the comparison to existing work (Reisinger et al., Kerimkulov et al., Zhou-Lu) is honest, and the adjoint-operator idea is genuinely new and likely useful even if the current proof needs repair. The numerical sections are extensive, with code, but the theory does not cover the approximation errors of the neural network implementation; the authors acknowledge this.

The citation pattern looks fine: the self-citation to Davey-Zheng is about the DC2BSDE scheme, not the theoretical claim. The main theorem does not rest on their own prior work.

Bottom line: I would send this to a serious referee. The gap is real and load-bearing, but the paper is substantive, and an expert will quickly tell whether the missing integrability can be supplied. I would not cite the convergence theorem in its present form, but I'd keep the adjoint representation in mind once the proof is tightened.

Referee Report

3 major / 4 minor

Summary. The paper studies a proximal policy gradient method (PPGM) for continuous-time stochastic control problems in which the control enters both the drift and the diffusion of a linear state process, with costs that need only be strongly convex in either the running or terminal term. The main theoretical result, Theorem 2.4, claims linear convergence in H2 norm of the control iterates to a stationary point under a sufficiently large strong-convexity parameter µ and a sufficiently small step size τ. The proof strategy is to represent the Hamiltonian gradient via adjoint operators of the linearized state dynamics, avoiding pointwise regularity of the BSDE component Z, and then to establish a contraction-type estimate for successive control differences. The paper also presents an ODE-based algorithm for the unconstrained LQ problem and a deep-learning implementation for the general constrained problem, together with numerical experiments in dimensions up to 100.

Significance. If the central convergence theorem is correct, this is a meaningful extension of the existing PPGM literature: it removes the assumption that the diffusion is independent of the control, which was a key limitation of earlier works such as Reisinger et al. (2023). The H2-space formulation and the adjoint-operator representation are promising tools for treating the Z component of the adjoint BSDE, and the sufficient conditions in Assumption 1 are stated in terms of model coefficients rather than abstract regularity of the value function. The numerical section is substantial: it includes non-convex costs, control constraints, dimension scaling, and public code, all of which support the practical claims. Overall, the contribution is potentially valuable, and the identified proof gaps appear to be repairable with standard estimates rather than fundamental obstructions.

major comments (3)

[Appendix A, Lemma A.2 and proof of (29)] The Brownian-motion terms in the Itô expansion are discarded with the justification that X^{x,u} ∈ S² and u,Z^{η,ξ} ∈ H², citing Zhang (2017, Problem 2.10.7). This is not directly justified: the integrands are products such as (X^{x,u})^T Z^{η,ξ}, and the cited condition for a stochastic integral to have zero mean is typically H ∈ L²(P×dt), which is not implied by the separate memberships. The gap is repairable: by Cauchy–Schwarz, E[(∫|X^T Z|²ds)^{1/2}] ≤ ||X||_{S²}||Z||_{H²} < ∞, so the quadratic variation of the stochastic integral has integrable square root and the integral is a true martingale by the BDG criterion; the same argument applies to the integral in d|L⁰u|² with integrand (L⁰u)^T(C L⁰u + D u). The proof should add this verification explicitly, since the representation (22) and the coercivity bound (29) both rely on it.
[Section 2, Theorem 2.4] The proof of Theorem 2.4 establishes only the successive-difference bound ∥u^{k+1}-u^k∥_{H²} ≤ ĉ∥u^k-u^{k-1}∥_{H²} from Theorem 2.3. This makes the sequence Cauchy and yields convergence, but it does not by itself give the claimed error estimate ∥u^k-u*∥_{H²} ≤ ĉ^k∥u⁰-u*∥_{H²}; the difference contraction would only give a bound of the form C ĉ^k∥u¹-u⁰∥_{H²}. To obtain the stated linear rate with the same constant ĉ, the iteration map T(u) = prox_U(u - τ ∂uH(X^u,u,Y^u,Z^u)) must be shown to be a contraction in H². The estimates already present in the proof of Theorem 2.3 can be rerun for arbitrary u,v ∈ U in place of u^k,u^{k-1}, so the fix is straightforward, but the contraction property of T should be stated and proved explicitly.
[Section 3.1, Theorem 3.1 and Eqs. (13)–(15)] There is a factor-of-two inconsistency between the LQ data as defined in (11) and the adjoint BSDE terminal condition. Since g(x) = x^T G x with symmetric G, the terminal condition of the adjoint BSDE (7) is Y_T = ∇g(X_T) = 2G X_T, so the coefficient a in the ansatz Y^{t,x,φ}_s = a_s X^{t,x,φ}_s must satisfy a_T = 2G (or G+G^T), not a_T = G as stated in Theorem 3.1. Alternatively, if a denotes the coefficient of the quadratic value function, then the update formula (15) is missing the factor 2 multiplying B^T_t a^k_t and D^T_t a^k_t(C_t + D_t α^k_t). Also, the proof says 'we propose v^{t,φ}_s(x̃)=a_s x̃', but the value function is scalar and cannot equal a vector a_s x̃; the ansatz should be for the adjoint process Y. These issues affect the correctness of Algorithm 1 and the associated numerical validation.

minor comments (4)

[Section 1.1] There is a typo: 'meaurable' should be 'measurable'. The special characters such as 'Fr´echet' and 'Itˆo' are also misencoded.
[Section 3.2.1] The discretized Y-update contains Y_{i+1} on the right-hand side, making the scheme implicit; the text should either specify how the implicit equation is solved or write the scheme in the standard explicit form with all drift terms evaluated at i.
[Section 3.2.2] The loss function G(ν, χ^k) contains a factor 1/(BN) but the batch-size symbol B is not defined; presumably it should be M or another declared quantity.
[Section 4.1.5, Example 4.3] The statement that 'we have µ = ∥R∥∞ = r' is only true under the convention that the strong-convexity parameter of the running cost is the smallest eigenvalue of R; specifying the relationship would avoid ambiguity.

Circularity Check

0 steps flagged · score 1.0 of 10

No circularity identified: the convergence proof rests on external BSDE/optimization results, and the only self-citation is a non-load-bearing numerical implementation reference.

full rationale

Walking the derivation chain, Theorem 2.4 is obtained from Theorem 2.3, which uses the stationary-point characterization of Reisinger et al. (2023, Theorem 3.10), Nesterov's strong-convexity inequalities, standard BSDE well-posedness and regularity from Zhang (2017), and the adjoint-operator representation in Lemma A.2. Lemma A.2 is derived by applying Itô's formula to the inner product of the forward and backward processes and using the adjoint of the linear operator L_t; this is a mathematical derivation, not a definition of the convergence rate in terms of itself. The coercivity bound (29) is likewise obtained from Itô's formula and the model coefficients, not from the desired contraction. No fitted parameter is renamed as a prediction, and the numerical LQ experiments are benchmarked against closed-form Riccati solutions. The only self-citation, Davey and Zheng (2022), appears in Section 3.2 to motivate the DC2BSDE-style deep learning implementation; it is not used in Assumption 1 or in the proofs of Theorems 2.3 and 2.4. The skeptical concern about Lemma A.2 and (29) is that the stochastic integrals may fail to be true martingales because the product integrability E∫|X|^2|Z|^2 ds < ∞ is not implied by X∈S^2 and Z∈H^2. That is a possible proof gap relative to the external result from Zhang (2017), but it is not circular: it does not reduce the theorem to an input assumption or to a self-citation. Therefore, the circularity burden is minimal.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The theoretical derivation is self-contained and introduces no fitted parameters. It relies on standard results in FBSDE theory and convex analysis, plus the explicit Assumption 1. The singular case adds verifiable conditions on the model matrices. The numerical experiments use hand-chosen hyperparameters and coefficient scalings, but these do not affect the central theorem.

free parameters (2)

Numerical hyperparameters (τ, δ, ρ, batch size, hidden units) = τ small, ρ=δ=0.01, M=50, d1=d2=10
Hand-chosen in Section 4 to achieve convergence and stability; they do not enter the theoretical guarantee.
Coefficient scaling factor n^{-1} = n^{-1}
Chosen by inspection in Section 4.1.5 to ensure the random coefficients satisfy the assumptions for each dimension n; it affects the numerical experiment but not the theoretical statement.

assumptions (6)

standard math Well-posedness and S2/H2 regularity of linear FBSDEs (Zhang 2017, Theorems 3.2.2 and 4.2.1)
Used throughout to guarantee existence, uniqueness, and bounds for (X^k, Y^k, Z^k) and the operator L.
standard math Martingale property of stochastic integrals with S2/H2 integrands (Zhang 2017, Problem 2.10.7)
Invoked in Lemma A.2 and in the Itô expansion for |L0u|^2 in (29) to drop martingale terms with zero expectation.
standard math Nesterov's inequalities for smooth strongly convex functions (Nesterov 2003, Theorem 2.1.12)
Used to lower bound the inner products involving ∂u f2 and ∇g.
standard math Stationary point characterization through prox of the indicator of a convex set (Reisinger et al. 2023, Theorem 3.10)
Connects the fixed point of the update to a stationary point of J.
domain assumption Assumption 1: sufficiently large μ strong convexity of f^2 (standard) or g with matrix conditions (singular)
This is the stated sufficient condition for the contraction; it is verifiable from the problem data.
domain assumption Linear state dynamics in SDE (1)
The adjoint operator representation and the coercivity result rely on linearity; the paper notes nonlinear dynamics are outside its scope.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Convergence of Proximal Policy Gradient Method for Problems with Control Dependent Diffusion Coefficients." pith.science (2026). https://pith.science/paper/NMNULTSF

@misc{pith2026250518379,
  author       = {Pith},
  title        = {Pith review of: Convergence of Proximal Policy Gradient Method for Problems with Control Dependent Diffusion Coefficients},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NMNULTSF}},
  note         = {Machine review of arXiv:2505.18379}
}

read the original abstract

We prove convergence of the proximal policy gradient method for a class of constrained stochastic control problems with control in both the drift and diffusion of the state process. The problem requires either the running or terminal cost to be strongly convex, but other terms may be non-convex. The inclusion of control-dependent diffusion introduces additional complexity in regularity analysis of the associated backward stochastic differential equation. We provide sufficient conditions under which the control iterates converge linearly to the optimal control, by deriving representations and estimates of solutions to the adjoint backward stochastic differential equations. We introduce numerical algorithms that implement this method using deep learning and ordinary differential equation based techniques. These approaches enable high accuracy and scalability for stochastic control problems in higher dimensions. We provide numerical examples to demonstrate the accuracy and validate the theoretical convergence guarantees of the algorithms.

Figures

Figures reproduced from arXiv: 2505.18379 by the authors.

**Figure 2.** Comparison between algorithm errors for the unconstrained LQ problem. [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗

**Figure 3.** Comparison between algorithm errors for the unconstrained LQ problem. [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Comparison between algorithm errors for the constrained LQ problem. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

**Figure 5.** Figure 5: Control and value errors and runtime of unconstrained PGM applied to LQ problem. [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

**Figure 6.** Figure 6: Algorithm results for non-strongly-convex running cost. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 14 canonical work pages

[1]

Acciaio, B., Backhoff-Veraguas, J., and Carmona, R. (2019). Extended mean field control problems: stochastic maximum principle and transport perspective. SIAM journal on Control and Optimization , 57(6):3666--3693

work page 2019
[2]

and Zheng, H

Davey, A. and Zheng, H. (2022). Deep learning for constrained utility maximisation. Methodology and Computing in Applied Probability , 24(2):661--692

work page 2022
[3]

Fazel, M., Ge, R., Kakade, S., and Mesbahi, M. (2018). Global convergence of policy gradient methods for the linear quadratic regulator. In International Conference on Machine Learning , pages 1467--1476. PMLR

work page 2018
[4]

Giegrich, M., Reisinger, C., and Zhang, Y. (2024). Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems. SIAM Journal on Control and Optimization , 62(2):1060--1092

work page 2024
[5]

Hambly, B., Xu, R., and Yang, H. (2023). Policy gradient methods find the Nash equilibrium in n-player general-sum linear-quadratic games. Journal of Machine Learning Research , 24(139):1--56

work page 2023
[6]

and Zhou, X

Hu, Y. and Zhou, X. Y. (2005). Constrained stochastic lq control with random coefficients, and application to portfolio selection. SIAM Journal on Control and Optimization , 44(2):444--466

work page 2005
[7]

Kerimkulov, B., S i s ka, D., and Szpruch, L. (2021). A modified MSA for stochastic control problems. Applied Mathematics & Optimization , 84(3):1--20

work page 2021
[8]

Y., Pinkus, A., and Schocken, S

Leshno, M., Lin, V. Y., Pinkus, A., and Schocken, S. (1993). Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural networks , 6(6):861--867

work page 1993

Show all 15 references

[9]

Nesterov, Y. (2003). Introductory lectures on convex optimization: A basic course , volume 87. Springer Science & Business Media

2003
[10]

Reisinger, C., Stockinger, W., and Zhang, Y. (2023). Linear convergence of a policy gradient method for some finite horizon continuous time control problems. SIAM Journal on Control and Optimization , 61(6):3526--3558

2023
[11]

Sun, J., Li, X., and Yong, J. (2016). Open-loop and closed-loop solvabilities for stochastic linear quadratic optimal control problems. SIAM Journal on Control and Optimization , 54(5):2274--2308

2016
[12]

Wang, H., Zariphopoulou, T., and Zhou, X. Y. (2020). Reinforcement learning in continuous time and space: A stochastic control approach. The Journal of Machine Learning Research , 21(1):8145--8178

2020
[13]

Wang, W., Han, J., Yang, Z., and Wang, Z. (2021). Global convergence of policy gradient for linear-quadratic mean-field control/game in continuous time. In International Conference on Machine Learning , pages 10772--10782. PMLR

2021
[14]

Zhang, J. (2017). Backward Stochastic Differential Equations . Springer

2017
[15]

and Lu, J

Zhou, M. and Lu, J. (2023). A policy gradient framework for stochastic optimal control problems with global convergence guarantee. arXiv:2302.05816

2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Acciaio, B., Backhoff-Veraguas, J., and Carmona, R. (2019). Extended mean field control problems: stochastic maximum principle and transport perspective. SIAM journal on Control and Optimization , 57(6):3666--3693

work page 2019

[2] [2]

and Zheng, H

Davey, A. and Zheng, H. (2022). Deep learning for constrained utility maximisation. Methodology and Computing in Applied Probability , 24(2):661--692

work page 2022

[3] [3]

Fazel, M., Ge, R., Kakade, S., and Mesbahi, M. (2018). Global convergence of policy gradient methods for the linear quadratic regulator. In International Conference on Machine Learning , pages 1467--1476. PMLR

work page 2018

[4] [4]

Giegrich, M., Reisinger, C., and Zhang, Y. (2024). Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems. SIAM Journal on Control and Optimization , 62(2):1060--1092

work page 2024

[5] [5]

Hambly, B., Xu, R., and Yang, H. (2023). Policy gradient methods find the Nash equilibrium in n-player general-sum linear-quadratic games. Journal of Machine Learning Research , 24(139):1--56

work page 2023

[6] [6]

and Zhou, X

Hu, Y. and Zhou, X. Y. (2005). Constrained stochastic lq control with random coefficients, and application to portfolio selection. SIAM Journal on Control and Optimization , 44(2):444--466

work page 2005

[7] [7]

Kerimkulov, B., S i s ka, D., and Szpruch, L. (2021). A modified MSA for stochastic control problems. Applied Mathematics & Optimization , 84(3):1--20

work page 2021

[8] [8]

Y., Pinkus, A., and Schocken, S

Leshno, M., Lin, V. Y., Pinkus, A., and Schocken, S. (1993). Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural networks , 6(6):861--867

work page 1993