Pith. sign in

REVIEW 2 major objections 4 minor 40 references

Deterministic policies make continuous-time extended mean-field control learnable without stochastic kernels or known action dependence.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 07:42 UTC pith:OT7L34OL

load-bearing objection Solid, usable model-free DPG for continuous-time extended MFC via deterministic policies; the classical C^{1,2} regularity is the only real soft spot and is typical of the subfield. the 2 major comments →

arxiv 2607.11005 v1 pith:OT7L34OL submitted 2026-07-13 math.OC cs.AIcs.LGstat.ML

Actor-Critic Learning for Extended Mean Field Control with Deterministic Policies

Ziheng Cheng , Xin Guo , Huy\^en Pham , Yufei Zhang This is my paper
classification math.OC cs.AIcs.LGstat.ML MSC 49N8093E2068T0560H10
keywords extended mean field controldeterministic policy gradientMcKean-Vlasov dynamicsadvantage-rate functioncontinuous-time reinforcement learningactor-criticWasserstein space
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that continuous-time extended mean-field control—where both dynamics and rewards can depend on the joint law of states and controls—can be solved by model-free reinforcement learning once one restricts to deterministic feedback policies. Under that restriction the joint state–action law is simply the push-forward of the state law, so policy search reduces to ordinary parameter optimization rather than optimization over stochastic kernels. From a model-free sensitivity formula for parameterized McKean–Vlasov dynamics the authors derive a deterministic policy gradient expressed through an advantage-rate function on the Wasserstein space; they then refine it into local value and advantage-rate functions of state, action and joint law. The resulting martingale characterization is turned into a continuous-time deep deterministic policy-gradient algorithm that uses particle approximations, measure-dependent networks, temporal-difference updates and either action- or parameter-space exploration. Experiments on Cucker–Smale consensus and crowded liquidation confirm that the method converges stably even when the control distribution appears explicitly in the coefficients.

Core claim

Under natural regularity, the gradient of the value with respect to a deterministic-policy parameter equals the integral of the derivative of a local advantage-rate function that depends on state, action and the joint state–action law; this identity supplies a martingale characterization that can be learned model-free and yields a practical continuous-time deep DPG algorithm for extended mean-field control.

What carries the argument

The deterministic policy-gradient formula (Theorems 3.1–3.2): the policy gradient is expressed via both the ordinary action derivative and an L-derivative with respect to the control marginal of a local advantage-rate function that is identified by a martingale condition along observed trajectories.

Load-bearing premise

The lifted value function is assumed already to be a classical C1,2 solution on the Wasserstein space for every policy parameter, rather than being proved from the coefficients.

What would settle it

Run the CT-DDPG algorithm on the linear-quadratic Cucker–Smale or liquidation problem with known closed-form optimum; if the learned return systematically fails to approach the analytic optimum as particle number and episode count increase, the claimed gradient and martingale characterizations are false.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper develops a model-free continuous-time actor-critic framework for extended mean-field control (MFC), in which both dynamics and rewards may depend on the joint law of states and controls. Deterministic feedback policies are used so that the state-action measure is the push-forward of the state law, avoiding optimization over stochastic kernels. A model-free sensitivity formula for parameterized McKean-Vlasov dynamics (Theorems 2.1-2.3) yields a deterministic policy-gradient identity on the Wasserstein space (Theorem 3.1). This is refined via local value and advantage-rate functions of (state, action, joint law), producing a gradient that contains both ordinary action derivatives and L-derivatives with respect to the control marginal (Theorem 3.2). The local objects are characterized by a martingale condition that is turned into a continuous-time deep DPG algorithm (CT-DDPG, Algorithm 1) with particle approximations, measure-dependent networks, TD learning, and action- or parameter-space exploration. Numerical experiments on stochastic Cucker-Smale consensus and optimal liquidation with trade crowding illustrate efficiency and robustness, including settings with explicit control-distribution dependence.

Significance. If the regularity assumptions hold, the work supplies a clean, first-principles policy-gradient theory for continuous-time extended MFC that removes the separable-structure and known-control-dependence restrictions of earlier exploratory-policy methods. The local martingale characterization (3.20) and the resulting CT-DDPG algorithm are practically useful and are supported by consistent numerical evidence on both LQ and non-LQ problems. The deterministic-policy route is a genuine conceptual contribution relative to the stochastic-policy literature, and the paper is careful to state the classical-solution hypotheses under which the identities are derived.

major comments (2)
  1. Assumption 2.2 (and the induced Assumption 3.1) postulates that the lifted value V(·,·,θ) already belongs to C^{1,2}([0,T]×P_2(R^n)) for every policy parameter θ and that the advantage-rate map A[w] is continuously differentiable in θ. This classical regularity is used both for the sensitivity formula (Theorem 2.1) and for the martingale characterization that drives learning (Theorem 3.2 / (3.20)). The paper does not derive it from the coefficients; a short discussion of sufficient conditions (or a pointer to existing viscosity/regularity results for McKean-Vlasov HJB equations) would make the scope of the claims clearer.
  2. Section 3.3 constructs candidate local functions V_D^θ and q_D via a decoupled dynamics and the integrated Hamiltonian (3.27). While this shows existence under extra smoothness, the uniqueness claim in Theorem 3.2 is only for the integrated objects V̂ and q̂. The algorithm learns the local networks V^φ and q^ψ; a brief remark on whether different local representatives can produce the same integrated gradient (and therefore the same policy update) would strengthen the link between theory and practice.
minor comments (4)
  1. In (3.18) and (3.22) the independent copy is written eξ / eX; a single consistent notation (e.g., ξ̃) would improve readability.
  2. Figure 1 caption states that AC and q-Learning exploit the LQ structure while CT-DDPG does not; the main text already makes this clear, but the caption could briefly note that the comparison is therefore not fully model-agnostic.
  3. The terminal-penalty weight w=0.002 and soft-update τ=0.1 appear only in the experimental section; a short sensitivity remark (or a default recommendation) would help reproducibility.
  4. A few typographical inconsistencies remain (e.g., “T echnical” in the section heading of 6.1, occasional missing spaces after commas in displayed equations).

Circularity Check

0 steps flagged

No significant circularity: policy-gradient identities and martingale characterizations are derived from performance-difference + Itô under postulated classical regularity, without fitting the target or load-bearing self-citation chains.

full rationale

The core claims (Theorems 2.1–2.3, 3.1–3.2) follow from a performance-difference identity (Proposition 6.2, extending classical control lemmas via Itô on the flow of measures) plus the linear PDE satisfied by the lifted value (Lemma 6.1) and a chain-rule expansion of the push-forward (3.18). The advantage-rate A[w] is defined from the generator (2.5–2.6) and then shown to recover the parameter gradient by direct differentiation under the integral; the local representations VD, qD are constructed explicitly as the decoupled value and integrated Hamiltonian (3.23–3.27) and verified to satisfy the martingale condition (3.20). Regularity (Assumptions 2.2/3.1) is postulated rather than proved, but that is an existence gap, not a circular reduction of the claimed identities to their inputs. Self-citations ([7], [11], [27], etc.) supply related continuous-time or discrete-time analogues; none is invoked as a uniqueness theorem that forces the present formulae. Numerical experiments validate the resulting CT-DDPG algorithm on independent benchmarks and do not enter the theoretical statements. The derivation is therefore self-contained against its own equations.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 1 invented entities

The theoretical claims rest on standard Lipschitz/growth conditions for McKean-Vlasov coefficients, classical C^{1,2} regularity of the lifted value function (postulated), and the existence of local decompositions of value and advantage. Experimental free parameters (learning rates, exploration noise, network widths, particle count) affect only the numerical illustrations, not the theorems.

free parameters (4)
  • learning rate η = 3×10^{-4}
    Adam learning rate fixed at 3e-4 for all networks; chosen by hand and affects observed convergence speed.
  • exploration noise σ_epl = 0.1 (default)
    Gaussian noise scale for action- or parameter-space exploration; default 0.1, ablated over {0.001,0.01,0.1}.
  • particle count M = 50
    Number of particles used to approximate the McKean-Vlasov law; fixed at 50.
  • soft-update τ and terminal weight w = τ=0.1, w=0.002
    Target-network EMA coefficient and terminal-loss weight; set to 0.1 and 0.002 respectively.
axioms (3)
  • domain assumption Coefficients b,σ,r,g satisfy local Lipschitz continuity in (x,μ) and linear growth uniformly in the policy parameter (Assumption 2.1 / 3.1).
    Standard for well-posedness of McKean-Vlasov SDEs; invoked throughout Sections 2–3.
  • ad hoc to paper For every policy parameter θ the lifted value function V(·,·,θ) belongs to C^{1,2}([0,T]×P_2(R^n)) and the advantage-rate map A[w] is continuously differentiable in θ (Assumption 2.2).
    Classical-solution regularity is postulated rather than derived from the coefficients; used for the sensitivity formula (Thm 2.1) and martingale characterization.
  • ad hoc to paper Local value and advantage-rate functions admit continuous L-derivatives with respect to the control marginal and satisfy the chain rule (Assumption 3.2).
    Needed to pass from the integrated policy gradient to the local expression containing both ∂_a and ∂_ν terms (Thm 3.2).
invented entities (1)
  • local advantage-rate function q_D(t,x,a,Γ) no independent evidence
    purpose: Decomposes the lifted advantage rate so that both ordinary action derivatives and measure derivatives w.r.t. the control law appear in the policy gradient; enables martingale TD learning from individual trajectories.
    Defined in (3.27) as the integrated Hamiltonian; existence is shown under the regularity assumptions but the object itself is introduced for the learning algorithm.

pith-pipeline@v1.1.0-grok45 · 31864 in / 3021 out tokens · 29648 ms · 2026-07-14T07:42:00.399139+00:00 · methodology

0 comments
read the original abstract

This paper develops a model-free reinforcement learning framework for continuous--time extended mean field control problems, where both the dynamics and reward may depend on the joint distribution of states and controls. We adopt deterministic feedback policies, under which the state--action distribution is induced directly as a push--forward of the state law. This avoids optimization over stochastic kernels and bypasses key limitations of existing approaches in extended mean field settings. We first establish a model--free sensitivity formula for parameterized McKean--Vlasov dynamics and use it to derive a deterministic policy gradient formula expressed through an advantage--rate function on the Wasserstein space. We then refine this formula by introducing local value and advantage--rate representations that depend on the state, action, and joint state--action distribution, yielding a policy gradient that includes both action derivatives and measure--derivative terms with respect to the control distribution. These characterizations lead to a martingale--based learning principle and motivate a continuous--time deep deterministic policy gradient algorithm combining particle approximations, measure--dependent neural networks, temporal--difference learning, and exploration in either action or parameter space. Numerical experiments on stochastic Cucker--Smale consensus control and optimal liquidation with trade crowding demonstrate the efficiency, stability, and robustness of the proposed method, including problems with explicit dependence on the control distribution.

Figures

Figures reproduced from arXiv: 2607.11005 by Huy\^en Pham, Xin Guo, Yufei Zhang, Ziheng Cheng.

Figure 1
Figure 1. Figure 1: Results for C-S model with γ = 0. AC and q-Learning exploit the LQ structure and corresponding parameterization, while CT-DDPG uses neural network parameterized policy. Results for LQ case. We first set γ = 0, C = 1, c = 0.1, σ = 0.3, T = 1 in C-S dynamics and the initial state distribution is µ0 = Unif([0, 1] n ) × Unif([−1, 1] n ). The optimal policy is given explic￾itly in (5.4). We shall examine the ef… view at source ↗
Figure 2
Figure 2. Figure 2: Results for C-S model with γ = 1. Different designs of mean-field distribution embed￾dings in CT-DDPG lead to different convergence rates. 0 60 120 180 240 300 Episodes 1.1 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 Action epl=0.001 epl=0.01 epl=0.1 0 60 120 180 240 300 Episodes 1.1 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 Parameter epl=0.001 epl=0.01 epl=0.1 [PITH_FULL_IMAGE:figures/full_fig_p019_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Results for C-S model with γ = 1, h = 0.0005, where CT-DDPG is implemented with different exploration strategies. Observe that CT-DDPG achieves the fastest convergence to the optimal value, despite not ex￾plicitly exploiting the LQ structure. In contrast, AC and q-Learning adopt an LQ-based param￾eterization and nevertheless underperform CT-DDPG, further demonstrating the effectiveness of the proposed meth… view at source ↗
Figure 4
Figure 4. Figure 4: Results of CT-DDPG for liquidation problem with [PITH_FULL_IMAGE:figures/full_fig_p021_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 9 linked inside Pith

  1. [1]

    Acciaio, J

    B. Acciaio, J. Backhoff-Veraguas, and R. Carmona. Extended mean field control problems: stochastic maximum principle and transport perspective.SIAM journal on Control and Opti- mization, 57(6):3666–3693, 2019

    2019

  2. [2]

    Bayraktar, N

    E. Bayraktar, N. B ¨auerle, and A. D. Kara. Finite approximations for mean-field type multi- agent control and their near optimality.Applied Mathematics & Optimization, 92(1):7, 2025

    2025

  3. [3]

    Bayraktar, M

    E. Bayraktar, M. Hernandez, Q. Yan, and Y. Zhu. Policy gradient for continuous-time mean- field control.arXiv preprint arXiv:2605.20718, 2026. 24

    Pith/arXiv arXiv 2026

  4. [4]

    Buckdahn, J

    R. Buckdahn, J. Li, S. Peng, and C. Rainer. Mean-field stochastic differential equations and associated pdes.Annals of Probability: An official journal of the Institute of Mathematical Statistics, 45(2):824–878, 2017

    2017

  5. [5]

    Carmona and F

    R. Carmona and F. Delarue.Probabilistic Theory of Mean Field Games with Applications I: Mean Field FBSDEs, Control, and Games, volume 83. Springer, 2018

    2018

  6. [6]

    Carmona, M

    R. Carmona, M. Lauri `ere, and Z. Tan. Model-free mean-field reinforcement learning: mean- field MDP and mean-field Q-learning.The Annals of Applied Probability, 33(6B):5334–5381, 2023

    2023

  7. [7]

    Cheng, X

    Z. Cheng, X. Guo, and Y. Zhang. Deterministic policy gradient for reinforcement learning with continuous time and state.arXiv preprint arXiv:2509.23711, 2025

    arXiv 2025

  8. [8]

    Cosso, F

    A. Cosso, F. Gozzi, I. Kharroubi, H. Pham, and M. Rosestolato. Optimal control of path- dependent McKean–Vlasov SDEs in infinite-dimension.The Annals of Applied Probability, 33(4):2863–2918, 2023

    2023

  9. [9]

    Cucker and S

    F. Cucker and S. Smale. Emergent behavior in flocks.IEEE Transactions on automatic control, 52(5):852–862, 2007

    2007

  10. [10]

    Foerster, I

    J. Foerster, I. A. Assael, N. De Freitas, and S. Whiteson. Learning to communicate with deep multi-agent reinforcement learning.Advances in neural information processing systems, 29, 2016

    2016

  11. [11]

    Frikha, M

    N. Frikha, M. Germain, M. Lauri `ere, H. Pham, and X. Song. Actor-critic learning for mean- field control in continuous time.Journal of Machine Learning Research, 26(127):1–42, 2025

    2025

  12. [12]

    Germain, M

    M. Germain, M. Lauri `ere, H. Pham, and X. Warin. Deepsets and their derivative networks for solving symmetric PDEs.Journal of Scientific Computing, 91(2):63, 2022

    2022

  13. [13]

    H. Gu, X. Guo, X. Wei, and R. Xu. Dynamic programming principles for mean-field controls with learning.Operations Research, 71(4):1040–1054, 2023

    2023

  14. [14]

    H. Gu, X. Guo, X. Wei, and R. Xu. Mean-field multiagent reinforcement learning: A decen- tralized network approach.Mathematics of Operations Research, 50(1):506–536, 2025

    2025

  15. [15]

    X. Guo, Y. Huang, and X. Yu. Deterministic policy gradient for learning equilibrium in time- inconsistent control problems.arXiv preprint arXiv:2606.11798, 2026

    Pith/arXiv arXiv 2026

  16. [16]

    Haarnoja, A

    T. Haarnoja, A. Zhou, P . Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. InInternational Conference on Machine Learning, pages 1861–1870. Pmlr, 2018

    2018

  17. [17]

    Y. Jia, D. Ouyang, and Y. Zhang. Accuracy of discretely sampled stochastic policies in continuous-time reinforcement learning.SIAM Journal on Control and Optimization, 64(3):1889–1929, 2026

    1929

  18. [18]

    Jia and X

    Y. Jia and X. Y. Zhou. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms.Journal of Machine Learning Research, 23(275):1–50, 2022

    2022

  19. [19]

    Jia and X

    Y. Jia and X. Y. Zhou. q-learning in continuous time.Journal of Machine Learning Research, 24(161):1–61, 2023. 25

    2023

  20. [20]

    T. P . Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning.arXiv preprint arXiv:1509.02971, 2015

    Pith/arXiv arXiv 2015

  21. [21]

    R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch. Multi-agent actor- critic for mixed cooperative-competitive environments.Advances in neural information process- ing systems, 30, 2017

    2017

  22. [22]

    Ma and Z

    J. Ma and Z. Liu. Continuous dependence for McKean-Vlasov SDEs under distribution- dependent Lyapunov conditions.Discrete and Continuous Dynamical Systems-S, 18(11):3282– 3301, 2025

    2025

  23. [23]

    Mekkaoui, H

    S. Mekkaoui, H. Pham, and X. Warin. Learning operators on labelled conditional distribu- tions with applications to mean field control of non exchangeable systems.arXiv preprint arXiv:2603.21683, 2026

    arXiv 2026

  24. [24]

    Meunier, H

    M. Meunier, H. Pham, and C. Reisinger. Model-free policy gradient for discrete-time mean- field control.arXiv preprint arXiv:2601.11217, 2026

    arXiv 2026

  25. [25]

    Motte and H

    M. Motte and H. Pham. Mean-field markov decision processes with common noise and open- loop controls.The Annals of Applied Probability, 32(2):1421–1458, 2022

    2022

  26. [26]

    Nourian, P

    M. Nourian, P . E. Caines, and R. P . Malham ´e. Mean field analysis of controlled cucker- smale type flocking: Linear analysis and perturbation equations.IFAC Proceedings Volumes, 44(1):4471–4476, 2011

    2011

  27. [27]

    Pham and X

    H. Pham and X. Warin. Actor-critic learning algorithms for mean-field control with moment neural networks.Methodology and Computing in Applied Probability, 27(1):13, 2025

    2025

  28. [28]

    Pham and X

    H. Pham and X. Wei. Bellman equation and viscosity solutions for mean-field stochastic control problem.ESAIM: Control, Optimisation and Calculus of Variations, 24(1):437–461, 2018

    2018

  29. [29]

    Picarelli, M

    A. Picarelli, M. Scaratti, and J. Tam. Extended mean field control: a global numerical solution via finite-dimensional approximation.arXiv preprint arXiv:2503.20510, 2025

    arXiv 2025

  30. [30]

    Plappert, R

    M. Plappert, R. Houthooft, P . Dhariwal, S. Sidor, R. Y. Chen, X. Chen, T. Asfour, P . Abbeel, and M. Andrychowicz. Parameter space noise for exploration.arXiv preprint arXiv:1706.01905, 2017

    Pith/arXiv arXiv 2017

  31. [31]

    Reisinger, W

    C. Reisinger, W. Stockinger, M. O. Tsianni, and Y. Zhang. Convergence rates of time dis- cretization in extended mean field control.arXiv preprint arXiv:2509.00904, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Reisinger, W

    C. Reisinger, W. Stockinger, and Y. Zhang. A fast iterative pde-based algorithm for feedback controls of nonsmooth mean-field control problems.SIAM Journal on Scientific Computing, 46(4):A2737–A2773, 2024

    2024

  33. [33]

    Z. Ren, X. Wei, X. Yu, and X. Y. Zhou. Continuous-time q-learning for mean-field control with common noise, part-i: Theoretical foundations.arXiv preprint arXiv:2604.27372, 2026

    Pith/arXiv arXiv 2026

  34. [34]

    Z. Ren, X. Wei, X. Yu, and X. Y. Zhou. Continuous-time q-learning for mean-field control with common noise, part-ii: q-learning algorithms.arXiv preprint arXiv:2604.27378, 2026

    Pith/arXiv arXiv 2026

  35. [35]

    Sethi, D

    D. Sethi, D. ˇSiˇska, and Y. Zhang. Entropy annealing for policy mirror descent in continuous time and space.SIAM Journal on Control and Optimization, 63(4):3006–3041, 2025. 26

    2025

  36. [36]

    H. M. Soner, J. Teichmann, and Q. Yan. Learning algorithms for mean field optimal control. arXiv preprint arXiv:2503.17869, 2025

    Pith/arXiv arXiv 2025

  37. [37]

    Szpruch, T

    L. Szpruch, T. Treetanthiploet, and Y. Zhang. Optimal scheduling of entropy regularizer for continuous-time linear-quadratic reinforcement learning.SIAM Journal on Control and Optimization, 62(1):135–166, 2024

    2024

  38. [38]

    Wei and X

    X. Wei and X. Yu. Continuous time q-learning for mean-field control problems.Applied Mathematics & Optimization, 91(1):10, 2025

    2025

  39. [39]

    X. Wei, X. Yu, and F. Yuan. Unified continuous-time q-learning for mean-field game and mean-field control problems.arXiv preprint arXiv:2407.04521, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    J. Yong. Linear-quadratic optimal control problems for mean-field stochastic differen- tial equations—time-consistent solutions.Transactions of the American Mathematical Society, 369(8):5467–5523, 2017. 27

    2017