Pith. sign in

REVIEW 3 major objections 4 minor 41 references

For entropy-regularized N-player stochastic differential games, a Nash equilibrium exists exactly when a cross-partial compatibility condition on the optimal q-functions holds.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 11:16 UTC pith:EAUP5AH3

load-bearing objection Solid continuous-time RL extension to finite-N differential games, but the thing called a Nash equilibrium is really a correlated/conditional equilibrium, so the central existence claims need honest reframing before this is publishable as stated. the 3 major comments →

arxiv 2607.19928 v1 pith:EAUP5AH3 submitted 2026-07-22 math.OC

Continuous-Time Reinforcement Learning for N-Player Stochastic Differential Games with Exploratory Policies

Jisheng Liu , Jing Zhang This is my paper
classification math.OC MSC 91A1593E2060H1049L2091A10
keywords stochastic differential gamescontinuous-time reinforcement learningexploratory policiesentropy regularizationNash equilibriumq-learningcompatibility conditionapproximate correlated equilibrium
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish a complete equilibrium theory for N-player stochastic differential games in which each player explores with an entropy-regularized stochastic policy. Given the other players' realized actions, each player's optimal response is a Gibbs (softmax) distribution over actions; the paper defines a Nash equilibrium as the existence of a joint density whose conditionals are exactly these Gibbs responses. The central result is that such a Nash equilibrium exists if and only if a simple cross-partial identity holds between any two players' optimal q-functions, (1/γ_i)∂²q_i*/∂u_j∂u_i = (1/γ_j)∂²q_j*/∂u_i∂u_j. Decoupled and symmetric games always satisfy the identity, and when it fails the paper constructs an approximate correlated equilibrium with explicit quadratic KL-divergence and value-loss bounds that vanish as the exploration weights grow. This matters because it turns the existence question for multi-agent continuous-time RL into a computable condition on learned quantities, and supplies a fallback equilibrium concept when exact Nash play is impossible.

Core claim

The paper's core claim is that in the exploratory formulation of N-player stochastic differential games, the natural equilibrium concept—simultaneously maximizing every player's Hamiltonian in the coupled HJB system—is equivalent to compatibility of the conditional Gibbs optimal policies π_i^*(u_i|u_{-i}). Compatibility is characterized three ways, most usefully as the cross-partial condition on optimal q-functions: for every pair i≠j, (1/γ_i)∂²q_i^*/∂u_j∂u_i = (1/γ_j)∂²q_j^*/∂u_i∂u_j. When this holds, the equilibrium joint density is unique and takes a Gibbs form built by path-ordered integration of the q-function gradients; when it fails, the coordinate path-integral construction still yie

What carries the argument

The central object is the family of conditional Gibbs best responses π_i^*(u_i;t,x,u_{-i}) ∝ exp{q_i^*(t,x,u)/γ_i}, where q_i^* is player i's optimal q-function (Hamiltonian minus time-decay and discount) and γ_i is that player's exploration weight. The argument is carried by the compatibility condition: the N conditional densities are the conditionals of one joint density on U^N. The paper proves this is equivalent to the cross-partial identity on the log-densities, which becomes the q-function criterion (1/γ_i)∂²q_i^*/∂u_j∂u_i = (1/γ_j)∂²q_j^*/∂u_i∂u_j; a Poincaré-lemma path integral then constructs the equilibrium joint density. When the identity fails, the same path-integral construction

Load-bearing premise

The load-bearing premise is that a 'Nash equilibrium' may be defined as a joint density whose conditionals are each player's Gibbs best response to the others' realized actions; in the classical game from Section 3.1, controls depend only on (t,x), so strategies that condition on u_{-i} switch the solution concept to a correlated (or conditional quantal response) equilibrium, and the unconditional existence results are not proved for the classical Nash definition.

What would settle it

Take a two-player, two-dimensional action game with non-separable drift (e.g., b_1(x,u)=c u_1 u_2) and compute the optimal q-functions by solving the coupled HJB system; if the cross-partial gap Δ_12 is nonzero at some (t,x,u) while a classical open-loop or Markov-perfect Nash equilibrium (with F-adapted controls) exists, then the paper's criterion characterizes its correlated equilibrium, not the classical Nash equilibrium. Alternatively, simulate the paper's off-policy algorithm in a game with nonzero gap and check whether the learned joint policy matches the predicted approximate-correlated

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • In decoupled games (dynamics and rewards depend only on each player's own state and action) and in symmetric games with equal exploration weights, the cross-partial identity holds automatically, so a Nash equilibrium exists unconditionally and is unique.
  • When the identity holds, the equilibrium joint density is explicitly constructible as exp of a path-ordered integral of q-function gradients; no fixed-point iteration is needed to find it.
  • When the identity fails, the approximate correlated equilibrium has KL divergence at most (e^{2(N−1)ε|U|^2}−1)^2 from the individually optimal conditional policies, and the per-player value loss is at most γ_i(e^{2(N−1)ε|U|^2}−1)^2/β_i, so both vanish locally uniformly at O(1/γ) as exploration weights grow.
  • The q_i-functions satisfy a weak martingale characterization: a candidate q-function equals the true one iff a certain discounted process is a martingale under any admissible policy, which justifies off-policy TD learning without knowing the model.
  • The same results carry over to the infinite-horizon ergodic setting, with identical O_R(1/γ) local rates for policy uniformization, compatibility gap, and value sub-optimality.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the compatibility criterion is a purely algebraic, finite-dimensional condition on q-functions that could be monitored online during multi-agent training; the paper notes this empirically but does not prove convergence of the policy-iteration dynamics to a compatible profile.
  • Editorial extension: because each player's strategy is allowed to depend on the other players' realized actions, the solution concept is closer to a correlated (conditional quantal response) equilibrium than to the classical Nash equilibrium of the original differential game with F-adapted controls; if one insists on the classical definition, the unconditional existence results apply to the correl
  • Editorial extension: the potential-game parallel (a single potential Φ with 1/γ̄ = Σ1/γ_i) suggests that when compatibility holds the equilibrium is a potential-game equilibrium; testing whether learning dynamics converge to this potential in finite-N continuous-time games is a natural next step.
  • Editorial extension: a sharp test of the theory would be to compute the compatibility gap in an asymmetric linear-quadratic game with cross-coupling in drift; the numerical section's LQ example has zero gap by construction, so the approximate-correlated-equilibrium bounds remain numerically untested there.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper develops an entropy-regularized, continuous-time reinforcement-learning framework for N-player nonzero-sum stochastic differential games. For each player, and for a fixed value of the other players' actions u_-i, the optimal exploratory response is a Gibbs density π_i^*(·|u_-i). A joint density ψ is called a Nash equilibrium (Definition 10) if its conditional densities coincide with these Gibbs responses. The authors prove that this compatibility is equivalent to simultaneous Hamiltonian maximization, characterize it by a cross-partial condition on the optimal q-functions, establish unconditional existence for decoupled and symmetric games, and construct an approximate correlated equilibrium with explicit KL-divergence bounds when compatibility fails. They also provide q-learning algorithms based on weak martingale characterizations, with an ergodic extension and a two-player linear-quadratic numerical experiment.

Significance. If the equilibrium notion is framed correctly, the paper makes a useful contribution: it extends continuous-time exploratory RL to finite-N general-sum games, gives a computable compatibility criterion in terms of q-functions, and provides explicit approximate-equilibrium bounds that vanish as γ→∞. The proofs are detailed and the construction in Theorem 24 is explicit. However, the central terminology is problematic: the object called 'Nash equilibrium' is a correlated (or conditional quantal response) equilibrium, not a Nash equilibrium in mixed strategies. This issue pervades the abstract, introduction, and main existence theorems, so the central claim as stated is not supported. The underlying mathematics is largely coherent, and the paper can be made correct by re-framing the equilibrium concept or by proving product-form existence in the symmetric/decoupled cases.

major comments (3)
  1. [Definition 10; Theorems 15 and 19; Theorem 24] The equilibrium object defined in Definition 10 is not a Nash equilibrium in the standard sense. A Nash equilibrium in mixed strategies requires a product-form joint distribution ψ(u)=∏_i π_i(u_i), so that each player's randomization is independent of the others' realized actions. Definition 10 instead allows an arbitrary joint density whose conditionals are the Gibbs best responses π_i^*(·|u_-i); this is the definition of a correlated equilibrium (or conditional quantal response equilibrium). In coupled games π_i^* genuinely depends on u_-i (see Section 4.2, Eq. (17)), so any compatible ψ is non-product. The paper itself signals this in Theorem 24, where the ε=0 object is called 'an exact correlated equilibrium which coincides with the Nash equilibrium of Definition 10.' Since the abstract and Introduction claim 'Nash equilibria exist unconditionally' on the basis of Theorems 18–19, the
  2. [Theorem 19] The proof of unconditional existence for symmetric games is incomplete as written. It asserts that all q_i^* are 'the same function q^* up to relabeling' and then applies Schwarz's theorem to a single function q^*. Symmetry under simultaneous permutation of state coordinates and controls gives relations at permuted states x^τ, not at the same non-symmetric state x. The player-dependent running reward f_i(t,x,u_i) also distinguishes q_i from q_j at the same x. The cross-partial condition may still hold under the stated symmetry—because ∂²f_i/∂u_j∂u_i=0 and the symmetric dynamics give the same mixed partial—but this needs to be shown directly. As written, the proof does not establish the claimed existence of a compatible joint density.
  3. [Section 8.4 and Table 2] The numerical verification does not actually test the exploratory equilibrium. The theoretical equilibrium policies are conditional Gaussian densities with variance depending on γ_i and on the q-function parameters; the experiment instead simulates deterministic feedback laws a_i=K_i x_t, and the exploration weights γ_i are never specified. The value function is fixed at the Riccati solution rather than learned. The parameter convergence results are suggestive, but the claim that 'the learned conditional policies reproduce the Nash equilibrium stationary distribution' is not supported by the reported experiment. This should be corrected or the claims substantially softened.
minor comments (4)
  1. [Remark 22 / Abstract] Remark 22 correctly notes that the O_R(1/γ) rate reflects uniformization of policies, not structural alignment. This caveat should be stated in the abstract, where the notation O_R(1/γ) currently appears without definition.
  2. [Appendix A] There are several typographical issues: 'Assumption2alreadyyields' is missing spaces, and 'F unding' in the author footnote is a typo. The appendix is otherwise clear.
  3. [Proposition 17] The proposition states 'additionally assuming V~i is C^1 in u_-i', but Remark 3 and Assumption 2 are meant to provide this. Please make the cross-reference explicit so the assumption is not introduced ad hoc.
  4. [Section 7 and Remark 38] Remark 38 correctly notes that compatibility is not preserved under policy iteration; this is an important limitation and should be given more prominence in the conclusions. The sentence is currently buried in the algorithmic section.

Circularity Check

2 steps flagged

Compatibility theorem is mathematically substantive, but 'Nash equilibrium' is defined as the joint-density compatibility condition, and the LQ experiment feeds the known Riccati value into the TD target.

specific steps
  1. self definitional [Definition 10; Proposition 11; Theorem 12]
    "For any (t,x)∈[0,T]×R^N, if there exists a probability space ... possessing a joint density ψ∈P(U^N) such that π_i^*(·;t,x,u_{−i}) is the conditional density of A_i(t,x) given {A_j(t,x)}_{j≠i} for each i=1,...,N, then we say a Nash equilibrium of the exploratory game (2)–(3) exists."

    The 'Nash equilibrium' is defined to be exactly the existence of a joint density whose conditional densities are the Gibbs best responses. Proposition 11 restates this definition (π_i^* = ψ/∫ψ du_i), and Theorem 12's equivalence between natural equilibrium and compatibility is a comparison of two formulations of the same conditional-maximization requirement. Thus the central existence claims in Theorems 18–19 are true by the paper's stipulated solution concept; they do not establish a product-form classical Nash equilibrium of the F-adapted game in Section 3.1, and Theorem 24 itself calls the ε=0 object 'an exact correlated equilibrium.'

  2. fitted input called prediction [Section 8.2–8.4, esp. eqs. (40)-(41) and Section 8.3]
    "The value function V_i(x) = −P_i x^2 is fixed at the Riccati solution, eliminating collinearity ... The TD target for player i ... is target_i = −(Q_i x^2 + R_i(a_i)^2)Δt + δ_ρ V_i(x′)."

    The experiment's conclusion that the learned policies 'approximate the Nash equilibrium stationary distribution' is obtained with the Nash value function V_i = −P_i x^2 placed inside the TD target. The true gains K_i and the q-function parameters in (41) are derived from the same Riccati P_i, so the target already encodes the equilibrium. The numerical section is therefore a consistency check that the estimator recovers a functional of its own target, not an independent prediction of the equilibrium.

full rationale

The core analytical results are not circular in the fitting sense: q_i^* and π_i^* are obtained from the entropy-regularized HJB equations, and Theorem 15 is a direct but contentful substitution of log π_i^* = q_i^*/γ_i into the cross-partial compatibility condition, whose closed-form potential (16) follows from Poincaré's lemma. Theorem 24's approximate-correlated-equilibrium construction and the KL/value bounds are genuine inequalities given an externally checkable ε. The paper's self-references to [21] and [36] are to prior single-agent CT-RL work by other authors, used as lemmas/tools; they are not a self-citation chain that forces the conclusions. What prevents a clean 0–2 score is that the paper's headline 'Nash equilibrium exists' claims are tied to Definition 10, which defines Nash equilibrium as the existence of a joint density with Gibbs conditionals—a correlated/conditional equilibrium concept. Theorem 12 then 'proves' an equivalence that is essentially definitional, so the unconditional existence theorems are statements about the paper's own solution concept rather than the classical Nash equilibrium. The LQ experiment compounds this by embedding the known Riccati value function in the TD target, making its 'reproduce the Nash stationary distribution' check partly assume the target it claims to validate. Weighing both issues as partial, the central compatibility mathematics is still independent and nontrivial, so the overall circularity is limited.

Axiom & Free-Parameter Ledger

0 free parameters · 7 axioms · 0 invented entities

No fitted parameters enter the theorems; the exploration weights γ_i are model inputs, not fitted values. The O_R(1/γ) constants (M_R, C_R) are existence constants. The paper introduces no new physical entities but relies on a nonstandard equilibrium definition (Definition 10), which is the load-bearing conceptual assumption.

axioms (7)
  • domain assumption Assumptions 1-2: Lipschitz/growth conditions, uniform ellipticity, C^{1,2+α,2} smoothness of coefficients.
    Used throughout for well-posedness of the exploratory SDE, classical solutions of the HJB system (Remark 3), and sensitivity estimates in Proposition 21.
  • ad hoc to paper Conditional reduction (Lemma 8): optimal response depends on opponents only through the current-time marginal π_{-i}(·;t,x).
    Justifies freezing u_{-i} as a parameter and the Dirac-measure parameterization (Remark 6); central to the entire conditional framework.
  • ad hoc to paper Definition 10 identifies a Nash equilibrium with the existence of a joint density whose conditionals are the Gibbs best responses.
    This is not the standard Nash equilibrium concept for SDGs; it is a correlated-equilibrium condition. All existence and approximation results are statements about this object.
  • standard math Poincaré lemma on simply connected U^N (Theorem 13).
    Used to pass from the cross-partial condition to the existence of a global potential joint density.
  • standard math Interior Schauder estimates and maximum principle (Proposition 21).
    Provides the O_R(1/γ) uniform bounds and boundedness of spatial derivatives of the value functions.
  • domain assumption Compact convex U containing 0 (Theorems 20 and 24).
    Needed for the coordinate path integral starting at 0 and for finiteness of the partition function.
  • domain assumption Ergodic well-posedness conditions (i) dissipativity and (ii) at-most-quadratic growth (Section 5.2).
    Ensures classical solutions of the elliptic HJB equation.

pith-pipeline@v1.3.0-alltime-deepseek · 32326 in / 16518 out tokens · 156170 ms · 2026-08-01T11:16:19.814769+00:00 · methodology

0 comments
read the original abstract

We study continuous-time reinforcement learning for $N$-player noncooperative stochastic differential games. Each player adopts an entropy-regularized exploratory policy; given the others' actions, the optimal response is a Gibbs distribution, and a Nash equilibrium requires these $N$ conditional distributions to be jointly compatible. We prove that the natural equilibrium concept -- simultaneous Hamiltonian maximization -- is equivalent to this compatibility, and establish a necessary and sufficient condition expressed as a computable cross-partial criterion on the optimal $q$-functions. Nash equilibria exist unconditionally for decoupled and symmetric games. When compatibility fails, a coordinate path integral construction yields an approximate correlated equilibrium with explicit quadratic KL-divergence bounds that vanish locally uniformly as the exploration weight $\gamma\to\infty$. A $q$-function framework for the $N$-player game extends the single-agent $q$-learning theory of [21], with weak martingale characterizations motivating model-free on-policy and off-policy algorithms. The framework extends to the ergodic (infinite-horizon) setting with the same locally uniform $O_R(1/\gamma)$ asymptotic rates.

Figures

Figures reproduced from arXiv: 2607.19928 by Jing Zhang, Jisheng Liu.

Figure 1
Figure 1. Figure 1: Convergence of all six q-function parameters (ψ1, ψ2, ψ3, η1, η2 for each player) over 3×106 off-policy TD steps. Red dashed: true Nash values; gray dotted: learning-rate annealing point at step 2 × 106 . The zero-mean property is preserved in both cases. This is consistent with the conditional q-learning algorithm producing strategies that, when used in the ergodic game, approxi￾mate the Nash equilibrium … view at source ↗
Figure 2
Figure 2. Figure 2: Enlarged view of the opponent-state coupling parameter [PITH_FULL_IMAGE:figures/full_fig_p034_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Ergodic equilibrium verification. Left: state trajectory under the learned feed￾back policy (blue, K1 learned = −0.449, K2 learned = −0.316) vs. Nash policy (red dashed, K1 = −0.549, K2 = −0.443). Right: empirical stationary distribution of Xt (blue his￾togram) vs. theoretical Nash stationary density (black dashed). games as N → ∞, including convergence of the conditional compatibility condition to the McK… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 7 linked inside Pith

  1. [1]

    Başar and G.J

    T. Başar and G.J. Olsder.Dynamic Noncooperative Game Theory. SIAM, 2nd edition, 1999

    1999

  2. [2]

    Bouchard and N

    B. Bouchard and N. Touzi. Weak dynamic programming principle for viscosity solu- tions.SIAM Journal on Control and Optimization, 49(3):948–962, 2011

    2011

  3. [3]

    Buckdahn, P

    R. Buckdahn, P. Cardaliaguet, and C. Rainer. Nash equilibrium payoffs for nonzero- sum stochastic differential games.SIAM Journal on Control and Optimization, 43(2):624–642, 2004

    2004

  4. [4]

    Buşoniu, R

    L. Buşoniu, R. Babuşka, and B. De Schutter. A comprehensive survey of multiagent reinforcement learning.IEEE Transactions on Systems, Man, and Cybernetics, Part C, 38(2):156–172, 2008

    2008

  5. [5]

    Carmona and F

    R. Carmona and F. Delarue.Probabilistic Theory of Mean Field Games with Appli- cations I–II. Springer, 2018. 36

    2018

  6. [6]

    Dockner, S

    E.J. Dockner, S. Jørgensen, N.V. Long, and G. Sorger.Differential Games in Eco- nomics and Management Science. Cambridge University Press, 2000

    2000

  7. [7]

    Fleming and H.M

    W.H. Fleming and H.M. Soner.Controlled Markov Processes and Viscosity Solutions. Springer, 2nd edition, 2006

    2006

  8. [8]

    X. Gao, L. Li, and X.Y. Zhou. Reinforcement learning for jump-diffusions, with financial applications.Mathematical Finance, 2026. Available at: https://doi.org/10.1111/mafi.70027

    doi:10.1111/mafi.70027 2026

  9. [9]

    Springer, 2001

    D.GilbargandN.S.Trudinger.Elliptic Partial Differential Equations of Second Order. Springer, 2001

    2001

  10. [10]

    X. Guo, Y. Xu, and T. Zariphopoulou. Entropy regularization for mean field games with learning.Mathematics of Operations Research, 47(4):3239–3260, 2022

    2022

  11. [11]

    Anα-potential game framework forn-player dynamic games.SIAM Journal on Control and Optimization, 63(4):2964–3005, 2025

    Xin Guo, Xinyu Li, and Yufei Zhang. Anα-potential game framework forn-player dynamic games.SIAM Journal on Control and Optimization, 63(4):2964–3005, 2025

    2025

  12. [12]

    Entropy regularized reinforce- ment learning for zero-sum stochastic differential games in a regime-switching jump- diffusion process.arXiv:2606.28669, 2026

    Congde Hu, Zhuo Jin, Danping Li, and Lin Xu. Entropy regularized reinforce- ment learning for zero-sum stochastic differential games in a regime-switching jump- diffusion process.arXiv:2606.28669, 2026

    Pith/arXiv arXiv 2026

  13. [13]

    Entropy-regularized reinforce- ment learning for linear-quadratic stackelberg differential games in regime-switching diffusion models.arXiv:2606.28671, 2026

    Congde Hu, Danping Li, Lin Xu, and Wenying Xu. Entropy-regularized reinforce- ment learning for linear-quadratic stackelberg differential games in regime-switching diffusion models.arXiv:2606.28671, 2026

    Pith/arXiv arXiv 2026

  14. [14]

    Recent developments in machine learning meth- ods for stochastic control and games.Numerical Algebra, Control and Optimization, 14(3):435–525, 2024

    Ruimeng Hu and Mathieu Laurière. Recent developments in machine learning meth- ods for stochastic control and games.Numerical Algebra, Control and Optimization, 14(3):435–525, 2024

    2024

  15. [15]

    Huang, P.E

    M. Huang, P.E. Caines, and R.P. Malhamé. Large-population cost-coupled LQG problems with nonuniform agents: individual-mass behavior and decentralizedϵ-Nash equilibria.IEEE Transactions on Automatic Control, 52(9):1560–1571, 2007

    2007

  16. [16]

    Continuous-time reinforcement learning for optimal switching over multiple regimes.arXiv:2512.04697, 2025

    Yijie Huang, Mengge Li, Xiang Yu, and Zhou Zhou. Continuous-time reinforcement learning for optimal switching over multiple regimes.arXiv:2512.04697, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    Sublinear regret for a class of continuous- time linear-quadratic reinforcement learning problems.SIAM Journal on Control and Optimization, 63(5):3452–3474, 2025

    Yilie Huang, Yanwei Jia, and Xun Yu Zhou. Sublinear regret for a class of continuous- time linear-quadratic reinforcement learning problems.SIAM Journal on Control and Optimization, 63(5):3452–3474, 2025

    2025

  18. [18]

    Data-driven exploration for a class of continuous-time indefinite linear-quadratic reinforcement learning problems.arXiv:2507.00358, 2025

    Yilie Huang and Xun Yu Zhou. Data-driven exploration for a class of continuous-time indefinite linear-quadratic reinforcement learning problems.arXiv:2507.00358, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    Jia and X.Y

    Y. Jia and X.Y. Zhou. Policy gradient and actor–critic learning in continuous time and space: Theory and algorithms.Journal of Machine Learning Research, 23(275):1–50, 2022

    2022

  20. [20]

    Jia and X.Y

    Y. Jia and X.Y. Zhou. Policy gradient in continuous time and space: A martingale approach.Journal of Machine Learning Research, 23(155):1–40, 2022. 37

    2022

  21. [21]

    Jia and X.Y

    Y. Jia and X.Y. Zhou.q-Learning in continuous time.Journal of Machine Learning Research, 24(161):1–61, 2023

    2023

  22. [22]

    Karatzas and S.E

    I. Karatzas and S.E. Shreve.Brownian Motion and Stochastic Calculus. Springer, 2nd edition, 1991

    1991

  23. [23]

    Krylov.Nonlinear Elliptic and Parabolic Equations of the Second Order

    N.V. Krylov.Nonlinear Elliptic and Parabolic Equations of the Second Order. D. Reidel Publishing Company, 1987

    1987

  24. [24]

    Ladyzhenskaya, V.A

    O.A. Ladyzhenskaya, V.A. Solonnikov, and N.N. Ural’tseva.Linear and Quasi-linear Equations of Parabolic Type. American Mathematical Society, 1968

    1968

  25. [25]

    Lasry and P.-L

    J.-M. Lasry and P.-L. Lions. Mean field games.Japanese Journal of Mathematics, 2(1):229–260, 2007

    2007

  26. [26]

    Actor-critic reinforcement learning al- gorithms for mean field games in continuous time, state and action spaces.Applied Mathematics & Optimization, 89(3):73, 2024

    Hong Liang, Zhiping Chen, and Kaili Jing. Actor-critic reinforcement learning al- gorithms for mean field games in continuous time, state and action spaces.Applied Mathematics & Optimization, 89(3):73, 2024

    2024

  27. [27]

    R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch. Multi-agent actor- critic for mixed cooperative-competitive environments. InAdvances in Neural Infor- mation Processing Systems, 2017

    2017

  28. [28]

    Mnih et al

    V. Mnih et al. Human-level control through deep reinforcement learning.Nature, 518:529–533, 2015

    2015

  29. [29]

    Dov Mondererand Lloyd S. Shapley. Potentialgames.Games and Economic Behavior, 14:124–143, 1996

    1996

  30. [30]

    J. Nash. Non-cooperative games.Annals of Mathematics, 54(2):286–295, 1951

    1951

  31. [31]

    Learning distributed equilibria in linear-quadratic stochastic differential games: Anα-potential approach.arXiv:2602.16555, 2026

    Philipp Plank and Yufei Zhang. Learning distributed equilibria in linear-quadratic stochastic differential games: Anα-potential approach.arXiv:2602.16555, 2026

    arXiv 2026

  32. [32]

    Continuous-time q- learning for mean-field control with common noise, part I: Theoretical foundations

    Zhenjie Ren, Xiaoli Wei, Xiang Yu, and Xun Yu Zhou. Continuous-time q- learning for mean-field control with common noise, part I: Theoretical foundations. arXiv:2604.27372, 2026

    Pith/arXiv arXiv 2026

  33. [33]

    Reinforcement learning for exploratory linear- quadratic two-person zero-sum stochastic differential games.Applied Mathematics and Computation, 442:127763, 2023

    Zhongshi Sun and Guangyan Jia. Reinforcement learning for exploratory linear- quadratic two-person zero-sum stochastic differential games.Applied Mathematics and Computation, 442:127763, 2023

    2023

  34. [34]

    Sutton and A.G

    R.S. Sutton and A.G. Barto.Reinforcement Learning: An Introduction. MIT Press, 2nd edition, 2018

    2018

  35. [35]

    Regret of exploratory policy improvement and q- learning.arXiv:2411.01302, 2024

    Wenpin Tang and Xun Yu Zhou. Regret of exploratory policy improvement and q- learning.arXiv:2411.01302, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    H. Wang, T. Zariphopoulou, and X.Y. Zhou. Reinforcement learning in continuous timeandspace: Astochasticcontrolapproach.Journal of Machine Learning Research, 21(198):1–34, 2020. 38

    2020

  37. [37]

    L. Wang. On the regularity theory of fully nonlinear parabolic equations: II.Com- munications on Pure and Applied Mathematics, 45(2):141–178, 1992

    1992

  38. [38]

    X. Wang. Schauder estimates for elliptic and parabolic equations.Chinese Annals of Mathematics, Series B, 27:637–642, 2006

    2006

  39. [39]

    Continuoustimeq-learningformean-fieldcontrolproblems.Applied Mathematics & Optimization, 91(1):10, 2025

    X.WeiandX.Yu. Continuoustimeq-learningformean-fieldcontrolproblems.Applied Mathematics & Optimization, 91(1):10, 2025

    2025

  40. [40]

    Unified continuous-time q-learning for mean- field game and mean-field control problems.arXiv:2407.04521, 2024

    Xiaoli Wei, Xiang Yu, and Fengyi Yuan. Unified continuous-time q-learning for mean- field game and mean-field control problems.arXiv:2407.04521, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Zhang, Z

    K. Zhang, Z. Yang, and T. Başar. Multi-agent reinforcement learning: A selective overview. InHandbook of Reinforcement Learning and Control. Springer, 2021. 39

    2021