Pith. sign in

REVIEW 1 major objections 5 minor 63 references

Policy Optimization for Continuous-time Linear-Quadratic Graphon Mean Field Games

T0 review · 1 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A bilevel policy-gradient algorithm provably converges to the Nash equilibrium of continuous-time linear-quadratic graphon mean field games, with linear inner rates and epsilon-global accuracy.

desk verdict First policy-gradient result for continuous-time GMFGs with a genuinely useful cost decomposition, but the numerical section runs outside the theorem's smallness regime and should be caveated. read the letter →

arxiv 2506.05894 v1 pith:KMQWTXS2 submitted 2025-06-06 math.OC cs.LGmath.PR

Philipp Plank , Yufei Zhang This is my paper
classification math.OCcs.LGmath.PR MSC 68Q2591A1549N8091A0791A4349N10
keywords graphonmeanfieldgameslinear-quadraticpolicygradientNashequilibriuminfinite-dimensionalspacebileveloptimizationaffinepoliciesheterogeneousagents
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Graphon mean field games model large populations of heterogeneous agents who interact through a network limit, but until now provable learning algorithms existed only in discrete time. This paper proposes a bilevel policy optimization method for continuous-time, finite-horizon linear-quadratic graphon mean field games and proves that it converges to the Nash equilibrium: inner policy gradient steps converge linearly to best responses, and the outer loop returns policy slope, player-specific intercepts, and population mean within $\varepsilon$ of equilibrium values in $\mathcal{O}(\log(1/\varepsilon)^2)$ iterations under a contraction condition. The argument works because each player's best response is affine with a shared slope, which lets the algorithm optimize the slope once for all players and then update each intercept separately. If correct, this gives the first global convergence guarantee for policy optimization in continuous-time graphon mean field games and a parameterization whose cost grows linearly, not quadratically, in the number of players.

What carries the argument

The load-bearing machinery is the cost decomposition of Theorem 2.3 together with two gradient updates. The slope update (2.16) uses the covariance-normalized gradient $\nabla_K J_1^\alpha (\vartheta^{\ell,\alpha})^{-1}$, which is player-independent and keeps iterates uniformly bounded; the intercept update (2.19) exploits uniform strong convexity and Lipschitz smoothness of $G \mapsto J_2^\alpha(K^{(\ell)},G,Z^\alpha)$, proved with Volterra resolvent bounds. The outer loop's contraction is Assumption 2.9, which bounds the graphon's $L^2$ norm and horizon so that each mean-field update shrinks the error to the Nash mean.

What would settle it

Run Algorithm 1 on a problem where the forward-backward system (2.5) is solved numerically and Assumption 2.9 is verified, with $\varepsilon=10^{-3}$ and oracle errors obeying (2.27); if the three bounds of Theorem 2.10 are violated after the prescribed number of outer and inner iterations, the central convergence claim is false.

Watch

Extended reading notes

Core claim

The paper establishes that the Nash equilibrium of an LQ-GMFG can be found by search over affine policies $\phi^\alpha_\theta(t,x)=K_t x + G^\alpha_t$, with one slope $K$ shared by the whole population and intercepts $G^\alpha$ encoding each player's network position. The central identity is the cost decomposition of Theorem 2.3: $J^\alpha(K,G^\alpha,Z^\alpha)=J_1^\alpha(K)+J_2^\alpha(K,G^\alpha,Z^\alpha)$, where $J_1^\alpha$ depends on the slope only through the state covariance and the optimal intercept of $J_2$ is independent of $K$. That decoupling allows a sequential algorithm: covariance-normalized gradient descent on $K$, followed by gradient descent on each $G^\alpha$, with linear convergence to best responses (Theorems 2.5 and 2.8). Under a contraction condition on the graphon interaction, the alternating scheme converges globally, returning parameters and mean within $\varepsilon$ of the Nash equilibrium (Theorem 2.10).

Load-bearing premise

The whole convergence result rests on Assumption 2.9, a smallness condition on the horizon, coefficient norms, and the graphon's $L^2$ norm that makes the best-response map a contraction; if the interaction is too strong, the outer mean-field loop is not guaranteed to shrink errors.

Editorial extensions

If this is right

  • With known model coefficients and an oracle satisfying (2.27), Algorithm 1 returns $K^{(N)}$, $G^{(N)}$, $\mu^{(N)}$ within $\varepsilon$ of the Nash equilibrium.
  • The slope update does not depend on the graphon, so the graphon's structure affects only the intercept and mean-field convergence rate, through its $L^2$ norm.
  • Affine policy search with a shared slope has parameter complexity roughly $k(d+N)$ rather than $k d N^2$, making the method scalable to large populations and high state dimensions.
  • Both full-model pathwise differentiation and model-free zeroth-order implementations show consistent linear error decay across graphons, discretization meshes, and noise levels in the numerical experiments.
  • The inner policy gradient converges linearly for finite-horizon affine LQ problems, closing a gap left by existing analyses confined to infinite-horizon stationary policies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the contraction assumption is the true bottleneck, the same bilevel template should extend to non-LQ dynamics by replacing the exact decomposition with a surrogate loss, although the linear-rate constants would be lost.
  • The covariance normalization trick likely transfers to single-agent and $N$-player finite-horizon affine LQ problems, where existing analyses use second-moment normalization and do not give player-independent updates.
  • A fully model-free guarantee would require a finite-sample bound for the mean-field oracle (2.27); the paper verifies the oracle empirically but does not quantify its sample complexity, so that is the natural next test.
  • For sparse graphs with small $\|W\|_{L^2}$, the theorem predicts faster mean-field convergence; this is visible in the plateau ordering across the four test graphons and is directly testable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper studies continuous-time finite-horizon linear-quadratic graphon mean field games (LQ-GMFGs) and proposes a bilevel policy optimization algorithm (Algorithm 1) in which each player's policy is affine with a shared slope parameter and a player-specific intercept. The main theoretical contributions are: a cost decomposition (Theorem 2.3) that separates slope and intercept optimization; linear convergence of a covariance-normalized policy gradient method for the slope (Theorem 2.5); strong convexity and smoothness properties of the intercept cost that hold uniformly over players, slopes, and graphon aggregates (Propositions 2.6 and 2.7), implying linear convergence of the intercept updates (Theorem 2.8); and global convergence of the full bilevel algorithm to the Nash equilibrium (Theorem 2.10) under a contraction-type condition (Assumption 2.9) and an oracle accuracy condition (2.27). Numerical experiments with pathwise-differentiation and zeroth-order gradient estimates demonstrate linear convergence on several graphons.

Significance. If the results hold, this is a substantial contribution: it appears to be the first provably convergent policy optimization algorithm for continuous-time graphon mean field games, and it extends affine-policy policy gradient analysis to infinite-dimensional time-dependent policy spaces. The paper introduces novel landscape characterizations, including a covariance-normalized gradient update and a Volterra-resolvent-based strong convexity argument, and it provides detailed proofs with explicit constants and dependencies. The numerical study is extensive and includes a model-free zeroth-order extension, giving useful evidence of practical performance. The main limitations are the LQ structure and the contraction Assumption 2.9, which is stronger than the known existence condition for the NE; this is acknowledged in Remark 2.6. The oracle condition (2.27) is stated abstractly, and a finite-sample justification for the model-free case is not provided, but this is a reasonable modeling choice for a first theory.

major comments (1)
  1. [Section 3, Theorem 2.10, Assumption 2.9] All reported numerical runs use parameters for which Assumption 2.9 fails. For the setup in §3.1 (T=1, A=-0.25, B=0.5, \bar A=0.25, Q=0.25, R=0.5, H=1, \bar Q=0.05, \bar H=1, K(0)=-1), solving (2.18) for P(0) gives approximately P(0)≈0.40, so the uniform slope bound in Proposition 4.6 is C_K0 = ||K(0)||∞ + (||B||∞/λ_R)||P(0)||∞ ≈ 1.40. Consequently M1 = T(||A||∞ + ||B||∞ C_K0 + ||\bar A||∞ ||W||_{L2}) = 0.25 + 0.70 + 0.25||W||_{L2}, which exceeds 1 for all four graphons (with ||W||_{L2} ∈ {1/√2, 1/2, 1/√6}, M1 ∈ [1.05,1.13]). The text states that the experiments 'validate our theoretical findings' (Section 3) and 'confirm' the results (Section 1), but no run lies in the hypothesis regime of Theorem 2.10. Please either add an experiment with parameters satisfying M1<1 (e.g., K(0)=0) or explicitly caveat that the numerical study explores a regime beyond the theoretical guarantees and is provided as empirical evidence of robustness, not as a validation of Theorem 2.10.
minor comments (5)
  1. [Section 4.3, proof of Theorem 2.8] The displayed inequalities (4.16) and (4.18) are garbled; the expressions involving '1 L 2η^2 − η' and 'η^2 L 2η^2 − η − 2η' are not meaningful as printed. Please rewrite these lines using the standard descent lemma, e.g., J(x_{k+1}) ≤ J(x_k) − η(1 − Lη/2)||∇J(x_k)||², so that the linear-rate calculation is transparent.
  2. [Section 4.2, Lemma 4.7 and Proposition 4.6] The displayed formula for the upper bound of ϑ(ℓ),α has an unclosed parenthesis: the expression exp(2T(||A||+||B||C_K0)) should be closed before '=: Mϑ'. Additionally, Proposition 4.6 states 'For ℓ ∈ N' but the conclusion is intended for all ℓ ∈ N0; please fix the quantifier.
  3. [Section 3.3] The zeroth-order experiment sets the terminal cost \bar Q = 0, whereas the rest of the paper and the previous experiments use \bar Q = 0.05. Please justify this change and state whether the convergence comparison is still with respect to the NE of the original model with \bar Q = 0.05.
  4. [Theorem 2.10, condition (2.27)] The claim that the mean field oracle can be realized in the model-free case by averaging state trajectories is not quantified. It would be helpful to state how many trajectories (or what error probability) are needed to meet the accuracy ω_n/3, or to explicitly list this finite-sample analysis as future work.
  5. [Throughout Section 4] The term 'Gâteaux' is typeset as 'Gˆ ateaux' in several places (e.g., Lemmas 4.3 and 4.9); please fix the encoding and any other OCR-type artifacts in the displayed equations.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the NE convergence proof is self-contained; the only self-citation is a non-load-bearing adaptation of single-agent estimates, and the numerical regime mismatch is a correctness caveat, not circularity.

full rationale

The paper's derivation chain is self-contained. Theorem 2.3's cost decomposition is algebraic (J = J1 + J2 with Sigma = mu mu^T + vartheta). The slope convergence (Theorem 2.5) is proved from the Riccati equation, gradient dominance (Lemma 4.5), and uniform covariance bounds (Lemma 4.7); the intercept convergence (Theorem 2.8) is proved from strong convexity and smoothness established via Volterra resolvent bounds (Lemmas 4.12 and 4.13). Theorem 2.10's global convergence is a contraction argument under Assumption 2.9 and the oracle accuracy condition (2.27); these are explicit premises, not conclusions. The NE characterization is imported from [21], which is not authored by the present authors, so it is external support. The only self-citation is [23] (coauthor Yufei Zhang), used as a template for single-agent slope convergence: the paper says 'The proof of Theorem 2.5 adapts the analysis for the single-agent LQ control problem in [23] to our setting.' This is not load-bearing because the paper re-derives the necessary player-dependent estimates (e.g., Lemma 4.7 and Corollary 4.8) instead of assuming the GMFG result. No fitted parameter is renamed as a prediction. One non-circular correctness caveat: the Section 3 experiments use T = 1, A = -0.25, B = 0.5, A-bar = 0.25, Q = 0.25, R = 0.5, giving M1 in [1.05, 1.13] > 1 for all four graphons, so the runs lie outside Assumption 2.9; the paper states the experiments 'validate our theoretical findings' without displaying a run satisfying M1 < 1. This affects applicability but does not make the analysis circular.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The theoretical claims introduce no free parameters fitted to data. The proof constants (CK0, Mres, m, L, MG) are all explicit functions of the model coefficients and are not tuned to match the numerical results. The main assumptions are the structural conditions on the LQ-GMFG and the contraction condition in Assumption 2.9.

assumptions (6)
  • standard math Standard functional analysis: L2 spaces, Banach fixed point theorem, Volterra integral operator resolvent bounds, Polyak-Lojasiewicz inequalities.
    Used in Section 4 to establish gradient dominance, strong convexity, and Lipschitz smoothness; these are unproved background facts the paper relies on.
  • domain assumption Unique Nash equilibrium of the LQ-GMFG exists and is characterized by the forward-backward system (2.5), as in [21, Proposition 2].
    Invoked in Section 2.1 and throughout; if the FBSDE system has no unique solution, the NE policy class and the convergence target are not well defined.
  • domain assumption R(t) is uniformly positive definite and Q(t), Qbar are positive semidefinite, with uniform eigenvalue bounds.
    Assumed in Section 2.3 and used in Lemmas 4.4, 4.5 and the strong convexity constant in Lemma 4.13.
  • domain assumption Assumption 2.4: initial state covariances are uniformly bounded and uniformly positive definite.
    Required to normalize the slope gradient by the covariance in (2.16) and to obtain uniform covariance bounds in Lemma 4.7.
  • ad hoc to paper Assumption 2.9: M1 < 1 and M2 < 1, where M1 and M2 are smallness constants defined in (2.24).
    This is the contraction condition for the outer loop; it is stronger than the existence condition (2.25) and is introduced specifically to prove Theorem 2.10.
  • ad hoc to paper The mean field oracle Phi-hat satisfies the accuracy condition (2.27): the returned population mean is within omega_n / 3 of the exact mean under the current policy.
    Theorem 2.10 assumes such an oracle; in model-free settings it is not shown with finite sample guarantees, it is only asserted to be satisfiable by many trajectories.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Policy Optimization for Continuous-time Linear-Quadratic Graphon Mean Field Games." pith.science (2026). https://pith.science/paper/KMQWTXS2

@misc{pith2026250605894,
  author       = {Pith},
  title        = {Pith review of: Policy Optimization for Continuous-time Linear-Quadratic Graphon Mean Field Games},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KMQWTXS2}},
  note         = {Machine review of arXiv:2506.05894}
}
read the original abstract

Multi-agent reinforcement learning, despite its popularity and empirical success, faces significant scalability challenges in large-population dynamic games. Graphon mean field games (GMFGs) offer a principled framework for approximating such games while capturing heterogeneity among players. In this paper, we propose and analyze a policy optimization framework for continuous-time, finite-horizon linear-quadratic GMFGs. Exploiting the structural properties of GMFGs, we design an efficient policy parameterization in which each player's policy is represented as an affine function of their private state, with a shared slope function and player-specific intercepts. We develop a bilevel optimization algorithm that alternates between policy gradient updates for best-response computation under a fixed population distribution, and distribution updates using the resulting policies. We prove linear convergence of the policy gradient steps to best-response policies and establish global convergence of the overall algorithm to the Nash equilibrium. The analysis relies on novel landscape characterizations over infinite-dimensional policy spaces. Numerical experiments demonstrate the convergence and robustness of the proposed algorithm under varying graphon structures, noise levels, and action frequencies.

Figures

Figures reproduced from arXiv: 2506.05894 by the authors.

Figure 1
Figure 1. Graphons for numerical experiments Algorithm 1 is implemented by approximating the continuous policy space (2.8) with the space of piecewise constant policies. Specifically, we consider a uniform time mesh (τi) Npolicy+1 i=0 ⊂ [0, T] with τi = i∆τ , for some ∆τ > 0 and Npolicy ∈ N such that (Npolicy + 1)∆τ = T, and consider piecewise constant policy slope and intercept parameters on this grid (see e.g., [23]). We fu… view at source ↗
Figure 2
Figure 2. presents the RMSEs of the policy parameters throughout the execution of Algorithm 1, for different graphon interactions (with the policy discretization mesh size fixed at ∆τ = 1/30). Recall that there are 10 outer iterations, each comprising 15 policy gradient updates, resulting in a total of 150 gradient iterations. One can clearly observe that both policy parameters converge linearly to the NE policy, even when on… view at source ↗
Figure 3
Figure 3. shows that Algorithm 1 achieves a robust linear convergence for both policy paramet￾ers, regardless of the policy discretization mesh size ∆τ , until the optimization error is overtaken by the discretization error. As highlighted in Remark 3.1, scaling the policy gradient by the discret￾ization mesh size is crucial for ensuring mesh-independent convergence, as standard PG methods typically exhibit performance degrad… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Convergence of Algorithm 1 with different noise levels 20 [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: Convergence of Algorithm 1 with zeroth-order gradient estimations We take the same piecewise constant policy parameterization as in Section 3.2, and update the policies according to (3.3) with smaller stepsizes ηK = ηG = 0.01 to enhance algorithm stability. The number …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 53 canonical work pages

  1. [21]

    Caines, and Minyi Huang

    Shuang Gao, Peter E. Caines, and Minyi Huang. LQG graphon mean field games: Analysis via graphon-invariant subspaces. IEEE Transactions on Automatic Control , 68:7482–7497, 2023

    work page 2023

  2. [1]

    Value iteration algorithm for mean- field games

    Berkay Anahtarcı, Can Deha Karıksız, and Naci Saldi. Value iteration algorithm for mean- field games. Systems & Control Letters , 143(104744), 2020

    work page 2020

  3. [2]

    Unified reinforcement q-learning for mean field game and control problems

    Andrea Angiuli, Jean-Pierre Fouque, and Mathieu Lauri` ere. Unified reinforcement q-learning for mean field game and control problems. Mathematics of Control, Signals, and Systems , 34:217–271, 2022

    work page 2022

  4. [3]

    Stochastic graphon games: II

    Alexander Aurell, Ren´ e Carmona, and Mathieu Lauri` ere. Stochastic graphon games: II. the linear-quadratic case. Applied Mathematics & Optimization , 85(39), 2022

    work page 2022

  5. [4]

    Berahas, Liyuan Cao, Krzysztof Choromanski, and Katya Scheinberg

    Albert S. Berahas, Liyuan Cao, Krzysztof Choromanski, and Katya Scheinberg. A theor- etical and empirical comparison of gradient approximations in derivative-free optimization. Foundations of Computational Mathematics , 22:507–560, 2022

    work page 2022

  6. [5]

    An Lp theory of sparse graph convergence i: Limits, sparse random graph models, and power law distributions

    Christian Borgs, Jennifer Chayes, Henry Cohn, and Yufei Zhao. An Lp theory of sparse graph convergence i: Limits, sparse random graph models, and power law distributions. Transactions of the American Mathematical Society , 372:3019–3062, 2019

    work page 2019

  7. [6]

    Chayes, Henry Cohn, and Yufei Zhao

    Christian Borgs, Jennifer T. Chayes, Henry Cohn, and Yufei Zhao. An Lp theory of sparse graph convergence ii: LD convergence, quotients, and right convergence. The Annals of Probability, 46:337–396, 2018

    work page 2018

  8. [7]

    Functional Analysis, Sobolev Spaces and Partial Differential Equations

    Haim Brezis. Functional Analysis, Sobolev Spaces and Partial Differential Equations . Springer, New York, 2011

    work page 2011

Show all 63 references
  1. [8]

    Policy gradient-based algorithms for continuous-time linear quadratic control

    Jingjing Bu, Afshin Mesbahi, and Mehran Mesbahi. Policy gradient-based algorithms for continuous-time linear quadratic control. arXiv preprint arXiv:2006.09178 , 2020

    2006 arXiv

  2. [9]

    Caines and Minyi Huang

    Peter E. Caines and Minyi Huang. Graphon mean field games and their equations. SIAM Journal on Control and Optimization , 59:4373–4399, 2021

    2021

  3. [10]

    Financial Mathematics

    Ren´ e Carmona.Lectures on BSDEs, Stochastic Control, and Stochastic Differential Games with Financial Applications . Financial Mathematics. SIAM, Society for Industrial and Ap- plied Mathematics, Philadelphia, 2016

    2016

  4. [11]

    Cooney, Christy V

    Ren´ e Carmona, Daniel B. Cooney, Christy V. Graves, and Mathieu Lauri` ere. Stochastic graphon games: I. the static case. Mathematics of Operations Research, 47:750–778, 2022

    2022

  5. [12]

    Deep learning for mean field games and mean field control with applications to finance

    Ren´ e Carmona and Mathieu Lauri` ere. Deep learning for mean field games and mean field control with applications to finance. In Machine Learning and Data Sciences for Financial Markets, pages 369–392. Cambridge University Press, 2023. 41

    2023

  6. [13]

    Linear-quadratic mean-field reinforce- ment learning: Convergence of policy gradient methods

    Ren´ e Carmona, Mathieu Lauri` ere, and Zongjun Tan. Linear-quadratic mean-field reinforce- ment learning: Convergence of policy gradient methods. arXiv preprint arXiv:1910.04295 , 2019

    1910 arXiv

  7. [14]

    Approximately solving mean field games via entropy-regularized deep reinforcement learning

    Kai Cui and Heinz Koeppl. Approximately solving mean field games via entropy-regularized deep reinforcement learning. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , volume 130 of Proceedings of Machine Learning Research, pages 1...

    1909

  8. [15]

    Learning graphon mean field games and approximate Nash equilibria

    Kai Cui and Heinz Koeppl. Learning graphon mean field games and approximate Nash equilibria. In International Conference on Learning Representations , 2022

    2022

  9. [16]

    Dechevski and Lars Erik Persson

    Ljubomir T. Dechevski and Lars Erik Persson. Sharp generalized Carleman inequalities with minimal information about the spectrum. Mathematische Nachrichten , 168:61–77, 1994

    1994

  10. [17]

    On the convergence of model free learning in mean field games

    Romuald Elie, Julien P´ erolat, Mathieu Lauri` ere, Matthieu Geist, and Olivier Pietquin. On the convergence of model free learning in mean field games. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 7143–7150, 2020

    2020

  11. [18]

    Learning sparse graphon mean field games

    Christian Fabian, Kai Cui, and Heinz Koeppl. Learning sparse graphon mean field games. In International Conference on Artificial Intelligence and Statistics , volume 206, pages 4486–

  12. [19]

    Global convergence of policy gradient methods for the linear quadratic regulator

    Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. In Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research, pages 1467...

    2018

  13. [20]

    Full error analysis of policy gradient learning algorithms for exploratory linear quadratic mean-field control problem in continuous time with common noise

    Noufel Frikha, Huyˆ en Pham, and Xuanye Song. Full error analysis of policy gradient learning algorithms for exploratory linear quadratic mean-field control problem in continuous time with common noise. arXiv preprint arXiv:2408.02489 , 2024

    2024 arXiv

  14. [22]

    Shuang Gao, Rinel Foguen Tchuendom, and Peter E. Caines. Linear quadratic graphon field games. Communications in Information and Systems , 21:341–369, 2021

    2021

  15. [23]

    Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems

    Michael Giegrich, Christoph Reisinger, and Yufei Zhang. Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems. SIAM Journal on Control and Optimization , 62:1060–1092, 2024

    2024

  16. [24]

    Volterra Integral and Functional Equations

    Gustaf Gripenberg, Stig-Olof Londen, and Olof Staffans. Volterra Integral and Functional Equations. Cambridge University Press, 1990

    1990

  17. [25]

    Learning mean-field games

    Xin Guo, Anran Hu, Renyuan Xu, and Junzi Zhang. Learning mean-field games. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019

    2019

  18. [26]

    Policy gradient methods for the noisy lin- ear quadratic regulator over a finite horizon

    Ben Hambly, Renyuan Xu, and Huining Yang. Policy gradient methods for the noisy lin- ear quadratic regulator over a finite horizon. SIAM Journal on Control and Optimization , 59:3359–3391, 2021. 42

    2021

  19. [27]

    Policy gradient methods find the nash equilib- rium in n-player general-sum linear-quadratic games

    Ben Hambly, Renyuan Xu, and Huining Yang. Policy gradient methods find the nash equilib- rium in n-player general-sum linear-quadratic games. Journal of Machine Learning Research, 24:1–56, 2023

    2023

  20. [28]

    Mf-oml: Online mean-field reinforcement learning with occupa- tion measures for large population games

    Anran Hu and Junzi Zhang. Mf-oml: Online mean-field reinforcement learning with occupa- tion measures for large population games. arXiv preprint arXiv:2405.00282 , 2024

    2024 arXiv

  21. [29]

    Model-based RL for mean-field games is not statistically harder than single-agent RL

    Jiawei Huang, Niao He, and Andreas Krause. Model-based RL for mean-field games is not statistically harder than single-agent RL. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 19816– 19870....

    2024

  22. [30]

    On the statistical efficiency of mean-field reinforcement learning with general function approximation

    Jiawei Huang, Batuhan Yardim, and Niao He. On the statistical efficiency of mean-field reinforcement learning with general function approximation. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , volume 238 of Proceedings of Machi...

    2024

  23. [31]

    Malham´ e, and Peter E

    Minyi Huang, Roland P. Malham´ e, and Peter E. Caines. Large population stochastic dynamic games: Closed-loop McKean-Vlasov systems and the nash certainty equivalence principle. Communications in Information and Systems , 6:221 – 252, 2006

    2006

  24. [32]

    Reinforcement learning for SBM graphon games with re-sampling

    Peihan Huo, Oscar Peralta, Junyu Guo, Qiaomin Xie, and Andreea Minca. Reinforcement learning for SBM graphon games with re-sampling. arXiv preprint arXiv:2310.16326 , 2023

    2023 arXiv

  25. [33]

    Real-time bidding with multi-agent reinforcement learning in display advertising

    Junqi Jin, Chengru Song, Han Li, Kun Gai, Jun Wang, and Weinan Zhang. Real-time bidding with multi-agent reinforcement learning in display advertising. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management , pages 2193–2201, 2018

    2018

  26. [34]

    A natural policy gradient

    Sham M Kakade. A natural policy gradient. In Advances in Neural Information Processing Systems, volume 14. MIT Press, 2001

    2001

  27. [35]

    A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in polish spaces

    Bekzhan Kerimkulov, James-Michael Leahy, David Siska, Lukasz Szpruch, and Yufei Zhang. A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in polish spaces. arXiv preprint arXiv:2310.02951 , 2024

    2024 arXiv

  28. [36]

    Actor-critic algorithms

    Vijay Konda and John Tsitsiklis. Actor-critic algorithms. In Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999

    1999

  29. [37]

    A label-state formulation of stochastic graphon games and approximate equilibria on large networks.Mathematics of Operations Research, 48:1811–2382, 2022

    Daniel Lacker and Agathe Soret. A label-state formulation of stochastic graphon games and approximate equilibria on large networks.Mathematics of Operations Research, 48:1811–2382, 2022

    2022

  30. [38]

    Mean field games

    Jean-Michel Lasry and Pierre-Louis Lions. Mean field games. Japanese Journal of Mathem- atics, 2:229–260, 2007

    2007

  31. [39]

    Learning in mean field games: A survey

    Mathieu Lauri` ere, Sarah Perrin, Julien P´ erolat, Sertan Girgin, Paul Muller, Romuald ´Elie, Matthieu Geist, and Olivier Pietquin. Learning in mean field games: A survey. arXiv preprint arXiv:2205.12944, 2024

    2024 arXiv

  32. [40]

    American Mathematical Society Col- loquium Publications

    L´ aszl´ o Lov´ asz.Large Networks and Graph Limits . American Mathematical Society Col- loquium Publications. American Mathematical Society, 2012. 43

    2012

  33. [41]

    On the global con- vergence rates of softmax policy gradient methods

    Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global con- vergence rates of softmax policy gradient methods. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6820–...

    2020

  34. [42]

    Stochastic graphon games with memory

    Eyal Neuman and Sturmius Tuschmann. Stochastic graphon games with memory. arXiv preprint arXiv:2411.05896, 2024

    2024 arXiv

  35. [43]

    OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemys law D´ ebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal J´ ozefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d O. Pinto, Jona...

    1912 arXiv

  36. [44]

    Graphon games: A statistical framework for network games and interventions

    Francesca Parise and Asuman Ozdaglar. Graphon games: A statistical framework for network games and interventions. Econometrica, 91:191–225, 2023

    2023

  37. [45]

    Time discretization-invariant safe action re- petition for policy gradient methods

    Seohong Park, Jaekyeom Kim, and Gunhee Kim. Time discretization-invariant safe action re- petition for policy gradient methods. In Advances in Neural Information Processing Systems, volume 34, pages 267–279. Curran Associates, Inc., 2021

    2021

  38. [46]

    Entropy annealing for policy mirror descent in continuous time and space

    Deven Sethi, David ˇSiˇ ska, and Yufei Zhang. Entropy annealing for policy mirror descent in continuous time and space. arXiv preprint arXiv:2405.20250 , 2025

    2025 arXiv

  39. [47]

    Safe, multi-agent, reinforce- ment learning for autonomous driving

    Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforce- ment learning for autonomous driving. arXiv preprint arXiv:1610.03295 , 2016

    2016 arXiv

  40. [48]

    David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanc- tot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Made...

    2016

  41. [49]

    Deterministic policy gradient algorithms

    David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Ried- miller. Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference on Machine Learning , volume 32 of Proceedings of Machine Learning Research, pages 387–3...

    2014

  42. [50]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. Reinforcement learning. MIT Press, 2018

    2018

  43. [51]

    Policy gradient methods for reinforcement learning with function approximation

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, volume 12, pages 1057–1063. MIT Press, 1999

    1999

  44. [52]

    Caines, and Minyi Huang

    Rinel Foguen Tchuendom, Shuang Gao, Peter E. Caines, and Minyi Huang. Infinite horizon LQG graphon mean field games: Explicit Nash values and local minima. Systems & Control Letters, 187(105780), 2024

    2024

  45. [53]

    Ordinary Differential Equations and Dynamical Systems, volume 140 ofGradu- ate Studies in Mathematics

    Gerald Teschl. Ordinary Differential Equations and Dynamical Systems, volume 140 ofGradu- ate Studies in Mathematics . American Mathematical Society, Providence, 2012. 44

    2012

  46. [54]

    Global convergence of policy gradient for linear-quadratic mean-field control/game in continuous time

    Weichen Wang, Jiequn Han, Zhuoran Yang, and Zhaoran Wang. Global convergence of policy gradient for linear-quadratic mean-field control/game in continuous time. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learn...

    2021

  47. [55]

    Learning while playing in mean-field games: Convergence and optimality

    Qiaomin Xie, Zhuoran Yang, Zhaoran Wang, and Andreea Minca. Learning while playing in mean-field games: Convergence and optimality. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11436–1144...

    2021

  48. [56]

    Linear-quadratic graphon mean field games with common noise

    De xuan Xu, Zhun Gou, Nan jing Huang, and Shuang Gao. Linear-quadratic graphon mean field games with common noise. arXiv preprint arXiv:2401.09030 , 2024

    2024 arXiv

  49. [57]

    Policy mirror ascent for efficient and independent learning in mean field games

    Batuhan Yardim, Semih Cayci, Matthieu Geist, and Niao He. Policy mirror ascent for efficient and independent learning in mean field games. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 397...

    2023

  50. [58]

    When is mean-field reinforcement learning tractable and relevant? In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems , page 2038–2046

    Batuhan Yardim, Artur Goldman, and Niao He. When is mean-field reinforcement learning tractable and relevant? In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems , page 2038–2046. International Foundation for Autonomous Agents and Mu...

    2024

  51. [59]

    Stochastic Controls: Hamiltonian Systems and HJB Equations

    Jiongmin Yong and Xun Yu Zhou. Stochastic Controls: Hamiltonian Systems and HJB Equations. Stochastic Modelling and Applied Probability. Springer, New York, 1999

    1999

  52. [60]

    Learning regularized monotone graphon mean-field games

    Fengzhuo Zhang, Vincent Tan, Zhaoran Wang, and Zhuoran Yang. Learning regularized monotone graphon mean-field games. In Advances in Neural Information Processing Systems, volume 36, pages 67297–67308. Curran Associates, Inc., 2023

    2023

  53. [61]

    Fengzhuo Zhang, Vincent Y. F. Tan, Zhaoran Wang, and Zhuoran Yang. Learning regularized graphon mean-field games with unknown graphons. Journal of Machine Learning Research , 25:1–95, 2024

    2024

  54. [62]

    Convergence of policy gradient for stochastic linear quad- ratic optimal control problems in infinite horizon

    Xinpei Zhang and Guangyan Jia. Convergence of policy gradient for stochastic linear quad- ratic optimal control problems in infinite horizon. Journal of Mathematical Analysis and Applications, 547(129264), 2025

    2025

  55. [63]

    Graphon mean field games with a representative player: Analysis and learning algorithm

    Fuzhong Zhou, Chenyu Zhang, Xu Chen, and Xuan Di. Graphon mean field games with a representative player: Analysis and learning algorithm. In Proceedings of the 41st Inter- national Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research, pages 6...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.