Pith. sign in

REVIEW 3 major objections 6 minor 99 references

Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean-Field Games

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proves that a trust-region policy optimization algorithm, alternating an entropy-regularized TRPO policy update with a mean-field population update, reaches an ε-approximate mean-field Nash equilibrium in finite ergodic games…

desk verdict Genuinely new TRPO-for-MFG analysis with clean exact proofs, but the 'model-free' label overstates the oracle: the step rule takes µ as a query argument, so the Õ(1/ε^6) guarantee is really for a mean-field simulator. read the letter →

arxiv 2505.22781 v1 pith:HFQM5I4M submitted 2025-05-28 stat.ML cs.LGmath.STstat.TH

Antonio Ocello , Daniil Tiapkin , Lorenzo Mancini , Mathieu Laurière , Eric Moulines This is my paper
classification stat.MLcs.LGmath.STstat.TH MSC 91A1690C40
keywords mean-fieldgamestrustregionpolicyoptimizationNashequilibriumsamplecomplexityentropyregularizationmirrordescentreinforcementlearningexploitability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Mean-field games approximate many-agent interactions by a representative agent facing a population distribution. The paper claims that a trust-region policy optimization algorithm, built from an inner entropy-regularized TRPO loop and an outer population-distribution update, converges to an approximate mean-field Nash equilibrium in finite state-action ergodic games. For the sample-based version it proves a high-probability bound on exploitability and an overall cost of Õ(1/$ε^{6}$) environment calls. The result matters because it brings model-free TRPO-style guarantees to mean-field games under weaker assumptions than prior proximal methods, which often required policies to stay uniformly randomized.

What carries the argument

The load-bearing object is the two-loop iteration: an inner entropy-regularized TRPO loop whose policy update is a closed-form softmax mirror-descent step π_{ℓ+1}(a|s) ∝ π_ℓ(a|s) exp(α_ℓ(Q(s,a) − η log π_ℓ(a|s))), and an outer mean-field update that moves μ toward the M-step forward population μ($P^{{π}}$_μ)^M. The proof hinges on the exploitability decomposition φ(π,μ) ≤ [max_{π'} J(π',μ,μ) − J(π,μ,μ)] + C_φ‖$λ^{{π,μ}}$ − μ‖₂, which separates how wrong the policy is from how far the population is from its own stationary distribution. Assumptions A-1 to A-4, namely Lipschitzness, a monotone contraction of the M-step update, unichain ergodicity, and a finite concentrability coefficient, are what make the two errors contract at geometric and O(log L/L) rates.

What would settle it

Implement the algorithm on a small finite mean-field game satisfying Assumptions A-1 to A-4 whose equilibrium is computable by policy iteration; with the ν-restart oracle, record the number of environment calls needed to make the exploitability of the output uniform-mixture policy first fall below ε. If this count grows faster than Õ(1/$ε^{6}$), or never falls, the central sample-complexity claim is wrong.

Watch

Extended reading notes

Core claim

The central claim is that alternating a TRPO best-response step for a fixed population μ with an M-step population update μ_{k+1} = μ_k + β_k(μ_k($P^{{π_k}}$_{μ_k})^M − μ_k) drives the pair to the unique ε-MFNE. Exact MF-TRPO attains exploitability ε_K ≤ C exp(−(τ/4)Σ_{j=1}^K β_j) + C√(log L/L), and Sample-Based MF-TRPO attains a matching high-probability bound with extra sampling-error terms, as stated in Corollary D.5. The key point is that the equilibrium error decomposes into a policy-value gap and a population-stationarity gap, each controlled by a different loop, yielding a total sample complexity of Õ(1/$ε^{6}$) without requiring policies to be uniformly bounded away from zero.

Load-bearing premise

The sample-based algorithm assumes the environment oracle lets the learner reset and query transitions under any chosen population distribution μ, whereas in a genuinely multi-agent system the population distribution is an emerging statistic the learner cannot simply dial in.

Editorial extensions

If this is right

  • A model-free learner can find an ε-MFNE in a finite ergodic mean-field game with Õ(1/ε^6) calls to a ν-restart environment, matching the sample rate of single-agent TRPO.
  • Previous assumptions forcing policies to be uniformly bounded away from zero, or forcing the induced Markov chain to be absolutely continuous with respect to the uniform distribution, are relaxed; unichain ergodicity plus the monotonicity condition A-2 suffice.
  • The sample-based guarantee applies to the uniform mixture of the inner TRPO iterates, not the last iterate, so averaging over iterates is what the theory protects.
  • To reach an ε-MFNE, the paper sets the inner horizon L = Õ(1/ε²), the outer iteration count K = Õ(log(1/ε)), and the per-iteration episode counts P and I_ℓ = Õ(1/ε⁴), giving the total Õ(1/ε^6) call complexity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The two-error decomposition suggests that any policy-optimization method with a decaying value gap and a contractive population update would inherit the same Õ(1/ε^6) complexity, so the specific TRPO update may not be essential to the rate.
  • If the environment oracle cannot accept μ as an input, because the population distribution only emerges from play, the algorithm's inner rollouts cannot be generated as written; an online variant that uses the realized μ_t would be a natural extension whose rate the paper leaves open.
  • Since the theorem protects the uniform mixture of inner policies, a practical implementation should sample from that mixture via the paper's Uniform-Mixture subroutine; comparing last-iterate versus mixture exploitability in the grid experiments would test how conservative the averaging guarantee is.
  • The Õ(1/ε^6) result is an upper bound, and the paper provides no matching lower bound, so it is plausible that a tighter analysis or a different algorithm could reach ε^4 or ε^5 in this mean-field setting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper proposes MF-TRPO, an entropy-regularized trust-region policy optimization algorithm for finite state-action mean-field games, and analyzes both an exact version (Algorithms 1–2) and a sample-based version (Algorithms 6–8). The exact version combines an inner TRPO loop, whose convergence is imported from Shani et al. (2020), with an outer mean-field population update; under Assumptions A-1–A-3 the authors prove an exploitability bound of the form O(exp(-τ∑β_j/4) + sqrt(log L / L)). The sample-based version adds sampling error from inner Q-estimation and outer population rollouts and claims, under Assumptions A-1–A-4 and a ν-restart oracle, a high-probability exploitability bound with total Õ(1/ε^6) environment interactions to reach an ε-MFNE. The main text presents the algorithms and informal statements; the appendices contain the proofs and the sample-complexity accounting.

Significance. If the claims hold, this would be the first finite-sample high-probability guarantee for TRPO-type methods in mean-field games with explicit population updates, under assumptions weaker than uniform exploration, and the rate Õ(1/ε^6) would align with single-agent regularized TRPO. The paper is generally careful with constants, clearly separates the inner-loop bound (Shani et al.) from the outer contraction argument, and the exact version's analysis is plausible. However, the sample-based claim's applicability to model-free mean-field games depends on an environment oracle that accepts the mean-field distribution as an input; this is a substantive gap, along with a missing union bound over the outer iterations and an apparent arithmetic error in the complexity statement.

major comments (3)
  1. [Section 2 (Interactions with the environment) and Algorithm 8] The environment oracle is defined so that the step operation takes the mean-field distribution profile µ as an input and samples from P(·|s,a,µ). Algorithm 6 lines 8-9 and Algorithm 8 line 15 both generate rollouts under a user-specified \hatµ_{k-1}. In a model-free mean-field game, µ is an emergent population statistic rather than a queryable control; resetting the initial state to ν does not reset the population. Therefore the finite-sample guarantee in Corollary D.5 and the Õ(1/ε^6) complexity in Remark D.7 do not apply to the claimed model-free setting unless the authors either reframe the result as requiring a mean-field simulator (a generative-model-strength assumption at the population level) or modify the algorithm to use only transitions under the realized µ. This is load-bearing because it determines whether the algorithm can be executed at all outside the oracle.
  2. [Theorem D.4 and Corollary D.5] The high-probability statements are not union-bounded over the K outer iterations. Theorem D.2 and Proposition D.3 each hold with probability at least 1-δ for a fixed call (a fixed µ, a fixed k), but the recursion in Theorem D.4 requires all K outer iterations and all L inner iterations to be simultaneously good. As written, one needs to replace δ by δ/(K L) (or an equivalent schedule) in the trajectory counts, and while this only changes polylogarithmic factors, the stated 'with probability at least 1-δ' bound is not justified without that step. This affects the main high-probability exploitability guarantee.
  3. [Remark D.7] The sample-complexity accounting is internally inconsistent. With P and I_ℓ both declared to be Õ(1/ε^4), the claimed mean-field update cost Õ(P × I_ℓ × K) is Õ(1/ε^8), not Õ(1/ε^2), and the subsequent overall Õ(1/ε^6) total does not follow from the stated component costs. Moreover, the mean-field update in Algorithm 8 uses P rollouts per outer iteration, not P × I_ℓ rollouts, so the displayed product conflates the inner policy-estimation budget with the population-estimation budget. The authors should provide a corrected end-to-end count of environment calls over K outer iterations.
minor comments (6)
  1. [Assumption A-2] The displayed inequality in Assumption A-2 is missing the universal quantifier over µ and µ′; it currently reads 'for µ ∈ P(S)' even though both µ and µ′ appear. Also, the condition is an upper bound on an inner product and is used in the proofs as a one-sided Lipschitz bound, so calling it 'monotonicity' without further explanation is confusing.
  2. [Section 5, paragraph after Proposition 5.2] The sentence 'as explicit knowledge of the environment's transition dynamics or reward structure is required' should read 'is not required'; as written it contradicts the stated model-free motivation of Sample-Based MF-TRPO.
  3. [Algorithm 8, line 15] The notation 'P_{a∈A} P(·|s_{m-1,p,k}, a, \hatµ_{k-1}) \hatπ_k(a|s_{m-1,p,k})' should be written as an explicit sum over a; as displayed it could be misread as a product or a set-valued transition.
  4. [Appendix D.4, equation (28)] In the recursion displayed before equation (28), the term '∥\hatµ_{k−1} + µ⋆∥²' appears where the minus sign is clearly intended; this is a typo in an otherwise detailed proof.
  5. [Proposition D.3] The proof applies Hoeffding's inequality to an ℓ1-type deviation and then concludes a bound on the Euclidean norm ∥ϵ_k∥₂ ≤ ε; the intermediate display 'P(1/P ∑ ... ≥ ε/4)' and the final constant 64 should be reconciled, since the displayed argument appears to give an ℓ1 bound with a slightly different constant.
  6. [Remark D.1 and Corollary D.5] The notation for the uniform mixture policy is inconsistent: Remark D.1 and Algorithm 6 use \hatπ^{Unif,µ}_L, while Corollary D.5 writes \hatπ^{Unif,\hatµ_k}_L. The two should be harmonized to avoid confusion about which mean-field parameter the mixture is evaluated under.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: the MF-TRPO guarantees are derived from explicit assumptions plus the external Shani et al. TRPO bound; the only self-citations are definitional and non-load-bearing.

full rationale

The derivation chain is self-contained relative to its stated assumptions. Exact MF-TRPO convergence (Theorem C.5) follows from Assumptions A-1, A-2, A-3 and Theorem C.1, which is Theorem 16 of Shani et al. (2020), an external bound with no fitted target conclusion. The supporting lemmas on Lipschitzness of optimal policies, value gaps, and exploitability (Propositions E.2-E.5, Corollary E.4) are proved from the assumptions. The sample-based analysis (Theorem D.2, Theorem D.4, Corollary D.5) adds high-probability estimation lemmas (Proposition D.3) and the external Theorem 5 of Shani et al.; no fitted parameter is renamed as a prediction, and the õ(1/ε^6) complexity is obtained by multiplying the explicitly stated L, Iℓ, P, and K scalings. Assumption A-2 is stated as an assumption rather than derived as a conclusion, and a sufficient primitive condition for it is proved in Lemma E.6 from A-1 and A-3, so the monotonicity assumption is not circularly imported. The only self-citations are to Laurière et al. (2022) for the discounted-stationary MFG formulation and the exploitability definition; these frame the problem but are not load-bearing for the convergence rates. A genuine scope caveat, not a circularity, is that the interaction oracle in Section 2 lets the algorithm query transitions under a user-supplied population profile μ; this is a modeling-assumption concern about whether the algorithm is truly model-free, not a circular reuse of the target result. Some typographical and notational inconsistencies (e.g., k versus K in Corollary 5.3, and the garbled sentence in Section 5 about explicit knowledge being required) do not constitute circularity. Overall the circularity burden is very low.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim rests on four structural assumptions (A-1 to A-4), a strong environment oracle that accepts a user-specified mean-field distribution, and two external TRPO theorems from Shani et al. (2020). No new physical or mathematical entities are postulated. The algorithm hyperparameters η, M, βk, and γ appear in the bounds and are hand-chosen in the experiments, not fitted to data.

free parameters (4)
  • entropy regularization weight η = 0.05 and 0.3 in experiments
    Appears in the objective and the step size αℓ = 1/(η(ℓ+2)). The theory holds for any fixed η>0; the reported bounds depend on it. Not fitted, but hand-chosen.
  • mean-field update horizon M = 100 in experiments
    The population update uses the M-step transition operator. Assumption A-2 requires some M with Cop,MFG<1; the bounds depend on M through CErg,M.
  • outer step size βk = 0.01 (exact), 0.1 (sample-based) in experiments
    Must respect the upper bounds (17)/(23). Chosen constant in experiments; the theory requires Σβk = ∞.
  • discount factor γ = 0.9 in experiments
    Defines the discounted approximation of the ergodic MFG; constants in all bounds depend on 1/(1-γ).
assumptions (6)
  • domain assumption Assumption A-1: r and P are Lipschitz in µ (Lr_µ, LP_µ)
    Smoothness of the MFG data; used to bound policy and value sensitivity in Corollary E.4, Lemma E.1, and the exploitability decomposition.
  • domain assumption Assumption A-2: M-step mean-field operator is strongly monotone with Cop,MFG<1
    Yields uniqueness of the fixed point (8) and the exponential contraction of the outer loop. A sufficient primitive condition is derived in Lemma E.6, but the condition itself is assumed to hold.
  • domain assumption Assumption A-3: unichain plus uniform mixing with rate CErg ρ^t
    Gives a unique stationary distribution and exponential mixing, used throughout the policy and population error bounds in Lemma E.1 and Theorem C.1.
  • domain assumption Assumption A-4: concentrability of discounted occupation measure d^{πµ}_{µ,µ}/ν is bounded
    Required by the sample-based TRPO theorem D.2 to control importance-weighting errors under the ν-restart oracle.
  • domain assumption Environment oracle: reset to ν and step with a user-specified µ
    The sample-based algorithm's rollouts and Q-estimates require querying P(·|s,a,µ) for arbitrary µ supplied by the algorithm, stated in Section 2. In a true model-free MFG, µ is emergent, not a queryable input.
  • standard math Shani et al. (2020), Theorem 16 (exact TRPO value-gap bound) and Theorem 5 (sample-based TRPO bound)
    Imported as black boxes; the paper marks Theorem C.1 as 'Theorem 16 in Shani et al.' and Theorem D.2 as 'Based on Theorem 5'. The central rates inherit from these external results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean-Field Games." pith.science (2026). https://pith.science/paper/HFQM5I4M

@misc{pith2026250522781,
  author       = {Pith},
  title        = {Pith review of: Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean-Field Games},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HFQM5I4M}},
  note         = {Machine review of arXiv:2505.22781}
}
read the original abstract

We introduce Mean-Field Trust Region Policy Optimization (MF-TRPO), a novel algorithm designed to compute approximate Nash equilibria for ergodic Mean-Field Games (MFG) in finite state-action spaces. Building on the well-established performance of TRPO in the reinforcement learning (RL) setting, we extend its methodology to the MFG framework, leveraging its stability and robustness in policy optimization. Under standard assumptions in the MFG literature, we provide a rigorous analysis of MF-TRPO, establishing theoretical guarantees on its convergence. Our results cover both the exact formulation of the algorithm and its sample-based counterpart, where we derive high-probability guarantees and finite sample complexity. This work advances MFG optimization by bridging RL techniques with mean-field decision-making, offering a theoretically grounded approach to solving complex multi-agent problems.

Figures

Figures reproduced from arXiv: 2505.22781 by the authors.

Figure 1
Figure 1. Exploitability achieved by the Sample-Based MF-TRPO algorithm in the 5 × 5 Grid-Based Crowd Modeling game with the bottom-right corner being a point of interest. The left plot corresponds to η = 0.05, and the right to η = 0.3 with results averaged over 10 and 3 random seeds, respectively. The environment considered is a Grid-Based Crowd Model￾ing game where, from a given initial distribution, agents are tasked with … view at source ↗
Figure 2
Figure 2. Evolution of the mean field distribution for η = 0.05 in the 5×5 Grid-Based Crowd Modeling game with the bottom-right corner being a point of interest. From left to right: step 0, step 10 and step 200. 7. Conclusion In this work, we introduced Exact MF-TRPO, a novel algorithm for computing MFNE in ergodic MFG. By lever￾aging the trust-region policy optimization framework, we established explicit non-asymptotic conve… view at source ↗
Figure 3
Figure 3. The reading order is (from left to right): Four Rooms Crowd Modeling, Two-Islands-Graph Crowd Modeling, and Four Rooms Crowd Modeling with a point of interest. Solid lines denote η = 0.05, whereas dashed lines indicate η = 0.3. 44 [PITH_FULL_IMAGE:figures/full_fig_p044_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Evolution of the mean field distribution for η = 0.05 in the Four Rooms Crowd Modeling game. From left to right: step 0, step 1000 and step 5000. 45 [PITH_FULL_IMAGE:figures/full_fig_p045_4.png]
Figure 5
Figure 5. Figure 5: Evolution of the mean field distribution for η = 0.05 in the Two-Islands Graph Crowd Modeling game. From left to right: step 0, step 2000 and step 5000 [PITH_FULL_IMAGE:figures/full_fig_p046_5.png]
Figure 6
Figure 6. Figure 6: Evolution of the mean field distribution for η = 0.05 in the Four Rooms Graph Crowd Modeling game with the bottom-right corner being a point of interest. From left to right: step 0, step 1000 and step 5000. 46 [PITH_FULL_IMAGE:figures/full_fig_p046_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

99 extracted references · 63 canonical work pages

  1. [1]

    and Capuzzo-Dolcetta, I

    Achdou, Y. and Capuzzo-Dolcetta, I. Mean field games: numerical methods. SIAM Journal on Numerical Analysis, 48 0 (3): 0 1136--1162, 2010

    2010

  2. [2]

    and Porretta, A

    Achdou, Y. and Porretta, A. Convergence of a finite difference scheme to weak solutions of the system of partial differential equations arising in mean field games. SIAM Journal on Numerical Analysis, 54 0 (1): 0 161--186, 2016

    2016

  3. [3]

    Mean field games: numerical methods for the planning problem

    Achdou, Y., Camilli, F., and Capuzzo-Dolcetta, I. Mean field games: numerical methods for the planning problem. SIAM Journal on Control and Optimization, 50 0 (1): 0 77--109, 2012

    2012

  4. [4]

    Mean field games and applications: Numerical aspects

    Achdou, Y., Cardaliaguet, P., Delarue, F., Porretta, A., Santambrogio, F., Achdou, Y., and Lauri \`e re, M. Mean field games and applications: Numerical aspects. Mean Field Games: Cetraro, Italy 2019, pp.\ 249--307, 2020

    2019

  5. [5]

    Agarwal, A., Kakade, S., and Yang, L. F. Model-based reinforcement learning with a generative model is minimax optimal. In Conference on Learning Theory, pp.\ 67--83. PMLR, 2020

    2020

  6. [6]

    An extended mean field game for storage in smart grids

    Alasseur, C., Ben Taher, I., and Matoussi, A. An extended mean field game for storage in smart grids. Journal of Optimization Theory and Applications, 184: 0 644--670, 2020

    2020

  7. [7]

    Regularization of the policy updates for stabilizing mean field games

    Algumaei, T., Solozabal, R., Alami, R., Hacid, H., Debbah, M., and Tak \'a c , M. Regularization of the policy updates for stabilizing mean field games. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp.\ 361--372. Springer, 2023

    2023

  8. [8]

    D., and Saldi, N

    Anahtarci, B., Kariksiz, C. D., and Saldi, N. Learning mean-field games with discounted and average costs. Journal of Machine Learning Research, 24 0 (17): 0 1--59, 2023 a

    2023

Show all 99 references
  1. [9]

    D., and Saldi, N

    Anahtarci, B., Kariksiz, C. D., and Saldi, N. Q-learning in regularized mean-field games. Dynamic Games and Applications, 13 0 (1): 0 89--117, 2023 b

    2023

  2. [10]

    Mean-field sampling for cooperative multi-agent reinforcement learning

    Anand, E., Karmarkar, I., and Qu, G. Mean-field sampling for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2412.00661, 2024

    2024

  3. [11]

    Unified reinforcement q-learning for mean field game and control problems

    Angiuli, A., Fouque, J.-P., and Laurière, M. Unified reinforcement q-learning for mean field game and control problems. arXiv preprint arXiv:2006.13912, 2021

    2006 arXiv

  4. [12]

    Convergence of multi-scale reinforcement Q learning algorithms for mean field game and control problems

    Angiuli, A., Fouque, J.-P., Lauri \`e re, M., and Zhang, M. Convergence of multi-scale reinforcement Q learning algorithms for mean field game and control problems. arXiv preprint arXiv:2312.06659, 2023

    2023 arXiv

  5. [13]

    On solutions of mean field games with ergodic cost

    Arapostathis, A., Biswas, A., and Carroll, J. On solutions of mean field games with ergodic cost. Journal de Math \'e matiques Pures et Appliqu \'e es , 107 0 (2): 0 205--251, 2017

    2017

  6. [14]

    Lipschitz continuity in model-based reinforcement learning

    Asadi, K., Misra, D., and Littman, M. Lipschitz continuity in model-based reinforcement learning. In International Conference on Machine Learning, pp.\ 264--273. PMLR, 2018

    2018

  7. [15]

    Inapproximability of np-complete variants of nash equilibrium

    Austrin, P., Braverman, M., and Chlamt \'a c , E. Inapproximability of np-complete variants of nash equilibrium. In International Workshop on Approximation Algorithms for Combinatorial Optimization, pp.\ 13--25. Springer, 2011

    2011

  8. [16]

    M., Munos, R., and Kappen, H

    Azar, G. M., Munos, R., and Kappen, H. J. Minimax pac bounds on the sample complexity of reinforcement learning with a generative model. Machine learning, 91: 0 325--349, 2013

    2013

  9. [17]

    and Priuli, F

    Bardi, M. and Priuli, F. S. Linear-quadratic n-person and mean-field games with ergodic cost. SIAM Journal on Control and Optimization, 52 0 (5): 0 3022--3052, 2014

    2014

  10. [18]

    A mean-field game model of electricity market dynamics

    Bassi \`e re, A., Dumitrescu, R., and Tankov, P. A mean-field game model of electricity market dynamics. In Quantitative Energy Finance: Recent Trends and Developments, pp.\ 181--219. Springer, 2024

    2024

  11. [19]

    and Hesse, S

    Becherer, D. and Hesse, S. Common noise by random measures: Mean-field equilibria for competitive investment and hedging. arXiv preprint arXiv:2408.01175, 2024

    2024

  12. [20]

    First-order methods in optimization

    Beck, A. First-order methods in optimization. SIAM, 2017

    2017

  13. [21]

    and Russo, D

    Bhandari, J. and Russo, D. Global optimality guarantees for policy gradient methods. Operations Research, 2024

    2024

  14. [22]

    A., Ortega, P

    Braun, D. A., Ortega, P. A., Theodorou, E., and Schaal, S. Path integral control and bounded rationality. In 2011 IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL), pp.\ 202--209. IEEE, 2011

    2011

  15. [23]

    The master equation and the convergence problem in mean field games:(ams-201)

    Cardaliaguet, P., Delarue, F., Lasry, J.-M., and Lions, P.-L. The master equation and the convergence problem in mean field games:(ams-201). Princeton University Press, 2019

    2019

  16. [24]

    and Lauri \`e re, M

    Carmona, R. and Lauri \`e re, M. Convergence analysis of machine learning algorithms for the numerical solution of mean field control and games i: The ergodic case. SIAM Journal on Numerical Analysis, 59 0 (3): 0 1455--1485, 2021

    2021

  17. [25]

    Mean field games and systemic risk

    Carmona, R., Fouque, J.-P., and Sun, L.-H. Mean field games and systemic risk. arXiv preprint arXiv:1308.2172, 2013

    2013 arXiv

  18. [26]

    Probabilistic theory of mean field games with applications I-II

    Carmona, R., Delarue, F., et al. Probabilistic theory of mean field games with applications I-II. Springer, 2018

    2018

  19. [27]

    Numerical method for fbsdes of mckean--vlasov type

    Chassagneux, J.-F., Crisan, D., and Delarue, F. Numerical method for fbsdes of mckean--vlasov type. The Annals of Applied Probability, 29 0 (3): 0 1640--1684, 2019

    2019

  20. [28]

    Cover, T. M. Elements of information theory. John Wiley & Sons, 1999

    1999

  21. [29]

    and Koeppl, H

    Cui, K. and Koeppl, H. Approximately solving mean field games via entropy-regularized deep reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp.\ 1909--1917. PMLR, 2021

    1909

  22. [30]

    and Koeppl, H

    Cui, K. and Koeppl, H. Learning graphon mean field games and approximate nash equilibria. In International Conference on Learning Representations, 2022

    2022

  23. [31]

    A mean field game analysis of sir dynamics with vaccination

    Doncel, J., Gast, N., and Gaujal, B. A mean field game analysis of sir dynamics with vaccination. Probability in the Engineering and Informational Sciences, 36 0 (2): 0 482--499, 2022

    2022

  24. [32]

    and Touzi, N

    Espinosa, G.-E. and Touzi, N. Optimal investment under relative performance concerns. Mathematical Finance, 25 0 (2): 0 221--257, 2015

    2015

  25. [33]

    and Silva, F

    Fischer, M. and Silva, F. J. On the asymptotic nature of first order mean field games. Applied Mathematics & Optimization, 84: 0 2327--2357, 2021

    2021

  26. [34]

    N-player games and mean field games of moderate interactions

    Flandoli, F., Ghio, M., and Livieri, G. N-player games and mean field games of moderate interactions. Applied Mathematics & Optimization, 85 0 (3): 0 38, 2022

    2022

  27. [35]

    Counterfactual multi-agent policy gradients

    Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. Counterfactual multi-agent policy gradients. Proceedings of the AAAI conference on artificial intelligence, 32 0 (1), 2018

    2018

  28. [36]

    Convergence of adaptive and interacting markov chain monte carlo algorithms

    Fort, G., Moulines, E., and Priouret, P. Convergence of adaptive and interacting markov chain monte carlo algorithms. The Annals of Statistics, 39 0 (6): 0 3262--3289, 2011

    2011

  29. [37]

    Taming the noise in reinforcement learning via soft updates

    Fox, R., Pakman, A., and Tishby, N. Taming the noise in reinforcement learning via soft updates. Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence, 2016

    2016

  30. [38]

    A theory of regularized markov decision processes

    Geist, M., Scherrer, B., and Pietquin, O. A theory of regularized markov decision processes. In International Conference on Machine Learning, pp.\ 2160--2169. PMLR, 2019

    2019

  31. [39]

    Concave utility reinforcement learning: the mean-field game viewpoint, 2022

    Geist, M., Pérolat, J., Laurière, M., Elie, R., Perrin, S., Bachem, O., Munos, R., and Pietquin, O. Concave utility reinforcement learning: the mean-field game viewpoint, 2022. URL https://arxiv.org/abs/2106.03787

    2022 arXiv

  32. [40]

    Numerical resolution of mckean-vlasov fbsdes using neural networks

    Germain, M., Mikael, J., and Warin, X. Numerical resolution of mckean-vlasov fbsdes using neural networks. Methodology and Computing in Applied Probability, 24 0 (4): 0 2557--2586, 2022

    2022

  33. [41]

    and Diepold, K

    Gronauer, S. and Diepold, K. Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review, 55 0 (2): 0 895--943, 2022

    2022

  34. [42]

    Learning mean-field games

    Guo, X., Hu, A., Xu, R., and Zhang, J. Learning mean-field games. Advances in neural information processing systems, 32, 2019

    2019

  35. [43]

    A general framework for learning mean-field games

    Guo, X., Hu, A., Xu, R., and Zhang, J. A general framework for learning mean-field games. Mathematics of Operations Research, 48 0 (2): 0 656--686, 2023

    2023

  36. [44]

    Reinforcement learning with deep energy-based policies

    Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pp.\ 1352--1361. PMLR, 2017

    2017

  37. [45]

    J., Liaw, C., Plan, Y., and Randhawa, S

    Harvey, N. J., Liaw, C., Plan, Y., and Randhawa, S. Tight analyses for non-smooth stochastic gradient descent. In Conference on Learning Theory, pp.\ 1579--1613. PMLR, 2019

    2019

  38. [46]

    Howard, R. A. Dynamic programming and markov processes. John Wiley, 1960

    1960

  39. [47]

    Howard, R. A. and Matheson, J. E. Risk-sensitive markov decision processes. Management science, 18 0 (7): 0 356--369, 1972

    1972

  40. [48]

    E., and Malham \'e , R

    Huang, M., Caines, P. E., and Malham \'e , R. P. Individual and mass behaviour in large population stochastic wireless power control problems: centralized and nash equilibrium solutions. In 42nd IEEE international conference on decision and control (IEEE cat. No. 03CH37475), v...

    2003

  41. [49]

    P., and Caines, P

    Huang, M., Malham \'e , R. P., and Caines, P. E. Nash equilibria for large-population linear stochastic systems of weakly coupled agents. In Analysis, control and optimization of complex dynamic systems, pp.\ 215--252. Springer, 2005

    2005

  42. [50]

    P., and Caines, P

    Huang, M., Malham \'e , R. P., and Caines, P. E. Large population stochastic dynamic games: closed-loop mckean-vlasov systems and the nash certainty equivalence principle. Commun. Inf. Syst., 6 0 (1): 0 221--252, 2006 a

    2006

  43. [51]

    P., and Caines, P

    Huang, M., Malham \'e , R. P., and Caines, P. E. Nash certainty equivalence in large population stochastic dynamic games: Connections with the physics of interacting particle systems. In Proceedings of the 45th IEEE Conference on Decision and Control, pp.\ 4921--4926. IEEE, 2006 b

    2006

  44. [52]

    and Sha, F

    Iqbal, S. and Sha, F. Actor-attention-critic for multi-agent reinforcement learning. In International conference on machine learning, pp.\ 2961--2970. PMLR, 2019

    2019

  45. [53]

    and Langford, J

    Kakade, S. and Langford, J. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pp.\ 267--274, 2002

    2002

  46. [54]

    Kakade, S. M. On the sample complexity of reinforcement learning. University of London, University College London (United Kingdom), 2003

    2003

  47. [55]

    and Singh, S

    Kearns, M. and Singh, S. Finite-sample convergence rates for q-learning and indirect algorithms. Advances in neural information processing systems, 11, 1998

    1998

  48. [56]

    A., and Peters, J

    Kober, J., Bagnell, J. A., and Peters, J. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32 0 (11): 0 1238--1274, 2013

    2013

  49. [57]

    and Zariphopoulou, T

    Lacker, D. and Zariphopoulou, T. Mean field and n-agent games for optimal investment under relative performance criteria. Mathematical Finance, 29 0 (4): 0 1003--1038, 2019

    2019

  50. [58]

    and Lions, P.-L

    Lasry, J.-M. and Lions, P.-L. Jeux \`a champ moyen. i--le cas stationnaire. Comptes Rendus Math \'e matique , 343 0 (9): 0 619--625, 2006 a

    2006

  51. [59]

    and Lions, P.-L

    Lasry, J.-M. and Lions, P.-L. Jeux \`a champ moyen. ii--horizon fini et contr \^o le optimal. Comptes Rendus. Math \'e matique , 343 0 (10): 0 679--684, 2006 b

    2006

  52. [60]

    and Lions, P.-L

    Lasry, J.-M. and Lions, P.-L. Mean field games. Japanese journal of mathematics, 2 0 (1): 0 229--260, 2007

    2007

  53. [61]

    Learning mean field games: A survey

    Lauri \`e re, M., Perrin, S., Geist, M., and Pietquin, O. Learning mean field games: A survey. arXiv preprint arXiv:2205.12944, pp.\ 19--49, 2022

    2022 arXiv

  54. [62]

    and Tankov, P

    Lavigne, P. and Tankov, P. Decarbonization of financial markets: a mean-field game approach. arXiv preprint arXiv:2301.09163, 2023

    2023

  55. [63]

    G., and Castro, P

    Le Lan, C., Bellemare, M. G., and Castro, P. S. Metrics and continuity in reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35 0 (9): 0 8261--8269, 2021

    2021

  56. [64]

    A mean-field game approach to cloud resource management with function approximation

    Mao, W., Qiu, H., Wang, C., Franke, H., Kalbarczyk, Z., Iyer, R., and Basar, T. A mean-field game approach to cloud resource management with function approximation. Advances in Neural Information Processing Systems, 35: 0 36243--36258, 2022

    2022

  57. [65]

    I., Fern \'a ndez-Gaucherand, E., Hern \'a ndez-Hernandez, D., Coraluppi, S., and Fard, P

    Marcus, S. I., Fern \'a ndez-Gaucherand, E., Hern \'a ndez-Hernandez, D., Coraluppi, S., and Fard, P. Risk sensitive markov decision processes. In Systems and control in the twenty-first century, pp.\ 263--279. Springer, 1997

    1997

  58. [66]

    J., and Le Fort-Piat, N

    Matignon, L., Laurent, G. J., and Le Fort-Piat, N. Hysteretic q-learning: an algorithm for decentralized reinforcement learning in cooperative multi-agent teams. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.\ 64--69. IEEE, 2007

    2007

  59. [67]

    On the global convergence rates of softmax policy gradient methods

    Mei, J., Xiao, C., Szepesvari, C., and Schuurmans, D. On the global convergence rates of softmax policy gradient methods. In International conference on machine learning, pp.\ 6820--6829. PMLR, 2020

    2020

  60. [68]

    Asynchronous methods for deep reinforcement learning

    Mnih, V. Asynchronous methods for deep reinforcement learning. International Conference on Machine Learning, pp.\ 1928--1937, 2016

    1928

  61. [69]

    Equilibrium points in n-person games

    Nash, J. Equilibrium points in n-person games. Proceedings of the national academy of sciences, 36 0 (1): 0 48--49, 1950

    1950

  62. [70]

    Non-cooperative games

    Nash, J. Non-cooperative games. Annals of Mathematics, 54 0 (2): 0 286--295, 1951

    1951

  63. [71]

    A unified view of entropy-regularized markov decision processes

    Neu, G., Jonsson, A., and G \'o mez, V. A unified view of entropy-regularized markov decision processes. arXiv preprint arXiv:1705.07798, 2017

    2017 arXiv

  64. [72]

    Combining policy gradient and q-learning

    O'Donoghue, B., Munos, R., Kavukcuoglu, K., and Mnih, V. Combining policy gradient and q-learning. 5th International Conference on Learning Representations, 2017

    2017

  65. [73]

    Scaling mean field games by online mirror descent

    P \'e rolat, J., Perrin, S., Elie, R., Lauri \`e re, M., Piliouras, G., Geist, M., Tuyls, K., and Pietquin, O. Scaling mean field games by online mirror descent. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pp.\ 1028--1037, 2022

    2022

  66. [74]

    Fictitious play for mean field games: C ontinuous time analysis and applications

    Perrin, S., P \'e rolat, J., Lauri \`e re, M., Geist, M., Elie, R., and Pietquin, O. Fictitious play for mean field games: C ontinuous time analysis and applications. Advances in neural information processing systems, 33: 0 13199--13213, 2020

    2020

  67. [75]

    Mean field games flock! the reinforcement learning way

    Perrin, S., Lauri \`e re, M., P \'e rolat, J., Geist, M., \'E lie, R., and Pietquin, O. Mean field games flock! the reinforcement learning way. arXiv preprint arXiv:2105.07933, 2021

    2021 arXiv

  68. [76]

    Generalization in mean field games by learning master policies

    Perrin, S., Lauri \`e re, M., P \'e rolat, J., \'E lie, R., Geist, M., and Pietquin, O. Generalization in mean field games by learning master policies. Proceedings of the AAAI Conference on Artificial Intelligence, 36 0 (9): 0 9413--9421, 2022

    2022

  69. [77]

    Relative entropy policy search

    Peters, J., Mulling, K., and Altun, Y. Relative entropy policy search. Proceedings of the AAAI Conference on Artificial Intelligence, 24 0 (1): 0 1607--1612, 2010

    2010

  70. [78]

    Puterman, M. L. and Shin, M. C. Modified policy iteration algorithms for discounted markov decision problems. Management Science, 24 0 (11): 0 1127--1137, 1978

    1978

  71. [79]

    Risk-averse dynamic programming for markov decision processes

    Ruszczy \'n ski, A. Risk-averse dynamic programming for markov decision processes. Mathematical programming, 125: 0 235--261, 2010

    2010

  72. [80]

    Discrete-time average-cost mean-field games on polish spaces

    Saldi, N. Discrete-time average-cost mean-field games on polish spaces. Turkish Journal of Mathematics, 44 0 (2): 0 463--480, 2020

    2020

  73. [81]

    S., Farquhar, G., Nardelli, N., Rudner, T

    Samvelyan, M., Rashid, T., De Witt, C. S., Farquhar, G., Nardelli, N., Rudner, T. G., Hung, C.-M., Torr, P. H., Foerster, J., and Whiteson, S. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043, 2019

    1902 arXiv

  74. [82]

    and Geist, M

    Scherrer, B. and Geist, M. Local policy search in a convex space and conservative policy iteration as boosted policy search. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Part I...

    2014

  75. [83]

    Trust region policy optimization

    Schulman, J., Levine, S., Moritz, P., Jordan, M., and Abbeel, P. Trust region policy optimization. In Proceedings of the 32nd International Conference on International Conference on Machine Learning-Volume 37, pp.\ 1889--1897, 2015

    2015

  76. [84]

    Safe, multi-agent, reinforcement learning for autonomous driving

    Shalev-Shwartz, S., Shammah, S., and Shashua, A. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295, 2016

    2016 arXiv

  77. [85]

    Adaptive trust region policy optimization: Global convergence and faster rates for regularized mdps

    Shani, L., Efroni, Y., and Mannor, S. Adaptive trust region policy optimization: Global convergence and faster rates for regularized mdps. Proceedings of the AAAI Conference on Artificial Intelligence, 34 0 (04): 0 5668--5675, 2020

    2020

  78. [86]

    Near-optimal time and sample complexities for solving markov decision processes with a generative model

    Sidford, A., Wang, M., Wu, X., Yang, L., and Ye, Y. Near-optimal time and sample complexities for solving markov decision processes with a generative model. Advances in Neural Information Processing Systems, 31, 2018

    2018

  79. [87]

    and Barto, A

    Sutton, R. and Barto, A. G. Reinforcement learning: An introduction. SIAM Rev, 6 0 (2): 0 423, 2018 a

    2018

  80. [88]

    Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018 b

    2018

  81. [89]

    Algorithms for reinforcement learning

    Szepesv \'a ri, C. Algorithms for reinforcement learning. Springer nature, 2022

    2022

  82. [90]

    and Zhou, X

    Tangpi, L. and Zhou, X. Optimal investment in a large population of competitive and heterogeneous agents. Finance and Stochastics, 28 0 (2): 0 497--551, 2024

    2024

  83. [91]

    Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations

    Weinan, E., Han, J., and Jentzen, A. Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations. Communications in Mathematics and Statistics, 4 0 (5): 0 349--380, 2017

    2017

  84. [92]

    Wiering, M. A. et al. Multi-agent reinforcement learning for traffic light control. In Machine Learning: Proceedings of the Seventeenth International Conference (ICML'2000), pp.\ 1151--1158, 2000

    2000

  85. [93]

    Williams, R. J. and Peng, J. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3 0 (3): 0 241--268, 1991

    1991

  86. [94]

    Policy mirror ascent for efficient and independent learning in mean field games

    Yardim, B., Cayci, S., Geist, M., and He, N. Policy mirror ascent for efficient and independent learning in mean field games. In International Conference on Machine Learning, pp.\ 39722--39754. PMLR, 2023

    2023

  87. [95]

    Zaman, M. A. U., Koppel, A., Bhatt, S., and Basar, T. Oracle-free reinforcement learning in mean-field games along a single sample path. In International Conference on Artificial Intelligence and Statistics, pp.\ 10178--10206. PMLR, 2023

    2023

  88. [96]

    Multi-agent reinforcement learning: A selective overview of theories and algorithms

    Zhang, K., Yang, Z., and Ba s ar, T. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, pp.\ 321--384, 2021

    2021

  89. [97]

    Ziebart, B. D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010

    2010

  90. [98]

    D., Bagnell, J

    Ziebart, B. D., Bagnell, J. A., and Dey, A. K. Modeling interaction via the principle of maximum causal entropy. International Conference on Machine Learning (ICML), pp.\ 1255–--1262, 2010

    2010

  91. [99]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.