Pith. sign in

REVIEW 4 major objections 5 minor 40 references

Population-aware Online Mirror Descent for Mean-Field Games with Common Noise by Deep Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A deep RL algorithm learns one policy that reaches Nash equilibria for any initial crowd and common noise.

desk verdict A useful empirical DRL algorithm for master policies in MFGs with common noise, but the theoretical justification is thinner than advertised because the implemented update deviates from the proven one. read the letter →

arxiv 2509.03030 v1 pith:D4D3JGL5 submitted 2025-09-03 cs.LG cs.MAcs.ROcs.SYeess.SY

Zida Wu , Mathieu Lauriere , Matthieu Geist , Olivier Pietquin , Ankur Mehta This is my paper · ORCID
classification cs.LGcs.MAcs.ROcs.SYeess.SY
keywords mean-fieldgamescommonnoisemasterpolicyonlinemirrordescentdeepreinforcementlearningNashequilibriumexploitabilityMunchausen
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a single deep reinforcement learning agent can compute population-dependent Nash equilibria in finite-horizon mean-field games even when the initial population distribution is unknown and the whole population is hit by common noise. The proposed algorithm, Master OMD, replaces fictitious play's averaging of past policies with an online mirror descent update realized through a Munchausen regularizer, so no historical policy or distribution sampling is needed. On seven benchmark tasks it reports lower exploitability than deep fictitious play baselines, and a master policy trained on several initial distributions generalizes to unseen ones. If right, the paper supplies a practical route to master policies in settings where prior methods require costly best-response computation or a known initial condition.

What carries the argument

Theorem 1 is the load-bearing identity: for $\tau>0$, the softmax of the cumulative Q-sum equals $\mathrm{softmax}((Q^k + \tau \ln \pi^{k-1})/\tau)$, so the Munchausen-regularized current Q-value stands in for the sum of historical Q-functions. It carries the argument by letting a single deep Q-network, trained with the target in Eq. (4), implement online mirror descent implicitly, without storing or averaging past neural networks.

What would settle it

In a small tabular mean-field game where all exact Q-functions are computable, run the Algorithm 1 update with target (4) alongside the true OMD policy $\mathrm{softmax}(\tfrac{1}{\tau}\sum_{i=0}^k Q^i)$ and measure the KL divergence between the two policies per iteration; if the divergence remains bounded away from zero, the modified target no longer implements online mirror descent and the claimed convergence guarantee is not in force.

Watch

Extended reading notes

Core claim

The central claim is that Master OMD learns a master policy: at each stage of the game, the policy maps the representative agent's state, the current population distribution, and, when present, common noise to action probabilities, and this single policy is an equilibrium policy for every initial distribution in its training set and beyond. The key step is a population-dependent analogue of Theorem 1: the softmax of the cumulative historical Q-functions, which defines classic online mirror descent, equals the softmax of the current Q-function plus $\tau$ times the log of the previous policy. That equality justifies training one deep Q-network with a Munchausen-style target, so the network implicitly sums past Q-values without storing them. The paper extends the update to common noise by feeding the noise history into the network, and reports that this architecture matches the model-based equilibrium in the beach-bar and linear-quadratic examples.

Load-bearing premise

The load-bearing premise is that changing the target policy in Eq. (4) from the previous-iteration policy $\pi^{k-1}$, for which Theorem 1 is proved, to the current-iteration target policy $\pi^k_{\theta'}$ preserves the implicit summation of historical Q-values; no proof covers the modified update.

Editorial extensions

If this is right

  • Master OMD should produce near-zero exploitability on held-out initial distributions without retraining, because the learned policy is a function of the current mean-field state rather than of one fixed initial condition.
  • Unlike fictitious play, whose uniform averaging gives past iterations diminishing weight, M-OMD maintains a constant effective update rate, so convergence should not slow as the iteration count grows.
  • Memory cost per iteration stays bounded by one current network plus a replay buffer, whereas fictitious play's need to evaluate averaged historical policies grows linearly in the number of iterations.
  • With common-noise history as part of the network input, the same algorithm extends to aggregate shocks; the experiments indicate that population trajectories track the noise rather than expending effort to counteract it.
  • Because time is encoded as part of the input, the method applies to finite-horizon, non-stationary policies without requiring a separate network per timestep.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The stability-motivated change from $\pi^{k-1}$ to the current-iteration target $\pi^k_{\theta'}$ in Eq. (4) is not covered by Theorem 1; whether the implicit-summation property survives this change is a testable question rather than an established fact.
  • If the cumulative-Q interpretation does survive the target change, OMD-style updates could replace fictitious play in other master-equation settings, since they avoid both best-response optimization and decaying averaging over past iterations.
  • The ad-hoc teaming experiments suggest that generalization to mid-game population shocks may require injecting such shocks during training; one natural extension is to sample random population-join events into the replay buffer.
  • The linear-quadratic case, where M-OMD converges more slowly than fictitious play, hints that the regularizer strength and network capacity control whether history-free OMD beats averaging, so sweeping $\tau$ per environment may restore the advantage.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes Master OMD (M-OMD), a deep reinforcement learning algorithm for finite-horizon mean-field games with population-dependent (master) policies, initially unknown initial distributions, and common noise. The algorithm uses a Munchausen-style regularized Q-function to mimic the cumulative Q-summation of classical online mirror descent, combined with a replay buffer that is reset at each outer iteration and a mean-field input to the Q-network. The authors state and prove a theorem (Theorem 1) relating the softmax of the regularized Q-function to the softmax of cumulative historical Q-functions, extend the architecture to incorporate common noise, and report experiments on seven benchmarks within three environments (exploration, beach bar, and linear-quadratic), comparing M-OMD against vanilla and master fictitious play and against OMD baselines. The central advertised findings are faster convergence and lower exploitability, particularly when the policy must handle multiple initial distributions.

Significance. If the algorithm indeed learns master policies with the reported speed, it would provide a practical DRL method for a setting that existing OMD-based mean-field-game learners do not cover: finite-horizon problems with population-dependent policies, variable initial distributions, and common noise. The empirical study is reasonably broad, with several baselines, multiple seeds, a buffer-size sweep, and an additional 30-distribution experiment in Appendix K where exploitability is computed by dynamic programming rather than by a learned best response. The paper also gives credit to prior work and clearly identifies the modifications it makes relative to Munchausen RL and fictitious play. However, the theoretical anchor of the paper is a standard identity whose proof in Appendix B contains algebraic slips, and the implemented update in Eq. (4) replaces the policy under which the next-state expectation is taken without a proof that the cumulative-Q interpretation survives. The contribution is therefore currently an empirically demonstrated algorithm with an unverified theoretical justification.

major comments (4)
  1. [Section III(b), Eq. (4); Appendix A, Eq. (6)] The target in Eq. (4) uses π^k_{θ'} inside the next-state expectation, and the same substitution appears in Eq. (6) of Appendix A. Theorem 1 is stated and proved only for the Bellman evaluation of π^{k-1}; for the stated identity to hold, the next-state expectation must be taken under π^{k-1}. With π^k_{θ'} inside the Bellman operator, the Q-network is trained toward a self-consistent equation involving the policy being learned, and no theorem shows that its solution equals softmax(1/τ Σ_i Q^i) or satisfies any OMD-type recursion. Since Section III(b) explicitly says this is a stability-motivated modification, the abstract's claim that the algorithm reaches equilibria "without relying on averaging or historical sampling" is not established. Please either prove that the modified target preserves the cumulative-Q property, or state precisely the weaker guarantee that is actually proved.
  2. [Appendix B, Eqs. (14)-(15)] The proof of Theorem 1 contains algebraic errors. In Eq. (14), π^{k-1} is expanded as e^{1/τ(Q^{k-1}+λ1)-1}, which is not the definition of π^{k-1}: the softmax contains the whole sum Σ_{i<k} Q^i and a log-partition normalization factor. The repeated "-1" terms then produce an e^{-(k+1)} factor, and Eq. (15) treats C1·C2(λ) as a constant even though C2 depends on λ and must be chosen so that Σ_a π(a)=1. As written, the derivation does not correctly establish the softmax normalization and therefore does not prove Theorem 1. Since Theorem 1 is the sole theoretical support for the implicit averaging mechanism, this proof must be corrected.
  3. [Section V-A and Appendix K] The exploitability values in Figures 1-6 and Table I appear to be computed with a DQN-based best response; Appendix K states that dynamic programming is used to compute the "true" exploitability only in the 30-distribution experiments. If the main comparisons use an approximate best response, the reported gap between M-OMD and M-FP may not reflect the true exploitability defined in Eq. (2). Please report, for at least the main benchmarks, the DP-based exploitability, or otherwise justify that the DQN-based metric is an accurate proxy.
  4. [Section IV and Section VII] The paragraph in Section IV that begins "Following [24], the proof of convergence relies on constructing a similarity function" appears to sketch a convergence argument, but no theorem or proof is given for the common-noise extension. Section VII defers convergence proofs to future work. Please clarify whether any convergence guarantee is being claimed and, if so, provide the precise statement and proof; if not, state explicitly that the convergence results are empirical only.
minor comments (5)
  1. [Section II(a)] The phrase "on aE" should be "on E", and the notation µ^{µ0,π} in Eq. (1) is confusing and should be cleaned up.
  2. [Section V-D and Appendix D] For the state space X={−L,...,L}, the number of states is 2L+1, not 2L−1 as written in the LQ model descriptions.
  3. [Algorithm 1] The pseudocode has malformed loops: "for distribution µk in (µk,µ0)µ0∈D do" should be something like "for each µ0 in D do", and the line "Update mean-field sequence µk with πk−1 sampled by agents N" is unclear and should be rewritten.
  4. [Table I] Table I does not report standard deviations or multiple-seed information; without these, the reader cannot judge whether the differences, especially in the LQ row, are statistically meaningful.
  5. [Section V-B] In the discussion of Figure 2, the text refers to "F-FP" whereas the baseline is elsewhere called V-FP; please correct the acronym.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Theorem 1 is proven from softmax/KL identities and exploitability is measured against independent best responses; the target-policy mismatch is a proof gap, not a circular reduction.

full rationale

The derivation chain is self-contained at the point where circularity could occur. Theorem 1 defines tilde Q^k = Q^k + tau ln pi^{k-1} and proves pi^k = softmax((1/tau) tilde Q^k) through a Lagrangian optimization and an algebraic identity in Appendix B, so the 'implicit summation of historical Q-values' is re-derived rather than imported from prior work. The experimental claim is also externally grounded: exploitability is computed against a separately solved dynamic-programming best response (Appendix K) using testing distributions, so the reported convergence is not fitted to the target quantity. The main caveat is a proof gap, not circularity: Eq. (4) and Appendix Eq. (6) replace the next-state expectation under pi^{k-1} with the current target policy pi^k_{theta'}, whereas Theorem 1 establishes the cumulative-Q identity only when the expectation is under pi^{k-1}. Section III(b) explicitly acknowledges this stability-motivated modification, but no theorem covers it; the implemented update may therefore not implement the proved OMD recursion. This is missing support for the theoretical claim, not a reduction of the claim to its own inputs by construction. Self-citations to [21], [24], [25], and [28] are present but not load-bearing because the key equivalence is proved locally and the baselines are evaluated independently. No circular step is exhibited.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim depends mainly on hyperparameter choices, monotonicity assumptions, and function-approximation generalization, not on new physical or conceptual entities.

free parameters (5)
  • Munchausen temperature τ = 50 for M-OMD and V-OMD2, 5.0 for V-OMD1 in Exploration tasks (Table II)
    Controls the KL regularization and softmax temperature; swept in Appx. I and chosen for best exploitability.
  • Entropy coefficient α = 1.0 (Table II)
    Taken from [21]'s Munchausen DQN; affects the strength of the Munchausen log-policy term.
  • Clip threshold = 10^-6
    Used in the Munchausen log term to avoid singularities; an implementation-specific hand choice.
  • Training initial distribution set size = 5 distributions, with 30 in Appx. K
    The master policy is trained only on these µ0; performance on other µ0 depends on generalization.
  • Number of agents for mean-field simulation = 500
    Empirical approximation of the infinite population mean field; chosen by hand.
assumptions (3)
  • domain assumption Lasry-Lions monotonicity condition ensures the similarity function decreases under OMD updates in the common-noise setting
    Invoked in Section IV to argue convergence, but no theorem is proven for the deep RL algorithm.
  • domain assumption DQN with function approximation converges to the tabular fixed point in this non-stationary MFG training setting
    The algorithm inherits DQN-style target network and replay buffer; standard DQN convergence is not established for repeated mean-field updates.
  • ad hoc to paper The finite training set of initial distributions is representative enough to yield a master policy on all ∆X
    Section V and Appx. G train on 5 or 30 distributions; the master-policy property over any µ0 is assumed to hold approximately via generalization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Population-aware Online Mirror Descent for Mean-Field Games with Common Noise by Deep Reinforcement Learning." pith.science (2026). https://pith.science/paper/D4D3JGL5

@misc{pith2026250903030,
  author       = {Pith},
  title        = {Pith review of: Population-aware Online Mirror Descent for Mean-Field Games with Common Noise by Deep Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D4D3JGL5}},
  note         = {Machine review of arXiv:2509.03030}
}
read the original abstract

Mean Field Games (MFGs) offer a powerful framework for studying large-scale multi-agent systems. Yet, learning Nash equilibria in MFGs remains a challenging problem, particularly when the initial distribution is unknown or when the population is subject to common noise. In this paper, we introduce an efficient deep reinforcement learning (DRL) algorithm designed to achieve population-dependent Nash equilibria without relying on averaging or historical sampling, inspired by Munchausen RL and Online Mirror Descent. The resulting policy is adaptable to various initial distributions and sources of common noise. Through numerical experiments on seven canonical examples, we demonstrate that our algorithm exhibits superior convergence properties compared to state-of-the-art algorithms, particularly a DRL version of Fictitious Play for population-dependent policies. The performance in the presence of common noise underscores the robustness and adaptability of our approach.

Figures

Figures reproduced from arXiv: 2509.03030 by the authors.

Figure 1
Figure 1. Example 2 in Env1: Exploration in one room. (a): [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Example 1 in Env1: Exploration in four connected [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. Example 1 in Env3: Linear Quadratic with the [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (11 more)
Figure 5
Figure 5. Figure 5: Example 2 in Env2: Beach bar problem. (a) and (b) [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Example 2 in Env3: Linear quadratic model. (a) shows [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 10
Figure 10. Figure 10: Model size comparison for M-OMD (ours) and M [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 8
Figure 8. Figure 8: Ad-hoc teaming test simulates new team joining into [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Exploitability versus map size. We compared two [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 12
Figure 12. Figure 12: Training and testing sets for Exploration in Four [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: Hyperparameter sweeping (a) Total time per iteration (b) Learn policy (c) Update distribution (d) Compute exploitability [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]
Figure 15
Figure 15. Figure 15: Training on two exploration tasks with 30 distribu [PITH_FULL_IMAGE:figures/full_fig_p014_15.png]
Figure 17
Figure 17. Figure 17: Exploration in Four Rooms task: Exploitability for 30 [PITH_FULL_IMAGE:figures/full_fig_p014_17.png]
Figure 18
Figure 18. Figure 18: Training sets with 30 distributions for two exploration [PITH_FULL_IMAGE:figures/full_fig_p014_18.png]
Figure 19
Figure 19. Figure 19: Testing sets with 30 distributions for two exploration [PITH_FULL_IMAGE:figures/full_fig_p014_19.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 29 canonical work pages

  1. [24]

    Scaling mean field games by online mirror descent,

    J. Perolat, S. Perrin, R. Elie, M. Lauri `ere, G. Piliouras, M. Geist, K. Tuyls, and O. Pietquin, “Scaling mean field games by online mirror descent,” Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems , pp. 1028–1037, 2022

    work page 2022

  2. [1]

    Multi-agent systems: A survey,

    A. Dorri, S. S. Kanhere, and R. Jurdak, “Multi-agent systems: A survey,” Ieee Access, vol. 6, pp. 28573–28593, 2018

    2018

  3. [2]

    Flocking for multi-agent dynamic systems: Algorithms and theory,

    R. Olfati-Saber, “Flocking for multi-agent dynamic systems: Algorithms and theory,” IEEE Transactions on automatic control , vol. 51, no. 3, pp. 401–420, 2006

    2006

  4. [3]

    Emergent behavior in flocks,

    F. Cucker and S. Smale, “Emergent behavior in flocks,” IEEE Transactions on automatic control , vol. 52, no. 5, pp. 852–862, 2007

    work page 2007

  5. [4]

    Application of multi- agent systems in traffic and transportation,

    B. Burmeister, A. Haddadi, and G. Matylis, “Application of multi- agent systems in traffic and transportation,” IEE Proceedings-Software, vol. 144, no. 1, pp. 51–60, 1997

    work page 1997

  6. [5]

    A survey on aerial swarm robotics,

    S.-J. Chung, A. A. Paranjape, P. Dames, S. Shen, and V . Kumar, “A survey on aerial swarm robotics,” IEEE Transactions on Robotics , vol. 34, no. 4, pp. 837–855, 2018

    2018

  7. [6]

    Multi-agent actor-critic for mixed cooperative-competitive environments,

    R. Lowe, Y . I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” Advances in neural information processing systems , vol. 30, 2017

    2017

  8. [7]

    Monotonic value function factorisation for deep multi-agent reinforcement learning,

    T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson, “Monotonic value function factorisation for deep multi-agent reinforcement learning,” The Journal of Machine Learning Research, vol. 21, no. 1, pp. 7234–7284, 2020

    work page 2020

Show all 40 references
  1. [8]

    Mean field games,

    J.-M. Lasry and P.-L. Lions, “Mean field games,” Japanese journal of mathematics, vol. 2, no. 1, pp. 229–260, 2007

    2007

  2. [9]

    Large-population cost- coupled lqg problems with nonuniform agents: individual-mass behavior and decentralized ε-nash equilibria,

    M. Huang, P. E. Caines, and R. P. Malham ´e, “Large-population cost- coupled lqg problems with nonuniform agents: individual-mass behavior and decentralized ε-nash equilibria,” IEEE transactions on automatic control, vol. 52, no. 9, pp. 1560–1571, 2007

    2007

  3. [10]

    Carmona and F

    R. Carmona and F. Delarue, Probabilistic theory of mean field games with applications I-II . Springer, 2018

    2018

  4. [11]

    Bensoussan, J

    A. Bensoussan, J. Frehse, and P. Yam, Mean field games and mean field type control theory , vol. 101. Springer, 2013

    2013

  5. [12]

    Efficient ridesharing order dispatching with mean field multi-agent reinforcement learning,

    M. Li, Z. Qin, Y . Jiao, Y . Yang, J. Wang, C. Wang, G. Wu, and J. Ye, “Efficient ridesharing order dispatching with mean field multi-agent reinforcement learning,” in The world wide web conference , pp. 983– 994, 2019

    2019

  6. [13]

    Learning mean-field games,

    X. Guo, A. Hu, R. Xu, and J. Zhang, “Learning mean-field games,” Advances in Neural Information Processing Systems , vol. 32, 2019

    2019

  7. [14]

    Approximately solving mean field games via entropy-regularized deep reinforcement learning,

    K. Cui and H. Koeppl, “Approximately solving mean field games via entropy-regularized deep reinforcement learning,” in International Conference on Artificial Intelligence and Statistics , pp. 1909–1917, PMLR, 2021

    1909

  8. [15]

    Q-learning in regularized mean-field games,

    B. Anahtarci, C. D. Kariksiz, and N. Saldi, “Q-learning in regularized mean-field games,” Dynamic Games and Applications , vol. 13, no. 1, pp. 89–117, 2023

    2023

  9. [16]

    Iterative solution of games by fictitious play,

    G. W. Brown, “Iterative solution of games by fictitious play,” Act. Anal. Prod Allocation, vol. 13, no. 1, p. 374, 1951

    1951

  10. [17]

    Brown’s original fictitious play,

    U. Berger, “Brown’s original fictitious play,” Journal of Economic Theory, vol. 135, no. 1, pp. 572–578, 2007

    2007

  11. [18]

    Learning in mean field games: the fictitious play,

    P. Cardaliaguet and S. Hadikhanloo, “Learning in mean field games: the fictitious play,” ESAIM: Control, Optimisation and Calculus of Variations, vol. 23, no. 2, pp. 569–591, 2017

    2017

  12. [19]

    Finite mean field games: fictitious play and convergence to a first order continuous mean field game,

    S. Hadikhanloo and F. J. Silva, “Finite mean field games: fictitious play and convergence to a first order continuous mean field game,” Journal de Math ´ematiques Pures et Appliqu ´ees, vol. 132, pp. 369–397, 2019

    2019

  13. [20]

    Fictitious play for mean field games: Continuous time analysis and applications,

    S. Perrin, J. P ´erolat, M. Lauri `ere, M. Geist, R. Elie, and O. Pietquin, “Fictitious play for mean field games: Continuous time analysis and applications,” Advances in Neural Information Processing Systems , vol. 33, pp. 13199–13213, 2020

    2020

  14. [21]

    Scalable deep rein- forcement learning algorithms for mean field games,

    M. Lauri `ere, S. Perrin, S. Girgin, P. Muller, A. Jain, T. Cabannes, G. Piliouras, J. P ´erolat, R. ´Elie, and O. Pietquin, “Scalable deep rein- forcement learning algorithms for mean field games,” in International Conference on Machine Learning , pp. 12078–12095, PMLR, 2022

    2022

  15. [22]

    Learning in anonymous nonatomic games with applications to first-order mean field games,

    S. Hadikhanloo, “Learning in anonymous nonatomic games with applications to first-order mean field games,” arXiv preprint arXiv:1704.00378, 2017

    2017 arXiv

  16. [23]

    Hadikhanloo, Learning in mean field games

    S. Hadikhanloo, Learning in mean field games . PhD thesis, Universit ´e Paris sciences et lettres, 2018

    2018

  17. [25]

    Munchausen reinforcement learning,

    N. Vieillard, O. Pietquin, and M. Geist, “Munchausen reinforcement learning,” Advances in Neural Information Processing Systems , vol. 33, pp. 4235–4246, 2020

    2020

  18. [26]

    Learning in mean field games: A survey,

    M. Lauri`ere, S. Perrin, J. P ´erolat, S. Girgin, P. Muller, R. ´Elie, M. Geist, and O. Pietquin, “Learning in mean field games: A survey,” arXiv preprint arXiv:2205.12944, 2022

    2022 arXiv

  19. [27]

    Cardaliaguet, F

    P. Cardaliaguet, F. Delarue, J.-M. Lasry, and P.-L. Lions, The master equation and the convergence problem in mean field games:(ams-201) . Princeton University Press, 2019

    2019

  20. [28]

    Generalization in mean field games by learning master policies,

    S. Perrin, M. Lauri `ere, J. P ´erolat, R. ´Elie, M. Geist, and O. Pietquin, “Generalization in mean field games by learning master policies,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, pp. 9413–9421, 2022

    2022

  21. [29]

    Population-aware online mirror descent for mean-field games by deep reinforcement learning,

    Z. Wu, M. Lauri`ere, S. J. C. Chua, M. Geist, O. Pietquin, and A. Mehta, “Population-aware online mirror descent for mean-field games by deep reinforcement learning,” arXiv preprint arXiv:2403.03552 , 2024

    2024 arXiv

  22. [30]

    Computing approximate equilibria in sequential adversarial games by exploitability descent,

    E. Lockhart, M. Lanctot, J. P ´erolat, J.-B. Lespiau, D. Morrill, F. Tim- bers, and K. Tuyls, “Computing approximate equilibria in sequential adversarial games by exploitability descent,” in Proceedings of the 28th International Joint Conference on Artificial Intelligence , pp...

    2019

  23. [31]

    Leverage the average: an analysis of kl regularization in reinforcement learning,

    N. Vieillard, T. Kozuno, B. Scherrer, O. Pietquin, R. Munos, and M. Geist, “Leverage the average: an analysis of kl regularization in reinforcement learning,” Advances in Neural Information Processing Systems, vol. 33, pp. 12163–12174, 2020

    2020

  24. [32]

    Human-level control through deep reinforcement learning,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, and G. Ostrovski, “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015

    2015

  25. [33]

    Prioritized experience replay,

    T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay,” arXiv preprint arXiv:1511.05952 , 2015

    2015 arXiv

  26. [34]

    Overcoming catastrophic forgetting in neural networks,

    J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, and A. Grabska-Barwinska, “Overcoming catastrophic forgetting in neural networks,” Proceedings of the national academy of sciences , vol. 114, no. 13, pp. 3521–3526, 2017

    2017

  27. [35]

    Mean field games with common noise,

    R. Carmona, F. Delarue, and D. Lacker, “Mean field games with common noise,” Annals of Probability , vol. 44, pp. 3740–3803, 2016

    2016

  28. [36]

    Training larger networks for deep reinforcement learning,

    K. Ota, D. K. Jha, and A. Kanezaki, “Training larger networks for deep reinforcement learning,” arXiv preprint arXiv:2102.07920 , 2021

    2021 arXiv

  29. [37]

    Concave utility reinforcement learning: the mean-field game viewpoint,

    M. Geist, J. P ´erolat, M. Lauri `ere, R. Elie, S. Perrin, O. Bachem, R. Munos, and O. Pietquin, “Concave utility reinforcement learning: the mean-field game viewpoint,” arXiv preprint arXiv:2106.03787 , 2021

    2021 arXiv

  30. [38]

    Control of McKean– Vlasov dynamics versus mean field games,

    R. Carmona, F. Delarue, and A. Lachapelle, “Control of McKean– Vlasov dynamics versus mean field games,” Mathematics and Financial Economics, vol. 7, pp. 131–166, 2013

    2013

  31. [39]

    Linear- quadratic mean field games,

    A. Bensoussan, K. Sung, S. C. P. Yam, and S.-P. Yung, “Linear- quadratic mean field games,” Journal of Optimization Theory and Applications, vol. 169, pp. 496–529, 2016. APPENDIX https://drive.google.com/file/d/ 1nXKRwdVhSw-HogyzcnoGbz_9Pxw6wx3b/view? usp=sharing See Algo. 2 f...

    2016

  32. [40]

    This set comprises 10 distributions originating from fixed points, 10 following Gaussian distributions, and 10 distributed across random points

    To extend our exploration of its adaptability across a broader spectrum of distributions, we examine an ensemble of 30 distributions depicted in Fig 18. This set comprises 10 distributions originating from fixed points, 10 following Gaussian distributions, and 10 distributed a...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.