Robust Q-learning algorithm with convergence and finite-time bounds for mean-field control under Wasserstein uncertainty in common noise.
Learning in mean field games: A survey
9 Pith papers cite this work, alongside 13 external citations. Polarity classification is still indexing.
9
Pith papers citing it
13
external citations · Pith
abstract
Non-cooperative and cooperative games with a very large number of players have many applications but remain generally intractable when the number of players increases. Introduced by Lasry and Lions, and Huang, Caines and Malham\'e, Mean Field Games (MFGs) rely on a mean-field approximation to allow the number of players to grow to infinity. Traditional methods for solving these games generally rely on solving partial or stochastic differential equations with a full knowledge of the model. Recently, Reinforcement Learning (RL) has appeared promising to solve complex problems at scale. The combination of RL and MFGs is promising to solve games at a very large scale both in terms of population size and environment complexity. In this survey, we review the quickly growing recent literature on RL methods to learn equilibria and social optima in MFGs. We first identify the most common settings (static, stationary, and evolutive) of MFGs. We then present a general framework for classical iterative methods (based on best-response computation or policy evaluation) to solve MFGs in an exact way. Building on these algorithms and the connection with Markov Decision Processes, we explain how RL can be used to learn MFG solutions in a model-free way. Last, we present numerical illustrations on a benchmark problem, and conclude with some perspectives.
citation-role summary
background 2
citation-polarity summary
roles
background 2polarities
background 2representative citing papers
Introduces a mean field game model for active travel-time tomography with a two-stage diffusion-plus-MFG inversion pipeline and reports stable numerical recovery.
Proves first UATs for k-times differentiable nonlinear operators and their derivatives via OL architectures uniformly on compact sets in weighted Bastiani-Sobolev spaces on general Banach spaces.
Model-free DQN learning achieves suboptimality bounds of O(1/sqrt(Ns)) + O(1/N) in Karma DPGs at equilibrium, and deep RL combined with fictitious play empirically reaches near-Stationary Nash Equilibrium from scratch.
LQ mean-field games on hybrid manifold graphs admit FBPDE Nash limits with high-probability tracking error O((log N)^{-1/2}) sparse and O(N^{-γ(p,s)}) dense.
The authors propose actor-critic q-learning algorithms for mean-field control with common noise based on martingale orthogonality conditions and relaxed controls, establish convergence of inner iterations in the linear-quadratic case, and demonstrate performance on examples.
Mean-field game equilibrium calibrated to historical de-pegs attributes most peg recovery to primary-market arbitrage and identifies a nonlinear stress threshold for slow recovery.
Neural mean-field games integrate mean-field game theory with neural SDEs to learn strategic interactions from data in a model-free way, demonstrated on games and viral dynamics.
A monograph develops the probabilistic and control-theoretic framework connecting multi-agent reinforcement learning to mean field control, including analyses of Q-learning, policy gradients, and numerical methods for linear-quadratic and general models.
citing papers explorer
-
Robust $Q$-learning for mean-field control under Wasserstein uncertainty in common noise
Robust Q-learning algorithm with convergence and finite-time bounds for mean-field control under Wasserstein uncertainty in common noise.
-
Travel-time tomography from mean field game dynamics
Introduces a mean field game model for active travel-time tomography with a two-stage diffusion-plus-MFG inversion pipeline and reports stable numerical recovery.
-
Universal Approximation of Nonlinear Operators and Their Derivatives
Proves first UATs for k-times differentiable nonlinear operators and their derivatives via OL architectures uniformly on compact sets in weighted Bastiani-Sobolev spaces on general Banach spaces.
-
Towards Model-Free Learning in Dynamic Population Games: An Application to Karma Economies
Model-free DQN learning achieves suboptimality bounds of O(1/sqrt(Ns)) + O(1/N) in Karma DPGs at equilibrium, and deep RL combined with fictitious play empirically reaches near-Stationary Nash Equilibrium from scratch.
-
Linear-Quadratic Mean Field Games with Hybrid Local-Global Interactions on Manifolds
LQ mean-field games on hybrid manifold graphs admit FBPDE Nash limits with high-probability tracking error O((log N)^{-1/2}) sparse and O(N^{-γ(p,s)}) dense.
-
Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms
The authors propose actor-critic q-learning algorithms for mean-field control with common noise based on martingale orthogonality conditions and relaxed controls, establish convergence of inner iterations in the linear-quadratic case, and demonstrate performance on examples.
-
Who Restores the Peg? A Mean-Field Game Approach to Model Stablecoin Market Dynamics
Mean-field game equilibrium calibrated to historical de-pegs attributes most peg recovery to primary-market arbitrage and identifies a nonlinear stress threshold for slow recovery.
-
Neural Mean-Field Games: Extending Mean-Field Game Theory with Neural Stochastic Differential Equations
Neural mean-field games integrate mean-field game theory with neural SDEs to learn strategic interactions from data in a model-free way, demonstrated on games and viral dynamics.
-
Mean Field Reinforcement Learning
A monograph develops the probabilistic and control-theoretic framework connecting multi-agent reinforcement learning to mean field control, including analyses of Q-learning, policy gradients, and numerical methods for linear-quadratic and general models.