Recurrent Structural Policy Gradient for Partially Observable Mean Field Games

Andreas Schaab; Benjamin Moll; Chiyuan Wang; Clarisse Wibault; George Whittle; Jakob Foerster; Johannes Forkel; Juan Duque; Maike Osborne; Sebastian Towers

REVIEW 3 major objections 4 minor 1 cited by

RSPG makes history-dependent equilibrium policies computationally tractable in partially observable mean-field games by restricting memory to shared aggregate observations.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 21:23 UTC pith:JL7VAAAG

load-bearing objection A solid conceptual extension of SPG with a tractable shared-observation special case, but the exploitability metric cannot support the 'state-of-the-art / solves' claims; send to peer review with requests for restricted-class or absolute exploitability. the 3 major comments →

arxiv 2602.20141 v2 pith:JL7VAAAG submitted 2026-02-23 cs.AI

Recurrent Structural Policy Gradient for Partially Observable Mean Field Games

Clarisse Wibault , Johannes Forkel , Sebastian Towers , Tiphaine Wibault , Juan Duque , George Whittle , Andreas Schaab , Yucheng Yang

show 4 more authors

Chiyuan Wang Maike Osborne Benjamin Moll Jakob Foerster

This is my paper

classification cs.AI

keywords mean-field gamespartial observabilitycommon noiserecurrent policy gradienthybrid structural methodsshared aggregate observationsmacroeconomicsexploitability

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to show that partially observable mean-field games with common noise—where agents see only public information about the aggregate population—can be solved with a hybrid structural method that keeps the low-variance exact expectations of fully observable settings. It identifies a special case: when every agent receives the same deterministic observation of the aggregate state and conditions only on that shared history plus its own individual state, the mean-field update stays tractable. On this basis it builds RSPG, a recurrent policy-gradient algorithm, and reports that it reaches low exploitability and converges roughly an order of magnitude faster in wall-clock time than model-free reinforcement learning, including on a macroeconomics model with common noise and history-dependent policies. A sympathetic reader would care because this is a route to computing history-dependent equilibrium policies without sampling individual trajectories or enumerating all possible histories.

Core claim

The central claim is that history-dependence does not force a return to sample-based reinforcement learning in partially observable mean-field games. If the observation is a shared deterministic function of the aggregate state, o_t = U(µ_t, z_t), and the policy conditions only on the current individual state and the history of those shared observations, then the population distribution can still be updated exactly using the analytic mean-field update; the shared history is a single trajectory rather than an exponentially large distribution over individual histories. RSPG uses a recurrent policy whose hidden state is independent of the individual state, so the exact expected return can be dif

What carries the argument

The load-bearing object is the reduced policy together with the shared-observation condition. With o_t = U(µ_t, z_t), a policy at ∼ π(· | s_t, o_{0:t}) makes the mean-field update µ_{t+1} = Φ^π(µ_t, z_t, o_{0:t}) identical to a fully observable update, so exact expectations over individual transitions can be computed and only the common noise needs to be sampled. In RSPG the recurrent hidden state processes only the shared observation history, keeping its size independent of the individual state; this preserves the functional, non-materialised form of the update and keeps the per-step cost linear in the number of states rather than quadratic.

Load-bearing premise

The method stays tractable only when the public observation is a deterministic shared function of the aggregate state and the policy restricts memory to the history of those shared observations; if observations carry individual-specific information, the distribution over histories grows exponentially and the analytic mean-field update becomes intractable.

What would settle it

Construct a mean-field game where each agent observes the aggregate price plus an individual-specific signal, so o_t depends on s_t, and run RSPG. If per-iteration memory or computation grows exponentially with the horizon, or if exploitability stops decreasing while a recurrent model-free method succeeds, the special-case restriction is load-bearing and the tractability claim applies only to shared aggregate observations.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

If RSPG's claims hold, history-dependent Nash equilibria in partially observable mean-field games become computable at essentially the same cost as memoryless equilibria whenever observations are shared.
The method promises an order-of-magnitude reduction in wall-clock training time versus model-free reinforcement learning in settings where individual transition dynamics are known, making equilibrium search feasible on larger population models.
In macroeconomic models the learned policies exhibit anticipatory behaviour—agents spend wealth before the end of the episode and move away from a bar before it may close—behaviour that memoryless policies cannot represent.
The paper's unified treatment of dynamic programming, reinforcement learning, and hybrid structural methods clarifies when each class of algorithm is applicable to mean-field games with common noise.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the shared-observation restriction fails—for instance, if agents receive private signals correlated with their individual state—RSPG's exact update becomes intractable; a natural extension would be to combine the recurrent policy with a learned belief state or generative model of the history distribution.
The same design pattern, shared source of randomness plus exact individual expectation, could extend to games with major players or multiple mean-fields by treating the major player's action as part of the common observation.
A testable prediction follows from the architecture: in any environment where the observation already reveals the full aggregate state and the timestep, RSPG should perform similarly to its memoryless counterpart; if it does not, the recurrent memory is encoding spurious information.
The reported speedup is measured in wall-clock GPU time; whether it transfers to settings with very large discrete state spaces depends on the functional update avoiding explicit construction of the transition matrix.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A solid conceptual extension of SPG with a tractable shared-observation special case, but the exploitability metric cannot support the 'state-of-the-art / solves' claims; send to peer review with requests for restricted-class or absolute exploitability.

read the letter

RSPG is a clean, well-motivated extension of SPG to shared-observation partial information. The special case in Section 5.1 — a recurrent policy whose hidden state is independent of the individual state — is the real contribution: it keeps the analytic mean-field update exact and tractable while allowing history-dependent behaviour. The formal parts look correct, and the wall-clock speedups over model-free RL are plausible given the structural advantage.

I largely agree with the stress-test on the empirical claims. The exploitability metric is an upper bound over a more expressive perfect-information policy class, not the exploitability over the policy's own information class. The paper explicitly says this, but then uses the resulting numbers to claim 'state-of-the-art' and 'solves' in the abstract and conclusion. An upper bound cannot rank two policies by true equilibrium distance unless it is equally tight for both, and it will systematically penalize memoryless policies that cannot see history. RSPG's lower perfect-information exploitability may just mean its RNN partially recovers the richer information set, not that it is closer to equilibrium. This is a serious problem for the paper's main empirical conclusions.

Other soft spots are minor by comparison. No absolute exploitability values are reported in the text, only confidence-band plots, which makes the relative claims hard to verify. The hyperparameters were swept and best configurations reported, which is common but should be flagged as selection. And the code URL is given as clarisse-wibault.github.io/rspg/ in the abstract but github.com/CWibault/mfax in the body — a small reproducibility issue.

What the paper does well: the POMFG-CN formalization is a genuine new definition, the special case is clean, and MFAX looks like a useful framework. The empirical section is honest about the upper-bound nature of the metric, but the framing needs to be brought in line with the evidence. This is exactly the kind of paper a serious referee should see: the algorithmic idea is sound and novel within the MFG/heterogeneous-agent macro community, and the evaluation can be fixed by reporting restricted-class exploitability or at least absolute numbers and toning down the claims.

Referee Report

3 major / 4 minor

Summary. The paper formalizes Partially Observable Mean Field Games with Common Noise (POMFG-CN), in which agents receive observations that may depend on the individual state and the aggregate state. It observes that the general IAOH-based mean-field update is intractable, then identifies a tractable special case: deterministic shared observations o_t = U(µ_t, z_t) and policies restricted to π(a|s_t, o_{0:t}). The proposed algorithm, Recurrent Structural Policy Gradient (RSPG), is a hybrid structural method that uses known individual transition dynamics to perform analytic mean-field updates and exact expected-return computations conditioned on sampled common-noise sequences, while an RNN processes only the shared observation history so that the hidden state remains independent of the individual state. The paper also introduces MFAX, a JAX-based framework supporting both analytic and sample-based mean-field updates. Experiments on partially observable Linear Quadratic, Beach Bar, and a Krusell–Smith macroeconomics model claim state-of-the-art exploitability, an order-of-magnitude wall-clock speedup over RL baselines, and the first solution of a partially observable macroeconomics MFG with history-aware policies.

Significance. If validated, the paper makes a useful algorithmic and software contribution. The key derivation — that the mean-field update remains tractable when the policy conditions on the current individual state and the history of shared aggregate observations — is correct for the stated special case, and the gradient flow is properly restricted to individual transitions rather than through the mean-field update. The design of MFAX, with explicit white-box vs. black-box transition access and functional mean-field updates, is a practical contribution that the paper backs with benchmarking code and public release. The main weaknesses are in the evaluation and in the formalization of the solution concept: the reported exploitability is an upper bound against a different, more expressive policy class, which does not support the paper's comparative claims or the 'solves' conclusion; and the restricted policy class is not tied to a formally defined restricted equilibrium. These issues are load-bearing for the headline claims but appear addressable within the manuscript's scope.

major comments (3)

[§8.2.1, §8.3, §9] The exploitability metric is computed as J*_evol(π) over 'a common, more expressive class of best-response policies with perfect information' and is explicitly called a 'conservative upper bound on the true distance to equilibrium.' An upper bound cannot rank policies by their true exploitability: a large upper bound for a memoryless policy may simply reflect that the perfect-information benchmark uses information (z_t, t, full µ_t) that the policy class is structurally unable to use, even if the policy is at a Nash equilibrium of its own restricted class. Therefore the conclusions in §8.3 that RSPG 'consistently achieves among the lowest exploitability' and the Abstract/§9 claims of 'state-of-the-art performance' and 'solves, for the first time, a macroeconomics MFG' are not supported by the evidence. Please compute exploitability with best responses restricted to the same information s
[§5.1, Definition 5.1] Definition 5.1 defines a mean-field Nash equilibrium over policies conditioning on individual-action-observation histories, while the tractable algorithm in §5.1 restricts the search to the reduced class π(a|s_t, o_{0:t}). The paper does not define a restricted equilibrium for this reduced class, nor does it prove that an equilibrium within the reduced class coincides with an equilibrium of the original POMFG-CN. Since the macroeconomics 'solves' claim refers to the original game, this is a conceptual gap. Please state the restricted solution concept explicitly and scope all equilibrium claims to that concept.
[§6.1, Algorithm 1, §B.4] Algorithm 1 passes d_t to the policy, but d_t is never defined in the paper. If d_t is the timestep, the policy is actually π(a|s_t, o_{0:t}, t), not π(a|s_t, o_{0:t}) as stated in §5.1. Moreover, in the Linear Quadratic and Beach Bar environments the observation is said not to include the timestep, and the qualitative 'history-aware anticipatory behavior' in §8.3 (Figures 3–4) may then be driven by direct time information rather than by the recurrent memory. Please define d_t, adjust the problem specification if time is known to the agent, and reinterpret the history-awareness evidence accordingly.

minor comments (4)

[§5, Eq. (11)] In Eq. (11), Φπ(˜µ_t, z_{0:∞}) appears where the first argument should presumably be z_t or the resulting µ_{t+1}; the notation is inconsistent with Eq. (10).
[Abstract / Contribution 3] The claim 'first history-aware HSM' is broader than the actual contribution, which is restricted to shared deterministic aggregate observations. Suggest wording such as 'first HSM with history-dependent policies conditioned on shared aggregate observations.'
[Figure 2] The y-axis label 'exploitability' should be 'upper bound on exploitability' or similar, since the metric is not the true exploitability of the learned policy class.
[§7.1, Table 2] The speedup claims (10x vs. OpenSpiel, 1000x vs. MFGLib) are based on a single environment and a single update; please state the benchmark conditions more carefully and consider reporting on multiple environments if this is meant as a general framework claim.

Circularity Check

0 steps flagged

No significant circularity: RSPG's derivation is self-contained; the exploitability caveat is a metric limitation, not a definitional circularity.

full rationale

The paper's central derivation is not circular. RSPG takes known individual dynamics T, the analytic mean-field update, and the reduced policy class pi(a_t | s_t, o_{0:t}) as inputs, then computes the exact discounted return with gradients flowing through individual transitions and expected rewards but not through the mean-field update (Algorithm 1; Eq. 14). The empirical claims are tested against external baselines (SPG, IPPO, RIPPO, M-OMD) and an exploitability metric computed by backwards induction over a more expressive perfect-information policy class, as stated in §8.2.1: 'we compute J*_evol(pi) over a common, more expressive class of best-response policies with perfect information. Even though this might render it impossible to reach zero exploitability, it provides a conservative upper bound on the true distance to equilibrium and enables meaningful comparison across algorithms.' This quoted passage is a self-acknowledged limitation of the metric for the 'solves / state-of-the-art' wording, but it is not a case of a fitted input being renamed as a prediction: the policy gradient objective does not reduce to the exploitability benchmark, and the best-response class is external to the learned policy class. The self-citations (Moll 2025 for the macroeconomic environment; Yang et al. 2025 for the memoryless SPG baseline) are used to supply the benchmark setting and the algorithmic predecessor, but neither is load-bearing for the new history-aware special-case tractability argument in §5.1, which is derived from the paper's own equations. No load-bearing uniqueness theorem, ansatz smuggled via self-citation, or renaming of a known result is present. The primary risk is that the reported exploitability is only an upper bound and may rank policy classes unequally, which is a correctness/interpretation concern, not circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central algorithm is an empirical method, not a derivation from first principles. The ledger makes explicit that the tractability and the speed claims rely on known dynamics, shared aggregate observations, a restricted recurrent policy class, and hand-chosen or swept training hyperparameters. No new physical or mathematical entity is introduced.

free parameters (4)

RSPG learning rate = selected from {1e-4, 1e-3, 1e-2}
Only hyperparameter swept for RSPG (Appendix C.2); final choice is the one giving low exploitability and fast convergence, so the speed claim depends on this choice.
Network hidden sizes = 64/128; GRU hidden 64
Set by hand for all algorithms in Appendix C.2; no sensitivity analysis beyond the ordinal-vs-categorical policy ablation.
Number of parallel environments = 8 (LQ, macro), 128 (Beach Bar)
Chosen per environment; directly enters the wall-clock training-time comparison.
Sample-based mean-field agent count = 10,000 agents
Used for RL baselines; chosen to be much larger than |S|, but contributes to RL wall-clock overhead compared with HSMs.

axioms (5)

domain assumption Known individual transition dynamics T and exact analytic mean-field update operator Φπ are available
RSPG is an HSM; it requires white-box access to T to compute exact expectations (Information Box 3 and Section 6.1).
domain assumption Shared observation o_t is a function only of the aggregate state (µ_t,z_t)
Introduced in Section 5.1 to keep the mean-field update over |S| states rather than over exponentially growing histories; this is the load-bearing restriction.
ad hoc to paper Policy class restricted to π(a|s_t,o_{0:t}) with hidden state independent of s_t
Architectural choice (Figure 1) with no proof that this class contains a mean-field Nash equilibrium; it makes updates tractable.
domain assumption Finite/discrete state and action spaces, with |A| much smaller than |S| for efficiency
Equations (2) and (14) are matrix-vector forms; the macro environment is discretized to 1000 states and 20 actions.
standard math Atomless agents: individual actions do not affect the mean-field
Standard MFG reduction used in J_evol (equations 3 and 12); without it the population/individual feedback loop breaks.

pith-pipeline@v1.3.0-alltime-deepseek · 20917 in / 14240 out tokens · 136904 ms · 2026-08-02T21:23:46.305623+00:00 · methodology

0 comments

read the original abstract

Mean Field Games (MFGs) provide a principled framework for modelling interactions in large population systems. However, algorithmic progress has been limited since model-free methods are high variance and exact methods scale poorly. Recent Hybrid Structural Methods (HSMs) reduce variance while maintaining tractability by leveraging low-dimensional individual state and action spaces and known transition dynamics to compute the exact expected return conditioned on Monte Carlo rollouts of common noise. However, HSMs have not been extended to partially observable settings. We propose Recurrent Structural Policy Gradient (RSPG), the first history-aware HSM for MFGs with public partial information. RSPG achieves an order-of-magnitude faster convergence than model-free RL methods while learning history-aware behaviour, unlike current HSMs. To facilitate research into MFGs, we also introduce MFAX, our JAX-based framework for MFGs that supports both analytic and sample-based mean-field updates. MFAX and usage examples can be found at https://clarisse-wibault.github.io/rspg/.

Figures

Figures reproduced from arXiv: 2602.20141 by Andreas Schaab, Benjamin Moll, Chiyuan Wang, Clarisse Wibault, George Whittle, Jakob Foerster, Johannes Forkel, Juan Duque, Maike Osborne, Sebastian Towers, Tiphaine Wibault, Yucheng Yang.

**Figure 1.** Figure 1: Top left: the analytic mean-field update computes the exact expectation over next states. Bottom left: a sample-based mean-field update re-approximates the mean-field at each step, by tracking individual agents. Right: network architecture for the reduced policy used in RSPG. The hidden state is independent of the individual state such that the analytic mean-field update has the same asymptotic computation… view at source ↗

**Figure 2.** Figure 2: Exploitability versus training wall-clock time for partially observable Linear Quadratic, Beach Bar, and Macroeconomics environments. All experiments were conducted on NVIDIA L40S GPUs (48 GB). HSMs (SPG, RSPG) are an order of magnitude faster than RL methods, with history-aware RSPG consistently achieving among the lowest exploitability. Shaded regions indicate 95%ile CI for the mean over 10 seeds. 7.2 Sp… view at source ↗

**Figure 3.** Figure 3: Heatmaps: mean-field distribution (income on y-axis and wealth on x-axis) at specific timesteps during the episode for the Macroeconomics environment (with total episode length of 128 steps). Interest rates (first column) and wages (second column) are determined by the mean-field distribution. The environment is implemented as a finite horizon: with RSPG, agents learn anticipatory behaviour, spending more … view at source ↗

**Figure 4.** Figure 4: Mean-field distribution (y-axis) versus time (x-axis) for the Beach Bar environment. Agents are rewarded for being next to the bar when it is open, and penalised for being directly next to the bar when it is closed, or just before it closes, which can occur halfway through the episode (white-line). With RSPG and RIPPO, agents learn to apprehend the time at which the bar might close, moving away from the ba… view at source ↗

**Figure 5.** Figure 5: Mean-field distribution (y-axis) versus time (x-axis) for the Beach Bar environment (top). Learned policy (middle) versus best response policy (bottom). Agents are rewarded for being next to the bar when it is open, and penalised for being directly next to the bar when it is closed, or just before it closes, which can occur halfway through the episode (white-line). Here the bar stays open, which is why age… view at source ↗

**Figure 6.** Figure 6: Mean-field distribution (y-axis) versus time (x-axis) for the Beach Bar environment (top). Learned policy (middle) versus best response policy (bottom). Agents are rewarded for being next to the bar when it is open, and penalised for being directly next to the bar when it is closed, or just before it closes, which can occur halfway through the episode (white-line). Here the bar stays closes, which is why a… view at source ↗

**Figure 7.** Figure 7: Mean-field distribution (y-axis) versus time (x-axis) for the Linear Quadratic environment (top). Learned policy (middle) versus best response policy (bottom). Agents are subject to one of two realisations of common noise, pushing the entire population downwards or upwards. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_7.png] view at source ↗

**Figure 8.** Figure 8: Heatmaps: mean-field distribution (income on y-axis and wealth on x-axis) at specific timesteps during the episode for the Macroeconomics environment (with total episode length of 128 steps). Interest rates (first column) and wages (second column) are determined by the mean-field distribution. The environment is implemented as a finite horizon: with RSPG and RIPPO, we see that agents learn anticipatory beh… view at source ↗

**Figure 9.** Figure 9: Learned actions (top) and best response actions (middle) and (difference) between the two (bottom) for the Macroeconomics environment. For M-OMD, the learned policy is much coarser, which we attribute to the fact that it is not parameterised by an underlying continuous action distribution [PITH_FULL_IMAGE:figures/full_fig_p024_9.png] view at source ↗

**Figure 10.** Figure 10: Underlying continuous actions (second row), and their associated continuous consumptions (third row), rewards (final row) and cumulative discounted rewards (top row) over the entire episode for the Macroeconomics environment. We include these plots for macroeconomics interest. M-OMD is omitted, since it does not have an underlying continuous action distribution. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_… view at source ↗

**Figure 11.** Figure 11: Exploitability versus training wall-clock time for partially observable Macroeconomics environment with ordinal versus categorical policies. Experiments were run on NVIDIA A40 GPUs. Having an underlying continuous distribution (Ordinal) consistently outperforms the underlying categorical distribution (Discrete), which does not encode information about the ordinal nature of the action space. 26 [PITH_FULL… view at source ↗

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Faithful Decoding
econ.GN 2026-07 conditional novelty 5.0

Fixed-point solutions can be transferred exactly between a high-dimensional problem S=D∘E and a transformed low-dimensional problem T=E∘D, enabling exact dimensionality reduction and debiased stochastic approximation.

Reference graph

Works this paper leans on

11 extracted references · 4 linked inside Pith · cited by 1 Pith paper

[1]

a single state step function, which adds idiosyncratic noise to the deterministic next state and returns the next state indices as well as their probabilities
[2]

an aggregate observation function, which returns the common observation of the aggregate state
[3]

arXiv:2110.05038 [cs]

URLhttp://arxiv.org/abs/2110.05038. arXiv:2110.05038 [cs]. Perolat, J., Perrin, S., Elie, R., Lauri`ere, M., Piliouras, G., Geist, M., Tuyls, K., and Pietquin, O. Scaling up Mean Field Games with Online Mirror Descent, February 2021. URL http://arxiv.org/abs/2103.00623. arXiv:2103.00623 [cs]. Perrin, S. Scaling up Multi-agent Reinforcement Learning with M...

Pith/arXiv arXiv 2021
[8]

analytic mean-field

and a reset function, which determines the exact initial mean-field distribution. Given these functions, the generic environment analytic mean-field update wrapper handles updating the entire mean-field distribution (which involves computing A⊤µ), calculating expectations over next states (such as Av, for example) and returning the reward matrixR. 15 RSPG...
[9]

a single agent step function, which steps a single agent forward given the current individual state, aggregate state and action by sampling idiosyncratic noise and adding that to the deterministic next state from the base environment
[10]

a local observation function, which returns the individual observation of the aggregate state
[11]

a reset function, which samples individual agents from an initial mean-field distribution. Given these functions, the generic environment sample-based wrapper handles updating the entire mean-field distribution by stepping a fixed number of agents forwards and re-computing the mean-field statistics based on the updated samples. Mean-field sequenceThe pseu...

2025
[687]

URLhttps://ieeexplore.ieee.org/document/8030547/

doi: 10.1109/MWC.2017.1600114. URLhttps://ieeexplore.ieee.org/document/8030547/. Yang, J., Ye, X., Trivedi, R., Xu, H., and Zha, H. Learning Deep Mean Field Games for Modeling Large Population Behavior, April 2018b. URLhttp://arxiv.org/abs/1711.03156. arXiv:1711.03156 [cs]. Yang, Y ., Wang, C., Schaab, A., and Moll, B. Structural Reinforcement Learning fo...

arXiv 2017
[2022]

URLhttps://arxiv.org/abs/2112.14377. Hu, A. and Zhang, J. MF-OML: Online Mean-Field Reinforcement Learning with Occupation Measures for Large Population Games, September 2025. URLhttp://arxiv.org/abs/2405.00282. arXiv:2405.00282 [math]. Huang, M., Malhame, R., and Caines, P. Large population stochastic dynamic games: closed-loop mckean-vlasov systems and ...

Pith/arXiv arXiv 2025
[2024]

arXiv:2311.10090 [cs]

URLhttp://arxiv.org/abs/2311.10090. arXiv:2311.10090 [cs]. 13 RSPG for Partially Observable Mean Field Games Saldi, N., Bas ¸ar, T., and Raginsky, M. Approximate Nash Equilibria in Partially Observed Stochastic Games with Mean-Field Interactions.Mathematics of Operations Research, 44(3):1006–1033, August 2019. ISSN 0364-765X, 1526-5471. doi: 10.1287/moor....

Pith/arXiv arXiv 2019
[2025]

arXiv:2304.08630 [cs] version: 2

URLhttp://arxiv.org/abs/2304.08630. arXiv:2304.08630 [cs] version: 2. Han, J., Yang, Y ., and E, W. Deepham: A global solution method for heterogeneous agent models with aggregate shocks,

Pith/arXiv arXiv

[1] [1]

a single state step function, which adds idiosyncratic noise to the deterministic next state and returns the next state indices as well as their probabilities

[2] [2]

an aggregate observation function, which returns the common observation of the aggregate state

[3] [3]

arXiv:2110.05038 [cs]

URLhttp://arxiv.org/abs/2110.05038. arXiv:2110.05038 [cs]. Perolat, J., Perrin, S., Elie, R., Lauri`ere, M., Piliouras, G., Geist, M., Tuyls, K., and Pietquin, O. Scaling up Mean Field Games with Online Mirror Descent, February 2021. URL http://arxiv.org/abs/2103.00623. arXiv:2103.00623 [cs]. Perrin, S. Scaling up Multi-agent Reinforcement Learning with M...

Pith/arXiv arXiv 2021

[4] [8]

analytic mean-field

and a reset function, which determines the exact initial mean-field distribution. Given these functions, the generic environment analytic mean-field update wrapper handles updating the entire mean-field distribution (which involves computing A⊤µ), calculating expectations over next states (such as Av, for example) and returning the reward matrixR. 15 RSPG...

[5] [9]

a single agent step function, which steps a single agent forward given the current individual state, aggregate state and action by sampling idiosyncratic noise and adding that to the deterministic next state from the base environment

[6] [10]

a local observation function, which returns the individual observation of the aggregate state

[7] [11]

a reset function, which samples individual agents from an initial mean-field distribution. Given these functions, the generic environment sample-based wrapper handles updating the entire mean-field distribution by stepping a fixed number of agents forwards and re-computing the mean-field statistics based on the updated samples. Mean-field sequenceThe pseu...

2025

[8] [687]

URLhttps://ieeexplore.ieee.org/document/8030547/

doi: 10.1109/MWC.2017.1600114. URLhttps://ieeexplore.ieee.org/document/8030547/. Yang, J., Ye, X., Trivedi, R., Xu, H., and Zha, H. Learning Deep Mean Field Games for Modeling Large Population Behavior, April 2018b. URLhttp://arxiv.org/abs/1711.03156. arXiv:1711.03156 [cs]. Yang, Y ., Wang, C., Schaab, A., and Moll, B. Structural Reinforcement Learning fo...

arXiv 2017

[9] [2022]

URLhttps://arxiv.org/abs/2112.14377. Hu, A. and Zhang, J. MF-OML: Online Mean-Field Reinforcement Learning with Occupation Measures for Large Population Games, September 2025. URLhttp://arxiv.org/abs/2405.00282. arXiv:2405.00282 [math]. Huang, M., Malhame, R., and Caines, P. Large population stochastic dynamic games: closed-loop mckean-vlasov systems and ...

Pith/arXiv arXiv 2025

[10] [2024]

arXiv:2311.10090 [cs]

URLhttp://arxiv.org/abs/2311.10090. arXiv:2311.10090 [cs]. 13 RSPG for Partially Observable Mean Field Games Saldi, N., Bas ¸ar, T., and Raginsky, M. Approximate Nash Equilibria in Partially Observed Stochastic Games with Mean-Field Interactions.Mathematics of Operations Research, 44(3):1006–1033, August 2019. ISSN 0364-765X, 1526-5471. doi: 10.1287/moor....

Pith/arXiv arXiv 2019

[11] [2025]

arXiv:2304.08630 [cs] version: 2

URLhttp://arxiv.org/abs/2304.08630. arXiv:2304.08630 [cs] version: 2. Han, J., Yang, Y ., and E, W. Deepham: A global solution method for heterogeneous agent models with aggregate shocks,

Pith/arXiv arXiv