Pith. sign in

REVIEW 4 major objections 5 minor 62 references

An Agent-Centric Dynamical Systems Perspective on Multi-Agent Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that much of the apparent randomness in multi-agent reinforcement learning is low-dimensional dynamical structure in the agents' parameter updates, and that stability and sensitivity can be diagnosed from training traces us

desk verdict A useful agent-centric framing for diagnosing MARL stability from parameter traces, but the Markov assumption in Eq. (12) is unexamined for deep RL and the empirical support needs error bars, artifacts, and a more careful novelty claim. read the letter →

arxiv 2512.07588 v3 pith:3RACWTCY submitted 2025-12-08 cs.MA

James Rudd-Jones , María Pérez-Ortiz , Mirco Musolesi This is my paper
classification cs.MA
keywords multi-agentreinforcementlearningdynamicalsystemsstabilityanalysissensitivityLyapunovexponentsrecurrenceplotsstationarydistributionsparameter-space
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-agent reinforcement learning often looks random and unstable: the same algorithm produces divergent, oscillating, or abrupt training runs. This paper argues that much of that apparent randomness is actually low-dimensional structure in a coupled stochastic dynamical system whose state is the agents' parameters. Treating the parameter updates of each agent as coupled maps—rather than averaging them into population-level replicator dynamics—lets existing tools from dynamical systems theory (stationary distributions, Lyapunov exponents, recurrence plots, fractal dimension) measure whether a run is converging, cycling, or becoming chaotic. The paper shows these metrics track observed training behaviour across four repeated games and a cooperative kitchen task, and that sweeping hyperparameters such as discount factor and exploration decay changes the diagnostics in interpretable ways. If the picture holds, practitioners gain a data-driven way to tell stable from fragile training runs and to steer hyperparameters toward stable regimes.

What carries the argument

The load-bearing object is the coupled stochastic map θ_{h+1}=g(θ_h, s_t, π)+ν_h, treated as a Markov process whose invariant distribution ρ(θ) replaces the classical fixed point. Around that map the paper wraps four standard diagnostics: the Frobenius norm of the covariance of ρ, the maximal Lyapunov exponent (divergence rate of nearby parameter traces), recurrence plots (revisitation patterns, with diagonal lines indicating determinism), and the correlation dimension D2 (attractor geometry, D2≈0 fixed point, ≈1 limit cycle, non-integer >1 strange attractor). All are computed empirically from parameter traces, making the framework data-driven; agent coupling enters through the shared state

What would settle it

Compute the maximal Lyapunov exponent and correlation dimension from a long IDQN training trace in Matching Pennies, then recompute both after randomly permuting the temporal order of the parameter vectors. If the diagnostics change only marginally under permutation, the apparent dynamical structure is an artefact of the marginal distribution rather than of the assumed Markov update map; if they change sharply, the temporal coupling is real. This directly tests whether θ_{h+1}=g(θ_h,s_t,π)+ν_h describes the trace.

Watch

Extended reading notes

Core claim

Central claim: individual-agent MARL training can be read as a coupled discrete-time stochastic dynamical system in parameter space. Each agent's update is θ_{h+1}=g(θ_h, s_t, π)+ν_h, with ν_h bundling exploration, transition, and gradient noise; stability is then a property of the stationary distribution ρ(θ) rather than an idealized fixed point. From parameter traces, the paper computes a covariance norm, a maximal Lyapunov exponent, recurrence plots, and a correlation dimension, and uses them to classify runs as noisy fixed points, quasi-cycles, or chaos. Evidence: Prisoner's Dilemma shows a concentrated stationary distribution with near-zero λ_max and low correlation dimension (a stable

Load-bearing premise

The load-bearing premise is that each parameter update is a Markov process in the current parameter vector alone—θ_{h+1}=g(θ_h, s_t, π)+ν_h—so that replay buffers, target networks, and older behaviour policies do not make the next update depend on more than the present θ_h; if that history dependence is real, the stationary distribution, Lyapunov exponents, and recurrence plots computed from raw traces are not well-defined properties of the learning system.

Editorial extensions

If this is right

  • Training runs can be classified post-hoc as noisy fixed point, quasi-cycle, or chaos from parameter traces alone, without needing population-level or mean-field assumptions.
  • Metrics like λ_max and D2 give a quantitative handle on when a MARL system is approaching instability, complementing reward curves that may look similar across regimes.
  • Hyperparameter sensitivity can be mapped: for IDQN in Matching Pennies, increasing the discount factor γ reduces cycling as γ→1, and setting the exploration end value to zero drives D2 and λ_max toward zero, signalling convergence to a fixed point.
  • The diagnostics scale to high-dimensional and deep function-approximation settings, where replicator dynamics or direct phase-portrait visualisation become intractable.
  • The framework suggests control: stability metrics could be used as pseudo-rewards or meta-learning objectives to keep learning in stable, predictable regimes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Markov assumption in Eq. (12) is violated by replay buffers, target networks, and behaviour policies from earlier in training, the diagnostics computed from raw traces may reflect the sampling schedule as much as the learning dynamics; a natural test is to compare metrics on temporally shuffled versus ordered traces.
  • The same parameter-space dynamical view could be applied to single-agent RL or to other stochastic optimisation processes, offering a unified way to compare instability across algorithms.
  • Because the diagnostics are estimated from finite traces, confidence intervals and surrogate-data tests are needed before using them as gates in safety-critical deployments.
  • A testable extension: if λ_max and D2 are valid order parameters, then interventions that reduce them (for example, lowering exploration noise) should, on average, reduce policy oscillation in held-out environments.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes an agent-centric dynamical-systems perspective on MARL, modeling the parameter updates of individual agents as coupled stochastic discrete-time dynamical systems. It introduces a set of established DS diagnostics—stationary distributions, covariance norms, maximal Lyapunov exponents, recurrence plots, and correlation dimension—and applies them to tabular Q-learning, policy gradient, and IDQN agents in four stateless games plus Overcooked. The central claim is that these metrics correspond closely to observed training behavior and thereby provide a practical framework for diagnosing stability and sensitivity of individual-agent learning. The paper also performs hyperparameter sweeps over the discount factor and exploration decay to demonstrate sensitivity analysis, and discusses how these diagnostics could guide control and algorithm design.

Significance. If the methodology is validated, this would be a genuinely useful individual-level complement to population-level replicator-dynamics analysis, with practical implications for stability-aware MARL design and safety assessment. The paper's strengths include a clear formalization of parameter-space dynamics, the pairing of empirical traces with replicator-dynamics vector fields for tabular and policy-gradient baselines, and the use of multiple complementary diagnostics. However, the central empirical claim currently rests on an unvalidated Markov/autonomous assumption for deep RL and on estimator outputs that are reported without error bars or surrogate validation. These gaps prevent the paper from fully establishing the claimed correspondence between DS metrics and observed training behavior, especially for IDQN and the sensitivity sweeps.

major comments (4)
  1. [§3.1, Eq. (12) and §5.1] Eq. (12) models the parameter update as θ_{h+1}=g(θ_h,s_t,π)+ν_h, a time-homogeneous Markov process in θ alone, and Eq. (13) defines a stationary distribution ρ_θ. For IDQN, the update depends on a replay buffer containing transitions generated by older behavior policies, a target network that is a delayed copy of earlier parameters, and an ε-greedy schedule that decays with h. The process is therefore non-autonomous and history-dependent; stationary distributions, Lyapunov exponents, and recurrence plots computed from raw traces may not be well-defined properties of the learning system. The paper states that the definition 'can be extended' to replay buffers and target networks, but no such extension is derived or tested. This is load-bearing for the headline claim, which explicitly includes IDQN and Overcooked. Please provide a validated extension (e.g., by augmenting the state with bu
  2. [Table 1 and Appendix B] Table 1 reports ∥Σ∥_F, λ_max, and D2 as point estimates with no error bars, and Appendix B defines nsteps, nruns, and nburn but never gives their values. Given the substantial stochasticity visible in Figure 1, the regime distinctions (e.g., λ_max≈0 vs. 0.039; D2=0.438 vs. 0.760) may not be statistically meaningful. Please report confidence intervals or seed-wise distributions, and state the actual simulation parameters. Without this, the empirical support for the central claim is under-specified.
  3. [§5.1 and §B.3] The maximal Lyapunov exponent is estimated from finite, noisy parameter traces using a nearest-neighbor divergence method. This estimator is designed for deterministic dynamical systems; on stochastic processes with i.i.d. noise, log-divergence can grow linearly before saturating, producing spurious positive λ_max. The same concern applies to the recurrence-plot determinism interpretation, which is sensitive to autocorrelation. The paper should validate the estimation pipeline on surrogate data (noise-only series and known deterministic systems) and ideally show that the reported distinctions survive such controls. Without this, the claimed connection between positive λ_max and chaotic or quasi-cyclic learning dynamics is not established.
  4. [§5.2 / Figure 4] The sensitivity sweep over γ and ε_end reports λ_max and D2 as functions of the swept parameter, but again with no error bars, replicate counts, or statistical tests. The text makes strong claims such as 'increasing γ reduces cycling in Matching Pennies drastically as γ→1' and that ε_end=0 results in convergence to a fixed point. Given the finite-horizon, non-autonomous issues above, these conclusions require evidence from multiple seeds and, ideally, a statistical comparison across parameter configurations. As presented, Figure 4 may overstate the smoothness and reliability of the parameter dependence.
minor comments (5)
  1. [Abstract] The phrase 'for the first time' is a strong novelty claim; given prior dynamical-systems analyses of learning (e.g., [6,20,34,47]), please temper or clarify precisely what is new with respect to these works.
  2. [Table 1] The entry 'Prisoners’ Dilemma' should be 'Prisoner’s Dilemma' for consistency with the text.
  3. [§5.1] The Lyapunov exponent definition uses the Euclidean norm ||θ_h−θ'_h||; for high-dimensional neural parameters, the nearest-neighbor estimator requires explicit choices of norm and normalization. Please specify these choices.
  4. [§B.4] The recurrence threshold ε is said to be chosen for a desired recurrence rate (e.g., 8%), but the actual rate used in each figure is not stated; please provide the value and any sensitivity analysis.
  5. [Figure 1] It is unclear how the IDQN outputs are mapped to the [0,1] axis for action probabilities. Since the text says IDQN uses Boltzmann exploration, please specify whether the plotted quantity is the Boltzmann softmax probability or something else.

Circularity Check

1 steps flagged · score 3.0 of 10

The 'correspondence' between dynamical-systems metrics and observed MARL behaviour is partly constructed: regime labels are read off the same stationary-distribution estimates that define them, though the replicator-dynamics anchor provides some independent support.

  1. self definitional [§5.1, Eqs. (12)-(13) and diagnostics list; Table 1]
    "Formally, let the update rule for one agent be written as a Markov process: θ_{h+1}=g(θ_h, s_t, π)+ν_h. We define a stationary distribution ρθ that satisfies: θ_h∼ρθ ⇒ θ_{h+1}∼ρθ. ... If the distribution focuses on a contained area of the phase space then we may have a fixed point. ... In Prisoner’s Dilemma λmax≈0 and a fractal dimension D2≈0 indicate a stable fixed point is reached."

    The qualitative regime (fixed point, quasi-cycle, chaos) is defined by properties of the same empirical stationary distribution / recurrence plot from which the diagnostic metrics are computed; there is no held-out or independent measure of 'observed training behaviour' being predicted. The agreement in Table 1 and Figures 2-3 is therefore a re-description of the trace in dynamical-systems vocabulary, not an empirical confirmation. Some external support exists (replicator-dynamics vector field for tabular Q-learning and policy gradient), which prevents this from fully reducing the paper's contribution to its inputs.

full rationale

Apart from the regime-labelling step, the paper's core derivation is not circular: no parameter is fitted to a subset of data and then called a prediction, no uniqueness theorem from the authors is invoked, and the sensitivity sweeps (Figures 4 and 6) vary hyperparameters and report metrics directly. The main self-citations ([3,4,15,43,44]) are contextual and not load-bearing. A separate validity gap, not circularity per se, is that Eq. (12) models θ_h as a Markov process in θ alone, while the IDQN experiments use replay buffers, target networks, and decaying ε-greedy exploration; the paper asserts this 'can be extended' without deriving or testing the extension, so the invariant-distribution estimates for IDQN may not be well-defined. This weakens the central empirical claim but is not an input-output equivalence. Overall, one partial re-description step justifies a score of 3 rather than 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central framework rests on treating parameter updates as Markov autonomous stochastic systems and on interpreting finite noisy traces through attractor diagnostics. Several analysis thresholds and run parameters are chosen by hand and not reported; no new physical or algorithmic entities are introduced.

free parameters (5)
  • Recurrence-plot threshold ε = chosen to achieve recurrence rate ~8%
    Recurrence plots depend on this threshold; chosen by hand, not reported per figure.
  • Theiler window w = not reported
    Used to exclude temporal neighbours when estimating the maximum Lyapunov exponent; not specified in Appendix B.3.
  • Embedding dimension m and lag τ = not reported
    Required for Grassberger-Procaccia D2 estimation; not specified in Appendix B.5.
  • Training horizon n_steps, seeds n_runs, burn-in n_burn = not reported
    Appendix B.1 uses placeholder names; stationary-distribution estimates depend on these values.
  • Learning rates for Q-learning and policy gradient = not reported
    The authors state these were 'tuned to follow replicator dynamics by using very small learning rates'; exact values are omitted, and the replicator-dynamics comparison depends on them.
assumptions (5)
  • domain assumption Parameter updates form a Markov process in θ (Eq. 12)
    Justified for simple stateless learners, but false for IDQN with replay buffers and target networks; the paper does not show diagnostics remain valid under history-dependent updates.
  • domain assumption Empirical trajectories approximate an ergodic stationary distribution ρ_θ
    Stationary distributions are estimated from finite runs after a burn-in; no mixing or ergodicity argument is given.
  • ad hoc to paper Lyapunov exponents estimated from finite noisy parameter traces are meaningful
    Standard Lyapunov-exponent estimation assumes deterministic smooth dynamics; applying it to stochastic MARL traces requires a correction or justification that is not provided.
  • standard math Replicator-dynamics correspondence for tabular Q-learning and REINFORCE with Boltzmann exploration
    Imported from cited results [8,10,11]; used as the external benchmark in §5.1.
  • domain assumption Attractor diagnostics computed on the joint parameter vector summarize high-dimensional learning dynamics
    Used for IDQN and Overcooked, where parameters are high-dimensional; no theoretical support is offered.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Agent-Centric Dynamical Systems Perspective on Multi-Agent Reinforcement Learning." pith.science (2026). https://pith.science/paper/3RACWTCY

@misc{pith2026251207588,
  author       = {Pith},
  title        = {Pith review of: An Agent-Centric Dynamical Systems Perspective on Multi-Agent Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3RACWTCY}},
  note         = {Machine review of arXiv:2512.07588}
}
read the original abstract

Analysing learning in Multi-Agent Reinforcement Learning (MARL) environments is challenging, in particular with respect to \textit{individual} decision-making. Practitioners frequently struggle to compare training runs due to the inherent stochasticity in algorithms arising from random dithering exploration, environment transition noise, and stochastic gradient updates to name a few. Traditional analytical approaches, such as replicator dynamics, oft rely on mean-field approximations to remove stochastic effects, but this simplification, whilst able to provide general overall trends, can lead to dissonance between analytical predictions and actual agent realisations. We propose modelling MARL training as a \textit{coupled stochastic dynamical systems}, capturing both agent interactions and environmental characteristics. Leveraging tools from dynamical systems theory, we pragmatically analyse the stability and sensitivity of agent behaviour, which are key dimensions for their practical deployments, for example, in presence of strict safety requirements. This framework allows us to rigorously study the inherent stochasticity of MARL, providing a deeper understanding of system behaviour.

Figures

Figures reproduced from arXiv: 2512.07588 by the authors.

Figure 1
Figure 1. Comparison between replicator dynamics and realisations of Policy Gradient, Tabular Q-learning, and IDQN with [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Stationary distributions calculated from realisations of training for two IDQN agents in Prisoner’s Dilemma and [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Recurrence plots from a realisation of training two [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Varying 𝛾, the discounting parameter in IDQN, and 𝜖End the end value for 𝜖-greedy exploration in IDQN, to understand their impact on the coupled dynamical system attractor via the Max Lyapunov Exponent and fractal dimension 𝐷2. 0 5000 10000 Time 0 2000 4000 6000 8000 1…
Figure 5
Figure 5. Figure 5: Recurrence plot calculated from a realisation of [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Recurrence plot calculated from a realisation of training two IDQN agents in Matching Pennies. All four combinations [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

62 extracted references · 4 linked inside Pith

  1. [1]

    Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. 2021. Deep reinforcement learning at the edge of the statistical precipice.Advances in Neural Information Processing Systems (NeurIPS’21)(2021)

    2021

  2. [2]

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in AI safety.arXiv preprint arXiv:1606.06565 (2016)

    arXiv 2016

  3. [3]

    Nicolas Anastassacos, Julian Garcia, Stephen Hailes, and Mirco Musolesi. 2021. Cooperation and Reputation Dynamics with Reinforcement Learning . InProceed- ings of the 20th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’21)

    2021

  4. [4]

    Nicolas Anastassacos, Stephen Hailes, and Mirco Musolesi. 2020. Partner Se- lection for the Emergence of Cooperation in Multi-Agent Systems using Rein- forcement Learning. InProceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI’20)

    2020

  5. [5]

    Robert Axelrod and William D Hamilton. 1981. The evolution of cooperation. Science211, 4489 (1981), 1390–1396

    1981

  6. [6]

    Wolfram Barfuss, Jonathan F Donges, and Jürgen Kurths. 2019. Deterministic limit of temporal difference reinforcement learning for stochastic games.Physical Review E99, 4 (2019), 043305

    2019

  7. [7]

    Randall D Beer. 1995. A Dynamical Systems Perspective on Agent-Environment Interaction.Artificial Intelligence72, 1-2 (1995), 173–215

    1995

  8. [8]

    Martino Bernasconi, Federico Cacciamani, Simone Fioravanti, Nicola Gatti, and Francesco Trovò. 2025. The evolutionary dynamics of soft-max policy gradient in multi-agent settings.Theoretical Computer Science1027 (2025), 115011

    2025

Show all 62 references
  1. [9]

    Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. 2019. Dota 2 with large scale deep reinforcement learning.arXiv preprint arXiv:1912.06680(2019)

    2019 arXiv

  2. [10]

    Daan Bloembergen, Karl Tuyls, Daniel Hennes, and Michael Kaisers. 2015. Evo- lutionary Dynamics of Multi-Agent Learning: A Survey.Journal of Artificial Intelligence Research53 (2015)

    2015

  3. [11]

    Tilman Börgers and Rajiv Sarin. 1997. Learning through Reinforcement and Replicator Dynamics.Journal of Economic Theory77, 1 (1997), 1–14

    1997

  4. [12]

    Albrecht Böttcher and David Wenzel. 2008. The Frobenius norm and the commu- tator.Linear algebra and its Applications429, 8-9 (2008), 1864–1885

    2008

  5. [13]

    Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. 2019. On the utility of learning about humans for human-ai coordination. InAdvances in Neural Information Processing Systems (NeurIPS’19)

    2019

  6. [14]

    John G Cross. 1973. A Stochastic Learning Model of Economic Behavior.The Quarterly Journal of Economics87, 2 (1973), 239–266

    1973

  7. [15]

    Nayana Dasgupta and Mirco Musolesi. 2025. Investigating the impact of direct punishment on the emergence of cooperation in multi-agent reinforcement learning systems.Autonomous Agents and Multi-Agent Systems39, 1 (2025), 1–37

    2025

  8. [16]

    Jean-Pierre Eckmann Eckmann, S Oliffson Kamphorst, and David Ruelle. 1995. Recurrence plots of dynamical systems. InTurbulence, Strange Attractors and Chaos. World Scientific, 441–445

    1995

  9. [17]

    Andrea Facchini and Holger Kantz. 2007. Curved structures in recurrence plots: The role of the sampling time.Physical Review E75, 3 (2007), 036215

    2007

  10. [18]

    J Doyne Farmer. 1982. Information dimension and the probabilistic structure of chaos.Zeitschrift für Naturforschung A37, 11 (1982), 1304–1326

    1982

  11. [19]

    Luis H Favela. 2020. Dynamical Systems Theory in Cognitive Science and Neu- roscience.Philosophy Compass15, 8 (2020), e12695

    2020

  12. [20]

    Tobias Galla and J Doyne Farmer. 2013. Complex dynamics in learning com- plicated games.Proceedings of the National Academy of Sciences110, 4 (2013), 1232–1236

    2013

  13. [21]

    Aram Galstyan. 2013. Continuous Strategy Replicator Dynamics for Multi-agent Q-learning.Autonomous Agents and Multi-agent Systems26, 1 (2013), 37–53

    2013

  14. [22]

    Javier Garcıa and Fernando Fernández. 2015. A comprehensive survey on safe reinforcement learning.Journal of Machine Learning Research16, 1 (2015), 1437– 1480

    2015

  15. [23]

    David Goll, Jobst Heitzig, and Wolfram Barfuss. 2024. Deterministic Model of In- cremental Multi-Agent Boltzmann Q-Learning: Transient Cooperation, Metasta- bility, and Oscillations.arXiv preprint arXiv:2501.00160(2024)

    2024 arXiv

  16. [24]

    2023.JAX: Autograd and XLA for high-performance machine learning research

    Brain Team Google Research. 2023.JAX: Autograd and XLA for high-performance machine learning research. https://github.com/google/jax

    2023

  17. [25]

    Peter Grassberger and Itamar Procaccia. 1984. Dimensions and entropies of strange attractors from a fluctuating dynamics approach.Physica D: Nonlinear Phenomena13, 1-2 (1984), 34–54

    1984

  18. [26]

    Harris, K

    Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fe...

    2020

  19. [27]

    Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. 2018. Deep reinforcement learning that matters. InProceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI’18)

    2018

  20. [28]

    Daniel Hennes, Dustin Morrill, Shayegan Omidshafiei, Rémi Munos, Julien Pero- lat, Marc Lanctot, Audrunas Gruslys, Jean-Baptiste Lespiau, Paavo Parmas, Edgar Duéñez-Guzmán, et al. 2020. Neural Replicator Dynamics: Multiagent Learning via Hedging Policy Gradients. InProceedings...

    2020

  21. [29]

    Daniel Hennes, Karl Tuyls, and Matthias Rauterberg. 2009. State-Coupled Replica- tor Dynamics. InProceedings of the 21st International Joint Conference on Artificial Intelligence (IJCAI’09)

    2009

  22. [30]

    1998.Evolutionary Games and Population Dynamics

    Josef Hofbauer and Karl Sigmund. 1998.Evolutionary Games and Population Dynamics. Cambridge University Press

    1998

  23. [31]

    Harold Hotelling. 1933. Analysis of a complex of statistical variables into principal components.Journal of Educational Psychology24, 6 (1933), 417

    1933

  24. [32]

    Aamal Hussain, Francesco Belardinelli, and Georgios Piliouras. 2023. Beyond Strict Competition: Approximate Convergence of Multi Agent Q-Learning Dy- namics. InProceedings of the 32nd International Joint Conference on Artificial Intelligence (IJCAI’23)

    2023

  25. [33]

    Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Grae- pel

    Joel Z. Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Grae- pel. 2017. Multi-Agent Reinforcement Learning in Sequential Social Dilemmas. InProceedings of the 16th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’17)

    2017

  26. [34]

    Chin-wing Leung, Shuyue Hu, and Ho-fung Leung. 2023. The stochastic evo- lutionary dynamics of softmax policy gradient in games. InProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS’24)

    2023

  27. [35]

    Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. 2021. Fourier Neural Operator for Parametric Partial Differential Equations. InProceedings of the 9th International Conference on Learning Representatio...

    2021

  28. [36]

    Aleksandr Mikhailovich Lyapunov. 1992. The general problem of the stability of motion.Internat. J. Control55, 3 (1992), 531–534

    1992

  29. [37]

    Norbert Marwan, M Carmen Romano, Marco Thiel, and Jürgen Kurths. 2007. Recurrence plots for the analysis of complex systems.Physics Reports438, 5-6 (2007), 237–329

    2007

  30. [38]

    1982.Evolution and the Theory of Games

    John Maynard Smith. 1982.Evolution and the Theory of Games. Cambridge University Press

    1982

  31. [39]

    Ratliff, Michael I

    Eric Mazumdar, Lillian J. Ratliff, Michael I. Jordan, and S. Shankar Sastry. 2020. Policy-Gradient Algorithms Have No Guarantees of Convergence in Linear Qua- dratic Games. InProceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’20)

    2020

  32. [40]

    Guido Novati, Hugues Lascombes de Laroussilhe, and Petros Koumoutsakos

  33. [41]

    Martin A Nowak. 2006. Five Rules for the Evolution of Cooperation.Science314, 5805 (2006), 1560–1563

    2006

  34. [42]

    Hannes Risken. 1989. Fokker-Planck equation. InThe Fokker-Planck Equation: Methods of Solution and Applications. Springer, 63–95

    1989

  35. [43]

    James Rudd-Jones, Mirco Musolesi, and María Pérez-Ortiz. 2025. Multi-Agent Reinforcement Learning Simulation for Environmental Policy Synthesis. InPro- ceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’25)

    2025

  36. [44]

    James Rudd-Jones, Fiona Thendean, and María Pérez-Ortiz. 2025. Craft- ing desirable climate trajectories with reinforcement learning explored socio- environmental simulations.Environmental Data Science4 (2025), e41

    2025

  37. [45]

    Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Garðar Ingvarsson Juto, Timon Willi, Ravi Hammond, Akbir Khan, Chris- tian Schroeder de Witt, et al. 2024. JaxMARL: Multi-Agent Reinforcement Learn- ing Environments and Algorithms in JAX.Advance...

    2024

  38. [46]

    James BT Sanders, J Doyne Farmer, and Tobias Galla. 2018. The prevalence of chaotic dynamics in games with many players.Scientific Reports8, 1 (2018), 4902

    2018

  39. [47]

    Yuzuru Sato, Eizo Akiyama, and J Doyne Farmer. 2002. Chaos in learning a simple two-person game.Proceedings of the National Academy of Sciences99, 7 (2002), 4748–4751

    2002

  40. [48]

    Jeffrey D Scargle. 1989. An introduction to chaotic and random time series analysis.International Journal of Imaging Systems and Technology1, 2 (1989), 243–253

    1989

  41. [49]

    2005.Deterministic Chaos: An Introduc- tion

    Heinz Georg Schuster and Wolfram Just. 2005.Deterministic Chaos: An Introduc- tion. Wiley

    2005

  42. [50]

    2024.Nonlinear Dynamics and Chaos: with Applications to Physics, Biology, Chemistry, and Engineering

    Steven H Strogatz. 2024.Nonlinear Dynamics and Chaos: with Applications to Physics, Biology, Chemistry, and Engineering. Chapman and Hall/CRC

    2024

  43. [51]

    Pol Suárez, Francisco Alcántara-Ávila, Arnau Miró, Jean Rabault, Bernat Font, Oriol Lehmkuhl, and Ricardo Vinuesa. 2025. Active Flow Control for Drag Reduction Through Multi-agent Reinforcement Learning on a Turbulent Cylinder at𝑅𝑒 𝐷 =3900.Flow, Turbulence and Combustion114, 3...

    2025

  44. [52]

    George Sugihara, Robert May, Hao Ye, Chih-hao Hsieh, Ethan Deyle, Michael Fogarty, and Stephan Munch. 2012. Detecting causality in complex ecosystems. Science338, 6106 (2012), 496–500

    2012

  45. [53]

    Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Kor- jus, Juhan Aru, Jaan Aru, and Raul Vicente. 2017. Multiagent cooperation and competition with deep reinforcement learning.PLOS One12, 4 (2017), e0172395

    2017

  46. [54]

    James Theiler. 1990. Estimating fractal dimension.Journal of the Optical Society of America A7, 6 (1990), 1055–1073

    1990

  47. [55]

    Karl Tuyls, Pieter Jan’T Hoen, and Bram Vanschoenwinkel. 2006. An Evolutionary Dynamical Analysis of Multi-Agent Learning in Iterated Games.Autonomous Agents and Multi-Agent Systems12, 1 (2006), 115–153

    2006

  48. [56]

    Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, An- drew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. 2019. Grandmaster level in StarCraft II using multi-agent reinforcement learning.Nature575, 7782 (2019), 350–354

    2019

  49. [57]

    1989.Learning from Delayed Re- wards

    Christopher John Cornish Hellaby Watkins. 1989.Learning from Delayed Re- wards. PhD Thesis. King’s College, University of Cambridge, Cambridge, United Kingdom

    1989

  50. [58]

    Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang

  51. [59]

    Yaodong Yang and Jun Wang. 2020. An overview of multi-agent reinforcement learning from game theoretical perspective.arXiv preprint arXiv:2011.00583 (2020)

    2020 arXiv

  52. [60]

    Tianyu Zhang, Andrew Williams, Phillip Wozny, Kai-Hendrik Cohrs, Koen Ponse, Marco Jiralerspong, Soham Phade, Sunil Srinivasa, Lu Li, Yang Zhang, Prateek Gupta, Erman Acar, Irina Rish, Yoshua Bengio, and Stephan Zheng. 2025. AI for Global Climate Cooperation: Modeling Global C...

    2025

  53. [2018]

    InProceedings of the 35th International Conference on Machine Learning (ICML’18), Vol

    Mean Field Multi-Agent Reinforcement Learning. InProceedings of the 35th International Conference on Machine Learning (ICML’18), Vol. 80. 5571–5580

  54. [2021]

    Nature Machine Intelligence3, 1 (2021), 87–96

    Automating turbulence modelling by multi-agent reinforcement learning. Nature Machine Intelligence3, 1 (2021), 87–96

    2021

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.