Pith. sign in

REVIEW 1 cited by

Deterministic Model of Incremental Multi-Agent Boltzmann Q-Learning: Transient Cooperation, Metastability, and Oscillations

T0 review · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A first-principles model of incremental Boltzmann Q-learning shows apparent cooperation in the Prisoner's Dilemma is a metastable transient, and discounting drives oscillations through a Neimark-Sacker bifurcation.

desk verdict A serious, mostly sound paper that correctly separates FAQL/BQL from incremental Q-learning and offers a plausible 4D mean-field model; its main weakness is that the link to the actual stochastic algorithm is empirical, not theorem-backed. read the letter →

arxiv 2501.00160 v1 pith:UTT4ZYNC submitted 2024-12-30 cs.MA nlin.AOphysics.soc-ph

David Goll , Jobst Heitzig , Wolfram Barfuss This is my paper · ORCID
classification cs.MAnlin.AOphysics.soc-ph
keywords independentQ-learningBoltzmannexplorationdeterministicapproximationmodelPrisoner'sDilemmaNeimark-Sackerbifurcationmetastabilitymulti-agentreinforcementlearningmovingtargetproblem
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that standard deterministic approximations of independent Boltzmann Q-learning in games—the frequency-adjusted and batch Q-learning models—describe modified algorithms rather than the original incremental update rule, in which only the chosen action's Q-value changes. It introduces a discrete-time map built from the one-step conditional expectation of the stochastic update, replacing the random update indicator by the action's current Boltzmann probability, and shows that this map reproduces the qualitative behaviour of stochastic runs in the two-agent Prisoner's Dilemma. The central results are that the logit Quantal Response equilibrium is stable only for small discount factors, loses stability at $\gamma \approx 0.75$ through a supercritical Neimark-Sacker bifurcation, and that apparent long-term cooperation is a metastable phase lasting up to billions of steps before collapsing. If the model is right, prior policy-space analyses of Q-learning have been studying different algorithms, and quasi-equilibrium cooperation observed in multi-agent learning should be treated as potentially exploitable transient behaviour.

What carries the argument

The load-bearing object is the one-step conditional-expectation map that replaces the indicator $\delta_{A_i(t),a_i}$ in the stochastic update with the Boltzmann probability $\pi_i^{a_i}(t)$ of choosing that action. This keeps the learning dynamics in four-dimensional Q-space instead of collapsing them into two-dimensional policy space, because the update probabilities multiply the reward and target terms in a way that cannot be rewritten using only the Q-value difference $\Delta Q_i$. The map's fixed point $Q^*$ is characterised by Eq. (15), and the stability analysis is carried out on the eigenvalues of the Jacobian at that fixed point; for $T=1$ the critical eigenvalue pair crosses the unit circle at $\gamma_{cr}\approx0.75$, giving the supercritical Neimark-Sacker bifurcation and the stable limit cycle. The metastable phases are driven by the slow updating of a rarely chosen action's Q-value: when $\pi_i^C$ is near zero, $Q_i^C$ is updated so infrequently that it takes millions to billions of steps to approach its target, so the policy looks frozen even though the system is far from equilibrium.

What would settle it

Run many long stochastic simulations of the original algorithm in this Prisoner's Dilemma with $T=1$, $\alpha=0.01$, $\gamma=0.8$, and an asymmetric initial policy such as $\pi_C=(0.5,0.48)$: if over tens of millions of steps the policy distribution does not settle onto the limit cycle predicted by the deterministic map, or instead converges to the logit QRE for most seeds, the central claim fails. For the metastability claim, start from $\pi_C=0.9$ with $Q_{base}=0$ and $\gamma=0.8$ and check whether the escape from mutual cooperation occurs on the roughly billion-step timescale predicted by the map, or whether the predicted dependence of escape time on learning rate and discount factor is contradicted by rescaled shorter runs.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the Kronecker-delta choice of which Q-value gets updated is not a technical detail: it changes the asymptotic dynamics. The conditional-expectation model $E[Q_i^a(t+1)|Q(t)] = Q_i^a(t) + \alpha \pi_i^a(t)\left(E_{A^{-i}\sim\pi^{-i}} R_i^{a,A^{-i}} + \gamma \max_b Q_i^b(t) - Q_i^a(t)\right)$ shares its policy-space fixed point with the FAQL/BQL models—the logit Quantal Response equilibrium (the soft best-response equilibrium), with $\pi_C^*\approx0.227$ for $T=1$—but that fixed point's stability depends on $\gamma$. At $T=1,\alpha=0.01$ it is a stable focus for $\gamma\lesssim0.75$; at $\gamma_{cr}\approx0.75$ a supercritical Neimark-Sacker bifurcation makes it an unstable focus encircled by a stable limit cycle, so asymmetric initial conditions lead to permanent oscillations rather than convergence. For $\gamma\gtrsim0.95$ the fixed point becomes a saddle. The apparent convergence to mutual cooperation, e.g. from initial $\pi_C=0.9$ at $\gamma=0.8$, is shown to be a metastable phase lasting up to roughly four billion steps, after which the trajectory escapes; the paper's stability analysis of all four Q-values, not just the two policies, explains both the slow escape and the oscillations.

Load-bearing premise

The entire analysis assumes that iterating the one-step average of the random update gives an accurate picture of what the stochastic algorithm does over long times, even though no formal limit theorem ties the deterministic map's attractors or transient durations to the random process; the support shown is a visual comparison with five stochastic runs.

Editorial extensions

If this is right

  • With discount factors above the bifurcation threshold, two Boltzmann Q-learners in the Prisoner's Dilemma do not converge to their logit Quantal Response equilibrium; they settle onto a stable limit cycle and keep oscillating indefinitely.
  • The discount factor, though it does not move the fixed-point policy, becomes a control parameter for convergence: small values give convergence, intermediate values give oscillations, and large values turn the fixed point into a saddle.
  • Apparent cooperation in social dilemmas observed over finite simulation horizons should not be read as equilibrium behaviour, because the metastable phase eventually collapses and is exploitable in the meantime.
  • Results derived from frequency-adjusted or batch Q-learning models cannot be transferred to the original incremental algorithm without re-deriving them under update frequencies.
  • The four-dimensional deterministic map provides a cheap way to map parameter regimes into convergence, oscillation, or metastable behaviour before running long stochastic simulations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested extension of the same mechanism: in multi-state environments or games with rarely visited states, the same update-frequency lag should create analogous metastable and oscillatory phases; a generalised conditional-expectation map could be checked against simulations before attributing those phenomena to other causes.
  • Because the deterministic map omits update variance, adding a small noise term should predict not just qualitative regimes but quantitative exit times from metastable phases and the average period of the limit cycle, which would be a sharper test of the map's faithfulness.
  • For symmetric two-action games more generally, the critical discount factor could be derived as a function of payoffs, temperature, and learning rate; if the predicted bifurcation threshold matched stochastic simulations across a family of payoff matrices, the mechanism would be confirmed as general rather than specific to the Prisoner's Dilemma.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the deterministic model is derived from the algorithm's update rule, not from its conclusions.

full rationale

The paper's central derivation is self-contained rather than circular. The deterministic approximation (14) is constructed explicitly from the stochastic update (2) by replacing the Kronecker delta δ_{A_i(t),a_i} with the Boltzmann probability π_i^{a_i}(t), i.e., by taking the one-step conditional expectation of the update, and no parameter is fitted to the phenomena (metastability, oscillations, bifurcation) that the model then explains. The fixed point of (14) is computed from the model's own stationarity condition (15), and the claimed equality in policy space with the logit QRE of the BQL model is a derived algebraic consequence, not an input. The stability and bifurcation analysis is a numerical linearization of the derived four-dimensional map, and the visual comparisons with stochastic runs in Figures 1 and 2 are offered as supporting evidence rather than as fitted calibration. The paper does cite prior work by its own authors, notably Barfuss et al. (2019) and Barfuss (2022), but these citations are used to describe and critique the previous BQL model that the paper explicitly argues is a different algorithm, not to justify the paper's own positive claims. Other self-citations, such as Kittel et al. (2017) and Barfuss and Meylahn (2023), are contextual and not load-bearing. The main weakness—that iterating the one-step conditional expectation (14) is not formally proven to reproduce the stochastic process (2) for fixed α, without an α→0 limit or noise-rescaling argument—is a validity or correctness concern about the model's faithfulness, not a circularity in the derivation. The paper itself acknowledges this limitation by stating that the model 'cannot precisely capture the exact timing of specific runs' and that adding noise would improve predictions about exit times and oscillation periods. Therefore, no step reduces to its own input by construction, and the circularity score is low.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central model rests on one domain assumption: that iterating the conditional expectation (14) is a faithful proxy for the stochastic Q-learning process, especially over very long timescales. The paper provides visual comparisons with a few stochastic runs but no formal approximation theorem. A second assumption fixes the maximizing action to defection in the local stability analysis; this holds at the fixed point and in its neighborhood because the expected payoff of defection always exceeds cooperation in the Prisoner's Dilemma, but its global validity along limit cycles is not examined. The model is parameter-free in the sense that simulation settings (α=0.01, T=1) are inputs, not fitted values, and the bifurcation threshold γ_cr is computed from the model.

assumptions (3)
  • domain assumption Iterating the one-step conditional expectation (14) yields trajectories that faithfully represent the stochastic Q-learning process, including asymptotic regimes such as limit cycles and extremely long transients.
    No formal limit theorem (e.g., stochastic approximation with α→0) is provided; support is empirical comparison in figures 1-3.
  • domain assumption At the unique fixed point Q*, each agent's maximizing action is defection, Q_D* > Q_C*, so the max term in (14) is Q_D for both agents in the linear stability analysis.
    Used in Appendix D to simplify the Jacobian; holds because the expected reward for D exceeds that for C in the Prisoner's Dilemma for any opponent policy.
  • domain assumption The single-state repeated environment with discount factor γ retained, interpreted as the continuation probability of the interaction, is the correct formalization for the finitely repeated normal-form game.
    Stated in Section 2.1; alternative formalizations set γ=0 by entering a terminal state after each round, which would remove the oscillation mechanism.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deterministic Model of Incremental Multi-Agent Boltzmann Q-Learning: Transient Cooperation, Metastability, and Oscillations." pith.science (2026). https://pith.science/paper/UTT4ZYNC

@misc{pith2026250100160,
  author       = {Pith},
  title        = {Pith review of: Deterministic Model of Incremental Multi-Agent Boltzmann Q-Learning: Transient Cooperation, Metastability, and Oscillations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UTT4ZYNC}},
  note         = {Machine review of arXiv:2501.00160}
}
read the original abstract

Multi-Agent Reinforcement Learning involves agents that learn together in a shared environment, leading to emergent dynamics sensitive to initial conditions and parameter variations. A Dynamical Systems approach, which studies the evolution of multi-component systems over time, has uncovered some of the underlying dynamics by constructing deterministic approximation models of stochastic algorithms. In this work, we demonstrate that even in the simplest case of independent Q-learning with a Boltzmann exploration policy, significant discrepancies arise between the actual algorithm and previous approximations. We elaborate why these models actually approximate interesting variants rather than the original incremental algorithm. To explain the discrepancies, we introduce a new discrete-time approximation model that explicitly accounts for agents' update frequencies within the learning process and show that its dynamics fundamentally differ from the simplified dynamics of prior models. We illustrate the usefulness of our approach by applying it to the question of spontaneous cooperation in social dilemmas, specifically the Prisoner's Dilemma as the simplest case study. We identify conditions under which the learning behaviour appears as long-term stable cooperation from an external perspective. However, our model shows that this behaviour is merely a metastable transient phase and not a true equilibrium, making it exploitable. We further exemplify how specific parameter settings can significantly exacerbate the moving target problem in independent learning. Through a systematic analysis of our model, we show that increasing the discount factor induces oscillations, preventing convergence to a joint policy. These oscillations arise from a supercritical Neimark-Sacker bifurcation, which transforms the unique stable fixed point into an unstable focus surrounded by a stable limit cycle.

Figures

Figures reproduced from arXiv: 2501.00160 by the authors.

Figure 1
Figure 1. Comparison between a single run of independent Q-learning on the Prisoner’s Dilemma [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Comparison between averaged policy trajectories of independent Q-learning on the [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Projection of our 4D deterministic approximation model of independent Q-learning on [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Stability analysis of our model, defined by [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Projection of 4D deterministic dynamics of independent Q-learning on the Prisoner’s [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. An Agent-Centric Dynamical Systems Perspective on Multi-Agent Reinforcement Learning

    cs.MA 2025-12 conditional novelty 5.0 of 10

    Treating MARL training as coupled stochastic dynamical systems lets Lyapunov exponents, recurrence plots, and fractal dimensions characterize individual-agent stability and sensitivity.

Reference graph

Works this paper leans on

62 extracted references · 49 canonical work pages · cited by 1 Pith paper

  1. [1]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL http://incompleteideas.net/book/the-book-2nd.html. 1, 4, 13

    work page 2018

  2. [2]

    Q-learning

    Christopher Watkins and Peter Dayan. Q-learning. Machine Learning, 8(3):279–292, 1992. ISSN 1573-0565. doi: 10.1007/BF00992698. 1, 3, 4, 14

    doi:10.1007/bf00992698 1992

  3. [3]

    A neural substrate of prediction and reward

    Wolfram Schultz, Peter Dayan, and P Read Montague. A neural substrate of prediction and reward. Science, 275(5306):1593–1599, 1997. 1

    work page 1997

  4. [4]

    Reinforcement learning: the good, the bad and the ugly

    Peter Dayan and Yael Niv. Reinforcement learning: the good, the bad and the ugly. Current opinion in neurobiology , 18(2):185–196, 2008

    work page 2008

  5. [5]

    Read Montague

    Wolfram Schultz, Peter Dayan, and P. Read Montague. A neural substrate of prediction and reward. Science, 275(5306):1593–1599, 1997. doi: 10.1126/science.275.5306.1593. URL https://www.science.org/doi/abs/10.1126/science.275.5306.1593. 1

    arXiv 1997

  6. [6]

    Playing atari with deep reinforcement learning, 2013

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013. URL https://arxiv.org/abs/1312.5602. 1

    arXiv 2013

  7. [7]

    Human-level control through deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015. 16

    work page 2015

  8. [8]

    Mastering the game of go with deep neural networks and tree search

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587): 484–489, 2016. 1

    work page 2016

Show all 62 references
  1. [9]

    Open problems in cooperative ai

    Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R McKee, Joel Z Leibo, Kate Larson, and Thore Graepel. Open problems in cooperative ai. arXiv preprint arXiv:2012.08630, 2020. 1

    2012 arXiv

  2. [10]

    Cooperative ai: machines must learn to find common ground, 2021

    Allan Dafoe, Yoram Bachrach, Gillian Hadfield, Eric Horvitz, Kate Larson, and Thore Graepel. Cooperative ai: machines must learn to find common ground, 2021. 1

    2021

  3. [11]

    Albrecht, Filippos Christianos, and Lukas Sch¨ afer

    Stefano V. Albrecht, Filippos Christianos, and Lukas Sch¨ afer. Multi-Agent Reinforcement Learning: Foundations and Modern Approaches . MIT Press, 2024. URL https://www. marl-book.com. 1, 2, 3, 4, 13, 14

    2024

  4. [12]

    Multi-agent reinforcement learning: independent vs

    Ming Tan. Multi-agent reinforcement learning: independent vs. cooperative agents , page 487–494. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1997. ISBN 1558604952. 1

    1997

  5. [13]

    A survey of learning in multiagent environments: Dealing with non-stationarity

    Pablo Hernandez-Leal, Michael Kaisers, Tim Baarslag, and Enrique Munoz De Cote. A survey of learning in multiagent environments: Dealing with non-stationarity. arXiv preprint arXiv:1707.09183, 2017. 1, 2, 4, 13

    2017 arXiv

  6. [14]

    Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems

    Laetitia Matignon, Guillaume J Laurent, and Nadine Le Fort-Piat. Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems. The Knowledge Engineering Review , 27(1):1–31, 2012. 2

    2012

  7. [15]

    A survey and critique of multiagent deep reinforcement learning

    Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. A survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Multi-Agent Systems , 33(6):750–797,

  8. [16]

    Benchmark- ing multi-agent deep reinforcement learning algorithms in cooperative tasks

    Georgios Papoudakis, Filippos Christianos, Lukas Sch¨ afer, and Stefano V Albrecht. Benchmark- ing multi-agent deep reinforcement learning algorithms in cooperative tasks. arXiv preprint arXiv:2006.07869, 2020. 2

    2006 arXiv

  9. [17]

    Learning through reinforcement and replicator dynam- ics

    Tilman B¨ orgers and Rajiv Sarin. Learning through reinforcement and replicator dynam- ics. Journal of Economic Theory , 77(1):1–14, 1997. ISSN 0022-0531. doi: https:// doi.org/10.1006/jeth.1997.2319. URL https://www.sciencedirect.com/science/article/ pii/S002205319792319X. 2, 4

    1997

  10. [18]

    A selection-mutation model for q-learning in multi-agent systems

    Karl Tuyls, Katja Verbeeck, and Tom Lenaerts. A selection-mutation model for q-learning in multi-agent systems. In Proceedings of the Second International Joint Conference on Autonomous Agents and Multiagent Systems , AAMAS ’03, page 693–700, New York, NY, USA,

  11. [19]

    Doyne Farmer

    Yuzuru Sato, Eizo Akiyama, and J. Doyne Farmer. Chaos in learning a simple two-person game. Proceedings of the National Academy of Sciences , 99(7):4748–4751, 2002. doi: 10.1073/ pnas.032086299. URL https://www.pnas.org/doi/abs/10.1073/pnas.032086299

    2002 doi

  12. [20]

    Crutchfield

    Yuzuru Sato and James P. Crutchfield. Coupled replicator equations for the dynamics of learning in multiagent systems. Physical Review E , 67(1), January 2003. ISSN 1095-3787. doi: 10.1103/physreve.67.015206. URL http://dx.doi.org/10.1103/PhysRevE.67.015206. 2, 4

    2003 doi

  13. [21]

    Crutchfield

    Yuzuru Sato, Eizo Akiyama, and James P. Crutchfield. Stability and diversity in collective adaptation. Physica D: Nonlinear Phenomena , 210(1–2):21–57, October 2005. ISSN 0167-

    2005

  14. [22]

    Individual q-learning in normal form games

    David S Leslie and Edmund J Collins. Individual q-learning in normal form games. SIAM Journal on Control and Optimization , 44(2):495–514, 2005. 2, 5, 9 17

    2005

  15. [23]

    Reinforcement learning dynamics in social dilemmas

    Segismundo S Izquierdo, Luis R Izquierdo, and Nicholas M Gotts. Reinforcement learning dynamics in social dilemmas. Journal of Artificial Societies and Social Simulation , 11(2):1,

  16. [24]

    Learning and equilibrium

    Drew Fudenberg and David K Levine. Learning and equilibrium. Annu. Rev. Econ. , 1(1): 385–420, 2009

    2009

  17. [25]

    Intrinsic noise in game dynamical learning

    Tobias Galla. Intrinsic noise in game dynamical learning. Phys. Rev. Lett. , 103:198702, Nov

  18. [26]

    A theoretical analysis of temporal difference learning in the iterated prisoner’s dilemma game

    Naoki Masuda and Hisashi Ohtsuki. A theoretical analysis of temporal difference learning in the iterated prisoner’s dilemma game. Bulletin of mathematical biology , 71:1818–1850, 2009. 2

    2009

  19. [27]

    Classes of multiagent q-learning dynamics with epsilon-greedy exploration

    Michael Wunder, Michael L Littman, and Monica Babes. Classes of multiagent q-learning dynamics with epsilon-greedy exploration. In Proceedings of the 27th International Conference on Machine Learning (ICML-10) , pages 1167–1174, 2010. 3

    2010

  20. [28]

    Numerical analysis of a reinforcement learning model with the dynamic aspiration level in the iterated prisoner’s dilemma

    Naoki Masuda and Mitsuhiro Nakamura. Numerical analysis of a reinforcement learning model with the dynamic aspiration level in the iterated prisoner’s dilemma. Journal of theoretical biology, 278(1):55–62, 2011. 2

    2011

  21. [29]

    Cycles of cooperation and defection in imperfect learning

    Tobias Galla. Cycles of cooperation and defection in imperfect learning. Journal of Statistical Mechanics: Theory and Experiment , 2011(08):P08007, aug 2011. doi: 10.1088/1742-5468/ 2011/08/P08007. URL https://dx.doi.org/10.1088/1742-5468/2011/08/P08007

    2011 doi

  22. [30]

    Dynamics of boltzmann q learning in two-player two-action games

    Ardeshir Kianercy and Aram Galstyan. Dynamics of boltzmann q learning in two-player two-action games. Phys. Rev. E , 85:041145, Apr 2012. doi: 10.1103/PhysRevE.85.041145. URL https://link.aps.org/doi/10.1103/PhysRevE.85.041145. 2, 3

    2012 doi

  23. [31]

    Continuous strategy replicator dynamics for multi-agent q-learning

    Aram Galstyan. Continuous strategy replicator dynamics for multi-agent q-learning. Au- tonomous agents and multi-agent systems , 26:37–53, 2013. 2, 3

    2013

  24. [32]

    Doyne Farmer

    Tobias Galla and J. Doyne Farmer. Complex dynamics in learning complicated games. Proceedings of the National Academy of Sciences , 110(4):1232–1236, 2013. doi: 10.1073/pnas. 1109672110. URL https://www.pnas.org/doi/abs/10.1073/pnas.1109672110

    2013 doi

  25. [33]

    Evolutionary dynamics of multi-agent learning: A survey

    Daan Bloembergen, Karl Tuyls, Daniel Hennes, and Michael Kaisers. Evolutionary dynamics of multi-agent learning: A survey. Journal of Artificial Intelligence Research , 53:659–697, 2015. 2

    2015

  26. [34]

    James B. T. Sanders, J. Doyne Farmer, and Tobias Galla. The prevalence of chaotic dynamics in games with many players. Scientific Reports , 8(1):4902, 2018. ISSN 2045-2322. doi: 10.1038/s41598-018-22013-5. URL https://doi.org/10.1038/s41598-018-22013-5

    2018 doi

  27. [35]

    Donges, and J¨ urgen Kurths

    Wolfram Barfuss, Jonathan F. Donges, and J¨ urgen Kurths. Deterministic limit of temporal difference reinforcement learning for stochastic games. Phys. Rev. E , 99:043305, Apr 2019. doi: 10.1103/PhysRevE.99.043305. URL https://link.aps.org/doi/10.1103/PhysRevE. 99.043305. 2, 5, 9

    2019 doi

  28. [36]

    Modelling the dynamics of multiagent q-learning in repeated symmetric games: a mean field theoretic approach

    Shuyue Hu, Chin-wing Leung, and Ho-fung Leung. Modelling the dynamics of multiagent q-learning in repeated symmetric games: a mean field theoretic approach. Advances in Neural Information Processing Systems, 32, 2019

    2019

  29. [37]

    Dynamical systems as a level of cognitive analysis of multi-agent learning

    Wolfram Barfuss. Dynamical systems as a level of cognitive analysis of multi-agent learning. Neural Computing and Applications , 34:1653–1671, 2022. doi: 10.1007/s00521-021-06117-0. URL https://doi.org/10.1007/s00521-021-06117-0 . 2, 5

    2022 doi

  30. [38]

    The dynamics of q- learning in population games: A physics-inspired continuity equation model

    Shuyue Hu, Chin-Wing Leung, Ho-fung Leung, and Harold Soh. The dynamics of q- learning in population games: A physics-inspired continuity equation model. arXiv preprint arXiv:2203.01500, 2022. 3, 9, 10

    2022 arXiv

  31. [39]

    A formal model for multiagent q-learning dynamics on regular graphs

    Chen Chu, Yong Li, Jinzhuo Liu, Shuyue Hu, Xuelong Li, and Zhen Wang. A formal model for multiagent q-learning dynamics on regular graphs. In IJCAI, pages 194–200, 2022. 18

    2022

  32. [40]

    Modeling the effects of environmental and perceptual uncertainty using deterministic reinforcement learning dynamics with partial observability

    Wolfram Barfuss and Richard P Mann. Modeling the effects of environmental and perceptual uncertainty using deterministic reinforcement learning dynamics with partial observability. Physical Review E , 105(3):034409, 2022

    2022

  33. [41]

    Exploration-exploitation in multi-agent learning: Catastrophe theory meets game theory

    Stefanos Leonardos and Georgios Piliouras. Exploration-exploitation in multi-agent learning: Catastrophe theory meets game theory. Artificial Intelligence, 304:103653, 2022. 2, 3

    2022

  34. [42]

    Frequency adjusted multi-agent q-learning

    Michael Kaisers and Karl Tuyls. Frequency adjusted multi-agent q-learning. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: volume 1-Volume 1, pages 309–316, 2010. 2, 3, 5, 9

    2010

  35. [43]

    Evolutionary multi-agent reinforcement learning in group social dilemmas

    Brian Mintz and Feng Fu. Evolutionary multi-agent reinforcement learning in group social dilemmas. arXiv preprint arXiv:2411.10459 , 2024. 2, 3

    2024 arXiv

  36. [44]

    Sandholm and Robert H

    Tuomas W. Sandholm and Robert H. Crites. Multiagent reinforcement learning in the iterated prisoner’s dilemma. Biosystems, 37(1):147–166, 1996. ISSN 0303-2647. doi: https: //doi.org/10.1016/0303-2647(95)01551-5. URL https://www.sciencedirect.com/science/ article/pii/0303264795...

    1996

  37. [45]

    Faq-learning in matrix games: Demonstrating convergence near nash equilibria, and bifurcation of attractors in the battle of sexes

    Michael Kaisers and Karl Tuyls. Faq-learning in matrix games: Demonstrating convergence near nash equilibria, and bifurcation of attractors in the battle of sexes. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence , 2011

    2011

  38. [46]

    L´ evy noise promotes cooperation in the prisoner’s dilemma game with reinforcement learning

    Lu Wang, Danyang Jia, Long Zhang, Peican Zhu, Matjaˇ z Perc, Lei Shi, and Zhen Wang. L´ evy noise promotes cooperation in the prisoner’s dilemma game with reinforcement learning. Nonlinear Dynamics, 108(2):1837–1845, 2022

    2022

  39. [47]

    Limiting dynamics for q-learning with memory one in symmetric two-player, two-action games

    Janusz M Meylahn and Lars Janssen. Limiting dynamics for q-learning with memory one in symmetric two-player, two-action games. Complexity, 2022(1):4830491, 2022

    2022

  40. [48]

    Q-learners can provably collude in the iterated prisoner’s dilemma

    Quentin Bertrand, Juan Duque, Emilio Calvano, and Gauthier Gidel. Q-learners can provably collude in the iterated prisoner’s dilemma. arXiv preprint arXiv:2312.08484 , 2023

    2023

  41. [49]

    Symmetric equilibrium of multi-agent reinforcement learning in repeated prisoner’s dilemma

    Yuki Usui and Masahiko Ueda. Symmetric equilibrium of multi-agent reinforcement learning in repeated prisoner’s dilemma. Applied Mathematics and Computation , 409:126370, 2021. 2

    2021

  42. [50]

    Q-learning in two-player two-action games

    Monica Babes, Michael Wunder, and Michael Littman. Q-learning in two-player two-action games. In Proc. AAMAS, pages 1–6, 2009. 3

    2009

  43. [51]

    Melioration learning in iterated public goods games: The impact of exploratory noise

    Johannes Zschache. Melioration learning in iterated public goods games: The impact of exploratory noise. The Journal of Mathematical Sociology , 42(1):1–16, 2018. 3

    2018

  44. [52]

    Rein- forcement learning and decision making in monkeys during a competitive game

    Daeyeol Lee, Michelle L Conroy, Benjamin P McGreevy, and Dominic J Barraclough. Rein- forcement learning and decision making in monkeys during a competitive game. Cognitive brain research, 22(1):45–58, 2004. 4

    2004

  45. [53]

    Valuation of uncertain and delayed rewards in primate prefrontal cortex

    Soyoun Kim, Jaewon Hwang, Hyojung Seo, and Daeyeol Lee. Valuation of uncertain and delayed rewards in primate prefrontal cortex. Neural Networks, 22(3):294–304, 2009. 4

    2009

  46. [54]

    John G. Cross. A Stochastic Learning Model of Economic Behavior*. The Quarterly Journal of Economics, 87(2):239–266, 05 1973. ISSN 0033-5533. doi: 10.2307/1882186. URL https: //doi.org/10.2307/1882186. 4

    1973 doi

  47. [55]

    Batch Reinforcement Learning, pages 45–73

    Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch Reinforcement Learning, pages 45–73. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 978-3-642-27645-3. doi: 10.1007/978-3-642-27645-3 2. URL https://doi.org/10.1007/978-3-642-27645-3_2 . 5

    2012 doi

  48. [56]

    Quantal response equilibria for normal form games

    Richard D McKelvey and Thomas R Palfrey. Quantal response equilibria for normal form games. Games and economic behavior , 10(1):6–38, 1995. 6

    1995

  49. [57]

    High-stakes failures of backward induction

    Bouke Klein Teeselink, Dennie van Dolder, Martijn J van den Assem, and Jason D Dana. High-stakes failures of backward induction. Games and Economic Behavior , 2024. 6

    2024

  50. [58]

    Timing of transients: quantifying reaching times and transient behavior in complex systems

    Tim Kittel, Jobst Heitzig, Kevin Webster, and J¨ urgen Kurths. Timing of transients: quantifying reaching times and transient behavior in complex systems. New Journal of Physics , 19(8): 083005, 2017. 13 19

    2017

  51. [59]

    Wolfram Barfuss and Janusz M. Meylahn. Intrinsic fluctuations of reinforcement learning promote cooperation. Scientific Reports, 13(1), January 2023. ISSN 2045-2322. doi: 10. 1038/s41598-023-27672-7. URL https://www.nature.com/articles/s41598-023-27672-7 . Number: 1 Publisher:...

    2023

  52. [2003]

    ISBN 1581136838

    Association for Computing Machinery. ISBN 1581136838. doi: 10.1145/860575.860687. URL https://doi.org/10.1145/860575.860687. 2, 3, 4

  53. [2009]

    URL https://link.aps.org/doi/10.1103/ PhysRevLett.103.198702

    doi: 10.1103/PhysRevLett.103.198702. URL https://link.aps.org/doi/10.1103/ PhysRevLett.103.198702. 2

    doi

  54. [2789]

    URL http://dx.doi.org/10.1016/j.physd.2005

    doi: 10.1016/j.physd.2005.06.031. URL http://dx.doi.org/10.1016/j.physd.2005. 06.031

    2005 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.