Pith. sign in

REVIEW 4 major objections 5 minor 71 references

Dilution, Diffusion and Symbiosis in Spatial Prisoner's Dilemma with Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding a persist action lets cooperation survive in a spatial prisoner's dilemma.

desk verdict A genuinely new persist-action mutualism observation in Q-learning spatial PD, but the convergence evidence is self-contradictory and the central claim needs error bars and payoff measurements before it can be taken as established. read the letter →

arxiv 2507.02211 v2 pith:ZR4BB4IX submitted 2025-07-03 cs.AI cs.NEphysics.comp-ph

Gustavo C. Mangold , Heitor C. M. Fernandes , Mendeli H. Vainstein This is my paper
classification cs.AIcs.NEphysics.comp-ph
keywords spatialprisoner'sdilemmareinforcementlearningindependentmulti-agentQ-learningcooperationdilutionmobilitypersistactionsymbiosis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that adding a "persist" action (do nothing, stay put) to independent multi-agent Q-learning in a diluted, mobile spatial prisoner's dilemma creates a mutualistic symbiosis between persist agents and copy-the-best agents. In this symbiosis, persist agents cluster around copy-the-best players and shield them from mobile defectors, while the persist agents gain higher payoffs by joining cooperative clusters. As a result, cooperation endures in parameter regions—low mobility and densities just below the percolation threshold—where the same algorithm without persist yields total defection. The paper also argues that games with fixed update rules can be qualitatively reproduced by learned update rules, positioning the reinforcement-learning framework as a benchmark for classical spatial game results.

What carries the argument

The central machinery is independent multi-agent Q-learning, where each agent maintains a Q-table over states {C,D} and an action set, updated by the rule Q(s,a) <- (1-alpha)Q(s,a) + alpha(pi + gamma max Q(s',a')). The load-bearing action is persist (P), defined as doing neither strategy change nor movement; its complement, copy-the-best (B), copies the strategy of the neighbour with the highest payoff; move (M) randomly diffuses to a vacant neighbouring site with a given mobility rate. The paper uses the percolation threshold of the square lattice (rho ≈ 0.593) as the density around which cooperation peaks appear, and analyses action-state correlations to show that persist and copy-the-best agents together form cooperative clusters that resist invasion by mobile defectors.

What would settle it

Run the {persist, copy-the-best, move} simulations with the averaging window doubled, or with epsilon-greedy exploration switched off after a warm-up period, and check whether the persist-copy cooperative clusters and the cooperation fraction remain unchanged; if the cooperation drifts or the clusters dissolve, the claimed steady-state symbiosis is an artifact of finite-time learning rather than a stable outcome.

Watch

Extended reading notes

Core claim

The paper's central claim is that introducing the action set {persist, copy-the-best, move} into an otherwise standard independent Q-learning spatial prisoner's dilemma produces a stable cooperative phase driven by mutualism between two subpopulations. Agents that choose persist neither change their strategy nor move, while agents that choose copy-the-best adopt the strategy of the highest-payoff neighbor; persistent agents surround the copy-the-best players, reducing their exposure to defectors, and in return receive higher payoffs from belonging to cooperative clusters. This symbiotic arrangement sustains cooperation in the low-mobility, sub-percolation region where the copy-the-best/move action set alone leads to complete defection, and it manifests as aligned clusters in state space and action space, as shown in snapshots and correlation plots.

Load-bearing premise

The results are averaged over the last N/10 Monte Carlo steps under the assumption that the independent multi-agent Q-learning dynamics have converged to a steady state, even though the paper itself notes that non-stationarity invalidates single-agent convergence guarantees in multi-agent settings.

Editorial extensions

If this is right

  • Cooperation can appear in parameter regions (low mobility, densities just below percolation) where the same algorithm without the persist action produces total defection.
  • Learned update rules can reproduce the qualitative behaviour of fixed update rules, as shown by the copy-the-best action matching known mobility results, so the reinforcement-learning framework can act as a benchmark for classical spatial game dynamics.
  • In the no-knowledge case with only cooperate/defect actions, cooperation does not vanish at high temptation b because agents learn that universal defection yields zero reward, leading to a mixed cooperator-defector steady state.
  • Because the action set is arbitrary, the framework can be extended to other movement styles or directional actions, making it a flexible tool for exploring how action choices shape emergent social behaviour.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The persist-barrier effect is probably not unique to Q-learning: any mechanism that reduces a cooperator's exposure to defectors while rewarding the protecting agents could yield the same mutualism, so the result should transfer to fixed-rule spatial games if persist is treated as a strategy.
  • The symbiosis may depend on exploration rate, learning rate, or finite simulation time; because multi-agent Q-learning lacks convergence guarantees, the persistence of the cluster pattern over much longer runs or with exploration switched off should be tested.
  • Since the paper notes the action set is arbitrary, adding persist to other social dilemmas (for example, the public goods game or snowdrift game) is a natural next test of whether the barrier effect generalises beyond the prisoner's dilemma.
  • The correlation between state space and action space suggests that action choices themselves can act as observable markers of hidden cooperative structure, which could be exploited to detect or encourage cooperation in multi-agent systems.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies the spatial prisoner's dilemma with independent multi-agent Q-learning, introducing dilution (lattice vacancies) and diffusive mobility. It defines several action sets: static {C,D}, mobile {C,D,M}, copy-the-best {B,M}, and {B,P,M} with a persist action. The central claims are (i) that adding the persist action to copy-the-best agents produces a mutualistic symbiosis that sustains cooperation in low-mobility, near-percolation regimes where it would otherwise vanish (Sec. III B 2, Figs. 6-7), and (ii) that games with fixed update rules can be qualitatively equivalent to games with learned update rules (Abstract, Sec. III B 1). The paper also reproduces known percolation-related cooperation effects as a benchmark and reports non-vanishing cooperation under high temptation in the no-knowledge case.

Significance. If the symbiosis result is robust, it would be a genuinely new observation in multi-agent reinforcement learning applied to evolutionary games, with potential relevance to understanding how learned and fixed update rules relate. The paper is methodologically transparent: it uses a standard, interpretable Q-learning algorithm, states parameters explicitly, and the code is publicly available (footnote 45). The benchmark comparison with fixed-rule spatial games is a useful contribution. However, the central claim is currently supported only by averaged simulation curves without error bars or convergence diagnostics, and the proposed mechanism for the symbiosis is inferred from spatial correlations rather than measured payoffs. These gaps make the main conclusion provisional.

major comments (4)
  1. [Section II, paragraph on epsilon-greedy; Conclusions] The convergence argument for the multi-agent algorithm is invalid as stated. The text in Section II says that epsilon-greedy exploration guarantees sufficient visits to all state-action pairs, 'which is a condition for convergence of the single agents version of algorithm. This, in turn, also determines the convergence of our independent multi-agent algorithm.' This is a non sequitur because each agent's environment is non-stationary: other agents are learning and moving concurrently, so the Markov property is lost. The Conclusions explicitly concede that non-stationarity 'invalidates the convergence guarantees typically associated with single-agent reinforcement learning.' Since the paper averages the last N/10 Monte Carlo steps and calls the result a steady state without any convergence diagnostics (no time-series plots of fC over N, no block-averaging check, no Q-value stability measure), the reported 'steady state' may be a finite-time transient controlled by alpha, epsilon, and run length rather than an emergent equilibrium. I request convergence diagnostics or a clearly stated empirical criterion for stationarity before the symbiosis claim can be accepted.
  2. [Figures 3 and 6 (Sec. III B)] The central comparison between the aB and aB-P action sets is presented through heat maps and curves without any error bars or confidence intervals, despite 20 independent samples. This matters for the load-bearing claim: Fig. 6 shows cooperation in regions where Fig. 3 shows zero cooperation, but the color scale and absence of uncertainty make it impossible to assess whether the difference is statistically significant or within run-to-run variability. The paper should report means and standard deviations (or standard errors) for representative parameter points, especially around the percolation threshold and low mobility values where the symbiosis is claimed.
  3. [Sec. III B 2, Fig. 7] The proposed mechanism for mutualistic symbiosis—that persist (P) agents act as a barrier around copy-the-best (B) agents, reducing their exposure to defectors, and in return P agents receive higher payoffs—is inferred from snapshots and Pearson correlations between the state and action spaces. The claimed payoff benefit to P agents is never measured. Without quantitative evidence, such as the mean payoff of P agents inside versus outside cooperative clusters, or an ablation in which P agents are removed or forced to choose B, the interpretation as symbiosis remains an appealing but unsupported narrative. I recommend adding a direct payoff comparison or an intervention test.
  4. [Sec. III B (parameter choices)] The temptation value is fixed at b = 1.4 'as a transition in cooperation clusters with similar rulesets appears around this limit [9],' but no sweep over b is reported for the aB and aB-P cases. The key claim that persist sustains cooperation where it otherwise disappears may be strongly b-dependent; a single value is insufficient to establish robustness. Additionally, the learning parameters differ between the no-knowledge section (epsilon = 0.02, N = 2e4, 10 runs) and the action-exploration section (epsilon = 0.15, N = 1e5, 20 runs), so quantitative comparisons of cooperation levels across these sections (e.g., Figs. 1-2 versus Fig. 6) are confounded by the parameter changes.
minor comments (5)
  1. [Sec. III A 1 (Fig. 1 discussion)] The text says that at low densities 'players are more likely to be isolated and lack neighbors, leading them to choose their actions randomly – essentially flipping a coin at each round.' But the exploration probability is epsilon = 0.02 in that section, so random action selection occurs only 2% of the time. The explanation for fC near 0.5 at low density needs clarification; perhaps the intended mechanism is that isolated players receive zero payoff and their Q-values remain tied, making the argmax choice arbitrary.
  2. [Sec. II, Eq. (4)] The update rule uses max Q(s', a') for the future state s', but for the movement action M the next state is determined by a probabilistic diffusion attempt and for the persist action P the agent does not change state. The text should define s' explicitly for each action, or state that s' is the player's own state after the action is executed.
  3. [Sec. II, 'off-policy' vs 'offline'] The sentence 'in the reinforcement learning setting, asynchronous updates would be classified as off-policy [39] updates when considering the simulation as a whole and an offline [40, 41] update, when taking each round as a separate training step' seems to conflate off-policy and offline learning. This is a minor terminology issue, but it should be corrected for precision, particularly because the paper elsewhere describes its updates as on-policy (Sec. III B 1).
  4. [Fig. 6 caption] The caption says 'we see the same cooperative region shifted to the left,' but it is not immediately clear what it is being compared to. State explicitly that the comparison is with Fig. 3a, or describe the shift direction in terms of density.
  5. [Sec. IV, Conclusions] The sentence 'In fixed update rules settings such as using choose the best or the Fermi rule, when players do not learn, cooperation quickly goes to zero [9, 47]' is too sweeping; for example, spatial structure can sustain cooperation under certain parameters even with fixed rules. Qualify the statement to avoid overgeneralization.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the persist–copy-the-best symbiosis emerges from simulation rather than being encoded in the setup or fitted to the outcome.

full rationale

The paper's derivation chain is a Monte Carlo simulation of a standard independent Q-learning rule (Eq. 4) with a fixed payoff matrix (Eq. 1). No parameter is fitted to the reported cooperation fractions, and no result is presented as a prediction that was already an input to the model. The persist action is defined operationally in Sec. II (the action set a_{B-P} = {B, P, M}), and the cooperative phase in Figs. 6–7 is an emergent numerical observation; the proposed barrier mechanism for B and P agents is an interpretive explanation offered after the simulation, not a constraint placed into the update rule. Prior self-citations ([15], [20], [53], [55]) are used as benchmarks, qualitative comparisons, or examples of previously observed symbiosis, but none functions as a load-bearing premise that forces the new mutualism claim. The paper's own convergence caveat in the Conclusions (“the non-stationary nature of the environment, which invalidates the convergence guarantees typically associated with single-agent reinforcement learning”) is a genuine reliability risk against the Sec. II assertion that sufficient visits determine convergence of the multi-agent algorithm, and it could make the reported steady state a finite-time artifact. However, that is a correctness and verification concern, not a circular reduction: the averaged last N/10 Monte Carlo steps are not defined so as to equal any fitted target, and no equation-level identity between inputs and outputs is present. The claimed qualitative equivalence between fixed and learned update rules is a benchmark comparison with earlier literature, not a renaming of prior results. Therefore no significant circularity is found.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No physical entities are introduced. The model freedom consists of hand-set hyperparameters and simulation details, and the central results are conditional on these choices and on the assumed convergence of independent Q-learning.

free parameters (4)
  • b (temptation to defect) = 1.4
    Fixed at 1.4 for all action-space runs because a transition appears around this limit (Sec. III B); central symbiosis and equivalence claims are only evaluated at this single value.
  • epsilon (exploration probability) = 0.15 for action-space runs, 0.02 for no-knowledge runs
    Chosen for convergence; different exploration rates across sections complicate comparisons and may affect emergent cooperation.
  • alpha (learning rate) = 0.75
    Chosen for convergence purposes; not swept across values.
  • gamma (discount factor) = 0.8
    Chosen for convergence purposes; not swept across values.
assumptions (4)
  • domain assumption Epsilon-greedy exploration guarantees sufficient visits to all state-action pairs and therefore convergence of the independent multi-agent Q-learning process.
    Invoked in Sec. II to justify convergence; the paper later acknowledges in the Conclusions that non-stationarity invalidates single-agent convergence guarantees, so this is a strong assumption.
  • domain assumption Averaging the last N/10 steps over 10 to 50 independent runs yields a steady state representative of the learned dynamics.
    Assumed throughout Sec. III; no stationarity or convergence diagnostic is shown.
  • domain assumption The weak prisoner's dilemma payoffs (R=1, P=0, S=0, T=b with b in (1,2)) preserve the phenomenon of interest.
    Model setup in Sec. II; restricts all results to the weak prisoner's dilemma.
  • domain assumption The population-policy equivalence lets a player switching actions be interpreted as an individual switching population.
    Introduced in Sec. I and used in Sec. III B 2 to label the observed mutualism as a between-population effect.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dilution, Diffusion and Symbiosis in Spatial Prisoner's Dilemma with Reinforcement Learning." pith.science (2026). https://pith.science/paper/ZR4BB4IX

@misc{pith2026250702211,
  author       = {Pith},
  title        = {Pith review of: Dilution, Diffusion and Symbiosis in Spatial Prisoner's Dilemma with Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZR4BB4IX}},
  note         = {Machine review of arXiv:2507.02211}
}
read the original abstract

Recent studies in the spatial prisoner's dilemma games with reinforcement learning have shown that static agents can learn to cooperate through a diverse sort of mechanisms, including noise injection, different types of learning algorithms and neighbours' payoff knowledge. In this work, using an independent multi-agent Q-learning algorithm, we study the effects of dilution and mobility in the spatial version of the prisoner's dilemma. Within this setting, different possible actions for the algorithm are defined, connecting with previous results on the classical, non-reinforcement learning spatial prisoner's dilemma, showcasing the versatility of the algorithm in modeling different game-theoretical scenarios and the benchmarking potential of this approach. As a result, a range of effects is observed, including evidence that games with fixed update rules can be qualitatively equivalent to those with learned ones, as well as the emergence of a symbiotic mutualistic effect between populations that forms when multiple actions are defined.

Figures

Figures reproduced from arXiv: 2507.02211 by the authors.

Figure 2
Figure 2. Cooperation as a function of mobility and occupation density for the no-knowledge case with set aM for b = 1.4. (a) Heat map as a function of the density and the mobility rate, where the color bar shows the fraction of cooperators, which lies only in a small regime. (b) Curves for specific densities, showcasing the weak dependence of cooperation on mobility for this set of actions. After considering the static case,… view at source ↗
Figure 3
Figure 3. Fraction of cooperators as a function of the density of occupation and mobility with choosing the best player set aB, showing that there is an extensive region of null cooperation for almost all parameters, with a cooperation peak for low mobility around the percolation threshold. (a) Heat map, where the color bar represents the fraction of cooperators and the pd axis is in logarithmic scale. (b) Cooperation curves … view at source ↗
Figure 4
Figure 4. Snapshots with a specific striped initial configuration, taken at the percolation limit ρ ≈ 0.593. Time increases to the right with different scales, with the last snapshot in the upper row at step 110 and in the bottom row at step 105 representing, respectively, pd = 1 and pd = 0.01, showing that fast agents tend toward defection quickly, while slow agents can gather in clusters and resist invasion from defectors. … view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Typical snapshots showing both the state space s variables and the action space aB variables, together with the correlation between the state space and action space shown in the first row. The showcased snapshots’ sample is in bold color in the correlation plot, while …
Figure 6
Figure 6. Figure 6: Cooperation (a) Heat map of the fraction of cooperators for different levels of mobility, varying together with the density of the lattice, where we see the same cooperative region shifted to the left. The axis pd is in logarithmic scale. (b) Curves for specific values…
Figure 7
Figure 7. Figure 7: Typical snapshots from the simulation with the action set aPB, where the middle row shows the state space, with blue being C, red being D and black being holes. The lower row shows holes in black as well, while the agents’ actions space is represented in the color purp…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

71 extracted references · 64 canonical work pages

  1. [9]

    (8) This results in a dynamic environment similar to the one described in the previous section, as evidenced by com- paring the curves in Fig

    Persist and copy-the-best Finally, we use an action set that involves three differ- ent actions with our last Q-table: QPB = QCP QCB QCM QDP QDB QDM . (8) This results in a dynamic environment similar to the one described in the previous section, as evidenced by com- paring the curves in Fig. 6a with those in Fig. 3a. As with the action setaB, there is a ...

    work page 2023

  2. [1]

    We initialize anL × L square lattice by populating it partially, with a densityρ of players occupying it randomly

    work page

  3. [2]

    Each player is randomly assigned its first role, or state, as cooperator or defector and their Q-table is initialized to zeros

    work page

  4. [3]

    A player is sampled at random, choosing its action either randomly with probabilityϵ or according to the maximum value on the Q-table, with probabil- ity 1 − ϵ, to obtain a payoff

    work page

  5. [4]

    (4), and the state of the player is updated based on the action a

    The Q-table is then updated according to Eq. (4), and the state of the player is updated based on the action a

    work page

  6. [5]

    A Monte Carlo Step (MCS) consists of L2 repe- titions of items 3 and 4 to complete a learning episode. The game is iterated through a maximum ofN = 105 steps on a100 × 100 lattice to complete an asynchronous, single-agent update, Monte Carlo simulation; simulations are evolved for at least 2 × 104 MCS until the system reaches a steady state. We also carry...

    work page

  7. [6]

    (5) In Fig

    Static agents With no mobility, that is, in the diluted setting with an exact number of holes, we use the first set of actions aS as described in Section II, defining thus the Q-table: Qs = QCC QCD QDC QDD . (5) In Fig. 1, we observe a monotonic increase in cooperation levels as the density ρ decreases. At low densities, the fraction of cooperatorsfC beco...

    work page

  8. [7]

    2: Cooperation as a function of mobility and occupation density for the no-knowledge case with set aM for b = 1.4

    Diffusing agents (a) (b) Fig. 2: Cooperation as a function of mobility and occupation density for the no-knowledge case with set aM for b = 1.4. (a) Heat map as a function of the density and the mobility rate, where the color bar shows the fraction of cooperators, which lies only in a small regime. (b) Curves for specific densities, showcasing the weak de...

    work page

Show all 71 references
  1. [8]

    This is achieved through the ability to choose an action that reveals the identity of the best-performing player nearby

    Copy-the-best We begin by simulating agents using the action set aB = {B, M}, which explicitly provides them with in- formation about their surroundings. This is achieved through the ability to choose an action that reveals the identity of the best-performing player nearby. Th...

  2. [10]

    T. D. Seeley, Honeybee democracy, Princeton University Press, 2011

    2011

  3. [11]

    Michener, The Social Behavior of the Bees: A Com- parative Study, Vol

    C. Michener, The Social Behavior of the Bees: A Com- parative Study, Vol. 404, Harvard University Press (Belk- nap Press), 1974

    1974

  4. [12]

    K. v. Frisch, The dance language and orientation of bees, Harvard University Press, 1993

    1993

  5. [13]

    Hosokawa, Y

    T. Hosokawa, Y. Ishii, N. Nikoh, M. Fujie, N. Satoh, T. Fukatsu, Obligate bacterial mutualists evolving from environmental bacteria in natural insect populations, Na- ture Microbiology 1 (1) (2016) 1–7

    2016

  6. [14]

    J. L. Bronstein, R. Alarcón, M. Geber, The evolution of plant–insect mutualisms, New Phytologist 172 (3) (2006) 412–428

    2006

  7. [15]

    D. P. Vázquez, N. Blüthgen, L. Cagnolo, N. P. Chacoff, Uniting pattern and process in plant–animal mutualis- tic networks: a review, Annals of Botany 103 (9) (2009) 1445–1457

    2009

  8. [16]

    A. D. Vaudo, L. A. Dyer, A. S. Leonard, Pollen nutrition structures bee and plant community interactions, Pro- 10 ceedings of the National Academy of Sciences 121 (3) (2024) e2317228120

    2024

  9. [17]

    Rapoport, A

    A. Rapoport, A. M. Chammah, Prisoner’s dilemma: A study in conflict and cooperation, Vol. 165, University of Michigan press, 1965

    1965

  10. [18]

    M. A. Nowak, R. M. May, Evolutionary games and spa- tial chaos, nature 359 (6398) (1992) 826–829

    1992

  11. [19]

    Axelrod, W

    R. Axelrod, W. D. Hamilton, The evolution of coopera- tion, Science 211 (4489) (1981) 1390–1396

    1981

  12. [20]

    Vincent, Evolutionary game theory, natural selection, and darwinian dynamics, Cambridge University Press, 2005

    T. Vincent, Evolutionary game theory, natural selection, and darwinian dynamics, Cambridge University Press, 2005

    2005

  13. [21]

    L. S. Flores, M. A. Amaral, M. H. Vainstein, H. C. Fer- nandes, Cooperation in regular lattices, Chaos, Solitons & Fractals 164 (2022) 112744

    2022

  14. [22]

    Takesue, Effects of updating rules on the coevolving prisoner’s dilemma, Physica A: Statistical Mechanics and its Applications 513 (2019) 399–408

    H. Takesue, Effects of updating rules on the coevolving prisoner’s dilemma, Physica A: Statistical Mechanics and its Applications 513 (2019) 399–408

    2019

  15. [23]

    Kaiping, G

    G. Kaiping, G. Jacobs, S. Cox, T. Sluckin, Nonequiva- lence of updating rules in evolutionary games under high mutation rates, Physical Review E 90 (4) (2014) 042726

    2014

  16. [24]

    M. H. Vainstein, J. J. Arenzon, Disordered environments inspatialgames, PhysicalReviewE64(5)(2001)051905

    2001

  17. [25]

    J. M. Pacheco, F. L. Pinheiro, F. C. Santos, Popula- tion structure induces a symmetry breaking favoring the emergence of cooperation, PLoS computational biology 5 (12) (2009) e1000596

    2009

  18. [26]

    McAvoy, C

    A. McAvoy, C. Hauert, Structural symmetry in evolu- tionary games, Journal of The Royal Society Interface 12 (111) (2015) 20150420

    2015

  19. [27]

    J. Du, Z. Wu, Asymmetric game promotes the emergence of cooperation in dynamic networks, Knowledge-Based Systems 284 (2024) 111287

    2024

  20. [28]

    L. S. Flores, M. H. Vainstein, H. C. Fernandes, M. A. Amaral, Heterogeneous contributions can jeopardize co- operation in the public goods game, Physical Review E 108 (2) (2023) 024111

    2023

  21. [29]

    M. H. Vainstein, A. T. Silva, J. J. Arenzon, Does mobil- ity decrease cooperation?, Journal of theoretical biology 244 (4) (2007) 722–728

    2007

  22. [30]

    R. S. Sutton, A. G. Barto, Reinforcement learning: An introduction, MIT press, 2018

    2018

  23. [31]

    T. W. Sandholm, R. H. Crites, Multiagent reinforcement learning in the iterated prisoner’s dilemma, Biosystems 37 (1) (1996) 147–166

    1996

  24. [32]

    R.Axelrod, Theemergenceofcooperationamongegoists, American political science review 75 (2) (1981) 306–318

    1981

  25. [33]

    C. J. Watkins, P. Dayan, Q-learning, Machine learning 8 (1992) 279–292

    1992

  26. [34]

    Zhang, J.-Q

    S.-P. Zhang, J.-Q. Zhang, Z.-G. Huang, B.-H. Guo, Z.-X. Wu, J. Wang, Collective behavior of artificial intelligence population: transition from optimization to game, Non- linear Dynamics 95 (2019) 1627–1637

    2019

  27. [35]

    Zhang, J.-Q

    S.-P. Zhang, J.-Q. Zhang, L. Chen, X.-D. Liu, Oscillatory evolution of collective behavior in evolutionary games played with reinforcement learning, Nonlinear Dynamics 99 (4) (2020) 3301–3312

    2020

  28. [36]

    L. Wang, D. Jia, L. Zhang, P. Zhu, M. Perc, L. Shi, Z. Wang, Lévy noise promotes cooperation in the pris- oner’s dilemma game with reinforcement learning, Non- linear Dynamics 108 (2) (2022) 1837–1845

    2022

  29. [37]

    C. Zhao, G. Zheng, C. Zhang, J. Zhang, L. Chen, Emergence of cooperation under punishment: A reinforcement learning perspective, Chaos: An In- terdisciplinary Journal of Nonlinear Science 34 (7) (2024) 073123. arXiv:https://pubs.aip.org/aip/ cha/article-pdf/doi/10.1063/5.02157...

    2024 doi

  30. [38]

    Z. Yan, L. Li, J. Shang, H. Zhao, Periodic update rule with q-learning promotes evolution of cooperation in gametransitionwithpunishmentmechanism, Neurocom- puting 609 (2024) 128510

    2024

  31. [39]

    L. Wang, L. Fan, L. Zhang, R. Zou, Z. Wang, Synergis- tic effects of adaptive reward and reinforcement learn- ing rules on cooperation, New Journal of Physics 25 (7) (2023) 073008

    2023

  32. [40]

    Zhang, T

    H. Zhang, T. An, P. Yan, K. Hu, J. An, L. Shi, J. Zhao, J. Wang, Exploring cooperative evolution with tunable payoff’s loners using reinforcement learning, Chaos, Soli- tons & Fractals 178 (2024) 114358

    2024

  33. [41]

    Meloni, A

    S. Meloni, A. Buscarino, L. Fortuna, M. Frasca, J. Gómez-Gardeñes, V. Latora, Y. Moreno, Effects of mobility in a population of prisoner’s dilemma players, Physical Review E 79 (6) (2009) 067101

    2009

  34. [42]

    Tan, Multi-agent reinforcement learning: Indepen- dent vs

    M. Tan, Multi-agent reinforcement learning: Indepen- dent vs. cooperative agents, in: Proceedings of the tenth international conference on machine learning, 1993, pp. 330–337

    1993

  35. [43]

    Busoniu, R

    L. Busoniu, R. Babuska, B. De Schutter, A comprehen- sive survey of multiagent reinforcement learning, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 38 (2) (2008) 156–172

    2008

  36. [44]

    Wei, Y.-T

    C.-Y. Wei, Y.-T. Hong, C.-J. Lu, Online reinforcement learning in stochastic games, Advances in Neural Infor- mation Processing Systems 30 (2017)

    2017

  37. [45]

    H. Guo, Z. Wang, Z. Song, Y. Yuan, X. Deng, X. Li, Ef- fect of state transition triggered by reinforcement learn- ing in evolutionary prisoner’s dilemma game, Neurocom- puting 511 (2022) 187–197

    2022

  38. [46]

    L.Matignon, G.J.Laurent, N.LeFort-Piat, Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems, The Knowledge Engineering Review 27 (1) (2012) 1–31

    2012

  39. [47]

    Bettini, R

    M. Bettini, R. Kortvelesy, A. Prorok, Neural diversity is key to collective artificial learning, arXiv preprint arXiv:2412.16244 (2024)

    2024 arXiv

  40. [48]

    Uehara, C

    M. Uehara, C. Shi, N. Kallus, A review of off-policy evaluation in reinforcement learning, arXiv preprint arXiv:2212.06355 (2022)

    2022 arXiv

  41. [49]

    Levine, A

    S. Levine, A. Kumar, G. Tucker, J. Fu, Offline rein- forcement learning: Tutorial, review, and perspectives on open problems, arXiv preprint arXiv:2005.01643 (2020)

    2020 arXiv

  42. [50]

    R. F. Prudencio, M. R. Maximo, E. L. Colombini, A sur- vey on offline reinforcement learning: Taxonomy, review, and open problems, IEEE Transactions on Neural Net- works and Learning Systems (2023)

    2023

  43. [51]

    K. Soma, Y. Bouteiller, H. Hamann, G. Beltrame, Bridg- ing swarm intelligence and reinforcement learning, arXiv preprint arXiv:2410.17517 (2024)

    2024 arXiv

  44. [52]

    Bloembergen, K

    D. Bloembergen, K. Tuyls, D. Hennes, M. Kaisers, Evo- lutionary dynamics of multi-agent learning: A survey, Journal of Artificial Intelligence Research 53 (2015) 659– 697

    2015

  45. [53]

    Toffoli, N

    T. Toffoli, N. Margolus, Cellular automata machines: a new environment for modeling, MIT press, 1987

    1987

  46. [54]

    All code is available on github.com/gustavomangold/

  47. [55]

    Z. Yang, L. Zheng, M. Perc, Y. Li, Interaction state q- learning promotes cooperation in the spatial prisoner’s 11 dilemma game, Applied Mathematics and Computation 463 (2024) 128364

    2024

  48. [56]

    M. Perc, A. Szolnoki, Social diversity and promotion of cooperation in the spatial prisoner’s dilemma game, PhysicalReviewE—Statistical, Nonlinear, andSoftMat- ter Physics 77 (1) (2008) 011904

    2008

  49. [57]

    Newman, R

    M. Newman, R. M. Ziff, Efficient monte carlo algorithm and high-precision results for percolation, Physical Re- view Letters 85 (19) (2000) 4104

    2000

  50. [58]

    Benesty, J

    J. Benesty, J. Chen, Y. Huang, I. Cohen, Pearson Cor- relation Coefficient, Springer Berlin Heidelberg, Berlin, Heidelberg, 2009, pp. 1–4

    2009

  51. [59]

    Li, R.-R

    M. Li, R.-R. Liu, L. Lü, M.-B. Hu, S. Xu, Y.-C. Zhang, Percolation on complex networks: Theory and applica- tion, Physics Reports 907 (2021) 1–68

    2021

  52. [60]

    Z. Wang, A. Szolnoki, M. Perc, If players are sparse social dilemmas are too: Importance of percolation for evolu- tion of cooperation, Scientific reports 2 (1) (2012) 369

    2012

  53. [61]

    Z. Wang, A. Szolnoki, M. c. v. Perc, Percolation thresh- old determines the optimal population density for public cooperation, Phys. Rev. E 85 (2012) 037101

    2012

  54. [62]

    F. R. Leivas, H. C. M. Fernandes, M. H. Vainstein, Anomalous behavior of replicator dynamics for the pris- oner’s dilemma on diluted lattices, Phys. Rev. E 111 (2025) 024123. doi:10.1103/PhysRevE.111.024123. URL https://link.aps.org/doi/10.1103/PhysRevE. 111.024123

    2025 doi

  55. [63]

    J. L. Bronstein, Mutualism, Oxford University Press, 2015

    2015

  56. [64]

    L. S. Flores, H. C. Fernandes, M. A. Amaral, M. H. Vain- stein, Symbiotic behaviour in the public goods game with altruistic punishment, Journal of Theoretical Biology 524 (2021) 110737

    2021

  57. [65]

    G. M. Viswanathan, V. Afanasyev, S. V. Buldyrev, S. Havlin, M. G. da Luz, E. P. Raposo, H. E. Stanley, Lévy flights in random searches, Physica A: Statistical Mechanics and its Applications 282 (1-2) (2000) 1–12

    2000

  58. [66]

    Canese, G

    L. Canese, G. C. Cardarilli, L. Di Nunzio, R. Fazzolari, D. Giardino, M. Re, S. Spanò, Multi-agent reinforcement learning: A review of challenges and applications, Ap- plied Sciences 11 (11) (2021) 4948

    2021

  59. [67]

    Papoudakis, F

    G. Papoudakis, F. Christianos, A. Rahman, S. V. Al- brecht, Dealing with non-stationarity in multi-agent deep reinforcement learning, arXiv preprint arXiv:1906.04737 (2019)

    2019 arXiv

  60. [68]

    Lanctot, V

    M. Lanctot, V. Zambaldi, A. Gruslys, A. Lazaridou, K. Tuyls, J. Pérolat, D. Silver, T. Graepel, A uni- fiedgame-theoreticapproachtomultiagentreinforcement learning, AdvancesinNeuralInformationProcessingSys- tems 30 (2017)

    2017

  61. [69]

    Rajeswaran, I

    A. Rajeswaran, I. Mordatch, V. Kumar, A game theo- retic framework for model based reinforcement learning, in: International conference on machine learning, PMLR, 2020, pp. 7953–7963

    2020

  62. [70]

    Y. Yang, J. Wang, An overview of multi-agent reinforce- ment learning from game theoretical perspective, arXiv preprint arXiv:2011.00583 (2020)

    2020 arXiv

  63. [71]

    De La Fuente, G

    N. De La Fuente, G. Casadellà, et al., Game the- ory and multi-agent reinforcement learning: From Nash equilibria to evolutionary dynamics, arXiv preprint arXiv:2412.20523 (2024)

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.