Pith. sign in

REVIEW 3 major objections 5 minor 101 references

Equilibrium stability as a driver of cooperation among Q-learners

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Cooperation can dominate the long-run behavior of Q-learning agents even when exploration never fades, and the paper derives the payoff threshold at which it does.

desk verdict A sharp, honestly-labeled heuristic boundary for time-averaged cooperation in Q-learning; solid simulations, but the stability argument ignores fluctuation variance and the stationarity conjecture is unproven. read the letter →

arxiv 2607.13607 v1 pith:DEWWOZWL submitted 2026-07-15 cs.MA econ.GNecon.THq-fin.EC

Janusz M. Meylahn , Maximilian Schäfer This is my paper · ORCID
classification cs.MAecon.GNecon.THq-fin.EC MSC 68T0560J2091A05
keywords algorithmiccollusionQ-learningiteratedPrisoner'sDilemmaconstantexplorationoccupationtimeequilibriumstabilitywin-staylose-shiftepsilon-greedy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper studies two epsilon-greedy Q-learning agents playing the iterated Prisoner's Dilemma with one-period memory, keeping both exploration and learning rates constant. In this regime the agents never settle permanently into one strategy, so the paper shifts the question from convergence to time-averaged behavior: what fraction of time do the agents spend cooperating? The authors claim that cooperative play, centered on the win-stay, lose-shift equilibrium, can dominate in this time-averaged sense, and they derive a closed-form boundary on the discount factor that separates cooperative from defection-dominated regimes. The boundary is obtained by comparing the stability of the All-Defect and WSLS equilibria, measured by the smallest gap between Q-values in each. Large-scale simulations show that this boundary predicts the observed transition away from defection-dominated behavior with high classification accuracy.

What carries the argument

The machinery is a stability comparison between the All-Defect and Win-Stay, Lose-Shift equilibria of the expected Q-learning dynamics. For each equilibrium, the minimum over the four states of the optimal Q-value difference ΔQ(s) = Q(s,argmax_a Q(s,a)) - Q(s,argmin_a Q(s,a)) is computed from the Bellman optimality equation; that minimum represents the height of the smallest Q-value barrier. Setting the two minima equal and solving for δ gives the critical discount factor δ_c, which is the paper's predictor. The derivation is heuristic—it deliberately ignores differences in fluctuation sizes across equilibria—but the resulting boundary is the object that organizes the simulations and the cla

What would settle it

Simulate epsilon-greedy Q-learning in the same repeated Prisoner's Dilemma and sweep the discount factor across δ_c while measuring the average occupation time of cooperative profiles; if the empirical transition occurs at a markedly different δ, or fails to shift in the predicted direction when ε changes, the boundary is falsified. A complementary check is to measure the Q-value distributions around the AD and WSLS equilibria: if the difference in their fluctuation widths changes which equilibrium is more likely to be exited, the barrier-only heuristic will misclassify.

Watch

Extended reading notes

Core claim

The central claim is that the relative stability of two Nash equilibria of the memory-one repeated Prisoner's Dilemma—All-Defect (AD) and Win-Stay, Lose-Shift (WSLS)—controls which equilibrium the persistent fluctuations of constant-exploration Q-learning favor. Stability is measured by the smallest optimal Q-value gap min_s ΔQ(s), interpreted as the barrier random exploration must overcome to reverse a player's action preference in some state. Equating this barrier for WSLS and AD and solving for the discount factor yields δ_c = 2(T+P-(R+S)) / ((1-ε)[2(R-P)+ε(P+S-(R+T))]): for δ > δ_c, WSLS is the more stable equilibrium, so the long-run stationary distribution should place more occupation

Load-bearing premise

The central prediction collapses if an equilibrium's stability is not captured by its smallest Q-value gap, or if the occupation-time distribution does not converge to a unique stationary law; the paper leaves both as assumptions, and the fluctuation-size part is explicitly conceded not to hold literally.

Editorial extensions

If this is right

  • Cooperation can be sustained among reinforcement-learning pricing algorithms without vanishing exploration; the relevant measure of collusive harm becomes the long-run fraction of time spent in cooperative or supra-competitive play.
  • The boundary δ_c depends on payoffs and exploration rate but not on the learning rate, so within the accessible stationary regime the prediction should transfer across choices of α.
  • The boundary is best interpreted as locating the transition away from defection-dominated behavior rather than a sharp prediction of full mutual cooperation, since asymmetric states occupy a non-negligible share of time near the boundary.
  • The time-averaged perspective implies that social-welfare effects of algorithmic collusion depend on how long supra-competitive prices are sustained by fluctuating learning dynamics, not only on whether such prices can be learned.
  • The prediction presupposes convergence to a unique stationary occupation-time distribution; in practice the theory applies only where the simulation horizon is long enough for stationarity to be reached.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the barrier-only stability heuristic is correct, the same Q-value-gap comparison could be applied to any pair of equilibria in other Q-learning games with a finite state space, yielding a general time-averaged dominance predictor.
  • A direct test beyond the paper's grid would be to measure the stationary fluctuation width of Q-values around AD and WSLS; if those widths differ enough to reverse the ordering of exit probabilities, the boundary should misclassify in a detectable way.
  • Because the boundary itself is independent of α but the accessible stationary regime depends strongly on α, a possible extension is a finite-horizon mixing-time correction that extends the prediction to non-stationary parameter regions.
  • For regulators, the result suggests that auditing algorithms for collusion should include the fraction of time spent pricing supra-competitively, since deployed algorithms with constant exploration may never converge to a fixed policy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies two Q-learners playing the iterated prisoner's dilemma with one-period memory, constant exploration epsilon, and constant learning rate alpha. Because exploration never vanishes, the authors propose measuring cooperation by the long-run fraction of time spent in cooperative strategy profiles rather than by convergence to a cooperative equilibrium. They prove a positive-probability recurrence result for a set of best-response profiles (Theorem 4.1), conjecture uniqueness of a stationary distribution (Conjecture 4.2), and then derive a heuristic boundary delta_c (Eq. 4.9) by equating the minimal Q-value gaps of the All-Defect (AD) and Win-Stay-Lose-Shift (WSLS) equilibria. The paper evaluates this boundary on a large simulation grid, reporting macro F1 scores of 0.85-0.95 in the empirically stationary region, with additional state-space robustness checks and perturbation analyses showing local optimality of the boundary.

Significance. If the theoretical derivation were sound, the paper would offer a valuable new perspective on algorithmic collusion: instead of asking whether Q-learners converge to collusion, it asks how resilient collusive strategies are to persistent exploration noise. The manuscript has clear strengths: the boundary is derived analytically from Bellman equations rather than fitted, the simulation campaign is extensive (136,000 trajectories), and the authors include careful robustness checks (state-space outcomes, adversarial precision/recall, boundary perturbations). The distinction between convergence-based and time-averaged notions of collusion is well motivated and relevant to the algorithmic collusion debate. However, the theoretical core is heuristic and rests on a conjecture, and the derivation of the central boundary contains an internal inconsistency that must be resolved before the paper's main claim can be accepted as stated.

major comments (3)
  1. [§4, Eq. (4.7)-(4.9) and Appendix B] The quantity called Delta Q_WSLS in the derivation is not the max-min difference defined in Eq. (4.7). The formulas in Appendix B compute, up to the common factor 1/(1-delta), the signed difference Q(s, pi_WSLS(s)) - Q(s, other action), which coincides with Eq. (4.7) only when WSLS is a Nash equilibrium. The paper itself notes (footnote 7) that WSLS is an equilibrium only in part of the parameter space, yet the boundary is evaluated over the full (R,P) grid. In non-equilibrium regions the signed gap is negative, so equating it to the positive AD gap is not a meaningful stability condition. For example, with T=1, R=0.6, P=0.5, S=0, epsilon=0.2, delta=0.7, Eq. (B.10) gives -0.416 while the actual max-min gap from (B.3)-(B.4) is positive. The theoretical derivation should be restricted to the WSLS-NE region, or the heuristic should be reformulated using the actual max-min gaps, and the simu
  2. [§4, after Conjecture 4.2, and footnote 6] The stability criterion equates equilibrium stability with the smallest Q-value gap and explicitly ignores the size of fluctuations inside each equilibrium. Footnote 6 concedes that Q-value fluctuation sizes differ across equilibria because they depend on rewards and continuation payoffs. In a stochastic Q-learning process, the relevant exit rate is approximately exp(-Delta Q / sigma), so equal stability requires Delta Q_AD / sigma_AD = Delta Q_WSLS / sigma_WSLS, not Delta Q_AD = Delta Q_WSLS. The simulation F1 scores do not discriminate between these two conditions because sigma differences may be small or correlated with payoffs on the tested grid. The authors should either derive a variance-corrected boundary, or provide empirical estimates of sigma_AD and sigma_WSLS showing that the omitted term is negligible in the region where the boundary is claimed to hold.
  3. [§4, Conjecture 4.2 and §5.2] The central theoretical framing relies on the existence of a unique stationary distribution of the hidden Markov process, but Conjecture 4.2 is unproven. The simulation-based stationarity check (Delta O < 0.05 across ten initialisations) is a finite-horizon heuristic and does not establish uniqueness of an invariant measure. Moreover, the accessible parameter range is defined by this same heuristic and then used to select the configurations for the F1 evaluation, so the reported scores are conditional on the conjecture. Since Appendix E shows that performance degrades outside this range, the paper should either provide a proof or at least a substantially stronger argument for Conjecture 4.2, or explicitly re-frame the contribution as a heuristic boundary with empirical validation in the finite-horizon stationary regime.
minor comments (5)
  1. [§4, Theorem 4.1] The set in Eq. (4.4) is written as {tilde pi^i = BR(pi) for some pi in Pi_epsilon}; this should specify that BR is the best response to the opponent's strategy, not to an arbitrary strategy in Pi_epsilon. As written it is ambiguous.
  2. [§5.2 and Figure 3] The text states that the discount-factor grid is Theta_delta = {0.65, 0.70, ..., 0.85}, but Figure 3 shows panels for delta = 0.55 and Figure 4 also uses delta = 0.55. The grid definition should be corrected.
  3. [§5.3, Eq. (5.4) and Table 1] The normalisation in Eq. (5.4) divides by the total occupation of the five focal profiles, but Table 1 shows this total is only 0.77-0.90 depending on the hyperparameters. The interpretation of 'dominance' should explicitly note that the classifier is relative to the focal set, not to the full strategy-profile space.
  4. [Abstract and §4] The phrase 'derive a boundary' overstates the status of the result, since the derivation is a heuristic resting on an unproven conjecture. I suggest 'propose and empirically validate a heuristic boundary'.
  5. [Figure 6] The perturbation plot along the delta dimension uses the x-axis 'delta - delta_c' with truncation to [-1,1]. Since delta_c can be outside [0,1], the truncation and the interpretation of the resulting scores should be explained.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the boundary δ_c is derived analytically from Bellman Q-values, and the F1 evaluation is an independent simulation check rather than a fit.

full rationale

The central boundary (Eq. 4.9) is obtained by solving the Bellman optimality equations in Appendix B for Q_AD and Q_WSLS, defining per-state gaps (Eq. 4.7), and equating the minimal gaps (Eq. 4.8). This is a parameter-free analytic derivation, not a fitted curve: no coefficient in δ_c is estimated from the simulation outcomes, and the boundary is independent of the learning rate α. The subsequent macro-F1 evaluation is an out-of-sample check against simulated occupation times, with several complementary outcome labels (strategy-profile and state-based variants) and a local-optimality/perturbation analysis; the boundary is not chosen to maximize F1. Footnotes and text explicitly acknowledge the heuristic nature of equating stability with barrier height (footnote 6: fluctuations in Q-value estimates also depend on rewards and continuation payoffs) and Conjecture 4.2 is left unproven; these are correctness/robustness limitations, not circular reductions. Self-citations (Meylahn and Janssen 2022; Meylahn 2025; Barfuss and Meylahn 2023) are used for equilibrium classification and motivation, but the paper recomputes the Q-values and the boundary from the model equations, so the central claim does not reduce to those citations. The only data-informed choices are the restriction to the accessible parameter range and the inclusion of AC and AGT as focal profiles based on preliminary simulations; these influence the evaluation metric but do not enter the derivation of δ_c. Hence the claimed prediction is not equivalent by construction to its inputs.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The boundary derivation is self-contained given the equilibrium Q-value formulas, but it imports the equilibrium set and best-response structure from earlier work and relies on unproved or explicitly heuristic premises: unique stationary distribution and barrier-only stability.

assumptions (6)
  • domain assumption Payoff ordering T > R > P > S, discounted rewards with δ∈(0,1), and epsilon-greedy Q-learning with constant α and ε, memory-one state space.
    Section 3 defines the entire model; the boundary and simulations apply only within this setting.
  • domain assumption Best responses are unique for almost all parameter values.
    Section 3, just before Eq. (3.8): uniqueness is needed to define the IBR graph and the Nash equilibria used in the boundary.
  • ad hoc to paper Equilibrium stability is determined by the smallest Q-value gap (barrier height) alone; fluctuation sizes are ignored.
    Section 4: the boundary solves min_s ΔQ_WSLS = min_s ΔQ_AD; footnote 6 admits Q-value fluctuations differ across equilibria.
  • ad hoc to paper For the boundary derivation, stationary occupation mass is assumed to concentrate on Nash equilibria.
    Section 4, footnote 5 explicitly states: 'This premise is not satisfied in our case, but the argument serves as a basis for deriving our heuristic boundary.'
  • ad hoc to paper Conjecture 4.2: the occupation-time distribution converges to a unique stationary distribution.
    Section 4 states this as a conjecture; the strong Feller condition required by one proof route is explicitly left unresolved.
  • standard math Beck-Srikant constant-step-size Q-learning error bounds are valid in the proof of Theorem 4.1.
    Appendix A, Eq. (A.3), uses the Beck-Srikant bound to show Q-values can get close to best-response values with positive probability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Equilibrium stability as a driver of cooperation among Q-learners." pith.science (2026). https://pith.science/paper/DEWWOZWL

@misc{pith2026260713607,
  author       = {Pith},
  title        = {Pith review of: Equilibrium stability as a driver of cooperation among Q-learners},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DEWWOZWL}},
  note         = {Machine review of arXiv:2607.13607}
}
read the original abstract

Algorithmic collusion among pricing algorithms has raised concerns about sustained supra-competitive prices and their implications for social welfare. Existing work has largely focused on the probability that reinforcement-learning algorithms converge to cooperative strategies, typically under the assumption that exploration vanishes over time. Motivated by the observation that algorithms deployed in practice are likely to continue exploring in order to remain adaptive to changing environments, we study learning dynamics under constant exploration. In this setting, the relevant question is no longer whether an algorithm converges to a particular strategy profile, but rather what fraction of time the algorithms spend playing cooperative strategies. Even in the benchmark case of the repeated Prisoner's Dilemma with one-period memory, this yields high-dimensional stochastic learning dynamics, for which a complete analytic treatment is intractable. We show that cooperative strategies can be dominant in this time-averaged sense and derive a boundary predicting when such dominance arises, based on the expected dynamics of the Q-learning process. Extensive simulations show that this boundary is a strong predictor for non-defection-dominated behaviour under epsilon-greedy Q-learning.

Figures

Figures reproduced from arXiv: 2607.13607 by the authors.

Figure 1
Figure 1. Critical condition for the stability of WSLS being greater than that of AD. On [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Examples of occupation-time trajectories for ten initialisations under three pa [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Heatmaps of ∆O(θ). Each panel shows values of ∆O(θ) across ten initialisations for all combinations of α and ϵ at a fixed value of δ. Regions with ∆O(θ) < 0.05 are classified as stationary at time T = 2 × 108 . 14 [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Heatmaps of average empirical occupation times for [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Macro F1 scores across algorithmic hyperparameters. For each value of a given [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Macro F1 scores under perturbations of the boundary in Equation (4.9) along [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Asymmetric state occupations. The left panel plots the average total occupation [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Macro F1 scores across algorithmic hyperparameters. For each value of a given [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Heatmaps of the average empirical occupation time of [PITH_FULL_IMAGE:figures/full_fig_p028_9.png]
Figure 10
Figure 10. Figure 10: Heatmaps of the average empirical occupation time of [PITH_FULL_IMAGE:figures/full_fig_p029_10.png]
Figure 11
Figure 11. Figure 11: Heatmaps of the average empirical occupation time of [PITH_FULL_IMAGE:figures/full_fig_p030_11.png]
Figure 12
Figure 12. Figure 12: Heatmaps of the average empirical occupation time of [PITH_FULL_IMAGE:figures/full_fig_p031_12.png]
Figure 13
Figure 13. Figure 13: Heatmaps of the average empirical occupation time of [PITH_FULL_IMAGE:figures/full_fig_p032_13.png]
Figure 14
Figure 14. Figure 14: Heatmaps of the average empirical occupation time of [PITH_FULL_IMAGE:figures/full_fig_p033_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

101 extracted references · 6 canonical work pages

  1. [1]

    International Journal of Game Theory , volume=

    Best-response dynamics, playing sequences, and convergence to equilibrium in random games , author=. International Journal of Game Theory , volume=. 2023 , publisher=

    2023

  2. [2]

    Error bounds for constant step-size

    Beck, Carolyn L and Srikant, Rayadurgam , journal=. Error bounds for constant step-size. 2012 , publisher=

    2012

  3. [3]

    SIAM Journal on Control and Optimization , volume=

    Asymptotic properties of stochastic approximations with constant coefficients , author=. SIAM Journal on Control and Optimization , volume=. 1981 , publisher=

    1981

  4. [4]

    SIAM Journal on Control and Optimization , volume=

    Averaging methods for the asymptotic analysis of learning and adaptive systems, with small adjustment rate , author=. SIAM Journal on Control and Optimization , volume=. 1981 , publisher=

    1981

  5. [5]

    IEEE Transactions on Neural Networks , volume=

    Convergence of learning algorithms with constant learning rates , author=. IEEE Transactions on Neural Networks , volume=. 1991 , publisher=

    1991

  6. [6]

    Final iteration convergence bound of

    Lee, Donghwan , journal=. Final iteration convergence bound of. 2024 , publisher=

    2024

  7. [7]

    Automatica , volume=

    Finite-sample analysis of nonlinear stochastic approximation with applications in reinforcement learning , author=. Automatica , volume=. 2022 , publisher=

    2022

  8. [8]

    Finite-time error bounds for linear stochastic approximation and

    Srikant, Rayadurgam and Ying, Lei , booktitle=. Finite-time error bounds for linear stochastic approximation and. 2019 , organization=

    2019

Show all 101 references
  1. [9]

    Li, Gen and Cai, Changxiao and Chen, Yuxin and Wei, Yuting and Chi, Yuejie , journal=. Is. 2024 , publisher=

    2024

  2. [10]

    Durmus, Alain and Jim. On. International Conference on Artificial Intelligence and Statistics , pages=. 2021 , organization=

    2021

  3. [11]

    Lecture notes , volume=

    Convergence of Markov processes , author=. Lecture notes , volume=

  4. [12]

    Nicolas Kryloff and Nicolas Bogoliouboff , journal =. La Th\'

  5. [13]

    Unified finite-time error analysis of soft

    Jeong, Narim and Lee, Donghwan , journal=. Unified finite-time error analysis of soft. 2025 , publisher=

    2025

  6. [14]

    arXiv preprint arXiv:2409.01147 , year=

    On mechanism underlying algorithmic collusion , author=. arXiv preprint arXiv:2409.01147 , year=

    arXiv

  7. [15]

    arXiv preprint arXiv:2312.02644 , year=

    Algorithmic collusion under competitive design , author=. arXiv preprint arXiv:2312.02644 , year=

    arXiv

  8. [16]

    arXiv preprint arXiv:2309.10609 , year=

    Game connectivity and adaptive dynamics , author=. arXiv preprint arXiv:2309.10609 , year=

    arXiv

  9. [17]

    arXiv preprint arXiv:2407.05460 , year=

    Basins of attraction in two-player random ordinal potential games , author=. arXiv preprint arXiv:2407.05460 , year=

    arXiv

  10. [18]

    Games and Economic Behavior , volume=

    Best-response dynamics in two-person random games with correlated payoffs , author=. Games and Economic Behavior , volume=. 2024 , publisher=

    2024

  11. [19]

    Self-Play

    Bertrand, Quentin and Duque, Juan Agustin and Calvano, Emilio and Gidel, Gauthier , booktitle=. Self-Play

  12. [20]

    SIAM Journal on Control and Optimization , volume=

    Learning algorithms for Markov decision processes with average cost , author=. SIAM Journal on Control and Optimization , volume=. 2001 , publisher=

    2001

  13. [21]

    Proceedings of the ACM on Measurement and Analysis of Computing Systems , volume=

    Adaptive discretization for episodic reinforcement learning in metric spaces , author=. Proceedings of the ACM on Measurement and Analysis of Computing Systems , volume=. 2019 , publisher=

    2019

  14. [22]

    Advances in Neural Information Processing Systems , volume=

    Adaptive discretization for model-based reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  15. [23]

    Journal of Economic Theory , volume=

    Analogy-based expectation equilibrium , author=. Journal of Economic Theory , volume=. 2005 , doi=

    2005

  16. [24]

    arXiv preprint arXiv:2202.05946 , year=

    Artificial intelligence and spontaneous collusion , author=. arXiv preprint arXiv:2202.05946 , year=

    arXiv

  17. [25]

    Proceedings of the 24th ACM Conference on Economics and Computation , pages =

    Banchio, Martino and Mantegazza, Giacomo , title =. Proceedings of the 24th ACM Conference on Economics and Computation , pages =. 2023 , isbn =. doi:10.1145/3580507.3597726 , abstract =

    2023

  18. [26]

    arXiv preprint arXiv:2508.14766 , year=

    Algorithmic Collusion is Algorithm Orchestration , author=. arXiv preprint arXiv:2508.14766 , year=

  19. [27]

    Physical Review E , volume=

    Evolutionary dynamics, intrinsic noise, and cycles of cooperation , author=. Physical Review E , volume=. 2010 , publisher=

    2010

  20. [28]

    Available at SSRN 4943829 , year=

    On Mechanism Underlying Algorithmic Collusion , author=. Available at SSRN 4943829 , year=

  21. [29]

    arXiv preprint arXiv:2303.13539 , year=

    Decentralized Multi-Agent Reinforcement Learning for Continuous-Space Stochastic Games , author=. arXiv preprint arXiv:2303.13539 , year=

    arXiv

  22. [30]

    Theory and Decision , volume=

    A classification of weakly acyclic games , author=. Theory and Decision , volume=. 2015 , doi =

    2015

  23. [31]

    Decentralized

    Arslan, G. Decentralized. IEEE Transactions on Automatic Control , volume=. 2016 , doi =

    2016

  24. [32]

    Journal of Political Economy , doi =

    Assad, Stephanie and Clark, Robert and Ershov, Daniel and Xu, Lei , title =. Journal of Political Economy , doi =

  25. [33]

    , year =

    Axelrod, Robert and Hamilton, William D. , year =. The. Science , volume =. doi:10.1126/science.7466396 , url =

    doi

  26. [34]

    Scientific Reports , volume=

    Intrinsic fluctuations of reinforcement learning promote cooperation , author=. Scientific Reports , volume=. doi:10.1038/s41598-023-27672-7 , year=

    doi

  27. [35]

    Games , volume=

    Names for games: locating 2 2 games , author=. Games , volume=. 2015 , url=. doi:10.3390/g6040495 , publisher=

    2015 doi

  28. [36]

    Available at SSRN: 4403058 , doi=

    Coordinated vs Efficient Prices: The Impact of Algorithmic Pricing on Multifamily Rental Markets , author=. Available at SSRN: 4403058 , doi=

  29. [37]

    Protecting consumers from collusive prices due to

    Calvano, Emilio and Calzolari, Giacomo and Denicol. Protecting consumers from collusive prices due to. Science , volume=. 2020 , doi=

    2020

  30. [38]

    2020 , journal =

    Artificial Intelligence, Algorithmic Pricing, and Collusion , author =. 2020 , journal =. doi:10.1257/aer.20190623 , url=

    2020 doi

  31. [39]

    Algorithmic Competition: O

    OECD , url=. Algorithmic Competition: O

  32. [40]

    den Boer and Janusz M

    Arnoud V. den Boer and Janusz M. Meylahn and Maarten Pieter Schinkel , journal=. Artificial Collusion: Examining Supracompetitive Pricing by. doi:10.2139/ssrn.4213600 , year=

    doi

  33. [41]

    2022 , publisher =

    The Algorithmic Learning Equations: Evolving Strategies in Dynamic Games , journal =. 2022 , publisher =. doi:10.2139/ssrn.4175239 , url =

    2022 doi

  34. [42]

    Chaos: An Interdisciplinary Journal of Nonlinear Science , volume=

    Quantifying the likelihood of learning collusive strategy equilibria , author=. Chaos: An Interdisciplinary Journal of Nonlinear Science , volume=. 2025 , publisher=

    2025

  35. [43]

    Available at SSRN 4293831 , year=

    Algorithmic Collusion and a Folk Theorem from Learning with Bounded Rationality , author=. Available at SSRN 4293831 , year=

  36. [44]

    Physical Review E , volume=

    Deterministic limit of temporal difference reinforcement learning for stochastic games , author=. Physical Review E , volume=

  37. [45]

    American Economic Review , volume=

    Learning to coordinate: A study in retail gasoline , author=. American Economic Review , volume=. 2019 , publisher=

    2019

  38. [46]

    Algorithmic Game Theory: Third International Symposium, SAGT 2010, Athens, Greece, October 18-20, 2010

    On the structure of weakly acyclic games , author=. Algorithmic Game Theory: Third International Symposium, SAGT 2010, Athens, Greece, October 18-20, 2010. Proceedings 3 , pages=. 2010 , url=. doi:10.1007/978-3-642-16170-4_12 , organization=

    2010 doi

  39. [47]

    Algorithmic collusion in assortment games , author=

  40. [48]

    2023 , url=

    Reinforcement learning and collusion , author=. 2023 , url=

    2023

  41. [49]

    arXiv preprint arXiv:2412.15707 , year=

    Online Optimization Algorithms in Repeated Price Competition: Equilibrium Learning and Algorithmic Collusion , author=. arXiv preprint arXiv:2412.15707 , year=

  42. [50]

    , booktitle=

    Fabrikant, Alex and Papadimitriou, Christos H. , booktitle=. The complexity of game dynamics:. 2008 , doi=

    2008

  43. [51]

    Structural models:

    Harary, Frank and Norman, Robert Zane and Cartwright, Dorwin , publisher=. Structural models:

  44. [52]

    V-Learning--A Simple, Efficient, Decentralized Algorithm for Multiagent

    Jin, Chi and Liu, Qinghua and Wang, Yuanhao and Yu, Tiancheng , journal=. V-Learning--A Simple, Efficient, Decentralized Algorithm for Multiagent

  45. [53]

    Finite-sample analysis of decentralized

    Gao, Zuguang and Ma, Qianqian and Ba. Finite-sample analysis of decentralized. arXiv preprint arXiv:2112.07859 , year=

    arXiv

  46. [54]

    Journal of Statistical Mechanics: Theory and Experiment , volume=

    Cycles of cooperation and defection in imperfect learning , author=. Journal of Statistical Mechanics: Theory and Experiment , volume=. 2011 , doi=

    2011

  47. [55]

    den Boer , title =

    Thomas Loots and Arnoud V. den Boer , title =. Production and Operations Management , volume =. doi:10.1111/poms.13919 , url=

    doi

  48. [56]

    and Skopalik, Alexander , booktitle=

    Mirrokni, Vahab S. and Skopalik, Alexander , booktitle=. On the complexity of. doi:10.1145/1566374.1566376 , pages=

  49. [57]

    Proceedings of the 39th International Conference on Machine Learning , volume =

    On improving model-free algorithms for decentralized multi-agent reinforcement learning , author=. Proceedings of the 39th International Conference on Machine Learning , volume =. 2022 , publisher =

    2022

  50. [58]

    Provably efficient reinforcement learning in decentralized general-sum

    Mao, Weichao and Ba. Provably efficient reinforcement learning in decentralized general-sum. Dynamic Games and Applications , volume=. 2023 , url=. doi:10.1007/s13235-021-00420-0 , publisher=

    2023 doi

  51. [59]

    Independent and decentralized learning in

    Maheshwari, Chinmay and Wu, Manxi and Pai, Druv and Sastry, Shankar , journal=. Independent and decentralized learning in

  52. [60]

    Available at SSRN: 4594415 , year=

    Does an intermediate price facilitate algorithmic collusion? , author=. Available at SSRN: 4594415 , year=

  53. [61]

    and Janssen, Lars , journal=

    Meylahn, Janusz M. and Janssen, Lars , journal=. Limiting dynamics for. 2022 , url=. doi:10.1155/2022/4830491 , publisher=

    2022 doi

  54. [62]

    Manufacturing & Service Operations Management , volume=

    Learning to collude in a pricing duopoly , author=. Manufacturing & Service Operations Management , volume=. 2022 , publisher=. doi:10.1287/msom.2021.1074 , url =

    2022

  55. [63]

    A Strategy of

    Nowak, Martin and Sigmund, Karl , year =. A Strategy of. Nature , volume =. doi:10.1038/364056a0 , url =

    doi

  56. [64]

    2020 , publisher=

    Independent learning approaches: Overcoming multi-agent learning pathologies in team-games , author=. 2020 , publisher=

    2020

  57. [65]

    International Conference on Artificial Intelligence and Statistics , pages=

    Actor-critic fictitious play in simultaneous move multistage games , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2018 , url=

    2018

  58. [66]

    Discrete Mathematics , volume=

    An enumeration of distinct and non-isomorphic functional quasi-order relations , author=. Discrete Mathematics , volume=. doi:10.1016/j.disc.2022.113039 , year=

    2022

  59. [67]

    Decentralized

    Sayin, Muhammed and Zhang, Kaiqing and Leslie, David and Basar, Tamer and Ozdaglar, Asuman , journal=. Decentralized

  60. [68]

    Proceedings of the tenth international conference on machine learning , volume=

    A reinforcement learning method for maximizing undiscounted rewards , author=. Proceedings of the tenth international conference on machine learning , volume=

  61. [69]

    Internet and Network Economics: 8th International Workshop, WINE 2012, Liverpool, UK, December 10-12, 2012

    Choosing products in social networks , author=. Internet and Network Economics: 8th International Workshop, WINE 2012, Liverpool, UK, December 10-12, 2012. Proceedings 8 , pages=. 2012 , organization=

    2012

  62. [70]

    Economics bulletin , volume=

    The pure Nash equilibrium property and the quasi-acyclic condition , author=. Economics bulletin , volume=

  63. [71]

    A theoretical and empirical analysis of

    Van Seijen, Harm and Van Hasselt, Hado and Whiteson, Shimon and Wiering, Marco , booktitle=. A theoretical and empirical analysis of. 2009 , doi=

    2009

  64. [72]

    1998 , publisher=

    Individual strategy and social structure: An evolutionary theory of institutions , author=. 1998 , publisher=

    1998

  65. [73]

    2004 , publisher=

    Strategic learning and its limits , author=. 2004 , publisher=

    2004

  66. [74]

    Econometrica: Journal of the Econometric Society , pages=

    The evolution of conventions , author=. Econometrica: Journal of the Econometric Society , pages=. 1993 , url=. doi:10.2307/2951778 , publisher=

    1993 doi

  67. [75]

    Modelling the dynamics of multiagent

    Hu, Shuyue and Leung, Chin-wing and Leung, Ho-fung , journal=. Modelling the dynamics of multiagent

  68. [76]

    SIAM Journal on Mathematics of Data Science , volume=

    Satisficing paths and independent multiagent reinforcement learning in stochastic games , author=. SIAM Journal on Mathematics of Data Science , volume=. 2023 , doi =

    2023

  69. [77]

    Available at SSRN 4891033 , year=

    Algorithmic Collusion: Where Are We and Where Should We Be Going? , author=. Available at SSRN 4891033 , year=

  70. [78]

    Games and Economic Behavior , volume=

    Reinforcement learning in a prisoner's dilemma , author=. Games and Economic Behavior , volume=. 2024 , doi=

    2024

  71. [79]

    European Journal of Operational Research , year=

    Collusion by mistake: Does algorithmic sophistication drive supra-competitive profits? , author=. European Journal of Operational Research , year=. doi:https://doi.org/10.1016/j.ejor.2024.06.006 , url=

    2024 doi

  72. [80]

    Available at SSRN 4498926 , year=

    Less than meets the eye: simultaneous experiments as a source of algorithmic seeming collusion , author=. Available at SSRN 4498926 , year=

  73. [81]

    , author=

    Best-Response Mechanisms. , author=. ICS , pages=. 2011 , organization=

    2011

  74. [82]

    Applied Mathematics and Computation , volume=

    Symmetric equilibrium of multi-agent reinforcement learning in repeated prisoner's dilemma , author=. Applied Mathematics and Computation , volume=. 2021 , doi=

    2021

  75. [83]

    Economics Letters , volume=

    On algorithmic collusion and reward--punishment schemes , author=. Economics Letters , volume=. 2024 , publisher=

    2024

  76. [84]

    The RAND Journal of Economics , volume=

    Autonomous algorithmic collusion: Q-learning under sequential pricing , author=. The RAND Journal of Economics , volume=. 2021 , publisher=

    2021

  77. [85]

    AEA Papers and Proceedings , volume=

    Artificial intelligence, algorithm design, and pricing , author=. AEA Papers and Proceedings , volume=. 2022 , organization=

    2022

  78. [86]

    Journal of Economics & Management Strategy , volume=

    The impact of artificial intelligence design on pricing , author=. Journal of Economics & Management Strategy , volume=. 2024 , publisher=

    2024

  79. [87]

    American Economic Review , volume=

    Artificial intelligence, algorithmic pricing, and collusion , author=. American Economic Review , volume=. 2020 , publisher=

    2020

  80. [88]

    2012 , publisher=

    Markov chains and stochastic stability , author=. 2012 , publisher=

    2012

  81. [89]

    Devraj, Adithya M and Meyn, Sean , journal=. Zap

  82. [90]

    and Wentzell, Alexander D

    Freidlin, Mark I. and Wentzell, Alexander D. Random Perturbations of Dynamical Systems. 2012. doi:10.1007/978-3-642-25847-3_1

    2012 doi

  83. [91]

    Marketing Science , volume=

    Frontiers: Algorithmic collusion: Supra-competitive prices via independent algorithms , author=. Marketing Science , volume=. 2021 , publisher=

    2021

  84. [92]

    Management Science , volume=

    The effect of outsourcing pricing algorithms on market competition , author=. Management Science , volume=. 2022 , publisher=

    2022

  85. [93]

    Journal of Economic Dynamics and Control , volume=

    Q-learning agents in a Cournot oligopoly model , author=. Journal of Economic Dynamics and Control , volume=. 2008 , publisher=

    2008

  86. [94]

    International journal of industrial organization , volume=

    Algorithmic collusion with imperfect monitoring , author=. International journal of industrial organization , volume=. 2021 , publisher=

    2021

  87. [95]

    Available at SSRN 5012923 , year=

    A (mathematical) definition of algorithmic collusion , author=. Available at SSRN 5012923 , year=

  88. [96]

    Journal of Political Economy , volume=

    Algorithmic pricing and competition: Empirical evidence from the German retail gasoline market , author=. Journal of Political Economy , volume=. 2024 , publisher=

    2024

  89. [97]

    American Economic Journal: Microeconomics , volume=

    Competition in pricing algorithms , author=. American Economic Journal: Microeconomics , volume=. 2023 , publisher=

    2023

  90. [98]

    Dynamic pricing algorithms, consumer harm, and regulatory response , author=. Wash. UL Rev. , volume=. 2022 , publisher=

    2022

  91. [99]

    Journal of Competition Law & Economics , volume=

    Developing competition law for collusion by autonomous artificial agents , author=. Journal of Competition Law & Economics , volume=. 2018 , publisher=

    2018

  92. [100]

    Proceedings of the 2024 Symposium on Computer Science and Law , pages=

    Regulation of algorithmic collusion , author=. Proceedings of the 2024 Symposium on Computer Science and Law , pages=

    2024

  93. [101]

    Management Science , volume=

    Artificial intelligence: Can seemingly collusive outcomes be avoided? , author=. Management Science , volume=. 2023 , publisher=

    2023

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.