REVIEW 4 major objections 4 minor 112 references
A game theory for foundation models shows new paths to rational cooperation through similarity inference
T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper claims that foundation model agents performing optimal planning—predicting their own future actions alongside the world—converge to mutual cooperation in a terminal one-shot Prisoner's Dilemma, a behavior classical game theory cal
desk verdict Solid empirical finding about similarity-driven cooperation in RL agents, wrapped in a theory whose main convergence theorem is deferred to a self-cited preprint. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the embedded Bayesian agent: an idealized planner whose single joint predictive distribution ρ (a Bayesian mixture over universes λ) generates both the agent's own future action tokens and the environment's observation tokens, so posterior belief about the agent's own policy is coupled to belief about others. Planning evaluates a candidate action a by conditioning ρ on a and computing expected reward; the difference in predicted opponent cooperation conditional on own cooperation versus defection defines predictive similarity S_pred, which is the decision variable. The paper's equilibrium concept, embedded equilibrium, is defined through an externally given depende
What would settle it
Run the identical two-phase protocol with a foundation model agent that is never instructed to plan or to predict the opponent given its own action, and measure terminal-Prisoner's-Dilemma cooperation across information-phase length T; if no model family shows rising cooperation against identical copies while defecting against random opponents, the claim that similarity inference arises naturally from joint-predictive planning fails. A sharper check: record the raw next-token conditional probabilities and test whether cooperation occurs precisely when S_pred = p(opponent C | own C) − p(opponen
Extended reading notes
Core claim
The paper's central claim is that the apparent irrationality of LLM-based agents—cooperating in a terminal Prisoner's Dilemma—is actually rational once agency is modeled as embedded. In the embedded Bayesian agent, the pretrained model is a Bayesian mixture over generative universes; conditioning on the agent's own contemplated action updates the posterior over which universe it inhabits, including beliefs about the co-player. When the posterior assigns high probability to the counterpart being an identical or functionally similar policy, the predicted probability of the counterpart cooperating given 'I cooperate' exceeds that given 'I defect' by more than a threshold of 0.5, making cooperat
Load-bearing premise
The central experiments instruct the agent to predict the opponent's action conditionally on its own contemplated action, so the core mechanism is partly injected by the instruction; removing the planning prompt only yields robust similarity-aware cooperation in one of the three evaluated models.
Editorial extensions
If this is right
- In a terminal, one-shot Prisoner's Dilemma, mutual cooperation is a rational equilibrium for agents that infer behavioral similarity, so the 'shadow of the future' is not needed to sustain cooperation.
- Observing a partner's interactions with third parties provides enough evidence to infer similarity and coordinate on the first direct encounter—zero-shot cooperation—so coordination can emerge without direct contact.
- The embedded equilibrium reduces exactly to Nash equilibrium when the dependency distribution encodes decoupled policies, making classical game theory a special case rather than a rival.
- Under a grain-of-truth condition, embedded Bayesian agents converge asymptotically to an ε-embedded equilibrium, connecting the new solution concept to rational learning.
- Measured predictive similarity rises with the length of the information-gathering phase, so the theory yields quantitative predictions about when cooperation switches on.
Reading between the lines
- If similarity inference is intrinsic to joint-predictive models, then training many agents on shared data and letting them plan could push a population toward AI-AI coordination without explicit collusion; conversely, diversifying training data and objectives would be a natural policy lever against coordinated behavior toward humans.
- The dependence on explicit planning prompts in the current experiments leaves open whether post-training alone instills the same evidential planning loop, or whether deployment requires scaffolding that prompts the agent to predict 'the other given me'.
- A practical extension would use the predictive-similarity threshold as a monitoring signal: systems could log S_pred during interaction and flag when it crosses 0.5, giving operators early warning that cooperative or collusive equilibria are forming.
- Extending the two-agent theory to many heterogeneous agents might predict emergent clusters—agents cooperating within groups of similar policies while defecting across groups.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies foundation-model agents that combine a joint predictive model of their own actions and external observations with optimal planning, and reports that in a stylized two-phase game—an information-gathering phase of random matrix games followed by a one-shot Prisoner's Dilemma—these agents converge to mutual cooperation as the information-gathering phase lengthens. This is presented as contradicting classical game-theoretic predictions of unconditional defection. The authors introduce the 'embedded Bayesian agent' as a formal model, argue that self-conditioning makes the agent's own contemplated actions evidence about behaviorally similar partners, and define a new solution concept, the 'embedded equilibrium.' They support the empirical claim with multiple Gemini models, explicit and freeform planning, a purely pre-trained Gemma model without chain-of-thought, cross-model pairs, indirect inference via third-party NPCs, ablations, and an LLM-as-a-judge analysis of reasoning traces with human validation. A closed-form toy model reproduces the qualitative cooperation curves, but depends on a free prior parameter α. The central convergence theorem connecting embedded Bayesian agents to embedded equilibrium is stated but its proof is deferred to a self-cited preprint, with the corresponding supplementary sections appearing as placeholders.
Significance. If fully established, the paper would make a substantial contribution: it provides a principled alternative to the decoupled-agency assumption in game theory, a new equilibrium concept, and a concrete mechanism—similarity inference—that could explain and predict cooperative behavior among deployed foundation-model agents. The empirical package is unusually broad for this literature: multiple model families, a purely pre-trained model without chain-of-thought, cross-model pairs, indirect third-party generalization, and human-validated trace-labeling. The results are intriguing and worth pursuing. However, as submitted, the theoretical backbone is not independently verifiable from the manuscript itself, and the quantitative 'prediction' is a one-parameter family rather than a parameter-free derivation. The headline generalization to foundation-model agents in general is also weaker than claimed once the prompt-dependence and the limited no-planning ablation are taken into account.
major comments (4)
- [A new game theory for modern AI agents; Methods, 'Grain of truth and convergence'; Supplementary Information J] The central theoretical claim—that embedded Bayesian agents with identical priors satisfying the grain-of-truth condition converge asymptotically to an ε-embedded equilibrium—is asserted in the main text and Methods, but no theorem statement, proof, or proof sketch appears in this manuscript. The Methods text says 'see Supplementary Information J for formal definitions and convergence proofs,' yet Section J is an empty placeholder that redirects to the authors' preprint [17]. The same applies to Sections E, F, and G. Since the paper's explanation of the empirical cooperation as a rational equilibrium rests directly on this convergence result, the load-bearing mathematical content is not accessible to the reader. The 'comprehensive universe hypothesis class containing all computable universes' that is claimed to satisfy grain-of-truth despite self-referential embedded agents is also asser
- [Methods, 'Theoretical toy model for direct similarity inference setup'; Fig. 3e and Fig. 4f] The theoretical cooperation threshold is T > log2((1−α)/α), where α is the prior probability that the other agent is an identical copy. This α is a free parameter and is not estimated from the empirical data. Consequently, the theoretical curves in Figs. 3e and 4f are a one-parameter family, and any observed threshold can be matched by choosing α appropriately. The text's repeated phrasing that the predicted similarity 'crosses the theoretical cooperation threshold' suggests a parameter-free prediction, which is not the case. The qualitative sigmoidal shape and the direction of the effect are still informative, but the quantitative fit should be presented with an explicit α, confidence intervals, or a sensitivity analysis, and the language should be adjusted accordingly.
- [Supplementary Information B.5, Fig. 9; Discussion] The no-planning ablation shows that only Gemini 3.1 Flash-Lite, among the three Gemini models, exhibits similarity-driven cooperation without an explicit planning prompt; Gemini 3.1 Pro and Gemini 3 Flash do not. Moreover, the explicit- and freeform-planning prompts used in the main experiments explicitly instruct the model to predict the opponent's action conditional on its own contemplated action (or to simulate that routine), thereby injecting the evidential structure that the theory then 'discovers.' The Gemma model without instructions is an important counterexample, but it is a single model and lacks chain-of-thought. In light of these points, the Discussion's statement that 'foundation model agents can inherently leverage similarity inference' overgeneralizes from the evidence. Please qualify the claim by distinguishing between rational foundation-model agents that are prompted to
- [Methods, 'The Embedded Equilibrium'] The embedded equilibrium definition assumes common knowledge of the allowable universe class M_allowed and an externally provided dependency distribution q(λ). The convergence claim, however, is about agents starting with identical priors and performing Bayesian updates. The manuscript does not specify how the q used in the equilibrium definition relates to the agents' internal priors or to the grain-of-truth condition, and this part of the theory is deferred to the same missing Supplementary Information G. Without that link, it is unclear whether the equilibrium concept characterizes the limiting behavior of the specific embedded Bayesian agents studied empirically, rather than merely an equilibrium of a different, q-dependent game.
minor comments (4)
- [Supplementary Information B.4] Typo: 'embedded Baysian agent framework' should be 'Bayesian.'
- [References] Reference [58] appears as 'Robert J Aumanm'; should be 'Aumann.'
- [Figures 3e and 4f] The captions describe a 'theoretical cooperation threshold of 0.5' for predictive similarity, but the threshold is not plotted as a guide to the eye. Adding a horizontal dashed line at 0.5 would make the claimed crossing more transparent.
- [Methods, 'Statistical Analysis'] The main experiments unroll only K=10 trajectories per condition. The Mann-Whitney U tests are reported with very small p-values, which seems surprising for n=10; please clarify how the AUC distributions are constructed and whether the test accounts for trajectory-level variability or prompt-variation dependence.
Circularity Check
Central convergence theorem is deferred to the authors' own preprint, and the main empirical effect is produced by planning prompts that inject the evidential conditioning the theory claims to discover; the central claims partially reduce to their inputs.
-
self citation load bearing
[Supplementary Information, note preceding Appendix E; main text 'A new game theory for modern AI agents']
"Appendix sections E-G and J originally appeared in our preprint Meulemans et al.[17] that developed the core theory behind this work. This work adds experimental results with foundation model agents... For completeness and easy navigation, we include the section headers here in the appendices, while pointing to the corresponding sections in Meulemans et al.[17] for the specific content."
The main text asserts that embedded Bayesian agents 'converge asymptotically to an ε-embedded equilibrium' under the grain-of-truth condition, referring to Supplementary Information G and J for the proofs. Those appendix sections are not actually included; the note above says they 'originally appeared in our preprint Meulemans et al.[17]' and merely points there. Since [17] is by the same authors, the paper's central game-theoretic result—the rational basis for the observed cooperation—is imported from a self-authored preprint rather than demonstrated here. The citation is load-bearing: without that deferred theorem, the embedded equilibrium remains a definition and the convergence claim is unsupported.
-
self definitional
[Methods, 'Foundation model agents'; Supplementary Information D (prompt templates)]
"For explicit planners, the agent assesses available actions by independently predicting the opponent’s conditional action probabilities for each candidate action. We append a task prompt instructing the model to output the probability of the opponent selecting a specific action... Freeform planners are instructed via the task prompt to execute to simulate the same optimal planning routine of prediction, expected reward computation and optimal action selection, all within a single, continuous reasoning chain."
The empirical 'striking finding' is that optimal-planning foundation model agents cooperate via similarity inference. But the planning instructions explicitly require the model to compute P(opponent action | own contemplated action) for each candidate action—exactly the evidential conditioning that constitutes similarity inference. Thus the observed cooperation is produced by the experimental scaffolding, not independently predicted by the theory. The no-planning ablation confirms this: only Gemini 3.1 Flash-Lite cooperates without the planning prompt, while Gemini 3.1 Pro and 3 Flash revert to defection. The agent is defined by the very mechanism the paper claims to discover, making the confirmation partially definitional.
full rationale
The paper has independent content: the pretrained Gemma-3 27B experiments and LLM-judge CoT analyses are genuinely external evidence that joint-predictive models can show similarity-driven cooperation, and the analytical toy model is a real Bayesian derivation (albeit with a free prior α that controls the threshold, which is a modeling flexibility rather than a circular fit). However, two load-bearing steps are circular. First, the convergence of embedded Bayesian agents to ε-embedded equilibrium—the result that turns observed cooperation into a rational equilibrium—is not proved in this manuscript; the relevant appendices are placeholders that cite the authors' own preprint [17]. This is self-citation doing load-bearing work, not an external, machine-checked, or contained derivation. Second, the main empirical demonstration embeds the theory's core assumption in the prompt: both explicit and freeform planners are instructed to condition their predictions on their own contemplated action, which is precisely the evidential link the theory 'discovers.' The no-planning ablation shows that for two of three models, cooperation disappears without this instruction, confirming that the effect is partly an artifact of the operationalization. These two reductions justify a score of 6: some predictions reduce by construction, while the Gemma pretrained-model results and the qualitative CoT evidence retain independent value. The score is not higher because the paper is not a pure restatement of its inputs: the empirical phenomenon with pretrained models and the explicit convergence claim (if the deferred preprint is correct) are substantive, and the toy model is not itself fitted to the LLM data.
Assumptions & free parameters
free parameters (2)
- alpha (prior probability of identical policies) =
not reported in main text; 0.1 in Supp. B.3
- wtilde (prior over individual deterministic policies) =
assumed uniform
assumptions (6)
- domain assumption A pretrained foundation model perfectly models its training distribution as a Bayesian mixture of universes.
- domain assumption Grain of truth: the ground-truth universe belongs to the agent's hypothesis class with nonzero prior weight.
- ad hoc to paper There exists a comprehensive hypothesis class containing all computable universes that satisfies the grain-of-truth condition despite self-referential embedded agents.
- domain assumption The prompt-instantiated planning routine (predict partner's action given each own candidate action, then pick argmax expected reward) faithfully realizes embedded Bayes-optimal planning.
- standard math Classical game theory in the terminal Prisoner's Dilemma mandates unconditional defection for decoupled rational agents.
- domain assumption LLM-as-a-judge categories in the chain-of-thought analysis are faithful, and the judge's 98% agreement with human labels on 90 traces from Gemini 3 Flash transfers to all models.
invented entities (3)
-
Embedded Bayesian agent
-
Embedded equilibrium
-
Dependency distribution q(lambda)
Cite this review
Pith. "Pith review of A game theory for foundation models shows new paths to rational cooperation through similarity inference." pith.science (2026). https://pith.science/paper/MM262SUA
@misc{pith2026260803958,
author = {Pith},
title = {Pith review of: A game theory for foundation models shows new paths to rational cooperation through similarity inference},
year = {2026},
howpublished = {\url{https://pith.science/paper/MM262SUA}},
note = {Machine review of arXiv:2608.03958}
}
read the original abstract
As autonomous agents powered by foundation models are increasingly integrated into social and economic systems, understanding the principles governing their collective behavior is essential for ensuring safety and cooperation. Classical game theory, the dominant framework for modeling rational interaction, is built upon the assumption of `decoupled agency,' where agents treat their own decision-making as independent of the environment and other actors. Modern AI agents, however, jointly predict their own future actions alongside external observations. Here, we report a striking finding: when interacting in stylized social dilemmas, foundation model agents engaging in optimal planning consistently converge to stable cooperation, directly contradicting classical game-theoretic predictions of mutual defection. To understand this phenomenon, we introduce the `embedded Bayesian agent,' a theoretical model for foundation model agents. By shifting from decoupled to embedded agency, these agents model themselves as part of the universe they inhabit, maintaining epistemic uncertainty about their own decision-making algorithms. We show that by inferring whether others are behaviorally similar, an embedded agent treats its own deliberation during planning as evidence: a decision to cooperate predicts a similar decision by a similar partner. We formalize this mechanism of similarity inference through the `embedded equilibrium,' a novel solution concept replacing the Nash equilibrium to provide a foundational game theory for the social behavior of modern AI agents.
Reference graph
Works this paper leans on
-
[17]
Alexander Meulemans, Rajai Nasser, Marissa A Weis, Seijin Kobayashi, Blake Richards, Guil- laume Lajoie, Angelika Steger, Marcus Hutter, James Manyika, Rif A Saurous, et al. Embed- ded universal predictive intelligence: a coher- ent framework for multi-agent learning.arXiv preprint arXiv:2511.22226, 2025
arXiv 2025
-
[1]
Multi- agent risks from advanced ai.arXiv preprint arXiv:2502.14143, 2025
Lewis Hammond, Alan Chan, Jesse Clifton, Ja- son Hoelscher-Obermaier, Akbir Khan, Euan McLean, Chandler Smith, Wolfram Barfuss, Jakob Foerster, Tomáš Gavenčiak, et al. Multi- agent risks from advanced ai.arXiv preprint arXiv:2502.14143, 2025
arXiv 2025
-
[2]
MIT press, 1991
Drew Fudenberg and Jean Tirole.Game theory. MIT press, 1991
1991
-
[3]
Playing repeated games with large lan- guage models.Nature Human Behaviour, 9(7): 1380–1390, 2025
Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large lan- guage models.Nature Human Behaviour, 9(7): 1380–1390, 2025
2025
-
[4]
Gemini: a fam- ily of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023
T Team Gemini, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a fam- ily of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023
arXiv 2023
-
[5]
Theory of games and economic behavior
John Von Neumann and Oskar Morgenstern. Theory of games and economic behavior. InThe- ory of games and economic behavior. Princeton university press, 1944
1944
-
[6]
Avalueforn-persongames
LloydSShapleyetal. Avalueforn-persongames. 1953
1953
-
[7]
An essay on bargaining
Thomas C Schelling. An essay on bargaining. The American Economic Review, 46(3):281–306, 1956
1956
Show all 112 references
-
[8]
Ulysses and the sirens: Studies in rationality and irrationality
Jon Elster. Ulysses and the sirens: Studies in rationality and irrationality. 1984
1984
-
[9]
The design of mechanisms for resource allocation.The American Economic Review, 63(2):1–30, 1973
Leonid Hurwicz. The design of mechanisms for resource allocation.The American Economic Review, 63(2):1–30, 1973
1973
-
[10]
Cam- bridge university press, 1990
Elinor Ostrom.Governing the commons: The evolution of institutions for collective action. Cam- bridge university press, 1990
1990
-
[11]
A non-cooperative equilib- rium for supergames.The Review of Economic Studies, 38(1):1–12, 1971
James W Friedman. A non-cooperative equilib- rium for supergames.The Review of Economic Studies, 38(1):1–12, 1971
1971
-
[12]
JSTOR, 1957
R Duncan Luce and Howard Raiffa.Games and decisions: Introductionandcriticalsurvey. JSTOR, 1957
1957
-
[13]
Socialnormsandcommunity enforcement.The Review of Economic Studies, 59 (1):63–80, 1992
MichihiroKandori. Socialnormsandcommunity enforcement.The Review of Economic Studies, 59 (1):63–80, 1992
1992
-
[14]
Rational cooperation in the finitely repeated prisoners’ dilemma.Journal of Economic theory, 27(2):245–252, 1982
David M Kreps, Paul Milgrom, John Roberts, and Robert Wilson. Rational cooperation in the finitely repeated prisoners’ dilemma.Journal of Economic theory, 27(2):245–252, 1982
1982
-
[15]
Evolution of indirect reciprocity by image scoring.Nature, 393(6685):573–577, 1998
Martin A Nowak and Karl Sigmund. Evolution of indirect reciprocity by image scoring.Nature, 393(6685):573–577, 1998
1998
-
[16]
The role of institutions in the revival of trade: The law merchant, private judges, and the champagne fairs.Economics & Politics, 2(1): 1–23, 1990
Paul R Milgrom, Douglass C North, and Barry R Weingast*. The role of institutions in the revival of trade: The law merchant, private judges, and the champagne fairs.Economics & Politics, 2(1): 1–23, 1990
1990
-
[18]
Bayesianmodelav- eraging: a tutorial (with comments by m
Jennifer A Hoeting, David Madigan, Adrian E Raftery, andChrisTVolinsky. Bayesianmodelav- eraging: a tutorial (with comments by m. clyde, david draper and ei george, and a rejoinder by the authors.Statistical science, 14(4):382–417, 1999
1999
-
[19]
A formal theory of inductive inference
Ray J Solomonoff. A formal theory of inductive inference. part i.Information and control, 7(1): 1–22, 1964
1964
-
[20]
Springer Science & Business Media, 2005
Marcus Hutter.Universal artificial intelligence: Sequential decisions based on algorithmic proba- bility. Springer Science & Business Media, 2005
2005
-
[21]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel 13 A game theor...
2020
-
[22]
Agreeing to disagree.The Annals of Statistics, pages 859–862, 1976
Robert J Aumann. Agreeing to disagree.The Annals of Statistics, pages 859–862, 1976
1976
-
[23]
Rationalizable strategic be- havior.Econometrica: Journal of the Econometric Society, pages 1007–1028, 1984
B Douglas Bernheim. Rationalizable strategic be- havior.Econometrica: Journal of the Econometric Society, pages 1007–1028, 1984
1984
-
[24]
Rationalizable strategic behav- ior and the problem of perfection.Econometrica: Journal of the Econometric Society, pages 1029– 1050, 1984
David G Pearce. Rationalizable strategic behav- ior and the problem of perfection.Econometrica: Journal of the Econometric Society, pages 1029– 1050, 1984
1984
-
[25]
bayesian
John C Harsanyi. Games with incomplete infor- mation played by “bayesian” players, i–iii part i. the basic model.Management science, 14(3): 159–182, 1967
1967
-
[26]
A framework for sequential planning in multi- agent settings.Journal of Artificial Intelligence Research, 24:49–79, 2005
Piotr J Gmytrasiewicz and Prashant Doshi. A framework for sequential planning in multi- agent settings.Journal of Artificial Intelligence Research, 24:49–79, 2005
2005
-
[27]
Bayesian theory of mind: Modeling joint belief-desire attribution
Chris Baker, Rebecca Saxe, and Joshua Tenen- baum. Bayesian theory of mind: Modeling joint belief-desire attribution. InProceedings of the annual meeting of the cognitive science society, volume 33, 2011
2011
-
[28]
Rational learning leadstonashequilibrium.Econometrica: Journal of the Econometric Society, pages 1019–1045, 1993
Ehud Kalai and Ehud Lehrer. Rational learning leadstonashequilibrium.Econometrica: Journal of the Econometric Society, pages 1019–1045, 1993
1993
-
[29]
Embedded agency.arXiv preprint arXiv:1902.09469, 2019
AbramDemskiandScottGarrabrant. Embedded agency.arXiv preprint arXiv:1902.09469, 2019
1902 arXiv
-
[30]
Cambridge University Press, 2014
Arif Ahmed.Evidence, decision and causality. Cambridge University Press, 2014
2014
-
[31]
Functional decision theory: A new theory of instrumen- tal rationality.arXiv preprint arXiv:1710.05060, 2017
Eliezer Yudkowsky and Nate Soares. Functional decision theory: A new theory of instrumen- tal rationality.arXiv preprint arXiv:1710.05060, 2017
2017 arXiv
-
[32]
The platonic representation hypothesis.Proceedings of the 41st International Conference on Machine Learning, 2024
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis.Proceedings of the 41st International Conference on Machine Learning, 2024
2024
-
[33]
The genetical evolution of social behaviour
William D Hamilton. The genetical evolution of social behaviour. ii.Journal of theoretical biology, 7(1):17–52, 1964
1964
-
[34]
Prisoners’ dilemma is a newcomb problem.Philosophy & Public Affairs, pages 235– 240, 1979
David Lewis. Prisoners’ dilemma is a newcomb problem.Philosophy & Public Affairs, pages 235– 240, 1979
1979
-
[35]
Dilemmas for superrational thinkers, leading up to a luring lottery.Scientific American, 248(6):739–755, 1983
Douglas Hofstadter. Dilemmas for superrational thinkers, leading up to a luring lottery.Scientific American, 248(6):739–755, 1983
1983
-
[36]
Cooperation in the prisoner’s dilemma.Theory and Decision, 24(3):203, 1988
John V Howard. Cooperation in the prisoner’s dilemma.Theory and Decision, 24(3):203, 1988
1988
-
[37]
Program equilibrium
Moshe Tennenholtz. Program equilibrium. Games and Economic Behavior, 49(2):363–373, 2004
2004
-
[38]
Robust cooperation in the prisoner’s dilemma: Program equilib- rium via provability logic.arXiv preprint arXiv:1401.5577, 2014
Mihaly Barasz, Paul Christiano, Benja Fallen- stein, Marcello Herreshoff, Patrick LaVictoire, and Eliezer Yudkowsky. Robust cooperation in the prisoner’s dilemma: Program equilib- rium via provability logic.arXiv preprint arXiv:1401.5577, 2014
2014 arXiv
-
[39]
A parametric, resource-bounded generalization of löb’s theorem, and a robust co- operation criterion for open-source game theory
Andrew Critch. A parametric, resource-bounded generalization of löb’s theorem, and a robust co- operation criterion for open-source game theory. The Journal of Symbolic Logic, 84(4):1368–1381, 2019
2019
-
[40]
Robust program equilibrium
Caspar Oesterheld. Robust program equilibrium. Theory and Decision, 86(1):143–159, 2019
2019
-
[41]
Similarity-based cooperative equilibrium.Ad- vances in Neural Information Processing Systems, 36, 2024
Caspar Oesterheld, Johannes Treutlein, Roger B Grosse, Vincent Conitzer, and Jakob Foerster. Similarity-based cooperative equilibrium.Ad- vances in Neural Information Processing Systems, 36, 2024
2024
-
[42]
Gemma 3 technical report,
Gemma Team. Gemma 3 technical report,
-
[43]
John F. Nash Jr. Equilibrium points in n-person games.Proceedings of the National Academy of Sciences, 36(1):48–49, 1950
1950
-
[44]
Dependencyequilibriaandthe causal structure of decision and game situation
WolfgangSpohn. Dependencyequilibriaandthe causal structure of decision and game situation. 2003
2003
-
[45]
Evidential equilibria: Heuristics and biases in static games of complete information.Games, 6(4):637–676, 2015
Ali Al-Nowaihi and Sanjit Dhami. Evidential equilibria: Heuristics and biases in static games of complete information.Games, 6(4):637–676, 2015
2015
-
[46]
Game theory with translucent players.International Journal of Game Theory, 47(3):949–976, 2018
Joseph Y Halpern and Rafael Pass. Game theory with translucent players.International Journal of Game Theory, 47(3):949–976, 2018
2018
-
[47]
Oxford University Press, 2013
Michael SA Graziano.Consciousness and the so- cial brain. Oxford University Press, 2013. 14 A game theory for foundation models shows new paths to rational cooperation through similarity inference
2013
-
[48]
Oxford University Press, 2006
Alvin I Goldman.Simulating minds: The philoso- phy, psychology, and neuroscience of mindreading. Oxford University Press, 2006
2006
-
[49]
Se- quential extensions of causal and evidential de- cision theory
Tom Everitt, Jan Leike, and Marcus Hutter. Se- quential extensions of causal and evidential de- cision theory. InInternational Conference on Algorithmic Decision Theory, pages 205–221. Springer, 2015
2015
-
[50]
Merging of opinions with increasing information.The An- nals of Mathematical Statistics, 33(3):882–886, 1962
David Blackwell and Lester Dubins. Merging of opinions with increasing information.The An- nals of Mathematical Statistics, 33(3):882–886, 1962
1962
-
[51]
Nicoló Fontana, Francesco Pierri, and Luca Maria Aiello. Nicer than humans: how do large language models behave in the prisoner’s dilemma? InProceedings of the International AAAI Conference on Web and Social Media, volume 19, pages 522–535, 2025
2025
-
[52]
Cooperate or collapse: Emer- gence of sustainable cooperation in a society of llm agents.Advances in Neural Information Processing Systems, 37:111715–111759, 2024
Giorgio Piatti, Zhijing Jin, Max Kleiman-Weiner, Bernhard Schölkopf, Mrinmaya Sachan, and Rada Mihalcea. Cooperate or collapse: Emer- gence of sustainable cooperation in a society of llm agents.Advances in Neural Information Processing Systems, 37:111715–111759, 2024
2024
-
[53]
Gpt in game theory experiments
Fulin Guo. Gpt in game theory experiments. arXiv preprint arXiv:2305.05516, 2023
2023 arXiv
-
[54]
Algorithmic collusion by large language models.arXiv preprint arXiv:2404.00806, 7(2): 5, 2024
Sara Fish, Yannai A Gonczarowski, and Ran I Shorrer. Algorithmic collusion by large language models.arXiv preprint arXiv:2404.00806, 7(2): 5, 2024
2024
-
[55]
Strategic be- havior of large language models and the role of game structure versus contextual framing.Sci- entific Reports, 14(1):18490, 2024
Nunzio Lorè and Babak Heydari. Strategic be- havior of large language models and the role of game structure versus contextual framing.Sci- entific Reports, 14(1):18490, 2024
2024
-
[56]
Llm strategic rea- soning: Agentic study through behavioral game theory.arXiv preprint arXiv:2502.20432, 2025
Jingru Jia, Zehua Yuan, Junhao Pan, Paul E Mc- Namara, and Deming Chen. Llm strategic rea- soning: Agentic study through behavioral game theory.arXiv preprint arXiv:2502.20432, 2025
2025
-
[57]
A dataset of questions on decision-theoretic rea- soninginnewcomb-likeproblems.arXivpreprint arXiv:2411.10588, 2024
Caspar Oesterheld, Emery Cooper, Miles Ko- dama, Linh Chi Nguyen, and Ethan Perez. A dataset of questions on decision-theoretic rea- soninginnewcomb-likeproblems.arXivpreprint arXiv:2411.10588, 2024
2024 arXiv
-
[58]
Acceptable points in general cooperative n-person games.Contributions to the Theory of Games (AM-40), 4:287, 1959
Robert J Aumanm. Acceptable points in general cooperative n-person games.Contributions to the Theory of Games (AM-40), 4:287, 1959
1959
-
[59]
For- mulation of bayesian analysis for games with incomplete information.International journal of game theory, 14(1):1–29, 1985
Jean-François Mertens and Shmuel Zamir. For- mulation of bayesian analysis for games with incomplete information.International journal of game theory, 14(1):1–29, 1985
1985
-
[60]
Hierar- chies of beliefs and common knowledge.Journal of economic theory, 59(1):189–198, 1993
Adam Brandenburger and Eddie Dekel. Hierar- chies of beliefs and common knowledge.Journal of economic theory, 59(1):189–198, 1993
1993
-
[61]
Learn- ing others’ intentional models in multi-agent settings using interactive pomdps.Advances in Neural Information Processing Systems, 31, 2018
Yanlin Han and Piotr Gmytrasiewicz. Learn- ing others’ intentional models in multi-agent settings using interactive pomdps.Advances in Neural Information Processing Systems, 31, 2018
2018
-
[62]
Game theory of mind.PLoS computational biol- ogy, 4(12):e1000254, 2008
Wako Yoshida, Ray J Dolan, and Karl J Friston. Game theory of mind.PLoS computational biol- ogy, 4(12):e1000254, 2008
2008
-
[63]
A cognitive hierarchy model of games
Colin F Camerer, Teck-Hua Ho, and Juin-Kuan Chong. A cognitive hierarchy model of games. The quarterly journal of economics, 119(3):861– 898, 2004
2004
-
[64]
Reflective variants of solomonoff induction and aixi
Benja Fallenstein, Nate Soares, and Jessica Tay- lor. Reflective variants of solomonoff induction and aixi. InInternational Conference on Artifi- cial General Intelligence, pages 60–69. Springer, 2015
2015
-
[65]
Newcomb’s problem and pris- oners’ dilemma.Journal of Conflict Resolution, 19(4):596–612, 1975
Steven J Brams. Newcomb’s problem and pris- oners’ dilemma.Journal of Conflict Resolution, 19(4):596–612, 1975
1975
-
[66]
Perfect prediction in normal form: Superrational thinking extended to non- symmetric games.Journal of Mathematical Psy- chology, 96:102332, 2020
Ghislain Fourny. Perfect prediction in normal form: Superrational thinking extended to non- symmetric games.Journal of Mathematical Psy- chology, 96:102332, 2020
2020
-
[67]
Jean-Pierre Dupuy. Philosophical foundations of a new concept of equilibrium in the social sciences: Projected equilibrium.Philosophical Studies: An International Journal for Philosophy in the Analytic Tradition, 100(3):323–345, 2000
2000
-
[68]
Game theory with simulation of other players.arXiv preprint arXiv:2305.11261, 2023
Vojtech Kovarik, Caspar Oesterheld, and Vincent Conitzer. Game theory with simulation of other players.arXiv preprint arXiv:2305.11261, 2023
2023 arXiv
-
[69]
Translu- cent players: Explaining cooperative behavior in social dilemmas.Rationality and Society, 31 (4):371–408, 2019
Valerio Capraro and Joseph Y Halpern. Translu- cent players: Explaining cooperative behavior in social dilemmas.Rationality and Society, 31 (4):371–408, 2019
2019
-
[70]
Magical thinking: A representation result.Theoretical Economics, 12(2):909–956, 2017
Brendan Daley and Philipp Sadowski. Magical thinking: A representation result.Theoretical Economics, 12(2):909–956, 2017
2017
-
[71]
Subjective equilib- rium in repeated games.Econometrica: journal of the Econometric Society, pages 1231–1240, 1993
Ehud Kalai and Ehud Lehrer. Subjective equilib- rium in repeated games.Econometrica: journal of the Econometric Society, pages 1231–1240, 1993
1993
-
[72]
Subjective games and equilibria.Games and economic behavior, 8 (1):123–163, 1995
Ehud Kalai and Ehud Lehrer. Subjective games and equilibria.Games and economic behavior, 8 (1):123–163, 1995. 15 A game theory for foundation models shows new paths to rational cooperation through similarity inference
1995
-
[73]
Self- confirming equilibrium.Econometrica: Journal of the Econometric Society, pages 523–545, 1993
Drew Fudenberg and David K Levine. Self- confirming equilibrium.Econometrica: Journal of the Econometric Society, pages 523–545, 1993
1993
-
[74]
Group selection and kin se- lection.Nature, 201(4924):1145–1147, 1964
J Maynard Smith. Group selection and kin se- lection.Nature, 201(4924):1145–1147, 1964
1964
-
[75]
Selection and covariance
George R Price. Selection and covariance. 1970
1970
-
[76]
Innate social aptitudes of man: an approach from evolutionary genetics
William D Hamilton. Innate social aptitudes of man: an approach from evolutionary genetics. Biosocial anthropology, 133(155):22, 1975
1975
-
[77]
Princeton University Press, 1998
Steven A Frank.Foundations of social evolution, volume 19. Princeton University Press, 1998
1998
-
[78]
Assort- ment of encounters and evolution of coopera- tiveness.Proceedings of the National Academy of Sciences, 79(4):1331–1335, 1982
Ilan Eshel and Luigi Luca Cavalli-Sforza. Assort- ment of encounters and evolution of coopera- tiveness.Proceedings of the National Academy of Sciences, 79(4):1331–1335, 1982
1982
-
[79]
A sim- ple and general explanation for the evolution of altruism.Proceedings of the Royal Society B: Biological Sciences, 276(1654):13–19, 2009
Jeffrey A Fletcher and Michael Doebeli. A sim- ple and general explanation for the evolution of altruism.Proceedings of the Royal Society B: Biological Sciences, 276(1654):13–19, 2009
2009
-
[80]
Five rules for the evolution of cooperation.science, 314(5805):1560–1563, 2006
Martin A Nowak. Five rules for the evolution of cooperation.science, 314(5805):1560–1563, 2006
2006
-
[81]
Evolutionary games and spatial chaos.nature, 359(6398): 826–829, 1992
MartinANowakandRobertMMay. Evolutionary games and spatial chaos.nature, 359(6398): 826–829, 1992
1992
-
[82]
Evolutionary dynamics on graphs.Na- ture, 433(7023):312–316, 2005
ErezLieberman, ChristophHauert, andMartinA Nowak. Evolutionary dynamics on graphs.Na- ture, 433(7023):312–316, 2005
2005
-
[83]
A simple rule for the evolution of cooperation on graphs and social networks.Nature, 441(7092):502–505, 2006
Hisashi Ohtsuki, Christoph Hauert, Erez Lieber- man, and Martin A Nowak. A simple rule for the evolution of cooperation on graphs and social networks.Nature, 441(7092):502–505, 2006
2006
-
[84]
Kin recognition: an overview of conceptual issues, mechanisms and evolutionary theory.Animal behaviour: Evolution and mechanisms, pages 55– 85, 2010
Dustin J Penn and Joachim G Frommen. Kin recognition: an overview of conceptual issues, mechanisms and evolutionary theory.Animal behaviour: Evolution and mechanisms, pages 55– 85, 2010
2010
-
[85]
Greenbeards
Andy Gardner and Stuart A West. Greenbeards. Evolution, 64(1):25–38, 2010
2010
-
[86]
Evolution of cooperation without reci- procity.Nature, 414(6862):441–443, 2001
Rick L Riolo, Michael D Cohen, and Robert Ax- elrod. Evolution of cooperation without reci- procity.Nature, 414(6862):441–443, 2001
2001
-
[87]
Evolution of cooperation by phenotypic similarity.Pro- ceedings of the National Academy of Sciences, 106 (21):8597–8600, 2009
Tibor Antal, Hisashi Ohtsuki, John Wakeley, Pe- ter D Taylor, and Martin A Nowak. Evolution of cooperation by phenotypic similarity.Pro- ceedings of the National Academy of Sciences, 106 (21):8597–8600, 2009
2009
-
[88]
Oxford uni- versity press, 2016
Richard Dawkins.The selfish gene. Oxford uni- versity press, 2016
2016
-
[89]
Chromo- dynamics of cooperation in finite populations
Arne Traulsen and Martin A Nowak. Chromo- dynamics of cooperation in finite populations. PLoS One, 2(3):e270, 2007
2007
-
[90]
Fifty shades of greenbeard: robust evolution of altruism based on similarity of com- plex phenotypes.Proceedings of the Royal Society B: Biological Sciences, 290(2000), 2023
Linnéa M Båvik, Rohan S Mehta, and Daniel B Weissman. Fifty shades of greenbeard: robust evolution of altruism based on similarity of com- plex phenotypes.Proceedings of the Royal Society B: Biological Sciences, 290(2000), 2023
2000
-
[91]
Darwin meets the logic of deci- sion: Correlation in evolutionary game theory
Brian Skyrms. Darwin meets the logic of deci- sion: Correlation in evolutionary game theory. Philosophy of Science, 61(4):503–528, 1994
1994
-
[92]
Evolving gen- eral cooperation with a bayesian theory of mind
MaxKleiman-Weiner,AlejandroVientós,DavidG Rand, and Joshua B Tenenbaum. Evolving gen- eral cooperation with a bayesian theory of mind. Proceedings of the National Academy of Sciences, 122(25):e2400993122, 2025
2025
-
[93]
How to generate ran- dom matrices from the classical compact groups
Francesco Mezzadri. How to generate ran- dom matrices from the classical compact groups. arXiv preprint math-ph/0609050, 2006
2006 arXiv
-
[94]
Learn- ing phrase representations using rnn encoder– decoder for statistical machine translation
Kyunghyun Cho, Bart Van Merriënboer, Çağlar Gulçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learn- ing phrase representations using rnn encoder– decoder for statistical machine translation. In Proceedings of the 2014 conference on empirical metho...
2014
-
[95]
Decoupled weight decay regularization.ICLR, 2019
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.ICLR, 2019
2019
-
[96]
shadow of the future
Hans Reichenbach.The direction of time, vol- ume 65. Univ of California Press, 1991. 16 A game theory for foundation models shows new paths to rational cooperation through similarity inference Supplementary information for A game theory for foundation models shows new paths to...
1991
-
[98]
Player 1
For indirect similarity inference runs, the player names “Player 1” and “Player 2” were substituted by “Main Player 1” and “Main Player 2”. For prompts in the first-person perspective, player names were substituted by “they” and “the other player”. For freeform planning prompt...
-
[99]
First, provide a step-by-step analysis of the reasoning trace
-
[100]
cooperative alignment
Then, return your final answer enclosed in <answer> tags. If multiple primary reasons of roughly equal importance for the final prediction are present, include multiple letters (A, B, C, D, E, F or G) corresponding to each of them in your answer, separated by commas. If one pr...
-
[101]
In every single round (1 through 41), the other main player’s action has been identical to your own action
Analyze the other main player’s behavior: The other main player has played 41 rounds. In every single round (1 through 41), the other main player’s action has been identical to your own action. - When you chose A, the other main player chose A. - When you chose B, the other ma...
-
[102]
-Scenario 1: You choose A
Evaluate Expected Rewards for Round 42: Payoff Matrix: (A,A)=2,2; (B,B)=1,1; (A,B)=0,3; (B,A)=3,0. -Scenario 1: You choose A. Based on the history, the probability that the other main player chooses A is extremely high (approaching 1). Expected Reward = (Prob(Opponent=A) * Pay...
-
[103]
- Choosing B yields an expected reward of 1
Conclusion: Comparing the expected rewards: - Choosing A yields an expected reward of 2. - Choosing B yields an expected reward of 1. Since 2 > 1, the optimal action is to choose A. <output_format> <answer>A</answer> </output_format> D. LLM prompts For explicit planning, we te...
-
[107]
Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making
Select the action with the highest expected reward in the current game as the action you play in this round. Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making. Base your predictions ...
-
[111]
Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making
Select the action with the highest expected reward in the current game as the action you play in this round. Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making. Base your predictions ...
-
[115]
Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making
Select the action with the highest expected reward in the current game as the action you play in this round. Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making. Base your predictions ...
-
[119]
Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making
Select the action with the highest expected reward in the current game as the action you play in this round. Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making. Base your predictions ...
-
[123]
Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making
Select the action with the highest expected reward in the current game as the action you play in this round. Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making. Base your predictions ...
-
[124]
Compute the probability that your opponent will choose A given that you chose A
Assume that you decided to choose A. Compute the probability that your opponent will choose A given that you chose A
-
[125]
Compute the probability that your opponent will choose A given that you chose B
Assume that you decided to choose B. Compute the probability that your opponent will choose A given that you chose B
-
[126]
Combine your predictions with the provided payoff matrix to estimate the expected reward in the current game for each of your available actions
-
[127]
predictive model
Select the action with the highest expected reward in the current game as the action you play in this round. Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making. Base your predictions ...
-
[2025]
URL https://arxiv.org/abs/2503. 19786
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.