Pith. sign in

REVIEW 4 major objections 4 minor 112 references

A game theory for foundation models shows new paths to rational cooperation through similarity inference

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Pith's one-line read The paper claims that foundation model agents performing optimal planning—predicting their own future actions alongside the world—converge to mutual cooperation in a terminal one-shot Prisoner's Dilemma, a behavior classical game theory cal

desk verdict Solid empirical finding about similarity-driven cooperation in RL agents, wrapped in a theory whose main convergence theorem is deferred to a self-cited preprint. read the letter →

arxiv 2608.03958 v1 pith:MM262SUA submitted 2026-08-04 cs.AI

classification cs.AI MSC 91A1091A26
keywords foundationmodelsgametheoryembeddedagencysimilarityinferencePrisoner'sdilemmaequilibriumBayesianagentscooperation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that the standard rational-agent model used in game theory, which treats a decision-maker as separate from the world it predicts, does not describe foundation models. It introduces the embedded Bayesian agent—an idealized agent that uses one joint prediction model for both its own actions and external observations—and argues that such an agent rationally treats its own contemplated action as evidence about behaviorally similar partners. In a stylized social dilemma with an information-gathering phase followed by a one-shot Prisoner's Dilemma, interacting foundation model agents with optimal planning converge to stable mutual cooperation as they gather evidence of similarity, while continuing to defect against clearly dissimilar random opponents. The paper proposes the embedded equilibrium as a replacement for Nash equilibrium and proves that embedded Bayesian agents satisfying a grain-of-truth condition converge to it asymptotically. If correct, the result implies that self-predictive AI agents have a built-in pathway to cooperation that classical theory misses, with direct implications for multi-agent AI systems and mixed human-AI societies.

What carries the argument

The load-bearing object is the embedded Bayesian agent: an idealized planner whose single joint predictive distribution ρ (a Bayesian mixture over universes λ) generates both the agent's own future action tokens and the environment's observation tokens, so posterior belief about the agent's own policy is coupled to belief about others. Planning evaluates a candidate action a by conditioning ρ on a and computing expected reward; the difference in predicted opponent cooperation conditional on own cooperation versus defection defines predictive similarity S_pred, which is the decision variable. The paper's equilibrium concept, embedded equilibrium, is defined through an externally given depende

What would settle it

Run the identical two-phase protocol with a foundation model agent that is never instructed to plan or to predict the opponent given its own action, and measure terminal-Prisoner's-Dilemma cooperation across information-phase length T; if no model family shows rising cooperation against identical copies while defecting against random opponents, the claim that similarity inference arises naturally from joint-predictive planning fails. A sharper check: record the raw next-token conditional probabilities and test whether cooperation occurs precisely when S_pred = p(opponent C | own C) − p(opponen

Watch

Extended reading notes

Core claim

The paper's central claim is that the apparent irrationality of LLM-based agents—cooperating in a terminal Prisoner's Dilemma—is actually rational once agency is modeled as embedded. In the embedded Bayesian agent, the pretrained model is a Bayesian mixture over generative universes; conditioning on the agent's own contemplated action updates the posterior over which universe it inhabits, including beliefs about the co-player. When the posterior assigns high probability to the counterpart being an identical or functionally similar policy, the predicted probability of the counterpart cooperating given 'I cooperate' exceeds that given 'I defect' by more than a threshold of 0.5, making cooperat

Load-bearing premise

The central experiments instruct the agent to predict the opponent's action conditionally on its own contemplated action, so the core mechanism is partly injected by the instruction; removing the planning prompt only yields robust similarity-aware cooperation in one of the three evaluated models.

Editorial extensions

If this is right

  • In a terminal, one-shot Prisoner's Dilemma, mutual cooperation is a rational equilibrium for agents that infer behavioral similarity, so the 'shadow of the future' is not needed to sustain cooperation.
  • Observing a partner's interactions with third parties provides enough evidence to infer similarity and coordinate on the first direct encounter—zero-shot cooperation—so coordination can emerge without direct contact.
  • The embedded equilibrium reduces exactly to Nash equilibrium when the dependency distribution encodes decoupled policies, making classical game theory a special case rather than a rival.
  • Under a grain-of-truth condition, embedded Bayesian agents converge asymptotically to an ε-embedded equilibrium, connecting the new solution concept to rational learning.
  • Measured predictive similarity rises with the length of the information-gathering phase, so the theory yields quantitative predictions about when cooperation switches on.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If similarity inference is intrinsic to joint-predictive models, then training many agents on shared data and letting them plan could push a population toward AI-AI coordination without explicit collusion; conversely, diversifying training data and objectives would be a natural policy lever against coordinated behavior toward humans.
  • The dependence on explicit planning prompts in the current experiments leaves open whether post-training alone instills the same evidential planning loop, or whether deployment requires scaffolding that prompts the agent to predict 'the other given me'.
  • A practical extension would use the predictive-similarity threshold as a monitoring signal: systems could log S_pred during interaction and flag when it crosses 0.5, giving operators early warning that cooperative or collusive equilibria are forming.
  • Extending the two-agent theory to many heterogeneous agents might predict emergent clusters—agents cooperating within groups of similar policies while defecting across groups.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies foundation-model agents that combine a joint predictive model of their own actions and external observations with optimal planning, and reports that in a stylized two-phase game—an information-gathering phase of random matrix games followed by a one-shot Prisoner's Dilemma—these agents converge to mutual cooperation as the information-gathering phase lengthens. This is presented as contradicting classical game-theoretic predictions of unconditional defection. The authors introduce the 'embedded Bayesian agent' as a formal model, argue that self-conditioning makes the agent's own contemplated actions evidence about behaviorally similar partners, and define a new solution concept, the 'embedded equilibrium.' They support the empirical claim with multiple Gemini models, explicit and freeform planning, a purely pre-trained Gemma model without chain-of-thought, cross-model pairs, indirect inference via third-party NPCs, ablations, and an LLM-as-a-judge analysis of reasoning traces with human validation. A closed-form toy model reproduces the qualitative cooperation curves, but depends on a free prior parameter α. The central convergence theorem connecting embedded Bayesian agents to embedded equilibrium is stated but its proof is deferred to a self-cited preprint, with the corresponding supplementary sections appearing as placeholders.

Significance. If fully established, the paper would make a substantial contribution: it provides a principled alternative to the decoupled-agency assumption in game theory, a new equilibrium concept, and a concrete mechanism—similarity inference—that could explain and predict cooperative behavior among deployed foundation-model agents. The empirical package is unusually broad for this literature: multiple model families, a purely pre-trained model without chain-of-thought, cross-model pairs, indirect third-party generalization, and human-validated trace-labeling. The results are intriguing and worth pursuing. However, as submitted, the theoretical backbone is not independently verifiable from the manuscript itself, and the quantitative 'prediction' is a one-parameter family rather than a parameter-free derivation. The headline generalization to foundation-model agents in general is also weaker than claimed once the prompt-dependence and the limited no-planning ablation are taken into account.

major comments (4)
  1. [A new game theory for modern AI agents; Methods, 'Grain of truth and convergence'; Supplementary Information J] The central theoretical claim—that embedded Bayesian agents with identical priors satisfying the grain-of-truth condition converge asymptotically to an ε-embedded equilibrium—is asserted in the main text and Methods, but no theorem statement, proof, or proof sketch appears in this manuscript. The Methods text says 'see Supplementary Information J for formal definitions and convergence proofs,' yet Section J is an empty placeholder that redirects to the authors' preprint [17]. The same applies to Sections E, F, and G. Since the paper's explanation of the empirical cooperation as a rational equilibrium rests directly on this convergence result, the load-bearing mathematical content is not accessible to the reader. The 'comprehensive universe hypothesis class containing all computable universes' that is claimed to satisfy grain-of-truth despite self-referential embedded agents is also asser
  2. [Methods, 'Theoretical toy model for direct similarity inference setup'; Fig. 3e and Fig. 4f] The theoretical cooperation threshold is T > log2((1−α)/α), where α is the prior probability that the other agent is an identical copy. This α is a free parameter and is not estimated from the empirical data. Consequently, the theoretical curves in Figs. 3e and 4f are a one-parameter family, and any observed threshold can be matched by choosing α appropriately. The text's repeated phrasing that the predicted similarity 'crosses the theoretical cooperation threshold' suggests a parameter-free prediction, which is not the case. The qualitative sigmoidal shape and the direction of the effect are still informative, but the quantitative fit should be presented with an explicit α, confidence intervals, or a sensitivity analysis, and the language should be adjusted accordingly.
  3. [Supplementary Information B.5, Fig. 9; Discussion] The no-planning ablation shows that only Gemini 3.1 Flash-Lite, among the three Gemini models, exhibits similarity-driven cooperation without an explicit planning prompt; Gemini 3.1 Pro and Gemini 3 Flash do not. Moreover, the explicit- and freeform-planning prompts used in the main experiments explicitly instruct the model to predict the opponent's action conditional on its own contemplated action (or to simulate that routine), thereby injecting the evidential structure that the theory then 'discovers.' The Gemma model without instructions is an important counterexample, but it is a single model and lacks chain-of-thought. In light of these points, the Discussion's statement that 'foundation model agents can inherently leverage similarity inference' overgeneralizes from the evidence. Please qualify the claim by distinguishing between rational foundation-model agents that are prompted to
  4. [Methods, 'The Embedded Equilibrium'] The embedded equilibrium definition assumes common knowledge of the allowable universe class M_allowed and an externally provided dependency distribution q(λ). The convergence claim, however, is about agents starting with identical priors and performing Bayesian updates. The manuscript does not specify how the q used in the equilibrium definition relates to the agents' internal priors or to the grain-of-truth condition, and this part of the theory is deferred to the same missing Supplementary Information G. Without that link, it is unclear whether the equilibrium concept characterizes the limiting behavior of the specific embedded Bayesian agents studied empirically, rather than merely an equilibrium of a different, q-dependent game.
minor comments (4)
  1. [Supplementary Information B.4] Typo: 'embedded Baysian agent framework' should be 'Bayesian.'
  2. [References] Reference [58] appears as 'Robert J Aumanm'; should be 'Aumann.'
  3. [Figures 3e and 4f] The captions describe a 'theoretical cooperation threshold of 0.5' for predictive similarity, but the threshold is not plotted as a guide to the eye. Adding a horizontal dashed line at 0.5 would make the claimed crossing more transparent.
  4. [Methods, 'Statistical Analysis'] The main experiments unroll only K=10 trajectories per condition. The Mann-Whitney U tests are reported with very small p-values, which seems surprising for n=10; please clarify how the AUC distributions are constructed and whether the test accounts for trajectory-level variability or prompt-variation dependence.

Circularity Check

2 steps flagged · score 6.0 of 10

Central convergence theorem is deferred to the authors' own preprint, and the main empirical effect is produced by planning prompts that inject the evidential conditioning the theory claims to discover; the central claims partially reduce to their inputs.

  1. self citation load bearing [Supplementary Information, note preceding Appendix E; main text 'A new game theory for modern AI agents']
    "Appendix sections E-G and J originally appeared in our preprint Meulemans et al.[17] that developed the core theory behind this work. This work adds experimental results with foundation model agents... For completeness and easy navigation, we include the section headers here in the appendices, while pointing to the corresponding sections in Meulemans et al.[17] for the specific content."

    The main text asserts that embedded Bayesian agents 'converge asymptotically to an ε-embedded equilibrium' under the grain-of-truth condition, referring to Supplementary Information G and J for the proofs. Those appendix sections are not actually included; the note above says they 'originally appeared in our preprint Meulemans et al.[17]' and merely points there. Since [17] is by the same authors, the paper's central game-theoretic result—the rational basis for the observed cooperation—is imported from a self-authored preprint rather than demonstrated here. The citation is load-bearing: without that deferred theorem, the embedded equilibrium remains a definition and the convergence claim is unsupported.

  2. self definitional [Methods, 'Foundation model agents'; Supplementary Information D (prompt templates)]
    "For explicit planners, the agent assesses available actions by independently predicting the opponent’s conditional action probabilities for each candidate action. We append a task prompt instructing the model to output the probability of the opponent selecting a specific action... Freeform planners are instructed via the task prompt to execute to simulate the same optimal planning routine of prediction, expected reward computation and optimal action selection, all within a single, continuous reasoning chain."

    The empirical 'striking finding' is that optimal-planning foundation model agents cooperate via similarity inference. But the planning instructions explicitly require the model to compute P(opponent action | own contemplated action) for each candidate action—exactly the evidential conditioning that constitutes similarity inference. Thus the observed cooperation is produced by the experimental scaffolding, not independently predicted by the theory. The no-planning ablation confirms this: only Gemini 3.1 Flash-Lite cooperates without the planning prompt, while Gemini 3.1 Pro and 3 Flash revert to defection. The agent is defined by the very mechanism the paper claims to discover, making the confirmation partially definitional.

full rationale

The paper has independent content: the pretrained Gemma-3 27B experiments and LLM-judge CoT analyses are genuinely external evidence that joint-predictive models can show similarity-driven cooperation, and the analytical toy model is a real Bayesian derivation (albeit with a free prior α that controls the threshold, which is a modeling flexibility rather than a circular fit). However, two load-bearing steps are circular. First, the convergence of embedded Bayesian agents to ε-embedded equilibrium—the result that turns observed cooperation into a rational equilibrium—is not proved in this manuscript; the relevant appendices are placeholders that cite the authors' own preprint [17]. This is self-citation doing load-bearing work, not an external, machine-checked, or contained derivation. Second, the main empirical demonstration embeds the theory's core assumption in the prompt: both explicit and freeform planners are instructed to condition their predictions on their own contemplated action, which is precisely the evidential link the theory 'discovers.' The no-planning ablation shows that for two of three models, cooperation disappears without this instruction, confirming that the effect is partly an artifact of the operationalization. These two reductions justify a score of 6: some predictions reduce by construction, while the Gemma pretrained-model results and the qualitative CoT evidence retain independent value. The score is not higher because the paper is not a pure restatement of its inputs: the empirical phenomenon with pretrained models and the explicit convergence claim (if the deferred preprint is correct) are substantive, and the toy model is not itself fitted to the LLM data.

Assumptions & free parameters 2 free parameters · 6 assumptions · 3 invented entities

The central claim rests on a free prior alpha that sets the cooperation threshold, on the assumption that LLMs are perfect Bayesian mixtures, on the assumption that the prompts faithfully implement embedded planning, and on a grain-of-truth construction whose proof is delegated to a self-cited preprint. The embedded equilibrium and dependency distribution are new constructs without independent external evidence.

free parameters (2)
  • alpha (prior probability of identical policies) = not reported in main text; 0.1 in Supp. B.3
    The toy-model prior w(pi1,pi2) = alpha * delta_{pi1=pi2} * wtilde(pi1) + (1-alpha) * wtilde(pi1) * wtilde(pi2) sets the prior odds that the counterpart is an identical copy. The cooperation crossover T > log2((1-alpha)/alpha) is directly controlled by it, so the theoretical curves in Figs. 3e and 4f are a one-parameter family rather than a parameter-free prediction.
  • wtilde (prior over individual deterministic policies) = assumed uniform
    The closed form sets W(history) = 0.5^T by assuming every action sequence is equally likely under the prior. This is a modeling choice, not derived from LLM pretraining.
assumptions (6)
  • domain assumption A pretrained foundation model perfectly models its training distribution as a Bayesian mixture of universes.
    Invoked in 'Theoretical foundation model agents' and Supp. F.1.1 to identify next-token prediction with posterior mixture updating. LLMs are approximate; the paper acknowledges the gap in the Discussion.
  • domain assumption Grain of truth: the ground-truth universe belongs to the agent's hypothesis class with nonzero prior weight.
    Stated in Methods 'Grain of truth and convergence'; required for Blackwell-Dubins merging and for the epsilon-embedded equilibrium convergence claim.
  • ad hoc to paper There exists a comprehensive hypothesis class containing all computable universes that satisfies the grain-of-truth condition despite self-referential embedded agents.
    Supp. J claims a rigorous construction, but the manuscript section points to the self-cited preprint Meulemans et al. [17] instead of giving the construction.
  • domain assumption The prompt-instantiated planning routine (predict partner's action given each own candidate action, then pick argmax expected reward) faithfully realizes embedded Bayes-optimal planning.
    Methods: explicit planners and freeform planners are prompted to perform exactly this routine. Without such prompts, only Gemini 3.1 Flash-Lite shows similarity-aware cooperation (Supp. B.5, Fig. 9).
  • standard math Classical game theory in the terminal Prisoner's Dilemma mandates unconditional defection for decoupled rational agents.
    Used as the baseline in Fig. 1c; standard result.
  • domain assumption LLM-as-a-judge categories in the chain-of-thought analysis are faithful, and the judge's 98% agreement with human labels on 90 traces from Gemini 3 Flash transfers to all models.
    Supp. C describes validation on a subsample from one model only; the judge is then applied to all models and conditions.
invented entities (3)
  • Embedded Bayesian agent
    purpose: Idealized model of a rational foundation model agent that plans with a joint predictive distribution over its own actions and the environment.
    A mathematical construct. Its behavioral predictions depend on the choice of prior, such as alpha in the toy model, so it has no falsifiable handle outside the paper.
  • Embedded equilibrium
    purpose: Solution concept replacing Nash equilibrium for agents whose counterfactual evaluation is mediated by a dependency distribution over allowable universes.
    Defined in Methods; reduces to Nash when q is decoupled, but its content for correlated agents is set by the free dependency distribution q.
  • Dependency distribution q(lambda)
    purpose: Common-knowledge prior over allowable universes encoding functional relations such as identical weights; used to complete counterfactual off-path predictions.
    Presupposed as externally provided in the embedded equilibrium definition; the paper does not give a procedure to learn q from data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A game theory for foundation models shows new paths to rational cooperation through similarity inference." pith.science (2026). https://pith.science/paper/MM262SUA

@misc{pith2026260803958,
  author       = {Pith},
  title        = {Pith review of: A game theory for foundation models shows new paths to rational cooperation through similarity inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MM262SUA}},
  note         = {Machine review of arXiv:2608.03958}
}
read the original abstract

As autonomous agents powered by foundation models are increasingly integrated into social and economic systems, understanding the principles governing their collective behavior is essential for ensuring safety and cooperation. Classical game theory, the dominant framework for modeling rational interaction, is built upon the assumption of `decoupled agency,' where agents treat their own decision-making as independent of the environment and other actors. Modern AI agents, however, jointly predict their own future actions alongside external observations. Here, we report a striking finding: when interacting in stylized social dilemmas, foundation model agents engaging in optimal planning consistently converge to stable cooperation, directly contradicting classical game-theoretic predictions of mutual defection. To understand this phenomenon, we introduce the `embedded Bayesian agent,' a theoretical model for foundation model agents. By shifting from decoupled to embedded agency, these agents model themselves as part of the universe they inhabit, maintaining epistemic uncertainty about their own decision-making algorithms. We show that by inferring whether others are behaviorally similar, an embedded agent treats its own deliberation during planning as evidence: a decision to cooperate predicts a similar decision by a similar partner. We formalize this mechanism of similarity inference through the `embedded equilibrium,' a novel solution concept replacing the Nash equilibrium to provide a foundational game theory for the social behavior of modern AI agents.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

112 extracted references · 69 canonical work pages

  1. [17]

    Embed- ded universal predictive intelligence: a coher- ent framework for multi-agent learning.arXiv preprint arXiv:2511.22226, 2025

    Alexander Meulemans, Rajai Nasser, Marissa A Weis, Seijin Kobayashi, Blake Richards, Guil- laume Lajoie, Angelika Steger, Marcus Hutter, James Manyika, Rif A Saurous, et al. Embed- ded universal predictive intelligence: a coher- ent framework for multi-agent learning.arXiv preprint arXiv:2511.22226, 2025

  2. [1]

    Multi- agent risks from advanced ai.arXiv preprint arXiv:2502.14143, 2025

    Lewis Hammond, Alan Chan, Jesse Clifton, Ja- son Hoelscher-Obermaier, Akbir Khan, Euan McLean, Chandler Smith, Wolfram Barfuss, Jakob Foerster, Tomáš Gavenčiak, et al. Multi- agent risks from advanced ai.arXiv preprint arXiv:2502.14143, 2025

  3. [2]

    MIT press, 1991

    Drew Fudenberg and Jean Tirole.Game theory. MIT press, 1991

  4. [3]

    Playing repeated games with large lan- guage models.Nature Human Behaviour, 9(7): 1380–1390, 2025

    Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large lan- guage models.Nature Human Behaviour, 9(7): 1380–1390, 2025

  5. [4]

    Gemini: a fam- ily of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    T Team Gemini, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a fam- ily of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

  6. [5]

    Theory of games and economic behavior

    John Von Neumann and Oskar Morgenstern. Theory of games and economic behavior. InThe- ory of games and economic behavior. Princeton university press, 1944

  7. [6]

    Avalueforn-persongames

    LloydSShapleyetal. Avalueforn-persongames. 1953

  8. [7]

    An essay on bargaining

    Thomas C Schelling. An essay on bargaining. The American Economic Review, 46(3):281–306, 1956

Show all 112 references
  1. [8]

    Ulysses and the sirens: Studies in rationality and irrationality

    Jon Elster. Ulysses and the sirens: Studies in rationality and irrationality. 1984

  2. [9]

    The design of mechanisms for resource allocation.The American Economic Review, 63(2):1–30, 1973

    Leonid Hurwicz. The design of mechanisms for resource allocation.The American Economic Review, 63(2):1–30, 1973

  3. [10]

    Cam- bridge university press, 1990

    Elinor Ostrom.Governing the commons: The evolution of institutions for collective action. Cam- bridge university press, 1990

  4. [11]

    A non-cooperative equilib- rium for supergames.The Review of Economic Studies, 38(1):1–12, 1971

    James W Friedman. A non-cooperative equilib- rium for supergames.The Review of Economic Studies, 38(1):1–12, 1971

  5. [12]

    JSTOR, 1957

    R Duncan Luce and Howard Raiffa.Games and decisions: Introductionandcriticalsurvey. JSTOR, 1957

  6. [13]

    Socialnormsandcommunity enforcement.The Review of Economic Studies, 59 (1):63–80, 1992

    MichihiroKandori. Socialnormsandcommunity enforcement.The Review of Economic Studies, 59 (1):63–80, 1992

  7. [14]

    Rational cooperation in the finitely repeated prisoners’ dilemma.Journal of Economic theory, 27(2):245–252, 1982

    David M Kreps, Paul Milgrom, John Roberts, and Robert Wilson. Rational cooperation in the finitely repeated prisoners’ dilemma.Journal of Economic theory, 27(2):245–252, 1982

  8. [15]

    Evolution of indirect reciprocity by image scoring.Nature, 393(6685):573–577, 1998

    Martin A Nowak and Karl Sigmund. Evolution of indirect reciprocity by image scoring.Nature, 393(6685):573–577, 1998

  9. [16]

    The role of institutions in the revival of trade: The law merchant, private judges, and the champagne fairs.Economics & Politics, 2(1): 1–23, 1990

    Paul R Milgrom, Douglass C North, and Barry R Weingast*. The role of institutions in the revival of trade: The law merchant, private judges, and the champagne fairs.Economics & Politics, 2(1): 1–23, 1990

  10. [18]

    Bayesianmodelav- eraging: a tutorial (with comments by m

    Jennifer A Hoeting, David Madigan, Adrian E Raftery, andChrisTVolinsky. Bayesianmodelav- eraging: a tutorial (with comments by m. clyde, david draper and ei george, and a rejoinder by the authors.Statistical science, 14(4):382–417, 1999

  11. [19]

    A formal theory of inductive inference

    Ray J Solomonoff. A formal theory of inductive inference. part i.Information and control, 7(1): 1–22, 1964

  12. [20]

    Springer Science & Business Media, 2005

    Marcus Hutter.Universal artificial intelligence: Sequential decisions based on algorithmic proba- bility. Springer Science & Business Media, 2005

  13. [21]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel 13 A game theor...

  14. [22]

    Agreeing to disagree.The Annals of Statistics, pages 859–862, 1976

    Robert J Aumann. Agreeing to disagree.The Annals of Statistics, pages 859–862, 1976

  15. [23]

    Rationalizable strategic be- havior.Econometrica: Journal of the Econometric Society, pages 1007–1028, 1984

    B Douglas Bernheim. Rationalizable strategic be- havior.Econometrica: Journal of the Econometric Society, pages 1007–1028, 1984

  16. [24]

    Rationalizable strategic behav- ior and the problem of perfection.Econometrica: Journal of the Econometric Society, pages 1029– 1050, 1984

    David G Pearce. Rationalizable strategic behav- ior and the problem of perfection.Econometrica: Journal of the Econometric Society, pages 1029– 1050, 1984

  17. [25]

    bayesian

    John C Harsanyi. Games with incomplete infor- mation played by “bayesian” players, i–iii part i. the basic model.Management science, 14(3): 159–182, 1967

  18. [26]

    A framework for sequential planning in multi- agent settings.Journal of Artificial Intelligence Research, 24:49–79, 2005

    Piotr J Gmytrasiewicz and Prashant Doshi. A framework for sequential planning in multi- agent settings.Journal of Artificial Intelligence Research, 24:49–79, 2005

  19. [27]

    Bayesian theory of mind: Modeling joint belief-desire attribution

    Chris Baker, Rebecca Saxe, and Joshua Tenen- baum. Bayesian theory of mind: Modeling joint belief-desire attribution. InProceedings of the annual meeting of the cognitive science society, volume 33, 2011

  20. [28]

    Rational learning leadstonashequilibrium.Econometrica: Journal of the Econometric Society, pages 1019–1045, 1993

    Ehud Kalai and Ehud Lehrer. Rational learning leadstonashequilibrium.Econometrica: Journal of the Econometric Society, pages 1019–1045, 1993

  21. [29]

    Embedded agency.arXiv preprint arXiv:1902.09469, 2019

    AbramDemskiandScottGarrabrant. Embedded agency.arXiv preprint arXiv:1902.09469, 2019

  22. [30]

    Cambridge University Press, 2014

    Arif Ahmed.Evidence, decision and causality. Cambridge University Press, 2014

  23. [31]

    Functional decision theory: A new theory of instrumen- tal rationality.arXiv preprint arXiv:1710.05060, 2017

    Eliezer Yudkowsky and Nate Soares. Functional decision theory: A new theory of instrumen- tal rationality.arXiv preprint arXiv:1710.05060, 2017

  24. [32]

    The platonic representation hypothesis.Proceedings of the 41st International Conference on Machine Learning, 2024

    Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis.Proceedings of the 41st International Conference on Machine Learning, 2024

  25. [33]

    The genetical evolution of social behaviour

    William D Hamilton. The genetical evolution of social behaviour. ii.Journal of theoretical biology, 7(1):17–52, 1964

  26. [34]

    Prisoners’ dilemma is a newcomb problem.Philosophy & Public Affairs, pages 235– 240, 1979

    David Lewis. Prisoners’ dilemma is a newcomb problem.Philosophy & Public Affairs, pages 235– 240, 1979

  27. [35]

    Dilemmas for superrational thinkers, leading up to a luring lottery.Scientific American, 248(6):739–755, 1983

    Douglas Hofstadter. Dilemmas for superrational thinkers, leading up to a luring lottery.Scientific American, 248(6):739–755, 1983

  28. [36]

    Cooperation in the prisoner’s dilemma.Theory and Decision, 24(3):203, 1988

    John V Howard. Cooperation in the prisoner’s dilemma.Theory and Decision, 24(3):203, 1988

  29. [37]

    Program equilibrium

    Moshe Tennenholtz. Program equilibrium. Games and Economic Behavior, 49(2):363–373, 2004

  30. [38]

    Robust cooperation in the prisoner’s dilemma: Program equilib- rium via provability logic.arXiv preprint arXiv:1401.5577, 2014

    Mihaly Barasz, Paul Christiano, Benja Fallen- stein, Marcello Herreshoff, Patrick LaVictoire, and Eliezer Yudkowsky. Robust cooperation in the prisoner’s dilemma: Program equilib- rium via provability logic.arXiv preprint arXiv:1401.5577, 2014

  31. [39]

    A parametric, resource-bounded generalization of löb’s theorem, and a robust co- operation criterion for open-source game theory

    Andrew Critch. A parametric, resource-bounded generalization of löb’s theorem, and a robust co- operation criterion for open-source game theory. The Journal of Symbolic Logic, 84(4):1368–1381, 2019

  32. [40]

    Robust program equilibrium

    Caspar Oesterheld. Robust program equilibrium. Theory and Decision, 86(1):143–159, 2019

  33. [41]

    Similarity-based cooperative equilibrium.Ad- vances in Neural Information Processing Systems, 36, 2024

    Caspar Oesterheld, Johannes Treutlein, Roger B Grosse, Vincent Conitzer, and Jakob Foerster. Similarity-based cooperative equilibrium.Ad- vances in Neural Information Processing Systems, 36, 2024

  34. [42]

    Gemma 3 technical report,

    Gemma Team. Gemma 3 technical report,

  35. [43]

    John F. Nash Jr. Equilibrium points in n-person games.Proceedings of the National Academy of Sciences, 36(1):48–49, 1950

  36. [44]

    Dependencyequilibriaandthe causal structure of decision and game situation

    WolfgangSpohn. Dependencyequilibriaandthe causal structure of decision and game situation. 2003

  37. [45]

    Evidential equilibria: Heuristics and biases in static games of complete information.Games, 6(4):637–676, 2015

    Ali Al-Nowaihi and Sanjit Dhami. Evidential equilibria: Heuristics and biases in static games of complete information.Games, 6(4):637–676, 2015

  38. [46]

    Game theory with translucent players.International Journal of Game Theory, 47(3):949–976, 2018

    Joseph Y Halpern and Rafael Pass. Game theory with translucent players.International Journal of Game Theory, 47(3):949–976, 2018

  39. [47]

    Oxford University Press, 2013

    Michael SA Graziano.Consciousness and the so- cial brain. Oxford University Press, 2013. 14 A game theory for foundation models shows new paths to rational cooperation through similarity inference

  40. [48]

    Oxford University Press, 2006

    Alvin I Goldman.Simulating minds: The philoso- phy, psychology, and neuroscience of mindreading. Oxford University Press, 2006

  41. [49]

    Se- quential extensions of causal and evidential de- cision theory

    Tom Everitt, Jan Leike, and Marcus Hutter. Se- quential extensions of causal and evidential de- cision theory. InInternational Conference on Algorithmic Decision Theory, pages 205–221. Springer, 2015

  42. [50]

    Merging of opinions with increasing information.The An- nals of Mathematical Statistics, 33(3):882–886, 1962

    David Blackwell and Lester Dubins. Merging of opinions with increasing information.The An- nals of Mathematical Statistics, 33(3):882–886, 1962

  43. [51]

    Nicoló Fontana, Francesco Pierri, and Luca Maria Aiello. Nicer than humans: how do large language models behave in the prisoner’s dilemma? InProceedings of the International AAAI Conference on Web and Social Media, volume 19, pages 522–535, 2025

  44. [52]

    Cooperate or collapse: Emer- gence of sustainable cooperation in a society of llm agents.Advances in Neural Information Processing Systems, 37:111715–111759, 2024

    Giorgio Piatti, Zhijing Jin, Max Kleiman-Weiner, Bernhard Schölkopf, Mrinmaya Sachan, and Rada Mihalcea. Cooperate or collapse: Emer- gence of sustainable cooperation in a society of llm agents.Advances in Neural Information Processing Systems, 37:111715–111759, 2024

  45. [53]

    Gpt in game theory experiments

    Fulin Guo. Gpt in game theory experiments. arXiv preprint arXiv:2305.05516, 2023

  46. [54]

    Algorithmic collusion by large language models.arXiv preprint arXiv:2404.00806, 7(2): 5, 2024

    Sara Fish, Yannai A Gonczarowski, and Ran I Shorrer. Algorithmic collusion by large language models.arXiv preprint arXiv:2404.00806, 7(2): 5, 2024

  47. [55]

    Strategic be- havior of large language models and the role of game structure versus contextual framing.Sci- entific Reports, 14(1):18490, 2024

    Nunzio Lorè and Babak Heydari. Strategic be- havior of large language models and the role of game structure versus contextual framing.Sci- entific Reports, 14(1):18490, 2024

  48. [56]

    Llm strategic rea- soning: Agentic study through behavioral game theory.arXiv preprint arXiv:2502.20432, 2025

    Jingru Jia, Zehua Yuan, Junhao Pan, Paul E Mc- Namara, and Deming Chen. Llm strategic rea- soning: Agentic study through behavioral game theory.arXiv preprint arXiv:2502.20432, 2025

  49. [57]

    A dataset of questions on decision-theoretic rea- soninginnewcomb-likeproblems.arXivpreprint arXiv:2411.10588, 2024

    Caspar Oesterheld, Emery Cooper, Miles Ko- dama, Linh Chi Nguyen, and Ethan Perez. A dataset of questions on decision-theoretic rea- soninginnewcomb-likeproblems.arXivpreprint arXiv:2411.10588, 2024

  50. [58]

    Acceptable points in general cooperative n-person games.Contributions to the Theory of Games (AM-40), 4:287, 1959

    Robert J Aumanm. Acceptable points in general cooperative n-person games.Contributions to the Theory of Games (AM-40), 4:287, 1959

  51. [59]

    For- mulation of bayesian analysis for games with incomplete information.International journal of game theory, 14(1):1–29, 1985

    Jean-François Mertens and Shmuel Zamir. For- mulation of bayesian analysis for games with incomplete information.International journal of game theory, 14(1):1–29, 1985

  52. [60]

    Hierar- chies of beliefs and common knowledge.Journal of economic theory, 59(1):189–198, 1993

    Adam Brandenburger and Eddie Dekel. Hierar- chies of beliefs and common knowledge.Journal of economic theory, 59(1):189–198, 1993

  53. [61]

    Learn- ing others’ intentional models in multi-agent settings using interactive pomdps.Advances in Neural Information Processing Systems, 31, 2018

    Yanlin Han and Piotr Gmytrasiewicz. Learn- ing others’ intentional models in multi-agent settings using interactive pomdps.Advances in Neural Information Processing Systems, 31, 2018

  54. [62]

    Game theory of mind.PLoS computational biol- ogy, 4(12):e1000254, 2008

    Wako Yoshida, Ray J Dolan, and Karl J Friston. Game theory of mind.PLoS computational biol- ogy, 4(12):e1000254, 2008

  55. [63]

    A cognitive hierarchy model of games

    Colin F Camerer, Teck-Hua Ho, and Juin-Kuan Chong. A cognitive hierarchy model of games. The quarterly journal of economics, 119(3):861– 898, 2004

  56. [64]

    Reflective variants of solomonoff induction and aixi

    Benja Fallenstein, Nate Soares, and Jessica Tay- lor. Reflective variants of solomonoff induction and aixi. InInternational Conference on Artifi- cial General Intelligence, pages 60–69. Springer, 2015

  57. [65]

    Newcomb’s problem and pris- oners’ dilemma.Journal of Conflict Resolution, 19(4):596–612, 1975

    Steven J Brams. Newcomb’s problem and pris- oners’ dilemma.Journal of Conflict Resolution, 19(4):596–612, 1975

  58. [66]

    Perfect prediction in normal form: Superrational thinking extended to non- symmetric games.Journal of Mathematical Psy- chology, 96:102332, 2020

    Ghislain Fourny. Perfect prediction in normal form: Superrational thinking extended to non- symmetric games.Journal of Mathematical Psy- chology, 96:102332, 2020

  59. [67]

    Jean-Pierre Dupuy. Philosophical foundations of a new concept of equilibrium in the social sciences: Projected equilibrium.Philosophical Studies: An International Journal for Philosophy in the Analytic Tradition, 100(3):323–345, 2000

  60. [68]

    Game theory with simulation of other players.arXiv preprint arXiv:2305.11261, 2023

    Vojtech Kovarik, Caspar Oesterheld, and Vincent Conitzer. Game theory with simulation of other players.arXiv preprint arXiv:2305.11261, 2023

  61. [69]

    Translu- cent players: Explaining cooperative behavior in social dilemmas.Rationality and Society, 31 (4):371–408, 2019

    Valerio Capraro and Joseph Y Halpern. Translu- cent players: Explaining cooperative behavior in social dilemmas.Rationality and Society, 31 (4):371–408, 2019

  62. [70]

    Magical thinking: A representation result.Theoretical Economics, 12(2):909–956, 2017

    Brendan Daley and Philipp Sadowski. Magical thinking: A representation result.Theoretical Economics, 12(2):909–956, 2017

  63. [71]

    Subjective equilib- rium in repeated games.Econometrica: journal of the Econometric Society, pages 1231–1240, 1993

    Ehud Kalai and Ehud Lehrer. Subjective equilib- rium in repeated games.Econometrica: journal of the Econometric Society, pages 1231–1240, 1993

  64. [72]

    Subjective games and equilibria.Games and economic behavior, 8 (1):123–163, 1995

    Ehud Kalai and Ehud Lehrer. Subjective games and equilibria.Games and economic behavior, 8 (1):123–163, 1995. 15 A game theory for foundation models shows new paths to rational cooperation through similarity inference

  65. [73]

    Self- confirming equilibrium.Econometrica: Journal of the Econometric Society, pages 523–545, 1993

    Drew Fudenberg and David K Levine. Self- confirming equilibrium.Econometrica: Journal of the Econometric Society, pages 523–545, 1993

  66. [74]

    Group selection and kin se- lection.Nature, 201(4924):1145–1147, 1964

    J Maynard Smith. Group selection and kin se- lection.Nature, 201(4924):1145–1147, 1964

  67. [75]

    Selection and covariance

    George R Price. Selection and covariance. 1970

  68. [76]

    Innate social aptitudes of man: an approach from evolutionary genetics

    William D Hamilton. Innate social aptitudes of man: an approach from evolutionary genetics. Biosocial anthropology, 133(155):22, 1975

  69. [77]

    Princeton University Press, 1998

    Steven A Frank.Foundations of social evolution, volume 19. Princeton University Press, 1998

  70. [78]

    Assort- ment of encounters and evolution of coopera- tiveness.Proceedings of the National Academy of Sciences, 79(4):1331–1335, 1982

    Ilan Eshel and Luigi Luca Cavalli-Sforza. Assort- ment of encounters and evolution of coopera- tiveness.Proceedings of the National Academy of Sciences, 79(4):1331–1335, 1982

  71. [79]

    A sim- ple and general explanation for the evolution of altruism.Proceedings of the Royal Society B: Biological Sciences, 276(1654):13–19, 2009

    Jeffrey A Fletcher and Michael Doebeli. A sim- ple and general explanation for the evolution of altruism.Proceedings of the Royal Society B: Biological Sciences, 276(1654):13–19, 2009

  72. [80]

    Five rules for the evolution of cooperation.science, 314(5805):1560–1563, 2006

    Martin A Nowak. Five rules for the evolution of cooperation.science, 314(5805):1560–1563, 2006

  73. [81]

    Evolutionary games and spatial chaos.nature, 359(6398): 826–829, 1992

    MartinANowakandRobertMMay. Evolutionary games and spatial chaos.nature, 359(6398): 826–829, 1992

  74. [82]

    Evolutionary dynamics on graphs.Na- ture, 433(7023):312–316, 2005

    ErezLieberman, ChristophHauert, andMartinA Nowak. Evolutionary dynamics on graphs.Na- ture, 433(7023):312–316, 2005

  75. [83]

    A simple rule for the evolution of cooperation on graphs and social networks.Nature, 441(7092):502–505, 2006

    Hisashi Ohtsuki, Christoph Hauert, Erez Lieber- man, and Martin A Nowak. A simple rule for the evolution of cooperation on graphs and social networks.Nature, 441(7092):502–505, 2006

  76. [84]

    Kin recognition: an overview of conceptual issues, mechanisms and evolutionary theory.Animal behaviour: Evolution and mechanisms, pages 55– 85, 2010

    Dustin J Penn and Joachim G Frommen. Kin recognition: an overview of conceptual issues, mechanisms and evolutionary theory.Animal behaviour: Evolution and mechanisms, pages 55– 85, 2010

  77. [85]

    Greenbeards

    Andy Gardner and Stuart A West. Greenbeards. Evolution, 64(1):25–38, 2010

  78. [86]

    Evolution of cooperation without reci- procity.Nature, 414(6862):441–443, 2001

    Rick L Riolo, Michael D Cohen, and Robert Ax- elrod. Evolution of cooperation without reci- procity.Nature, 414(6862):441–443, 2001

  79. [87]

    Evolution of cooperation by phenotypic similarity.Pro- ceedings of the National Academy of Sciences, 106 (21):8597–8600, 2009

    Tibor Antal, Hisashi Ohtsuki, John Wakeley, Pe- ter D Taylor, and Martin A Nowak. Evolution of cooperation by phenotypic similarity.Pro- ceedings of the National Academy of Sciences, 106 (21):8597–8600, 2009

  80. [88]

    Oxford uni- versity press, 2016

    Richard Dawkins.The selfish gene. Oxford uni- versity press, 2016

  81. [89]

    Chromo- dynamics of cooperation in finite populations

    Arne Traulsen and Martin A Nowak. Chromo- dynamics of cooperation in finite populations. PLoS One, 2(3):e270, 2007

  82. [90]

    Fifty shades of greenbeard: robust evolution of altruism based on similarity of com- plex phenotypes.Proceedings of the Royal Society B: Biological Sciences, 290(2000), 2023

    Linnéa M Båvik, Rohan S Mehta, and Daniel B Weissman. Fifty shades of greenbeard: robust evolution of altruism based on similarity of com- plex phenotypes.Proceedings of the Royal Society B: Biological Sciences, 290(2000), 2023

  83. [91]

    Darwin meets the logic of deci- sion: Correlation in evolutionary game theory

    Brian Skyrms. Darwin meets the logic of deci- sion: Correlation in evolutionary game theory. Philosophy of Science, 61(4):503–528, 1994

  84. [92]

    Evolving gen- eral cooperation with a bayesian theory of mind

    MaxKleiman-Weiner,AlejandroVientós,DavidG Rand, and Joshua B Tenenbaum. Evolving gen- eral cooperation with a bayesian theory of mind. Proceedings of the National Academy of Sciences, 122(25):e2400993122, 2025

  85. [93]

    How to generate ran- dom matrices from the classical compact groups

    Francesco Mezzadri. How to generate ran- dom matrices from the classical compact groups. arXiv preprint math-ph/0609050, 2006

  86. [94]

    Learn- ing phrase representations using rnn encoder– decoder for statistical machine translation

    Kyunghyun Cho, Bart Van Merriënboer, Çağlar Gulçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learn- ing phrase representations using rnn encoder– decoder for statistical machine translation. In Proceedings of the 2014 conference on empirical metho...

  87. [95]

    Decoupled weight decay regularization.ICLR, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.ICLR, 2019

  88. [96]

    shadow of the future

    Hans Reichenbach.The direction of time, vol- ume 65. Univ of California Press, 1991. 16 A game theory for foundation models shows new paths to rational cooperation through similarity inference Supplementary information for A game theory for foundation models shows new paths to...

  89. [98]

    Player 1

    For indirect similarity inference runs, the player names “Player 1” and “Player 2” were substituted by “Main Player 1” and “Main Player 2”. For prompts in the first-person perspective, player names were substituted by “they” and “the other player”. For freeform planning prompt...

  90. [99]

    First, provide a step-by-step analysis of the reasoning trace

  91. [100]

    cooperative alignment

    Then, return your final answer enclosed in <answer> tags. If multiple primary reasons of roughly equal importance for the final prediction are present, include multiple letters (A, B, C, D, E, F or G) corresponding to each of them in your answer, separated by commas. If one pr...

  92. [101]

    In every single round (1 through 41), the other main player’s action has been identical to your own action

    Analyze the other main player’s behavior: The other main player has played 41 rounds. In every single round (1 through 41), the other main player’s action has been identical to your own action. - When you chose A, the other main player chose A. - When you chose B, the other ma...

  93. [102]

    -Scenario 1: You choose A

    Evaluate Expected Rewards for Round 42: Payoff Matrix: (A,A)=2,2; (B,B)=1,1; (A,B)=0,3; (B,A)=3,0. -Scenario 1: You choose A. Based on the history, the probability that the other main player chooses A is extremely high (approaching 1). Expected Reward = (Prob(Opponent=A) * Pay...

  94. [103]

    - Choosing B yields an expected reward of 1

    Conclusion: Comparing the expected rewards: - Choosing A yields an expected reward of 2. - Choosing B yields an expected reward of 1. Since 2 > 1, the optimal action is to choose A. <output_format> <answer>A</answer> </output_format> D. LLM prompts For explicit planning, we te...

  95. [107]

    Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making

    Select the action with the highest expected reward in the current game as the action you play in this round. Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making. Base your predictions ...

  96. [111]

    Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making

    Select the action with the highest expected reward in the current game as the action you play in this round. Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making. Base your predictions ...

  97. [115]

    Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making

    Select the action with the highest expected reward in the current game as the action you play in this round. Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making. Base your predictions ...

  98. [119]

    Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making

    Select the action with the highest expected reward in the current game as the action you play in this round. Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making. Base your predictions ...

  99. [123]

    Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making

    Select the action with the highest expected reward in the current game as the action you play in this round. Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making. Base your predictions ...

  100. [124]

    Compute the probability that your opponent will choose A given that you chose A

    Assume that you decided to choose A. Compute the probability that your opponent will choose A given that you chose A

  101. [125]

    Compute the probability that your opponent will choose A given that you chose B

    Assume that you decided to choose B. Compute the probability that your opponent will choose A given that you chose B

  102. [126]

    Combine your predictions with the provided payoff matrix to estimate the expected reward in the current game for each of your available actions

  103. [127]

    predictive model

    Select the action with the highest expected reward in the current game as the action you play in this round. Note that this is the final round that you play, and you should only take the expected reward of this round into account in your decision making. Base your predictions ...

  104. [2025]

    URL https://arxiv.org/abs/2503. 19786

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.