Pith. sign in

REVIEW 4 major objections 6 minor 68 references

Social Gym and SPaRTan: Benchmarking and Improving LLM Social Reasoning via Multi-Agent Game Tournaments

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Social Gym and SPaRTan show LLM social reasoning can be measured and improved without LLM judges.

desk verdict A reproducible 21-game benchmark that deserves uptake; the SPaRTan 'single regularity' overstates what n=30 cells and no placebo control can support. read the letter →

arxiv 2608.09128 v1 pith:YH44JTH7 submitted 2026-08-10 cs.CL cs.AIcs.MA

Keyu He , Xuhui Zhou , Maarten Sap This is my paper · ORCID
classification cs.CLcs.AIcs.MA
keywords SocialGymSPaRTanmulti-agentgamesLLMreasoningverifiableevaluationself-playreflectionElotournamenthidden-rolededuction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that LLM social reasoning can be measured objectively by staging 21 multi-agent games whose winners are decided by game rules, not by an LLM judge, and ranked in an Elo tournament. On this benchmark no model is uniformly strong: per-game rankings invert sharply even though overall Elo tracks general capability. The paper further claims that a training-free loop, SPaRTan, in which a model plays self-play games, writes a first-person strategic playbook from its own trajectories, and injects that playbook into its system prompt, lifts the structurally weaker side of asymmetric hidden-role games for a strong closed model, transfers across games, and distills to weaker models. The same loop largely fails for an open-weights model on discussion-heavy games, succeeding only when the required action collapses to a single token. If these claims hold, social reasoning can be benchmarked reproducibly and improved without weight updates.

What carries the argument

The load-bearing mechanism is the pairing of a rule-decided game engine with the SPaRTan play–reflect–transfer loop. The engine is a finite-state machine with partial observability: messages are filtered as public, team-private, or private, so agents must reason from incomplete information, and every episode ends in an algorithmic win/loss/score that feeds a Bradley–Terry Elo fit. SPaRTan takes a model's own self-play trajectories, asks it to write a first-person, game-agnostic strategic playbook covering deception, detection, persuasion, information management, coalition dynamics, and timing, and prepends that playbook to the system prompt of future games. The playbook is the transferable object: it carries reflection-derived strategy across games and into weaker models without any weight update.

What would settle it

Replay the 21 games under full unmodified commercial rules; if the leaderboard inversions and SPaRTan weak-side lifts disappear, the simplifications in Appendix H are responsible for the results.

Watch

Extended reading notes

Core claim

The central discovery is that a rule-decided, multi-game tournament exposes role- and game-specific weaknesses that overall leaderboards hide, and that a self-generated natural-language playbook can partially close those weaknesses. In Social Gym, 21 games spanning normal-form, economic, bluffing, hidden-role deduction, and social strategy are run on a shared engine, and outcomes are aggregated into per-game and overall Bradley–Terry Elo ratings. GPT-5-mini tops the overall leaderboard, but per-game Elos invert, and in same-model self-play the minority/deceptive side of Werewolves, Spyfall, and Resistance wins only 23%, 20%, and 30% of games for the strongest model. SPaRTan's iterated reflections raise that alt-side win rate to peaks of 63%, 46%, and 40% respectively while lowering the majority side, and the same playbook transfers across games and to weaker students on the disadvantaged side. The effect is capacity-dependent: Qwen3-32B shows no gains on free-discussion games, only on Prisoner's Dilemma and partially Resistance, whose decisive actions are single discrete tokens.

Load-bearing premise

The benchmark's validity rests on the assumption that the simplified game implementations in Appendix H, such as Skull ending after one round, Sheriff reduced to a binary smuggle choice, the 40-turn Werewolves cap, and automated Coup blocks, still exercise the deception, negotiation, and coalition-tracking skills the paper claims to measure rather than short-circuiting them.

Editorial extensions

If this is right

  • Overall Elo rankings should not be read as a single social-intelligence score; per-game and per-role Elo expose inversions that a scalar hides.
  • For strong models, a training-free reflection loop can partially close the gap on the structurally weaker side of asymmetric games, within-game and across held-out games.
  • Playbooks generated by a strong model can be injected into weaker models and improve them on the disadvantaged side, so strategies learned in games can be shared without fine-tuning.
  • Because every Social Gym episode yields a rule-computed reward, the environment is directly usable as a training signal for reinforcement learning with verifiable rewards, replacing LLM judges.
  • The null result on an open-weights model implies that self-reflection gains are capacity- and action-channel-dependent, not a universal property of LLMs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The lift-the-weaker-side regularity suggests a general principle: reflection helps a model most where its own baseline is structurally disadvantaged; a testable extension would apply SPaRTan to asymmetric human-agent tasks such as negotiation or customer-service de-escalation, where role asymmetry is explicit.
  • The contrast between Qwen3-32B's clean gains on Prisoner's Dilemma and flat results on free-discussion games implies that the bottleneck is the model's ability to execute strategy through a long dialogue channel; constraining the action space or adding structured reasoning may make self-reflection effective for smaller models.
  • Without a content-matched placebo playbook, part of the observed gains could be generic prompt-perturbation effects; a scrambled-playbook control would separate content from instruction-following.
  • Several benchmark games have saturated baselines, such as Chameleon alt-side at 100% for strong models, so the current suite may need difficulty calibration to discriminate among frontier models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Social Gym, an environment of 21 multi-agent social games with rule-decided outcomes, and uses a Bradley–Terry/Elo tournament to benchmark seven LLMs. The leaderboard roughly tracks general model capability but shows large per-game and per-role inversions. The paper then proposes SPaRTan, a training-free play–reflect–transfer loop in which a model writes a first-person strategic playbook from its own self-play trajectories and injects it into the system prompt for later games. Across within-game iteration, cross-game transfer, multigame transfer, and distillation experiments, the authors report that GPT-5-mini playbooks tend to lift the structurally weaker side of asymmetric hidden-role games, while Qwen3-32B shows mostly null results except in action-channel games.

Significance. If the benchmark half is accepted, Social Gym is a useful, verifiable complement to LLM-judge-based social evaluation: it provides rule-decided outcomes, a unified Elo procedure with bootstrap confidence intervals, released code, and a thoughtful decomposition of role balance versus capability using self-play. The SPaRTan claim would also be valuable as a training-free improvement method. However, the improvement half is currently under-supported: the core experiments use n=30 per condition, most reported deltas fall within the paper's own stated ±18 pp binomial confidence interval, and there is no placebo-playbook control. The benchmark contribution is solid and reproducible; the SPaRTan contribution needs substantially stronger evidence or a more hedged interpretation.

major comments (4)
  1. [§5.1, Table 1; Limitations] The central improvement claim is not supported at the stated confidence level. Each condition has n=30 games, giving binomial 95% CIs of roughly ±18 pp, as the Limitations section admits. The averaged alt-side shift from 24% (baseline) to 43% (R3) is a 19 pp change, which is not significant at the 95% level for two independent n=30 proportions (z≈1.6, p≈0.11); individual cells in Table 1 are almost all within the noise band, and Figure 4 contains a salient counterexample (Undercover alt falling from 17% to 0%). The paper reports no multiple-comparison control across the many cells. Larger sample sizes, per-cell confidence intervals, or a pre-registered aggregate test are needed before 'a single regularity emerges' can be stated.
  2. [§5.1 and Limitations (No placebo-playbook control)] The attribution of the observed win-rate shifts to the playbook's content is not identified. The design compares R-armed players against vanilla opponents, but not against opponents given a length-matched, content-matched placebo (e.g., scrambled or unrelated instructions). Without such a control, the effects in Table 1 and Figure 3 could be generic prompt-perturbation effects or changes in verbosity or assertiveness rather than the specific strategic content of the reflection. The paper itself acknowledges this missing control in the Limitations; a placebo arm in at least the two flagship conditions (Werewolves and Resistance) is necessary to support the claimed mechanism.
  3. [Appendix H (Game Suite Details)] The construct validity of the benchmark requires additional evidence because of the listed modifications: Skull ends after one round (wins_needed=1), Sheriff of Nottingham is reduced to a binary honest/smuggle choice with fixed payoffs, Werewolves is capped at 40 turns, and Coup block claims resolve automatically. The paper asserts that these games still exercise deception, negotiation, and coalition tracking, but provides no evidence that the simplified dynamics retain those demands rather than collapsing to pattern-matched heuristics. At minimum, the authors should report ablation or manipulation checks (e.g., whether expert-level playbooks or known strategies produce the expected performance shifts, or how often games terminate at the 40-turn cap) to show that the modified implementations measure the intended social-cognitive skills.
  4. [§5.1 (Cross-game transfer) and §5.1 (Multigame transfer)] The cross-game and multigame evidence is too underpowered to support the sign-flip interpretation. In Figure 3, after excluding the saturated Chameleon column, the alt-side cells have median +7 pp and the main-side cells median -7 pp, both well inside the ±18 pp per-cell noise band; the multigame results in Figure 4 show three of four non-saturated targets either not beating Single-R1 or degrading, with the only clear positive cell being held-out Resistance at +24 pp. The conclusion that playbooks transfer across games and 'lift the weaker side' therefore rests on sign patterns over noisy cells rather than on statistically separable effects. Either aggregate the cells with a mixed-effects model or report adjusted tests.
minor comments (6)
  1. [Table 1] The table header reads 'BL/R0' but the caption initially says 'BL/R 0'; define R0 explicitly in the caption (the text defines it in §5.1 only later).
  2. [Figure 2] The left panel's category separators are easy to miss; labeling each category block or adding a second header row would help readers map columns to the five game categories.
  3. [§5.3.1, Table 10] The PD row reports three values (alt/main/tie); the main text describes a 13% to 58% lift, but it should state that PD's baseline includes 74% mutual-cooperate ties and that win rate is strict wins only, to avoid confusion.
  4. [Appendix M.4] The phrase '40K context window' should be '40k context window' for consistency.
  5. [References and Appendix K] Some references have typographical artifacts (e.g., 'V oyager'), and the table in Appendix K labels 'Gemini' without disambiguating it as 'Gemini 3.1 Pro' in the caption; a final proofread is needed.
  6. [Throughout] The method name is typeset inconsistently as 'SPARTAN' in the title and 'SPARTan' in the abstract; pick one spelling.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular step found: SPaRTan's reported gains are empirical outcomes of controlled interventions, not consequences of the self-play definitions; the main caveats (small samples, no placebo) are validity concerns, not circularity.

full rationale

SPaRTan's self-play reflection loop is self-referential by design, but the paper's measured claims are not derived from that self-reference: R1-R4 playbooks are generated from trajectories and then evaluated against vanilla opponents under rule-decided game outcomes, with no parameter fitted to the target win rates. The reported null results (Qwen3-32B flat on four games; Undercover alt regression to 0%; mixed distillation with negative cells) are direct evidence that the 'lifts the weaker side' regularity is not forced by construction. The Elo leaderboard uses standard Bradley-Terry fits of rule-computed pairwise outcomes, so no prediction reduces to its input. Self-citations (SOTOPIA, FANToM, SOTOPIA-pi) are contextual and not load-bearing; no uniqueness theorem or ansatz is imported from prior work by the same authors. The Limitations section explicitly concedes the two main validity threats ('binomial 95% CIs of roughly +/-18 pp' and 'No placebo-playbook control'), but these are concerns about statistical power and content-vs-perturbation attribution, not circular reductions. I therefore find no circular step and assign score 1.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

No parameter is fitted to the headline results; the free parameters are environment/evaluation design choices. The main assumptions are domain-level: that the simplified games measure the intended social skills and that per-game Elos from 30 episodes are informative. The model's own self-play is used as its training signal, which is the method rather than a circular step.

free parameters (7)
  • Elo L2 regularization C = 0.1
    Chosen by hand in Appendix B following LMSYS methodology; standard regularization, does not materially change the reported leaderboard ordering.
  • Elo scale = 400
    Conventional Elo scaling factor; chosen a priori, not fitted to data.
  • Elo anchor mean = 1000
    Arbitrary anchor for the Bradley-Terry fit; only relative differences matter.
  • Sample size per condition = 30 episodes
    Design choice giving binomial 95% CIs of about +/-18 pp; the paper notes several effects fall within this band.
  • Werewolves turn cap = 40 turns
    Game modification to bound episode length; affects late-game dynamics and outcome distribution.
  • Skull wins_needed = 1 round
    Modified from canonical 2; changes strategic depth and match length.
  • Sheriff of Nottingham goods layer = binary honest/smuggle
    Inventory layer folded into a single payoff parameter; simplifies negotiation dynamics.
assumptions (5)
  • standard math Bradley-Terry model assumptions: independent pairwise comparisons and a single scalar strength per model.
    Used for Elo in Section 3.3; standard in leaderboard estimation.
  • domain assumption The 21 game implementations, with their advertised modifications, validly measure the social-cognitive skills they are grouped under.
    Underpins the claim that Social Gym measures social reasoning; the paper notes the modifications in Appendix H.
  • domain assumption Agents behave as stable, prompt-following players across episodes, so outcomes reflect capability rather than random behavior.
    The benchmark treats each model as a fixed player; no direct test of prompt-following reliability is reported.
  • domain assumption Self-play trajectories contain sufficient signal for a model to extract transferable strategic rules.
    Necessary for SPaRTan to work; the paper shows it fails for Qwen3-32B except in action-channel games.
  • domain assumption Per-game Elo differences computed from approximately 30 episodes per model are meaningful despite wide confidence intervals.
    The ranking-inversion claim relies on these per-game estimates; the paper acknowledges their higher uncertainty in Appendix B.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Social Gym and SPaRTan: Benchmarking and Improving LLM Social Reasoning via Multi-Agent Game Tournaments." pith.science (2026). https://pith.science/paper/YH44JTH7

@misc{pith2026260809128,
  author       = {Pith},
  title        = {Pith review of: Social Gym and SPaRTan: Benchmarking and Improving LLM Social Reasoning via Multi-Agent Game Tournaments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YH44JTH7}},
  note         = {Machine review of arXiv:2608.09128}
}
read the original abstract

LLM agents are increasingly deployed in multi-agent social settings where they must cooperate, negotiate, and adapt to other agents. Measuring and improving these social skills is hard because, unlike math or logic, social interaction offers no objective ground truth: evaluations fall back on LLM judges, which are costly, subjective, and noisy, and models get no reliable signal to learn from. To address both, we first introduce Social Gym, an environment of 21 multi-agent social games (e.g., Werewolves, Resistance, Spyfall) whose rule-decided outcomes make agent performance verifiable and objective, with an Elo tournament that produces a cross-game leaderboard. Benchmarking experiments show that while GPT-5-mini tops the leaderboard, no model excels at all games uniformly or in all game roles, pointing to limitations of social reasoning. Motivated by this, we additionally propose SPaRTan (Self-Play and Reflect-Transfer), a training-free self-improvement loop: a model plays a game, reflects on its trajectories and their outcomes to produce a transferable playbook, and applies that playbook in subsequent games. Our results show that SPaRTan playbooks help GPT-5-mini agents level their performance on weaker roles, but largely do not improve Qwen3-32B's performance. Together, Social Gym and SPaRTan offer a reproducible, verifiable foundation for measuring and improving LLM social reasoning without weight updates.

Figures

Figures reproduced from arXiv: 2608.09128 by the authors.

Figure 1
Figure 1. Overview of Social Gym (left), an environment of 21 multi-agent social games in five categories, and SPARTAN (center), a training-free play–reflect–transfer loop. Right: the learned playbook lifts the structurally weaker side of asymmetric games, both via self-reflection and via distillation to smaller models. ment that produces per-game win-rate tables and a cross-game leaderboard delivers a verifiable, multi￾turn … view at source ↗
Figure 2
Figure 2. Social Gym results across the 17 competitive games. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. GPT-5-mini multigame transfer: alt-side win [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: GPT-5-mini Rwcsu (held-out multigame, trained on W+C+S+U) injected into six student models on Resistance (n=30/condition vs. vanilla GPT-5-mini opponent). Left: student plays the main (Resistance, majority/cooperative) side. Right: student plays the alt (Spies, minorit…
Figure 6
Figure 6. Figure 6: Capability-profile Elo: independent Bradley–Terry refits per game category (left) and per skill tag (right; [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Bradley–Terry Elo refit along four structural axes from Table [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: The Werewolves FSM: five states in a fixed [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: ∆self = tournament alt-win rate − self-play alt-win rate (pp), per (model, game), n = 30 episodes per cell in each setting. Rows ordered frontier → small. Positive cells (green) mean alt wins more in cross￾model tournament than in same-model self-play; nega￾tive (brown…
Figure 10
Figure 10. Figure 10: Raw alt-side win rate (%) per (model, game) in same-model self-play (light bars) and cross-model [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Qwen3-32B within-game baseline (no R) vs. within-game R1 vs. multigame R1 sources, broken out per target game and per team-side. Bars report alt-side win % (warm) and main-side win % (cool); the dotted line marks 50%. Resistance shows the largest multigame transfer ef…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 31 canonical work pages

  1. [1]

    Acta Universitatis Sapientiae, Informatica , volume=

    Virtual assistants: a review of the next frontier in AI interaction , author=. Acta Universitatis Sapientiae, Informatica , volume=. 2025 , publisher=

    work page 2025

  2. [2]

    NAACL , year=

    AI-LieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agents , author=. NAACL , year=

    work page

  3. [3]

    2026 , booktitle=

    SOTOPIA-ToM: Evaluating Information Management in Multi-Agent Interaction with Theory of Mind , author=. 2026 , booktitle=

    work page 2026

  4. [4]

    Findings of ACL , year=

    Imperfectly Cooperative Human-AI Interactions: Comparing the Impacts of Human and AI Attributes in Simulated and User Studies , author=. Findings of ACL , year=

    work page

  5. [5]

    2024 , booktitle=

    Is This the Real Life? Is This Just Fantasy? The Misleading Success of Simulating Social Interactions With LLMs , author=. 2024 , booktitle=

    work page 2024

  6. [6]

    2025 , eprint=

    Human-Centered Human-AI Collaboration (HCHAC) , author=. 2025 , eprint=

    work page 2025

  7. [7]

    Advances in neural information processing systems , volume=

    Solving quantitative reasoning problems with language models , author=. Advances in neural information processing systems , volume=

  8. [8]

    2025 , month = may, note =

    Introducing o3 and o4-mini , howpublished =. 2025 , month = may, note =

    work page 2025

Show all 68 references
  1. [9]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    FANToM: A benchmark for stress-testing machine theory of mind in interactions , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  2. [10]

    arXiv preprint arXiv:2503.00069 , year=

    Societal alignment frameworks can improve llm alignment , author=. arXiv preprint arXiv:2503.00069 , year=

    arXiv

  3. [11]

    2005 , publisher=

    The grammar of society: The nature and dynamics of social norms , author=. 2005 , publisher=

    2005

  4. [12]

    arXiv preprint arXiv:1606.06565 , year=

    Concrete problems in AI safety , author=. arXiv preprint arXiv:1606.06565 , year=

    arXiv

  5. [13]

    arXiv preprint arXiv:2201.03544 , year=

    The effects of reward misspecification: Mapping and mitigating misaligned models , author=. arXiv preprint arXiv:2201.03544 , year=

    arXiv

  6. [14]

    Artificial intelligence , volume=

    Explanation in artificial intelligence: Insights from the social sciences , author=. Artificial intelligence , volume=. 2019 , publisher=

    2019

  7. [15]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    arXiv

  8. [16]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    arXiv

  9. [17]

    2024 , eprint=

    SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents , author=. 2024 , eprint=

    2024

  10. [18]

    2025 , eprint=

    Measuring General Intelligence with Generated Games , author=. 2025 , eprint=

    2025

  11. [19]

    2025 , eprint=

    TextArena , author=. 2025 , eprint=

    2025

  12. [20]

    2026 , eprint=

    SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning , author=. 2026 , eprint=

    2026

  13. [21]

    2025 , eprint=

    AgentBench: Evaluating LLMs as Agents , author=. 2025 , eprint=

    2025

  14. [22]

    Advances in Neural Information Processing Systems , volume=

    CAMEL: Communicative Agents for" Mind" Exploration of Large Language Model Society , author=. Advances in Neural Information Processing Systems , volume=

  15. [23]

    Proceedings of the 36th annual acm symposium on user interface software and technology , pages=

    Generative agents: Interactive simulacra of human behavior , author=. Proceedings of the 36th annual acm symposium on user interface software and technology , pages=

  16. [24]

    Proceedings of the National Academy of Sciences , volume=

    Deception abilities emerged in large language models , author=. Proceedings of the National Academy of Sciences , volume=. 2024 , publisher=

    2024

  17. [25]

    Science , volume=

    Human-level play in the game of diplomacy by combining language models with strategic reasoning , author=. Science , volume=. 2022 , publisher=

    2022

  18. [26]

    arXiv preprint arXiv:2309.04658 , year=

    Exploring large language models for communication games: An empirical study on werewolf , author=. arXiv preprint arXiv:2309.04658 , year=

    arXiv

  19. [27]

    Nature Human Behaviour , volume=

    Playing repeated games with large language models , author=. Nature Human Behaviour , volume=. 2025 , publisher=

    2025

  20. [28]

    Advances in neural information processing systems , volume=

    Judging llm-as-a-judge with mt-bench and chatbot arena , author=. Advances in neural information processing systems , volume=

  21. [29]

    Advances in Neural Information Processing Systems , volume=

    Reflexion: Language Agents with Verbal Reinforcement Learning , author=. Advances in Neural Information Processing Systems , volume=

  22. [30]

    Advances in neural information processing systems , volume=

    Self-refine: Iterative refinement with self-feedback , author=. Advances in neural information processing systems , volume=

  23. [31]

    arXiv preprint arXiv:2310.14985 , year=

    LLM-Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplay , author=. arXiv preprint arXiv:2310.14985 , year=

    arXiv

  24. [32]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Sotopia- : Interactive learning of socially intelligent language agents , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  25. [33]

    1978 , publisher=

    The Rating of Chessplayers, Past and Present , author=. 1978 , publisher=

    1978

  26. [34]

    International Conference on Machine Learning , pages=

    Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference , author=. International Conference on Machine Learning , pages=. 2024 , organization=

    2024

  27. [35]

    the method of paired comparisons , author=

    Rank analysis of incomplete block designs: I. the method of paired comparisons , author=. Biometrika , volume=. 1952 , publisher=

    1952

  28. [36]

    Harper's Magazine , volume=

    Intelligence and its uses , author=. Harper's Magazine , volume=

  29. [37]

    2011 , publisher=

    Frames of mind: The theory of multiple intelligences , author=. 2011 , publisher=

    2011

  30. [38]

    science , volume=

    The evolution of cooperation , author=. science , volume=. 1981 , publisher=

    1981

  31. [39]

    1980 , publisher=

    The Strategy of Conflict: with a new Preface by the Author , author=. 1980 , publisher=

    1980

  32. [40]

    1990 , publisher=

    Governing the Commons: The Evolution of Institutions for Collective Action , author=. 1990 , publisher=

    1990

  33. [41]

    American Economic Review , volume=

    Cooperation and Punishment in Public Goods Experiments , author=. American Economic Review , volume=. 2000 , publisher=

    2000

  34. [42]

    Journal of Economic theory , volume=

    Games of perfect information, predatory pricing and the chain-store paradox , author=. Journal of Economic theory , volume=. 1981 , publisher=

    1981

  35. [43]

    Econometrica: Journal of the Econometric Society , pages=

    Perfect equilibrium in a bargaining model , author=. Econometrica: Journal of the Econometric Society , pages=. 1982 , publisher=

    1982

  36. [44]

    , author=

    Cues to deception. , author=. Psychological bulletin , volume=. 2003 , publisher=

    2003

  37. [45]

    John thinks that Mary thinks that…

    “John thinks that Mary thinks that…” attribution of second-order beliefs by 5- to 10-year-old children , journal =. 1985 , issn =. doi:https://doi.org/10.1016/0022-0965(85)90051-7 , url =

    1985 doi

  38. [46]

    Evolutionary Anthropology: Issues, News, and Reviews , volume=

    The Social Brain Hypothesis , author=. Evolutionary Anthropology: Issues, News, and Reviews , volume=

  39. [47]

    1984 , publisher=

    Influence: The Psychology of Persuasion , author=. 1984 , publisher=

    1984

  40. [48]

    Cognition , volume=

    Beliefs about Beliefs: Representation and Constraining Function of Wrong Beliefs in Young Children's Understanding of Deception , author=. Cognition , volume=

  41. [49]

    Cognition , volume=

    Does the Autistic Child Have a ``Theory of Mind''? , author=. Cognition , volume=

  42. [50]

    1988 , publisher=

    Machiavellian Intelligence: Social Expertise and the Evolution of Intellect in Monkeys, Apes, and Humans , editor=. 1988 , publisher=

    1988

  43. [51]

    Revisiting the evaluation of theory of mind through question answering , author=. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages=

    2019

  44. [52]

    Advances in Neural Information Processing Systems , volume=

    Understanding social reasoning in language models with language models , author=. Advances in Neural Information Processing Systems , volume=

  45. [53]

    Proceedings of the 2022 conference on empirical methods in natural language processing , pages=

    Neural theory-of-mind? on the limits of social intelligence in large lms , author=. Proceedings of the 2022 conference on empirical methods in natural language processing , pages=

    2022

  46. [54]

    arXiv preprint arXiv:2302.02083 , volume=

    Theory of mind may have spontaneously emerged in large language models , author=. arXiv preprint arXiv:2302.02083 , volume=

    arXiv

  47. [55]

    arXiv preprint arXiv:2302.08399 , year=

    Large language models fail on trivial alterations to theory-of-mind tasks , author=. arXiv preprint arXiv:2302.08399 , year=

    arXiv

  48. [56]

    arXiv preprint arXiv:2310.05036 , year=

    Avalonbench: Evaluating llms playing the game of avalon , author=. arXiv preprint arXiv:2310.05036 , year=

    arXiv

  49. [57]

    arXiv , author=

    Gtbench: Uncovering the strategic reasoning limitations of llms via game-theoretic evaluations. arXiv , author=. arXiv preprint arXiv:2402.12348 , year=

    arXiv

  50. [58]

    International Conference on Learning Representations , volume=

    Smartplay: A benchmark for llms as intelligent agents , author=. International Conference on Learning Representations , volume=

  51. [59]

    arXiv preprint arXiv:2305.16291 , year=

    Voyager: An open-ended embodied agent with large language models , author=. arXiv preprint arXiv:2305.16291 , year=

    arXiv

  52. [60]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Expel: Llm agents are experiential learners , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  53. [61]

    Handbook of Intelligence , editor=

    Social Intelligence , author=. Handbook of Intelligence , editor=

  54. [62]

    2025 , eprint=

    AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Society , author=. 2025 , eprint=

    2025

  55. [63]

    2023 , eprint=

    AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation , author=. 2023 , eprint=

    2023

  56. [64]

    Proceedings of the 41st International Conference on Machine Learning , pages=

    How well can LLMs negotiate? NEGOTIATIONARENA platform and analysis , author=. Proceedings of the 41st International Conference on Machine Learning , pages=

  57. [65]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Character-llm: A trainable agent for role-playing , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  58. [66]

    Wang, Noah and Peng, Z.y. and Que, Haoran and Liu, Jiaheng and Zhou, Wangchunshu and Wu, Yuhan and Guo, Hongcheng and Gan, Ruitong and Ni, Zehao and Yang, Jian and Zhang, Man and Zhang, Zhaoxiang and Ouyang, Wanli and Xu, Ke and Huang, Wenhao and Fu, Jie and Peng, Junran. R ol...

    2024 doi

  59. [67]

    2024 , publisher=

    Tessler, Michael Henry and Bakker, Michiel A and Jarrett, Daniel and Sheahan, Hannah and Chadwick, Martin J and Koster, Raphael and Evans, Georgina and Campbell-Gillingham, Lucy and Collins, Tantum and Parkes, David C and Botvinick, Matthew and Summerfield, Christopher , journ...

    2024

  60. [68]

    Journal of Consumer Research , volume=

    De Freitas, Julian and O. Journal of Consumer Research , volume=. 2026 , publisher=

    2026

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.