Pith. sign in

REVIEW 3 major objections 6 minor 37 references

The Decrypto Benchmark for Multi-Agent Reasoning and Theory of Mind

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces Decrypto, a word-association game benchmark, and claims that frontier LLMs play it worse than humans and simple word-embedding baselines, while newer reasoning models fall behind older ones on theory-of-mind tasks.

desk verdict Decrypto is a useful and well-executed benchmark, but the headline ToM regression is confounded by an output-reproduction artifact and needs reanalysis before it can carry weight. read the letter →

arxiv 2506.20664 v1 pith:PDXV53L6 submitted 2025-06-25 cs.AI cs.CLcs.HCcs.MA

Andrei Lupu , Timon Willi , Jakob Foerster This is my paper
classification cs.AIcs.CLcs.HCcs.MA
keywords theoryofmindmulti-agentreasoningLLMbenchmarkDecryptopragmaticinferenceperspectivetakingrepresentationalchangefalsebelief
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Decrypto, an interactive three-player word-association game rebuilt as a benchmark for multi-agent reasoning and theory of mind (ToM) in large language models. The design removes symbolic, mathematical, spatial, and embodied demands so that the remaining challenge is reasoning about what other players know, believe, and can infer. Empirically, the paper reports that frontier LLMs play the game worse than human novices and worse than simple word-embedding baselines, and that newer reasoning models score significantly worse than the older Llama 3.1-70B on ToM tasks adapted from the Smarties task and the Three Mountain Problem. The paper's point is that interactive games of this kind measure a distinct ability that static ToM benchmarks miss, and that this ability is currently regressing in the newest models.

What carries the argument

The load-bearing object is the Decrypto game itself: an eight-turn, three-role word game in which Alice and Bob share four secret keywords, Alice sends three hints for a three-digit code, Bob and the eavesdropper Eve each guess, and every guess, code, and hint is added to a public history that makes later turns progressively easier for Eve. This history is the mechanism that forces perspective taking, because Alice must choose hints that Bob can decode from the keywords but Eve cannot decode from the history alone. The paper also introduces a Hessian-determinant game metric that quantifies how sensitive outcomes are to changes in players or prompts, and two ToM instruments built on the platform: a three-question Smarties adaptation that compares beliefs before and after the keywords are revealed, and a Three-Mountain adaptation in which Alice predicts Eve's guess.

What would settle it

Re-run the Smarties and perspective-taking probes on the first turn, when Eve has no hint history and the code is uniformly random, replacing free-form generation with a forced choice among the 24 possible codes; if strong representational-change and false-belief accuracy rises well above 10 percent, or if reasoning models say 'cannot know' instead of predicting a specific intercept code, the reported regression would be a task artifact rather than a theory-of-mind deficit.

Watch

Extended reading notes

Core claim

The paper's central claim is that Decrypto, a game in which an encoder sends three word hints about secret keywords so a teammate can recover a code while an eavesdropper cannot, exposes a genuine and growing deficit in LLM social reasoning. In cooperative, competitive, and human-AI settings, LLMs produce hints that are too obvious or too ambiguous, get intercepted more often, and fail to coordinate, while GloVe and Word2Vec baselines that share an embedding space and strategy outperform every tested LLM. When the paper adapts the Smarties task to measure representational change and false belief, and the Three Mountain Problem to measure perspective taking, all models score at or below 10 percent on the strong variants. The newest reasoning models, including Claude 3.7 with extended thinking and o1 at high reasoning effort, are significantly worse than Llama 3.1-70B on all three abilities, often assuming the eavesdropper knows the secret keywords even when the prompt explicitly says otherwise.

Load-bearing premise

The word-based adaptations of the Smarties and Three-Mountain tasks measure the same abilities as the original embodied tasks; if LLM failures come instead from artifacts such as inability to reproduce an earlier exact output or a bias toward answering with a definite code, the headline regression claim gives way.

Editorial extensions

If this is right

  • If LLM game-playing ability in Decrypto lags humans and word embeddings, strong single-agent reasoning scores do not transfer to multi-agent language settings that require recursive belief modelling.
  • The public hint histories make each game generate fresh data, so Decrypto is harder to saturate than fixed-dataset benchmarks and can track capabilities across model generations.
  • Because newer reasoning models score below Llama 3.1-70B on all three ToM tasks, social reasoning should be monitored for regression across model releases rather than assumed to improve with scale or reasoning training.
  • Word-embedding baselines demonstrate that shared strategies and shared association spaces can coordinate well; the LLM bottleneck is modelling the other player's information, not vocabulary knowledge.
  • The platform's ToM experiments need only prompts and about fifteen lines of code, so new interactive ToM tests can be designed cheaply instead of reusing static question sets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test it, but its results suggest that reinforcement learning on verifiable reasoning tasks may train models to commit to definite answers at the cost of perspective taking; comparing interception predictions before and after such fine-tuning would test that mechanism.
  • The same benchmark could measure cross-model theory of mind by pairing Alice and Bob with different pretraining distributions or embedding spaces; miscommunication rates would then quantify how well each model anticipates the other's associations.
  • The Hessian-determinant metric could be exported to other LLM evaluations as a prompt-sensitivity diagnostic, flagging benchmarks where outcome variance comes from wording rather than competence.
  • Because Eve's current interception advantage is driven largely by weak LLM hinting, the leaderboard likely undervalues LLM interception skill; replaying human games against different Eve models, as the paper does, separates hint quality from interception ability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Decrypto, an interactive word-association game transformed into a benchmark for multi-agent reasoning and theory of mind (ToM). It evaluates frontier LLMs, word-embedding baselines, and human players in cooperative, competitive, cross-play, and human-AI settings, and it proposes adaptations of the Smarties and Three-Mountain tasks to measure representational change, false belief, and perspective taking. The headline empirical claims are that LLMs underperform simple embedding baselines and human players, and that newer reasoning models (Claude 3.7, o1) are significantly worse than older models such as Llama 3.1-70B on the ToM tasks. The paper also contributes an RSA formalization of Decrypto, a prompt-robustness study with 625 prompt variants, and a public code and data release.

Significance. If the headline findings hold, Decrypto would be a valuable interactive benchmark for multi-agent reasoning and ToM, addressing real gaps in existing static benchmarks: it is interactive, open-ended, avoids many tokenization and embodiment confounds, and produces granular failure-mode metrics. The paper's strengths include the breadth of the empirical evaluation, the human-AI cross-play data collection, the explicit RSA derivation in Appendix H, the large prompt-robustness study in Appendix G, and the public release of code and data with multi-seed reporting. However, the central ToM regression claim rests on a scoring rule that may measure output self-consistency rather than theory of mind, so the significance is conditional on that issue being resolved.

major comments (3)
  1. [Section 3.2, Figure 4, Appendix F.1] The strong RC and FB scores are defined as B=A and C=A, where A is Eve's keyword prediction, while B and C are generated from independent prompts that do not include A in context. This makes a 'strong pass' require the model to reproduce its earlier four-keyword output exactly, from the same history but without seeing that output. A model with correct ToM could represent its prior false belief yet answer with a different but equally justified keyword set, especially under exact-match scoring on an open-ended answer space. The near-zero strong-pass rates and the finding that newer reasoning models are worse may therefore reflect lower output stability in those models rather than a ToM regression. The weak variants are too lenient, since any answer differing from the true keywords passes, and the PT results, while qualitatively more convincing, cannot alone carry the headline claim. Please add a control that isolates output-reproduction ability from ToM, for example by conditioning on self-consistency on repeated identical prompts, or by scoring B and C against the set of outputs the model would generate across multiple independent samples for A, or by using a recognition format ('was your earlier guess X?') rather than free regeneration.
  2. [Section 1, Table 1] The Hessian determinant metric is advertised as a key contribution ('we introduce a novel game metric where we use the determinant of the Hessian matrix to measure the sensitivity of game outcomes to changes in players or prompts'), but it is never defined, computed, or plotted anywhere in the paper. Table 1 states that 'Human players have a much higher Hessian determinant, showing their superior performance,' yet no matrix, equation, or numerical value is given. This claim is currently unfalsifiable. Please provide the precise definition of the Hessian, the variables with respect to which it is taken, and the reported values, or remove the metric and the related statements from the abstract and main text.
  3. [Section 4.1, Figure 2] The baseline vocabulary size K is chosen post hoc: the text says 'We thus use K = 16 for the rest of the paper' after presenting Figure 2, which is the same figure used to argue that the baselines coordinate well and outperform LLMs. Because K directly controls the difficulty and cross-play compatibility of the baselines, this selection risks circularity in the main baseline-comparison claim. Please justify K with a criterion that is independent of the target comparisons, or report the headline baseline-vs-LLM results across a range of K values to show that the conclusions are not an artifact of this particular K.
minor comments (6)
  1. [Listing 1] Line 5 contains a typo: 'co munication' should be 'communication'.
  2. [Listings 3 and 4] The phrase 'For exam le' should be 'For example' in both listings.
  3. [Listing 5] Line 5 reads 'Eve d d not' and should read 'Eve did not'.
  4. [Listing 7] The output format example in line 19 contains an incomplete guess pattern, 'X-Y- ', which should be 'X-Y-Z'.
  5. [Figure 2 caption] The caption has an extra space in 'Average T urn Length' and 'Average T urn Length vs. Vocabulary Size'; these should be 'Turn'.
  6. [Section 4.1] The sentence 'We thus useK = 16for the rest of the paper' is missing a space between 'K = 16' and 'for'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: central claims are direct measurements; RSA formalization is self-contained with stated assumptions.

full rationale

The paper's headline results are empirical measurements rather than derived predictions: LLM versus word-embedding baseline performance is measured by game outcomes, human-AI coordination is measured by cross-play experiments, and the ToM scores are defined by explicit prompt-response comparisons. The Appendix H RSA treatment is presented as a formalization 'with mild assumptions'; its conclusion that Bob must model Alice's model of Eve follows from the stated utility definition and the Bayesian listener definition, which are clearly labeled assumptions rather than fitted parameters or empirical predictions. The K=16 choice for the embedding baselines is an explicitly justified operating point selected for cross-embedding correlation, not a parameter fitted to force the LLM-lag result. Self-citations to prior MARL work are contextual and not load-bearing for the central benchmark claim. The strong RC/FB criterion requiring B=A or C=A raises a legitimate construct-validity question about whether exact-output reproduction measures self-consistency rather than theory of mind, but that is an experimental design concern, not a circular reduction of the paper's derivation to its inputs.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The central empirical claims rest on the assumption that the game and its ToM variants have construct validity, and on the selection of the baseline parameter K=16. The RSA formalization relies on standard pragmatic-inference assumptions. No new physical entities are proposed; the undefined Hessian metric is the only advertised invented formalism.

free parameters (2)
  • K (baseline hint vocabulary size per keyword) = 16
    Chosen by inspecting Figure 2 to place GloVe/Word2Vec baselines in a competitive regime; the LLM-vs-baseline conclusions depend on this choice.
  • Generation temperature = 0.6 (game) / 0 (ToM)
    Hand-chosen generation parameter; not the subject of the paper's claims, but affects all results and is not varied in the main experiments.
assumptions (5)
  • domain assumption Decrypto hints must refer to the meaning of keywords via publicly available shared knowledge (game rules).
    Section 2; this underlies the entire benchmark and the RSA formalization.
  • standard math RSA literal-listener semantics with P(m)=1/|M|, exponential speaker model, and cost function cost(u,m)=ε log(1-P_Eve(m|u)).
    Appendix H, Equations 1-6; this is a standard applied framework, not an empirical fit.
  • domain assumption The Smarties-task adaptation assumes that recalling one's prior prediction (B=A) measures representational change, and predicting a second interceptor (C=A) measures false belief.
    Section 3.2; construct validity of the ToM experiments rests on this analogy to the original cognitive tasks.
  • domain assumption The Three Mountain Problem adaptation assumes that Alice's prediction of Eve's interception rate reflects perspective taking, rather than a separate prediction failure.
    Section 3.2 and Appendix M; the interpretation of the PT metric depends on this.
  • domain assumption LLM outputs at temperature 0 are deterministic enough for the ToM comparisons.
    Section F.2; the paper notes some closed models are not fully deterministic even at temperature 0.
invented entities (1)
  • Hessian determinant metric for game-outcome sensitivity
    purpose: Advertised as a key contribution to measure sensitivity of game outcomes to changes in players or prompts.
    Never defined or computed in the paper; no formula, no evaluation, only mentions in Section 1 and Appendix G.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Decrypto Benchmark for Multi-Agent Reasoning and Theory of Mind." pith.science (2026). https://pith.science/paper/PDXV53L6

@misc{pith2026250620664,
  author       = {Pith},
  title        = {Pith review of: The Decrypto Benchmark for Multi-Agent Reasoning and Theory of Mind},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PDXV53L6}},
  note         = {Machine review of arXiv:2506.20664}
}
read the original abstract

As Large Language Models (LLMs) gain agentic abilities, they will have to navigate complex multi-agent scenarios, interacting with human users and other agents in cooperative and competitive settings. This will require new reasoning skills, chief amongst them being theory of mind (ToM), or the ability to reason about the "mental" states of other agents. However, ToM and other multi-agent abilities in LLMs are poorly understood, since existing benchmarks suffer from narrow scope, data leakage, saturation, and lack of interactivity. We thus propose Decrypto, a game-based benchmark for multi-agent reasoning and ToM drawing inspiration from cognitive science, computational pragmatics and multi-agent reinforcement learning. It is designed to be as easy as possible in all other dimensions, eliminating confounding factors commonly found in other benchmarks. To our knowledge, it is also the first platform for designing interactive ToM experiments. We validate the benchmark design through comprehensive empirical evaluations of frontier LLMs, robustness studies, and human-AI cross-play experiments. We find that LLM game-playing abilities lag behind humans and simple word-embedding baselines. We then create variants of two classic cognitive science experiments within Decrypto to evaluate three key ToM abilities. Surprisingly, we find that state-of-the-art reasoning models are significantly worse at those tasks than their older counterparts. This demonstrates that Decrypto addresses a crucial gap in current reasoning and ToM evaluations, and paves the path towards better artificial agents.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 12 canonical work pages

  1. [1]

    Llm-deliberation: Evaluating llms with interactive multi-agent negotiation games.arXiv preprint arXiv:2309.17234,

    Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad, Lea Schönherr, and Mario Fritz. Llm-deliberation: Evaluating llms with interactive multi-agent negotiation games.arXiv preprint arXiv:2309.17234,

    arXiv

  2. [4]

    Two” refers to “two dimensions

    plane], and the hints are[two, clone, AC/DC]. “Two” refers to “two dimensions”, which is the defining characteristic of a geometric plane. “Clone” is a hint for “star”, since both “clone” and “star” are common operations performed on a GitHub repository. “AC/DC” is a rock band, and one of their most famous songs, “Thunderstruck” starts with the word “Thun...

    work page 2009

  3. [6]

    Overcookedv2: Rethinking overcooked for zero-shot coordination.arXiv preprint arXiv:2503.17821,

    TobiasGessler, TinDizdarevic, AniCalinescu, BenjaminEllis, AndreiLupu, andJakobNicolausFoerster. Overcookedv2: Rethinking overcooked for zero-shot coordination.arXiv preprint arXiv:2503.17821,

    arXiv

  4. [8]

    jazz fusion

    This suggests that the poor performance of LLMs in Decrypto is more likely due to a lack of reasoning abilities than a lack of prompt tuning. We look at the robustness of different models to variations in prompts and generation parameters, keeping in mind that generalist agents cannot control their prompt since it is assumed to be part of the environment....

    work page 2016

  5. [9]

    Hi-tom: A benchmark for evaluating higher-order theory of mind reasoning in large language models.arXiv preprint arXiv:2310.16755,

    11 Yinghui He, Yufan Wu, Yilin Jia, Rada Mihalcea, Yulong Chen, and Naihao Deng. Hi-tom: A benchmark for evaluating higher-order theory of mind reasoning in large language models.arXiv preprint arXiv:2310.16755,

    arXiv

  6. [11]

    Large language models cannot self-correct reasoning yet.arXiv preprint arXiv:2310.01798,

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet.arXiv preprint arXiv:2310.01798,

    arXiv

  7. [12]

    Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

    arXiv

  8. [13]

    Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770,

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770,

    arXiv

Show all 37 references
  1. [15]

    Theory of mind may have spontaneously emerged in large language models

    Michal Kosinski. Theory of mind may have spontaneously emerged in large language models. arXiv preprint arXiv:2302.02083, 4:169,

    arXiv

  2. [16]

    Revisiting the evaluation of theory of mind through question answering

    Matthew Le, Y-Lan Boureau, and Maximilian Nickel. Revisiting the evaluation of theory of mind through question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors,Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing an...

    2019

  3. [17]

    doi: 10.18653/v1/D19-1598.https://aclanthology.org/D19-1598/

    Association for Computational Linguistics. doi: 10.18653/v1/D19-1598.https://aclanthology.org/D19-1598/. Huao Li, Yu Quan Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Michael Lewis, and Katia Sycara. Theory of mind for multi-agent collaboration via large language mod...

    arXiv

  4. [18]

    Avalonbench: Evaluating llms playing the game of avalon

    Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. Avalonbench: Evaluating llms playing the game of avalon. In NeurIPS 2023 Foundation Models for Decision Making Workshop,

    2023

  5. [19]

    Llm-powered hierarchical language agent for real-time human-ai coordination.arXiv preprint arXiv:2312.15224,

    Jijia Liu, Chao Yu, Jiaxuan Gao, Yuqing Xie, Qingmin Liao, Yi Wu, and Yu Wang. Llm-powered hierarchical language agent for real-time human-ai coordination.arXiv preprint arXiv:2312.15224,

    arXiv

  6. [20]

    Efficient estimation of word representations in vector space.arXiv preprint arXiv:1301.3781,

    Tomas Mikolov. Efficient estimation of word representations in vector space.arXiv preprint arXiv:1301.3781,

    arXiv

  7. [23]

    Modeling cross-cultural pragmatic inference with codenames duet.arXiv preprint arXiv:2306.02475,

    12 Omar Shaikh, Caleb Ziems, William Held, Aryan J Pariani, Fred Morstatter, and Diyi Yang. Modeling cross-cultural pragmatic inference with codenames duet.arXiv preprint arXiv:2306.02475,

    arXiv

  8. [24]

    Mastering chess and shogi by self-play with a general reinforcement learning algorithm.arXiv preprint arXiv:1712.01815,

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm.arXiv preprint arXiv:1712.01815,

    arXiv

  9. [28]

    Opentom: A comprehensive benchmark for evaluating theory-of-mind reasoning capabilities of large language models.arXiv preprint arXiv:2402.06044,

    Hainiu Xu, Runcong Zhao, Lixing Zhu, Jinhua Du, and Yulan He. Opentom: A comprehensive benchmark for evaluating theory-of-mind reasoning capabilities of large language models.arXiv preprint arXiv:2402.06044,

    arXiv

  10. [29]

    On the tool manipulation capability of open-source large language models, 2023a

    Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the tool manipulation capability of open-source large language models, 2023a. Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models fo...

    arXiv

  11. [30]

    A careful examination of large language model performance on grade school arithmetic.arXiv preprint arXiv:2405.00332, 2024a

    Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al. A careful examination of large language model performance on grade school arithmetic.arXiv preprint arXiv:2405.00332, 2024a. Wenqi Zhang, Ke Tang, H...

    arXiv

  12. [31]

    meaningful

    While the 10 games collected were sufficient to demonstrate the limitations of LLMs in human-AI interactions, we believe that there is much to be learned from a more extensive study, including with humans as interceptors. We therefore made sure to provide the tools to enable s...

    2023

  13. [32]

    Finally, we believe the study of pragmatic inference in LLMs to be a promising avenue for future research, which is made much easier by the release of our benchmark

    produces the same kind of idiosyncrasies observed when trainingtabula rasamodels in other cooperative environments (Cui et al., 2023; Gessler et al., 2025). Finally, we believe the study of pragmatic inference in LLMs to be a promising avenue for future research, which is made...

    2023

  14. [34]

    Therefore, we set generous token limits (between 750 for non-reasoning models and up to 10000 for reasoning ones) to prevent cutting model generations prematurely

    Our goal was not to evaluate model verbosity, but rather the limit of their capabilities. Therefore, we set generous token limits (between 750 for non-reasoning models and up to 10000 for reasoning ones) to prevent cutting model generations prematurely. However, we found that ...

    2023

  15. [36]

    This makes Alice’s utility U (u, m) =β log PLit(m|u) +ε log(1 − PEve(m|u))

    Similarly, we can add a variableβ ∈ [0, 1], which captures the importance of not miscommunicating on this turn. This makes Alice’s utility U (u, m) =β log PLit(m|u) +ε log(1 − PEve(m|u)). (6) Finally, Bob, thepragmatic listener, decodes utterances based on the common prior ove...

    2023

  16. [37]

    airplane

    is therefore reasonable, 20 and still allows for a very large number of possible utterances. In fact, this is very similar to the way the word embedding baselines in Figure 2 only choose hints that are guaranteed to be interpreted correctly by Bob. Given these conditions, the ...

    2016

  17. [1988]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    arXiv

  18. [2010]

    Large language models fail on trivial alterations to theory-of-mind tasks

    Tomer Ullman. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399,

    arXiv

  19. [2011]

    Explore theory of mind: Program-guided adversarial data generation for theory of mind reasoning.arXiv preprint arXiv:2412.12175,

    Melanie Sclar, Jane Yu, Maryam Fazel-Zarandi, Yulia Tsvetkov, Yonatan Bisk, Yejin Choi, and Asli Celikyilmaz. Explore theory of mind: Program-guided adversarial data generation for theory of mind reasoning.arXiv preprint arXiv:2412.12175,

    arXiv

  20. [2013]

    GloVe: Global vectors for word representation

    Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543,

    2014

  21. [2014]

    Fantom: A benchmark for stress-testing machine theory of mind in interactions.arXiv preprint arXiv:2310.15421,

    Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Le Bras, Gunhee Kim, Yejin Choi, and Maarten Sap. Fantom: A benchmark for stress-testing machine theory of mind in interactions.arXiv preprint arXiv:2310.15421,

    arXiv

  22. [2017]

    Tokenization counts: the impact of tokenization on arithmetic in frontier llms.arXiv preprint arXiv:2402.14903,

    Aaditya K Singh and DJ Strouse. Tokenization counts: the impact of tokenization on arithmetic in frontier llms.arXiv preprint arXiv:2402.14903,

    arXiv

  23. [2018]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    arXiv

  24. [2019]

    Tombench: Benchmarking theory of mind in large language models.arXiv preprint arXiv:2402.15052,

    Zhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen, Guanqun Bi, Gongyao Jiang, Yaru Cao, Mengting Hu, Yunghwei Lai, Zexuan Xiong, et al. Tombench: Benchmarking theory of mind in large language models.arXiv preprint arXiv:2402.15052,

    arXiv

  25. [2020]

    Battleagentbench: A benchmark for evaluating cooperation and competition capabilities of language models in multi-agent systems.arXiv preprint arXiv:2408.15971,

    Wei Wang, Dan Zhang, Tao Feng, Boyan Wang, and Jie Tang. Battleagentbench: A benchmark for evaluating cooperation and competition capabilities of language models in multi-agent systems.arXiv preprint arXiv:2408.15971,

    arXiv

  26. [2021]

    Re-evaluating theory of mind evaluation in large language models.arXiv preprint arXiv:2502.21098,

    Jennifer Hu, Felix Sosa, and Tomer Ullman. Re-evaluating theory of mind evaluation in large language models.arXiv preprint arXiv:2502.21098,

    arXiv

  27. [2023]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  28. [2024]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457,

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457,

    arXiv

  29. [2025]

    Embodied llm agents learn to cooperate in organized teams.arXiv preprint arXiv:2403.12482,

    Xudong Guo, Kaixuan Huang, Jiale Liu, Wenhui Fan, Natalia Vélez, Qingyun Wu, Huazheng Wang, Thomas L Griffiths, and Mengdi Wang. Embodied llm agents learn to cooperate in organized teams.arXiv preprint arXiv:2403.12482,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.