Mind Your Theory: Theory of Mind Goes Deeper Than Reasoning

Eitan Wagner; Joseph M. Barnby; Nitay Alon; Omri Abend

REVIEW 2 major objections 5 minor 12 references

Mind Your Theory: Theory of Mind Goes Deeper Than Reasoning

T0 review · 2 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that LLM theory-of-mind benchmarks measure reasoning, not the decision to mentalize, and that a two-step view of ToM (invoke, then infer) is needed.

desk verdict A useful two-step ToM framing for LLM evaluation, but the pilot's proxy task doesn't measure invocation, so the empirical conclusion is overstated. read the letter →

arxiv 2412.13631 v3 pith:H2LC653Y submitted 2024-12-18 cs.AI cs.CL

Eitan Wagner , Nitay Alon , Joseph M. Barnby , Omri Abend This is my paper · ORCID

classification cs.AIcs.CL

keywords theoryofminddepthmentalizingLLMbenchmarksfalsebeliefreasoningvsinteractiveevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that current evaluations of theory of mind (ToM) in large language models measure only the second half of a two-step cognitive process. Step I is deciding whether to invoke ToM and at what Depth of Mentalizing (DoM) — the number of nested beliefs to consider; step II is applying the correct belief inference once that depth is fixed. The authors claim that benchmarks like HiToM and FANToM, along with prompting add-ons, treat ToM as a static logic problem, so a model that fails them may be failing at reasoning rather than at mentalizing. A small pilot shows that models can predict the required ToM order of a question (a proxy for step I) while still answering the belief questions incorrectly, which the authors read as evidence that the hard part in these benchmarks is step II. If the distinction holds, the debate about emergent ToM in LLMs needs to be reframed around invocation and adaptive depth, not raw accuracy on false-belief vignettes.

What carries the argument

The load-bearing object is the two-step decomposition of a ToM task: step I, invocation, sets the Depth of Mentalizing (DoM) — the level $k$ of recursive belief nesting, from $k=-1$ (no mentalizing) to $k\geq 1$ (nested beliefs about beliefs) — and step II performs the inference conditional on that depth. The empirical instrument is a label-prediction task: given a question (with or without its story), the model must output the ToM order that the benchmark's metadata assigns to it. Accuracy on this sub-task is used as a proxy for whether step I can be performed; the near-perfect performance of GPT-4o on the proxy, combined with much lower accuracy on the original belief questions, is the evidence that benchmark failures are Type C reasoning errors rather than failures to invoke ToM.

What would settle it

Find or build a benchmark in which the question does not reveal the required depth — the model must infer from the social context whether mentalizing is needed — and show that models with high ToM-order prediction accuracy on HiToM/FANToM nevertheless fail to invoke ToM when the context demands it; alternatively, show a model that answers belief questions correctly while unable to predict their ToM order, which would break the claimed link between step I and label prediction.

Watch

Extended reading notes

Core claim

The paper's central claim is that the dominant research paradigm for LLM theory of mind addresses only what the authors call step II — inference at a fixed depth of mentalizing — and therefore mistakes failures of logical reasoning for failures of social cognition. Formally, a ToM task is a distribution $P(Y \mid X)$ over answers $Y$ given a prompt $X$, decomposed into a meta-decision $do(\mathrm{DoM}(k))$ that sets the recursion depth, followed by the conditional $P(Y \mid do(\mathrm{DoM}(k)), X)$. Because current benchmarks announce the mentalizing demand through templatized questions about clearly described scenarios, the model never has to decide whether ToM is needed or how deep to go. The paper offers a case study on HiToM and FANToM in which GPT-4o and GPT-4o-mini predict the required ToM order nearly perfectly while struggling with the belief questions themselves, supporting the conclusion that the benchmark difficulty lies in step II. The authors therefore call for benchmarks that test whether and at what depth ToM is invoked, in dynamic interactive settings.

Load-bearing premise

The empirical argument equates succeeding at step I with predicting the ToM-order label of a question, so if label prediction is not a faithful measure of whether a model would actually decide to mentalize in an interactive setting, the pilot's support for the paper's central claim weakens.

Editorial extensions

If this is right

If the two-step distinction is correct, high scores on existing ToM benchmarks should not be read as evidence that a model decides when to mentalize; they show inference within an already-announced depth.
Low scores on the same benchmarks should not be read as ToM deficits, since they may be ordinary logical failures; error taxonomies need a Type B (under- or over-mentalizing) category alongside Type C.
ToM add-ons that fix a DoM in advance cannot improve invocation, so their gains are limited to step II; a model that never chooses to mentalize will not be helped by these modules.
Evaluation should move to dynamic, interactive settings where the agent must re-assess its DoM as new information arrives, mirroring cognitive tasks with biological agents.
Because mentalizing has a resource cost, correct invocation is also an efficiency question: models that over-mentalize waste compute, and models that under-mentalize miss deception.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's proxy equates invocation with predicting a question's metadata label; a stronger test would require the model to choose the depth from the social situation itself — e.g., competition vs. cooperation — without the question revealing it, and then act on that choice.
If the distinction generalizes, earlier 'ToM emergence' claims (e.g., on classic false-belief tasks) may overstate social cognition, since those tasks announce the need for mentalizing; conversely, failures on altered vignettes may be reasoning brittleness rather than absent ToM.
A testable extension: measure whether models adjust their DoM in a dialogue when a speaker is revealed as deceptive or when stakes change; the paper predicts such adaptation is currently missing.
The resource-cost argument suggests a concrete behavioral signature: an interactive agent that can conserve mentalizing in cooperative settings and escalate in competitive ones should outperform a fixed-DoM agent at equal parameter count.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things worth knowing about this paper. The conceptual core is solid: separating ToM into step I (deciding whether and at what depth to mentalize) from step II (applying the inference) is a genuine and useful reframing, and the paper is right that most LLM benchmarks and add-ons assume the invocation is already done for you. The survey of benchmarks, add-ons, and formal models is broad and fair, and the formalization via do-calculus is thoughtful.

Where it wobbles is the experiment. The pilot on HiToM and FANToM asks models to predict the ToM-order label of a question, and then infers that because they can do that while failing the belief questions, the hard part is reasoning, not invocation. That doesn't follow. Predicting the label from the question alone, which the results show is just as accurate as with the story, is a meta-classification task solvable from surface syntax (nested 'thinks that' clauses). It never tests spontaneous invocation, because the prompt explicitly asks for the order. The stress-test note is right on this: a model can state the correct order in a separate meta-task and still under-mentalize during the actual question. Appendix A.3's own admission that gold labels are contestable makes the ground truth shakier.

That said, the weakness is in the empirical support, not the conceptual argument. The paper is honest about the pilot being a case study, and the recommendations for interactive, dynamic benchmarks are reasonable and align with calls from other groups (Wang et al. 2025; Sclar et al. 2022), which are cited. The lack of code, data, and confidence intervals on a 100-example pilot is a minor additional concern, but the core issue is the proxy's validity.

For a reader, the value is in the framing: if you are building or evaluating ToM benchmarks, the Type A/B/C error taxonomy and the invocation-vs-inference distinction are worth taking seriously. The pilot should be treated as illustrative only. The paper deserves a serious referee because the conceptual claim is important and the writing is clear; the referee should ask for a better experiment or a more modest conclusion.

Referee Report

2 major / 5 minor

Summary. Wagner et al. argue that current AI and LLM Theory of Mind (ToM) research focuses on the second step of a two-step process—performing inference at a given Depth of Mentalizing (DoM)—while largely ignoring the first step, the decision of whether and at what depth to invoke ToM. They survey benchmarks, ToM add-ons, and formal models, and present a pilot experiment on HiToM and FANToM in which LLMs predict the ToM-order label of questions. They conclude from this pilot that failures on these benchmarks are due to reasoning (Type C) rather than to ToM invocation (Type B), and they recommend dynamic, interactive evaluations of ToM invocation.

Significance. If the two-step framing is accepted, the paper makes a valuable conceptual contribution by identifying a gap in LLM ToM evaluation and by connecting the AI literature to cognitive science. The survey of benchmarks, add-ons, and formal models is informative, and the proposed dynamic evaluation is a sensible direction. The paper also gives explicit, falsifiable predictions about where benchmark difficulty lies. However, the central empirical claim is not currently supported by the reported pilot, so the paper's main contribution rests on its conceptual argument and recommendations rather than on the experimental evidence.

major comments (2)

[Section 3.2, Appendix A.1–A.2] The experiment's proxy task does not measure step I as the paper defines it. Step I is the 'meta-decision regarding the DoM' that an agent makes while solving a social reasoning task; the pilot, however, explicitly instructs the model to output the ToM order as an answer, so the model never faces the spontaneous decision of whether to invoke ToM. The fact that Table 1 shows identical accuracy with and without the story for every model and setting strongly suggests that the order labels are predicted from the question's surface syntax rather than from any mental-state tracking. Consequently, the experiment cannot distinguish Type B from Type C errors, and the conclusion that benchmark failures are not due to ToM invocation is not supported by this experiment.
[Section 3.2, Appendix A.2–A.3] The strength of the conclusion is disproportionate to the evidence. The pilot uses 100 examples per benchmark, reports no confidence intervals, provides no code or data, and relies on gold labels that Appendix A.3 itself describes as contestable for some FANToM questions. At most the pilot demonstrates that a closely related meta-classification (label prediction) is easier than the original task; it does not show that invocation is not the bottleneck. The text should either present this as preliminary, suggestive evidence or be substantially expanded, for example with a control condition where the model must spontaneously decide whether a question requires ToM.

minor comments (5)

[Title] The title contains a formatting artifact: 'Mind Y our Theory' should read 'Mind Your Theory.'
[Appendix A.1] The reported FANToM baseline accuracies list '0.426 with GPT-4-turbo, and 0.497 with GPT-4-turbo'; the second model name appears to be a typo.
[Section 3.2] The assertion that 'the necessity of ToM is made obvious' in existing benchmarks would benefit from concrete examples or a citation to an analysis that demonstrates this property.
[Appendix A] Making the code, prompts, and sampled data publicly available would allow readers to verify the pilot's results and assess the validity of the proxy task.
[Limitations] The Limitations section does not mention the pilot's limitations, such as the small sample size, the lack of confidence intervals, and the potential invalidity of the proxy task; a sentence acknowledging these would improve the paper.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the framework borrows DoM from prior cognitive work by a co-author, but the central evaluation uses external benchmark metadata and no equation reduces to its own input.

full rationale

The paper's central claim is that existing LLM ToM benchmarks target only step II (inference at a fixed DoM) and neglect step I (whether and at what depth to mentalize). This claim is supported by a conceptual argument about vignette tasks making ToM necessity obvious, and by a pilot in which models predict the ToM-order metadata of HiToM and FANToM questions. No parameter is fitted to benchmark answers and then renamed as a prediction; the ToM-order labels are external metadata that guide the benchmarks' generation. The DoM concept is taken from Barnby et al. (2023), a prior cognitive-science paper co-authored by one of the present authors, but the same two-step distinction is also grounded in Schaafsma et al. (2015) and Leslie et al. (2004), so the self-citation is not load-bearing. The main weakness is construct validity, not circularity: Appendix A instructs models to report a ToM order, so high accuracy on that prompted meta-classification does not by itself prove spontaneous invocation, and Appendix A.3 concedes that gold labels are contestable. That is a substantive objection to the pilot's conclusion that failures are Type C errors, but it is not a case of a result being equivalent to its inputs by construction. The paper also flags this limitation when noting that a model can predict the correct depth yet stop incorrectly during reasoning. Accordingly, no circular step is established; the paper is self-contained against external benchmarks and the self-citations are background support.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper does not fit parameters to data; it relies on imported conceptual axioms. The key assumption is that predicting the ToM-order label of a question measures the invocation step, which is ad hoc to the paper's empirical argument.

assumptions (3)

domain assumption The two-step decomposition of ToM (first decide whether and at what depth to mentalize, then apply inference) is the correct way to evaluate ToM in LLMs.
Adopted from cognitive science (Schaafsma et al. 2015; Leslie et al. 2004; Barnby et al. 2023) and asserted in Section 3 as a formalization; not derived within the paper.
ad hoc to paper Predicting the ToM-order label of a question is a valid and sufficient test of the invocation step (step I).
The empirical argument in Appendix A uses label-prediction accuracy to conclude that step I is not the bottleneck; this equivalence is assumed, not independently validated.
domain assumption The ToM-order metadata in HiToM and FANToM provide reliable gold labels for the required mentalizing depth.
The experiment relies on benchmark metadata as ground truth, although Appendix A.3 itself documents ambiguities in FANToM labels and differing interpretations of implicit beliefs.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Mind Your Theory: Theory of Mind Goes Deeper Than Reasoning." pith.science (2026). https://pith.science/paper/H2LC653Y

@misc{pith2026241213631,
  author       = {Pith},
  title        = {Pith review of: Mind Your Theory: Theory of Mind Goes Deeper Than Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H2LC653Y}},
  note         = {Machine review of arXiv:2412.13631}
}

read the original abstract

Theory of Mind (ToM) capabilities in LLMs have recently become a central object of investigation. Cognitive science distinguishes between two steps required for ToM tasks: 1) determine whether to invoke ToM, which includes the appropriate Depth of Mentalizing (DoM), or level of recursion required to complete a task; and 2) applying the correct inference given the DoM. In this position paper, we first identify several lines of work in different communities in AI, including LLM benchmarking, ToM add-ons, ToM probing, and formal models for ToM. We argue that recent work in AI tends to focus exclusively on the second step which are typically framed as static logic problems. We conclude with suggestions for improved evaluation of ToM capabilities inspired by dynamic environments used in cognitive tasks.

Figures

Figures reproduced from arXiv: 2412.13631 by the authors.

**Figure 2.** Hierarchical Dynamic Theory of Mind (ToM). [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 7 canonical work pages

[6]

InProceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 15959–15983, Bangkok, Thailand

ToMBench: Benchmarking theory of mind in large language models. InProceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 15959–15983, Bangkok, Thailand. Association for Computational Linguistics. Fiery Cushman. 2024. Computational social psychol- ogy.Annual Review of Psychology, 75(1):625–6...

work page 2024
[9]

InProceedings of the 2024 Conference on Empir- ical Methods in Natural Language Processing, pages 19794–19809, Miami, Florida, USA

Perceptions to beliefs: Exploring precursory inferences for theory of mind in large language mod- els. InProceedings of the 2024 Conference on Empir- ical Methods in Natural Language Processing, pages 19794–19809, Miami, Florida, USA. Association for Computational Linguistics. Boaz Keysar, Shuhong Lin, and Dale J Barr. 2003. Limits on theory of mind use i...

arXiv 2024
[11]

Probing the Robustness of Theory of Mind in Large Language Models

Theory of mind for multi-agent collabora- tion via large language models. InProceedings of the 2023 Conference on Empirical Methods in Natu- ral Language Processing, pages 180–192, Singapore. Association for Computational Linguistics. Xiaomeng Ma, Lingyu Gao, and Qihui Xu. 2023a. ToM- Challenges: A principle-guided dataset and diverse evaluation tasks for...

work page Pith review arXiv 2023
[1985]

theory of mind

Does the autistic child have a “theory of mind”?Cognition, 21(1):37–46. Peter Bossaerts and Carsten Murawski. 2017. Com- putational complexity and human decision-making. Trends in cognitive sciences, 21(12):917–929. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, A...

arXiv 2017
[2011]

InProceedings of the an- nual meeting of the cognitive science society, vol- ume 33

Bayesian theory of mind: Modeling joint belief-desire attribution. InProceedings of the an- nual meeting of the cognitive science society, vol- ume 33. Cristian-Paul Bara, Sky CH-Wang, and Joyce Chai

work page
[2014]

Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gersten- berg, and Noah D

Theory of mind: did evolution fool us?PloS One, 9(2):e87619. Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gersten- berg, and Noah D. Goodman. 2023. Understanding social reasoning in language models with language models.Preprint, arXiv:2306.15448. Piotr J Gmytrasiewicz and Prashant Doshi. 2005. A framework for sequential planning in multi-agent settings.Jou...

arXiv 2023
[2017]

Brit Grosskopf and Rosemarie Nagel

How can memory-augmented neural networks pass a false-belief task? InCogSci. Brit Grosskopf and Rosemarie Nagel. 2007. Rational reasoning or adaptive behavior? evidence from two- person beauty contest games.Evidence from Two- Person Beauty Contest Games (June 2007). Harvard NOM Research Paper, (01-09). Guiyang Hou, Wenqi Zhang, Yongliang Shen, Linjuan Wu,...

arXiv 2007
[2019]

Revisiting the evaluation of theory of mind through question answering. InProceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5872–5877, Hong Kong, China. Association for Computational Linguistics. Alan M Leslie, Ori Friedma...

work page 2019

Show all 12 references

[2021]

InProceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 1112–1125, Online and Punta Cana, Dominican Republic

MindCraft: Theory of mind modeling for situ- ated dialogue in collaborative tasks. InProceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 1112–1125, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics....

2021
[2022]

What day is it today?

Symmetric machine theory of mind. InIn- ternational Conference on Machine Learning, pages 19450–19466. PMLR. Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. 2024. Clever hans or neural theory of mind? stre...

2023 arXiv
[2023]

Joseph M Barnby, Ryan J Dean, Henry Burgess, Jeffrey Kim, Alessa K Teunisse, Lisa Mackenzie, Gail A Robinson, Peter Dayan, and Linda J Richards

Formalising social representation to explain psychiatric symptoms.Trends in cognitive sciences, 27(3):317–332. Joseph M Barnby, Ryan J Dean, Henry Burgess, Jeffrey Kim, Alessa K Teunisse, Lisa Mackenzie, Gail A Robinson, Peter Dayan, and Linda J Richards. 2022. Increased persu...

2022
[2024]

Nitay Alon, Lion Schulz, Jeffrey S

(mal) adaptive mentalizing in the cognitive hierarchy, and its link to paranoia.Computational Psychiatry, 8(1):159. Nitay Alon, Lion Schulz, Jeffrey S. Rosenschein, and Peter Dayan. 2023. A (dis-)information theory of revealed and unrevealed preferences: Emerging de- ception a...

2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [6]

InProceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 15959–15983, Bangkok, Thailand

ToMBench: Benchmarking theory of mind in large language models. InProceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 15959–15983, Bangkok, Thailand. Association for Computational Linguistics. Fiery Cushman. 2024. Computational social psychol- ogy.Annual Review of Psychology, 75(1):625–6...

work page 2024

[2] [9]

InProceedings of the 2024 Conference on Empir- ical Methods in Natural Language Processing, pages 19794–19809, Miami, Florida, USA

Perceptions to beliefs: Exploring precursory inferences for theory of mind in large language mod- els. InProceedings of the 2024 Conference on Empir- ical Methods in Natural Language Processing, pages 19794–19809, Miami, Florida, USA. Association for Computational Linguistics. Boaz Keysar, Shuhong Lin, and Dale J Barr. 2003. Limits on theory of mind use i...

arXiv 2024

[3] [11]

Probing the Robustness of Theory of Mind in Large Language Models

Theory of mind for multi-agent collabora- tion via large language models. InProceedings of the 2023 Conference on Empirical Methods in Natu- ral Language Processing, pages 180–192, Singapore. Association for Computational Linguistics. Xiaomeng Ma, Lingyu Gao, and Qihui Xu. 2023a. ToM- Challenges: A principle-guided dataset and diverse evaluation tasks for...

work page Pith review arXiv 2023

[4] [1985]

theory of mind

Does the autistic child have a “theory of mind”?Cognition, 21(1):37–46. Peter Bossaerts and Carsten Murawski. 2017. Com- putational complexity and human decision-making. Trends in cognitive sciences, 21(12):917–929. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, A...

arXiv 2017

[5] [2011]

InProceedings of the an- nual meeting of the cognitive science society, vol- ume 33

Bayesian theory of mind: Modeling joint belief-desire attribution. InProceedings of the an- nual meeting of the cognitive science society, vol- ume 33. Cristian-Paul Bara, Sky CH-Wang, and Joyce Chai

work page

[6] [2014]

Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gersten- berg, and Noah D

Theory of mind: did evolution fool us?PloS One, 9(2):e87619. Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gersten- berg, and Noah D. Goodman. 2023. Understanding social reasoning in language models with language models.Preprint, arXiv:2306.15448. Piotr J Gmytrasiewicz and Prashant Doshi. 2005. A framework for sequential planning in multi-agent settings.Jou...

arXiv 2023

[7] [2017]

Brit Grosskopf and Rosemarie Nagel

How can memory-augmented neural networks pass a false-belief task? InCogSci. Brit Grosskopf and Rosemarie Nagel. 2007. Rational reasoning or adaptive behavior? evidence from two- person beauty contest games.Evidence from Two- Person Beauty Contest Games (June 2007). Harvard NOM Research Paper, (01-09). Guiyang Hou, Wenqi Zhang, Yongliang Shen, Linjuan Wu,...

arXiv 2007

[8] [2019]

Revisiting the evaluation of theory of mind through question answering. InProceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5872–5877, Hong Kong, China. Association for Computational Linguistics. Alan M Leslie, Ori Friedma...

work page 2019