REVIEW 4 major objections 5 minor 24 references
Small LLMs Do Not Learn a Generalizable Theory of Mind via Reinforcement Learning
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read For a small LLM, RL with verifiable rewards produces benchmark-specific mastery of Theory of Mind tasks, not a generalizable capability.
desk verdict A useful, credible negative result on RLVR for theory of mind, but the 'hacking' explanation for the largest held-out gain is an assumption rather than a demonstrated finding, and the headline claim needs softer wording and more statistical support. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The experimental machinery is RL with verifiable rewards (RLVR) run through the REINFORCE++ algorithm on Qwen2.5-7B-Instruct, using a rule-based reward that combines a format reward for enclosing reasoning in <think> and <answer> tags with a correctness reward for matching the ground-truth answer. Seven training configurations combine 900-sample subsets of HiToM, FANToM, and ExploreToM, while OpenToM, FANToM's list-generation tasks, and fourth-order HiToM are held out as generalization probes. The diagnostic that carries the argument is the inverted difficulty curve: a model trained only on lower-order HiToM questions performs best on the unseen, most complex fourth-order questions, which the authors take as evidence of dataset artifact exploitation rather than recursive belief reasoning.
What would settle it
Reproduce the same RLVR protocol on a different 7–8B model (or across several seeds) and test on OpenToM: a robust, seed-consistent improvement well above the 59.2% chain-of-thought baseline would falsify the paper's central claim, and so would eliminating the HiToM template artifacts and seeing the inverted difficulty curve disappear.
Extended reading notes
Core claim
The central claim is that, for small LLMs, applying RLVR to current Theory of Mind benchmarks does not lead to a genuine, general-purpose Theory of Mind. Training Qwen2.5-7B-Instruct with REINFORCE++ on 900-sample subsets of HiToM, FANToM, and ExploreToM lifted in-domain accuracy by 22 to 65 percentage points over baselines, yet held-out OpenToM accuracy stayed in a tight 56.9% to 61.8% band, essentially unchanged from the 59.2% chain-of-thought baseline. The learned skill also failed to transfer to list-format FANToM tasks, and a model trained on first- through third-order HiToM questions scored best on the held-out fourth-order questions, inverting the expected difficulty curve. The paper concludes that RLVR produces narrow, brittle pattern matching rather than abstract social intelligence.
Load-bearing premise
The headline conclusion assumes that one experimental configuration—a single 7B model, REINFORCE++, 900 training samples per dataset, 10 epochs, and one run—stands for the whole class of small LLMs trained with verifiable-reward RL.
Editorial extensions
If this is right
- In-domain accuracy gains from RLVR on ToM benchmarks should not be interpreted as evidence that the model acquired Theory of Mind.
- The success of RLVR in mathematical and logical reasoning does not automatically carry over to social reasoning for small models.
- Prolonged RL training can degrade out-of-distribution performance even while in-distribution accuracy climbs.
- Benchmarks with templated structure can be 'hacked' so that models appear to master higher-order reasoning they were never trained on.
- Adding more training benchmarks did not rescue generalization, since even the model trained on all three datasets failed to beat the chain-of-thought baseline on OpenToM.
Reading between the lines
- The paper's configuration space is narrow: one 7B model, one RL algorithm, 900 samples per dataset, 10 epochs, and a single run. The conclusion could be tested and sharpened by varying all five factors.
- A natural next experiment is to reward intermediate belief-state reasoning or explanation quality rather than only final answers; if OOD transfer then improves, the bottleneck is the reward signal, not the model class.
- The same benchmark-overfitting pattern may appear when RLVR is applied to other 'soft' reasoning domains, such as pragmatic inference or moral judgment, where correct answers are less verifiable than in math.
- If the finding holds across models, ToM evaluation will need adversarial held-out formats, multiple output styles, and process-level checks to keep RL from optimizing dataset statistics.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates whether small LLMs (Qwen2.5-7B-Instruct) can acquire a generalizable Theory of Mind through reinforcement learning with verifiable rewards (RLVR). The authors train models on various combinations of three ToM benchmarks (HiToM, FANToM, ExploreToM) and evaluate on held-out tasks (OpenToM, FANToM list-format tasks, and HiToM fourth-order examples). They report large in-distribution gains after RL training, no meaningful improvement on the held-out OpenToM and FANToM list tasks, and an inverted difficulty curve on HiToM in which a model trained on orders 1-3 performs best on the unseen fourth-order task. The paper interprets the inverted curve as evidence of statistical "hacking" of dataset artifacts and concludes that the learned behavior is narrow, brittle pattern matching rather than a genuine, generalizable ToM.
Significance. If the central claim is substantiated, the paper is a valuable cautionary result for the RLVR post-training literature: it would show that benchmark-specific reward optimization can produce large in-domain gains on social reasoning tasks without inducing transferable abstract competence. The paper's strengths are its systematic multi-regimen comparison across seven training configurations, the use of multiple held-out datasets spanning different generalization axes, and the explicit reporting of training-dynamics divergence between in-distribution and out-of-distribution accuracy (Figure 2). These design choices make the core empirical pattern—large in-domain improvement with flat OOD accuracy—internally consistent and worth communicating to the community. However, the strongest negative conclusion depends on an interpretive step that is not directly supported by the presented evidence, and the experimental basis is a single model, a single run, and small evaluation subsets; both issues limit the strength of the claims as currently stated.
major comments (4)
- [§4.2.3, Table 2, §5] The largest out-of-distribution gain in the paper is the HiToM fourth-order result: the Hi model trained on orders 1-3 reaches 94.2% on unseen O4, versus 34.2% for the CoT baseline and 35.8% for the untrained baseline. The paper discounts this result as evidence of "hacking" dataset artifacts, but no direct evidence is provided for that interpretation. There is no analysis isolating which surface features the model exploited, no ablation with corrupted or counterfactual templates, and no demonstration that the O4 gain disappears when template statistics are controlled. The inverted difficulty curve is suggestive, but the baseline difficulty curve is itself non-monotonic (O3 29.2%, O4 34.2%), so the inversion alone does not establish artifact exploitation. Because the conclusion that RLVR produces only narrow pattern matching depends on excluding this positive OOD transfer as an artifact, this is a load-bearing unverified step; the authors should either provide artifact analyses or substantially soften the hacking claim.
- [§4.1] All conclusions are based on a single base model (Qwen2.5-7B-Instruct), a single RL algorithm (REINFORCE++), one data budget (900 training samples per dataset, 10 epochs), and one run per configuration. No confidence intervals, standard errors, or significance tests are reported for any accuracy number. Consequently, differences such as the OpenToM scores clustered between 56.9% and 61.8% around the CoT baseline of 59.2% cannot be distinguished from sampling noise, and the generalized statement that "small LLMs" fail to learn generalizable ToM rests on a single point in configuration space. The authors should either run multiple seeds and report variability, or restrict the claim to the specific configuration studied.
- [§3.2, §4.2.2] The OpenToM held-out evaluation uses only 100 examples per subtask, with a single model evaluation per checkpoint. With 100 binary or multiple-choice items, the standard error of an accuracy estimate is roughly 3-5 percentage points, which is the same magnitude as the observed differences between trained models and baselines on OpenToM. The statement that trained models "did not significantly improve" is therefore not established without a significance test or confidence interval. The paper should report bootstrap intervals or per-example metrics across seeds, or explicitly acknowledge that the OpenToM comparison is underpowered.
- [§4.1] The paper selects checkpoints by best validation performance over 10 epochs. This procedure can inflate reported accuracy relative to a fixed training schedule, and it interacts with the small evaluation sets: a single lucky validation sample can pick an unrepresentative checkpoint. Since all reported numbers are single-run point estimates, the checkpoint-selection rule adds another source of variance that is not accounted for in the claims. The authors should either report the selected epoch and the validation scores, or evaluate at the final epoch as well, to show that the central pattern does not depend on the selection rule.
minor comments (5)
- [§3.2] The FANToM list-response tasks are cited as "(Xu et al., 2024)", but these tasks come from the FANToM benchmark of Kim et al. (2023). The citation should be corrected.
- [Table 1] The column headers abbreviate datasets inconsistently (e.g., "ExpToM" versus "ExploreToM" in the text), and the caption does not define the baseline row's exact prompt setting (zero-shot vs. chain-of-thought). Please make the table self-contained.
- [Figure 2] The learning curves plot average accuracies without error bars or per-epoch variability, yet the text claims "no significant changes" on out-set accuracies. Adding confidence bands or at least noting the absence of replication would make the figure more honest.
- [Abstract, §5] The abstract states that prolonged RL training "leads to models hacking the statistical patterns" as an established result, but the hacking claim is only an interpretation of the inverted difficulty curve. The abstract should reflect the weaker, evidence-backed claim that the models exploit dataset-specific cues, with the hacking mechanism as a hypothesis.
- [§4.3.1] The phrase "out-sets remain stagnant" and the following sentence are syntactically incomplete. Please revise for readability.
Circularity Check
No significant circularity: central claim rests on held-out evaluation, not on fitted parameters or self-citations.
full rationale
The paper's derivation chain is empirical: it trains Qwen2.5-7B-Instruct with REINFORCE++ on 900-sample subsets of HiToM, FANToM, and ExploreToM, selects checkpoints by validation performance, and measures held-out OpenToM, FANToM-list, and HiToM fourth-order accuracy. No conclusion quantity is defined in terms of its own evidence. The out-of-distribution tests were excluded from training and validation by design (Section 3.2), so in-domain gains and OOD stagnation are independent measurements rather than fitted-parameter predictions. The 'hacking' label attached to the inverted HiToM difficulty curve (Section 5) is an interpretation of observed accuracy patterns, not a mathematical reduction; even if that interpretation is contestable, it is not circular. The only self-citations (Sarangi et al., 2025) appear in related-work and introductory motivation and are not load-bearing for the RLVR transfer results. The reward function is a standard rule-based format-plus-correctness check and does not encode the paper's conclusion. No self-definitional step, fitted-input-as-prediction step, or self-citation chain appears, so the correct finding is no significant circularity.
Assumptions & free parameters
free parameters (5)
- Per-dataset training sample size =
900 train / 300 validation / 300 test
- RL hyperparameters =
batch=8, rollouts=8, lr=5e-7, temperature=0.6, 10 epochs
- ExploreToM genuine-ToM share =
70% ToM, 30% simpler mental-state tracking
- HiToM factual share =
10% factual examples
- OpenToM evaluation subset =
100 examples per task, label-balanced
assumptions (4)
- domain assumption Held-out benchmarks (OpenToM, FANToM List, HiToM O4) are valid measures of generalizable ToM ability.
- domain assumption Qwen2.5-7B-Instruct is representative of small LLMs for the purpose of the conclusion.
- domain assumption Accuracy on benchmark questions reflects ToM ability rather than unrelated linguistic regularities.
- ad hoc to paper Exploitable statistical artifacts exist in templated HiToM data and dominate higher-order performance.
Cite this review
Pith. "Pith review of Small LLMs Do Not Learn a Generalizable Theory of Mind via Reinforcement Learning." pith.science (2026). https://pith.science/paper/3JMZEFV2
@misc{pith2026250715788,
author = {Pith},
title = {Pith review of: Small LLMs Do Not Learn a Generalizable Theory of Mind via Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/3JMZEFV2}},
note = {Machine review of arXiv:2507.15788}
}
read the original abstract
Recent advancements in large language models (LLMs) have demonstrated emergent capabilities in complex reasoning, largely spurred by rule-based Reinforcement Learning (RL) techniques applied during the post-training. This has raised the question of whether similar methods can instill more nuanced, human-like social intelligence, such as a Theory of Mind (ToM), in LLMs. This paper investigates whether small-scale LLMs can acquire a robust and generalizable ToM capability through RL with verifiable rewards (RLVR). We conduct a systematic evaluation by training models on various combinations of prominent ToM datasets (HiToM, ExploreToM, FANToM) and testing for generalization on held-out datasets (e.g., OpenToM). Our findings indicate that small LLMs struggle to develop a generic ToM capability. While performance on in-distribution tasks improves, this capability fails to transfer to unseen ToM tasks with different characteristics. Furthermore, we demonstrate that prolonged RL training leads to models ``hacking'' the statistical patterns of the training datasets, resulting in significant performance gains on in-domain data but no change, or degradation of performance on out-of-distribution tasks. This suggests the learned behavior is a form of narrow overfitting rather than the acquisition of a true, abstract ToM capability.
Figures
Reference graph
Works this paper leans on
-
[1]
Simon Baron-Cohen. 1995. https://doi.org/10.7551/mitpress/4635.001.0001 Mindblindness: An Essay on Autism and Theory of Mind . The MIT Press
-
[2]
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...
arXiv 2025
-
[3]
Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D. Goodman. 2023. https://arxiv.org/abs/2306.15448 Understanding social reasoning in language models with language models . Preprint, arXiv:2306.15448
arXiv 2023
-
[4]
Jian Hu, Jason Klein Liu, and Wei Shen. 2025. https://arxiv.org/abs/2501.03262 Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models . Preprint, arXiv:2501.03262
arXiv 2025
-
[5]
Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Bras, Gunhee Kim, Yejin Choi, and Maarten Sap. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.890 FANT o M : A benchmark for stress-testing machine theory of mind in interactions . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14397--14413, Singapore. Assoc...
-
[6]
Michal Kosinski. 2023. https://arxiv.org/abs/2302.02083 Theory of mind may have spontaneously emerged in large language models . Preprint, arXiv:2302.02083
arXiv 2023
-
[7]
Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi...
arXiv 2025
-
[8]
Matthew Le, Y-Lan Boureau, and Maximilian Nickel. 2019. https://doi.org/10.18653/v1/D19-1598 Revisiting the evaluation of theory of mind through question answering . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5872-...
Show all 24 references
-
[9]
Yi-Long Lu, Chunhui Zhang, Jiajun Song, Lifeng Fan, and Wei Wang. 2025. https://arxiv.org/abs/2504.01698 Do theory of mind benchmarks need explicit human-like reasoning in language models? Preprint, arXiv:2504.01698
2025 arXiv
-
[10]
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...
2022 arXiv
-
[11]
David Premack and Guy Woodruff. 1978. https://doi.org/10.1017/S0140525X00076512 Does the chimpanzee have a theory of mind? Behavioral and Brain Sciences, 1(4):515--526
1978 doi
-
[12]
Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, S. M. Ali Eslami, and Matthew Botvinick. 2018. https://proceedings.mlr.press/v80/rabinowitz18a.html Machine theory of mind . In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Procee...
2018
-
[13]
Sneheel Sarangi, Maha Elgarf, and Hanan Salam. 2025. https://arxiv.org/abs/2501.09056 Decompose-tom: Enhancing theory of mind reasoning in large language models through simulation and task decomposition . Preprint, arXiv:2501.09056
2025 arXiv
-
[14]
Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. 2023. https://doi.org/10.18653/v1/2023.acl-long.780 Minding language models ' (lack of) theory of mind: A plug-and-play multi-character belief tracker . In Proceedings of the 61st Annual Meeti...
2023 doi
-
[15]
Melanie Sclar, Jane Yu, Maryam Fazel-Zarandi, Yulia Tsvetkov, Yonatan Bisk, Yejin Choi, and Asli Celikyilmaz. 2024. https://arxiv.org/abs/2412.12175 Explore theory of mind: Program-guided adversarial data generation for theory of mind reasoning . Preprint, arXiv:2412.12175
2024 arXiv
-
[16]
Karen Shanton and Alvin Goldman. 2010. https://doi.org/10.1002/wcs.33 Simulation theory . WIREs Cognitive Science, 1(4):527–538
2010 doi
-
[17]
Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. 2023. https://arxiv.org/abs/2305.14763 Clever hans or neural theory of mind? stress testing social reasoning in large language models . Preprint, arXiv:2305.14763
2023 arXiv
-
[18]
Tomer Ullman. 2023. https://arxiv.org/abs/2302.08399 Large language models fail on trivial alterations to theory-of-mind tasks . Preprint, arXiv:2302.08399
2023 arXiv
-
[19]
Alex Wilf, Sihyun Lee, Paul Pu Liang, and Louis-Philippe Morency. 2024. https://aclanthology.org/2024.acl-long.451 Think twice: Perspective-taking improves large language models ' theory-of-mind capabilities . In Proceedings of the 62nd Annual Meeting of the Association for Co...
2024
-
[20]
Yufan Wu, Yinghui He, Yilin Jia, Rada Mihalcea, Yulong Chen, and Naihao Deng. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.717 Hi- T o M : A benchmark for evaluating higher-order theory of mind reasoning in large language models . In Findings of the Association for Co...
2023 doi
-
[21]
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. 2025. https://arxiv.org/abs/2502.14768 Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning . Preprint, arXiv:2502.14768
2025 arXiv
-
[22]
Hainiu Xu, Runcong Zhao, Lixing Zhu, Jinhua Du, and Yulan He. 2024. https://aclanthology.org/2024.acl-long.466 O pen T o M : A comprehensive benchmark for evaluating theory-of-mind reasoning capabilities of large language models . In Proceedings of the 62nd Annual Meeting of t...
2024
-
[23]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[24]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.