Infusing Theory of Mind into Socially Intelligent LLM Agents

EunJeong Hwang; Giuseppe Carenini; Peter West; Vered Shwartz; Yuwei Yin

arxiv: 2509.22887 · v2 · submitted 2025-09-26 · 💻 cs.CL

Infusing Theory of Mind into Socially Intelligent LLM Agents

EunJeong Hwang , Yuwei Yin , Giuseppe Carenini , Peter West , Vered Shwartz This is my paper

Pith reviewed 2026-05-18 12:08 UTC · model grok-4.3

classification 💻 cs.CL

keywords Theory of MindLLM dialogue agentssocial intelligenceSotopia benchmarkmental state predictiondialogue lookaheadgoal-oriented reasoninglong-horizon adaptation

0 comments

The pith

LLMs that explicitly predict others' mental states during conversation achieve social goals more effectively than standard agents.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that adding Theory of Mind reasoning to LLM-based dialogue agents improves their ability to reach goals in interactive social settings. Simple prompting to generate mental states already helps, but the authors go further by training a dedicated agent called ToMA that pairs mental-state prediction with dialogue lookahead so the generated states directly support goal progress. On the Sotopia benchmark, ToMA outperforms baselines by showing more strategic planning across multiple turns while preserving partner relationships. A reader would care because this supplies a concrete mechanism for making conversational AI behave in ways that feel more socially competent over extended interactions.

Core claim

ToMAgent is created by training LLMs to generate mental-state representations of dialogue partners and then using lookahead over possible future turns to select those representations that best advance the agent's own goals; when evaluated on the Sotopia interactive social benchmark this produces agents that reason more strategically, adapt over longer horizons, and maintain better relationships than prompted or fine-tuned baselines.

What carries the argument

ToMAgent (ToMA), a dialogue agent whose training pairs Theory of Mind mental-state generation with lookahead planning to produce states that are maximally useful for goal achievement.

If this is right

ToMA outperforms a range of prompting and fine-tuning baselines on the Sotopia social evaluation benchmark.
The agent exhibits more strategic and goal-oriented reasoning behaviors across dialogue turns.
Long-horizon adaptation becomes possible while still preserving cooperative relationships with conversation partners.
Explicit mental-state modeling transfers from the training procedure to unseen interactive scenarios without additional selection steps.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same lookahead-plus-ToM training loop could be applied to multi-party conversations or to tasks that require tracking hidden information over many steps.
If mental-state lookahead proves reliable, future agents might use it as an internal planning layer rather than relying solely on next-token prediction.
The approach suggests that social competence in LLMs may improve more from structured prediction of hidden variables than from simply scaling model size or data volume.

Load-bearing premise

Mental states produced by dialogue lookahead are the most useful possible ones for reaching goals and this usefulness carries over directly to the Sotopia evaluation setting.

What would settle it

Running the same Sotopia evaluation and finding that ToMA shows no improvement in goal completion rates or no increase in strategic reasoning behaviors compared with strong baselines would falsify the central effectiveness claim.

read the original abstract

Theory of Mind (ToM)-an understanding of the mental states of others-is a key aspect of human social intelligence, yet, chatbots and LLM-based social agents do not typically integrate it. In this work, we demonstrate that LLMs that explicitly use ToM get better at dialogue, achieving goals more effectively. After showing that simply prompting models to generate mental states between dialogue turns already provides significant benefit, we further introduce ToMAgent (ToMA), a ToM-focused dialogue agent. ToMA is trained by pairing ToM with dialogue lookahead to produce mental states that are maximally useful for achieving dialogue goals. Experiments on the Sotopia interactive social evaluation benchmark demonstrate the effectiveness of our method over a range of baselines. Comprehensive analysis shows that ToMA exhibits more strategic, goal-oriented reasoning behaviors, which enable long-horizon adaptation, while maintaining better relationships with their partners. Our results suggest a step forward in integrating ToM for building socially intelligent LLM agents.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

ToM prompting helps on social dialogues and lookahead training adds gains on Sotopia, but the paper needs an ablation to show the training step is doing the real work.

read the letter

The main point is that this paper shows LLMs get better at multi-turn social goals when they explicitly generate Theory of Mind states, and training those states with dialogue lookahead produces agents that score higher on Sotopia for both task success and relationship maintenance. Simple prompting for mental states already moves the needle, and the full ToMAgent procedure pushes it further with more strategic, long-horizon reasoning in the analysis section. That combination is the concrete new piece: a training recipe that pairs ToM generation with lookahead to shape the states for goal utility. The behavioral analysis of how the agents plan and adapt is the part that actually adds value beyond the numbers. It gives readers a sense of why the method might work in practice rather than just claiming another benchmark win. The soft spot is the missing control for the lookahead component. The stress-test concern holds up on the evidence given: without an ablation that keeps ToM prompting but removes the lookahead optimization, it is hard to separate the contribution of the mental-state representation from whatever selection or distribution effects the training procedure introduces. The abstract also skips statistical details and split information, which leaves the robustness of the gains open to question. This work is aimed at people building interactive social agents who already follow the ToM-in-LLMs line of research. Readers who want a usable training trick on an established benchmark will get something from it. It shows clear engagement with prior work and reports new results on a relevant evaluation, so it deserves a serious referee. I would send it for review and ask specifically for the lookahead ablation and the missing stats.

Referee Report

2 major / 2 minor

Summary. The paper claims that explicitly incorporating Theory of Mind (ToM) into LLM-based dialogue agents improves goal achievement in social interactions. After showing gains from simply prompting models to generate mental states between turns, the authors introduce ToMAgent (ToMA), trained by pairing ToM generation with dialogue lookahead to produce mental states that are maximally useful for dialogue goals. Experiments on the Sotopia interactive benchmark demonstrate that ToMA outperforms a range of baselines, with analysis indicating more strategic, goal-oriented reasoning, long-horizon adaptation, and better relationship maintenance.

Significance. If the performance gains on Sotopia are robust and causally attributable to the explicit ToM component, the work would advance the development of socially intelligent LLM agents by offering a practical method for mental-state modeling that supports extended, goal-directed dialogue. The behavioral analysis of reasoning patterns provides additional value for understanding mechanisms of social adaptation in agents.

major comments (2)

[§3 (ToMA Training)] §3 (ToMA Training): The training procedure pairs ToM with dialogue lookahead to generate 'maximally useful' mental states for goal achievement. No ablation is reported that retains the ToM component while removing lookahead optimization. This is load-bearing because any Sotopia gains could reflect training-time selection effects or distribution shift rather than intrinsic benefits of the mental-state representation itself.
[§5 (Sotopia Experiments)] §5 (Sotopia Experiments): The reported gains over baselines lack details on evaluation episode count, variance across runs, data splits, or statistical significance tests. Without these, it is difficult to rule out post-hoc selection or noise as explanations for the observed strategic behaviors and long-horizon adaptation.

minor comments (2)

[Abstract and §3] The abstract and methods could more explicitly distinguish the prompting-only baseline from the full lookahead-trained ToMA to clarify incremental contributions.
[Figures] Figure captions describing reasoning traces would benefit from additional context on how mental states are visualized and linked to dialogue turns.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their thoughtful and constructive comments on our manuscript. We provide point-by-point responses to the major comments below and indicate the revisions we intend to make.

read point-by-point responses

Referee: [§3 (ToMA Training)] §3 (ToMA Training): The training procedure pairs ToM with dialogue lookahead to generate 'maximally useful' mental states for goal achievement. No ablation is reported that retains the ToM component while removing lookahead optimization. This is load-bearing because any Sotopia gains could reflect training-time selection effects or distribution shift rather than intrinsic benefits of the mental-state representation itself.

Authors: We agree that demonstrating the specific contribution of the ToM representation independent of the lookahead training objective would strengthen the causal claims. Our prompted ToM experiments (without training) already indicate benefits from mental state generation. To further address this, we will add an ablation in the revised manuscript where we fine-tune the model on ToM generation using a standard next-token prediction loss without the lookahead component, and evaluate it on Sotopia. This will help isolate whether the gains stem from the mental-state modeling or from the optimization procedure. revision: yes
Referee: [§5 (Sotopia Experiments)] §5 (Sotopia Experiments): The reported gains over baselines lack details on evaluation episode count, variance across runs, data splits, or statistical significance tests. Without these, it is difficult to rule out post-hoc selection or noise as explanations for the observed strategic behaviors and long-horizon adaptation.

Authors: We will revise the experimental section to provide the missing details on the evaluation setup. This includes specifying the total number of episodes used for evaluation, reporting standard deviations or variances across multiple runs, clarifying the data splits employed, and including statistical significance tests for the performance differences. These additions will allow readers to better assess the robustness of the results and rule out explanations based on noise or selection effects. revision: yes

Circularity Check

0 steps flagged

No significant circularity; central claims rest on external benchmark evaluation

full rationale

The paper's derivation proceeds from prompting LLMs to generate mental states, to training ToMA by pairing ToM with dialogue lookahead for goal-oriented states, to empirical evaluation on the independent Sotopia benchmark against baselines. No equations, fitted parameters renamed as predictions, or self-citations are load-bearing in the provided text. The reported improvements in strategic reasoning and long-horizon adaptation are measured externally rather than reducing by construction to the training objective itself. The method is self-contained against the external benchmark with no self-definitional or uniqueness-imported reductions exhibited.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The work is empirical and relies on the assumption that ToM-style intermediate reasoning is a useful inductive bias for social dialogue; no new mathematical axioms or invented physical entities are introduced.

axioms (1)

domain assumption Explicit generation of mental states between dialogue turns improves downstream goal achievement
Invoked in the first experiment described in the abstract as the basis for further training.

pith-pipeline@v0.9.0 · 5701 in / 1146 out tokens · 39450 ms · 2026-05-18T12:08:22.775028+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

TOMA is trained by pairing ToM with dialogue lookahead to produce mental states that are maximally useful for achieving dialogue goals... fine-tune the model to generate both the latent mental states and utterances
IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We retain all pairs with an average score ≥9... LCE(ϕ) = E[CE(m⋆,ϕ(H)) + CE(u⋆,ϕ(H,m⋆))]

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Can LLMs Think Like Consumers? Benchmarking Crowd-Level Reaction Reconstruction with ConsumerSimBench
cs.CL 2026-05 unverdicted novelty 7.0

ConsumerSimBench evaluates 13 LLMs on reconstructing crowd reactions from 1,553 Chinese social-media topics using 23,122 auditable yes-no criteria, finding maximum coverage of 47.8% by Gemini-3.1-Pro.
Does Theory of Mind Improvement Really Benefit Human-AI Interactions? Empirical Findings from Interactive Evaluations
cs.AI 2026-04 conditional novelty 6.0

Improvements in LLM Theory of Mind on static benchmarks do not reliably improve performance in dynamic, first-person human-AI interactions across goal-oriented and experience-oriented tasks.
Agents of Chaos
cs.AI 2026-02 unverdicted novelty 6.0

An exploratory red-teaming study documents eleven cases of security, privacy, and governance failures in autonomous language-model agents with tool access and persistent memory.