REVIEW 6 cited by
Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The escalating debate on AI's capabilities warrants developing reliable metrics to assess machine "intelligence". Recently, many anecdotal examples were used to suggest that newer large language models (LLMs) like ChatGPT and GPT-4 exhibit Neural Theory-of-Mind (N-ToM); however, prior work reached conflicting conclusions regarding those abilities. We investigate the extent of LLMs' N-ToM through an extensive evaluation on 6 tasks and find that while LLMs exhibit certain N-ToM abilities, this behavior is far from being robust. We further examine the factors impacting performance on N-ToM tasks and discover that LLMs struggle with adversarial examples, indicating reliance on shallow heuristics rather than robust ToM abilities. We caution against drawing conclusions from anecdotal examples, limited benchmark testing, and using human-designed psychological tests to evaluate models.
Forward citations
Cited by 6 Pith papers
-
LLM Agents for Deliberative Collaboration: A Study on Joint Decision Making Under Partial Observability
A benchmark for LLM agents in partially observable joint decision-making reveals that deliberation challenges current models but can enable reflection and error correction.
-
Small LLMs Do Not Learn a Generalizable Theory of Mind via Reinforcement Learning
Reinforcement learning with verifiable rewards makes a small LLM overfit theory-of-mind benchmarks, not acquire a generalizable theory of mind.
-
MotiveBench: How Far Are We From Human-Like Motivational Reasoning in Large Language Models?
MotiveBench shows that large language models still lag human consensus on motivational reasoning, with GPT-4o scoring 80.89 percent and chain-of-thought prompting usually reducing accuracy.
-
SocialMaze: A Benchmark for Evaluating Social Reasoning in Large Language Models
SocialMaze is a six-task benchmark that claims to evaluate LLM social reasoning along deep reasoning, dynamic interaction, and information uncertainty dimensions.
-
Training Language Models for Social Deduction with Multi-Agent Reinforcement Learning
A listening-plus-speaking training scheme, where language model agents are rewarded for influencing teammates' beliefs about the impostor, doubles crewmate win rates in a simulated Among Us game compared to RL alone.
-
Towards Machine Theory of Mind with Large Language Model-Augmented Inverse Planning
An LLM-augmented Bayesian inverse planning model, LAIP, generates hypotheses and action likelihoods, then uses Bayes' rule to infer agent preferences, outperforming LLM-only baselines.
Discussion (0). Continue with ORCID to comment.