REVIEW 7 cited by
LLMs achieve adult human performance on higher-order theory of mind tasks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
This paper examines the extent to which large language models (LLMs) have developed higher-order theory of mind (ToM); the human ability to reason about multiple mental and emotional states in a recursive manner (e.g. I think that you believe that she knows). This paper builds on prior work by introducing a handwritten test suite -- Multi-Order Theory of Mind Q&A -- and using it to compare the performance of five LLMs to a newly gathered adult human benchmark. We find that GPT-4 and Flan-PaLM reach adult-level and near adult-level performance on ToM tasks overall, and that GPT-4 exceeds adult performance on 6th order inferences. Our results suggest that there is an interplay between model size and finetuning for the realisation of ToM abilities, and that the best-performing LLMs have developed a generalised capacity for ToM. Given the role that higher-order ToM plays in a wide range of cooperative and competitive human behaviours, these findings have significant implications for user-facing LLM applications.
Forward citations
Cited by 7 Pith papers
-
Inverse Theory of Mind Modeling for Content Recommendation: From Web Browsing to Dynamic Intelligent Interfaces
A five-stage LLM pipeline infers explainable beliefs and personas from browsing traces, and these inferred profiles match or beat interview-derived profiles on several downstream prediction tasks.
-
Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis
Applying IIT 3.0/4.0 Φ estimates to LLM hidden-state sequences from Theory of Mind tests finds no robust statistical evidence of 'consciousness' phenomena, with span representations usually explaining score difference...
-
BeliefNest: A Joint Action Simulator for Embodied Agents with Theory of Mind
BeliefNest represents nested beliefs as hierarchical Minecraft simulators and shows LLM agents can use them to pass false-belief tasks.
-
Position: Theory of Mind Benchmarks are Broken for Large Language Models
The paper proposes that LLM theory-of-mind evaluation should measure functional adaptation to partners, not just literal prediction of their behavior, and shows the two can diverge sharply in simple games.
-
Xinyu AI Search: Enhanced Relevance and Comprehensive Results with Rich Answer Presentations
Xinyu, an integrated generative AI search engine with query decomposition, multi-source retrieval, and rich answer presentation, outperforms eight existing technologies in human evaluations.
-
Theory of Mind in Large Language Models: Assessment and Enhancement
A survey of recent story-based Theory of Mind benchmarks and enhancement strategies for large language models, organized by mental state coverage and method type.
-
A Survey of Theory of Mind in Large Language Models: Evaluations, Representations, and Safety Risks
A narrative review of behavioral and representational Theory of Mind in LLMs, with a taxonomy of safety risks and mitigation directions.
Discussion (0). Continue with ORCID to comment.