Pith. sign in

REVIEW 5 cited by

AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.19346 v2 pith:TDVNAJLA submitted 2024-10-25 cs.CL cs.CY

Xinyi Mou , Jingcong Liang , Jiayu Lin , Xinnong Zhang , Xiawei Liu , Shiyue Yang , Rong Ye , Lei Chen
show 3 more authors
Haoyu Kuang Xuanjing Huang Zhongyu Wei
This is my paper · ORCID
classification cs.CLcs.CY
keywords socialagentsscenariosagentsenselanguagebenchmarkingcomplexgoals
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large language models (LLMs) are increasingly leveraged to empower autonomous agents to simulate human beings in various fields of behavioral research. However, evaluating their capacity to navigate complex social interactions remains a challenge. Previous studies face limitations due to insufficient scenario diversity, complexity, and a single-perspective focus. To this end, we introduce AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios. Drawing on Dramaturgical Theory, AgentSense employs a bottom-up approach to create 1,225 diverse social scenarios constructed from extensive scripts. We evaluate LLM-driven agents through multi-turn interactions, emphasizing both goal completion and implicit reasoning. We analyze goals using ERG theory and conduct comprehensive experiments. Our findings highlight that LLMs struggle with goals in complex social scenarios, especially high-level growth needs, and even GPT-4o requires improvement in private information reasoning. Code and data are available at \url{https://github.com/ljcleo/agent_sense}.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    SAGE evaluates LLMs' higher-order social cognition by having a simulated 'sentient' user update its emotion score during multi-turn supportive dialogues, and reports that this score correlates with psychology-informed...

  2. Multi-Agent Simulator Drives Language Models for Legal Intensive Interaction

    cs.CL 2025-02 conditional novelty 6.0 of 10

    MASER generates synthetic interactive legal dialogues from real judgments and uses them to fine-tune LLMs that outperform GPT-4o on a new interactive complaint-drafting benchmark.

  3. PIORS: Personalized Intelligent Outpatient Reception based on Large Language Model with Multi-Agents Medical Scenario Simulation

    cs.CL 2024-11 conditional novelty 6.0 of 10

    A fine-tuned LLM receptionist trained on simulated patient conversations outperformed GPT-4o and other baselines in virtual outpatient triage tests.

  4. Auto-SLURP: A Benchmark Dataset for Evaluating Multi-Agent Frameworks in Smart Personal Assistant

    cs.CL 2025-04 conditional novelty 5.0 of 10

    Auto-SLURP is a new end-to-end benchmark for LLM multi-agent personal assistants, and the best current framework still fails a majority of user requests.

  5. From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents

    cs.CL 2024-12 conditional novelty 4.0 of 10

    A structured survey that categorizes LLM-based social simulation into individual, scenario, and society simulation, with associated methods, benchmarks, and observed trends.

Pith tools