The reviewed record of science sign in
Pith

arxiv: 2403.13679 · v4 · pith:G4EC6257 · submitted 2024-03-20 · cs.CL

SocialBench: Sociality Evaluation of Role-Playing Conversational Agents

Hongzhan Chen , Hehong Chen , Ming Yan , Wenshen Xu , Xing Gao , Weizhou Shen , Xiaojun Quan , Chenliang Li
show 3 more authors
Ji Zhang Fei Huang Jingren Zhou
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:G4EC6257record.jsonopen to challenge →

classification cs.CL
keywords agentsconversationalrole-playingbenchmarksocialbenchgroupsocialassessing
0
0 comments X
read the original abstract

Large language models (LLMs) have advanced the development of various AI conversational agents, including role-playing conversational agents that mimic diverse characters and human behaviors. While prior research has predominantly focused on enhancing the conversational capability, role-specific knowledge, and stylistic attributes of these agents, there has been a noticeable gap in assessing their social intelligence. In this paper, we introduce SocialBench, the first benchmark designed to systematically evaluate the sociality of role-playing conversational agents at both individual and group levels of social interactions. The benchmark is constructed from a variety of sources and covers a wide range of 500 characters and over 6,000 question prompts and 30,800 multi-turn role-playing utterances. We conduct comprehensive evaluations on this benchmark using mainstream open-source and closed-source LLMs. We find that agents excelling in individual level does not imply their proficiency in group level. Moreover, the behavior of individuals may drift as a result of the influence exerted by other agents within the group. Experimental results on SocialBench confirm its significance as a testbed for assessing the social interaction of role-playing conversational agents. The benchmark is publicly accessible at https://github.com/X-PLUG/SocialBench.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Character Beyond Speech: Leveraging Role-Playing Evaluation in Audio Large Language Models via Reinforcement Learning

    cs.LG 2026-04 unverdicted novelty 7.0

    RoleJudge is a multidimensional evaluation framework for speech-character alignment in audio LLMs, backed by the RoleChat dataset and multi-stage RL training with standard alignment to reduce reward issues.

  2. AudioRole: An Audio Dataset for Character Role-Playing in Large Language Models

    cs.SD 2025-09 unverdicted novelty 7.0

    AudioRole provides 1M+ character-grounded audio-text dialogues from TV series plus ARP-Eval to train and measure audio role-playing models, with ARP-Model showing 0.31 acoustic and 0.36 content personalization scores.

  3. Inertia in Moral and Value Judgments of Large Language Models

    cs.CL 2024-08 unverdicted novelty 4.0

    LLMs exhibit persistent inertia in value orientations, with harm avoidance and fairness remaining skewed across persona prompts.

  4. Intelligent Agents with Emotional Intelligence: Current Trends, Challenges, and Future Prospects

    cs.HC 2025-10 unverdicted novelty 2.0

    A holistic survey of affective computing for intelligent agents covering emotion understanding via multimodal data, affective cognition, emotional expression synthesis, key challenges, and future directions emphasizin...