The reviewed record of science sign in
Pith

arxiv: 2310.17976 · v4 · pith:U26ZUTHY · submitted 2023-10-27 · cs.CL

InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews

Xintao Wang , Yunze Xiao , Jen-tse Huang , Siyu Yuan , Rui Xu , Haoran Guo , Quan Tu , Yaying Fei
show 5 more authors
Ziang Leng Wei Wang Jiangjie Chen Cheng Li Yanghua Xiao
This is my paper

Reviewed by Pithpith:U26ZUTHYopen to challenge →

classification cs.CL
keywords rpascharactersincharacteragentsfidelitypersonalitiespersonalitypsychological
0
0 comments X
read the original abstract

Role-playing agents (RPAs), powered by large language models, have emerged as a flourishing field of applications. However, a key challenge lies in assessing whether RPAs accurately reproduce the personas of target characters, namely their character fidelity. Existing methods mainly focus on the knowledge and linguistic patterns of characters. This paper, instead, introduces a novel perspective to evaluate the personality fidelity of RPAs with psychological scales. Overcoming drawbacks of previous self-report assessments on RPAs, we propose InCharacter, namely Interviewing Character agents for personality tests. Experiments include various types of RPAs and LLMs, covering 32 distinct characters on 14 widely used psychological scales. The results validate the effectiveness of InCharacter in measuring RPA personalities. Then, with InCharacter, we show that state-of-the-art RPAs exhibit personalities highly aligned with the human-perceived personalities of the characters, achieving an accuracy up to 80.7%.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Character Beyond Speech: Leveraging Role-Playing Evaluation in Audio Large Language Models via Reinforcement Learning

    cs.LG 2026-04 unverdicted novelty 7.0

    RoleJudge is a multidimensional evaluation framework for speech-character alignment in audio LLMs, backed by the RoleChat dataset and multi-stage RL training with standard alignment to reduce reward issues.

  2. MARS-SQL: A multi-agent reinforcement learning framework for Text-to-SQL

    cs.CL 2025-11 unverdicted novelty 5.0

    MARS-SQL trains a multi-agent RL system with ReAct-style interaction and generative validation to produce SQL queries, reaching 77.84% execution accuracy on BIRD dev and 89.75% on Spider test.

  3. Inertia in Moral and Value Judgments of Large Language Models

    cs.CL 2024-08 unverdicted novelty 4.0

    LLMs exhibit persistent inertia in value orientations, with harm avoidance and fairness remaining skewed across persona prompts.