PersonaGym: Evaluating Persona Agents and LLMs
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:GGCSNKYDrecord.jsonopen to challenge →
read the original abstract
Persona agents, which are LLM agents conditioned to act according to an assigned persona, enable contextually rich and user aligned interactions across domains like education and healthcare. However, evaluating how faithfully these agents adhere to their personas remains a significant challenge, particularly in free-form settings that demand consistency across diverse, persona-relevant environments. We introduce PersonaGym, the first dynamic evaluation framework for persona agents, and PersonaScore, a human-aligned automatic metric grounded in decision theory that enables comprehensive large-scale evaluation. Our evaluation of 10 leading LLMs across 200 personas and 10,000 questions reveals significant advancement opportunities. For example, GPT-4.1 had the exact same PersonaScore as LLaMA-3-8b despite being a more recent and advanced closed source model. Importantly, increased model size and complexity do not necessarily enhance persona agent capabilities, underscoring the need for algorithmic and architectural innovation toward faithful, performant persona agents.
This paper has not been read by Pith yet.
Forward citations
Cited by 15 Pith papers
-
AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility
AgentBeats implements agentified evaluation of diverse AI agents through standardized interfaces, validated at scale in a five-month competition with 298 judges and 467 subjects plus a coding case study.
-
Beyond Recall: Behavioral Specification as an Interpretive Layer for AI Personalization
A Behavioral Specification interpretive layer improves representational accuracy for AI personalization by compressing user data into patterns, outperforming raw corpora and commercial memory systems on held-out behav...
-
BOOKMARKS: Efficient Active Storyline Memory for Role-playing
BOOKMARKS introduces searchable bookmarks as reusable answers to storyline questions, enabling active initialization and passive synchronization for more consistent role-playing agent memory than recurrent summarization.
-
RoleCDE:Benchmarking and Mitigating Role-Alignment Trade-offs in Role-Playing Agents
New benchmark RoleCDE reveals LLMs exhibit role value decoupling under conflicts and demonstrates mitigation via targeted fine-tuning.
-
Toward a Benchmark for Controllable Simulation of Imperfect Students with Large Language Models
Introduces a benchmark framework for steering LLMs via skill vectors to simulate students with controllable retained and missing competencies.
-
Reinforcing Human Behavior Simulation via Verbal Feedback
DITTO uses RL with verbal feedback to train LLMs for human behavior simulation, reporting 36% average gains over base models and outperforming GPT-5.4 on 6 of 10 SOUL benchmark tasks.
-
Dynamic In-Group Persona Generation for Enhancing Human-AI Rapport
Dynamic in-group persona generation for LLMs, conditioned on user concern, significantly improves perceived rapport and user experience over no-persona and minimal-disclosure baselines in a human-subject study.
-
StratMem-Bench: Evaluating Strategic Memory Use in Virtual Character Conversation Beyond Factual Recall
StratMem-Bench reveals that state-of-the-art LLMs distinguish required from irrelevant memories effectively but struggle to integrate supportive memories in character conversations.
-
Decision-Theoretic Safety Assessment of Persona-Driven Multi-Agent Systems in O-RAN
A persona-driven multi-agent framework with a three-dimensional decision-theoretic evaluation shows that agent-persona alignment significantly impacts performance and coordination in O-RAN optimization challenges.
-
StreetDesignAI: Broadening Designer Perspectives Through Multi-Persona Evaluation of Cycling Infrastructure
StreetDesignAI provides structured multi-persona feedback on cycling designs and a user study shows it broadens designers' grasp of diverse cyclist perspectives and improves design decision confidence.
-
StreetDesignAI: Broadening Designer Perspectives Through Multi-Persona Evaluation of Cycling Infrastructure
StreetDesignAI uses multi-persona AI feedback to help designers identify and resolve experiential conflicts in cycling infrastructure through structured evaluation and iterative modification.
-
MOA: Multi-Objective Alignment for Role-Playing Agents
MOA applies multi-objective RL with fine-grained rubrics and thought-augmented rollouts to role-playing agents, enabling an 8B model to match closed-source performance on PersonaGym and RoleMRC benchmarks.
-
Moral Susceptibility and Robustness under Persona Role-Play in Large Language Models
LLM moral robustness under persona role-play is largely determined by model family with Claude models most consistent, while susceptibility shows little family dependence.
-
TinyTroupe: An LLM-powered Multiagent Persona Simulation Toolkit
TinyTroupe provides a toolkit for fine-grained persona-based LLM multi-agent simulations with built-in support for population sampling, experimentation, and validation.
-
From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review
A survey consolidating benchmarks, agent frameworks, real-world applications, and protocols for LLM-based autonomous agents into a proposed taxonomy with recommendations for future research.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.