REVIEW 4 cited by
PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluation
read the original abstract
We introduce a benchmark for evaluating the role-playing capabilities of language models. Our approach leverages different language models to simulate users in dynamic, multi-turn conversations and assess the resulting dialogues. Our methodology involves three main components: a player model that adopts a specific character role, an interrogator model that simulates user behavior in a specific situation, and a judge model ensemble that evaluates conversation quality with 3 metrics: character consistency, entertainment value, and language fluency. We evaluated more than 40 models in both English and Russian, with each model participating in 64 conversations with 8 characters and 8 situations. We conducted experiments comparing automated evaluations with human annotations to validate our approach, demonstrating strong correlations across multiple criteria. This work provides a foundation for a robust and dynamic evaluation of different model capabilities in interactive scenarios.
Forward citations
Cited by 4 Pith papers
-
Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation
A role-playing benchmark that uses per-person simulated users and personalized rubrics to co-construct and score free multi-turn dialogues, showing fixed-history evaluation is history-biased and user-specific rankings...
-
Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation
PALATE trains five per-user simulators and personalized rubrics from real chat histories, then evaluates 16 role-playing agents on free multi-turn conversations, finding user-dependent winners.
-
Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
Psy-CoT decomposes reasoning into Interaction Perception, Psychological Empathy, and Logical Construction while RAPO asymmetrically weights role-specific tokens during policy optimization, outperforming prior CoT and ...
-
REAR: Test-time Preference Realignment through Reward Decomposition
REAR decomposes the reward into question and preference components, rescales their balance, and expresses the result as a linear combination of token log-probabilities for efficient integration with best-of-N and tree search.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.