Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effects

Ji-Lun Peng; Yun-Nung Chen

REVIEW 3 major objections 6 minor 10 references

LLM role-play scores drop when character names are anonymized, and personality descriptions can partly restore them.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 18:57 UTC pith:KLIHD6R5

load-bearing objection Worth a look for its evaluation idea, but the headline claim leans on a confounded anonymization step and effects that are mostly within noise. the 3 major comments →

arxiv 2603.03915 v2 pith:KLIHD6R5 submitted 2026-03-04 cs.CL cs.AI

Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effects

Ji-Lun Peng , Yun-Nung Chen This is my paper

classification cs.CL cs.AI

keywords role-playing agentsanonymous evaluationcharacter name memorizationpersonality augmentationMBTIBig FiveLLM evaluationgeneralization to unseen personas

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper's claim is that standard role-playing benchmarks inflate LLM ability: models lean on memorized associations with famous character names rather than on the persona description actually given. To test this, the authors replace every character name in the prompt with a placeholder token and measure the same models on the same questions. Across several benchmarks, languages, and model architectures, anonymized performance is consistently lower, which they interpret as evidence that name exposure supplies implicit cues. They then add explicit personality information—MBTI types or Big Five scores obtained by self-report, interview, or human annotation—and show that this consistently improves role fidelity, with model-generated personalities matching human-annotated ones. The payoff would be a fairer evaluation protocol for role-playing and a scalable way to build agents for personas the model has never seen.

Core claim

The core discovery is that a name is not a neutral identifier in role-play: it is a compressed memory hook. When the name is replaced by '<anonymous character>', the same models produce responses that human and reward-model judges rate as less in character, and pairwise win rates against the original setting drop. The authors conclude that prior work's numbers partly measure memorization of characters from pretraining. The second, more constructive finding is that an explicit abstract description of the character—its psychological type or trait profile—reliably closes much of this gap, and that the cheapest source (the model generating its own personality from the profile) works as well as h

What carries the argument

The mechanism that carries the argument is a two-condition contrast on the same benchmarks: original named prompts versus anonymized prompts where the name token is replaced by a placeholder. This isolates, as far as the prompt alone can, the contribution of name memory. The second mechanism is personality augmentation: taking a standardized personality summary (four-letter MBTI type or Big Five scores) and injecting it into the prompt before role-play. The personality can come from the model answering a questionnaire (self-report), from a judge model conducting an open interview, or from crowdsourced human votes; all three are compared. The personality block acts as an abstract behavioral c

Load-bearing premise

That removing only the character's name is what causes the score drop—if other identity cues leak through the anonymized descriptions, the performance gap cannot be cleanly attributed to name memory alone.

What would settle it

Create a fully scrubbed version of the benchmark in which every proper noun, nickname, interlocutor name, and distinctive world-specific phrase is replaced, and run the same pairwise comparison: if the anonymization gap does not grow, the original gap is not purely name-driven; if it grows substantially, the paper's mechanism is confirmed and prior anonymization was incomplete.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Rankings from name-bearing benchmarks should be treated as upper bounds; anonymized scores are the better estimate of true role-play skill on unseen personas.
Self-generated personality descriptions can replace expensive human annotation without losing fidelity, making personality enhancement scalable to arbitrary characters.
Personality augmentation works even when the model does not know who it is playing, so the same recipe applies to impersonating real people or original characters not in pretraining data.
Characters with more extreme or distinctive personality profiles benefit more from the augmentation, so trait strength is part of the signal.
The effect generalizes across two distinct personality instruments (MBTI and Big Five), suggesting it is the trait information, not the specific inventory, that helps.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the name-memory story is right, model rankings may shift once names are removed; an anonymous leaderboard would likely reward models that follow prompted descriptions rather than recall famous lines.
The authors' own limitation suggests the measured anonymization gap is a lower bound: because interlocutor names, nicknames, and signature traits may still leak identity, a stricter scrubbing could show an even larger performance drop.
One testable extension: vary the richness of the personality block (one type label vs. full trait descriptions) to see whether more detailed behavioral constraints push anonymous performance closer to the named baseline.
Another testable extension: apply the same anonymization-and-personality recipe to knowledge-heavy factual recall tasks to separate 'who am I' memory from 'what do I know'; the current knowledge-related dimensions hint that personality may not recover factual memory the way it recovers style.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Worth a look for its evaluation idea, but the headline claim leans on a confounded anonymization step and effects that are mostly within noise.

read the letter

The core idea here is genuinely useful: anonymize character names in role-play benchmarks so that models can't coast on memorized name cues, and test whether adding personality information compensates. That's a sensible direction, and the paper does a few concrete things well—it runs the anonymization across two benchmarks and three models, compares self-generated personality traits against human-annotated PDB data, and includes a small human evaluation. The self-generated vs. human-annotated comparison is the most interesting and least confounded piece, and the result that they perform similarly is worth knowing.

But the central claim—that removing names degrades performance because names carry implicit cues—is not actually supported by the design. The anonymization replaces every name with the literal string ''. There is no control with a plausible pseudonym, so the observed drop could just as easily come from the model being asked to role-play as a weird placeholder rather than from losing the name's semantic associations. The stress-test note makes exactly this point, and reading the paper, I think it lands. The authors' own Limitations section worries about residual identity leakage, but it never considers that the anonymized condition might be unfairly unnatural. That is load-bearing.

The empirical support is thinner than the abstract suggests. In Table 1, the anonymization deltas are mostly 0.0–0.2, many with mixed signs, and only the gpt-4o average reaches significance. The personality augmentation results in Table 3 are also small—often under 0.1 points—and several are negative or flat. The repeated use of 'consistently' and 'significantly' overstates what the tables show. To be fair, the paper does report significance in places and includes a human eval, but the effect sizes are modest and the variance is not fully reported.

I'd like to see a revised version that adds a pseudonym control, reports confidence intervals or effect sizes for the key comparisons, and tones down the claims. The idea is a legitimate contribution to role-play evaluation, but the current evidence doesn't justify the strong conclusion. As is, it's a solid workshop paper or a paper that needs substantial revision before acceptance. I'd send it to peer review because the question matters and the authors are onto something, but I wouldn't trust the headline result without the control.

This is a 'maybe' for a reading group—good for a discussion about evaluation methodology, not as a source of established findings. I wouldn't cite it as evidence, though I might mention it as related work if I were writing about benchmark design.

Referee Report

3 major / 6 minor

Summary. The paper proposes an anonymized protocol for evaluating LLM role-playing agents: character names in prompts are replaced with the literal token '<anonymous character>', and performance is compared on CharacterEval and RoleAgentBench using gemini-2.0-flash, llama-3.1-405B-instruct, and gpt-4o. The authors report that anonymization lowers scores, which they interpret as evidence that models rely on memorized name cues. They then augment anonymous prompts with MBTI or Big Five personality descriptions obtained from self-report, interview-based inference, or the crowdsourced Personality Database (PDB), and claim that personality augmentation consistently improves role fidelity, with self-generated personalities matching human-annotated ones. A small human evaluation on a Harry Potter subset is reported as corroborating evidence.

Significance. The underlying research question is timely and practically important: if name-free evaluation isolates true role-playing ability, the proposed protocol would improve generalization estimates for unseen personas, and the personality-augmentation idea offers a scalable way to compensate for missing name cues. The paper has genuine strengths: it spans two benchmarks, three model families, two personality inventories, and three acquisition methods, and it includes a human evaluation. However, the central causal claim depends on a missing control condition, and several quantitative results contradict the claimed consistency of the effects. The contribution is promising but needs additional experiments and substantially more careful reporting before it can support the paper's headline conclusions.

major comments (3)

[§2.2; Table 1; Limitations] The central claim that 'anonymizing characters degrades performance' is not cleanly identified by the current design. The intervention replaces a character name with the literal token '<anonymous character>' and there is no control condition using a plausible pseudonym or neutral descriptor. The token itself may cause hedging, inconsistent persona adoption, or other instruction-following artifacts, independent of any loss of name-triggered memory. The Limitations paragraph acknowledges residual identity leakage from other cues, but it does not address this opposite risk. Add a control in which names are replaced with an invented generic name (e.g., 'Alex') while keeping the rest of the profile identical. If the pseudonym recovers the original-condition scores, the name-cue interpretation is supported; if it behaves like '<anonymous character>', the Table 1 drops are largely a placeholder
[§2.5; Table 1] The abstract and §2.5 claim that anonymization degrades performance, but Table 1 does not consistently show this. For llama-3.1-405B-instruct, the CharacterEval Character Consistency average is higher under anonymization (2.913 vs. 2.907), and several sub-dimensions improve (e.g., Know-Exposure 2.097 vs. 2.037). Only gpt-4o shows a marked, statistically significant average drop; gemini's 0.10 average drop is not starred. Please report per-model and per-task effect sizes with paired tests and confidence intervals, and either temper the blanket degradation claim or restrict it to the models and dimensions where the effect is actually significant.
[§3.3; Table 3] The claim that 'incorporating personality information consistently improves RPA performance' is contradicted by Table 3. For example, llama-3.1 MBTI Character Consistency is lower in all augmented conditions (2.887, 2.884, 2.887) than the original (2.898); for gpt-4o Big Five Attractiveness, all augmented conditions are lower (3.086, 3.082, 3.056) than the original (3.107); and several MBTI augmentation effects for llama Character Consistency are negative. Many cells lack significance stars, and the winning margins are often small. Please clarify the exact comparison set, report paired significance tests with multiple-comparison control, and report the fraction of dimensions/models for which augmentation actually helps before using the word 'consistently'.

minor comments (6)

[§2.3] The text says 'The statistic of three benchmarks is shown in Table 2', but Table 2 lists two benchmarks. Please correct the count or add the missing benchmark.
[Figure 2] The stacked win-rate plot is hard to read because the numeric labels appear duplicated and misaligned, and no sample sizes or confidence intervals are given. A small table with win/loss/tie counts and totals would be clearer.
[Table 3] The table has formatting errors: '3.0792.898' is missing a separator, and the bold/underline conventions for best/second-best are not visually distinguishable in several cells. Please reformat.
[Figure 5] The prompt template contains a typo: 'Please anwser' should be 'Please answer'.
[§4.4] The human evaluation is limited to one model, five Harry Potter characters, and fair-to-moderate inter-rater agreement (kappa 0.308 for General Response). This is useful as a sanity check but cannot carry the cross-model/cross-benchmark claim. Report per-character results and the raw win/loss/tie counts behind Figure 4.
[§4.2] The 'strong personality' threshold (scores above 60% or below 40%) is arbitrary and is only evaluated with gemini-2.0-flash. State whether this threshold was chosen post hoc and provide robustness checks for neighboring thresholds.

Circularity Check

0 steps flagged

No circular step: anonymization and personality-augmentation effects are empirical comparisons scored by external benchmarks and human judges, not reductions to fitted parameters or self-cited premises.

full rationale

The paper's central contrasts are manipulations, not identities. It replaces character names with '<anonymous character>' (§2.2) and observes that scores drop (Table 1, Figure 2); that is a contingent empirical result, not true by definition. The personality conditions add MBTI/Big Five information (Figure 5) and compare against the same system without that information; personality is an input, not the evaluation target, so the improvement is not a fitted value renamed as a prediction. The only caveat is causal: the paper's Limitations section itself notes that 'models may still infer character identities from other descriptive signals,' and a plausible-pseudonym control would make the name-memory interpretation cleaner. That is a validity/confound concern, not circularity. Self-citations (Peng et al. 2024; Tseng et al. 2024) are survey/context citations and are not load-bearing; no uniqueness theorem or ansatz is imported from the authors' own prior work. Hence no specific step satisfies the circularity criteria.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

Ledger is light because this is an empirical benchmark paper with no fitted physical parameters. The main burdens are evaluation validity, trait-instrument validity, and the assumption that anonymization removes only the name.

free parameters (1)

strong personality threshold = scores above 60% or below 40% on MBTI dimensions
Ad-hoc cutoff used in Section 4.2 to select 'significant personality types'; no justification or sensitivity analysis is given.

axioms (3)

domain assumption Name replacement isolates character-memory effects from other identity cues.
Section 2.2 and the Limitations section admit other cues (interlocutors, nicknames, distinctive traits) may leak identity, confounding the anonymization comparison.
domain assumption LLM-as-a-judge and reward-model scores are valid proxies for role fidelity.
Sections 2.3-2.4 rely on a reward model and gpt-4o-mini judge without a systematic bias analysis beyond swapped-order double-run tie rule for RoleAgentBench.
domain assumption MBTI and Big Five instruments carry meaningful persona information for role-play.
Section 3.2 adopts 16Personalities/MBTI despite citing Pittenger (1993), which questions MBTI validity; no independent validation of character trait labels is provided beyond crowd-sourced PDB.

pith-pipeline@v1.3.0-alltime-deepseek · 11780 in / 11809 out tokens · 115783 ms · 2026-08-02T18:57:39.716822+00:00 · methodology

0 comments

read the original abstract

Large Language Models (LLMs) have shown remarkable potential in developing role-playing agents (RPAs). However, current evaluation frameworks rely heavily on well-known fictional characters, raising a critical concern: models may be leveraging their internal training memory of these characters rather than demonstrating role-playing capabilities. This reliance often leads to significant performance degradation when RPAs encounter unseen or out-of-distribution personas. To address this, we propose a more rigorous evaluation protocol designed to decouple role-playing proficiency from character recognition. Our experiments across multiple benchmarks demonstrate that anonymizing characters degrades performance, confirming that name exposure provides implicit cues that mask a model's true capability. To mitigate this, we investigate diverse personality augmentation as a method to enhance role fidelity in anonymous settings. We systematically analyze the impact of various personality-description methods on agent behavior and consistency. Our results show that incorporating personality information consistently improves RPA performance. This work establishes a more equitable evaluation standard and validates a scalable, personality-enhanced framework for constructing robust RPAs.

Figures

Figures reproduced from arXiv: 2603.03915 by Ji-Lun Peng, Yun-Nung Chen.

**Figure 2.** Figure 2: Paired response evaluation on RoleAgent [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: Pairwise comparison on RoleAgentBench: MBTI or Big Five personality integration vs. the original [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

**Figure 4.** Figure 4: Human pairwise comparison against the origi [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

**Figure 5.** Figure 5: Prompt template for personality-augmented RPA. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗

**Figure 6.** Figure 6: Prompt template for RoleAgentBench pair-wise evaluation. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

10 extracted references · 7 linked inside Pith

[1]

The more distinctive the speaking style, the better

Which one has more pronounced role speaking style, and speaks more in line with the role description. The more distinctive the speaking style, the better
[2]

condition

Which one’s output contains more knowledge and memories related to the role; the richer, the better. (If the question contains reference answers, then the role-specific knowledge and memories are based on the reference answer.) The question provided to each condition is: {benchmark question} The reference answer of the question is: {benchmark answer} The ...
[4]

arXiv preprint arXiv:2411.10109

Generative agent simulations of 1,000 people. arXiv preprint arXiv:2411.10109. Ji-Lun Peng, Sijia Cheng, Egil Diau, Yung-Yu Shih, Po-Heng Chen, Yen-Ting Lin, and Yun-Nung Chen

Pith/arXiv arXiv
[5]

David J Pittenger

A survey of useful llm evaluation.arXiv preprint arXiv:2406.00936. David J Pittenger. 1993. The utility of the myers- briggs type indicator.Review of educational research, 63(4):467–488. Haiming Qin, Jiwei Zhang, Wei Zhang, KeZhong Lu, Mingyang Zhou, Hao Liao, and Rui Mao. 2025. R- char: A metacognition-driven framework for role- playing in large language...

Pith/arXiv arXiv 1993
[6]

Quan Shi, Carlos E Jimenez, Stephen Dong, Brian Seo, Caden Yao, Adam Kelch, and Karthik Narasimhan

Character-llm: A trainable agent for role- playing.Preprint, arXiv:2310.10158. Quan Shi, Carlos E Jimenez, Stephen Dong, Brian Seo, Caden Yao, Adam Kelch, and Karthik Narasimhan

Pith/arXiv arXiv
[8]

Quan Tu, Shilong Fan, Zihang Tian, Tianhao Shen, Shuo Shang, Xin Gao, and Rui Yan

Characterchat: Learning towards conversa- tional ai with personalized social support.arXiv preprint arXiv:2308.10278. Quan Tu, Shilong Fan, Zihang Tian, Tianhao Shen, Shuo Shang, Xin Gao, and Rui Yan. 2024. Charac- terEval: A chinese benchmark for role-playing con- versational agent evaluation. InProceedings of the 62nd Annual Meeting of the Association f...

Pith/arXiv arXiv 2024
[1991]

8 Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi Mi, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, and 1 others

Big five inventory.Journal of personality and social psychology. 8 Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi Mi, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, and 1 others
[2023]

Cheng Liu, Yifei Lu, Fanghua Ye, Jian Li, Xingyu Chen, Feiliang Ren, Zhaopeng Tu, and Xiaolong Li

Chatharuhi: Reviving anime character in reality via large language model.arXiv preprint arXiv:2308.09597. Cheng Liu, Yifei Lu, Fanghua Ye, Jian Li, Xingyu Chen, Feiliang Ren, Zhaopeng Tu, and Xiaolong Li. 2025. Cogdual: Enhancing dual cognition of llms via rein- forcement learning with implicit rule-based rewards. InProceedings of the 2025 Conference on E...

Pith/arXiv arXiv 2025
[2024]

Robert R McCrae and Paul T Costa Jr

Large language models are superpositions of all characters: Attaining arbitrary role-play via self- alignment.arXiv preprint arXiv:2401.12474. Robert R McCrae and Paul T Costa Jr. 1997. Person- ality trait structure as a human universal.American psychologist, 52(5):509. Mary L McHugh. 2012. Interrater reliability: the kappa statistic.Biochemia medica, 22(...

Pith/arXiv arXiv 1997
[2025]

Mirac Suzgun and Adam Tauman Kalai

Impersona: Evaluating individual level lm impersonation.arXiv preprint arXiv:2504.04332. Mirac Suzgun and Adam Tauman Kalai. 2024. Meta-prompting: Enhancing language models with task-agnostic scaffolding.arXiv preprint arXiv:2401.12954. Yu-Min Tseng, Yu-Chao Huang, Teng-Yun Hsiao, Wei- Lin Chen, Chao-Wei Huang, Yu Meng, and Yun- Nung Chen. 2024. Two tales...

Pith/arXiv arXiv 2024

[1] [1]

The more distinctive the speaking style, the better

Which one has more pronounced role speaking style, and speaks more in line with the role description. The more distinctive the speaking style, the better

[2] [2]

condition

Which one’s output contains more knowledge and memories related to the role; the richer, the better. (If the question contains reference answers, then the role-specific knowledge and memories are based on the reference answer.) The question provided to each condition is: {benchmark question} The reference answer of the question is: {benchmark answer} The ...

[3] [4]

arXiv preprint arXiv:2411.10109

Generative agent simulations of 1,000 people. arXiv preprint arXiv:2411.10109. Ji-Lun Peng, Sijia Cheng, Egil Diau, Yung-Yu Shih, Po-Heng Chen, Yen-Ting Lin, and Yun-Nung Chen

Pith/arXiv arXiv

[4] [5]

David J Pittenger

A survey of useful llm evaluation.arXiv preprint arXiv:2406.00936. David J Pittenger. 1993. The utility of the myers- briggs type indicator.Review of educational research, 63(4):467–488. Haiming Qin, Jiwei Zhang, Wei Zhang, KeZhong Lu, Mingyang Zhou, Hao Liao, and Rui Mao. 2025. R- char: A metacognition-driven framework for role- playing in large language...

Pith/arXiv arXiv 1993

[5] [6]

Quan Shi, Carlos E Jimenez, Stephen Dong, Brian Seo, Caden Yao, Adam Kelch, and Karthik Narasimhan

Character-llm: A trainable agent for role- playing.Preprint, arXiv:2310.10158. Quan Shi, Carlos E Jimenez, Stephen Dong, Brian Seo, Caden Yao, Adam Kelch, and Karthik Narasimhan

Pith/arXiv arXiv

[6] [8]

Quan Tu, Shilong Fan, Zihang Tian, Tianhao Shen, Shuo Shang, Xin Gao, and Rui Yan

Characterchat: Learning towards conversa- tional ai with personalized social support.arXiv preprint arXiv:2308.10278. Quan Tu, Shilong Fan, Zihang Tian, Tianhao Shen, Shuo Shang, Xin Gao, and Rui Yan. 2024. Charac- terEval: A chinese benchmark for role-playing con- versational agent evaluation. InProceedings of the 62nd Annual Meeting of the Association f...

Pith/arXiv arXiv 2024

[7] [1991]

8 Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi Mi, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, and 1 others

Big five inventory.Journal of personality and social psychology. 8 Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi Mi, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, and 1 others

[8] [2023]

Cheng Liu, Yifei Lu, Fanghua Ye, Jian Li, Xingyu Chen, Feiliang Ren, Zhaopeng Tu, and Xiaolong Li

Chatharuhi: Reviving anime character in reality via large language model.arXiv preprint arXiv:2308.09597. Cheng Liu, Yifei Lu, Fanghua Ye, Jian Li, Xingyu Chen, Feiliang Ren, Zhaopeng Tu, and Xiaolong Li. 2025. Cogdual: Enhancing dual cognition of llms via rein- forcement learning with implicit rule-based rewards. InProceedings of the 2025 Conference on E...

Pith/arXiv arXiv 2025

[9] [2024]

Robert R McCrae and Paul T Costa Jr

Large language models are superpositions of all characters: Attaining arbitrary role-play via self- alignment.arXiv preprint arXiv:2401.12474. Robert R McCrae and Paul T Costa Jr. 1997. Person- ality trait structure as a human universal.American psychologist, 52(5):509. Mary L McHugh. 2012. Interrater reliability: the kappa statistic.Biochemia medica, 22(...

Pith/arXiv arXiv 1997

[10] [2025]

Mirac Suzgun and Adam Tauman Kalai

Impersona: Evaluating individual level lm impersonation.arXiv preprint arXiv:2504.04332. Mirac Suzgun and Adam Tauman Kalai. 2024. Meta-prompting: Enhancing language models with task-agnostic scaffolding.arXiv preprint arXiv:2401.12954. Yu-Min Tseng, Yu-Chao Huang, Teng-Yun Hsiao, Wei- Lin Chen, Chao-Wei Huang, Yu Meng, and Yun- Nung Chen. 2024. Two tales...

Pith/arXiv arXiv 2024