Pith. sign in

REVIEW 4 major objections 6 minor 26 references

Exploring the Potential of Large Language Models to Simulate Personality

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Big Five score prompts fail to reliably drive LLM personality in generated text.

desk verdict Useful dataset and framework for LLM personality simulation, but the key claim about trait bias rests on a classifier that is weakest on the very trait it implicates. read the letter →

arxiv 2502.08265 v1 pith:43FHQC5A submitted 2025-02-12 cs.CL cs.AI

Maria Molchanova , Anna Mikhailova , Anna Korzanova , Lidiia Ostyakova , Alexandra Dolidze This is my paper · ORCID
classification cs.CLcs.AI
keywords BigFivepersonalitylargelanguagemodelssimulationtextgenerationpromptingLLMevaluationBFI-44Neuroticism
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether telling an LLM to write as someone with a given Big Five personality score actually changes the personality expressed in the text it produces. The authors find that simple score-based prompting works for some traits and some models, but not reliably: several models drift toward a default 'helpful assistant' persona—low Neuroticism and high Agreeableness—even when told to simulate the opposite. The hardest trait is Neuroticism, which is often absent from generated texts or stuck at low scores. The paper releases the generated-text dataset and an analytical framework so other researchers can test personality simulation on their own models. The practical stakes are direct: chatbots and game agents are only as personalized as the language they actually produce.

What carries the argument

The argument runs on a two-stage evaluation protocol. First, each model answers BFI-44 questionnaire items under a prompt to adopt a high or low trait score, which tests whether the model knows the trait-behavior link. Second, each model answers open questions about preferences, perspectives, and life goals under a prompt specifying a trait score from 1 to 5, and those generated texts are scored by human annotators and by a GPT-4 Omni classifier built on the CARP clue-extraction method. Confusion matrices comparing prompted versus detected score groups are what turn observed text differences into a claim about personality simulation skill, supplemented by linguistic analyses of vocabulary across score levels.

What would settle it

Have three independent human annotators re-score a random sample of texts that the classifier labeled as low or non-detectable Neuroticism; if they frequently assign high Neuroticism scores, the paper's finding that models cannot simulate that trait would be an artifact of the classifier rather than a property of the models.

Watch

Extended reading notes

Core claim

The central discovery is that LLMs do not consistently translate an explicit Big Five trait score into a matching personality in free-form text, even though the same models answer personality questionnaires coherently. Using four models (GPT-3.5 Turbo, GPT-4 Omni, Mixtral 8x22B, and Claude 3 Haiku) prompted with scores from 1 to 5 and judged by human annotators plus a GPT-4 Omni classifier, the study finds that Openness to Experience is the most reliably simulated trait, while Agreeableness often comes out high and Neuroticism is frequently undetectable or biased low. Middle scores are particularly unstable: when a model has a default bias, a prompted middle score tends to fall into that biased group rather than sounding neutral. The authors interpret this pattern as evidence that models fall back on an agreeable, emotionally stable assistant persona when generating personality-related text.

Load-bearing premise

All comparisons of which model best simulates which trait assume the GPT-4 Omni classifier, which humans agree with only moderately and least on Neuroticism, is accurate enough on the full corpus to stand in for human judgment.

Editorial extensions

If this is right

  • Defining a persona by a single numeric Big Five score is not enough; generated text should be checked for score bias before deployment in chatbots or game characters.
  • Middle scores (around 3) are poor prompt targets; binary high/low trait definitions are more likely to yield distinguishable text.
  • For Neuroticism, the paper recommends setting the trait neutral or omitting it, because high-Neuroticism prompts fail to produce detectable emotional reactivity.
  • Questionnaire-based personality tests overstate a model's ability to simulate personality, since generation-task performance does not follow questionnaire performance.
  • Researchers can use the released framework to replicate the protocol on any model and compare trait-level bias directly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the default assistant persona is the cause, then overriding bias may require behavioral examples or system-level instruction rather than more extreme trait scores.
  • Because the classifier used for most of the corpus is the same model family as one of the generators, the measured trait gaps might be understated; a classifier from a different model family could rank the models differently.
  • The linguistic-lexicon results suggest a cheap extension: test whether prompts that specify vocabulary style improve trait simulation more than score descriptions alone.
  • A natural next experiment is to prompt full Big Five profiles rather than single traits and see whether trait interactions reduce or amplify the per-trait biases found here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper investigates whether LLMs can simulate Big Five personality traits in two tasks: answering the BFI-44 questionnaire and generating free-text responses under prompted trait scores. The questionnaire stage reports reliability metrics (Cronbach's alpha, Guttman's lambda) and score distributions per model. The text generation stage prompts GPT-3.5 Turbo, GPT-4 Omni, Claude 3 Haiku, and Mixtral 8x22B with single-trait scores from 1 to 5, then evaluates the outputs via a small human annotation study (288 texts, 8 annotators) and a GPT-4 Omni based automatic classifier. The paper concludes that generating personality-consistent text is still challenging, with Neuroticism the hardest trait and several models showing strong score biases, and it releases a dataset and an analytical framework.

Significance. If the central claims hold, the paper provides a useful negative result for personality prompting: simple Big Five score prompts do not reliably transfer to generated text, and questionnaire behavior can diverge from text-generation behavior. The released dataset and open-source framework are practical contributions, and the questionnaire stage is methodologically more solid, with explicit reliability metrics. However, the text-generation conclusions rest almost entirely on an automatic classifier that is validated on only 288 texts and is weakest exactly on Neuroticism, the trait carrying the headline result; until the classifier's systematic biases are characterized, the per-model conclusions in Section 5.3 and Figure 2 are not secure.

major comments (4)
  1. [Section 5.3, Tables 2-3] The automated classifier's Neuroticism scores are the least reliable (Level-2 weighted F1 = 0.50, MAE = 1.77), yet Neuroticism is the trait for which the paper makes its strongest claim ('the most challenging trait to simulate'). The paper reports only aggregate precision/recall/F1 and overall MAE, not the direction of classifier errors relative to human judgments. Without knowing whether the classifier systematically maps generated texts to low Neuroticism regardless of content, the confusion matrices in Figure 2 and the related per-model conclusions could be artifacts of classifier bias. Please report per-class and per-direction error analysis (e.g., confusion matrices of classifier vs. human scores for low/middle/high and Nondistinguishable), and if the bias is confirmed, re-analyze the generated-text corpus with a more reliable classifier or a substantially larger human-annotated sample.
  2. [Section 5.3 and Section 8] The classifier is built on GPT-4 Omni, which is also one of the models being evaluated for generation, and both generation and classification prompts use the same trait definitions and the same 'Nondistinguishable' instruction. Since Section 8 itself suggests that GPT-4 Omni may have a default persona of high Agreeableness and low Neuroticism, the classifier may share the very bias the paper attributes to the generators. This circularity is load-bearing: the per-model comparisons in Figure 2 and the bullet conclusions about model biases would collapse if the classifier's errors are systematically aligned with its own default persona. Please demonstrate robustness by using an independent classifier (e.g., a fine-tuned model not from the GPT family, or a different model family) or by providing evidence that the classifier's decisions are unbiased with respect to the prompted trait score and the generating model.
  3. [Section 5.1] The manuscript states that Claude responses 'were edited by masking direct references to personality characteristics or by removing content that did not align with the task objectives.' This editing is not quantified, and no criteria or inter-annotator reliability for the editing process are provided. If editing removes trait-relevant cues, it could directly lower the detectability of the prompted trait for Claude and thereby change the comparative conclusions in Section 5.3. Please specify how many responses were edited, what kinds of edits were made, whether the editors were blind to the prompted trait score, and ideally report results on both raw and edited texts to show that editing does not drive the findings.
  4. [Section 5.2 and Table 1] The human evaluation that serves as ground truth for the classifier is based on only 288 texts, with three annotators per text, and the inter-annotator agreement is moderate at best (Level-2 Fleiss kappa ranges from 0.57 for Neuroticism to 0.70 for Agreeableness). Given that Neuroticism also has the lowest human agreement, the conclusion that Neuroticism is especially hard to simulate is partly an artifact of noisier measurement for that trait. The paper should explicitly discuss how the measurement error in the human labels bounds the conclusions, and consider reporting analyses that pool scores into coarser categories or that use the full distribution of annotator scores rather than majority vote.
minor comments (6)
  1. [Section 5.1] The sentence 'The Claude model exhibited p performance in deciding whether to deliver a direct answer' appears to contain a typo; the word 'p' should likely be 'poor' or another adjective.
  2. [Section 5.2] The phrase 'Inner-annotation agreement metrics' should be 'inter-annotation agreement metrics', since the metrics measure agreement between annotators, not within a single annotation.
  3. [Tables 1-3 and throughout] The trait name is spelled inconsistently as both 'Extraversion' and 'Extroversion'; please choose one spelling (the standard Big Five term is 'Extraversion') and use it consistently.
  4. [Section 5.3 and Appendix B] The text refers to 'Figure 4 presents the results of the classifier's detection...' but the figure showing trait detection appears in Appendix B; please renumber or cross-reference the appendix figures clearly so that the reader can locate them.
  5. [Section 5.1 and C.2] The mapping between the 1-5 score used in generation prompts and the -2 to +2 scale used by human annotators and the classifier is not stated explicitly; please clarify how prompt scores were mapped to annotation and classification labels, in particular how score 3 (middle) is treated.
  6. [Limitations] The Limitations section mentions 'partial human dataset annotation', but the main body does not state that the classifier, rather than human judgment, is the primary instrument for all reported text-generation results; please move or echo this caveat in Section 5.3 where the classifier is first used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the main claims rest on a human-validated classifier and human questionnaire evidence, despite a self-referential evaluation risk.

full rationale

The paper's headline claim — that personality-related text generation remains challenging and that Neuroticism is especially hard to simulate — is an empirical finding, not a derivation. The automatic evaluation uses a GPT-4 Omni classifier (Section 5.3), and GPT-4 Omni is also one of the generators, so there is a genuine methodological risk that classifier bias drives the reported results. That risk is real and the paper itself flags it: "The most problematic trait for the classifier is Neuroticism, which also has a low inner-annotation agreement metric within the group," with Level-2 weighted F1 = 0.50 and MAE = 1.77 (Tables 2–3). However, circularity requires a specific reduction: a fitted input renamed as a prediction, a self-citation used as the sole load-bearing premise, or a target quantity defined in terms of itself. None of those occurs here. The classifier is validated against 288 human-annotated texts rather than fitted to the target outputs, and the shared trait definitions between generation and classification are a measurement requirement, not a definitional identity. The Neuroticism difficulty conclusion is also supported independently of the classifier: Section 7 states the trait "was not detected in the questionnaire answers or by human analysis," and human annotator agreement on Neuroticism is lowest (kappa 0.57–0.59, Table 1). The Limitations section acknowledges partial human annotation. These are validity and reliability concerns, not circular reasoning, so the appropriate verdict is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The study rests on psychological and methodological assumptions: the Big Five model's validity, the meaningfulness of LLM questionnaire responses, the reliability of human trait annotation, and the validity of the GPT-4 Omni classifier as a proxy for human judgment. No free parameters are fitted in the traditional sense, and no new entities are postulated.

assumptions (4)
  • domain assumption Big Five model is a valid representation of personality.
    Paper relies on the model to define trait scores and prompts; validity is cited (John et al., 2008) but not independently established here. Section 2.
  • domain assumption BFI-44 questionnaire responses from LLMs are meaningful indicators of personality.
    The study treats LLM questionnaire answers as expressions of prompted personality; reliability metrics are computed, but construct validity is assumed. Section 4.
  • domain assumption Human annotators can reliably infer Big Five traits from short generated texts.
    Human evaluation is used as ground truth; Fleiss kappa 0.57-0.71 shows only moderate agreement, so this is a load-bearing assumption. Section 5.2.
  • ad hoc to paper The GPT-4 Omni based classifier accurately reproduces human trait judgments on the full corpus.
    The classifier is the paper's own tool and is used to score all generated texts; validation on 288 texts with trait F1 as low as 0.50 means this assumption may fail for traits like Neuroticism. Section 5.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring the Potential of Large Language Models to Simulate Personality." pith.science (2026). https://pith.science/paper/43FHQC5A

@misc{pith2026250208265,
  author       = {Pith},
  title        = {Pith review of: Exploring the Potential of Large Language Models to Simulate Personality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/43FHQC5A}},
  note         = {Machine review of arXiv:2502.08265}
}
read the original abstract

With the advancement of large language models (LLMs), the focus in Conversational AI has shifted from merely generating coherent and relevant responses to tackling more complex challenges, such as personalizing dialogue systems. In an effort to enhance user engagement, chatbots are often designed to mimic human behaviour, responding within a defined emotional spectrum and aligning to a set of values. In this paper, we aim to simulate personal traits according to the Big Five model with the use of LLMs. Our research showed that generating personality-related texts is still a challenging task for the models. As a result, we present a dataset of generated texts with the predefined Big Five characteristics and provide an analytical framework for testing LLMs on a simulation of personality skills.

Figures

Figures reproduced from arXiv: 2502.08265 by the authors.

Figure 1
Figure 1. Distribution of BFI-44 trait scores for different LLMs prompted to simulate high or low Big Five traits. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Confusion matrix for prompted and detected trait level. L stands for low trait level, M for middle level and [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Averaged trait-level scores of the 5 most similar LLM-generated texts, grouped by the Big Five traits. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Distribution of texts where trait were non [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Distribution of the prompted personality level [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 14 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Ai, Yiming, Zhiwei He, Ziyin Zhang, Wenhong Zhu, Hongkun Hao, Kai Yu, Lingjun Chen, and Rui Wang. 2024. Is Cognition and Action Consistent or Not: Investigating Large Language Model's Personality. arXiv preprint arXiv:2402.14679 . https://arxiv.org/abs/2402.14679

    arXiv 2024

  4. [4]

    Goldberg

    Ashton, Michael C., Kibeom Lee, and Lewis R. Goldberg. 2004. A hierarchical analysis of 1,710 English personality-descriptive adjectives. Journal of Personality and Social Psychology , 87(5):707

    work page 2004

  5. [5]

    Big Five Personality Test Dataset. 2024. https://www.kaggle.com/datasets/tunguz/big-five-personality-test. Accessed: 2024-08-28

    work page 2024

  6. [6]

    Costa, P. T., Jr. and R. R. McCrae. 1992. Revised NEO Personality Inventory (NEO-PI-R) and NEO Five-Factor Inventory (NEO-FFI) professional manual. FL: Psychological Assessment Resources

    work page 1992

  7. [7]

    Costa, P. T. and R. R. McCrae. 1999. A five-factor theory of personality. In Handbook of personality: Theory and research. Guildford Press New York

    work page 1999

  8. [8]

    and Robert R

    Costa, Paul T. and Robert R. McCrae. 2008. The revised neo personality inventory (neo-pi-r). The SAGE handbook of personality theory and assessment , 2(2):179--198

    work page 2008

Show all 26 references
  1. [9]

    Goldberg, Lewis R. 1992. The development of markers for the big-five factor structure. Psychological assessment , 4(1):26

    1992

  2. [10]

    Chawla, Olaf Wiest, and Xiangliang Zhang

    Guo, Taicheng, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680

    2024 arXiv

  3. [11]

    Hilliard, Airlie, Cristian Mu \ n oz, Zekun Wu, and Adriano Soares Koshiyama. 2024. Eliciting personality traits in large language models. arXiv preprint , 15

    2024

  4. [12]

    Jiang, Hang, Xiajie Zhang, Xubo Cao, Jad Kabbara, and Deb Roy. 2023. Personallm: Investigating the ability of GPT -3.5 to express personality traits and gender differences. arXiv preprint arXiv:2305.02547

    2023 arXiv

  5. [13]

    Jiang, Guangyuan, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. 2024. Evaluating and inducing personality in pre-trained language models. Advances in Neural Information Processing Systems , 36

    2024

  6. [14]

    John, O. P. 1991. The ``big five'' inventory-versions 4a and 54. Berkeley: University of California, Berkeley, Institute of Personality and Social Research/Institute of Personality and Social Research

    1991

  7. [15]

    Naumann, and Christopher J

    John, Oliver P., Laura P. Naumann, and Christopher J. Soto. 2008. Paradigm shift to the integrative big five trait taxonomy. Handbook of personality: Theory and research , 3(2):114--158

    2008

  8. [16]

    Liu, Jianzhi, Hexiang Gu, Tianyu Zheng, Liuyu Xiang, Huijia Wu, Jie Fu, and Zhaofeng He. 2024. Dynamic generation of personalities with large language models. arXiv preprint arXiv:2404.07084

    2024 arXiv

  9. [17]

    and Alexander Waibel

    Polzin, Thomas S. and Alexander Waibel. 2000. Emotion-sensitive human-computer interfaces. In ISCA tutorial and research workshop (ITRW) on speech and emotion

    2000

  10. [18]

    Prendinger, Helmut, Junichiro Mori, and Mitsuru Ishizuka. 2005. Using human physiology to evaluate subtle expressivity of a virtual quizmaster in a mathematical game. International journal of human-computer studies , 62(2):231--245

    2005

  11. [19]

    Rashkin, Hannah, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2018. Towards empathetic open-domain conversation models: A new benchmark and dataset. arXiv preprint arXiv:1811.00207

    2018 arXiv

  12. [20]

    Schweiger, David M. 1985. Measuring managerial cognitive styles: on the logical validity of the myers-briggs type indicator. Journal of Business Research , 13(4):315--328. https://doi.org/10.1016/0148-2963(85)90004-9

    1985 doi

  13. [21]

    Serapio-Garc \' a, Greg, Mustafa Safdari, Cl \'e ment Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, and Maja Matari \'c . 2023. Personality traits in large language models. arXiv preprint arXiv:2307.00184

    2023 arXiv

  14. [22]

    Yamshchikov

    Sorokovikova, Aleksandra, Natalia Fedorova, Sharwin Rezagholi, and Ivan P. Yamshchikov. 2024. LLMs simulate big five personality traits: Further evidence. arXiv preprint arXiv:2402.01765

    2024 arXiv

  15. [23]

    Sun, Xiaofei, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Pei, Tianwei Zhang, and Guoyin Wang. 2023. Text classification via large language models. arXiv preprint arXiv:2305.08377

    2023 arXiv

  16. [24]

    Trofimova, Ira. 2014. Observer bias: an interaction of temperament traits with biases in the semantic perception of lexical material. PloS one , 9(1):e85677

    2014

  17. [25]

    Tseng, Yu-Min, Yu-Chao Huang, Teng-Yun Hsiao, Yu-Ching Hsu, Jia-Yin Foo, Chao-Wei Huang, and Yun-Nung Chen. 2024. Two tales of persona in llms: a survey of role-playing and personalization. arXiv preprint arXiv:2406.01171

    2024 arXiv

  18. [26]

    Yi, Zihao, Jiarui Ouyang, Yuwen Liu, Tianhao Liao, Zhe Xu, and Ying Shen. 2024. A survey on recent advances in llm-based multi-turn dialogue systems. arXiv preprint arXiv:2402.18013

    2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.