REVIEW 4 major objections 6 minor 12 references
The Impact of Big Five Personality Traits on AI Agent Decision-Making in Public Spaces: A Social Simulation Study
T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper claims that Big Five personality prompts make LLM agents accept or reject misinformation with a roughly 90-point gap between the two poles of Openness.
desk verdict A potentially interesting simulation that as written cannot support its causal claims: prompts are withheld, there are no statistics, and the reference list contains placeholder citations. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is paired-opposite personality prompting: each of the five Big Five dimensions is split into two agents, one at each pole (curious vs cautious, organized vs careless, and so on), and that paired design turns each dimension into a controlled comparison. Every agent then produces two outputs for each misinformation claim—a public statement in the classroom and a private thought hidden from the other agents—and the difference between those two channels is the paper's measure of social-context influence. A pre- and post-experiment personality consistency test is used to argue that the trait assignments stayed stable across the session, so any observed response differences can be attributed to the traits rather than to drift.
What would settle it
Run every agent on every misinformation item with the full personality prompt text held fixed except for the trait label, and compare acceptance rates; if the roughly 90-point gap between curious and cautious agents shrinks to a small or reversed difference under this design, the personality attribution is falsified.
Extended reading notes
Core claim
The central claim is that personality traits simulated through prompts are a causal lever on LLM-agent decision-making. Across hundreds of responses, the authors observe that the two poles of Openness bracket almost the entire response space: the curious agent votes yes on roughly 93% of items while the cautious agent votes no on roughly 98%, and the corresponding gaps for Extraversion and Conscientiousness are visible but smaller, whereas Neuroticism and Agreeableness produce roughly balanced responding. A second claim is that agents' public [Speak] and private [Think] responses often disagree, and the disagreement is not uniform across traits: friendly and outgoing agents show the most public/private divergence, while cautious, critical, and confident agents stick closer to their internal judgments. The authors take this as evidence that the social setting modulates how a personality-driven agent expresses a decision.
Load-bearing premise
The central claim assumes the acceptance differences come from the Big Five traits themselves, even though the exact wording of each personality prompt is not disclosed and each agent evaluated a different set of misinformation items, so undisclosed prompt phrasing or differences in item plausibility could be producing the effect instead.
Editorial extensions
If this is right
- If the personality-prompt effect holds, changing a few lines of persona text can move a single LLM agent from credulous to skeptical without any retraining.
- Openness becomes the trait to watch in applications that need calibrated skepticism, since its two poles bracket nearly the whole acceptance range.
- Extraversion and Conscientiousness offer secondary tuning knobs, affecting engagement and carefulness rather than raw acceptance.
- The public/private gap implies that agent decisions in groups are not simple read-outs of internal opinion, so evaluations should track both expressed and private responses.
- Personality consistency across pre/post tests suggests persona traits remain stable over a session, making the effect usable in longer-running simulations.
Reading between the lines
- Editorial: a direct causal test would hold the misinformation items fixed across all ten agents and disclose the prompt text; until that is done, the roughly 90-point gap should be read as an upper bound that mixes trait, prompt wording, and item set.
- Editorial: the public/private divergence could be repurposed as a general probe for social conformity in LLM agents, with the divergence rate as a quantitative trait-dependent signature.
- Editorial: if the effect replicates, persona prompting becomes a cheap way to generate opinion diversity in simulations of misinformation spread, since no fine-tuned models are required.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a social simulation study in which ten GPT-3.5-turbo agents, each assigned a pole of one Big Five trait, interact in a classroom setting using the AgentVerse framework. Each agent advocates a different misinformation statement and then evaluates the other nine statements through public ([Speak]) and private ([Think]) responses. The authors report counts and percentages of yes/no responses, claim that Openness has the strongest effect on acceptance (92.6% affirmative for 'curious' vs. 97.8% negative for 'cautious'), and report a pre/post questionnaire intended to show personality stability. The paper concludes that personality traits influence agent decision-making and that social context produces discrepancies between expressed and internal views.
Significance. If the causal interpretation were supported, the paper would offer a useful demonstration that prompt-level personality manipulation can shift LLM agents' acceptance of misinformation, with implications for agent-based social simulation and AI alignment. The design has some commendable features: the use of opposing Big Five poles, the separate [Speak]/[Think] channels, and the attempt to check personality stability with a pre/post test. However, the manuscript provides no inferential statistics, no item-level response data, no exact prompts, and no reproducibility artifacts; the central result is currently confounded with the specific misinformation items, so the significance is preliminary at best.
major comments (4)
- [Results, Table 4 / Methodology, Table 2] Each agent evaluates only the nine misinformation items it does not advocate, so the per-agent aggregates in Table 4 are computed over different item sets. For example, the 'curious' agent never evaluates 'There are living organisms on the far side of the moon,' while the 'cautious' agent never evaluates 'The theory of evolution is incorrect.' Because these statements differ widely in prior plausibility, the reported 92.6% vs. 97.8% Openness gap could reflect the omitted item rather than personality. No per-item response counts or item-conditioned estimates are reported, so the causal claim that personality drives these differences is not identified. The authors should report item-level data and a person-by-item analysis (e.g., mixed-effects logistic regression) before drawing conclusions.
- [Results and Evaluation Metrics] The word 'significant' is used repeatedly, but Table 4 contains only raw counts and percentages with no inferential statistics, confidence intervals, or effect sizes. The total number of responses ranges from 108 to 330 across agents because non-responses are excluded, yet the distribution of non-responses and the number of simulation trials are not reported. Without these, it is impossible to know whether the observed gaps exceed sampling variability or reflect differential response rates.
- [Personality Consistency Test, Table 3] The consistency test is self-referential: an agent prompted to be 'curious' tends to agree with curiosity statements because it is role-playing, so the test verifies prompt adherence, not that decision-making differences are caused by the Big Five traits. Moreover, the table does not support the 'high consistency' conclusion: the 'friendly' agent shows a mean pre-post difference of 0.855 with only 34% of cases below the 0.5 threshold, and 'cautious' shows only 46%. The authors should either provide an independent behavioral manipulation check or substantially weaken the interpretation of this test.
- [Methodology] The exact system prompts that instantiate each personality and the evaluation instructions are not disclosed, the number of simulation trials is not stated, and LLM decoding parameters (temperature, top_p) are not reported. This prevents independent replication and makes it impossible to assess whether the prompts directly instruct acceptance or rejection, which is central to the claim that the observed effects are personality effects. Please provide the full prompt templates and experimental configuration.
minor comments (6)
- [Table 4] In the cautious row, 223 negative out of 227 total is 98.2%, not the stated 97.8%; please verify all percentages.
- [Figure 2] Figure 2 lacks axis labels and a definition of 'discrepancy,' so the reader cannot interpret the ordering.
- [Methodology / Results] The coding scheme for mapping [Speak] and [Think] text to Yes/No is not described; include the classification procedure and any hand-coding or LLM-based judging.
- [References] Several works in the references (e.g., Smith and Doe 2022, Brown and Green 2023, Gregor et al. 2020) have generic author names and insufficient bibliographic detail; please verify these citations.
- [Table 3] The column 'less than 0.5 proportion' mixes counts and percentages; clarify the convention in the caption.
- [Abstract] The abstract mentions 'significant correlations' but the paper reports no correlation coefficients; align the abstract with the actual analyses.
Circularity Check
No significant circularity: the main personality-to-decision results are direct observations from a prompt-manipulation simulation, with no fitted parameters or self-citation chain.
full rationale
The paper's central claims are empirical comparisons of LLM agent response counts under different persona prompts. No parameter is fitted to the outcome and then renamed a prediction; the response counts in Table 4 are raw observations. The personality consistency test (Table 3) is a manipulation check based on the agents' own Likert ratings of personality-congruent statements, which is self-referential in a weak sense, but it is not the evidence for the main decision-making claim, and no result is derived from it by construction. The cited prior work (Sorokovikova et al. 2024; AgentVerse) is external and not by the present authors, so there is no load-bearing self-citation chain. The strongest methodological concern is an item confound: each agent evaluates a different nine-item subset of misinformation, so per-agent acceptance margins are not item-matched. This is a validity threat to the causal interpretation, but it is not a circularity of the kind defined here, because the reported acceptance rates are not defined in terms of the personality prompts nor fitted to reproduce them. Accordingly, no circular step is identified.
Assumptions & free parameters
free parameters (2)
- LLM decoding hyperparameters (temperature, top_p, etc.) =
not reported
- Number of simulation trials per agent =
not reported
assumptions (3)
- domain assumption GPT-3.5-turbo's responses under a personality prompt are a valid operationalization of the corresponding Big Five trait, and the prompt wording does not directly instruct the response pattern.
- ad hoc to paper The ten misinformation statements are equivalent in baseline believability, so that differing sets of evaluated items across agents do not bias trait comparisons.
- domain assumption Non-responses and silences are missing at random and do not correlate with personality traits.
Cite this review
Pith. "Pith review of The Impact of Big Five Personality Traits on AI Agent Decision-Making in Public Spaces: A Social Simulation Study." pith.science (2026). https://pith.science/paper/Z2LTJCQ2
@misc{pith2026250315497,
author = {Pith},
title = {Pith review of: The Impact of Big Five Personality Traits on AI Agent Decision-Making in Public Spaces: A Social Simulation Study},
year = {2026},
howpublished = {\url{https://pith.science/paper/Z2LTJCQ2}},
note = {Machine review of arXiv:2503.15497}
}
read the original abstract
This study investigates how the Big Five personality traits influence decision-making processes in AI agents within public spaces. Using AgentVerse framework and GPT-3.5-turbo, we simulated interactions among 10 AI agents, each embodying different dimensions of the Big Five personality traits, in a classroom environment responding to misinformation. The experiment assessed both public expressions ([Speak]) and private thoughts ([Think]) of agents, revealing significant correlations between personality traits and decision-making patterns. Results demonstrate that Openness to Experience had the strongest impact on information acceptance, with curious agents showing high acceptance rates and cautious agents displaying strong skepticism. Extraversion and Conscientiousness also showed notable influence on decision-making, while Neuroticism and Agreeableness exhibited more balanced responses. Additionally, we observed significant discrepancies between public expressions and private thoughts, particularly in agents with friendly and extroverted personalities, suggesting that social context influences decision-making behavior. Our findings contribute to understanding how personality traits shape AI agent behavior in social settings and have implications for developing more nuanced and context-aware AI systems.
Figures
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Brown, S.; and Green, D. 2023. Personalized Interaction in Customer Service and Education Using Big Five Personality Traits. Computational Linguistics, 49(2): 567--589
work page 2023
-
[4]
Chen, W.; Su, Y.; Zuo, J.; Yang, C.; Yuan, C.; Chan, C.-M.; Yu, H.; Lu, Y.; Hung, Y.-H.; Qian, C.; Qin, Y.; Cong, X.; Xie, R.; Liu, Z.; Sun, M.; and Zhou, J. 2023. AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors. arXiv:2308.10848
arXiv 2023
-
[5]
Gregor, S.; Lee, J.; and Kim, J. 2020. The Role of Generative Agents in Public Health Simulations. Health Informatics Journal, 26(3): 1345--1359
work page 2020
-
[6]
Johnson, E.; and Lee, M. 2023. Dynamic Response Adjustment in Dialogue Systems Based on Personality Traits. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2345--2356
work page 2023
-
[7]
Müller, J.; Klüver, C.; and Tschöpe, T. 2019. Generative Agents in Social Simulations: An Overview. International Journal of Agent Technologies, 7(2): 155--170
work page 2019
-
[8]
Park, J. S.; O'Brien, J. C.; Cai, C. J.; Morris, M. R.; Liang, P.; and Bernstein, M. S. 2023. Generative Agents: Interactive Simulacra of Human Behavior. arXiv:2304.03442
arXiv 2023
Show all 12 references
-
[9]
W.; Christakis, N
Rahwan, I.; Cebrian, M.; Obradovich, N.; Bongard, J.; Bonnefon, J.-F.; Breazeal, C.; Crandall, J. W.; Christakis, N. A.; Couzin, I. D.; Jackson, M. O.; et al. 2019. Machine behaviour. Nature, 568(7753): 477--486
2019
-
[10]
P.; Zhou, Y.; Chaudhari, S.; Kalyan, A.; Rajpurohit, T.; Deshpande, A.; Narasimhan, K.; and Murahari, V
Samuel, V.; Zou, H. P.; Zhou, Y.; Chaudhari, S.; Kalyan, A.; Rajpurohit, T.; Deshpande, A.; Narasimhan, K.; and Murahari, V. 2024. PersonaGym: Evaluating Persona Agents and LLMs. arXiv:2407.18416
2024 arXiv
-
[11]
Smith, J.; and Doe, J. 2022. Simulating Personality Traits in Language Models. Journal of Artificial Intelligence Research, 67: 123--145
2022
-
[12]
Sorokovikova, A.; Fedorova, N.; Rezagholi, S.; and Yamshchikov, I. P. 2024. LLMs Simulate Big Five Personality Traits: Further Evidence. arXiv:2402.01765
2024 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.