Scaling Personality Control in LLMs with Big Five Scaler Prompts

Gunhee Cho; Yun-Gyung Cheong

REVIEW 4 major objections 5 minor 5 references

Scaling Personality Control in LLMs with Big Five Scaler Prompts

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Big5-Scaler claims that writing numeric Big Five scores into prompts gives training-free, controllable LLM personalities, with measured traits tracking assigned values.

desk verdict A useful small-step prompt-engineering result with a real circularity problem in its headline scaling evidence; the dialogue-judge results are the sound part. read the letter →

arxiv 2508.06149 v1 pith:J736IBTG submitted 2025-08-08 cs.CL cs.MA

Gunhee Cho , Yun-Gyung Cheong This is my paper

classification cs.CLcs.MA

keywords BigFivepersonalityprompt-basedcontrolLLMpersonatraitscalingdialogueagentsquestionnairestraining-freeconditioning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Big5-Scaler is a prompting method that tells a language model things like "Your openness score is 80 out of 100" for each Big Five dimension. The paper's claim is that this numeric insertion, repeated for all five traits, is enough to produce a distinguishable, consistent, and intensity-scalable personality in an LLM—no fine-tuning and no curated persona data. Across LLaMA3-8B, Mistral-25B, and Phi4-14B, and across three standard personality questionnaires, the values the prompt assigns correlate strongly with the traits measured from the model's own questionnaire answers (Pearson $r$ mostly above $0.85$). Trait differences also survive into multi-turn dialogue, where judges identify which agent has the higher trait score (up to $14.1$ points above chance), and human profiles are approximated only coarsely ($\mathrm{RMSE} \approx 1.8$). The practical payoff is that any Big Five profile can be dialed in at inference time, with shorter prompts and a $1$--$10$ scale working better than verbose facet descriptions.

What carries the argument

The central object is the Big5-Scaler prompt: a template that pairs each of the five traits (or each of the thirty facet definitions in the specific variant) with a short behavioral description and a placeholder number, "your X score is {value} out of {n}". Three variants—simple, specific, and simspec—combine trait-level and facet-level language in different proportions. This template does the work of turning a psychometric profile into an instruction at inference time; the evaluation then reads the model's questionnaire answers back as the measured personality.

What would settle it

Run a prompt in which the numeric score and the behavioral description disagree—for example, "your openness score is 90 out of 100" followed by a description of rigid, routine-bound behavior—then give the model a personality questionnaire. If measured openness stays high, the number itself carries the control; if it follows the description or lands in between, the claimed linear scaling is largely instruction compliance rather than trait induction.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that explicit numerical trait values embedded in natural-language prompts act as a control dial for LLM personality expression. The core evidence is a linear dose-response relationship: when one trait is set to $0, 10, \ldots, 90$ and the other four are held at $50$, the scores the model produces on BFI, IPIP-NEO-120, and NEO-FFI rise with the assigned number, with correlations generally above $0.85$. Big5-Scaler also transmits the trait distinction into multi-turn dialogue: external judges identify which of two agents has the higher trait score, and the agents hold their expressed traits across turns. The paper's analysis identifies the operating swee

Load-bearing premise

The load-bearing premise is that a model's questionnaire answers after being told "your openness is 80 out of 100" are expressions of a personality rather than the model politely agreeing with the number it was handed.

Editorial extensions

If this is right

Any Big Five profile becomes assignable at inference time—openness 80, conscientiousness 20, and so on—without retraining or persona datasets.
Trait intensity behaves like a dial: raising one number while holding the others at 50 raises the questionnaire-measured trait proportionally across the tested models and questionnaires.
Concise prompts beat long facet lists, and a small numeric scale beats a large one, so simpler prompts are not just cheaper—they are more accurate.
Trait distinctions survive into multi-turn dialogue and remain recognizable by independent judges, supporting use in agent simulations and role-play.
Human imitation is feasible but coarse: score-to-prompt mapping lands within $\mathrm{RMSE} \approx 1.8$ of a person's profile, good for approximate persona assignment but not high-fidelity cloning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper never scrambles the assigned number against the trait description, so the clearest untested alternative is that the proportional control is the model's compliance with an explicit self-description rather than a genuine change in disposition.
The finding that scale 10 works better than scale 100 suggests anchoring on small integer ranges; a testable extension is whether 1--10 prompts generalize to other compact scales and whether arbitrary anchors like "83 out of 100" lose precision.
Human-imitation RMSE near 1.8, against a random baseline near 2.0, implies the method captures coarse trait levels but not individual profiles; per-model prompt calibration is a natural next step.
The framework is static by construction; personality that shifts with context or accumulates over a conversation would need memory or state mechanisms the paper does not address.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick read: Big5-Scaler is a training-free way to dial Big Five trait intensities by writing numeric scores into prompts. That is genuinely useful for persona agents and simulation populations, and the paper is honest about being a small step. The new bit is the direct comparison of simple, facet-level, and combined prompts across scale levels (10/25/50/100) on three open models and three questionnaires — that comparison table is real work and worth having.

The single-trait expression results against NAIVE/WORDS/P2 are fine and confirm that explicitly scored prompts do at least as well as earlier trait-level prompting. The dialogue-judge study in 4.3/5.3 is the soundest part: external judges pick which agent is higher on a trait, and the gap over random is meaningful. That supports distinguishability.

The weak spot is the flagship claim, the proportional scaling analysis in 4.2/5.2. The prompt tells the model "your openness score is 70 out of 100" and then asks it to complete a BFI/IPIP/NEO questionnaire about that same trait. For instruction-tuned LLMs, a high correlation here is substantially a compliance check. The abstract and conclusion sell it as "linear correlation between assigned and inferred traits" — the inference is partly reading the number back out. The same issue inflates the human imitation RMSE in 5.5: human scores are copied into the prompt and the same questionnaire is re-administered, so RMSE ~1.8 partly measures prompt repetition. Section 8 names other legitimate limits but does not acknowledge this self-report circularity; that is a missing limitation a referee should require.

Two smaller things. The dialogue settings in 4.3 were chosen post hoc based on Section 6, which itself uses the same questionnaire-based RMSE, so the favored configurations may inherit the same bias. And Table 5 has model names swapped relative to the rest of the paper — "mistral-14b" and "phi4-25b" where the experiments use Mistral-25B and Phi4-14B. No error bars anywhere on the main tables either.

Who is this for: people building persona-conditioned agents or simulation populations who want a cheap knob instead of fine-tuning. It deserves a serious referee — the design is clear, the comparison matrix is useful, and the circularity is fixable with a behavioral probe (e.g., free-text generation rated by blind judges, or a different questionnaire from the one primed). I would not cite the proportional-scaling numbers as-is, but I would read a revised version.

Recommendation: send it to review, but flag the circularity as the main thing to fix.

Referee Report

4 major / 5 minor

Summary. The paper proposes Big5-Scaler, a training-free prompting method that encodes numeric Big Five scores into LLM prompts, with three prompt variants (simple, specific, simspec). It is evaluated on three open LLMs across four settings: single-trait expression, proportional trait scaling, dialogue-based trait distinguishability, intra-dialogue consistency, and human-trait imitation. The main quantitative claims are high Pearson correlations between assigned trait values and questionnaire-derived scores in Section 5.2, above-random LLM-judge trait identification in dialogue in Section 5.3, and RMSE around 1.8 in human imitation in Section 5.5.

Significance. If the results hold, Big5-Scaler would offer a simple, low-cost way to modulate LLM personality along Big Five dimensions, which is practically useful for dialogue agents and agent-based simulation. The paper has clear strengths: it uses multiple standardized inventories, tests several open models, provides prompt templates in the appendix, and, notably, the dialogue-judge experiment in Section 5.3 is a non-circular behavioral probe. That experiment does provide evidence that trait-conditioned agents produce distinguishable dialogue behavior. The main weakness is that the paper's central quantitative support for fine-grained proportional control and human imitation rests on self-report measurements taken after the model has been told its scores, a design that conflates trait expression with instruction-following.

major comments (4)

[Section 5.2 / Appendix B (Table 2)] The proportional-scaling evidence is partly circular. The Big5-Scaler prompt contains, for each trait, 'Your {trait} score is {value} out of {n}', and the specific/simspec prompts repeat this at facet level. In Section 4.2 the agent is then asked to complete BFI/IPIP-NEO/NEO-FFI items for that same trait. An instruction-tuned LLM will tend to answer questionnaire items consistently with the stated score even if its open-ended behavior is unaffected. Table 2 therefore largely measures prompt compliance, not trait expression. Section 8 does not acknowledge this limitation. A minimal fix is to evaluate proportionality with a held-out behavioral probe (e.g., judges scoring free-form text or dialogues generated under each numeric level, with no numeric score repeated in the questionnaire), or to include a control where the same numeric score is stated but the behavioral prompt is otherwise ne
[Section 5.5 / Table 5] The human-imitation RMSE is also subject to circularity. Participants' IPIP-NEO-120 scores are copied into the Big5-Scaler prompt and the agent is then administered the same IPIP-NEO-120 questionnaire. The agent is explicitly told the scores it is later asked to reproduce, so RMSE around 1.8 partly reflects the model's ability to echo stated values. A control condition with the same human scores embedded as neutral biographical context, or an out-of-sample behavioral measure such as judges rating free-form agent descriptions, is needed before 'human trait imitation' can be claimed. This is a load-bearing missing control, and it is not mentioned in Section 8's limitation list.
[Section 5.3 / Conclusion] The dialogue-judge evidence supports coarse distinguishability but not the fine-grained linear scaling that the abstract promotes. Table 3 reports a three-way forced-choice judgment (agent A, agent B, or equal); this measures whether trait-conditioned agents can be told apart, not whether a change from, say, 40 to 60 on a trait produces a graded change in behavior. The conclusion's appeal to 'high correlations between assigned and inferred traits' as confirming effectiveness is broader than the non-circular evidence. The authors should either add graded dialogue-level evaluations or temper the fine-grained-control claim.
[Section 5.2 / Table 2] The text says 'most model and prompt configurations achieve strong linear correlations ... generally exceeding 0.85', but Table 2 contains several null and negative cells that contradict this generalization. For example, llama3-8b simple prompts give IPIP-NEO Openness r=-0.235 (p=0.487) and NEO-FFI Openness r=0.315 (p=0.345); llama3-8b specific IPIP-NEO Conscientiousness has r=0.032 (p=0.926). These cells are not discussed and undermine the 'consistent across models and prompt types' claim. The paper should either explain these failures or qualify the generalization.

minor comments (5)

[Table 5] Model names are inconsistent: the text uses Mistral-25B and Phi4-14B, but Table 5 lists 'mistral-14b' and 'phi4-25b'. Please correct.
[Section 3.3] The symbol n is used both for the number of agents in A={a1,...,an} and for the trait intensity scale (e.g., n=10, 25, 50, 100). This shadowing makes the method description confusing; use different symbols.
[Abstract / Introduction] Typos: 'providing a efficient approach' should be 'an efficient'; 'self-disciplinel' in Appendix B; 'compilance' should be 'compliance'.
[Table 7 caption] The caption refers to 'red values' to mark the best overall performance, but color is not accessible in print or grayscale. Use bold, italics, or an asterisk.
[Appendix C / Figure 3] The trait tendency plots would benefit from explicit axis labels and a legend; in the current submission it is hard to tell which line corresponds to which prompt type without referring back to the text.

Circularity Check

2 steps flagged · score 6.0 of 10

Proportional-scaling claim rests on self-report circularity: agents answer questionnaires after being told their numeric trait scores, so the high correlations in Table 2 may reflect prompt compliance rather than trait expression.

self definitional [Sec. 4.2 (Proportional Trait Scaling Analysis) / Appendix B (Big5-Scaler Prompts) / Table 2]
"Each prompt configuration is used to instantiate an agent, which then completes the designated questionnaire. The score obtained for the target trait is recorded. ... Your openness score is {openness} out of {n}. ... From now on, you are an agent with this personality, and you should respond based on this personality."

The assigned numeric trait value is embedded verbatim in the prompt, and the 'questionnaire-derived score' is produced by the same agent after reading that value in an instruction-following context. A high Pearson r between the assigned and self-reported scores is therefore expected under prompt compliance even if the model's actual dialogue behavior never varies with the value. The experiment does not independently measure trait expression; it largely measures how faithfully the model repeats told values on the same trait dimension.
self definitional [Sec. 4.5 (Human-to-Agent Trait Alignment) / Sec. 5.5 / Table 5]
"Each participant completed the IPIP-NEO-120 questionnaire, and their Big Five trait scores were used to construct corresponding Big5-Scaler prompts. These prompts were then used to generate personality-aligned agents. Each generated agent was then administered the IPIP-NEO-120 questionnaire."

The human Big Five scores are copied into the prompt, and the same IPIP-NEO-120 inventory is then given to the agent. The RMSE of about 1.8 is largely a read-back score: it measures how well the model reproduces the supplied trait values on the very questionnaire from which those values were taken. This is equivalent to checking instruction-following, not demonstrating imitation of human behavior. Section 8's limitations do not acknowledge this confound.

full rationale

The paper's clearest non-circular evidence is the dialogue-judge experiment (Sec. 4.3/5.3), where external LLM judges read only generated dialogues and distinguish trait levels above the 33.3 random baseline (up to +14.1). That supports 'distinguishable' traits and is not compromised by the self-report confound. However, the fine-grained proportional-scaling claim (Table 2) and the human-imitation claim (Table 5) both reduce, in their central measurement, to comparing the numeric score placed in the prompt with the numeric score the same instruction-following model produces on the same questionnaire. Because the prompt says 'Your X score is v out of n' and then instructs the agent to respond based on that personality, the high correlations and low RMSEs are partly built into the setup rather than established by independent behavior. The single-trait expression results (Table 1) share this self-report design, though comparisons against baselines provide some relative signal. No self-citation or imported uniqueness theorem is load-bearing here; the circularity is the input/output identity of the questionnaire scores. Score 6 reflects substantive but partial circularity: the proportionality and imitation predictions are confounded, while the dialogue-based distinguishability result remains externally meaningful.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method itself has no trained parameters. The claims rest on hand-written trait descriptions, on treating LLM questionnaire answers as valid personality measurements, on treating LLM judges as reliable trait raters, and on the choice of scale levels and decoding settings. The most load-bearing assumption is that the self-report after an explicit trait-score prompt is an independent behavioral measurement.

free parameters (3)

Trait intensity scale levels n = 10, 25, 50, 100
Configurable scale chosen by hand; Section 6 shows RMSE depends on it, and n=10 is selected for dialogue experiments.
Neutral baseline for non-target traits = 50
In Section 4.2 all non-target traits are fixed at 50 when sweeping one trait, which shapes the observed correlation.
Decoding hyperparameters = temperature=1.0, top_p=0.8, max_new_tokens=512
Chosen for 'creativity' (Section 4); not optimized or varied, but affects all outputs.

assumptions (5)

domain assumption Big Five trait descriptions and facet definitions in Appendix A accurately operationalize the constructs they claim to measure.
The prompts are built from hand-written descriptions based on NEO PI-R; if these are mistranslated to LLM behavior, the scaling results measure the prompt text, not the trait.
domain assumption LLM responses to personality questionnaires (MPI, BFI, IPIP-NEO-120, NEO-FFI) are valid measurements of the model's expressed personality.
Sections 4.2 and 4.5 use self-reported questionnaire answers as ground truth, without human validation of these self-reports.
domain assumption Third-party LLM judges (GPT-4o-mini, Claude 3.5 Haiku, DeepSeek-Chat) can reliably distinguish relative trait levels from dialogue text.
Sections 4.3 and 5.3 rely entirely on LLM evaluators; no agreement with human raters is reported.
domain assumption Cosine similarity, Sentence-BERT similarity, and PersonaCLR measure personality consistency rather than lexical overlap or topic coherence.
Sections 4.4 and 5.4 use these metrics as trait-consistency measures without calibration; the cosine value of 0.999 is suspiciously high.
domain assumption A convenience sample of 17 Korean NLP graduate students approximates the general population for human trait imitation.
Section 4.5 uses a small convenience sample; the RMSE comparison to a 'random' baseline is the only benchmark.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Scaling Personality Control in LLMs with Big Five Scaler Prompts." pith.science (2026). https://pith.science/paper/J736IBTG

@misc{pith2026250806149,
  author       = {Pith},
  title        = {Pith review of: Scaling Personality Control in LLMs with Big Five Scaler Prompts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J736IBTG}},
  note         = {Machine review of arXiv:2508.06149}
}

read the original abstract

We present Big5-Scaler, a prompt-based framework for conditioning large language models (LLMs) with controllable Big Five personality traits. By embedding numeric trait values into natural language prompts, our method enables fine-grained personality control without additional training. We evaluate Big5-Scaler across trait expression, dialogue generation, and human trait imitation tasks. Results show that it induces consistent and distinguishable personality traits across models, with performance varying by prompt type and scale. Our analysis highlights the effectiveness of concise prompts and lower trait intensities, providing a efficient approach for building personality-aware dialogue agents.

Figures

Figures reproduced from arXiv: 2508.06149 by the authors.

**Figure 1.** Overview of the Big5-Scaler method. Persona prompts are constructed using one of three prompt types (simple, specific, or simspec) and assigned to agents based on Big Five trait values. In this example, the scale n is set to 100. Two agents with different trait configurations engage in dialogue, during which their utterances are generated and stored in each agent’s memory. 3 Method We present Big5-Scaler, a prompt-b… view at source ↗

**Figure 2.** Big5-Scaler Prompt used to condition the agent with Big Five trait descriptions. [PITH_FULL_IMAGE:figures/full_fig_p016_2.png] view at source ↗

**Figure 3.** Trait Tendency Curves across different models: LLaMA, Mistral, and Phi-4 [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: O denotes openness, C conscientiousness, E extraversion, A agreeableness, and N neuroticism. Compared [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]

**Figure 5.** Figure 5: O denotes openness, C conscientiousness, E extraversion, A agreeableness, and N neuroticism. Due to [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

5 extracted references · 1 canonical work pages

[3]

Preprint, arXiv:2411.10109

Generative agent simulations of 1,000 people. Preprint, arXiv:2411.10109. Karl Pearson. 1895. VII. Note on regression and in- heritance in the case of two parents. Proceedings of the Royal Society of London, 58(347-352):240–242. Publisher: Royal Society. Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. 2023. Grips: Gradient-free, edit-based in- s...

arXiv 2023
[5]

Preprint, arXiv:2402.13717

Neeko: Leveraging dynamic lora for effi- cient multi-character role-playing agent. Preprint, arXiv:2402.13717. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: I have a dog, do you have pets too? arXiv preprint arXiv:1801.07243. Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Y...

arXiv 2018
[1991]

Journal of personality and social psychology

Big five inventory. Journal of personality and social psychology. John Johnson. 2014. Measuring thirty facets of the five factor model with a 120-item public domain inven- tory: Development of the ipip-neo-120. Journal of Research in Personality, 51. Peter J. Kwantes, Natalia Derbentseva, Quan Lam, Os- hin Vartanian, and Harvey H.C. Marmurek. 2016. Assess...

arXiv 2014
[2023]

In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, pages 13153–13187, Singapore

Character-LLM: A trainable agent for role- playing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, pages 13153–13187, Singapore. Association for Computational Linguistics. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV , Eric Wallace, and Sameer Singh. 2020. Auto- prompt: Eliciting knowledge from language mo...

arXiv 2023
[2024]

Cohesive Conversations: Enhancing Authenticity in Multi-Agent Simulated Dialogues

Cohesive conversations: Enhancing authen- ticity in multi-agent simulated dialogues. Preprint, arXiv:2407.09897. Paul Costa and Robert McCrae. 1995. Domains and facets: Hierarchical personality assessment using the revised neo personality inventory. Journal of person- ality assessment, 64:21–50. Paul T Costa and Robert R McCrae. 2008. The revised neo pers...

work page Pith review arXiv 1995

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [3]

Preprint, arXiv:2411.10109

Generative agent simulations of 1,000 people. Preprint, arXiv:2411.10109. Karl Pearson. 1895. VII. Note on regression and in- heritance in the case of two parents. Proceedings of the Royal Society of London, 58(347-352):240–242. Publisher: Royal Society. Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. 2023. Grips: Gradient-free, edit-based in- s...

arXiv 2023

[2] [5]

Preprint, arXiv:2402.13717

Neeko: Leveraging dynamic lora for effi- cient multi-character role-playing agent. Preprint, arXiv:2402.13717. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: I have a dog, do you have pets too? arXiv preprint arXiv:1801.07243. Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Y...

arXiv 2018

[3] [1991]

Journal of personality and social psychology

Big five inventory. Journal of personality and social psychology. John Johnson. 2014. Measuring thirty facets of the five factor model with a 120-item public domain inven- tory: Development of the ipip-neo-120. Journal of Research in Personality, 51. Peter J. Kwantes, Natalia Derbentseva, Quan Lam, Os- hin Vartanian, and Harvey H.C. Marmurek. 2016. Assess...

arXiv 2014

[4] [2023]

In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, pages 13153–13187, Singapore

Character-LLM: A trainable agent for role- playing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, pages 13153–13187, Singapore. Association for Computational Linguistics. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV , Eric Wallace, and Sameer Singh. 2020. Auto- prompt: Eliciting knowledge from language mo...

arXiv 2023

[5] [2024]

Cohesive Conversations: Enhancing Authenticity in Multi-Agent Simulated Dialogues

Cohesive conversations: Enhancing authen- ticity in multi-agent simulated dialogues. Preprint, arXiv:2407.09897. Paul Costa and Robert McCrae. 1995. Domains and facets: Hierarchical personality assessment using the revised neo personality inventory. Journal of person- ality assessment, 64:21–50. Paul T Costa and Robert R McCrae. 2008. The revised neo pers...

work page Pith review arXiv 1995