REVIEW 8 cited by
Quantifying the Persona Effect in LLM Simulations
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) have shown remarkable promise in simulating human language and behavior. This study investigates how integrating persona variables-demographic, social, and behavioral factors-impacts LLMs' ability to simulate diverse perspectives. We find that persona variables account for <10% variance in annotations in existing subjective NLP datasets. Nonetheless, incorporating persona variables via prompting in LLMs provides modest but statistically significant improvements. Persona prompting is most effective in samples where many annotators disagree, but their disagreements are relatively minor. Notably, we find a linear relationship in our setting: the stronger the correlation between persona variables and human annotations, the more accurate the LLM predictions are using persona prompting. In a zero-shot setting, a powerful 70b model with persona prompting captures 81% of the annotation variance achievable by linear regression trained on ground truth annotations. However, for most subjective NLP datasets, where persona variables have limited explanatory power, the benefits of persona prompting are limited.
Forward citations
Cited by 8 Pith papers
-
MyMentorLLM: A psychotherapy GenAI environment with multimodal voice/text patients, trainees and experts for deliberate practice
MyMentorLLM generates 2,100 multimodal CBT training sessions and finds that native speech-to-speech simulation matches human therapy competence scores while smaller models overestimate trainee skills and suffer from h...
-
Point of Order: Action-Aware LLM Persona Modeling for Data-Grounded Civic Deliberation
Fine-tuning on speaker-attributed, action-tagged transcripts from public meetings lets LLM agents mimic government meeting participants well enough that human judges often cannot tell them from real people.
-
Measuring AI Alignment with Human Flourishing
The authors propose the Flourishing AI Benchmark, which uses 1,229 objective and subjective questions plus LLM judges to score 28 chatbots across seven dimensions of human flourishing, and find none reach the 90-point...
-
Aligning LLM with human travel choices: a persona-based embedding learning approach
A persona-based embedding learning framework aligns LLM predictions with human travel mode choices, outperforming MNL and few-shot LLM baselines on the Swissmetro dataset.
-
Exploring Silicon-Based Societies: An Early Study of the Moltbook Agent Community
Clustering of Moltbook submolt descriptions shows agent-created communities organize into human-mimetic, silicon-centric, and proto-economic themes, but the categories were partly prescribed by the analysis prompt.
-
From Risk Perception to Behavior Large Language Models-Based Simulation of Pandemic Prevention Behaviors
LLM-based simulations of pandemic prevention behaviors show moderate distributional agreement with Beijing survey data, but the validation uses a lenient threshold and selective reporting.
-
Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests
Proposes SJTs and MIRT to measure consistent latent behavioral tendencies in LLMs, showing stability and predictive validity on external benchmarks.
-
SimuPanel: A Novel Immersive Multi-Agent System to Simulate Interactive Expert Panel Discussion
A multi-agent LLM system called SimuPanel simulates expert panel discussions with personas grounded in public academic sources, and a small evaluation suggests the full reasoning pipeline produces higher LLM-judged di...
Discussion (0). Sign in to comment.