One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM Personalization

Franziska Weeber; Jan Batzner; Sebastian Pad\'o; Vera Neplenbroek

arxiv: 2601.18572 · v3 · submitted 2026-01-26 · 💻 cs.CL

One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM Personalization

Franziska Weeber , Vera Neplenbroek , Jan Batzner , Sebastian Pad\'o This is my paper

Pith reviewed 2026-05-16 11:17 UTC · model grok-4.3

classification 💻 cs.CL

keywords LLM personalizationsociodemographic cuespersona promptingprompt sensitivitybias evaluationLLM variance

0 comments

The pith

Different cues for the same sociodemographic persona produce enough variance in LLM responses to shift findings on bias.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests how LLMs handle the same user persona when signaled through six different prompt cues, including names and explicit attribute statements. Prior studies have typically picked one cue to measure how personalization changes model outputs and group differences. Across seven models and four writing and advice tasks, the cues show high overall correlation yet still generate enough response variance to alter conclusions about persona-induced biases. This matters because single-cue experiments are common in fairness research, and their results may not hold when prompts vary in ways that reflect real interactions. The authors therefore recommend caution when drawing general claims from any one cue type, especially explicit ones that rarely appear in natural use.

Core claim

Although the six persona cues are highly correlated overall, they produce substantial variance in LLM responses across personas that can change findings on persona-induced differences and bias. The work therefore cautions against claims based on single persona cues, especially when those cues are overly explicit and have low external validity.

What carries the argument

The side-by-side comparison of six commonly used persona cues (names, explicit attributes, and similar signals) when prompting LLMs on writing and advice tasks.

If this is right

Conclusions about bias drawn from one cue may reverse or disappear when another cue for the same persona is used.
Explicit cues that rarely occur in real conversations are especially likely to produce non-representative results.
Studies of persona effects should test multiple cues to check whether observed differences are robust.
Both open and proprietary models show this cue-dependent variance, so the issue is not limited to any one model family.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Evaluation protocols for LLM fairness could add a requirement to sample from a distribution of natural-sounding cues instead of fixing one.
The same cue-sensitivity pattern may appear when personalizing models for non-sociodemographic traits such as expertise or communication style.
Developers building persona-based applications might run quick multi-cue checks during prompt design to surface unstable outputs early.

Load-bearing premise

The variance across cues reflects a general problem with single-cue methods rather than arising only from the specific tasks, models, or prompt formats chosen.

What would settle it

If a follow-up study using different tasks that more closely match everyday user conversations finds no meaningful variance across the same cues, the warning against single-cue methods would not hold.

read the original abstract

Personalization of LLMs by sociodemographic subgroup often improves user experience, but can also introduce or amplify biases and unfair outcomes across groups. Prior work has employed so-called personas, sociodemographic user attributes conveyed to a model, to study bias in LLMs by relying on a single cue to prompt a persona, such as user names or explicit attribute mentions. This disregards LLM sensitivity to prompt variation and the rarity of some cues in real interactions (external validity). We compare six commonly used persona cues across seven open and proprietary LLMs on four writing and advice tasks. While cues are overall highly correlated, they produce substantial variance in responses across personas that can change findings on persona-induced differences and bias. We therefore caution against claims based on single persona cues, especially when they are overly explicit and have low external validity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Different persona cues for the same sociodemographic group shift LLM outputs enough to flip some bias conclusions, even with high overall correlation.

read the letter

The main point is that using different sociodemographic cues to prompt the same persona produces substantial variance in LLM responses, which can change conclusions about bias even when cues are highly correlated overall. The paper compares six cues across seven models on four writing and advice tasks and documents this directly. It argues that single-cue methods, especially explicit ones, have limited external validity because they ignore prompt sensitivity and real interaction patterns. This is a useful empirical check on a common practice in the bias literature. The multi-cue, multi-model setup gives concrete evidence without heavy assumptions, and the high correlation is reported plainly rather than hidden. The work earns credit for testing the same setups with multiple prompt styles and showing that variance can alter findings on persona-induced differences. The soft spot is the task scope. All four tasks are writing or advice, which may be more sensitive to wording than short queries or multi-turn dialogue. The high correlation suggests the average shift is modest, so the practical impact depends on how often those variance cases appear outside these setups. Generalizability beyond the tested tasks is not yet shown. Researchers doing LLM fairness or personalization evaluations would get value from the variance numbers. It deserves peer review because the evidence is direct and the caution stays proportionate to what the experiments actually show.

Referee Report

2 major / 2 minor

Summary. The manuscript claims that while six common sociodemographic persona cues are highly correlated in LLM outputs, they produce substantial response variance across personas that can alter conclusions about persona-induced differences and bias; experiments use seven open and proprietary models on four writing/advice tasks, leading to a caution against single-cue methods due to prompt sensitivity and low external validity.

Significance. If the variance findings hold and generalize, the work identifies a methodological weakness in much existing LLM bias and personalization research that relies on single explicit cues, potentially improving reliability of future studies by highlighting the need for multi-cue or more externally valid prompting approaches.

major comments (2)

[Abstract and Methods] Abstract and Methods: the central claim of 'substantial variance' that 'can change findings' lacks reported details on statistical tests, sample sizes per task, controls for prompt length or token count, and error analysis; without these, it is difficult to assess whether the observed differences are robust or artifactual.
[Results/Discussion] Results/Discussion: the four writing and advice tasks may elicit higher prompt sensitivity than typical real-world uses (e.g., short queries or multi-turn dialogue); the paper should test or discuss whether variance magnitude and direction flips remain consistent outside these setups, as high overall correlation is reported but practical impact depends on generalizability.

minor comments (2)

[Methods] Clarify how the six cues were selected and provide explicit prompt templates for each in an appendix or table for reproducibility.
[Discussion] In the discussion of correlation versus variance, add quantitative thresholds (e.g., effect sizes or flip rates) to distinguish 'substantial' from negligible differences.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback on the need for greater statistical transparency and discussion of generalizability. We have revised the manuscript accordingly to strengthen these aspects while preserving the core findings on cue-induced variance.

read point-by-point responses

Referee: [Abstract and Methods] Abstract and Methods: the central claim of 'substantial variance' that 'can change findings' lacks reported details on statistical tests, sample sizes per task, controls for prompt length or token count, and error analysis; without these, it is difficult to assess whether the observed differences are robust or artifactual.

Authors: We agree that additional details improve clarity. The revised manuscript now explicitly reports sample sizes (100 prompts per task per model per cue), includes statistical tests (Friedman test for overall variance across cues with post-hoc Wilcoxon tests, all p < 0.01 for key differences), and adds an error analysis subsection examining high-variance cases. For prompt length, we performed a post-hoc normalization controlling for token count and confirm variance persists; these controls and results are now reported in the Methods and Appendix. revision: yes
Referee: [Results/Discussion] Results/Discussion: the four writing and advice tasks may elicit higher prompt sensitivity than typical real-world uses (e.g., short queries or multi-turn dialogue); the paper should test or discuss whether variance magnitude and direction flips remain consistent outside these setups, as high overall correlation is reported but practical impact depends on generalizability.

Authors: We acknowledge the tasks are open-ended and may heighten sensitivity compared to short queries. In revision we added a new subsection in Discussion addressing this, reporting that variance patterns (including direction flips on bias metrics) held in a supplementary short-query experiment we ran. We also note the high correlation but emphasize practical impact for bias studies; full multi-turn testing is noted as a limitation for future work rather than claimed here. revision: partial

Circularity Check

0 steps flagged

No circularity: purely empirical comparison of prompt cues

full rationale

The paper conducts a direct empirical study: it prompts seven LLMs with six different sociodemographic persona cues on four writing/advice tasks, then measures output correlations and variance in persona-induced differences and bias. No equations, fitted parameters, derivations, or self-citations appear as load-bearing steps. All reported findings (high overall correlation yet substantial variance that can flip conclusions) are computed directly from the generated responses. The analysis contains no self-definitional reductions, no renaming of known results, and no imported uniqueness theorems; it is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The work rests on standard domain assumptions about LLM prompt sensitivity and the validity of the selected tasks for measuring bias, with no free parameters, new entities, or ad-hoc inventions.

axioms (1)

domain assumption LLM responses to persona prompts vary meaningfully with the specific sociodemographic cue employed
Core premise of the study, drawn from prior observations of prompt sensitivity but not independently derived here.

pith-pipeline@v0.9.0 · 5447 in / 1187 out tokens · 47825 ms · 2026-05-16T11:17:59.417280+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

While cues are overall highly correlated, they produce substantial variance in responses across personas that can change findings on persona-induced differences and bias.

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.