REVIEW 3 cited by
Personas with Attitudes: Controlling LLMs for Diverse Data Annotation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We present a novel approach for enhancing diversity and control in data annotation tasks by personalizing large language models (LLMs). We investigate the impact of injecting diverse persona descriptions into LLM prompts across two studies, exploring whether personas increase annotation diversity and whether the impacts of individual personas on the resulting annotations are consistent and controllable. Our results show that persona-prompted LLMs produce more diverse annotations than LLMs prompted without personas and that these effects are both controllable and repeatable, making our approach a suitable tool for improving data annotation in subjective NLP tasks like toxicity detection.
Forward citations
Cited by 3 Pith papers
-
Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models
LLM probability estimates violate the law of total probability across partitions, and subgroup-aggregated estimates often beat direct population-level estimates (the macro fallacy).
-
LLM-Ideoplasticity: Measuring Ideological Plasticity in the Political Behavior of LLMs as a Context-Conditioned Distribution
LLM political ideology behaves as a context-conditioned distribution with large local shifts but a narrow global Overton envelope, not a fixed point.
-
Political Ideology Shifts in Large Language Models
LLMs shift their Political Compass answers when adopting synthetic personas, with shifts growing with scale, asymmetric between right- and left-leaning cues, and tracking persona themes.
Discussion (0). Sign in to comment.