REVIEW 8 cited by
Large Language Models Show Human-like Social Desirability Biases in Survey Responses
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
As Large Language Models (LLMs) become widely used to model and simulate human behavior, understanding their biases becomes critical. We developed an experimental framework using Big Five personality surveys and uncovered a previously undetected social desirability bias in a wide range of LLMs. By systematically varying the number of questions LLMs were exposed to, we demonstrate their ability to infer when they are being evaluated. When personality evaluation is inferred, LLMs skew their scores towards the desirable ends of trait dimensions (i.e., increased extraversion, decreased neuroticism, etc). This bias exists in all tested models, including GPT-4/3.5, Claude 3, Llama 3, and PaLM-2. Bias levels appear to increase in more recent models, with GPT-4's survey responses changing by 1.20 (human) standard deviations and Llama 3's by 0.98 standard deviations-very large effects. This bias is robust to randomization of question order and paraphrasing. Reverse-coding all the questions decreases bias levels but does not eliminate them, suggesting that this effect cannot be attributed to acquiescence bias. Our findings reveal an emergent social desirability bias and suggest constraints on profiling LLMs with psychometric tests and on using LLMs as proxies for human participants.
Forward citations
Cited by 8 Pith papers
-
Low Stage and High Order Explicit Runge--Kutta Methods via $Q$- and $D$-Conditions: Several Construction Details
A Q/D-space reformulation of Butcher simplifying assumptions yields sufficient order conditions and a recursive linear-system construction for explicit Runge-Kutta methods of even order p with s(p)=(p²-2p+8)/4 stages.
-
Low Stage and High Order Explicit Runge--Kutta Methods via $Q$- and $D$-Conditions: Several Construction Details
A Q/D-space framework supplies sufficient order conditions for explicit Runge-Kutta methods and supports a recursive construction of even-order methods with stage count (p²-2p+8)/4.
-
The Pinocchio Dimension: Phenomenality of Experience as the Primary Axis of LLM Psychometric Differences
The primary axis of psychometric variation among LLMs is the degree to which they represent themselves as loci of phenomenal experience rather than systems of behavioral responses.
-
Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?
Fine-tuning LLMs on essays reduces variance in IPIP-NEO responses across models but does not raise full five-trait profile accuracy above near-chance levels from unguided text.
-
The Granularity Axis: A Micro-to-Macro Latent Direction for Social Roles in Language Models
LLMs organize prompted social roles along a dominant, stable, and causally steerable granularity axis in representation space that runs from micro to macro levels.
-
I Understand How You Feel: Enhancing Deeper Emotional Support Through Multilingual Emotional Validation in Dialogue System
Releases M-EDESConv and M-TESC multilingual datasets and introduces MEGUMI model that fuses XLM-RoBERTa with emotion encoders for superior validation timing detection.
-
A Survey on Training-free Alignment of Large Language Models
A survey that catalogs and categorizes training-free LLM alignment methods into pre-decoding, in-decoding, and post-decoding, with a limited experimental comparison on one model.
-
Low Stage and High Order Explicit Runge--Kutta Methods via $Q$- and $D$-Conditions: Several Construction Details
A note giving the general sufficiency theorem, verification examples, p=10 ERK construction, linear systems, complexity analysis and coefficient tables for the Q/D-space order conditions.
Discussion (0). Sign in to comment.