Pith Number

pith:AO7CP7YR

pith:2025:AO7CP7YRR6UWOBKLXHMPQ4SXIG

not attested not anchored not stored refs pending

Human Psychometric Questionnaires Mischaracterize LLM Behavior

Dongmin Choi, Eun-Ju Lee, Jongwook Han, Woojung Song, Yohan Jo, Yoonah Park

LLM responses to human psychometric questionnaires substantially differ from their generation probabilities on real-world user queries.

arxiv:2509.10078 v4 · 2025-09-12 · cs.CL · cs.AI

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{AO7CP7YRR6UWOBKLXHMPQ4SXIG}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

The two profiles turn out to be substantially different and provide evidence that LLMs' responses to established questionnaires reflect desired behavior rather than stable psychological constructs, which challenges the consistent psychological dispositions of LLMs claimed in prior work.

C2weakest assumption

That generation probability scores of value- or personality-laden responses to real-world user queries accurately capture the LLMs' psychological characteristics expressed during interactions with users.

C3one line summary

Questionnaire-based and generation-based psychological profiles for LLMs are substantially different, indicating that established human questionnaires reflect desired behavior instead of stable psychological constructs.

Formal links

1 machine-checked theorem link

Cited by

1 paper in Pith

Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook

Receipt and verification

First computed	2026-06-01T02:03:26.332380Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

03be27ff118fa967054bb9d8f8725741837f2718c8d515ec63047df6ae320b2d

Aliases

arxiv: 2509.10078 · arxiv_version: 2509.10078v4 · doi: 10.48550/arxiv.2509.10078 · pith_short_12: AO7CP7YRR6UW · pith_short_16: AO7CP7YRR6UWOBKL · pith_short_8: AO7CP7YR

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/AO7CP7YRR6UWOBKLXHMPQ4SXIG \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 03be27ff118fa967054bb9d8f8725741837f2718c8d515ec63047df6ae320b2d

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "09be603bfdee750cb5415e56108a7c7d6e9a1f8991c91343038a881743e93d80",
    "cross_cats_sorted": [
      "cs.AI"
    ],
    "license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
    "primary_cat": "cs.CL",
    "submitted_at": "2025-09-12T09:14:42Z",
    "title_canon_sha256": "927259d1703836f385d2729fa5d0fef9560c5311460c0f82c681fc7375958df1"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2509.10078",
    "kind": "arxiv",
    "version": 4
  }
}