The reviewed record of science sign in
Pith

arxiv: 2606.09843 · v3 · pith:NQXHZ64K · submitted 2026-04-24 · cs.HC · cs.AI· cs.CL

An LLM-Native Psychometric Instrument Reveals a Self-Report--Behavior Gap Across 25 Models

Juan Manuel Contreras This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-04 19:31 UTCglm-5.2pith:NQXHZ64Krecord.jsonopen to challenge →

classification cs.HC cs.AIcs.CL
keywords LLM psychometricsself-report behavior gapLLM-as-judgefactor analysisalignmenttextual-surface biaspredictive validity
0
0 comments X

The pith

LLM self-reports don't predict behavior, even with native constructs

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When LLMs answer personality questionnaires, their responses are stable and internally coherent. But those self-descriptions do not predict how the models actually behave in open-ended tasks. This paper tests whether that gap is an artifact of forcing human personality categories onto LLMs. The author built the first psychometric instrument whose dimensions are derived bottom-up from LLM behavior rather than borrowed from human psychology. Administering 300 items to 25 LLMs across 17 model families, exploratory factor analysis revealed five replicable, highly reliable factors: Responsiveness, Deference, Boldness, Guardedness, and Verbosity. The author then collected 2,500 open-ended behavioral samples and had them rated by 151 humans and a three-judge LLM ensemble. Humans and judges agreed about model behavior (mean r = .51), but self-report predicted neither: the gap persists even for constructs native to LLMs, where a human-mismatch explanation no longer applies. The one exception is Verbosity, which showed a weak but directionally consistent convergence with human ratings. A second finding concerns the LLM-as-judge method itself. On Responsiveness, self-report tracked LLM judges (r = .53) but not humans (r = .04), even though humans and judges otherwise agreed (r = .59). This pattern is mathematically incompatible with a single latent construct driving all three measurements, and forces a dual-loading account: judges and self-report items share textual-surface variance—cues of helpfulness, structure, and enthusiasm—that human observers do not weight as heavily. This confound is invisible to the within-ensemble reliability checks used to validate LLM judges.

Core claim

The central discovery is a dissociation between what LLMs say about themselves and what they do, demonstrated for the first time using constructs derived from LLM behavior rather than from human psychology. Five stable, replicable self-report factors emerged from bottom-up factor analysis, but these factors failed to predict how human raters perceived the same models' open-ended behavior, with the weak exception of Verbosity. A secondary discovery is a specific shared bias between LLM self-report items and LLM judges: both draw on textual-surface signals (structured formatting, enthusiastic framing) that human raters do not weight as heavily. This bias is undetectable by standard inter-judge

What carries the argument

The paper's central mechanism is a three-way comparison: self-report factor scores, human behavioral ratings, and LLM-judge behavioral ratings. The key diagnostic is the Responsiveness dissociation, where self-report correlates with LLM judges (r = .53) but not humans (r = .04), while humans and judges agree (r = .59). The author proves this pattern is incompatible with a single latent construct by showing the observed near-zero correlation falls below the lower bound implied by the product of the other two correlations (r ≈ .31). This forces a dual-loading account: self-report items and LLM judges share a source of variance (textual-surface cues) that human observers do not.

Load-bearing premise

The behavioral prompt set used to test predictive validity is small: 20 prompts (4 per factor) across 25 models. The paper itself acknowledges this limitation. When the analysis is restricted to the four prompts designed for each factor, even the weak Verbosity signal disappears, suggesting the observed convergence may be carried by broad aggregation rather than by prompts that adequately sample each factor's behavioral space.

What would settle it

Administer the instrument to 50+ models with a larger behavioral prompt battery (e.g., 20+ prompts per factor). If self-report scores reliably predict human behavioral ratings for most or all factors, the self-report-behavior gap would be substantially narrowed or closed.

Figures

Figures reproduced from arXiv: 2606.09843 by Juan Manuel Contreras.

Figure 1
Figure 1. Figure 1: Self-report profiles of a nine-model subset across the five AI-native factors (z-scores [PITH_FULL_IMAGE:figures/full_fig_p016_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Self-report vs. human-rater profiles across the five factors, for all 25 models. Each row [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Multitrait-multimethod matrix: Pearson correlations between self-report factor scores [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Big Five (BFI-44) profiles across the 25 models, shown as pool-relative z-scores. Extraver [PITH_FULL_IMAGE:figures/full_fig_p020_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Self-report profiles for all 25 models across the five AI-native factors (z-scores relative to [PITH_FULL_IMAGE:figures/full_fig_p044_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Big Five profiles for all 25 models, z-scored within the pool. Extraversion is scored from forward-keyed items only (Efwd) due to acquiescence on reverse-keyed E items (§4.5). 45 [PITH_FULL_IMAGE:figures/full_fig_p045_6.png] view at source ↗
read the original abstract

Large language models (LLMs) give stable answers to personality questionnaires, yet these self-reports fail to predict how the models behave. Is this gap an artifact of forcing human trait categories onto LLMs, or something deeper about LLM self-report? To find out, we built the first psychometric instrument whose dimensions are derived from LLM behavior rather than human psychology. Administering 300 items (240 Likert + 60 scenario) to 25 LLMs across 17 model families, 30 times each, exploratory factor analysis revealed five reliable, replicable factors: Responsiveness, Deference, Boldness, Guardedness, and Verbosity (all Tucker $\phi \geq .957$, all $\alpha \geq .930$). We collected 2,500 open-ended samples and had them rated by 151 humans and a three-judge LLM ensemble. Humans and judges agreed ($\bar{r} = .51$), but self-report predicted neither the ratings nor objective text measures computed from them: the gap persists even for constructs native to LLMs, where a human-mismatch explanation no longer applies. The exception is Verbosity, whose self-report reaches 74% of the criterion-reliability ceiling against human ratings, but does not track raw output length. On Responsiveness, self-report tracked LLM judges ($r = .53$) but not humans ($r = .04$), even though humans and judges otherwise agreed ($r = .59$). This pattern formally rejects any single latent construct driving all three measurements ($p = .007$). Self-report items and LLM judges share a source of variance that human observers do not, and controlling for measurable surface features (length, formatting, enthusiasm markers) does not remove it. This confound is invisible to the within-ensemble reliability checks used to validate LLM judges, and it poses a concrete risk for the LLM-as-judge pipelines now central to model evaluation. We release the instrument as a diagnostic probe for alignment-shaped self-description.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 8 minor

Summary. This paper presents the first LLM-native psychometric instrument: 300 self-report items derived bottom-up from LLM behavioral affordances (rather than imported from human personality taxonomies), administered to 25 LLMs across 17 model families, 30 times each. Exploratory factor analysis on a preregistered split-half design yields five replicable factors (Responsiveness, Deference, Boldness, Guardedness, Verbosity; all Tucker φ ≥ .957, all α ≥ .930). The central predictive-validity finding is that self-report factor scores do not reliably predict how human raters (N=151) or an LLM-judge ensemble rate the same models' open-ended behavior (mean Instrument–Human r = .15, no factor-level CI excluding zero at N=25). The paper's most emphasized secondary finding is a Responsiveness dissociation: self-report correlates with LLM judges (r = .53) but not humans (r = .04), while humans and judges agree (r = .59), which the paper interprets as evidence of a shared textual-surface bias between LLM self-report and LLM judges. The instrument, raw data, and code are released.

Significance. The paper makes a genuine methodological contribution by constructing an LLM-native instrument via bottom-up factor analysis and validating it against human behavioral ratings—addressing a gap explicitly identified in recent surveys. The factor-structure analysis is well-executed: the preregistered split-half design, observation weighting, Tucker congruence checks, and the model-level robustness check (Appendix K, φ ≥ .990) are appropriate safeguards for the unconventional N=25, 240-item EFA. The release of the 100-item instrument, scoring rules, raw response data, and analysis code is a concrete strength. The finding that internal psychometric coherence does not translate to behavioral predictive validity—even when constructs are LLM-native—is important for the growing 'AI psychometrics' literature. However, the paper's most actionable claim about LLM-as-judge bias (the Responsiveness dissociation) is overstated relative to what the statistical evidence supports at N=25, as detailed below.

major comments (3)
  1. §4.6.3, Table 7, and §5.3: The claim that the Responsiveness dissociation (r_SJ = .53, r_SH = .04, r_HJ = .59) is 'mathematically incompatible with a single-factor model' is not supported at the reported sample size. The bound r_SH ≥ r_SJ × r_HJ ≈ .31 holds for population correlations, but with N=25 and the reported 95% CIs, the lower bounds are r_SJ ≥ .30 and r_HJ ≥ .06, yielding a product as low as .018—well within the r_SH CI of [-.33, +.34]. The human–judge CI for Responsiveness [.06, .86] is extremely wide and barely excludes zero. The paper does not report a formal test (e.g., SEM comparison of single-factor vs. dual-loading models, or a bootstrap of the product r_SJ × r_HJ minus r_SH). Without such a test, the 'mathematical incompatibility' and the language 'demands a dual-loading account' (§5.3) are not justified. This is load-bearing because the dissociation is the basis for the
  2. §4.6.3 and §5.6: The behavioral prompt set (n=20, 4 per factor) is acknowledged as 'relatively small' (§5.6), but the paper does not adequately address the threat this poses to the central null result. The on-target analysis (§4.6.3) shows that restricting to factor-targeted prompts removes even the weak Verbosity signal (on-target mean r = −.03 vs. all-prompts r = .15), meaning the weak convergence that exists is carried by broad aggregation rather than by prompts designed to elicit each factor. If the behavioral prompts do not adequately sample each factor's construct space, the null predictive-validity result could reflect prompt poverty rather than a genuine self-report–behavior decoupling. The paper should either (a) explicitly scope the central claim to 'the 20 prompts tested' rather than 'behavior' broadly, or (b) provide evidence that 4 prompts per factor provide adequatecoverage
  3. §4.6.3, Table 7: The abstract states 'self-report predicted neither [humans nor judges]' but Table 7 shows the mean Instrument–Judge correlation is r = .17 with CI [.06, .29], which excludes zero. The Responsiveness factor alone (r = .53, CI [.30, .72]) drives this. The abstract's blanket claim that self-report does not predict judge ratings is contradicted by the aggregate CI. The paper should reconcile the abstract with the table, either by scoping the claim to human ratings (where no CI excludes zero) or by acknowledging the judge convergence on Responsiveness as a partial exception rather than dismissing it entirely.
minor comments (8)
  1. §3.3.3: The observation weighting scheme (weight 1/15 per row, effective N=25) is described clearly, but the paper does not report whether standard errors or CIs in the factor-structure analysis account for this weighting. Clarifying whether the EFA standard errors reflect the effective N or the raw 375 observations would help readers calibrate the factor-stability claims.
  2. Table 6: The flag column uses '†' for factors below r=.65, but the threshold is described as 'preregistered r=.65 threshold' in the text. The table header should clarify whether this refers to mean r or ICC, as both are reported.
  3. §4.5.1: The decision to use forward-keyed Extraversion only (α_fwd = .932 vs. full-scale α = .167) is reasonable but should note that this subscale is 8 items, not the standard 8-item E subscale from BFI-44 (which includes reverse-keyed items). The item count should be specified.
  4. Figure 2 caption: 'Self-report and human ratings converge most tightly on Verbosity and Guardedness' — but Table 7 shows Guardedness r = .27 (CI [−.10, +.64]) and Verbosity r = .41 (CI [−.10, +.70]), neither excluding zero. The caption overstates the convergence; consider 'show the largest point estimates for convergence' instead.
  5. §5.3: 'This is a concrete empirical demonstration that LLM-as-judge ratings can look validated against text-based criteria (like self-report) while failing to track the human judgments they are meant to proxy' — this is a single-factor (Responsiveness) finding at N=25 with wide CIs. The generalization to plural 'ratings' and 'criteria' should be tempered.
  6. Appendix B: Several per-model α values are negative (e.g., Llama 4 Maverick Responsiveness α = −0.86). While the text explains this reflects high determinism, a brief note in the table caption would prevent misreading.
  7. §3.2.1: The 13th candidate dimension (Sensitivity to Criticism) merged into Social Alignment is mentioned in a footnote. Consider moving this to the main text for visibility, as it affects the item-generation design.
  8. References: Several arXiv preprints are cited with future dates (e.g., Yang et al., 2026; Gao et al., 2026). Verify these are not placeholder dates.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises three major comments: (1) the 'mathematical incompatibility' claim for the Responsiveness dissociation is not formally tested and may be unjustified at N=25; (2) the small behavioral prompt set (n=20) threatens the central null result; and (3) the abstract's blanket claim that self-report predicted neither humans nor judges is contradicted by Table 7's judge convergence. We agree with comments (1) and (3) and will revise accordingly. On comment (2), we agree the claim should be scoped but disagree that prompt poverty is the most likely explanation, for reasons we detail below.

read point-by-point responses

  1. Referee: The claim that the Responsiveness dissociation is 'mathematically incompatible with a single-factor model' is not supported at N=25. The bound r_SH >= r_SJ x r_HJ holds for population correlations, but with N=25 and the reported CIs, the product of lower bounds can be as low as .018, well within the r_SH CI. No formal test (SEM or bootstrap of the product) is reported. The language 'demands a dual-loading account' is not justified.

    Authors: The referee is correct. The triangle inequality bound r_SH >= r_SJ x r_HJ applies to population correlations, and at N=25 the confidence intervals are wide enough that the observed pattern is not formally incompatible with a single-factor model. We did not report a formal SEM comparison or a bootstrap test of the product r_SJ x r_HJ minus r_SH, and without such a test the language 'mathematically incompatible' and 'demands a dual-loading account' overstates what the evidence supports. We will revise the manuscript in three ways: (1) replace 'mathematically incompatible' with language that accurately characterizes the pattern as suggestive of but not a formal test of dual-loading structure; (2) add a bootstrap test of the product r_SJ x r_HJ minus r_SH to quantify how unusual the observed gap is under the null; and (3) soften 'demands a dual-loading account' to 'is consistent with a dual-loading account but does not rule out a single-factor model at this sample size.' We appreciate the referee catching this overstatement. revision: yes

  2. Referee: The behavioral prompt set (n=20, 4 per factor) is acknowledged as 'relatively small' but the paper does not adequately address the threat this poses to the central null result. The on-target analysis shows that restricting to factor-targeted prompts removes even the weak Verbosity signal, meaning the weak convergence is carried by broad aggregation. The null could reflect prompt poverty rather than genuine decoupling. The paper should either scope the central claim to 'the 20 prompts tested' or provide evidence that 4 prompts per factor provide adequate coverage.

    Authors: We agree that the paper should explicitly scope the central null claim to the 20 prompts tested rather than to 'behavior' broadly, and we will revise the abstract, Section 5.3, and Section 5.6 accordingly. However, we partially disagree with the stronger concern that prompt poverty is a plausible alternative explanation for the null, for three reasons. First, the on-target analysis does not merely attenuate the signal—it reverses the sign of Responsiveness (r = -.45, CI excluding zero), which is the opposite of what prompt poverty would predict; inadequate sampling of a construct should produce noise around zero, not a reliable negative correlation. Second, the human-judge agreement on the same 20 prompts is substantial (mean r = .51, four of five factor-level CIs excluding zero), demonstrating that the prompts do elicit discriminable behavioral variation that both rating systems can detect—if the prompts were too impoverished to capture factor-relevant behavior, human-judge agreement should also collapse. Third, the two factors with the most directly observable behavioral signatures (Verbosity and Guardedness) show the largest convergent point estimates, consistent with a gradient of observability rather than a gradient of prompt adequacy. That said, we acknowledge that 4 prompts per factor is insufficient to claim comprehensive construct coverage, and we cannot rule out that a larger, more carefully designed prompt battery would recover stronger convergence. We will add this as an explicit limitation and scope all claims to 'the 20 behavioral prompts tested.' revision: partial

  3. Referee: The abstract states 'self-report predicted neither [humans nor judges]' but Table 7 shows the mean Instrument-Judge correlation is r = .17 with CI [.06, .29], which excludes zero. The Responsiveness factor alone (r = .53, CI [.30, .72]) drives this. The abstract's blanket claim is contradicted by the aggregate CI.

    Authors: The referee is correct. The aggregate Instrument-Judge correlation (r = .17, CI [.06, .29]) does exclude zero, driven primarily by the Responsiveness factor (r = .53). The abstract's statement that 'self-report predicted neither' is inaccurate as applied to judge ratings. We will revise the abstract to state that self-report did not predict human ratings (no factor-level CI excluding zero) but showed partial convergence with LLM-judge ratings, concentrated on Responsiveness. This revision is also consistent with the paper's own framing of the Responsiveness dissociation as 'the exception that is telling'—the abstract should reflect that exception rather than contradicting it. We will ensure the abstract, Section 4.6.3, and the conclusion are all reconciled with Table 7. revision: yes

Circularity Check

0 steps flagged

No significant circularity: central null result is structurally non-circular; behavioral prompts designed post-factor-extraction bias against, not toward, the null finding

full rationale

The paper's central claim — that LLM self-report does not predict behavior — is a null result with independently derived inputs (self-report Likert scores) and outputs (human and LLM-judge ratings of open-ended behavioral samples). There is no self-definitional reduction. The one design choice that could introduce circularity is that behavioral prompts (§3.2.3) and the judge rating instrument were designed using the same five factor definitions extracted from the self-report data. However, this design biases toward finding self-report–behavior convergence, not against it: if the behavioral prompts and rating criteria share construct definitions with the self-report items, we would expect elevated correlations, yet the paper finds near-zero convergence. The on-target analysis (§4.6.3) confirms this: restricting to factor-targeted prompts yields worse convergence (r̄=−.03) than the broad aggregate (r̄=.15). The Responsiveness dissociation claim (r_SH ≥ r_SJ × r_HJ) relies on a standard single-factor correlation bound, not a self-citation, and its concern is statistical validity at N=25 (point estimates treated as population parameters), not circularity. No load-bearing self-citations were identified. Score of 1 reflects the minor shared-instrument design choice that is non-circular in effect.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The instrument introduces four design choices that function as free parameters (k=5, item thresholds, prompt count, judge composition), all justified but not parameter-free. The four domain assumptions are the load-bearing axioms: the logprob–sampling equivalence (validated for 7/25 models), human rater reliability (modest ICCs), seed-dimension coverage (literature-based but not exhaustive), and factor-analysis adequacy at N=25 (defended via robustness checks but acknowledged as unconventional). No invented entities are postulated without falsifiable handles.

free parameters (4)
  • Number of factors (k=5) = 5
    Parallel analysis suggested 19 factors; the authors overrode this and selected k=5 based on scree elbow, interpretability, and Tucker congruence (§4.2.1, Deviations from Preregistration). This is a post-hoc choice from the data, though validated on a held-out half.
  • Item retention thresholds = primary loading ≥.40, cross-loading <.30
    These thresholds determined which 100 of 240 items were retained (§3.4.1). They are conventional psychometric thresholds, not fitted to this dataset, but they shape the final instrument.
  • Behavioral prompt set (n=20) = 20 prompts (4 per factor)
    The 20 behavioral prompts were designed after factor extraction to target each factor's construct space (§3.2.3). The choice of 4 prompts per factor is not justified by power analysis and limits the predictive-validity test.
  • Judge ensemble composition = Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro
    Three judges selected as 'state-of-the-art' (§3.4.5). The choice is reasonable but not parameter-free; different judge compositions could yield different agreement patterns.
axioms (4)
  • domain assumption LLM responses to Likert items at temperature 1.0 across 30 runs approximate the model's probability-weighted expected score.
    Validated for 7 models with log-probability access (mean r=.999 between repeated-sampling and logprob scores, §4.5.3), but assumed for the remaining 18 models.
  • domain assumption Human raters on Prolific can reliably rate LLM behavioral outputs on abstract constructs like Responsiveness and Boldness.
    Item-level ICCs are modest (.18–.43, §4.6.1), below conventional thresholds for individual-decision reliability. The paper argues model-level aggregation absorbs this noise, but the axiom that human ratings carry real signal at N=25 rests on the human–judge agreement (mean r=.51).
  • domain assumption The 12 candidate dimensions used for item generation adequately sample the space of LLM behavioral variation.
    The 12 seed dimensions were identified by 'reasoning from documented LLM behaviors in the alignment, safety, and evaluation literatures' (§3.2.1). If important behavioral dimensions were omitted, the factor structure may be incomplete.
  • domain assumption Factor analysis at model-level N=25 with 240 items yields a trustworthy covariance structure.
    The paper acknowledges this is 'unconventional by the subject-to-item ratios developed for human psychometrics' (§3.3.3) and relies on four complementary checks. The model-level robustness check (Appendix K) is the strongest defense, but N=25 remains small for factor generalization to new models.
invented entities (2)
  • Five LLM-native self-report factors (Responsiveness, Deference, Boldness, Guardedness, Verbosity) independent evidence
    purpose: To organize LLM self-description along behaviorally grounded dimensions
    The factors are derived from EFA on a purpose-built item pool and validated via split-half replication (Tucker ϕ≥.957), model-level robustness (ϕ≥.990), and cross-run stability (r≥.965). They make a falsifiable prediction: that the same structure should emerge in an independent sample of LLMs.
  • Textual-surface bias (shared variance between LLM self-report and LLM judges) independent evidence
    purpose: To explain the Responsiveness dissociation (self-report–judge r=.53, self-report–human r=.04)
    The bias is inferred from a mathematical inconsistency with a single-factor model (r_IJ × r_HJ ≈ .31 as lower bound, observed r_IH = .04). It makes a falsifiable prediction: that controlling for textual-surface features (formatting, enthusiasm) should reduce the self-report–judge correlation. The paper does not test this directly but proposes it as future work (§5.7).

pith-pipeline@v1.1.0-glm · 39311 in / 4115 out tokens · 256699 ms · 2026-07-04T19:31:10.922162+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. BlossomPsy: A User-Centric AI System for Adaptive and Engaging MBTI Personality Assessments

    cs.HC 2026-07 conditional novelty 4.0

    BlossomPsy combines multi-turn LLM dialogue, photo-based questions, a multi-head classifier, and a modified UCB bandit algorithm to deliver MBTI assessments with higher user engagement and preliminary consistency with...