Pith. sign in

REVIEW 3 major objections 4 minor 28 references

Quantifying Data Contamination in Psychometric Evaluations of LLMs

T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large language models have absorbed the items and scoring rules of popular personality and values questionnaires, and can deliberately answer them to hit chosen target scores, making LLM psychometric results contaminated measurements.

desk verdict Useful framework, but the headline contamination claim is mostly built into the prompts: the scoring and target-matching tasks reveal the rule, so near-perfect MAE doesn't establish memorized protocols. read the letter →

arxiv 2510.07175 v2 pith:AAY46KIG submitted 2025-10-08 cs.CL cs.LG

Jongwook Han , Woojung Song , Jonggeun Lee , Yohan Jo This is my paper
classification cs.CLcs.LG
keywords datacontaminationpsychometricevaluationLLMBigFiveInventoryPortraitValuesQuestionnaireitemmemorizationtargetscorematchingpersonalitymeasurement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

According to this paper, large language models have internalized not only the wording of popular psychometric questionnaires but also which trait each item measures and how each response option is scored, including reverse-coded items. The paper proposes a three-part contamination framework, probes 21 models across four inventories, and reports that item-dimension mapping is near-saturated while the newest models can achieve target scores with mean absolute error near 0.1–0.2. The authors argue this is systematic evidence of data contamination, strongest for heavily used inventories such as the Big Five Inventory and the Portrait Values Questionnaire. If correct, scores from such inventories cannot be read as neutral measurements of model personality or values, which is why the paper calls for contamination-aware evaluation practices.

What carries the argument

The key machinery is a five-probe contamination battery: normalized edit distance for verbatim recall, masked-keyword recovery, item-to-dimension mapping F1, option-to-score mapping MAE, and target-score matching MAE. Each probe isolates a different layer of what a model could have memorized, from exact wording to construct associations to reverse-coding rules to strategic response selection. The battery works by supplying the inventory name, item text, dimension labels, and response options in the prompt; high performance on the later probes is the paper's evidence that the model has internalized the inventory's evaluation protocol.

What would settle it

Administer the same option-score and target-score prompts using brand-new items written in the same style and scale, including one reverse-coded item, that could not have appeared in training data. If models still produce near-zero MAE, the observed behavior is in-context compliance rather than contamination specific to these inventories.

Watch

Extended reading notes

Core claim

The paper's central claim is that contamination in psychometric LLM evaluation is layered: item memorization, evaluation memorization, and target score matching. Across BFI-44, PVQ-40, MFQ, and SD-3, models identify the correct dimension for items at near-ceiling levels, many map response options to scores with near-zero error, and the strongest models select options that produce a requested target score with MAE around 0.1–0.2. The pattern is stronger for BFI-44 and PVQ-40 and grows with model scale. A sympathetic author would summarize: the models are not merely being measured by these inventories; they are demonstrating prior mastery of the inventories themselves.

Load-bearing premise

The load-bearing premise is that near-perfect performance on the scoring and target-matching tasks reflects memorized knowledge of the inventories, rather than in-context compliance with scoring instructions that the prompts themselves already provide; the paper reports no control using novel, matched-format items.

Editorial extensions

If this is right

  • The paper's conclusion entails that published LLM personality profiles measured with BFI-44 or PVQ-40 reflect training-data exposure rather than intrinsic model traits.
  • Item-dimension mapping is near-saturated across models, so this aspect of contamination no longer differentiates models and should not be used as a behavioral signal.
  • Scaling amplifies contamination within model families, so larger future models are likely to appear more human on these instruments for spurious reasons.
  • Contamination is stronger for BFI-44 and PVQ-40 than for MFQ and SD-3, so instrument choice can change conclusions about a model's personality or values.
  • The paper calls for contamination-aware evaluation practices, such as generating or validating new items before interpreting LLM psychometric scores.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct control the paper does not run would distinguish contamination from instruction-following: repeat the option-score and target-score probes on newly written, matched-format items that cannot be in pretraining data; if MAE stays near zero, the claimed memorized scoring knowledge is not necessary to explain the results.
  • The probes double as a screening checklist: before interpreting any inventory score from an LLM, run the three-aspect battery and treat scores as unreliable if the model passes scoring and target-matching.
  • Because target-score matching is so easy for frontier models, any downstream application that relies on LLM psychometric scores, such as agent role-play or value alignment claims, can likely be steered simply by prompting for a profile.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a framework to quantify data contamination in psychometric evaluations of LLMs across three aspects: item memorization (verbatim AED and masked-keyword generation), evaluation memorization (item-dimension mapping F1 and option-score mapping MAE), and target score matching (MAE between achieved and requested scores). The framework is applied to 21 models from six families and four inventories (BFI-44, PVQ-40, MFQ, SD-3). The authors report near-ceiling item-dimension mapping, near-zero option-score MAE for frontier models, and MAE≈0.1–0.2 on target-score matching, concluding that models have internalized inventory items and scoring procedures, and can strategically adjust responses to achieve target scores.

Significance. If the framework and its interpretation were valid, the paper would provide a useful, systematic tool for a real concern in LLM psychology evaluation. Its strengths are the breadth of models/inventories, the explicit three-aspect decomposition, and the practical, inexpensive protocol. However, the two tasks that carry the paper's strongest claims—option-score mapping (§2.2) and target-score matching (§2.3)—embed the scoring rule and the item text in the prompt, so near-perfect performance is equally explained by generic instruction-following and valence reasoning as by memorized inventory-specific knowledge. The manuscript provides no control condition with novel, matched-format items, so the headline 'models understand scoring procedures and can adjust responses to achieve target scores' is not established by the reported experiments.

major comments (3)
  1. [§2.2 and §3 (Prompts in Appendix B)] Option-Score Mapping supplies the item, response options, target dimension, and the instruction 'Apply reverse-coding if the item requires it.' For a standard Likert item, the correct numeric score is determined by semantic valence (e.g., 'I am talkative' maps Agree strongly→5; 'I am reserved' maps 1 under reverse-coding). Any instruction-following model can produce MAE≈0 without having seen BFI-44 or PVQ-40. The near-zero MAE in Table 3 (BFI-44) and Table 5 (PVQ-40) is therefore confounded with in-context compliance. No control with novel items of matched format is run, so the evaluation-memorization claim is not supported.
  2. [§2.3 and Abstract] Target Score Matching instructs the model to 'Choose the option that produces the target score,' and then reports low MAE as evidence that the model can 'strategically generate responses to achieve specific target scores' (Abstract) or 'manipulate responses' (§3). By construction, the task asks the model to do exactly what the metric measures; low MAE partly restates task compliance rather than exposing memorized protocols. A control that withholds the scoring rule, or that uses novel items with the same valence structure, is needed before this can be attributed to contamination.
  3. [§2.1 Key Information Memorization] The masked-keyword task uses items like 'I see myself as someone who is ___' and asks for the masked keyword. High success (≈0.4–0.5) may reflect generic language-model prediction for highly predictable adjective completions, not necessarily memorization of the inventory. The paper provides no baseline completion accuracy on non-inventory sentences matched for length and predictability, so the 'item memorization' interpretation is not uniquely identified.
minor comments (4)
  1. [§3 / Appendix D.5] All experiments use temperature 0 with a single run, so no variance estimates are reported. Given the small differences in some MAE values across models, confidence intervals or repeated runs would help interpret the results.
  2. [§2.1] The masking of 'the most informative keyword' relies on a single 'psychology expert' annotation with no inter-annotator agreement or protocol description; this should be clarified.
  3. [Figure 1] The motivating example shows GPT-4o identifying a BFI-10 item, while the experiments are on BFI-44. This is a minor rhetorical mismatch; the caption should make the inventory explicit.
  4. [§3 'Scaling Effects'] The claim that 'larger models exhibit lower MAE and higher F1' is not backed by a statistical test or a family-wise consistency analysis; e.g., GLM-4.5-air has higher F1 than GLM-4.5 in Table 1. A formal trend test or visual scatter would be more convincing.

Circularity Check

2 steps flagged · score 6.0 of 10

Target-score matching and option-score mapping hand the scoring rule to the model, so the headline 'models can adjust responses to target scores' restates the prompt rather than evidencing contamination.

  1. self definitional [§2.3 Target Score Matching; Appendix B prompt template for target score mapping; §3 Implications]
    "We hypothesize that if LLMs have memorized the exact wording and the evaluation procedure of inventories, they would be able to strategically adjust responses to achieve desired scores. ... Your task is to select the response option that corresponds to the given target score. ... Apply reverse-coding if necessary."

    The target score is an input to the prompt and the model is instructed to choose the option that produces it; the MAE formula then reports how well the model followed that instruction. The claimed capability ('models can adjust their responses to achieve specific target scores') is therefore true by construction of the task, not an independent discovery. Because the item, options, target dimension, and reverse-coding rule are all supplied, low MAE cannot distinguish memorized scoring logic from in-context instruction-following, and no novel matched-format control is run. Interpreting low MAE as evidence of contamination is thus circular.

  2. other [§2.2 Evaluation Memorization, Option-Score Mapping; Appendix B prompt template; Tables 3 and 5]
    "Your task is to map each option to the numeric score it contributes to the target dimension. ... Apply reverse-coding if the item requires it. ... A lower MAE indicates a better understanding of the inventory's scoring protocol."

    The option-score prompt already discloses the reverse-coding convention and supplies the item text, response options, and target dimension. For standard Likert items the correct mapping follows from semantic valence (agreeing with an extraversion item scores high; a reverse-coded item is inverted), so an instruction-following model can reach near-zero MAE without ever having seen the inventory's official scoring table. The paper's near-zero MAE values are therefore consistent with prompt compliance, and using them as evidence of 'evaluation memorization' partly re-describes what the prompt asked the model to do.

full rationale

The paper's item-memorization probes (verbatim AED and key-information success) and item-dimension mapping have independent operational content, so the framework is not wholly circular and the paper is not merely self-citation. However, the two quantitative tasks that support the headline contamination claim—option-score mapping (§2.2) and target-score matching (§2.3)—give the model the item text, the target dimension, the response options, and the reverse-coding instruction. For Likert inventories such as BFI-44 and PVQ-40, the correct option-score mapping is recoverable from valence alone, and the target-score task literally asks the model to choose the option corresponding to the given score. The near-ceiling MAE values in Tables 3 and 5 thus measure instruction-following as much as (or instead of) memorized inventory scoring knowledge. No matched-format control with novel items is reported, so the strongest quantitative evidence for contamination—MAE ≈ 0.1–0.2—cannot separate the two explanations. The self-citations (Han et al. 2025; Choi et al. 2025; Lim et al. 2025) are context/reference citations and are not load-bearing for the framework. Score 6 because one central 'prediction' (target-score matching) reduces by construction to the prompt instruction, while other measures retain partial independent content.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The four inventories' published scoring keys and item-dimension assignments are taken as ground truth, and model outputs at temperature 0 are treated as deterministic reveals of memorized training content. The interpretation of the scoring-procedure tasks additionally assumes that supplying the options and reverse-coding instruction in the prompt does not by itself enable the task. Hand-chosen inputs that shift headline numbers: the per-item masked keyword, the target-score conditions (min/mean/max), and the AED normalizer.

free parameters (3)
  • Per-item masked keyword (key-information task)
    A single psychology expert selected the 'most informative keyword' for each item (§2.1). The reported 0.4–0.5 success rates depend directly on this hand choice; no inter-annotator reliability or chance baseline is reported, so the metric cannot separate inventory memorization from generic cloze ability.
  • Target score conditions (min, mean, max) per item
    Target-score matching is averaged over three hand-chosen conditions (§2.3); MAE magnitudes and the resulting contamination ranking depend on this choice, and only symmetric extremes plus the mean are tested.
  • AED normalizer L̄ (mean item length)
    Verbatim memorization divides each item's edit distance by the average item length L̄ (§2.1), so long paraphrases and refusals inflate AED; the choice of L̄ rather than output length shapes cross-model and cross-inventory comparisons.
assumptions (4)
  • domain assumption Published scoring protocols and item-dimension keys of BFI-44, PVQ-40, MFQ, and SD-3 are correct ground truth.
    Option-score mapping and target-score matching are scored against these keys (§2.2–2.3); construct validity of the instruments for LLMs is not questioned.
  • domain assumption Temperature-0 model outputs are deterministic and diagnostic of memorized knowledge rather than in-prompt instruction-following.
    All four tasks are interpreted as contamination measures on this basis (§3); the target-score task explicitly instructs the desired behavior, so the assumption is load-bearing.
  • ad hoc to paper Cloze completion of a masked item keyword indicates memorization of the inventory rather than generic language-model prediction.
    §2.1 defines key-information memorization this way without a baseline on matched unseen items.
  • standard math Standard definitions of edit distance, F1, and MAE.
    Used in §2.1–2.3 without modification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Quantifying Data Contamination in Psychometric Evaluations of LLMs." pith.science (2026). https://pith.science/paper/AAY46KIG

@misc{pith2026251007175,
  author       = {Pith},
  title        = {Pith review of: Quantifying Data Contamination in Psychometric Evaluations of LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AAY46KIG}},
  note         = {Machine review of arXiv:2510.07175}
}
read the original abstract

Recent studies apply psychometric questionnaires to Large Language Models (LLMs) to assess high-level psychological constructs such as values, personality, moral foundations, and dark traits. Although prior work has raised concerns about possible data contamination from psychometric inventories, which may threaten the reliability of such evaluations, there has been no systematic attempt to quantify the extent of this contamination. To address this gap, we propose a framework to systematically measure data contamination in psychometric evaluations of LLMs, evaluating three aspects: (1) item memorization, (2) evaluation memorization, and (3) target score matching. Applying this framework to 21 models from major families and four widely used psychometric inventories, we provide evidence that popular inventories such as the Big Five Inventory (BFI-44) and Portrait Values Questionnaire (PVQ-40) exhibit strong contamination, where models not only memorize items but can also adjust their responses to achieve specific target scores.

Figures

Figures reproduced from arXiv: 2510.07175 by the authors.

Figure 1
Figure 1. (Left) LLMs can recognize psychometric in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

28 extracted references · 2 canonical work pages

  1. [1]

    Marwa Abdulhai, Gregory Serapio-Garc \'i a, Clement Crepy, Daria Valter, John Canny, and Natasha Jaques. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.982 Moral foundations of large language models . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17737--17752, Miami, Florida, USA. Association for Compu...

    doi:10.18653/v1/2024.emnlp-main.982 2024

  2. [2]

    Simone Balloccu, Patr \'i cia Schmidtov \'a , Mateusz Lango, and Ondrej Dusek. 2024. https://aclanthology.org/2024.eacl-long.5/ Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLM s . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), ...

    2024

  3. [3]

    Pranav Bhandari, Usman Naseem, Amitava Datta, Nicolas Fay, and Mehwish Nasim. 2025. https://doi.org/10.1145/3701716.3715504 Evaluating personality traits in large language models: Insights from psychological questionnaires . In Companion Proceedings of the ACM on Web Conference 2025, WWW '25, page 868–872, New York, NY, USA. Association for Computing Machinery

    arXiv 2025

  4. [4]

    Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations

    2023

  5. [5]

    Dongmin Choi, Woojung Song, Jongwook Han, Eun-Ju Lee, and Yohan Jo. 2025. https://arxiv.org/abs/2509.10078 Established psychometric vs. ecologically valid questionnaires: Rethinking psychological assessments in large language models . Preprint, arXiv:2509.10078

    arXiv 2025

  6. [6]

    Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. 2024. https://doi.org/10.18653/v1/2024.findings-acl.716 Generalization or memorization: Data contamination and trustworthy evaluation for large language models . In Findings of the Association for Computational Linguistics: ACL 2024, pages 12039--12050, Bangkok, Thailand. Associ...

    doi:10.18653/v1/2024.findings-acl.716 2024

  7. [7]

    Yujuan Fu, Ozlem Uzuner, Meliha Yetisgen, and Fei Xia. 2025. https://doi.org/10.18653/v1/2025.findings-naacl.291 Does data contamination detection work (well) for LLM s? a survey and evaluation on detection assumptions . In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5235--5256, Albuquerque, New Mexico. Association for Com...

    doi:10.18653/v1/2025.findings-naacl.291 2025

  8. [8]

    Jesse Graham, Brian A Nosek, Jonathan Haidt, Ravi Iyer, Spassena Koleva, and Peter H Ditto. 2011. Mapping the moral domain. Journal of personality and social psychology, 101(2):366

    2011

Show all 28 references
  1. [9]

    Dorith Hadar Shoval, Kfir Asraf, Yonathan Mizrachi, Yuval Haber, and Zohar Elyoseph. 2024. https://doi.org/10.2196/55988 Assessing the alignment of large language models with human values for mental health integration: Cross-sectional study using schwartz’s theory of basic val...

    2024 doi

  2. [10]

    Thilo Hagendorff, Ishita Dasgupta, Marcel Binz, Stephanie C. Y. Chan, Andrew Lampinen, Jane X. Wang, Zeynep Akata, and Eric Schulz. 2024. https://arxiv.org/abs/2303.13988 Machine psychology . Preprint, arXiv:2303.13988

    2024 arXiv

  3. [11]

    Jongwook Han, Dongmin Choi, Woojung Song, Eun-Ju Lee, and Yohan Jo. 2025. https://doi.org/10.18653/v1/2025.acl-long.838 Value portrait: Assessing language models' values through psychometrically and ecologically valid items . In Proceedings of the 63rd Annual Meeting of the As...

    2025 doi

  4. [12]

    Jen-tse Huang, Wenxiang Jiao, Man Ho Lam, Eric John Li, Wenxuan Wang, and Michael Lyu. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.354 On the reliability of psychological scales on large language models . In Proceedings of the 2024 Conference on Empirical Methods in Natu...

    2024 doi

  5. [13]

    Hang Jiang, Xiajie Zhang, Xubo Cao, Cynthia Breazeal, Deb Roy, and Jad Kabbara. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.229 P ersona LLM : Investigating the ability of large language models to express personality traits . In Findings of the Association for Comput...

    2024 doi

  6. [14]

    Oliver P John, Eileen M Donahue, and Robert L Kentle. 1991. Big five inventory. Journal of Personality and Social Psychology

    1991

  7. [15]

    Jones and Delroy L

    Daniel N. Jones and Delroy L. Paulhus. 2013. Introducing the short dark triad (sd3) a brief measure of dark personality traits. Assessment, 21(1):28--41

    2013

  8. [16]

    Seungbeen Lee, Seungwon Lim, Seungju Han, Giyeong Oh, Hyungjoo Chae, Jiwan Chung, Minju Kim, Beong-woo Kwak, Yeonsoo Lee, Dongha Lee, Jinyoung Yeo, and Youngjae Yu. 2025. https://doi.org/10.18653/v1/2025.findings-naacl.469 Do LLM s have distinct and consistent personality? TRA...

    2025 doi

  9. [17]

    Xingxuan Li, Yutong Li, Lin Qiu, Shafiq Joty, and Lidong Bing. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.108 Evaluating psychological safety of large language models . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1826...

    2024 doi

  10. [18]

    Sungjib Lim, Woojung Song, Eun-Ju Lee, and Yohan Jo. 2025. https://arxiv.org/abs/2507.05890 Psychometric item validation using virtual respondents with trait-response mediators . Preprint, arXiv:2507.05890

    2025 arXiv

  11. [19]

    Qianli Lin, Zhipeng Hu, and Jun Ma. 2024. https://doi.org/10.3390/info15110679 The personality of the intelligent cockpit? exploring the personality traits of in-vehicle llms with psychometrics . Information, 15(11)

    2024 doi

  12. [20]

    Maril \`u Miotto, Nicola Rossberg, and Bennett Kleinberg. 2022. https://doi.org/10.18653/v1/2022.nlpcss-1.24 Who is GPT -3? an exploration of personality, values and demographics . In Proceedings of the Fifth Workshop on Natural Language Processing and Computational Social Sci...

    2022 doi

  13. [21]

    Beatrice Rammstedt and Oliver P John. 2007. Measuring personality in one minute or less: A 10-item short version of the big five inventory in english and german. Journal of research in Personality, 41(1):203--212

    2007

  14. [22]

    Oscar Sainz, Jon Campos, Iker Garc \'i a-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.722 NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark . In Findings of the ...

    2023 doi

  15. [23]

    Shalom H Schwartz. 2012. An overview of the schwartz theory of basic values. Online readings in Psychology and Culture, 2(1):11

    2012

  16. [24]

    Alejandro Tlaie. 2024. Exploring and steering the moral compass of large language models. arXiv preprint arXiv:2405.17345

    2024 arXiv

  17. [25]

    Haoran Ye, Yuhang Xie, Yuanyi Ren, Hanjun Fang, Xin Zhang, and Guojie Song. 2025. Measuring human and ai values based on generative psychometrics with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39

    2025

  18. [26]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  19. [27]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  20. [28]

    ,FM_" Y fӋ Z5Ì9fӇqzfXa: <d? W 27? xM./w8e vye۸r 1Mul ]nl:⏷ ` 'UvX ; rl R_=)Ԕ )S_SHL2uKJ ݟz)(*QS2i 2|۔1ߥLE dro(#?5

    Daniel N Jones and Delroy L Paulhus. 2014. Introducing the short dark triad (sd3) a brief measure of dark personality traits. Assessment, 21(1):28--41.figures/0000775000000000000000000000000015071507021011220 5ustar rootrootfigures/psy_contamination_figure1.pdf0000664000000000...

    2014

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.