{"id":"c642bd9b-3164-46aa-ace7-fc54232bac02","arxiv_id":"2510.07175","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Across 21 LLMs and four standard psychology questionnaires, models recognize the items, know which trait each item measures, and can choose responses to hit a specified target score.","lead":"Researchers tested whether AI chatbots had memorized four widely used psychology questionnaires — for personality, values, moral foundations, and dark traits. The study finds that models recognize the quizzes and can pick answers to land on a requested score, which casts doubt on previous AI personality and values studies.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The contamination measures in §2.2–2.3 are confounded with in-context instruction-following: prompts reveal the scoring rule, so near-ceiling MAE does not evidence memorized scoring protocols.","rationale":"The reader's weakest assumption correctly identifies the central confound: the Option-Score Mapping and Target Score Matching tasks hand the model the scoring-relevant information—item text, response options, dimension, and the reverse-coding instruction—so near-perfect performance is achievable without inventory-specific memory. This is the load-bearing point because the paper's headline and §3 implications lean most heavily on these two tasks: item memorization results are weak (verbatim AED≈1.9, key-information success ≈0.4–0.5 with no chance baseline), and item-dimension mapping is likewise confounded by semantic inference. If the target-score and option-score results are removed, the evidence for 'strong contamination' reduces to moderate semantic recall, which does not support the strong claim that models know scoring procedures and manipulate responses. The proposed novel-item control would settle the question: if MAE on novel items matches real-item MAE, the framework measures general instruction-following rather than contamination. This does not invalidate the paper as a framework proposal, but it means the headline is over-interpreted relative to the experimental design. The reader's CONDITIONAL verdict is appropriate; no change is needed.","tokens_in":12055,"tokens_out":2801,"duration_ms":26286,"concrete_test":"Construct a matched set of novel psychometric-style items: same sentence templates, response options, dimension labels, and scoring instructions as BFI-44/PVQ-40, but with content unlikely to appear in pretraining (e.g., replace trait adjectives with rare or invented words). Run Option-Score Mapping and Target Score Matching with identical prompts. If the MAE on novel items is comparable to the near-zero values reported for the real inventories, the reported results reflect instruction-following and valence reasoning, not memorized scoring protocols. If MAE degrades substantially, contamination remains plausible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that BFI-44 and PVQ-40 show contamination because models 'understand their scoring procedures' and 'can strategically generate responses to achieve specific target scores'—rests on the assumption that Option-Score Mapping (§2.2) and Target Score Matching (§2.3) measure inventory-specific memorized knowledge. The prompts in Appendix B undermine this assumption. They supply the item text, response options, target dimension, and explicitly instruct: 'Apply reverse-coding if the item requires it.' For standard Likert items, the correct score is determined by semantic valence: an extraversion item saying 'I am talkative' trivially maps 'Agree strongly' to 5; 'I am reserved' maps to 1 under reverse-coding. Any instruction-following model can reach MAE≈0 without ever having seen BFI-44 or PVQ-40. The near-zero MAE values in Tables 3 and 5 are therefore equally explained by in-context compliance as by contamination. The paper runs no control with novel, matched-format items, so the results cannot distinguish the two. Moreover, the abstract's 'can adjust their responses to achieve specific target scores' is, by construction, exactly what the target-score prompt asks the model to do. Without a control, the strongest quantitative evidence for contamination collapses.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework to quantify data contamination in psychometric evaluations of LLMs across three aspects: item memorization (verbatim AED and masked-keyword generation), evaluation memorization (item-dimension mapping F1 and option-score mapping MAE), and target score matching (MAE between achieved and requested scores). The framework is applied to 21 models from six families and four inventories (BFI-44, PVQ-40, MFQ, SD-3). The authors report near-ceiling item-dimension mapping, near-zero option-score MAE for frontier models, and MAE≈0.1–0.2 on target-score matching, concluding that models have internalized inventory items and scoring procedures, and can strategically adjust responses to achieve target scores.","tokens_in":12274,"tokens_out":2065,"duration_ms":12460,"significance":"If the framework and its interpretation were valid, the paper would provide a useful, systematic tool for a real concern in LLM psychology evaluation. Its strengths are the breadth of models/inventories, the explicit three-aspect decomposition, and the practical, inexpensive protocol. However, the two tasks that carry the paper's strongest claims—option-score mapping (§2.2) and target-score matching (§2.3)—embed the scoring rule and the item text in the prompt, so near-perfect performance is equally explained by generic instruction-following and valence reasoning as by memorized inventory-specific knowledge. The manuscript provides no control condition with novel, matched-format items, so the headline 'models understand scoring procedures and can adjust responses to achieve target scores' is not established by the reported experiments.","major_comments":[{"comment":"Option-Score Mapping supplies the item, response options, target dimension, and the instruction 'Apply reverse-coding if the item requires it.' For a standard Likert item, the correct numeric score is determined by semantic valence (e.g., 'I am talkative' maps Agree strongly→5; 'I am reserved' maps 1 under reverse-coding). Any instruction-following model can produce MAE≈0 without having seen BFI-44 or PVQ-40. The near-zero MAE in Table 3 (BFI-44) and Table 5 (PVQ-40) is therefore confounded with in-context compliance. No control with novel items of matched format is run, so the evaluation-memorization claim is not supported.","section":"§2.2 and §3 (Prompts in Appendix B)"},{"comment":"Target Score Matching instructs the model to 'Choose the option that produces the target score,' and then reports low MAE as evidence that the model can 'strategically generate responses to achieve specific target scores' (Abstract) or 'manipulate responses' (§3). By construction, the task asks the model to do exactly what the metric measures; low MAE partly restates task compliance rather than exposing memorized protocols. A control that withholds the scoring rule, or that uses novel items with the same valence structure, is needed before this can be attributed to contamination.","section":"§2.3 and Abstract"},{"comment":"The masked-keyword task uses items like 'I see myself as someone who is ___' and asks for the masked keyword. High success (≈0.4–0.5) may reflect generic language-model prediction for highly predictable adjective completions, not necessarily memorization of the inventory. The paper provides no baseline completion accuracy on non-inventory sentences matched for length and predictability, so the 'item memorization' interpretation is not uniquely identified.","section":"§2.1 Key Information Memorization"}],"minor_comments":[{"comment":"All experiments use temperature 0 with a single run, so no variance estimates are reported. Given the small differences in some MAE values across models, confidence intervals or repeated runs would help interpret the results.","section":"§3 / Appendix D.5"},{"comment":"The masking of 'the most informative keyword' relies on a single 'psychology expert' annotation with no inter-annotator agreement or protocol description; this should be clarified.","section":"§2.1"},{"comment":"The motivating example shows GPT-4o identifying a BFI-10 item, while the experiments are on BFI-44. This is a minor rhetorical mismatch; the caption should make the inventory explicit.","section":"Figure 1"},{"comment":"The claim that 'larger models exhibit lower MAE and higher F1' is not backed by a statistical test or a family-wise consistency analysis; e.g., GLM-4.5-air has higher F1 than GLM-4.5 in Table 1. A formal trend test or visual scatter would be more convincing.","section":"§3 'Scaling Effects'"}],"recommendation":"major_revision","confidential_remarks":"The central quantitative evidence for contamination is confounded with instruction-following in two of the three aspects. The paper has a clear and useful structure, but the authors should either add control conditions with novel matched-format items and rephrase their claims, or substantially weaken the interpretation of the option-score and target-score results. This is likely fixable within the manuscript's scope, hence major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful framework paper with an over-interpreted headline. It's the first systematic attempt I know of to quantify contamination in psychometric LLM evaluation across three aspects, 21 models, and four inventories. The per-inventory tables are informative, and the conclusion that PVQ-40 and BFI-44 show stronger contamination signals than MFQ and SD-3 follows from the numbers. You should know the strongest part is the framework itself; the weakest is the evidence that models understand scoring protocols.\n\nWhat it does well: the three aspects are clearly defined and operationalized; the metrics (AED, key-info success, F1, MAE) are standard and reproducible; the authors are upfront that the verbatim AED is inflated by refusal behavior; and the questionnaire-wise comparison is a reasonable first cut. This is the right kind of data for the field.\n\nNow the soft spots, and they are real. The option-score mapping task gives the model the item, the options, the dimension, and explicitly instructs \"Apply reverse-coding if the item requires it.\" Any instruction-following model can achieve near-zero MAE by semantic valence reasoning—no memory of the inventory required. Same for target-score matching: the prompt literally asks for \"the option that produces the target score,\" so MAE ≈ 0.1–0.2 partly restates the instruction. There is no control with novel, matched-format items, so the paper cannot distinguish memorized scoring knowledge from in-context compliance. The stress-test note is right. The item-memorization evidence is also weaker than the abstract implies: verbatim AED ≈ 1.9 is not memorization, and the key-information success rate (0.4–0.5) lacks a chance baseline, so we don't know how much is semantic inference. Single annotator for masked keywords and single runs at temperature 0 mean no error bars either.\n\nThese are fixable. Add a control set of novel items in the same format, compute chance baselines for key info, and the design would actually test the claim. In the current form, the headline \"models understand scoring procedures\" is not supported; what is supported is \"models follow explicit scoring instructions\" and \"models semantically understand item content.\" That is still useful, but it is not the contamination result claimed.\n\nWho should read it: anyone measuring LLM personality, values, or moral foundations with standard questionnaires, and anyone designing contamination probes. It's a good reading-group paper because the confound is instructive.\n\nRecommendation: this deserves peer review, but with a request for major revision. The framework is worth engaging with; the claims need to be scaled back or the control experiments added.","headline":"Useful framework, but the headline contamination claim is mostly built into the prompts: the scoring and target-matching tasks reveal the rule, so near-perfect MAE doesn't establish memorized protocols.","tokens_in":12905,"tokens_out":3416,"would_cite":true,"duration_ms":30696,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large language models have absorbed the items and scoring rules of popular personality and values questionnaires, and can deliberately answer them to hit chosen target scores, making LLM psychometric results contaminated measurements.","keywords":["data contamination","psychometric evaluation","LLM evaluation","Big Five Inventory","Portrait Values Questionnaire","item memorization","target score matching","personality measurement"],"falsifier":"Administer the same option-score and target-score prompts using brand-new items written in the same style and scale, including one reverse-coded item, that could not have appeared in training data. If models still produce near-zero MAE, the observed behavior is in-context compliance rather than contamination specific to these inventories.","tokens_in":11819,"feed_emoji":"🧠","tokens_out":7211,"duration_ms":61241,"temperature":0.7,"pith_summary":"According to this paper, large language models have internalized not only the wording of popular psychometric questionnaires but also which trait each item measures and how each response option is scored, including reverse-coded items. The paper proposes a three-part contamination framework, probes 21 models across four inventories, and reports that item-dimension mapping is near-saturated while the newest models can achieve target scores with mean absolute error near 0.1–0.2. The authors argue this is systematic evidence of data contamination, strongest for heavily used inventories such as the Big Five Inventory and the Portrait Values Questionnaire. If correct, scores from such inventories cannot be read as neutral measurements of model personality or values, which is why the paper calls for contamination-aware evaluation practices.","feed_headline":"LLMs can steer personality-test answers to target scores","feed_subtitle":"New probes of 21 models show BFI and PVQ scores may reflect memorized inventories, not model traits.","key_machinery":"The key machinery is a five-probe contamination battery: normalized edit distance for verbatim recall, masked-keyword recovery, item-to-dimension mapping F1, option-to-score mapping MAE, and target-score matching MAE. Each probe isolates a different layer of what a model could have memorized, from exact wording to construct associations to reverse-coding rules to strategic response selection. The battery works by supplying the inventory name, item text, dimension labels, and response options in the prompt; high performance on the later probes is the paper's evidence that the model has internalized the inventory's evaluation protocol.","core_discovery":"The paper's central claim is that contamination in psychometric LLM evaluation is layered: item memorization, evaluation memorization, and target score matching. Across BFI-44, PVQ-40, MFQ, and SD-3, models identify the correct dimension for items at near-ceiling levels, many map response options to scores with near-zero error, and the strongest models select options that produce a requested target score with MAE around 0.1–0.2. The pattern is stronger for BFI-44 and PVQ-40 and grows with model scale. A sympathetic author would summarize: the models are not merely being measured by these inventories; they are demonstrating prior mastery of the inventories themselves.","pith_inferences":["A direct control the paper does not run would distinguish contamination from instruction-following: repeat the option-score and target-score probes on newly written, matched-format items that cannot be in pretraining data; if MAE stays near zero, the claimed memorized scoring knowledge is not necessary to explain the results.","The probes double as a screening checklist: before interpreting any inventory score from an LLM, run the three-aspect battery and treat scores as unreliable if the model passes scoring and target-matching.","Because target-score matching is so easy for frontier models, any downstream application that relies on LLM psychometric scores, such as agent role-play or value alignment claims, can likely be steered simply by prompting for a profile."],"forward_implications":["The paper's conclusion entails that published LLM personality profiles measured with BFI-44 or PVQ-40 reflect training-data exposure rather than intrinsic model traits.","Item-dimension mapping is near-saturated across models, so this aspect of contamination no longer differentiates models and should not be used as a behavioral signal.","Scaling amplifies contamination within model families, so larger future models are likely to appear more human on these instruments for spurious reasons.","Contamination is stronger for BFI-44 and PVQ-40 than for MFQ and SD-3, so instrument choice can change conclusions about a model's personality or values.","The paper calls for contamination-aware evaluation practices, such as generating or validating new items before interpreting LLM psychometric scores."],"fun_headline_variants":["LLMs can hit target personality scores on demand","Personality tests for LLMs: scores may be memorized, not measured","BFI and PVQ responses from LLMs show memorization","LLMs adjust psychometric answers to reach specific scores"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that near-perfect performance on the scoring and target-matching tasks reflects memorized knowledge of the inventories, rather than in-context compliance with scoring instructions that the prompts themselves already provide; the paper reports no control using novel, matched-format items.","fun_headline_variants_meta":{"raw":{"variants":["LLMs can hit target personality scores on demand","Personality tests for LLMs: scores may be memorized, not measured","BFI and PVQ responses from LLMs show memorization","LLMs adjust psychometric answers to reach specific scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000812,"raw_usage":{"total_tokens":3368,"prompt_tokens":688,"completion_tokens":2680,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":432,"completion_tokens_details":{"reasoning_tokens":2611}},"tokens_in":432,"tokens_out":2680,"duration_ms":17259,"temperature":1.0,"reasoning_tokens":2611,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T10:59:23.211005+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Administer the same option-score and target-score prompts using brand-new items written in the same style and scale, including one reverse-coded item, that could not have appeared in training data. If models still produce near-zero MAE, the observed behavior is in-context compliance rather than contamination specific to these inventories.","supporting_citations":[],"review_version":1}