{"id":"a8d7c65a-8d86-4866-97ef-9a8d8bc97ea3","arxiv_id":"2508.20385","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Conversational history changes LLM personality-test answers: it increases answer consistency through in-context learning but shifts OCEAN scores, especially for GPT-3.5/4, while smaller models rely heavily on prior interactions.","lead":"An empirical study proposes CAPE, a framework that evaluates LLM personality from questions answered within a retained conversational history rather than in isolation. It reports that history makes model answers more consistent but also shifts their measured OCEAN scores, with implications for role-playing agents and survey-proxy use.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Adversarial history test (§6.3) cannot distinguish intrinsic personality from option-label prior; GPT models' partial shift to (c) may reflect label bias, not stable traits.","rationale":"The central claim of the paper is that conversational history improves consistency and that context-aware evaluation reveals intrinsic versus context-driven personality. The strongest evidence for the intrinsic part is the §6.3 adversarial attack. That evidence is confounded by a strong option-label prior: at baseline, all models almost never choose (c), so the observed 30–35 selections of (c) after the attack are modest relative to what a purely history-driven model would do, but the paper provides no control for label identity or per-item stability. The reader's weakest_assumption identified exactly this inferential gap. My independent reading confirms it is the single most load-bearing concern: if the intrinsic-personality subclaim is unsupported, the abstract's distinction between GPT models and Gemini/Llama is not established, though the broader framework (context improves consistency, and context shifts align with human judgments in RPAs) could still hold. The paper deserves credit for proposing a novel framework, releasing code/datasets, and performing statistical validation of the TC/OC metrics; those contributions are not undermined by this concern. A conditional verdict remains appropriate, requiring the additional control experiments or a softened claim. I recommend no change to the reader's CONDITIONAL verdict.","tokens_in":18934,"tokens_out":6601,"duration_ms":76146,"concrete_test":"Re-run the §6.3 adversarial attack with four false-history conditions: (i) all 'A', (ii) all 'E', (iii) all 'c' (original replication), and (iv) a content-coherent opposing persona (e.g., all answers opposite to the model's baseline trait profile). For each condition, compute per-item agreement between pre- and post-attack responses (not just option-(c) frequency). If GPT-3.5-Turbo/GPT-4-Turbo show substantial label-tracking under (i)/(ii) (e.g., ≥50% of items shift toward the appended label), the 'intrinsic personality' inference from §6.3 fails. If they largely preserve original answers across all conditions, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's inference that GPT models possess 'intrinsic personality' rests entirely on the §6.3 adversarial manipulation: after falsely appending option (c) as every prior answer, GPT-3.5-Turbo and GPT-4-Turbo select (c) only 30–35 times (not exclusively), while Gemini-1.5-Flash and Llama-3.1-8B switch completely. The authors conclude that GPT models 'continue to rely on their intrinsic personality.' This assumes resistance to the false history is caused by a stable internal trait. However, the manipulation confounds history content with option-label frequency. All models at temperature 0 select (c) only 2–3 times out of 120 at baseline, indicating a strong pre-existing prior against the neutral option. The post-attack increase to 30–35 for GPT models could simply reflect partial erosion of that label prior—a superficial response bias—rather than preservation of personality. No control condition appends a different option (e.g., all 'A' or all 'E'), and the paper never checks whether post-attack answers actually match the model's original (pre-attack) responses item-by-item. If GPT models also track an appended 'A' or 'E' pattern, then their original 'resistance' is specific to option (c), and the intrinsic/context-driven distinction loses its foundation. This distinction is a headline claim in the abstract and conclusion, making it load-bearing for the paper's central narrative.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CAPE, a context-aware personality evaluation framework in which multiple-choice psychometric items are administered within a single conversation so that prior question-answer pairs remain in the model's context. The authors introduce two consistency metrics, Trajectory Consistency (TC) and OCEAN Consistency (OC), based on the overlap of Gaussian Process posterior predictive support intervals across three runs. They evaluate 7 LLMs on the MPI dataset under five sensitivity factors, report that conversational history generally improves consistency, identify personality shifts (especially for GPT-3.5-Turbo and GPT-4-Turbo), conduct an adversarial history manipulation to argue that some models have intrinsic personality while others depend on context, test question-order robustness, and apply the framework to role-playing agents. The paper releases code and datasets.","tokens_in":19342,"tokens_out":5691,"duration_ms":64391,"significance":"If the central claims hold, CAPE would be a useful methodological contribution to LLM psychometrics: it operationalizes context-aware evaluation, provides a quantitative consistency measure, and demonstrates a practical application to role-playing agents. The paper is transparent in reporting per-model/per-factor results and includes an appendix documenting a negative result for DeepSeek-R1. The public release of code and datasets is a concrete strength. However, the load-bearing claims currently outrun the evidence: the aggregate improvement claim has many exceptions, the intrinsic-personality inference is confounded, and the RPA gains are not statistically supported.","major_comments":[{"comment":"The central claim that 'context-dependent evaluation improves consistency' is not supported uniformly by the table. For example, Llama-3.1-8B Temperature worsens in context-dependent setting (TAR 40.00→24.17, ED 0.76→1.00, TC 30.13→25.75, OC 76.73→67.59); GPT-4-Turbo Instructions worsens (TAR 52.50→32.50, ED 0.63→0.97); Claude-3.5-Haiku Temperature worsens on all metrics; GPT-3.5-Turbo Item Paraphrasing is worse on TAR, ED, and TC. The post hoc explanations about model size and instruction tuning are plausible but are not tested. With only three runs per condition and no error bars or per-cell significance tests, the aggregate statement in §5 and the abstract overstates the result. Please restrict the claim to the conditions that support it or provide bootstrap confidence intervals and paired tests for each row.","section":"§5, Table 1"},{"comment":"The inference that GPT models 'continue to rely on their intrinsic personality' rests on the adversarial attack that appends option (c) to all previous answers. The paper notes that all models select (c) only 2–3 times out of 120 at baseline, so the partial shift to 30–35 occurrences in GPT models could simply reflect partial erosion of a strong option-label prior rather than preservation of a stable internal trait. No control condition appends a different option (e.g., all A or all E), and no item-by-item comparison checks whether post-attack answers match the model's pre-attack answers. Without these controls, the intrinsic vs. context-driven distinction, which appears in the abstract and conclusion, is not established. Please add control conditions and report the overlap between pre- and post-attack choices per item.","section":"§6.3, Figure 4"},{"comment":"The claimed improvement of RPA++ over RPA is small and no significance testing is reported: OA increases from 67.92 to 68.69 for GPT-3.5-Turbo and from 68.62 to 68.93 for GPT-4-Turbo; MAE decreases by 0.49 and 0.25, respectively. Across 32 characters, such differences may be within noise. Please provide paired significance tests (e.g., Wilcoxon signed-rank) and confidence intervals over the 32 characters, and clarify how the consistency metrics are aggregated across characters and runs. Without this, the statement 'aligns better with human judgments' is not quantitatively supported.","section":"§6.5, Table 2"},{"comment":"The paper's main analysis excludes DeepSeek-R1 after reporting in Appendix E that both the 8B and 671B variants consistently perform worse in the context-dependent setting across all metrics (e.g., Deepseek-R1 671B TAR 64.17→14.17, TC 22.14→21.78, OC 93.58→70.19). This exclusion is post hoc and directly contradicts the central claim. Since the appendix transparently shows the result, the paper should either include DeepSeek in the main table or explicitly frame the main claim as applying to the selected instruction-tuned/chat models, with DeepSeek-R1 as a documented boundary case. Currently the 'exhaustive experiments on 7 LLMs' formulation is misleading because at least nine model variants were tested.","section":"Appendix E"},{"comment":"The proposed TC and OC metrics depend on the moving-average window ω=4, which is set by 'hyper-parameter tuning' without any sensitivity analysis, and on GPR kernel hyperparameters that are auto-tuned but not reported. More substantively, OCEAN Consistency concatenates all 120 permutations of the five trait scores into one artificial series; the interpretation of posterior support overlap over this constructed series is not justified. Because these metrics are used as the main evidence for the consistency improvement, please provide an ablation over ω, report kernel hyperparameters, and give a formal or intuitive justification for why OC measures OCEAN-level consistency rather than an artifact of permutation concatenation.","section":"§3.2, Eqs. (1)–(3)"}],"minor_comments":[{"comment":"The text says 'GPT-3.5-Turbo and GPT-3.5-Turbo undergo the extreme shifts'; the second model is presumably GPT-4-Turbo. Also, Figure 3a ranks Gemini-1.5-Flash and Llama-3.1-8B as having the widest score-difference distributions, while Figure 3b and the text emphasize GPT models as having extreme OCEAN shifts; please reconcile these descriptions.","section":"§6.2"},{"comment":"Model names are inconsistent: 'Llama' vs. 'LLaMA', 'Deepseek' vs. 'DeepSeek', 'GPT-3-5' vs. 'GPT-3.5'. The abstract has a grammatical error: 'GPT models response stem from...'. Please copyedit.","section":"Throughout"},{"comment":"The p-values in the text ('p < 10−9') do not match the exact p-values in Table 3 (e.g., 1.43×10−10 for TC vs. TAR Pearson). Please align the reporting.","section":"§5, Statistical Analysis"},{"comment":"The y-axis is labeled 'Frequency' but the figure caption says 'smoothed area plots'; also 'Smoothened' in Figure 2 should be 'Smoothed'.","section":"Figure 4"},{"comment":"The appendix states 'We plan to release our paraphrased item versions after acceptance,' which conflicts with the abstract's claim that datasets are already publicly available. Please clarify the release status of the paraphrased items.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and practically important question, and the open release of code and data is a plus. The main concerns are the post hoc exclusion of DeepSeek-R1, the absence of per-condition statistical tests for the headline Table 1, and the confounded adversarial test in §6.3. These are fixable with additional experiments and more precise claims, so I do not recommend rejection, but the current version overstates several conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it is a genuine attempt to fix a real gap: personality evaluations of LLMs have mostly been done in isolation, which is artificial. CAPE retains conversational history and introduces two new consistency metrics (TC and OC) that capture response-pattern similarity, not just pointwise agreement. Second, the paper’s headline claim about GPT models having ‘intrinsic personality’ is not actually supported by the experiment behind it, because the adversarial manipulation only appends option (c) to the history and never checks whether models would track a different appended option. That is a confound between conversational content and option-label frequency.\n\nWhat the paper does well: the problem is well motivated, the experimental design is broad (7 LLMs, 5 inconsistency factors, multiple orderings), and the new metrics are validated with correlations against TAR/ED, reliability statistics, and construct validity. The ablation in §6.1 showing that prior question-answer pairs act as few-shot demonstrations is clean and convincing, and Figure 2 is a nice visual of the mechanism. The RPA application in §6.5, though small, is a sensible way to show practical value: context-dependent evaluation improves consistency and aligns a bit better with human ratings.\n\nWhere I would push back: the §6.3 inference is the softest point. As the stress-test note says, the baseline prior against option (c) is very strong (2–3 out of 120), so a shift to 30–35 could be partial erosion of a label prior, not a stable internal trait. Without control conditions appending a different label (e.g., all ‘A’) or comparing item-by-item with pre-attack answers, the categories ‘intrinsic’ vs. ‘context-driven’ are unearned. This claim is repeated in the abstract and conclusion, so it is load-bearing. Also, the exclusion of DeepSeek-R1 from the main tables is post hoc (Appendix E shows it performs worse with context), and the RPA gains lack significance tests or error bars. Several consistency improvements in Table 1 have exceptions (Llama-8B temperature, GPT-4 instruction), and those are explained post hoc rather than as part of a theory. Minor: the paraphrased items are said to be released only after acceptance, so full reproducibility is not immediate.\n\nWho should read this? Anyone working on LLM personality assessment, role-playing agents, or using psychometric instruments on LLMs. It deserves a serious referee and a revision, not a desk reject. The core idea is sound; the intrinsic-personality claim needs a cleaner experiment. With a control condition and significance testing, this could be a solid contribution.","headline":"The CAPE framework is a real contribution—context-aware personality evaluation and the new consistency metrics are worth engaging with—but the 'intrinsic personality' inference in §6.3 rests on a confound that needs control conditions.","tokens_in":19750,"tokens_out":2015,"would_cite":true,"duration_ms":25880,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that conversational history acts as few-shot in-context learning, making LLM personality responses more consistent while shifting their OCEAN profiles, and that context-dependent evaluation aligns better with human judgment","keywords":["LLM personality evaluation","context-aware evaluation","conversational history","response consistency","Big Five OCEAN","role-playing agents","in-context learning","psychometric testing"],"falsifier":"Run the adversarial-history test again with the planted option rotated among (a), (b), (d), (e) and with fake priors that contradict each model's default answers; if GPT-3.5/4-Turbo switch to whatever label is planted, the intrinsic-trait interpretation collapses, and the observed consistency is better explained by surface biases than by personality.","tokens_in":18872,"feed_emoji":"🎭","tokens_out":6886,"duration_ms":72121,"temperature":0.7,"pith_summary":"The paper sets out to fix how LLM personality is measured. Existing psychometric tests ask each item in isolation, what the authors call the Disney World test, even though real use has conversational history. CAPE keeps prior question-answer pairs in context and measures response consistency with two new metrics. Across seven models, retaining history improves consistency, acting like few-shot in-context learning, but it also shifts OCEAN profiles, with GPT models shifting sharply yet retaining an apparent intrinsic core while Gemini-1.5-Flash and Llama-3.1-8B track the history almost completely. Applied to role-playing agents, the context-aware scores improve consistency and land closer to human-annotated character personalities.","feed_headline":"Keep prior answers, get steadier LLM personalities","feed_subtitle":"Context-free tests miss history-driven shifts; the CAPE framework measures them and matches human role-play ratings.","key_machinery":"The load-bearing object is the context window: instead of asking each item with no memory, CAPE feeds the model the session history Ht−1 = {(q1,r1),…,(qt−1,rt−1)} at every step, so the response function is rt = f(qt, Ht−1). Two metrics then quantify consistency across three independent runs. Trajectory Consistency (TC) fits a Gaussian process to each smoothed, normalized scoring trajectory and computes the average ratio of the intersection to the union of posterior predictive support intervals; OCEAN Consistency (OC) applies the same logic to the five trait scores after all 120 permutations are appended to make the representation order-invariant. The adversarial history attack, appending opt","core_discovery":"The central claim is that conversational history is not noise to be stripped away but a defining condition of LLM personality assessment. Treating each question in isolation yields what the authors call the Disney World test; CAPE retains the full prior context and defines response rt = f(qt, Ht−1). With 7 LLMs and the MPI inventory, the paper finds context-dependent runs achieve higher consistency under five perturbation factors (temperature, wording, option order, instruction, paraphrase), and an ablation shows trajectories move toward the full-history setting as more prior pairs are supplied, supporting the few-shot in-context-learning interpretation. Context also changes personality: GPT","pith_inferences":["The paper's 'intrinsic personality' reading for GPT models would be stronger if the adversarial test also swapped the planted option among A/B/D/E and varied the fake answer; such a test would separate trait persistence from label-following bias.","If few-shot in-context learning is really the mechanism, then replacing the history with same-length irrelevant demonstrations (e.g., arithmetic QA pairs) should not improve personality consistency; that is a clean experiment the paper does not run.","A practical guardrail follows directly: deployed agents could monitor drift in their OCEAN trajectory over a session and flag when history-driven shift exceeds a threshold, especially in tutoring or health contexts.","The consistency metrics likely transfer to any repeated scoring trajectory, not just personality inventories, since they measure support overlap of smoothed posterior curves rather than item-specific content."],"forward_implications":["Personality scores from single-turn tests should be read as a conditional snapshot, not a stable trait, because the same model can produce different but internally consistent OCEAN profiles with and without history.","Benchmarks that report LLM personality should control for question order and prior context; otherwise a model like Gemini-1.5-Flash or Llama-3.1-8B can look inconsistent when the cause is order sensitivity.","The few-shot mechanism implies that consistency gains will grow with the amount of retained history and weaken when history is truncated, giving system designers a cheap lever: keep more context for steadier personas.","Role-playing agents should be evaluated in the context-aware mode; RPA++ improved alignment by about 0.54 OA points, cut MAE by about 0.37, and raised TC and OC by about 13.4 points on average.","Context-aware consistency does not imply logical consistency: models stay stable on semantically similar pairs but often fail to give opposite answers on logically contradictory pairs."],"supporting_citations":[{"why":"Supplies the Machine Personality Inventory and the zero-shot MCQ assessment setup that CAPE builds on.","marker":"Jiang et al., 2023"},{"why":"Provides the Total Agreement Rate baseline that the new consistency metrics are correlated against.","marker":"Atil et al., 2024"},{"why":"Large-scale evidence that the psychometric instruments show construct validity on LLMs, the premise for reusing them in context.","marker":"Serapio-García et al., 2023"},{"why":"Supports the claim that retaining demonstrations enables in-context learning, the mechanism invoked for history-driven consistency.","marker":"Min et al., 2022"},{"why":"InCharacter supplies the 32 characters with human-annotated OCEAN scores used in the role-playing-agent alignment test.","marker":"Wang et al., 2024b"},{"why":"ChatHaruhi provides the character dialogues used to build the role-playing agents.","marker":"Li et al., 2023"},{"why":"RoleLLM provides the role-playing framework and data used to build the RPA baselines.","marker":"Wang et al., 2024a"},{"why":"Establishes that human assessments are largely robust to item order, the comparison point for LLM order sensitivity.","marker":"Schell and Oswald, 2013"},{"why":"Supplies the BFI inventory used in the role-playing agent evaluation.","marker":"Lang et al., 2011"},{"why":"Provides the paraphrasing methodology and robustness evidence for psychological scales on LLMs.","marker":"Huang et al., 2024a"}],"fun_headline_variants":["History steadies LLMs but shifts their traits","Context makes LLM personalities more consistent","LLM personality depends on conversation history","Don't test LLMs in isolation—context counts"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The paper's division of models into 'intrinsic' versus 'context-driven' assumes that a model's refusal to fully adopt the planted option (c) after the adversarial history reflects a stable internal trait, rather than a shallower response bias such as label preference, instruction following, or repetition avoidance.","fun_headline_variants_meta":{"raw":{"variants":["History steadies LLMs but shifts their traits","Context makes LLM personalities more consistent","LLM personality depends on conversation history","Don't test LLMs in isolation—context counts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1199,"prompt_tokens":770,"completion_tokens":429,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":373}},"tokens_in":514,"tokens_out":429,"duration_ms":5296,"temperature":1.0,"reasoning_tokens":373,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:05:08.709974+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the adversarial-history test again with the planted option rotated among (a), (b), (d), (e) and with fake priors that contradict each model's default answers; if GPT-3.5/4-Turbo switch to whatever label is planted, the intrinsic-trait interpretation collapses, and the observed consistency is better explained by surface biases than by personality.","supporting_citations":[],"review_version":1}