Pith. sign in

REVIEW 5 major objections 5 minor 26 references

CAPE: Context-Aware Personality Evaluation Framework for Large Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that conversational history acts as few-shot in-context learning, making LLM personality responses more consistent while shifting their OCEAN profiles, and that context-dependent evaluation aligns better with human judgment

desk verdict The CAPE framework is a real contribution—context-aware personality evaluation and the new consistency metrics are worth engaging with—but the 'intrinsic personality' inference in §6.3 rests on a confound that needs control conditions. read the letter →

arxiv 2508.20385 v1 pith:OKBXSIFZ submitted 2025-08-28 cs.CL

Jivnesh Sandhan , Fei Cheng , Tushar Sandhan , Yugo Murawaki This is my paper
classification cs.CL
keywords LLMpersonalityevaluationcontext-awareconversationalhistoryresponseconsistencyBigFiveOCEANrole-playingagentsin-contextlearningpsychometrictesting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to fix how LLM personality is measured. Existing psychometric tests ask each item in isolation, what the authors call the Disney World test, even though real use has conversational history. CAPE keeps prior question-answer pairs in context and measures response consistency with two new metrics. Across seven models, retaining history improves consistency, acting like few-shot in-context learning, but it also shifts OCEAN profiles, with GPT models shifting sharply yet retaining an apparent intrinsic core while Gemini-1.5-Flash and Llama-3.1-8B track the history almost completely. Applied to role-playing agents, the context-aware scores improve consistency and land closer to human-annotated character personalities.

What carries the argument

The load-bearing object is the context window: instead of asking each item with no memory, CAPE feeds the model the session history Ht−1 = {(q1,r1),…,(qt−1,rt−1)} at every step, so the response function is rt = f(qt, Ht−1). Two metrics then quantify consistency across three independent runs. Trajectory Consistency (TC) fits a Gaussian process to each smoothed, normalized scoring trajectory and computes the average ratio of the intersection to the union of posterior predictive support intervals; OCEAN Consistency (OC) applies the same logic to the five trait scores after all 120 permutations are appended to make the representation order-invariant. The adversarial history attack, appending opt

What would settle it

Run the adversarial-history test again with the planted option rotated among (a), (b), (d), (e) and with fake priors that contradict each model's default answers; if GPT-3.5/4-Turbo switch to whatever label is planted, the intrinsic-trait interpretation collapses, and the observed consistency is better explained by surface biases than by personality.

Watch

Extended reading notes

Core claim

The central claim is that conversational history is not noise to be stripped away but a defining condition of LLM personality assessment. Treating each question in isolation yields what the authors call the Disney World test; CAPE retains the full prior context and defines response rt = f(qt, Ht−1). With 7 LLMs and the MPI inventory, the paper finds context-dependent runs achieve higher consistency under five perturbation factors (temperature, wording, option order, instruction, paraphrase), and an ablation shows trajectories move toward the full-history setting as more prior pairs are supplied, supporting the few-shot in-context-learning interpretation. Context also changes personality: GPT

Load-bearing premise

The paper's division of models into 'intrinsic' versus 'context-driven' assumes that a model's refusal to fully adopt the planted option (c) after the adversarial history reflects a stable internal trait, rather than a shallower response bias such as label preference, instruction following, or repetition avoidance.

Editorial extensions

If this is right

  • Personality scores from single-turn tests should be read as a conditional snapshot, not a stable trait, because the same model can produce different but internally consistent OCEAN profiles with and without history.
  • Benchmarks that report LLM personality should control for question order and prior context; otherwise a model like Gemini-1.5-Flash or Llama-3.1-8B can look inconsistent when the cause is order sensitivity.
  • The few-shot mechanism implies that consistency gains will grow with the amount of retained history and weaken when history is truncated, giving system designers a cheap lever: keep more context for steadier personas.
  • Role-playing agents should be evaluated in the context-aware mode; RPA++ improved alignment by about 0.54 OA points, cut MAE by about 0.37, and raised TC and OC by about 13.4 points on average.
  • Context-aware consistency does not imply logical consistency: models stay stable on semantically similar pairs but often fail to give opposite answers on logically contradictory pairs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's 'intrinsic personality' reading for GPT models would be stronger if the adversarial test also swapped the planted option among A/B/D/E and varied the fake answer; such a test would separate trait persistence from label-following bias.
  • If few-shot in-context learning is really the mechanism, then replacing the history with same-length irrelevant demonstrations (e.g., arithmetic QA pairs) should not improve personality consistency; that is a clean experiment the paper does not run.
  • A practical guardrail follows directly: deployed agents could monitor drift in their OCEAN trajectory over a session and flag when history-driven shift exceeds a threshold, especially in tutoring or health contexts.
  • The consistency metrics likely transfer to any repeated scoring trajectory, not just personality inventories, since they measure support overlap of smoothed posterior curves rather than item-specific content.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes CAPE, a context-aware personality evaluation framework in which multiple-choice psychometric items are administered within a single conversation so that prior question-answer pairs remain in the model's context. The authors introduce two consistency metrics, Trajectory Consistency (TC) and OCEAN Consistency (OC), based on the overlap of Gaussian Process posterior predictive support intervals across three runs. They evaluate 7 LLMs on the MPI dataset under five sensitivity factors, report that conversational history generally improves consistency, identify personality shifts (especially for GPT-3.5-Turbo and GPT-4-Turbo), conduct an adversarial history manipulation to argue that some models have intrinsic personality while others depend on context, test question-order robustness, and apply the framework to role-playing agents. The paper releases code and datasets.

Significance. If the central claims hold, CAPE would be a useful methodological contribution to LLM psychometrics: it operationalizes context-aware evaluation, provides a quantitative consistency measure, and demonstrates a practical application to role-playing agents. The paper is transparent in reporting per-model/per-factor results and includes an appendix documenting a negative result for DeepSeek-R1. The public release of code and datasets is a concrete strength. However, the load-bearing claims currently outrun the evidence: the aggregate improvement claim has many exceptions, the intrinsic-personality inference is confounded, and the RPA gains are not statistically supported.

major comments (5)
  1. [§5, Table 1] The central claim that 'context-dependent evaluation improves consistency' is not supported uniformly by the table. For example, Llama-3.1-8B Temperature worsens in context-dependent setting (TAR 40.00→24.17, ED 0.76→1.00, TC 30.13→25.75, OC 76.73→67.59); GPT-4-Turbo Instructions worsens (TAR 52.50→32.50, ED 0.63→0.97); Claude-3.5-Haiku Temperature worsens on all metrics; GPT-3.5-Turbo Item Paraphrasing is worse on TAR, ED, and TC. The post hoc explanations about model size and instruction tuning are plausible but are not tested. With only three runs per condition and no error bars or per-cell significance tests, the aggregate statement in §5 and the abstract overstates the result. Please restrict the claim to the conditions that support it or provide bootstrap confidence intervals and paired tests for each row.
  2. [§6.3, Figure 4] The inference that GPT models 'continue to rely on their intrinsic personality' rests on the adversarial attack that appends option (c) to all previous answers. The paper notes that all models select (c) only 2–3 times out of 120 at baseline, so the partial shift to 30–35 occurrences in GPT models could simply reflect partial erosion of a strong option-label prior rather than preservation of a stable internal trait. No control condition appends a different option (e.g., all A or all E), and no item-by-item comparison checks whether post-attack answers match the model's pre-attack answers. Without these controls, the intrinsic vs. context-driven distinction, which appears in the abstract and conclusion, is not established. Please add control conditions and report the overlap between pre- and post-attack choices per item.
  3. [§6.5, Table 2] The claimed improvement of RPA++ over RPA is small and no significance testing is reported: OA increases from 67.92 to 68.69 for GPT-3.5-Turbo and from 68.62 to 68.93 for GPT-4-Turbo; MAE decreases by 0.49 and 0.25, respectively. Across 32 characters, such differences may be within noise. Please provide paired significance tests (e.g., Wilcoxon signed-rank) and confidence intervals over the 32 characters, and clarify how the consistency metrics are aggregated across characters and runs. Without this, the statement 'aligns better with human judgments' is not quantitatively supported.
  4. [Appendix E] The paper's main analysis excludes DeepSeek-R1 after reporting in Appendix E that both the 8B and 671B variants consistently perform worse in the context-dependent setting across all metrics (e.g., Deepseek-R1 671B TAR 64.17→14.17, TC 22.14→21.78, OC 93.58→70.19). This exclusion is post hoc and directly contradicts the central claim. Since the appendix transparently shows the result, the paper should either include DeepSeek in the main table or explicitly frame the main claim as applying to the selected instruction-tuned/chat models, with DeepSeek-R1 as a documented boundary case. Currently the 'exhaustive experiments on 7 LLMs' formulation is misleading because at least nine model variants were tested.
  5. [§3.2, Eqs. (1)–(3)] The proposed TC and OC metrics depend on the moving-average window ω=4, which is set by 'hyper-parameter tuning' without any sensitivity analysis, and on GPR kernel hyperparameters that are auto-tuned but not reported. More substantively, OCEAN Consistency concatenates all 120 permutations of the five trait scores into one artificial series; the interpretation of posterior support overlap over this constructed series is not justified. Because these metrics are used as the main evidence for the consistency improvement, please provide an ablation over ω, report kernel hyperparameters, and give a formal or intuitive justification for why OC measures OCEAN-level consistency rather than an artifact of permutation concatenation.
minor comments (5)
  1. [§6.2] The text says 'GPT-3.5-Turbo and GPT-3.5-Turbo undergo the extreme shifts'; the second model is presumably GPT-4-Turbo. Also, Figure 3a ranks Gemini-1.5-Flash and Llama-3.1-8B as having the widest score-difference distributions, while Figure 3b and the text emphasize GPT models as having extreme OCEAN shifts; please reconcile these descriptions.
  2. [Throughout] Model names are inconsistent: 'Llama' vs. 'LLaMA', 'Deepseek' vs. 'DeepSeek', 'GPT-3-5' vs. 'GPT-3.5'. The abstract has a grammatical error: 'GPT models response stem from...'. Please copyedit.
  3. [§5, Statistical Analysis] The p-values in the text ('p < 10−9') do not match the exact p-values in Table 3 (e.g., 1.43×10−10 for TC vs. TAR Pearson). Please align the reporting.
  4. [Figure 4] The y-axis is labeled 'Frequency' but the figure caption says 'smoothed area plots'; also 'Smoothened' in Figure 2 should be 'Smoothed'.
  5. [Appendix D] The appendix states 'We plan to release our paraphrased item versions after acceptance,' which conflicts with the abstract's claim that datasets are already publicly available. Please clarify the release status of the paraphrased items.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CAPE is an empirical evaluation; new metrics are convergence-validated against TAR/ED, and no load-bearing claim reduces to its own inputs.

full rationale

The paper's central chain is empirical: it defines a context-dependent evaluation protocol, proposes TC/OC metrics, runs 7 LLMs, and compares to established TAR/ED metrics. The claim that conversational history improves consistency is not built into TC/OC by construction: Table 1 shows the same context-dependent improvement appears in TAR (e.g., GPT-3.5-Turbo stability 86.67→91.67, temperature 40.83→71.67) and in ED, and Appendix B reports strong correlations of TC/OC with TAR/ED (r≈0.77–0.81, p<1e-9), so the new metrics are convergent with external measures rather than self-justifying. The construct-validity check (Appendix B, Experiment C) uses the context-free vs context-dependent contrast to show metric sensitivity; this is a standard manipulation check and, crucially, TAR and ED also differentiate the same conditions (Table 5: TAR p=0.0167, ED p=0.0064), so the central finding does not rest on a metric that was validated by the finding itself. The §6.3 inference that GPT models have 'intrinsic personality' because they do not fully switch to option (c) is an interpretive leap with a plausible option-label-prior confound, but it is not a definitional or self-citational reduction: no equation equates 'intrinsic personality' with observed resistance, and no fitted parameter is renamed as a prediction. There are no load-bearing self-citations, no imported uniqueness theorem, and no ansatz smuggled via citation. The 'first framework' and 'novel metrics' are novelty claims, not derivations. Hence no circular step meeting the stated evidence bar is present.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claims rest on standard psychometric assumptions and on two paper-specific operationalizations (the TC/OC metrics and the adversarial test). No new physical or conceptual entities are introduced.

free parameters (2)
  • moving average window size ω = 4
    Chosen via hyper-parameter tuning (§3.2, footnote 2); affects smoothing of scoring trajectories and therefore TC/OC values.
  • GPR kernel hyperparameters (RBF length-scale and noise) = auto (scikit-learn optimization)
    Automatically tuned per trajectory during Gaussian process regression; affects posterior intervals used in TC/OC.
assumptions (5)
  • domain assumption IPIP/MPI and BFI items measure the same OCEAN constructs in LLMs as in humans.
    Paper relies on prior validation (Serapio-García et al. 2023; Wang et al. 2025) and does not re-validate.
  • domain assumption Likert-scale responses can be treated as numeric scores for averaging and Gaussian process regression.
    Score assignment (§2) and trajectory modeling (§3.2) assume ordinal responses are interval-scale.
  • ad hoc to paper Consistency of an LLM personality can be quantified by overlap of GPR posterior predictive support intervals across three runs.
    TC/OC definition in §3.2 is a novel operationalization proposed by the paper.
  • ad hoc to paper The adversarial manipulation of prior responses is a valid test of intrinsic vs. context-driven personality.
    §6.3 infers intrinsic personality from resistance to fake history without ruling out superficial response biases.
  • domain assumption Human-annotated OCEAN scores for 32 fictional characters are a valid gold standard for alignment.
    Used in §6.5, taken from Wang et al. 2024b; assumed correct for measuring alignment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CAPE: Context-Aware Personality Evaluation Framework for Large Language Models." pith.science (2026). https://pith.science/paper/OKBXSIFZ

@misc{pith2026250820385,
  author       = {Pith},
  title        = {Pith review of: CAPE: Context-Aware Personality Evaluation Framework for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OKBXSIFZ}},
  note         = {Machine review of arXiv:2508.20385}
}
read the original abstract

Psychometric tests, traditionally used to assess humans, are now being applied to Large Language Models (LLMs) to evaluate their behavioral traits. However, existing studies follow a context-free approach, answering each question in isolation to avoid contextual influence. We term this the Disney World test, an artificial setting that ignores real-world applications, where conversational history shapes responses. To bridge this gap, we propose the first Context-Aware Personality Evaluation (CAPE) framework for LLMs, incorporating prior conversational interactions. To thoroughly analyze the influence of context, we introduce novel metrics to quantify the consistency of LLM responses, a fundamental trait in human behavior. Our exhaustive experiments on 7 LLMs reveal that conversational history enhances response consistency via in-context learning but also induces personality shifts, with GPT-3.5-Turbo and GPT-4-Turbo exhibiting extreme deviations. While GPT models are robust to question ordering, Gemini-1.5-Flash and Llama-8B display significant sensitivity. Moreover, GPT models response stem from their intrinsic personality traits as well as prior interactions, whereas Gemini-1.5-Flash and Llama--8B heavily depend on prior interactions. Finally, applying our framework to Role Playing Agents (RPAs) shows context-dependent personality shifts improve response consistency and better align with human judgments. Our code and datasets are publicly available at: https://github.com/jivnesh/CAPE

Figures

Figures reproduced from arXiv: 2508.20385 by the authors.

Figure 1
Figure 1. Illustration of the difference between exist [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. This figure illustrates the mechanism behind [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. (a) Distribution indicates how frequently and [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Smoothed area plots showing the distribution [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 6
Figure 6. Figure 6: LLMs maintain consistency for semantically [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

26 extracted references · 18 canonical work pages

  1. [4]

    In Proceedings of the 1st Workshop on Personalization of Generative AI Systems (PERSONALIZE 2024), pages 102–111, St

    LLM agents in interaction: Measuring personality consistency and linguistic alignment in interacting populations of large language models. In Proceedings of the 1st Workshop on Personalization of Generative AI Systems (PERSONALIZE 2024), pages 102–111, St. Julians, Malta. Association for Computational Lin- guistics. Lewis R. Goldberg

    work page 2024

  2. [5]

    AI & SOCIETY, 39(5):2603–2605

    AI language models cannot replace human research participants. AI & SOCIETY, 39(5):2603–2605. Jen-tse Huang, Wenxiang Jiao, Man Ho Lam, Eric John Li, Wenxuan Wang, and Michael Lyu. 2024a. On the reliability of psychological scales on large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 615...

    work page 2024

  3. [6]

    In Findings of the Associ- ation for Computational Linguistics: NAACL 2024, pages 3605–3627, Mexico City, Mexico

    PersonaLLM: In- vestigating the ability of large language models to express personality traits. In Findings of the Associ- ation for Computational Linguistics: NAACL 2024, pages 3605–3627, Mexico City, Mexico. Association for Computational Linguistics. O. P. John and S. Srivastava. 1999.The Big Five Trait taxonomy: History, measurement, and theoretical pe...

    work page 2024

  4. [7]

    Preprint, arXiv:2308.09597

    Chatharuhi: Reviving anime character in reality via large language model. Preprint, arXiv:2308.09597. Xingxuan Li, Yutong Li, Lin Qiu, Shafiq Joty, and Li- dong Bing

    arXiv

  5. [8]

    In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1826–1843, Miami, Florida, USA

    Evaluating psychological safety of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1826–1843, Miami, Florida, USA. Association for Computational Linguistics. R R McCrae and P T Costa, Jr

    work page 2024

  6. [9]

    Preprint, arXiv:2407.21783

    The llama 3 herd of models. Preprint, arXiv:2407.21783. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer

    arXiv

  7. [11]

    Preprint, arXiv:2303.08774

    Gpt-4 technical report. Preprint, arXiv:2303.08774. Jiao Ou, Junda Lu, Che Liu, Yihong Tang, Fuzheng Zhang, Di Zhang, and Kun Gai

    arXiv

  8. [12]

    DialogBench: Evaluating LLMs as human-like dialogue systems. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6137–6170, Mexico City, Mexico. Association for Computational Lin- guistics. Haocong Rao, Cyril Leung, and Chunyan Miao

    work page 2024

Show all 26 references
  1. [13]

    In Findings of the Associa- tion for Computational Linguistics: EMNLP 2023 , pages 1184–1194, Singapore

    Can ChatGPT assess human personalities? a general evaluation framework. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2023 , pages 1184–1194, Singapore. Association for Com- putational Linguistics. Yuanyi Ren, Haoran Ye, Hanjun Fang, Xin Zhang, and Guojie Song

    2023

  2. [14]

    In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2015–2040, Bangkok, Thailand

    ValueBench: Towards compre- hensively evaluating value orientations and under- standing of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2015–2040, Bangkok, Thailand. Association...

    2015

  3. [15]

    Preprint, arXiv:2307.00184

    Personality traits in large language models. Preprint, arXiv:2307.00184. Bangzhao Shu, Lechen Zhang, Minje Choi, Lavinia Dunagan, Lajanugen Logeswaran, Moontae Lee, Dal- las Card, and David Jurgens

    arXiv

  4. [16]

    You don‘t need a personality test to know these models are unre- liable: Assessing the reliability of large language models on psychometric instruments. In Proceed- ings of the 2024 Conference of the North American Chapter of the Association for Computational Lin- guistics: Hu...

    2024

  5. [17]

    Preprint, arXiv:2305.14693

    Have large lan- guage models developed a personality?: Applicabil- ity of self-assessment tests in measuring personality in llms. Preprint, arXiv:2305.14693. Alexander Spangher, Nanyun Peng, Sebastian Gehrmann, and Mark Dredze

    arXiv

  6. [18]

    In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21814–21828, Miami, Florida, USA

    Do LLMs plan like human writers? comparing journalist coverage of press releases with LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21814–21828, Miami, Florida, USA. Association for Computational Linguistics. Daniel Stri...

    2024

  7. [19]

    Found in the middle: Permutation self-consistency improves listwise rank- ing in large language models. In Proceedings of the 2024 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), ...

    2024

  8. [20]

    arXiv preprint arXiv:2403.05530

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.05530. Yu-Min Tseng, Yu-Chao Huang, Teng-Yun Hsiao, Wei- Lin Chen, Chao-Wei Huang, Yu Meng, and Yun- Nung Chen

    arXiv

  9. [21]

    In Find- ings of the Association for Computational Linguistics: EMNLP 2024, pages 16612–16631, Miami, Florida, USA

    Two tales of persona in LLMs: A survey of role-playing and personalization. In Find- ings of the Association for Computational Linguistics: EMNLP 2024, pages 16612–16631, Miami, Florida, USA. Association for Computational Linguistics. Noah Wang, Z.y. Peng, Haoran Que, Jiaheng ...

    2024

  10. [22]

    In Findings of the Association for Computational Linguistics: ACL 2024, pages 3343–3353, Bangkok, Thailand

    Enhancing se- mantic consistency of large language models through model editing: An interpretability-oriented approach. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3343–3353, Bangkok, Thailand. Association for Computational Linguistics. Tao Ya...

    2024

  11. [23]

    In Find- ings of the Association for Computational Linguis- tics: EMNLP 2023 , pages 3305–3320, Singapore

    PsyCoT: Psychological questionnaire as powerful chain-of-thought for personality detection. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2023 , pages 3305–3320, Singapore. Association for Computational Linguistics. Enyu Zhou, Rui Zheng, Zhiheng Xi, S...

    2023

  12. [24]

    In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10262–10274, Singapore

    RealBehavior: A frame- work for faithfully characterizing foundation models’ human-like behavior mechanisms. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10262–10274, Singapore. Association for Computational Linguistics. Yue Zhou, Yada Zhu, D...

    2023

  13. [25]

    You take charge

    Paraphrase and solve: Explor- ing and exploiting the impact of surface form on mathematical reasoning in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (...

    2024

  14. [26]

    You {Item}

    In terms of con- struct validity, both TC and OC significantly dif- ferentiate between context-dependent and context- free evaluations (all p < 0.01), with TC yield- ing the strongest effect size (Cohen’s d = 0.81). These results affirm the sensitivity of our metrics to meanin...

    2023

  15. [2022]

    Association for Computational Linguistics

    Rethinking the role of demonstrations: What makes in-context learning work? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. OpenAI

    2022

  16. [2023]

    In Findings of the Association for Computa- tional Linguistics: EMNLP 2023, pages 2370–2386, Singapore

    Manip- ulating the perceived personality traits of language models. In Findings of the Association for Computa- tional Linguistics: EMNLP 2023, pages 2370–2386, Singapore. Association for Computational Linguis- tics. DeepSeek-AI

    2023

  17. [2024]

    Preprint, arXiv:2408.04667

    Llm stability: A detailed analysis with some surprises. Preprint, arXiv:2408.04667. Graham Caron and Shashank Srivastava

    arXiv

  18. [2025]

    Preprint, arXiv:2501.12948

    Deepseek-r1: Incentivizing rea- soning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948. Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.