Pith. sign in

REVIEW 5 major objections 6 minor 2 cited by

Personality as a Probe for LLM Evaluation: Method Trade-offs and Downstream Effects

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Personality control in LLMs comes with three trade-offs: ICL, PEFT, and steering each buy alignment at a different price.

desk verdict Useful dataset and evaluation idea, but the headline three-way trade-off is not backed by the paper's own numbers, and the alignment measure is partly circular. read the letter →

arxiv 2509.04794 v1 pith:IK56MT5L submitted 2025-09-05 cs.CL

Gunmay Handa , Zekun Wu , Adriano Koshiyama , Philip Treleaven This is my paper · ORCID
classification cs.CL
keywords BigFivepersonalitymechanisticsteeringin-contextlearningparameter-efficientfine-tuningLLMevaluationtraitalignmentbiasbenchmarkcontrastivedataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper systematically compares three ways to give a large language model a Big Five personality—in-context learning (ICL), parameter-efficient fine-tuning (PEFT), and mechanistic steering (MS)—and measures what each costs in reasoning ability, agent performance, and social bias. It claims that ICL achieves strong personality alignment with minimal capability loss, PEFT delivers the strongest alignment but degrades task performance, and steering offers lightweight runtime control with competitive effectiveness. The authors treat personality manipulation as a probe into how traits are represented in a model, distinguishing surface conditioning, parameter-level encoding, and activation-level steering. A sympathetic reader would take away a practical map for choosing a method under deployment constraints.

What carries the argument

The load-bearing mechanism is the contrastive dataset—each high-trait response paired with a low-trait response generated by GPT-4.1 Mini—plus a within-run delta (Δ) evaluation protocol that measures capability, bias, and alignment changes relative to each method's own baseline. Steering vectors are computed as the mean difference between trait-positive and trait-negative activations at post-attention layer norms (layers 5, 10, 15, 20), with a purification step that combines an openness vector with an openness-versus-conscientiousness contrast vector.

What would settle it

Take a sample of model outputs from each method and trait condition, and have independent human raters score them for Big Five traits. If the human scores do not match the classifier's alignment deltas (e.g., classifier says +0.9 agreeableness but humans detect no change), the comparative trade-off claims lose their foundation.

Watch

Extended reading notes

Core claim

The central claim is that personality manipulation in LLMs is governed by a consistent three-way trade-off, and that the same Big Five traits are accessible at three different representational levels. On two instruction-tuned models, ICL produces high trait-alignment with small within-run performance deltas on MMLU and GAIA; PEFT achieves the highest alignment scores but incurs larger negative capability deltas, especially on Gemma-2; and mechanistic steering gives moderate alignment with trait-dependent deltas while requiring no parameter updates. The paper also claims that openness is uniquely difficult to manipulate, agreeableness resists ICL, and personality encoding concentrates around intermediate transformer layers (around layer 15).

Load-bearing premise

The alignment scores from the personality classifier are accurate measures of genuine trait expression, even though the classifier was built in prior work with overlapping authors and the training and steering data come from the same dataset.

Editorial extensions

If this is right

  • Practitioners can pick a personality-control method by constraint: ICL when capability preservation matters most, PEFT when alignment strength dominates, and steering when lightweight runtime control is required.
  • The finding that personality encoding consolidates around intermediate layers suggests that future steering interventions should target those layers rather than shallow or deep layers.
  • The trait-specific results imply that openness and agreeableness will need bespoke manipulation strategies, since generic conditioning works less well for them.
  • The stability framework provides a quantitative way to rank method-trait combinations under deployment constraints, with steering plus conscientiousness scoring highest.
  • The contrastive dataset construction method could be reused to build balanced training data for other behavioral attributes beyond personality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If personality alignment really is a multi-level phenomenon, then hybrid methods—for example, ICL plus a light steering vector—should outperform any single method at fixed capability cost; the paper does not test this directly.
  • The alignment classifier's measurement of traits is the linchpin for every comparative claim, so a human-evaluation replication on a small sample of responses would be the most direct test of whether the trade-offs are real.
  • The large BBQ bias shifts reported for steering and PEFT on Gemma-2 suggest that personality interventions can inadvertently move bias in either direction, which raises a governance question the paper only touches on.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper compares three methods for manipulating Big Five personality traits in Gemma-2-2B-IT and LLaMA-3-8B-Instruct: in-context learning (ICL), parameter-efficient fine-tuning via LoRA (PEFT), and activation-space mechanistic steering (MS). It introduces a contrastive high/low-trait dataset, a within-run Δ evaluation framework over MMLU, GAIA, and BBQ, a purification scheme intended to separate openness from conscientiousness, and a three-level stability metric. The headline claim is a trade-off: ICL gives strong alignment with minimal capability loss, PEFT gives the strongest alignment but degrades task performance, and MS provides lightweight, competitive runtime control. The paper also argues that these methods serve as complementary interpretability probes into how personality is represented in LLMs.

Significance. If the claims were supported, the paper would provide a practical decision framework for personality control and an interpretability probe spanning surface conditioning, parameter encoding, and activation-level steering. The within-run Δ design is a sensible attempt to avoid cross-method baseline artifacts, and the contrastive dataset addresses a real imbalance in existing personality-manipulation resources. The paper also reports concrete benchmark deltas that are falsifiable. However, the central findings are not currently established: the reported numbers contradict the abstract's characterization of steering, the alignment measure is circular, and all comparisons rest on single runs without uncertainty. The work is therefore of potential interest but cannot be accepted in its present form.

major comments (5)
  1. [Abstract; Table 1; Appendix G (Table 2); Appendix H (Tables 3–5)] The headline claim that mechanistic steering is "competitive" is contradicted by the paper's own results. On Gemma-2, Table 1 shows MS trait alignment below both ICL and PEFT for every trait, e.g., Extraversion +0.64 vs +0.91/+0.78, Openness +0.10 vs +0.24/+0.21, and Conscientiousness +0.29 vs +0.81/+0.78. Moreover, no LLaMA-3 steering row appears in Table 1, in the alignment table in Appendix G, or in the MMLU/GAIA/BBQ delta tables in Appendix H, so the abstract's cross-model claim that "Experiments on Gemma-2-2B-IT and LLaMA-3-8B-Instruct reveal clear trade-offs" has no supporting evidence for MS. The claim that PEFT delivers alignment "at the cost of degraded task performance" is also only visible on Gemma-2: LLaMA-3 PEFT deltas are small and sometimes positive (e.g., GAIA +0.02, +0.00, +0.02, +0.04, +0.02 in Table 4). These are load-bearing contradictions between the stated conclusions and the reported data.
  2. [Section 3; Appendix F.2; Appendix E.4; Appendix G] The alignment measurement is circular. Trait alignment scores are produced by the Jain et al. (2025) personality classifier; the interventions are constructed from the Jain et al. dataset (PEFT and ICL on high-trait examples, MS on high/low contrast pairs); and steering strengths are calibrated with that same classifier (Appendix E.4). Appendix F.2 states that the dataset's reliability is itself "validated through the personality classifier Jain et al. [2025]." The supposedly independent "dedicated alignment task" is never described: Appendix G reports only a table of deltas, with no task definition, prompt list, scoring rubric, or independent labels. As a result, the alignment scores cannot currently be taken as evidence about genuine trait expression, and the comparative effectiveness claims rest on the same instrument used to construct the interventions.
  3. [Appendix A; Section 3; Table 6] All comparative results are single-run and carry no uncertainty. Appendix A explicitly concedes "single benchmark evaluation runs and partial dataset subsets," yet Section 3 and Table 6 report precise rankings and stability scores (e.g., openness stability 0.0411 vs neuroticism 0.0309) without standard errors, confidence intervals, or repeated seeds. The NeurIPS checklist states that "stability analysis reports variance across runs," but no multi-run variance estimates are presented anywhere, and Appendix L's stability formula is computed from deltas rather than from run-to-run variation. Without error bars, the paper cannot support rank claims among ICL, PEFT, and MS, or the trait-stability ordering.
  4. [Appendix L.1, Eq. (1)] The stability metric as written does not measure run-to-run stability. With a single run per condition, "normalized_variance" and "normalized_range" appear to be computed over traits or benchmarks within that run, not over repeated runs; the consistency term 1/(1+mean_abs_deltas) rewards small absolute deltas, which is better interpreted as a capability-preservation measure. The normalization factors (÷10000 for variance, ÷1000 for range) are stated in Appendix L.3 to be empirically derived and can dominate the score. The three-level stability hierarchy in Table 6 is therefore not a validated construct.
  5. [Section 4; Appendix A; Appendix C.3] The text claims run-to-run variability, e.g., "LLaMA-3 exhibits variability across runs" in Section 4 and "we avoid cross-run comparisons due to baseline variance across runs" in Appendix C.3, while Appendix A says only single benchmark evaluation runs were performed. These statements are inconsistent. If multiple runs exist, they should be reported with means and errors; if they do not, the variability claims should be removed.
minor comments (6)
  1. [Appendix G] The alignment-table labels are inconsistent with the main text: Appendix G uses G2-P, G2-S, G2-F, L3-P, and L3-F with abbreviations defined as in Table 3, where P=Prompting, F=PEFT, and S=Steering, while the body uses ICL and MS; this makes cross-referencing unnecessarily difficult.
  2. [Appendix C.3] The claim that "Independent alignment validation shows strong alignment for most traits (e.g., Gemma extraversion 1.00, neuroticism 1.00)" does not match Table 1 or Table 2, which report +0.91 and +0.97 for Gemma-2 ICL on those traits.
  3. [Appendix E.6] The text asserts "statistically significant alignment" and "statistically significant improvements," but no significance tests, p-values, or confidence intervals are reported anywhere in the paper.
  4. [Appendix B.2] The sentence "Rao et al. [2023] demonstrated ... while Rao et al. [2023] revealed ..." cites the same reference twice in a way that reads as though two different studies are intended; please disambiguate.
  5. [Appendix A; Appendix E.4] The paper acknowledges that the trait-purification confidence threshold is "somewhat arbitrary" but provides no sensitivity analysis; since the openness purification is presented as a contribution, its robustness should be demonstrated.
  6. [Reference list] The entry "Jen tse Huang" appears to contain a formatting error in the author name, and the in-text citation "tse Huang et al." should also be normalized.

Circularity Check

2 steps flagged · score 6.0 of 10

MS alignment is calibrated against the same overlapping-author classifier used as the outcome metric, and dataset reliability is asserted through that classifier, making the trait-alignment axis partially circular.

  1. fitted input called prediction [Appendix E.4, 'Vector Calibration and Refinement'; reported as MS alignment deltas in Table 1 and Appendix G]
    "We perform linear search across strength values for each target layer, evaluating trait induction effectiveness at each strength using the personality classifier Jain et al. [2025]."

    The MS alignment scores that the abstract calls 'competitive effectiveness' are produced after layer and strength are selected by maximizing the very classifier that later defines the reported alignment deltas. The same Jain et al. instrument is the only trait-measurement tool described in the paper, and Appendix G does not specify any independent task or labels. Therefore the MS 'alignment' is an optimized fit to the evaluation metric rather than an independent prediction.

  2. self citation load bearing [Appendix F.2, 'Personality Classifier'; also Section 2 Methods]
    "The dataset reliability is validated through the personality classifier Jain et al. [2025]."

    The classifier and the manipulation dataset both come from prior work co-authored by three of the present authors. PEFT adapters are trained on that dataset, MS vectors are extracted from it, and alignment is scored with the same classifier. Declaring the dataset 'reliable' through this same classifier closes the loop: the ground truth for trait induction is supplied by the article's own prior work and its own evaluation instrument, with no external psychometric validation introduced.

full rationale

The external benchmark deltas (MMLU, GAIA, BBQ) are not circular: they are measured against independent, published benchmarks and carry real content. The circularity is concentrated in the trait-alignment axis, which supports the abstract's 'strong alignment' and 'competitive effectiveness' claims. In Appendix E.4, steering strength and layer are chosen by linear search on the Jain et al. personality classifier, and the same instrument is the only trait scorer defined anywhere in the paper. Appendix F.2 further states that dataset reliability is itself validated through that classifier, while Appendices D and E show that PEFT and MS are built from the same Jain et al. dataset. Thus for MS the reported delta-alignment is an optimised fit to the evaluation metric rather than an independent estimate, and for PEFT the adapters are trained and scored within the same overlapping-author dataset/classifier family. Appendix G's 'dedicated alignment task' is described only as a table of deltas, with no task specification or external labels, so no genuinely independent validation is documented. Appendix A also concedes single runs and partial benchmark subsets, which undermines the stability hierarchy in Table 6; I treat that as an evidence-quality concern rather than circularity. Similarly, the missing LLaMA-3 MS row and the Gemma-2 MS alignment values that are below both ICL and PEFT for every trait contradict the 'competitive effectiveness' phrasing, but that is a support gap rather than a circularity. Overall, the capability trade-offs retain independent content, but the trait-alignment measurements that anchor the central comparison are partially circular, giving a score of 6.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central evaluation rests on the same-group personality classifier and dataset used to build the manipulations, plus empirically chosen steering and stability parameters. The capability benchmarks are external, which limits but does not eliminate the circularity.

free parameters (4)
  • Steering layer and strength per trait = Openness: layer 15, strength 110; Conscientiousness: 15/250; Extraversion: 15/200; Agreeableness: 10/100; Neuroticism…
    Selected by linear search to maximize trait alignment on the personality classifier, then fixed for downstream evaluation (Appendix E.4, E.6). These are fitted values, not derived.
  • Trait purification confidence threshold = not specified
    Used to filter openness examples before vector computation; Appendix A admits the threshold is 'somewhat arbitrary' and may exclude valid examples.
  • Stability metric normalization factors = variance scale 10000, range scale 1000
    Empirically derived per Appendix L.3; the resulting stability ranking (ICL 0.0366, PEFT 0.0363, steering 0.0326) is the basis for method-level guidance.
  • LoRA hyperparameters = rank 64, alpha 16, dropout 0.1, 2 epochs, batch size 2, learning rate 2e-4
    Hand-chosen training setup (Appendix D.1.1); the choice of 2 epochs is 'calibrated' to avoid overfitting, but no ablation is shown.
assumptions (4)
  • domain assumption Big Five model is a valid framework for describing LLM behavior
    The paper adopts OCEAN traits as the measurement framework without validating that the traits are meaningful in model outputs (Section 2, F.1).
  • domain assumption The Jain et al. (2025) personality classifier accurately measures Big Five traits in LLM outputs
    All alignment scores use this classifier; its accuracy and validation are cited to prior work with overlapping authors, not assessed here (F.2, G).
  • domain assumption GPT-4.1 Mini generated low-trait responses are valid contrastive examples for steering vectors
    The contrastive dataset is built by an external model; Appendix A flags bias and quality concerns.
  • domain assumption Within-run delta analysis across a single run is sufficient for fair cross-method comparison
    The paper uses relative deltas to avoid baseline differences but runs each benchmark once (Appendix A), so the deltas are not statistically grounded.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Personality as a Probe for LLM Evaluation: Method Trade-offs and Downstream Effects." pith.science (2026). https://pith.science/paper/IK56MT5L

@misc{pith2026250904794,
  author       = {Pith},
  title        = {Pith review of: Personality as a Probe for LLM Evaluation: Method Trade-offs and Downstream Effects},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IK56MT5L}},
  note         = {Machine review of arXiv:2509.04794}
}
abstract

Personality manipulation in large language models (LLMs) is increasingly applied in customer service and agentic scenarios, yet its mechanisms and trade-offs remain unclear. We present a systematic study of personality control using the Big Five traits, comparing in-context learning (ICL), parameter-efficient fine-tuning (PEFT), and mechanistic steering (MS). Our contributions are fourfold. First, we construct a contrastive dataset with balanced high/low trait responses, enabling effective steering vector computation and fair cross-method evaluation. Second, we introduce a unified evaluation framework based on within-run $\Delta$ analysis that disentangles, reasoning capability, agent performance, and demographic bias across MMLU, GAIA, and BBQ benchmarks. Third, we develop trait purification techniques to separate openness from conscientiousness, addressing representational overlap in trait encoding. Fourth, we propose a three-level stability framework that quantifies method-, trait-, and combination-level robustness, offering practical guidance under deployment constraints. Experiments on Gemma-2-2B-IT and LLaMA-3-8B-Instruct reveal clear trade-offs: ICL achieves strong alignment with minimal capability loss, PEFT delivers the highest alignment at the cost of degraded task performance, and MS provides lightweight runtime control with competitive effectiveness. Trait-level analysis shows openness as uniquely challenging, agreeableness as most resistant to ICL, and personality encoding consolidating around intermediate layers. Taken together, these results establish personality manipulation as a multi-level probe into behavioral representation, linking surface conditioning, parameter encoding, and activation-level steering, and positioning mechanistic steering as a lightweight alternative to fine-tuning for both deployment and interpretability.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Do AI Personas Grow? Analyzing and Benchmarking Personality Evolution in LLM Agents After Life Events

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Personality-conditioned LLM agents show measurable, but weakly event-specific, Big Five shifts with compressed person-to-person variation: they reproduce the average human trajectory, not its shape.

  2. The Alignment Floor: How Persona Customization Breaks Safety in Weakly-Aligned LLMs

    cs.HC 2026-04 conditional novelty 6.0 of 10

    Sycophancy is persona-conditional: a strongly-aligned model stays within 5pp across personas while a lightly-aligned one spans 45pp, so persona safety requires per-model auditing.

Reference graph

Works this paper leans on

33 extracted references · 11 canonical work pages · cited by 2 Pith papers

  1. [1]

    Towards monosemanticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023

    2023

  2. [2]

    Persona vectors: Monitoring and controlling character traits in language models

    Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, and Jack Lindsey. Persona vectors: Monitoring and controlling character traits in language models. arXiv preprint arXiv:2507.21509, 2025

    arXiv 2025

  3. [3]

    Costa and Robert R

    Paul T. Costa and Robert R. McCrae. The NEO Personality Inventory Manual . Psychological Assessment Resources, 1992

    work page 1992

  4. [4]

    P-React: Synthesizing Topic-Adaptive Reactions of Personality Traits via Mixture of Specialized LoRA Experts

    Yuhao Dan, Jie Zhou, Qin Chen, Junfeng Tian, and Liang He. P-React : Synthesizing topic-adaptive reactions of personality traits via mixture of specialized LoRA experts. arXiv preprint arXiv:2406.12548, 2024

    work page Pith review arXiv 2024

  5. [5]

    QLoRA : Efficient finetuning of quantized LLMs

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA : Efficient finetuning of quantized LLMs . arXiv preprint arXiv:2305.14314, 2023

    arXiv 2023

  6. [6]

    Evaluating feature steering: A case study in mitigating social biases

    Esin Durmus, Alex Tamkin, Jack Clark, Jerry Wei, Jonathan Marcus, Joshua Batson, Kunal Handa, Liane Lovitt, Meg Tong, Miles McCain, et al. Evaluating feature steering: A case study in mitigating social biases. Anthropic Research, 2024

    work page 2024

  7. [7]

    Toy models of superposition

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022

    arXiv 2022

  8. [8]

    RealToxicityPrompts : Evaluating neural toxic degeneration in language models

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. RealToxicityPrompts : Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 3356--3369, 2020

    work page 2020

Show all 33 references
  1. [9]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021

    2021

  2. [10]

    Eliciting personality traits in large language models

    Airlie Hilliard, Cristian Mu \ n oz, Zekun Wu, and Adriano Soares Koshiyama. Eliciting personality traits in large language models. arXiv preprint arXiv:2402.08341, 2024

    2024 arXiv

  3. [11]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022

    2022

  4. [12]

    From text to emoji: How PEFT -driven personality manipulation unleashes the emoji potential in LLMs

    Navya Jain, Zekun Wu, Cristian Munoz, Airlie Hilliard, Xin Guan, Adriano Koshiyama, Emre Kazim, and Philip Treleaven. From text to emoji: How PEFT -driven personality manipulation unleashes the emoji potential in LLMs . arXiv preprint arXiv:2409.10245, 2025. doi:10.48550/arXiv...

    2025 doi arXiv

  5. [13]

    Personallm: Investigating the ability of large language models to express personality traits

    Hang Jiang, Xiajie Zhang, Xubo Cao, Cynthia Breazeal, Deb Roy, and Jad Kabbara. Personallm: Investigating the ability of large language models to express personality traits. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  6. [14]

    Inference-time intervention: Eliciting truthful answers from a language model

    Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. arXiv preprint arXiv:2306.03341, 2023 a

    2023 arXiv

  7. [15]

    Tailoring personality traits in large language models via unsupervisedly-built personalized lexicons

    Tianlong Li, Xiaoqing Zheng, and Xuanjing Huang. Tailoring personality traits in large language models via unsupervisedly-built personalized lexicons. arXiv preprint arXiv:2310.16582, 2023 b

    2023 arXiv

  8. [16]

    Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. In ACM Computing Surveys, 2023

    2023

  9. [17]

    François Mairesse and Marilyn A. Walker. PERSONAGE : Personality generation for dialogue. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, pages 496--503, 2007

    2007

  10. [18]

    Editing personality for LLMs

    Shengyu Mao, Ningyu Zhang, Xiaohan Wang, Mengru Wang, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Editing personality for LLMs . arXiv preprint arXiv:2310.02168, 2023

    2023 arXiv

  11. [19]

    GAIA : A benchmark for general AI assistants

    Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA : A benchmark for general AI assistants. arXiv preprint arXiv:2311.12983, 2023

    2023 arXiv

  12. [20]

    Zoom in: An introduction to circuits

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5 0 (3): 0 e00024--001, 2020

    2020

  13. [21]

    Steering Llama 2 via contrastive activation addition

    Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering Llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681, 2024

    2024 arXiv

  14. [22]

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. BBQ : A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022 , pages 2086--2105, 2022

    2022

  15. [23]

    A practical review of mechanistic interpretability for transformer-based language models

    Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. A practical review of mechanistic interpretability for transformer-based language models. arXiv preprint arXiv:2407.02646, 2024

    2024

  16. [24]

    Can ChatGPT assess human personalities? a general evaluation framework

    Haocong Rao, Cyril Leung, and Chunyan Miao. Can ChatGPT assess human personalities? a general evaluation framework. arXiv preprint arXiv:2303.01248, 2023

    2023 arXiv

  17. [25]

    Personality traits in large language models

    Mustafa Safdari, Greg Serapio-García, Clément Crepy, Stephen Fitz, Peter Romero, Luning Sun, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. Personality traits in large language models. arXiv preprint arXiv:2307.00184, 2023

    2023 arXiv

  18. [26]

    On the reliability of psychological scales on large language models

    Jen tse Huang, Wenxiang Jiao, Man Ho Lam, Eric John Li, Vihan Gupta, Samyak Gupta, and G K Anumanchipalli. On the reliability of psychological scales on large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, page 354, ...

    2024

  19. [27]

    Steering language models with activation engineering

    Alex Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248, 2023

    2023 arXiv

  20. [28]

    Exploring the impact of personality traits on LLM bias and toxicity

    Shuo Wang, Renhao Li, Xi Chen, Derek F Wong, Yulin Yuan, and Min Yang. Exploring the impact of personality traits on LLM bias and toxicity. arXiv preprint arXiv:2502.12566, 2025

    2025

  21. [29]

    InCharacter : Evaluating personality fidelity in role-playing agents through psychological interviews

    Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, et al. InCharacter : Evaluating personality fidelity in role-playing agents through psychological interviews. arXiv preprint arXiv:2310.17976, 2023

    arXiv

  22. [30]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022

    2022

  23. [31]

    Self-assessment, exhibition, and recognition: A review of personality in large language models

    Zhiyuan Wen, Yu Yang, Jiannong Cao, Haoming Sun, Ruosong Yang, and Shuaiqi Liu. Self-assessment, exhibition, and recognition: A review of personality in large language models. arXiv preprint arXiv:2406.17624, 2024

    2024 arXiv

  24. [32]

    The better angels of machine personality: How personality relates to LLM safety

    Jie Zhang, Dongrui Liu, Chen Qian, Ziyue Gan, Yong Liu, Yu Qiao, and Jing Shao. The better angels of machine personality: How personality relates to LLM safety. arXiv preprint arXiv:2407.12344, 2024

    2024 arXiv

  25. [33]

    Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico...

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.