Pith. sign in

REVIEW 4 major objections 6 minor 25 references

Exploring the Impact of Occupational Personas on Domain-Specific QA

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Giving an LLM an explicit job-title persona can slightly improve scientific QA accuracy, while personality-based personas such as 'scientific person' often degrade it.

desk verdict A useful persona taxonomy and a plausible negative result about personality personas, but the central contrast is confounded with prompt length and lexical form, and the neuron analysis is not load-bearing. read the letter →

arxiv 2505.24448 v1 pith:EGW7VLYN submitted 2025-05-30 cs.CL

Eojin Kang , Jaehyuk Yu , Juae Kim This is my paper
classification cs.CL
keywords personapromptingdomain-specificQAMMLUzero-shotquestionansweringlargelanguagemodelsprofession-basedpersonasoccupationalpersonalityneuronimportanceanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether giving an LLM a persona improves its answers to hard science multiple-choice questions. It distinguishes professional personas—explicit job titles like 'scientist' or 'chemist'—from occupational personality personas built from a six-type vocational personality model, such as 'scientific person' or 'artistic person.' Across college biology, chemistry, and physics questions, job-title personas occasionally and modestly beat the no-persona baseline, while personality personas often scored worse even when semantically relevant. The paper also reports that LLM neurons activate differently for the two persona types, suggesting the models separate professional expertise from personality traits rather than just keying on similar words. If correct, prompt engineers should prefer explicit expertise labels over personality descriptions.

What carries the argument

The paper's central contrast is between two prompt personas: Profession-Based Personas (job titles) and Occupational Personality-Based Personas (descriptors from the six-type vocational personality model). Its quantitative probe is the neuron-importance metric NeuronImp, which measures how much the feed-forward network's output changes when a single neuron is removed, aggregated per layer as PersonaImp. The thesis is carried by the pairing of near-identical lexical stimuli—'scientist' vs 'scientific person', 'artist' vs 'artistic person'—whose consistent accuracy differences are attributed to the model's internal separation of professional expertise from cognitive style.

What would settle it

Run a matched-length variant of the experiment—same word count and sentence structure for PBP and OPBP prompts—and check whether the accuracy gap and the neuron-distance pattern between 'Scientist' and 'Scientific Person' survive; if they vanish, the claimed semantic distinction is not the operative cause.

Watch

Extended reading notes

Core claim

This paper claims that persona relevance alone does not determine whether a persona helps or hurts domain-specific QA. The authors define Profession-Based Personas (PBPs), exactly the job title of a domain expert (e.g., scientist, biologist, chemist), and Occupational Personality-Based Personas (OPBPs), descriptors drawn from a six-type vocational personality taxonomy (e.g., scientific person, artistic person). In zero-shot multiple-choice QA on three science datasets from the MMLU benchmark, PBPs sometimes produced small gains over no persona—up to about three points on college biology with Llama-3-8B—while OPBPs frequently decreased accuracy, with the Artistic persona causing the largest drops. The gap between near-identical labels such as 'Scientist' versus 'Scientific Person,' and between 'Artist' versus 'Artistic Person,' is taken as evidence that LLMs differentiate the professional role from the personality trait. Supporting this, a neuron-importance analysis of the feed-forward layers shows distinct activation patterns across persona types, with the largest deviations from the no-persona baseline for the least domain-aligned personas.

Load-bearing premise

The whole comparison rests on the assumption that the accuracy gap between 'scientist' and 'scientific person' comes from the professional-versus-personality distinction, not from the different lengths and verbosity of the two prompt styles.

Editorial extensions

If this is right

  • Using a job-title persona such as 'Biologist' or 'Chemist' can yield small but real accuracy gains on college-level biology and chemistry QA, with the best cases around 2–3 percentage points over the no-persona baseline.
  • Personality-based personas like 'Artistic Person' can markedly reduce accuracy and raise refusal rates, so relevance to the topic is not enough to make a persona safe.
  • LLMs treat professional roles and personality traits as different internal states; near-identical word stems like 'Scientist' and 'Scientific Person' nonetheless produce different activations and outcomes.
  • Physics QA is less responsive to personas than biology and chemistry: in GPT-3.5 no persona beat the no-persona baseline on college physics, so domain-specific prompt design may be needed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A length-matched control would settle whether the PBP/OPBP gap is semantic or lexical: pad both conditions to identical word counts and check if the accuracy and neuron-distance gaps persist.
  • The same word-stem contrast ('scientist' vs 'scientific person') could be extended to other occupation pairs, such as 'doctor' vs 'caring person', to see if the professional/personality split generalizes.
  • The layered neuron analysis implies a practical tool: persona-condition distances could be used to screen prompt personas for downstream effect without running a full QA benchmark.
  • The observed abstention increases under mismatched personas could be either a harmful coverage loss or a useful guard against confident wrong answers; the paper does not settle which.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies whether assigning occupational personas to LLMs improves domain-specific QA. It distinguishes Profession-Based Personas (PBPs), i.e., job titles such as 'scientist', from Occupational Personality-Based Personas (OPBPs), derived from Holland's occupational personality themes, e.g., 'artistic person'. Experiments on college biology, chemistry, and physics subsets of MMLU with GPT-3.5 and Llama-3-8B (plus appendix results for GPT-4o mini and Llama-3-70B) report that PBPs can slightly improve accuracy while OPBPs often degrade it, and that the effect is mirrored in confession/abstention rates and in a neuron-impact analysis that the authors interpret as evidence that LLMs internally distinguish professional roles from personality traits.

Significance. If the empirical pattern were robust, the paper would make a practically useful contribution: it would tell prompt engineers to prefer explicit expertise labels over personality descriptors for domain QA, and it would offer evidence about how LLMs encode role versus trait information. The study addresses an underexamined distinction and the data collection is straightforward and reproducible in spirit. However, the central claims currently rest on small accuracy differences that are not supported by significance tests, the key PBP-versus-OPBP contrast is confounded with prompt length and lexical form, and the neuron analysis is under-specified and unvalidated. The significance is therefore conditional on the resolution of these load-bearing issues.

major comments (4)
  1. [Section 4.2, Table 4] The central PBP-versus-OPBP comparison is confounded with lexical form and prompt length. The flagship pair 'scientist' versus 'scientific person' differs not only in role-versus-personality framing but in word class, frequency, and concreteness, and 'scientific person' is not one of Holland's six types. The OPBP-Descriptive condition additionally embeds long personality paragraphs (Table 1) while the PBP condition inserts a short job title. No length- or frequency-matched control is run, so the observed accuracy gap cannot be attributed to the role/personality distinction. The authors should add matched controls or an ablation that holds the lexical stem constant and varies only the role-versus-trait framing.
  2. [Section 4, Tables 4, 7-10] No significance tests or confidence intervals are reported, and the text uses 'significantly' (e.g., 'a significant 23.96% drop' in Section 4.2) without a statistical test. Several claimed improvements are within one standard deviation of the baseline, for example GPT-3.5 Scientist on biology (+0.62 vs. sd 0.64) and Llama-3-8B Chemist on chemistry (+2.0 with sd around 4 from Table 4). With only nine runs per cell (three persona instructions times three seeds), the small deltas need paired tests with multiple-comparison control, or at least explicit effect sizes with uncertainty.
  3. [Table 4 vs. Table 7] The Artist row reports college chemistry accuracy of 37.78 in Table 4 but 30.56 in Table 7 for the same GPT-3.5 condition. This inconsistency directly affects the claim about the Artist persona degrading performance, and it suggests a data-processing error. The authors must correct the discrepancy and re-check all derived comparisons before the results can be relied upon.
  4. [Section 5, Eq. (1)] The neuron-impact analysis is not sufficient to support the internal-differentiation claim. Equation (1) is under-specified: the terms hffn, Mask[k], and Wdown are not defined precisely enough to reproduce the computation; raw NeuronImp values are reported without thresholding yet used to compute Euclidean distances; and no error bars or significance tests are given for the distances in Figure 3 or the layer-wise differences in Figure 4. Moreover, the input c includes the same confounded persona texts as the behavioral experiments, so the analysis cannot isolate role versus personality. The authors should provide a reproducible specification, validate the proxy against a known manipulation, and present the results as exploratory.
minor comments (6)
  1. [Section 8, Reference list] The claim in Section 8 that prompts designed for reasoning tasks do not improve knowledge-intensive tasks cites Li et al. (2024), but the corresponding reference is the CMMLU benchmark paper, which does not appear to be about prompt effects. Please verify and replace the citation.
  2. [Section 2, Table 4] Table 4 labels 'Scientific Person' as an OPBP, but Section 2 defines OPBPs via Holland's six types and 'scientific person' is not among them. The assignment rule should be stated explicitly, or a validation (e.g., human annotation of whether the label is perceived as a role or a trait) should be provided.
  3. [Section 3.2] The definitions of Confession Rate and Abstention Rate are loose: the formulas show Nlim and 'Nlim and no_ans' but the prose does not specify what counts as an acknowledgment of limitation or how the two metrics are annotated. A precise annotation protocol with positive and negative examples would help.
  4. [Section 4.1, Abstract] The text says personas 'significantly affect' or 'often degrade' performance without statistical support. The wording should be softened or paired with the requested significance tests.
  5. [Appendix A.2, Table 10] The Llama-3-70B Descriptived conditions show very large accuracy drops (e.g., Realistic Descriptive biology 46.76 vs. No Persona 71.53), yet the main text does not discuss this dramatic pattern. Please add an explanation or at least a caveat about the 70B results.
  6. [Section 3.1] The paper says performance was based on an average of 27 experiments, but each cell in Table 4 is an average of nine runs (three persona instructions times three seeds). Please clarify how the 27 figure is obtained and how the averaging hierarchy works.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's empirical comparisons and neuron analyses are measured outcomes, not fitted or self-referential derivations.

full rationale

This paper is an empirical prompt-comparison study, not a derivation. Its central claims — PBP accuracy gains, OPBP accuracy drops, and internal persona differentiation — are supported by measured model outputs and activations, not by fitting parameters to the conclusions. No parameter is fit to a subset of data and then relabeled as a prediction. Equation (1) defines a neuron-importance measure from the model's own feed-forward network, but the subsequent Euclidean distances and layer-wise sums are descriptive summaries of those measurements, not predictions entailed by the definition. The PBP/OPBP taxonomy is a labeling scheme; the assignment of 'Scientific Person' to OPBP is stipulative and the comparison of 'scientist' versus 'scientific person' is confounded with lexical concreteness and prompt length, but the accuracy advantage of 'Scientist' is not logically entailed by the label and must be observed empirically. There are no self-citations at all, and no uniqueness theorem is imported from the authors' prior work. The unsupported use of 'significantly' in Section 4.2 and the absence of length-matched controls are correctness and construct-validity concerns, not circularity: they do not make any result equivalent to its inputs by construction. No circular step is therefore identified.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No numbers were fitted to data. Model temperatures, top-p, max tokens, seeds, and quantization settings are fixed experimental hyperparameters rather than fitted parameters. The paper introduces prompt categories (PBPs, OPBPs) and metrics (CR, AR), but these are experimental constructs, not physical or conceptual entities requiring independent evidence.

assumptions (3)
  • domain assumption Holland's six occupational themes are valid categories for generating OPBP prompts.
    Section 2 defines OPBPs directly from Holland (1985); the experiment assumes this mapping captures personality traits distinct from professional expertise.
  • domain assumption The NeuronImp equation (Eq. 1) and PersonaImp layer comparison (Eq. 3) measure knowledge utilization and internal persona differentiation.
    Section 5 adopts the neuron detection method of Zhao et al. without task-specific validation; the interpretation that activations reveal knowledge retrieval is an assumption.
  • domain assumption Three MMLU college science subsets are representative of domain-specific QA for the study's conclusions.
    Section 3.1 selects 144/100/102 test items from biology, chemistry, and physics; the discussion extends findings to domain-specific QA broadly despite limited subject coverage.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring the Impact of Occupational Personas on Domain-Specific QA." pith.science (2026). https://pith.science/paper/EGW7VLYN

@misc{pith2026250524448,
  author       = {Pith},
  title        = {Pith review of: Exploring the Impact of Occupational Personas on Domain-Specific QA},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EGW7VLYN}},
  note         = {Machine review of arXiv:2505.24448}
}
read the original abstract

Recent studies on personas have improved the way Large Language Models (LLMs) interact with users. However, the effect of personas on domain-specific question-answering (QA) tasks remains a subject of debate. This study analyzes whether personas enhance specialized QA performance by introducing two types of persona: Profession-Based Personas (PBPs) (e.g., scientist), which directly relate to domain expertise, and Occupational Personality-Based Personas (OPBPs) (e.g., scientific person), which reflect cognitive tendencies rather than explicit expertise. Through empirical evaluations across multiple scientific domains, we demonstrate that while PBPs can slightly improve accuracy, OPBPs often degrade performance, even when semantically related to the task. Our findings suggest that persona relevance alone does not guarantee effective knowledge utilization and that they may impose cognitive constraints that hinder optimal knowledge application. Future research can explore how nuanced distinctions in persona representations guide LLMs, potentially contributing to reasoning and knowledge retrieval that more closely mirror human social conceptualization.

Figures

Figures reproduced from arXiv: 2505.24448 by the authors.

Figure 1
Figure 1. Example and performance of PBPs and OPBPs on scientific QA: The Scientist persona out￾performs No Persona (dotted line) by invoking do￾main knowledge. The performance gap between Scien￾tist and the personas Scientific and Investigative shows that LLMs can distinguish between profession-based and personality-based traits, rather than simply associ￾ating with similar input terms like “scient-”. The Artist persona show… view at source ↗
Figure 2
Figure 2. OPBPs’ Micro-averaged accuracy, CR and AR performance across three datasets in the GPT-3.5 model. Blue bars (left of each personality) show the accuracy for the Brief style and red bars (right of each personality) for Descriptive. two metrics: CR = Nlim Ntotal , AR = Nlim and no_ans Ntotal where Nlim is the number of acknowledgments of limitation, Nlim and no_ans is the number of ac￾knowledgments of limitation that … view at source ↗
Figure 4
Figure 4. Layer-wise neuron impact differences be [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 5 canonical work pages

  1. [1]

    Glass, and Pengcheng He

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. 2024. https://openreview.net/forum?id=Th6NyL07na Dola: Decoding by contrasting layers improves factuality in large language models . In The Twelfth International Conference on Learning Representations

    2024

  2. [2]

    Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. https://doi.org/10.18653/v1/2022.acl-long.581 Knowledge neurons in pretrained transformers . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493--8502, Dublin, Ireland. Association for Computational Linguistics

    doi:10.18653/v1/2022.acl-long.581 2022

  3. [3]

    Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.88 Toxicity in chatgpt: Analyzing persona-assigned language models . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1236--1270, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.findings-emnlp.88 2023

  4. [4]

    Silin Gao, Beatriz Borges, Soyoung Oh, Deniz Bayazit, Saya Kanno, Hiromi Wakaki, Yuki Mitsufuji, and Antoine Bosselut. 2023. https://doi.org/10.18653/v1/2023.acl-long.362 P ea C o K : Persona commonsense knowledge for consistent and engaging narratives . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: ...

    doi:10.18653/v1/2023.acl-long.362 2023

  5. [5]

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.446 Transformer feed-forward layers are key-value memories . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484--5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics

    doi:10.18653/v1/2021.emnlp-main.446 2021

  6. [6]

    Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan, Peter Clark, Ashish Sabharwal, and Tushar Khot. 2024. https://openreview.net/forum?id=kGteeZ18Ir Bias runs deep: Implicit reasoning biases in persona-assigned LLM s . In The Twelfth International Conference on Learning Representations

    2024

  7. [7]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://openreview.net/forum?id=d7KBjmI3GmQ Measuring massive multitask language understanding . In International Conference on Learning Representations

    2021

  8. [8]

    J.L. Holland. 1985. https://books.google.co.kr/books?id=8QxBAAAAMAAJ Making Vocational Choices: A Theory of Vocational Personalities and Work Environments . Prentice-Hall series in counseling and human development. Prentice-Hall

    work page 1985

Show all 25 references
  1. [9]

    Hang Jiang, Xiajie Zhang, Xubo Cao, Cynthia Breazeal, Deb Roy, and Jad Kabbara. 2024. https://arxiv.org/abs/2305.02547 Personallm: Investigating the ability of large language models to express personality traits . Preprint, arXiv:2305.02547

    2024 arXiv

  2. [10]

    Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, and Xiaohang Dong. 2024. https://arxiv.org/abs/2308.07702 Better zero-shot reasoning with role-play prompting . Preprint, arXiv:2308.07702

    2024 arXiv

  3. [11]

    Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi MI, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, Linkang Zhan, Yaokai Jia, Pingyu Wu, and Haozhen Sun. 2023. https://arxiv.org/abs/2308.09597 Chatharuhi: Reviving anime character in reality via large languag...

    2023 arXiv

  4. [12]

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, hai zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2024. https://openreview.net/forum?id=ck4SG9lnrQ CMMLU : Measuring massive multitask language understanding in chinese

    2024

  5. [13]

    Minsik Oh, Joosung Lee, Jiwei Li, and Guoyin Wang. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.1020 PK - ICR : Persona-knowledge interactive multi-context retrieval for grounded dialogue . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro...

    2023 doi

  6. [14]

    OpenAI. 2022. https://openai.com/index/chatgpt/ Introducing chatgpt . Accessed: 2024-09-15

    2022

  7. [15]

    O'Brien, Carrie J

    Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. https://arxiv.org/abs/2304.03442 Generative agents: Interactive simulacra of human behavior . Preprint, arXiv:2304.03442

    2023 arXiv

  8. [16]

    Minghai Qin. 2024. https://arxiv.org/abs/2408.15301 The uniqueness of llama3-70b with per-channel quantization: An empirical study . Preprint, arXiv:2408.15301

    2024 arXiv

  9. [17]

    Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. https://aclanthology.org/2023.emnlp-main.814/ Character- LLM : A trainable agent for role-playing . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13153--13187, Singapor...

    2023

  10. [18]

    Emily Sheng, Josh Arnold, Zhou Yu, Kai-Wei Chang, and Nanyun Peng. 2021. https://arxiv.org/abs/2104.08728 Revealing persona biases in dialogue systems . Preprint, arXiv:2104.08728

    2021 arXiv

  11. [19]

    Yu-Min Tseng, Yu-Chao Huang, Teng-Yun Hsiao, Wei-Lin Chen, Chao-Wei Huang, Yu Meng, and Yun-Nung Chen. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.969 Two tales of persona in LLM s: A survey of role-playing and personalization . In Findings of the Association for Com...

    2024 doi

  12. [20]

    Yixin Wan, Jieyu Zhao, Aman Chadha, Nanyun Peng, and Kai-Wei Chang. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.648 Are personalized stochastic parrots more dangerous? evaluating persona biases in dialogue systems . In Findings of the Association for Computational Li...

    2023 doi

  13. [21]

    Xintao Wang, Yunze Xiao, Jen tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, Jiangjie Chen, Cheng Li, and Yanghua Xiao. 2024 a . https://arxiv.org/abs/2310.17976 Incharacter: Evaluating personality fidelity in role-playing agents through ps...

    2024 arXiv

  14. [22]

    Zekun Wang, zhongyuan peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhu Chen, Jie Fu, and Junran Peng. 2024 b . https://openreview.net/forum?id=i4ULDEeBss Role LLM : Benchmark...

    2024

  15. [23]

    Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. 2024 c . https://arxiv.org/abs/2307.05300 Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration . Preprint, arXiv:2307.05300

    2024 arXiv

  16. [24]

    Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024. https://arxiv.org/abs/2402.18815 How do large language models handle multilingualism? Preprint, arXiv:2402.18815

    2024 arXiv

  17. [25]

    Mingqian Zheng, Jiaxin Pei, Lajanugen Logeswaran, Moontae Lee, and David Jurgens. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.888 When a helpful assistant is not really helpful: Personas in system prompts do not improve performances of large language models . In Find...

    2024 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.