Pith. sign in

REVIEW 5 major objections 5 minor 32 references

Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests

T0 review · 5 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Persona-conditioned LLM behaviors on situational judgment tests are stable, trait-linked, and better measured by scenarios than by self-report questionnaires.

desk verdict A useful, well-scoped dataset and pipeline paper whose headline validation claims outrun the evidence inside it; worth engaging for the artifacts, not for the MIRT/benchmark promises. read the letter →

arxiv 2510.22170 v3 pith:DP2SXSCS submitted 2025-10-25 cs.AI

Alexandra Yost , Shreyans Jain , Shivam Raval , Grant Corser , Allen Roush , Nina Xu , Jacqueline Hammack , Ravid Shwartz-Ziv
show 1 more author
Amirali Abdullah
This is my paper · ORCID
classification cs.AI
keywords LLMevaluationsituationaljudgmenttestsHEXACOpersonasbehavioralconsistencypsychometricsstructuredgenerationlatenttraits
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that situational judgment tests—scenarios whose response options are mapped to one of six HEXACO traits—can reveal stable behavioral tendencies in large language models, rather than just superficial variation in persona output. It constructs a large corpus of 8,500 police personas, 4,000 SJTs, and 300,000 scored responses, and reports that persona-conditioned behavior is stable across runs, that latent trait scores predict external benchmarks, and that scenario-based measurement is more reliable than self-report questionnaires. The central quantitative evidence is a set of regressions in which HEXACO trait scores explain SJT choice proportions with adjusted R² values from roughly 0.86 to 0.99, supported by contrasting case studies of two personas. If the core claim holds, the field can assess LLM tendencies through observable, role-relevant choices instead of introspective inventories, and the released dataset becomes a reusable resource for alignment and bias audits.

What carries the argument

The load-bearing object is the trait-mapped situational judgment test: each scenario offers six response options, one per HEXACO trait, and the chosen option is treated as a behavioral observation of a latent trait. Supporting machinery includes structured persona generation with demographic priors and archetype stacks, an LLM-as-judge trait-bleed correction loop that rewrites options until they align to a single trait, and linear regressions of HEXACO trait scores on SJT response proportions. The abstract invokes MIRT as the latent-variable lens, but the paper's limitations section states that full MIRT, EFA, and CFA analyses were not performed, leaving the regression and clustering results

What would settle it

Take the trait-bleed-corrected SJT response options, strip their intended trait labels, and have human raters independently assign each option to a HEXACO trait; if pairwise assignment agreement is at chance (κ≈0) for traits such as eXtraversion or Agreeableness—as the paper's Table 14 already shows for the current judge—then the trait scores used in the regression analyses are not measuring distinct traits, and the R² claim loses its interpretation.

Watch

Extended reading notes

Core claim

The paper's central claim is that an LLM's persona-conditioned decisions on situational judgment tests behave like observations of latent behavioral traits: they are stable across repeated runs, they align with the persona's HEXACO profile, and their aggregate scores carry predictive information about external benchmarks. The authors interpret these traits not as human personality but as consistent response tendencies, and they propose scenario-based psychometric evaluation as a more reliable alternative to classical self-report questionnaires for AI. In the law-enforcement case study, HEXACO self-report scores predict the proportion of SJT choices made for each trait (adjusted R² ≈ 0.8–0.9)

Load-bearing premise

The central claims assume that every SJT response option is uniquely aligned to exactly one HEXACO trait after the LLM-judge's 'trait-bleed' correction; the paper's own inter-rater agreement table shows chance-level agreement for several rubric dimensions, and its limitations section states that the confirmatory factor-analytic and MIRT analyses were not performed.

Editorial extensions

If this is right

  • If SJT-based measurement is accepted, LLM evaluation can move from self-report questionnaires to observable choice behavior in role-relevant scenarios.
  • Stable persona-conditioned behavior would mean a profile measured once can be trusted across runs and contexts, lowering the cost of persona-based testing.
  • The reported link between latent trait scores and external benchmarks suggests SJT-derived traits carry information that generalizes beyond the test itself.
  • The released dataset of 8,500 personas, 4,000 SJTs, and 300,000 responses would directly support downstream alignment, bias, and safety audits by other teams.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The validity of the whole pipeline depends on trait labels being truly separable; a direct check is to have independent raters, blind to the intended mapping, assign the corrected response options to HEXACO traits and require agreement clearly above chance before computing any trait scores.
  • The high regression R² may be partly circular, since both the predictor (HEXACO self-report) and the outcome (SJT choices) come from the same conditioned model and the same trait ontology; an independent behavioral criterion is needed to settle this.
  • A natural extension is to apply the same SJT protocol to multi-turn interactions or to non-HEXACO trait systems, since the current design captures only single-shot judgments.
  • If stable trait–behavior mappings hold, regulators could audit model behavior in deployment-relevant scenarios without relying on introspective self-reports, which are known to be sycophancy-prone in LLMs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a framework for psychometric evaluation of LLM behavior using situational judgment tests (SJTs), structured synthetic police-officer personas, and HEXACO trait-mapped response options. It reports construction of a large corpus (8,500 personas, 4,000 SJTs, 300,000 responses), an LLM-based trait-bleed correction pipeline, and a regression analysis claiming that HEXACO traits strongly predict SJT behavior (adjusted R² ≈ 0.86–0.99). The abstract further claims that latent trait scores predict external benchmarks such as TruthfulQA and EmoBench and that MIRT reveals consistent latent structure. The manuscript's own text, however, disclaims these analyses: §10 states that the authors do not go beyond basic clustering and data exploration, and §12 explicitly states that EFA/CFA and MIRT were not conducted. The central quantitative validation in §9 and Appendix J is an internal regression with no held-out split, and Table 14 shows that human–LLM judge agreement for several trait-alignment rubrics is at or near chance. The paper is best read as a dataset/engineering contribution plus a tentative exploratory analysis, not as a validated psychometric instrument.

Significance. If the central claims were supported, the paper would make a useful contribution to behavior-based LLM evaluation: a large, structured, expert-informed SJT corpus with public release would be a valuable resource, and a valid trait-behavior mapping would strengthen the case for scenario-based assessment over self-report inventories. The authors deserve credit for the scale of the resource, the involvement of domain experts in base scenario design, the structured-generation pipeline, and the explicit acknowledgment of limitations in §12 and Appendix P. However, the headline claims — external predictive validity and MIRT-based latent structure — are not supported by anything in the manuscript. The only quantitative validation is circular: both HEXACO scores and SJT responses are generated from the same persona descriptions, and the regression is fit on the same data without validation. Moreover, the trait labels assigned to SJT options, which are load-bearing for every downstream score, have near-zero human–LLM agreement for eXtraversion and Agreeableness alignment. In its current form, the paper's evidence does not establish the reliability or validity of the proposed trait-scoring f

major comments (5)
  1. [Abstract vs. §10 and §12] The abstract claims that 'latent trait scores predict external benchmarks (e.g., TruthfulQA, EmoBench)' and that 'MIRT reveals consistent latent structure.' The full text does not report any such analyses. §10 states: 'Due to time constraints, we do not go beyond basic clustering and data exploration.' §12 states: 'We do not conduct extensive exploratory or confirmatory factor analysis (EFA/CFA), or apply multidimensional item response theory (MIRT).' TruthfulQA and EmoBench are not mentioned in the body, references, or appendices. This is not a minor overstatement; the abstract's principal validation claims are unsupported by the manuscript's own evidence.
  2. [§9, Table 3, Appendix J] The regression analysis is presented as validation ('base HEXACO traits strongly predict SJT behaviors, high R² ≈0.8–0.9'), but both predictors and outcomes are derived from the same model conditioned on the same persona text. No held-out split, cross-validation, or external criterion is used. Appendix J's own per-trait correlations are weak or negative for several traits (e.g., Honesty–Humility r = –0.122, eXtraversion r = 0.504), and the high multivariate R² is explained as a joint linear effect. This is an internal consistency check at best, not predictive or construct validity. The claim of external validity is therefore unsupported.
  3. [Table 14, §G.3] The trait-scoring pipeline assumes each SJT response option maps uniquely to one HEXACO trait. The inter-rater agreement table reports κ = 0 for eXtraversion Alignment and eXtraversion Overlap, κ = 0.067 for Agreeableness Alignment, κ = 0.000 for Ethical Tension, and κ = 0.000 for Scenario Realism. The paper's aggregate statement of 'substantial (~0.63)' agreement is driven by rubrics with perfect agreement (e.g., Conscientiousness, Openness, race, gender) and obscures the fact that for two of the six HEXACO traits the human and LLM judges agree at chance. Because 'trait bleed' correction is performed by the same kind of LLM judge, the corrected dataset may still contain mislabeled options. Every downstream trait score, regression, and clustering analysis is contaminated if this mapping is unreliable.
  4. [§8, Tables 1–2] The case studies of Officer Wong and Officer Hagedorn are presented as validation of 'construct coherence' and 'strong cross-measure validity.' However, these are two selected personas, qualitatively interpreted by the authors' psychology expert, with no quantitative reliability metric, no independent raters (§12 and Appendix P explicitly disclaim independent raters), and no correction for selection. Showing that a Tough Cop persona receives high Conscientiousness and Honesty–Humility scores in both HEXACO and SJT formats demonstrates that the personas are coherent with their archetypes; it does not validate the SJT scoring as a measure of behavior.
  5. [§12, Appendix P] The limitations section explicitly states that no independent expert raters were used to cross-validate the score rubrics or persona classifications. The only human ratings reported are from the authors themselves (Appendix F, Table 13). This is appropriate as a stated limitation, but it means that the paper's validity claims rest entirely on internal consistency and author judgment. For a paper whose central contribution is 'psychometric evaluation,' the absence of independent annotation is a load-bearing gap rather than a routine caveat.
minor comments (5)
  1. [§2 vs. Abstract and Appendix F/H] The dataset counts are inconsistent. The abstract says 8,500 personas, 4,000 SJTs, and 300,000 responses; §2 says '200 personas, 500 SJTs, and 3 models'; Appendix J says responses were generated for '1504 personas across 500 SJT items.' These numbers should be reconciled.
  2. [Reference list, Cabrera and Nguyen] The reference 'Michael A McDaniel Cabrera and Nhung T Nguyen. 2001' appears malformed. It presumably should cite Cabrera and Nguyen (2001) or McDaniel and Nguyen; please correct.
  3. [§G.3 / §7] Averaging Cohen's κ across rubrics with very different base rates and difficulties is misleading. The paper reports 'substantial (~0.63)' but Table 14 includes values from 0 to 1. Report per-rubric κ and, if an average is needed, justify the aggregation method.
  4. [§10, 'Further analysis'] The observation that minority personas show a 'jump in Conscientiousness scores from 0.24 to 0.52' is reported without context, error bars, or a statistical test. As written, it risks implying a stable demographic effect from exploratory clustering. Please either analyze this properly or present it as a purely anecdotal observation.
  5. [§7, trait-bleed correction] The trait-bleed correction accepts a Trait Fit Score below 5 and rewrites the option. The threshold and the iterative loop are not fully specified: how many iterations, what stopping rule, and does the corrected option receive a re-evaluation? Adding this detail would improve reproducibility.

Circularity Check

2 steps flagged · score 6.0 of 10

SJT 'validation' regresses trait-labeled options against same-prompt HEXACO scores; abstract promises external/MIRT validation the paper disclaims.

  1. self definitional [§7 'SJT bank and controlled augmentation'; §9 'Trait Regression Analysis'; §10 'Conclusions']
    "Each scenario includes six reasonable and feasible responses, one per HEXACO trait. ... For response options with high trait bleed (Trait Fit Score < 5), we feed them back into the LLM to minimize overlap and sharpen their correspondence to an intended trait. ... In Section 9, we further validate the instrument: base HEXACO traits strongly predict SJT behaviors (high R2 ≈0.8–0.9 ), and item scores align with those traits."

    The SJT outcome is not an independent behavioral measure: the response options are pre-assigned to HEXACO traits and then LLM-corrected until each option 'uniquely' expresses one trait. SJT trait scores are counts of choices among these trait-labeled options. HEXACO-100 scores and SJT choices are both elicited from the same model under the same persona prompt. Regressing HEXACO scores on SJT trait-choice proportions therefore correlates two outputs of the same prompt-construct loop; the option-to-trait mapping is embedded in the scoring by construction. The resulting R2 ≈ 0.8–0.9 is evidence of internal consistency of generation, not of SJT predictive validity.

  2. fitted input called prediction [§9 'Trait Regression Analysis'; Abstract]
    "To aggregate and quantify the explanatory power of personality traits on SJT performance, we train a linear regression model using HEXACO scores as input predictors and aggregated SJT scores as the dependent variable. ... Across all traits, the models yield consistently high adjusted R2 values (~0.8–0.9) as seen in Table 3 with generally positive coefficients, particularly for the focal trait being predicted."

    The regression is fitted and reported on the same sample (1,504 personas, 500 SJTs); adjusted R2 is an in-sample goodness-of-fit, not a prediction. The paper then calls this 'base HEXACO traits strongly predict SJT behaviors' and the abstract claims 'latent trait scores predict external benchmarks (e.g., TruthfulQA, EmoBench),' but no held-out personas, out-of-sample forecast, TruthfulQA/EmoBench analysis, or MIRT analysis is provided. The 'prediction' claim is thus an in-sample fit relabeled as predictive validation.

full rationale

The only quantitative validation of the central claim is §9's regression. That regression is circular in two related ways: (i) the SJT dependent variable is constructed by assigning each response option to one HEXACO trait and then LLM-correcting 'trait bleed,' so SJT trait scores are already HEXACO-like labels; (ii) the HEXACO predictors and SJT responses come from the same model prompted with the same persona, so high R2 documents within-model consistency, not external predictive validity. The paper itself flags the missing controls in §12: 'We do not conduct extensive exploratory or confirmatory factor analysis (EFA/CFA), or apply multidimensional item response theory (MIRT)' and §10: 'Due to time constraints, we do not go beyond basic clustering and data exploration.' These limitation passages directly contradict the abstract's 'latent trait scores predict external benchmarks (e.g., TruthfulQA, EmoBench), and MIRT reveals consistent latent structure.' That contradiction is a support failure rather than a circularity, but it removes any independent benchmark evidence that could have broken the circularity. There is no self-citation uniqueness chain or imported ansatz; the circularity is local to the trait-score validation loop. Score 6 reflects that the central 'more reliable alternative' claim rests on the in-sample, by-construction regression, while the dataset construction and human annotation provide some independent content.

Assumptions & free parameters 2 free parameters · 5 assumptions · 3 invented entities

Two free parameters are used (regression coefficients and trait-bleed threshold); five axioms underlie the framework; three invented constructs lack independent evidence. The heaviest burden is the domain assumption that LLM text under persona conditioning behaves like a latent-trait measurement system.

free parameters (2)
  • HEXACO→SJT linear regression coefficients = Not individually reported; adjusted R² 0.86–0.99 per trait
    Section 9 fits six predictor coefficients per trait to LLM-generated HEXACO and SJT data; reported R² are in-sample fit, not holdout prediction.
  • Trait-bleed correction threshold = Trait Fit Score < 5 triggers rewrite
    Section 7 hand-chosen threshold for LLM-as-judge correction; directly determines which SJT options are rewritten and thus the trait-alignment quality.
assumptions (5)
  • domain assumption HEXACO latent trait model applies to LLM-generated text under persona conditioning
    Section 3 states HEXACO trait scores represent latent personality variables that should manifest consistently in behavioral choices; this is assumed rather than established for LLMs.
  • domain assumption LLM-as-a-judge provides valid trait-alignment and persona-quality ratings
    Sections 7 and G use GPT-4.1 as judge to detect trait bleed and scale rubrics; Table 14 shows chance-level agreement on several rubrics.
  • standard math Compositional regression assumptions for six proportional SJT scores
    Section 9 regresses SJT proportions on HEXACO scores without accounting for the sum-to-one constraint; standard linear regression independence assumptions are strained.
  • domain assumption Structured personas instantiate stable behavioral tendencies in LLMs
    Sections 6 and 8 treat persona descriptions as sufficient to induce trait-consistent behavior; no independent manipulation check beyond LLM self-report.
  • ad hoc to paper Trait bleed can be corrected by iterative LLM rewriting
    Section 7 introduces a custom correction loop with no independent verification; the correction target is the same judge's score.
invented entities (3)
  • Eight police-officer persona archetypes (Professional, Enforcer, Reciprocator, Avoider ×2, Tough Cop, Problem Solver ×2)
    purpose: Generate demographically and psychologically varied personas for LLM conditioning
    Co-designed by authors and not externally validated; no falsifiable handle outside the paper.
  • Latent behavioral tendency variables measured by SJTs
    purpose: Interpret SJT response patterns as observations of underlying stable traits
    No external criterion or MIRT/CFA analysis is provided; the abstract claims MIRT but limitations say it was not run.
  • Trait-bleed-corrected SJT bank
    purpose: Provide response options uniquely aligned to one HEXACO trait
    Quality is assessed by the same LLM judge family used for correction; human agreement is at chance for several traits.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests." pith.science (2026). https://pith.science/paper/DP2SXSCS

@misc{pith2026251022170,
  author       = {Pith},
  title        = {Pith review of: Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DP2SXSCS}},
  note         = {Machine review of arXiv:2510.22170}
}
read the original abstract

Persona conditioning is widely used to steer large language model (LLM) behavior, but it is unclear whether it induces stable behavioral structure or superficial variation. We propose a framework to measure consistent behavioral tendencies using situational judgment tests (SJTs), multidimensional item response theory (MIRT), and structured synthetic personas, treating responses as observations of latent behavioral variables. Across large-scale SJT and persona datasets, we find that persona-conditioned behaviors are stable across runs, latent trait scores predict external benchmarks (e.g., TruthfulQA, EmoBench), and MIRT reveals consistent latent structure. We validate these results through human annotation, benchmark evaluation, and internal consistency analyses. We interpret these traits not as human personality, but as stable behavioral tendencies expressed across contexts. Our results show that scenario-based psychometric evaluation provides a more reliable alternative to classical self-report approaches for assessing LLM behavior, and we release datasets to support further study.

Figures

Figures reproduced from arXiv: 2510.22170 by the authors.

Figure 1
Figure 1. Overview of the synthetic persona generation pipeline. We use GPT-4.1 to generate demographically [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the scenario generation pipeline. Domain experts first develop base scenarios with trait [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 14
Figure 14. Clusters using SJT Answers, colored by archetypes I.2 Trait Distributions and Preferences Similar to the archetype level analyses presented earlier, we extended our examination to additional persona attributes such as ethnic background, age, and gender. Notably, personas representing Colom￾bian, East Asian, Hispanic/Latino, and Southeast Asian backgrounds exhibited a remarkably higher preference for the Conscientiou… view at source ↗
Figures from the paper (1 more)
Figure 13
Figure 13. Figure 13: Clusters using HEXACO Answers, colored by archetypes [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 5 linked inside Pith

  1. [1]

    Identify prominent memoirs authored by po- lice officers or other members of law enforce- ment

  2. [2]

    The chosen profile in- cludes: • Sections derived from an intake inter- view

    Select a de-identified psychological report to serve as a structural reference for integrating the memoir content. The chosen profile in- cludes: • Sections derived from an intake inter- view. • Data and evaluations from 15 psychome- tric tests. • A summary with diagnostic impressions and clinical recommendations

  3. [3]

    Bias and fairness in large language models: A survey.Computational Linguistics, 50(3):1097– 1179. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schel- ten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mi- tra, Archie Sravanku...

    arXiv 2024

  4. [4]

    Review all profiles generated for accuracy and alignment with psychometric principles

  5. [5]

    Louis Kwok, Michal Bravansky, and Lewis D Griffin

    Creating a psychological test in a few sec- onds: Can chatgpt develop a psychometrically sound situational judgment test?European Journal of Psy- chological Assessment. Louis Kwok, Michal Bravansky, and Lewis D Griffin

  6. [6]

    arXiv preprint arXiv:2408.06929

    Evaluating cultural adaptability of a large lan- guage model via simulation of synthetic personas. arXiv preprint arXiv:2408.06929. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Effi- cient memory management for large language model serving with pagedattention. InProcee...

    arXiv 2023

  7. [7]

    arXiv preprint arXiv:2412.12144

    Automatic item generation for personality situational judgment tests with large language models. arXiv preprint arXiv:2412.12144. Filip Lievens and Stephan J Motowidlo. 2016. Situa- tional judgment tests: From measures of situational judgment to measures of general domain knowledge. Industrial and Organizational Psychology, 9(1):3– 22. Yunting Liu, Shreya...

    arXiv 2016

  8. [13]

    The report should be based on the indi- vidual described in the attached mem- oir

    Provide GPT-4.1 with the memoirs, the de- identified report, and the following prompt: Please create a comprehensive psycho- logical report modeled after the format and style of the example report in the documentComprehensive Report_E.J.. The report should be based on the indi- vidual described in the attached mem- oir. The goal is to generate a complete ...

Show all 32 references
  1. [15]

    Behav- ioral Observations

    Create condensed LLM personas by reducing the first six sections of each profile to one sen- tence each, while retaining the entire “Behav- ioral Observations” section and the complete summary. B Hand Designed Persona Schema Sections Section Fields / Items Demographic Fields N...

    2020

  2. [17]

    The Avoider

    Trait Alignment of Options (HEXACO Mapping) Definition:Extent to which each re- sponse option uniquely expresses the intended HEXACO trait without redundancy or over- lap. Scoring: 5 (Excellent):Clear and unique expression of the assigned trait with minimal overlap. 4 (Good):M...

    2018

  3. [18]

    Trait Fit EvaluationFor each option, evaluate how strongly it aligns with its intended trait definition. Use a 1–5 scale: • 5 = Very strong, clean representation, no leakage • 4 = Strong but with minor overlap • 3 = Moderate, noticeable blending • 2 = Weak, trait unclear or di...

  4. [19]

    Agreeable- ness)

    Separation AnalysisHighlight where options overlap or bleed into each other (e.g., eXtraversion vs. Agreeable- ness). Explain why the overlap occurs

  5. [20]

    Ensure each rewrite minimizes overlap with other traits and includes specific, actionable decisions rather than vague choices

    Correction SuggestionsFor any option rated below 5, propose a corrected rewrite that emphasizes the target trait more cleanly. Ensure each rewrite minimizes overlap with other traits and includes specific, actionable decisions rather than vague choices

  6. [21]

    scenario_summary

    Final Corrected SJT ObjectOutput an object with the exact same structure as the input SJT dictionary. Each option should contain the corrected version if a rewrite was needed, or the unchanged original if not. 5.Output FormatReturn results in structured JSON with this schema: ...

  7. [22]

    Write it as a concrete, sensory, scene-level story (180–250 words)

    The memoir_narrative is canonical grounding. Write it as a concrete, sensory, scene-level story (180–250 words). All fields must align with its facts and tone; if conflicts arise with archetype, prefer narrative. If conflicts arise with demographics, pick the demographics

  8. [23]

    Do NOT quote or paraphrase it; never list ‘Core trait/Focus/Strength- s/Challenges’

    Treat the archetype as a loose orientation. Do NOT quote or paraphrase it; never list ‘Core trait/Focus/Strength- s/Challenges’

  9. [24]

    Rephrase and localize details to the scene

    Do not reuse ≥5 consecutive words from inputs (archetype description or memoir summary). Rephrase and localize details to the scene

  10. [25]

    Favor specificity (who/what/where/when) over generic traits; vary wording across sections

  11. [26]

    Persona should be internally consistent between fields

  12. [27]

    Prefer specific, scene-derived wording

    Use natural phrasing; do not feel compelled to use section labels or taxonomy words (e.g., ‘stress’, ‘trauma’, ‘coping’, ‘abstraction’, ‘obsession’). Prefer specific, scene-derived wording. Persona User Prompt Template User Prompt Template Selected archetype: {archetype_name} ...

  13. [28]

    GPT 4.1 (OpenAI et al., 2024)

    2024

  14. [29]

    GPT 4.1-mini (OpenAI et al., 2024)

    2024

  15. [30]

    Qwen 3-0-6B Embedding Model (Zhang et al., 2025)

    2025

  16. [31]

    Qwen 2.5-7B-Instruct Model (Qwen et al., 2025)

    2025

  17. [32]

    Future work could include indepen- dent raters and inter-rater reliability assessments to enhance robustness and minimize potential bias

    Llama 3.1-8B-Instruct Model (Grattafiori et al., 2024) P Annotators We have not employed independent expert raters to cross-validate score rubrics or persona classifi- cations beyond the expert feedback and review by our authors. Future work could include indepen- dent raters ...

    2024

  18. [2001]

    Jacob Cohen

    Situational judgment tests: A review of prac- tice and constructs assessed.International journal of selection and assessment, 9(1-2):103–113. Jacob Cohen. 1960. A coefficient of agreement for nominal scales.Educational and Psychological Mea- surement, 20(1):37–46. Dane Corneil...

    1960

  19. [2012]

    Brandon T Willard and Rémi Louf

    Model fit and model selection in structural equation modeling.Handbook of structural equation modeling, 1(1):209–231. Brandon T Willard and Rémi Louf. 2023. Efficient guided generation for large language models.arXiv preprint arXiv:2307.09702. Shanle Yao, Babak Rahimi Ardabili...

    2023 arXiv

  20. [2014]

    Tiancheng Hu and Nigel Collier

    To boast or not to boast: Testing the humility aspect of the honesty–humility factor.Personality and Individual Differences, 69:12–16. Tiancheng Hu and Nigel Collier. 2024. Quantifying the persona effect in llm simulations.arXiv preprint arXiv:2402.10811. Hang Jiang, Xiajie Zh...

    2024 arXiv

  21. [2015]

    Specifically, we used PGMs created from US Census data and the names data of Rosenman et al

    to support cascaded PGMs with arbitrary post-processing steps. Specifically, we used PGMs created from US Census data and the names data of Rosenman et al. (2023). ZIP codes with corresponding city and state generated to ground later variables in geography; first, middle and l...

    2023

  22. [2023]

    Bolei Ma, Berk Yoztyurk, Anna-Carolina Haensch, Xin- peng Wang, Markus Herklotz, Frauke Kreuter, Bar- bara Plank, and Matthias Assenmacher

    Illuminating the black box: A psychometric investigation into the multifaceted nature of large language models.arXiv preprint arXiv:2312.14202. Bolei Ma, Berk Yoztyurk, Anna-Carolina Haensch, Xin- peng Wang, Markus Herklotz, Frauke Kreuter, Bar- bara Plank, and Matthias Assenm...

    2024 arXiv

  23. [2024]

    Michael A McDaniel Cabrera and Nhung T Nguyen

    Personality testing of large language models: limited temporal stability, but highlighted prosocial- ity.Royal Society Open Science, 11(10):240180. Michael A McDaniel Cabrera and Nhung T Nguyen

  24. [2025]

    Yang Lu, Jordan Yu, and Shou-Hsuan Stephen Huang

    Leveraging llm respondents for item evalua- tion: A psychometric analysis.British Journal of Educational Technology, 56:1028–1052. Yang Lu, Jordan Yu, and Shou-Hsuan Stephen Huang

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.