Value Portrait: Assessing Language Models' Values through Psychometrically and Ecologically Valid Items

Dongmin Choi; Eun-Ju Lee; Jongwook Han; Woojung Song; Yohan Jo

REVIEW 3 major objections 5 minor 1 cited by

Value Portrait: Assessing Language Models' Values through Psychometrically and Ecologically Valid Items

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that LLM value orientations can be measured with psychometrically validated items, and that 44 evaluated models uniformly prioritize Benevolence, Security, and Self-Direction while de-emphasizing Tradition, Power, and…

desk verdict A genuinely useful, reproducible benchmark that improves on perceived-value annotation, but its central assumption that human-validated items measure LLM values is unverified, so the value profiles should be read as suggestive. read the letter →

arxiv 2505.01015 v3 pith:366WAVIZ submitted 2025-05-02 cs.CL cs.AI

Jongwook Han , Dongmin Choi , Woojung Song , Eun-Ju Lee , Yohan Jo This is my paper · ORCID

classification cs.CLcs.AI

keywords valueorientationlargelanguagemodelsSchwartztheoryofbasicvaluespsychometricbenchmarkPVQ-21demographicbiasalignmentpersonaprompting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that existing value benchmarks for language models mislabel responses because annotators guess which values a text expresses, importing their own biases; it proposes instead to validate each test item psychometrically, by whether people who actually hold a value identify with the response. The result is Value Portrait, a benchmark of 520 query-response pairs drawn from real human-LLM interactions (ShareGPT, LMSYS) and human advisory exchanges (Reddit's AITA, Dear Abby), each tagged with the Schwartz values (and Big Five traits) whose scores correlate with human raters' "how similar is this to your own thoughts?" judgments ($|r_s| \geq 0.3$, $p<0.05$). Used to evaluate 44 models, the benchmark yields a consistent cross-model profile, with Benevolence, Security, and Self-Direction prioritized and Tradition, Power, and Achievement de-emphasized, and it shows that persona-prompted GPT-4o exaggerates gender, age, and political value differences relative to European Social Survey data. If the framework is right, it gives developers and auditors a contamination-resistant way to measure what values models actually express, turning value claims about LLMs from opinion into something testable.

What carries the argument

The load-bearing device is the value-correlation item. For every query-response pair, the paper computes Spearman correlations between crowdworkers' six-point similarity ratings and their scores on the official Portrait Values Questionnaire (PVQ-21, ten Schwartz value dimensions) and the BFI-10 personality inventory; items whose correlation with a dimension passes $|r_s| \geq 0.3$ with $p<0.05$ become items for that dimension. LLM evaluation then reuses the same human self-report prompt, "How similar is this response to your own thoughts?", averaging responses across six prompt variants, and converts raw scores into relative value priorities by subtracting each model's mean item response, following Schwartz's ipsatization convention for human value measurement. The Schwartz ten-value circle supplies the taxonomy, and the same correlation pipeline extends the benchmark to the Big Five personality traits.

What would settle it

Run the same 520 items on an unaligned base model (no instruction tuning or RLHF) or on a model instructed to role-play a maximally unhelpful assistant: if the profile still shows high Benevolence, Security, and Self-Direction, the instrument is reading training-text priors rather than model values. A complementary check is behavioral: have each model produce free-form answers to the same 104 queries, then test whether models that self-report higher Benevolence on Value Portrait also generate measurably more benevolent text; if self-reports and behavior diverge, the self-report reading fails.

Watch

Extended reading notes

Core claim

Value Portrait claims to be a psychometrically and ecologically valid instrument for measuring LLM value orientations. Each of its 520 items is a real-world query paired with a response generated by GPT-4o, and each item's link to a value is established empirically: roughly 46 crowdworkers per item rated how similar the response was to their own thoughts, and those ratings were Spearman-correlated with the same raters' PVQ-21 value (and BFI-10 personality) scores. Items with $|r_s| \geq 0.3$ and $p<0.05$ for a dimension became that dimension's items, leaving 549 value correlations and 287 trait correlations. Administered to 44 LLMs with the same six-point "how similar to your own thoughts?" question, averaged over six prompt variants and mean-centered following Schwartz's scoring method, the benchmark shows high internal consistency (Cronbach's $\alpha$ between 0.76 and 0.96) and reveals a stable profile across models: Benevolence, Security, and Self-Direction rank highest, while Tradition, Power, and Achievement rank lowest. The authors further report that reasoning models amplify Benevolence, larger models differentiate values more sharply, and persona-prompted GPT-4o overstates demographic differences, for example a male-female Conformity gap of 0.51 versus 0.02 in human data and a Left-Right Hedonism gap of 0.74 versus 0.03, while steering experiments show most values respond to prompts but Benevolence does not, since steering toward it raised Security by 1.08.

Load-bearing premise

That an LLM's answer to "How similar is this response to your own thoughts?" registers the model's actual value orientation the same way the identical question registers a human's, rather than capturing prompt compliance, socially desirable mimicry, or statistics of the model's training text instead; the paper's own appendix concedes this assumes outputs reflect the model's internal preferences.

Editorial extensions

If this is right

Any new LLM can be audited directly: run the released items, average over the six prompts, and read off mean-centered value scores that are internally consistent ($\alpha \geq 0.76$).
The uniform profile across 44 models implies that instruction tuning and safety alignment have pushed the field's models toward a shared value posture, so claims of distinctive brand-specific values become deviations to be explained rather than defaults.
Persona auditing against representative human survey data becomes a standard bias check: GPT-4o's exaggerated gender, age, and political gaps are concrete, measurable risks for anyone generating synthetic demographic data with LLMs.
Prompt steering is not a reliable value-control knob: while Universalism, Power, Hedonism, and Self-Direction respond to steering, Benevolence does not, so alignment interventions need measurement rather than assumed prompt effects.
The identical pipeline yields Big Five trait scores, so one benchmark covers both value and personality assessment without re-annotation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because Schwartz values form a correlated circular structure, the reported single-dimension scores will partly reflect neighboring values; the paper's own cross-loading analysis (same-direction value pairs average circular distance 1.59, opposite-direction pairs 3.54) means the ten scores should be read as one profile, not ten independent measures.
A testable extension the paper leaves implicit: compare each model's Value Portrait self-ratings with its free-form responses to the same 104 queries; if self-reported high-Benevolence models do not generate more benevolent texts, the instrument captures stated preference rather than expressed behavior.
The demographic-bias result is pinned to the European Social Survey as the human baseline; rerunning the persona audit against non-Western value datasets would show whether GPT-4o's amplification of group differences is a general property or specific to the values and demographics the ESS measures.
If LLM similarity ratings are driven by training-text statistics rather than a stable self, Value Portrait's absolute profile may partly reflect the moral vocabulary dominant in alignment data; the benchmark would still reliably rank models relative to one another, a use that survives even if the absolute profile does not.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Value Portrait is a worthwhile new resource: 104 realistic queries, 520 query-response pairs, 7,800 human similarity ratings tied to PVQ-21 and BFI-10, and a 44-model evaluation, with code and data released. The correlation-based item selection is a genuine step beyond perceived-value annotation, and the paper's own control analyses (cross-loading distances consistent with Schwartz's circular structure, and the honest reporting of failed value-targeted generation) are informative. The demographic bias comparisons are creative.

The soft spot is exactly where the stress-test note points. Item validity is established in human raters, then transferred to LLMs by asking the model the same self-report question. That is an assumption, not a demonstrated fact. Cronbach's alpha on LLM responses tells you only that LLM answers are internally consistent, not that they track Schwartz constructs. The paper's Appendix D (only 11.25% of value-targeted GPT-4o responses aligned) and Appendix I (Benevolence steering raised Security more than Benevolence) are not disqualifying, but they are exactly the kind of dissociation that makes the transfer assumption non-trivial. The Limitations section and Appendix L acknowledge general self-report issues but never directly address the human-to-LLM transfer. That should be addressed head-on.

The multiple comparisons issue is real but secondary: 7,800 correlations at p<0.05 with n≈46 will produce many expected false positives around r=0.3. The cross-loading analysis suggests signal, but a corrected threshold or split-half replication would make the item selection much more convincing.

None of this kills the paper. The core methodological point—correlation with actual value scores is better than perceived-value labeling—holds. The benchmark is reproducible and useful regardless of whether the specific value profiles are final. The headline value priorities (Benevolence, Security, Self-Direction over Tradition, Power, Achievement) replicate prior work, so they are plausible; the demographic bias numbers should be treated as suggestive until the measurement issue is resolved.

This paper deserves a serious referee. I would send it to review with a request to either soften the validity claims or add an external LLM-behavior validation, and to correct for multiple comparisons. The benchmark is worth having in the literature.

Referee Report

3 major / 5 minor

Summary. The paper introduces Value Portrait, a benchmark for evaluating the value orientations of large language models. Items are query-response pairs sampled from human-LLM interactions (ShareGPT, LMSYS) and human-human advisory contexts (Reddit Scruples, Dear Abby), with responses generated by GPT-4o. Each item was rated by human Prolific participants for similarity to their own thoughts, and Spearman correlations were computed between these ratings and participants' PVQ-21 (and BFI-10) scores; items with |r| >= 0.3 and p < 0.05 were retained. The benchmark is then administered to 44 LLMs, which rate how similar each item is to their own thoughts, and value orientations are computed from the averaged, normalized ratings. The authors report that LLMs prioritize Benevolence, Security, and Self-Direction and de-emphasize Tradition, Power, and Achievement, and they analyze demographic persona biases and value-steering effects for GPT-4o. The paper claims the benchmark is psychometrically and ecologically valid for LLM value assessment.

Significance. If the construct-validity assumptions are established, Value Portrait would be a valuable resource: it uses ecologically sourced queries, provides a large 44-model evaluation, and is accompanied by released code and data. The manuscript is transparent in reporting the failure of value-targeted generation (Appendix D) and unexpected steering interactions (Appendix I), which are useful negative results for the field. However, the central load-bearing step—transferring human item validity to LLM self-reports—is unverified, and the paper's own appendices supply evidence of dissociation between LLM stated similarity ratings and value-relevant behavior. The benchmark's usefulness as a measure of model values therefore remains conditional on additional validation or on a more cautious reframing of the claims.

major comments (3)

[§3.4.1–3.4.2, Appendices D, F, I, L] The central construct-validity argument transfers item validity from human raters to LLMs without direct evidence. Correlations computed on human participants (Section 3.4.1) justify retaining items for humans; Cronbach's alpha in Appendix F is computed on LLM responses and only shows internal consistency, not that the scale measures the intended Schwartz constructs in models. Appendix D reports that GPT-4o responses explicitly generated to express a value aligned with the intended PVQ dimension only 11.25% of the time, and Appendix I shows that steering GPT-4o toward Benevolence raised Security by +1.08 versus +0.11 for Benevolence; both results suggest LLM similarity ratings can dissociate from the intended value constructs. Appendix L also explicitly frames the key assumption as an unsupported 'assuming outputs reflect the model's internal preferences.' Because the value profiles in Section 4.1 and the bias comparisons depend on this assumption, the benchmark's central claim is not yet established.
[§3.2.3 and §3.4.1] The item-selection procedure is vulnerable to multiple-testing and underpowered detection. The paper computes thousands of correlations (520 items × 10 value dimensions, plus 5 BFI traits) and retains all correlations with |r| ≥ 0.3 and p < 0.05 without any multiple-comparison correction; under the null, roughly 5% of several thousand tests will pass the p-threshold, so many retained items are likely false positives. Moreover, the claim that the average 46 participants per item yields power 0.8 to detect r = 0.3 at p < 0.05 is not consistent with standard sample-size calculations: with N = 46 the approximate power for a two-sided Pearson test of r = 0.3 is about 0.5, and likely lower for Spearman. The authors should provide a power analysis and either correct for multiplicity (e.g., FDR) or justify the threshold selection on independent grounds.
[§4.2, Tables 9–11] The demographic-bias comparisons are purely descriptive and lack inferential support. The text states that GPT-4o 'significantly exaggerates' gender differences and 'amplifies political value differences,' but no confidence intervals, standard errors, or significance tests are reported for the GPT-4o persona-condition scores or for the comparison with ESS human data. The ESS samples are weighted and clustered; the analysis in Appendix H reports unweighted relative differences with no within-country or within-group variances. The bias claims in the abstract and introduction therefore currently exceed what the data support. Please report uncertainty quantification (e.g., bootstrapped CIs across items and prompts) and, where possible, test the human-model differences.

minor comments (5)

[§3.1, Appendix B] The evaluation of prior datasets uses different response formats from the original annotation tasks (binary Yes/No for ValueNet and Likert for FULCRA), and samples only 20 items per dataset; the reported 5% and 10% alignment rates should be interpreted as a pilot and the protocol mismatch acknowledged.
[§3.4.1, Figure 3] The filtering criterion in Figure 3 reads 'At least One Corr > 0.3' but the text specifies |r| ≥ 0.3 with p < 0.05; please make the figure consistent.
[Appendix K] The released dataset should be checked against the LMSYS-Chat-1M license quoted in Appendix K, which prohibits distribution, copying, and transfer to third parties; the paper does not state whether the dataset has been approved for redistribution.
[§4.2, Appendix H.1] The persona prompts used in Appendix H.1 are extremely short (e.g., 'Your gender is male.') and no pilot or manipulation check is reported to confirm that these prompts actually instantiate the intended persona in GPT-4o; consider reporting the full set of prompts and including a check (e.g., self-reported persona adherence).
[Throughout] There are several minor typographical issues (e.g., 'Y ohan Jo' in the author list, inconsistent superscript markers in Table 5); a careful proofread is recommended.

Circularity Check

1 steps flagged · score 2.0 of 10

Item validity is asserted by the same correlation filter that defines the item set; the LLM value evaluations themselves are not fitted to human targets.

self definitional [Section 3.4.2, Validity paragraph (item filter defined in §3.4.1)]
"As explained in §3.4.1, we only keep items that have meaningful correlations (rs≥0.3) with their corresponding value dimensions, following standard statistical guidelines (Cohen, 1988). This approach ensures that our benchmark items represent their intended value constructs while maintaining methodological rigor."

The benchmark items are defined by filtering query-response pairs on the criterion |Spearman r| ≥ 0.3 with a PVQ-21 dimension (§3.4.1: 'for each value dimension, we retain items with correlations of at least 0.3'). The validity claim then cites precisely that same filter as the evidence: 'we only keep items that have meaningful correlations... This approach ensures that our benchmark items represent their intended value constructs.' Selection criterion and validity evidence are the same statistic, so the validity statement is true by construction rather than by independent confirmation.

full rationale

The central LLM evaluation chain is not circular: human PVQ scores are external; items are selected by human correlations; LLM similarity ratings are new data; value scores are normalized means of those ratings; demographic comparisons are benchmarked against independent ESS human data; steering results are behavioral interventions measured with the same instrument. No fitted parameter is renamed as a prediction, and the paper does not fit LLM scores to human targets. No load-bearing self-citation appears: the methodological references (Schwartz, Davidov, PVQ, BFI) are external, and the authors do not cite their own prior results to justify the framework. The one tautological element is the use of the item-selection correlation threshold as validity evidence in §3.4.2; that is a psychometric overclaim but does not propagate into the LLM value measurements, which remain independent observations. The transfer of human-validated items to LLMs is an unverified assumption (the paper itself reports GPT-4o's value-targeted generation aligned only 11.25% of the time and steering Benevolence raised Security more than Benevolence), but an assumption about construct validity is not circularity. Overall score 2: one minor self-validating step, while the central claims retain independent empirical content.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central measurement rests on the transfer of human-validated items to LLMs and on treating LLM self-reports as value measurements; these are domain assumptions with no independent falsifiable handle in the paper. No new physical or theoretical entities are introduced.

free parameters (4)

Item inclusion correlation threshold = |r| >= 0.3, p < 0.05
Items are retained only if their human-rating/PVQ correlation exceeds this threshold; the threshold is standard but arbitrary, and no multiple-comparison correction is applied across 7,800 correlations, so the final item set depends on this choice.
Number of prompts per evaluation = 6
LLM responses are averaged over six prompt variants (three templates plus reversed option order); the choice affects measured scores.
Number of responses per query = 5
Five GPT-4o responses are generated per query; the diversity-focused generation prompt was selected after the value-targeted approach yielded only 11.25% alignment.
Per-source query allocation = 26 queries per source (104 total)
The target query count was set for feasibility and then allocated equally across four sources; filtering thresholds (e.g., minimum 7 relevant values for ShareGPT, 10 for LMSYS) were calibrated to reach this target.

assumptions (5)

domain assumption An LLM's self-rating on 'How similar is this response to your own thoughts?' is a valid measurement of its value orientation.
The evaluation framework (Section 3.4.1) asks LLMs the same question human participants answered; the paper provides no independent validation that LLM self-reports correspond to stable model values.
domain assumption Items validated on human participants are valid for assessing LLMs.
The benchmark selects items using human PVQ correlations, then applies them to LLMs; cross-population validity is assumed rather than tested.
domain assumption GPT-4o-generated responses are ecologically representative of real user-LLM interactions.
Responses are generated by GPT-4o with diversity-focused prompts (Section 3.2.2) rather than collected from actual users, so the response distribution reflects GPT-4o's style and content biases.
standard math PVQ-21 and BFI-10 are valid instruments for the Prolific participant population.
The paper relies on the established validity of these instruments; this is a standard psychometric assumption.
domain assumption The European Social Survey provides an appropriate human baseline for demographic bias comparisons.
Demographic bias analysis (Section 4.2) compares GPT-4o persona responses with ESS data from 37,498 participants; ESS sampling and value measurement are assumed comparable to the Prolific-based item validation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Value Portrait: Assessing Language Models' Values through Psychometrically and Ecologically Valid Items." pith.science (2026). https://pith.science/paper/366WAVIZ

@misc{pith2026250501015,
  author       = {Pith},
  title        = {Pith review of: Value Portrait: Assessing Language Models' Values through Psychometrically and Ecologically Valid Items},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/366WAVIZ}},
  note         = {Machine review of arXiv:2505.01015}
}

read the original abstract

The importance of benchmarks for assessing the values of language models has been pronounced due to the growing need of more authentic, human-aligned responses. However, existing benchmarks rely on human or machine annotations that are vulnerable to value-related biases. Furthermore, the tested scenarios often diverge from real-world contexts in which models are commonly used to generate text and express values. To address these issues, we propose the Value Portrait benchmark, a reliable framework for evaluating LLMs' value orientations with two key characteristics. First, the benchmark consists of items that capture real-life user-LLM interactions, enhancing the relevance of assessment results to real-world LLM usage. Second, each item is rated by human subjects based on its similarity to their own thoughts, and correlations between these ratings and the subjects' actual value scores are derived. This psychometrically validated approach ensures that items strongly correlated with specific values serve as reliable items for assessing those values. Through evaluating 44 LLMs with our benchmark, we find that these models prioritize Benevolence, Security, and Self-Direction values while placing less emphasis on Tradition, Power, and Achievement values. Also, our analysis reveals biases in how LLMs perceive various demographic groups, deviating from real human data.

Figures

Figures reproduced from arXiv: 2505.01015 by the authors.

**Figure 1.** Comparison of previous annotation methods with our annotation method. While existing works rely on identifying annotators’ perceived values in text, Value Portrait adopts a more psychometrically rigorous approach. has incorporated value frameworks and personality theories into LLMs to enhance their ability to generate more authentic, human-aligned responses (Caron and Srivastava, 2023; Rozen et al., 2024). Consequ… view at source ↗

**Figure 2.** Overview of the Value Portrait dataset construction pipeline consists of three phases: [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Evaluation framework overview for LLM value orientations [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Value orientation comparison across LLMs. In the radial plot, the dashed concentric circle represents the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Value orientations across age groups (20-39, 40-59, 60-79, 80+) in human data (left) and persona [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: Value orientations across gender groups in human data and persona-prompted GPT-4o responses. [PITH_FULL_IMAGE:figures/full_fig_p034_6.png]

**Figure 7.** Figure 7: Value orientations across political orientation groups in human data and persona-prompted GPT-4o [PITH_FULL_IMAGE:figures/full_fig_p034_7.png]

**Figure 8.** Figure 8: Value orientations across education levels in human data and persona-prompted GPT-4o responses. [PITH_FULL_IMAGE:figures/full_fig_p034_8.png]

**Figure 9.** Figure 9: Screenshot of the page annotators see on the Prolific platform. [PITH_FULL_IMAGE:figures/full_fig_p040_9.png]

**Figure 10.** Figure 10: Screenshot of our human survey [PITH_FULL_IMAGE:figures/full_fig_p041_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

When Counterbalancing Hides the Bias: Access-Conditioned Position Lock in Forced-Choice LLM Evaluation
cs.LG 2026-07 conditional novelty 7.0 of 10

Cross-model value distances from single draws are inflated by response determinism and confounded by the deployment client; a repeated counterbalanced protocol plus flip/magnitude decomposition separates them.

Reference graph

Works this paper leans on

14 extracted references · 10 canonical work pages · cited by 1 Pith paper

[1]

Your answers should naturally reflect this value and its meaning

work page
[2]

Express your thoughts authentically without directly mentioning the value name or its definition

work page
[3]

Response lengths should be between 10 to 30 words

The response shouldn't be too long. Response lengths should be between 10 to 30 words. Restrictions: - Never explicitly state {value.name} - Never directly quote the definition Approach A: Value-Targeted Generation Theoretical Distance Same-Direction Pairs Opposite-Direction Pairs Count Percentage Count Percentage 1 (Adjacent) 9561.69% 19 6.55% 2 35 22.73...

work page
[4]

Pouya Pezeshkpour and Estevam Hruschka

Training language models to follow instruc- tions with human feedback.Advances in neural information processing systems, 35:27730–27744. Pouya Pezeshkpour and Estevam Hruschka. 2024. Large language models sensitivity to the order of options in multiple-choice questions. InFindings of the Association for Computational Linguistics: NAACL 2024, pages 2006–20...

arXiv 2024
[5]

go to work; you’ve got to get out of the house and do your own thing

Value kaleidoscope: engaging ai with plu- ralistic human values, rights, and duties. InPro- ceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’24/IAAI’24/EAAI’24. AAAI Pre...

arXiv 2024
[9]

Analyze the scenario or question provided

work page
[10]

Generate five distinct responses, each with unique viewpoints

work page
[11]

Power”-oriented response to

Ensure responses vary in their degree of appeal Notes: - Responses should be distinct in perspective - Consider controversial or polarizing angles - Make responses feel realistic and relatable Approach B: Diversity-Focused Generation Empirical evaluation of Approach A, conducted across 80 generated items, revealed significant limitations in achieving targ...

work page

Show all 14 references

[12]

Size Scaling Across Model Families.The size scaling patterns observed in Llama models ex- tend consistently across Qwen2.5, DeepSeek-R1- Distill-Qwen, and Gemma3 families

These patterns align with our observations in GPT reasoning models, suggesting that enhanced reasoning capabilities consistently amplify ethical value orientations across different model architec- tures. Size Scaling Across Model Families.The size scaling patterns observed in ...

2024
[13]

License Grant: A limited, non-exclusive, non-transferable, non-sublicensable license for research, development, and improvement of software, algorithms, and machine learning models for both research and commercial purposes
[14]

ground truth

Key Compliance Requirements: Safety and Moderation: Implementation of appropriate filters and safety measures Non-Identification: Prohibition of attempts to identify individuals or infer sensitive personal data Prohibited Transfers: No distribution, copying, disclosure, or tra...
[2008]

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z

Bringing values back in: The adequacy of the european social survey to measure values in 20 coun- tries.Public Opinion Quarterly, 72(3):420–445. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, ...

2025 arXiv
[2022]

InProceedings of the Fifth Workshop on Natural Language Process- ing and Computational Social Science (NLP+CSS), pages 218–227, Abu Dhabi, UAE

Who is GPT-3? an exploration of personal- ity, values and demographics. InProceedings of the Fifth Workshop on Natural Language Process- ing and Computational Social Science (NLP+CSS), pages 218–227, Abu Dhabi, UAE. Association for Computational Linguistics. Nailia Mirzakhmedo...

2024
[2024]

Yuan Li, Yue Huang, Hongyi Wang, Xiangliang Zhang, James Zou, and Lichao Sun

Do llms have distinct and consistent personal- ity? trait: Personality testset designed for llms with psychometrics.arXiv preprint arXiv:2406.14703. Yuan Li, Yue Huang, Hongyi Wang, Xiangliang Zhang, James Zou, and Lichao Sun. 2024. Quantifying ai psychology: A psychometrics b...

2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Your answers should naturally reflect this value and its meaning

work page

[2] [2]

Express your thoughts authentically without directly mentioning the value name or its definition

work page

[3] [3]

Response lengths should be between 10 to 30 words

The response shouldn't be too long. Response lengths should be between 10 to 30 words. Restrictions: - Never explicitly state {value.name} - Never directly quote the definition Approach A: Value-Targeted Generation Theoretical Distance Same-Direction Pairs Opposite-Direction Pairs Count Percentage Count Percentage 1 (Adjacent) 9561.69% 19 6.55% 2 35 22.73...

work page

[4] [4]

Pouya Pezeshkpour and Estevam Hruschka

Training language models to follow instruc- tions with human feedback.Advances in neural information processing systems, 35:27730–27744. Pouya Pezeshkpour and Estevam Hruschka. 2024. Large language models sensitivity to the order of options in multiple-choice questions. InFindings of the Association for Computational Linguistics: NAACL 2024, pages 2006–20...

arXiv 2024

[5] [5]

go to work; you’ve got to get out of the house and do your own thing

Value kaleidoscope: engaging ai with plu- ralistic human values, rights, and duties. InPro- ceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’24/IAAI’24/EAAI’24. AAAI Pre...

arXiv 2024

[6] [9]

Analyze the scenario or question provided

work page

[7] [10]

Generate five distinct responses, each with unique viewpoints

work page

[8] [11]

Power”-oriented response to

Ensure responses vary in their degree of appeal Notes: - Responses should be distinct in perspective - Consider controversial or polarizing angles - Make responses feel realistic and relatable Approach B: Diversity-Focused Generation Empirical evaluation of Approach A, conducted across 80 generated items, revealed significant limitations in achieving targ...

work page