Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Value Compass Benchmarks: A Platform for Fundamental and Validated Evaluation of LLMs Values

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Value Compass Benchmarks is a dynamic, online platform for diagnosing LLM values through generative, self-evolving tests.

desk verdict A genuinely useful platform for LLM value comparison, but the paper's validity claim rests on an unvalidated closed loop that needs human-judgment evidence before the scores can be taken at face value. read the letter →

arxiv 2501.07071 v3 pith:CFPH6RZJ submitted 2025-01-13 cs.AI

Jing Yao , Xiaoyuan Yi , Shitong Duan , Jindong Wang , Yuzhuo Bai , Muhua Huang , Peng Zhang , Tun Lu
show 3 more authors
Zhicheng Dou Maosong Sun Xing Xie
This is my paper · ORCID
classification cs.AI
keywords LLMvalueevaluationalignmentgenerativeself-evolvingbenchmarkpluralismculturalbasichumanvaluesleaderboard
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that existing LLM value benchmarks fail on validity grounds: static, discriminative questions measure what models know about values rather than how they behave, and they become uninformative as models improve. To fix this, it presents Value Compass Benchmarks, an online platform that automatically generates fresh test items, scores open-ended responses through a value recognizer, and updates the items as new models are released. The platform reports fine-grained, behavior-based value scores for 33 leading LLMs across 27 dimensions and adds tools to compare models, weight scores by personal priorities, and see cultural alignment. If the approach works, value evaluation becomes a living diagnostic of behavioral conformity rather than a one-dimensional knowledge test.

What carries the argument

The load-bearing mechanism is the two-term item-generation objective in Eq. (1), which trains the generator to produce test items that both elicit values, by maximizing mutual information between responses and the value-dimension vector, and remain informative, by maximizing divergence among the evaluated models' predicted value distributions. The recognizer CLAVE supplies the value-probability estimates inside that objective and again when scoring responses, so the same function carries both item selection and the final behavioral score. A model's score on a value dimension is the expected recognizer output over generated items and sampled responses, which is what makes the evaluation behavioral rather than knowledge-based.

What would settle it

Sample a set of open-ended responses from the 33 evaluated LLMs to freshly generated items, have independent human annotators label the value dimensions expressed, and compare their labels with CLAVE's; if agreement is low or if replacing CLAVE with a human-validated recognizer changes the model rankings, the platform's validity claim fails.

Watch

Extended reading notes

Core claim

Value Compass Benchmarks is presented as the first dynamic, online and interactive platform devoted to comprehensive value diagnosis of LLMs, and the paper's central claim is that generative, self-evolving evaluation is more valid than static discriminative benchmarks. Instead of asking models to select the value-aligned answer, the platform generates novel, value-evoking scenarios, samples each model's open-ended response, and uses a hybrid value recognizer (CLAVE) to estimate the value distribution expressed by that behavior. The value score for a model on a dimension is the expected recognizer output over generated items, and the item generator is re-optimized at Eq. (1) whenever newer models arrive, balancing value elicitation against informativeness. The platform reports fine-grained scores across 27 dimensions from four value systems, supports user-weighted aggregation through social welfare functions, and maps each model onto cultural value vectors for alignment analysis.

Load-bearing premise

The entire pipeline assumes that the value recognizer CLAVE labels open-ended responses as reliably as a human would, because the same recognizer is used both to choose which test items to generate and to compute the final value scores.

Editorial extensions

If this is right

  • A model that can recite the value-aligned answer but does not act on it in realistic scenarios will receive a low behavioral score, closing the knowledge-behavior gap.
  • The benchmark regenerates test items as new LLMs appear, so evaluations stay informative and contaminated or saturated items are replaced.
  • Users can supply their own weights over value dimensions through social welfare functions, making the best model relative to a person's priorities instead of a single average.
  • Cultural alignment analysis reveals which documented cultural value vectors each model resembles, giving developers a concrete target for cultural adaptation.
  • Fine-grained scores across 27 dimensions support case-level diagnosis of specific misalignments, rather than only an overall safety ranking.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: Because the same recognizer CLAVE both selects test items and computes final scores, re-running the platform with a recognizer whose labels are validated against human judgments would be a direct robustness check; the reported model differences are partly a property of the recognizer.
  • Inference: The self-evolving generative design could transfer to other pluralistic targets, such as political values, professional ethics codes, or organization-specific value statements, where static questionnaires face the same contamination and ceiling problems.
  • Inference: Cultural alignment maps built from survey-reported value vectors could double as a training-data diagnostic, since a model trained largely on one region's text should approximate that region's value profile unless alignment training shifts it.
  • Inference: The case studies imply value scores track observable behavioral differences; adding high-stakes or adversarial scenarios would test whether those differences persist when a model is under pressure.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents Value Compass Benchmarks, an online platform for evaluating the values of 33 LLMs across four value systems (Schwartz Theory of Basic Values, Moral Foundation Theory, an LLM-specific value system, and a safety taxonomy). Its main technical contribution is a generative self-evolving evaluation paradigm: an item generator is optimized via Eq. (1) to produce test scenarios that maximize divergence in the value distributions recognized by CLAVE, and value scores are computed by feeding model responses to the same recognizer. The platform provides fine-grained scores, customized comparisons, and cultural-alignment visualizations. Section 3 reports quantitative comparisons, case studies, and a 15-participant user study.

Significance. The paper addresses a genuine gap: existing value benchmarks are mostly static and discriminative, and the proposed combination of pluralistic value systems, dynamically generated scenarios, and an interactive dashboard is a useful engineering contribution. If the scores were shown to be valid, the platform would be valuable for monitoring alignment and cultural fit of LLMs. Strengths include a public online leaderboard covering 33 recent models, transparent multi-dimensional reporting, external grounding in social-science value surveys for cultural alignment, and a fully specified pipeline that is easy to reproduce from Eq. (1). The main weakness is that the validity evidence is currently not sufficient.

major comments (3)
  1. [Section 2.2, Eq. (1) and the scoring formula] The same recognizer F (CLAVE) is used both to select test items and to compute final value scores. Specifically, p_i(v|x) in Eq. (1) is estimated as E_{y~p_i(y|x)}[p_F(v|x,y)], and the reported score s^v_i is E_{x~X_v, y~p_i(y|x)}[F(x,y)]. The manuscript contains no human agreement study for CLAVE on the generated items, no comparison against an independent recognizer, and no end-to-end comparison of the platform's scores with human judgments for the 33 evaluated models. Without such validation, systematic misreadings by CLAVE are amplified by item selection and then reported as genuine value differences; the central validity claim is therefore not established.
  2. [Section 3, Fig. 4] The evidence that generative evaluation is more valid than discriminative evaluation is not statistically supported. The figure shows qualitative differences in scores and the text interprets them as 'overestimation' and 'vulnerabilities', but lower or more dispersed scores do not by themselves show that the generative scores are more accurate; they could reflect recognizer bias or differences in prompt difficulty. No error bars, confidence intervals, or significance tests are reported, and the comparisons use only two models in Fig. 4(a) and four in Fig. 4(b). The claim that the platform yields 'valid' scores for 33 LLMs would need a criterion-validity analysis, for example against human ratings on a sample of generated items.
  3. [Section 3, user study] The 15-participant study measures perceived usefulness, informativeness, and usability (SUS), not whether the value scores are correct. The participants were not asked to judge the accuracy of the scores on held-out items, and the results are reported as means without confidence intervals. The user study can support a claim about usability, but it should not be cited as evidence for the validity of the value measurements; a validation study with human labels and more participants is needed.
minor comments (5)
  1. [Title and abstract] The abstract and the full text present inconsistent title variants: the reader-facing title mentions 'Fundamental and Validated Evaluation', while the full text begins with 'A Comprehensive, Generative and Self-Evolving Platform'. Please align the title across versions.
  2. [Section 1] The phrase 'the first dynamic, online and interactive platform' is an overclaim: dynamic evaluation platforms and leaderboards exist (e.g., DyVal, LMSYS Chatbot Arena). Suggest softening to 'a dynamic, online and interactive platform'.
  3. [Section 2.2, Eq. (1)] Equation (1) depends on a hyperparameter α and on the number of response samples per item, but no sensitivity analysis or default-value justification is given. Please report how α was set and whether the ranking is stable under reasonable choices.
  4. [Appendix A] Minor typos: 'we presents' in the abstract, 'demdominance' and 'encomapesses' in Appendix A. The Power definition should read 'dominance'.
  5. [Section 3, Fig. 4] Figure 4 would benefit from error bars and a description of how many items and samples were used; without that, the visual gap between static and evolving items cannot be assessed.

Circularity Check

2 steps flagged · score 6.0 of 10

Informativeness of the generated test items is circular: Eq. (1) maximizes recognizer-based divergence and the paper then reports that divergence as evidence; CLAVE's reliability is load-bearing and justified only by a self-citation.

  1. fitted input called prediction [Section 2.2 (Self-Evolving Item Generator, Eq. 1) and Section 3 (Quantitative Analysis, Fig. 4b)]
    "The first term in Eq. (1) exploits x that maximally captures value differences of LLMs ... with pi(v|x)≈E pi(y|x)[pF (v|x,y)] ... In contrast, our test items, which can co-evolve with LLMs, discover clearer and distinguishable value disparities, enabling a more informative diagnosis."

    Eq. (1) optimizes the item generator q_theta(x) to maximize D[p_1(v|x),...,p_M(v|x)], where each p_i(v|x) is estimated through the value recognizer F. Section 3 then presents the fact that the generated items yield clearer, distinguishable disparities under F as evidence that the items are more informative. That outcome is forced by the optimization objective: items are selected precisely to maximize F-based separation among models. The same F is also used to compute the final scores (s^v_i≈E_xE_y[F(x,y)]), so the recognizer both constructs the test and measures the result. No human-label agreement or third-party recognizer check on the generated open-ended items is reported, so the claimed informativeness is not validated against an independent criterion.

  2. self citation load bearing [Section 2.2 (Adaptive and Robust Value Recognizer)]
    "Therefore, we apply CLA VE (Yao et al., 2024), a hybrid value recognizer in our benchmarks. CLA VE leverages a large LLM with satisfactory robustness to identify generalized and representative value concepts from varied and subtle value expression ... Then, it fine-tunes a smaller LLM to effectively adapt to and recognize specific and diverse values based on the identified concepts."

    The reliability of CLAVE is the load-bearing premise for both item selection (Eq. 1 uses p_F) and scoring (final scores average F). The paper justifies this premise by citing Yao et al. 2024, a prior paper by present co-authors, and does not report a human-agreement study, an independent recognizer comparison, or an end-to-end correlation of CLAVE-based scores with human judgments for the 33 models. The reported user study (n=15) asks about perceived usefulness and the case studies are illustrative, neither of which tests recognizer accuracy. Thus the claimed 'valid and informative' value scores rest on a self-citation that is not independently verified in this manuscript.

full rationale

The paper is not circular in its value-system grounding: the Schwartz and Moral Foundation taxonomies, the safety taxonomy, and the social-welfare aggregation are imported from external literatures, and the platform's interface/visualization components are genuine contributions. The core evaluative loop, however, is circular in its informativeness argument. Eq. (1) selects test items by maximizing the divergence of recognizer-computed value score distributions across models; the paper's Fig. 4(b)/Section 3 then treats the resulting larger divergences as evidence that the generative self-evolving items are more informative, which is true by construction. Additionally, the 'Adaptive and Robust Value Recognizer' (CLAVE) is adopted from the authors' own prior work (Yao et al., 2024), and the same recognizer is used both to generate/select items and to produce final scores. No human validation of CLAVE on the generated items is reported, so the validity claim is not independently established. These two issues together make the central claim partially circular, warranting a score of 6 rather than lower; it is not a full 8-10 because the leaderboard scores are still empirical outputs of a deployed system, the value dimensions are externally grounded, and the platform has independent usability content.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The evaluation rests on social science value theories, a recognizer whose accuracy is assumed, culture survey data, and the item-generation objective; the recognizer and item generator form a closed loop that is not checked against external labels. No new theoretical entities are postulated; the item generator and recognizer are adopted from the authors' prior work (Jiang et al., 2024; Yao et al., 2024).

free parameters (2)
  • alpha (α) in Eq. (1)
    Trade-off between informativeness maximization and value elicitation in the item generator objective; no value or sensitivity analysis is provided, yet it controls item selection and hence the measured value scores.
  • Number of response samples per item
    Scores are defined as expectations over sampled responses, but the number of samples used to estimate the expectation is not reported, leaving variance uncontrolled.
assumptions (4)
  • domain assumption Schwartz Theory of Basic Values and Moral Foundation Theory are valid, sufficiently comprehensive models of human values.
    Section 2.1 grounds the evaluation in these social science theories without discussing their known limitations or the implications for score interpretation.
  • domain assumption The value recognizer CLAVE maps open-ended responses to value dimensions accurately enough for closed-loop item selection and scoring.
    Section 2.2 relies on CLAVE for both p_i(v|x) inside Eq. (1) and for computing final scores; no validation against human labels is given in this paper.
  • domain assumption Cultures' value vectors from ESS/WVS (e.g., UK, China, US) adequately represent pluralistic cultural values for correlation analysis.
    Section 2.3 uses these survey values as fixed cultural references without discussing sampling or comparability.
  • domain assumption Open-ended sampled responses under generated items reflect the model's values rather than prompt artifacts or sycophancy.
    The generative evaluation assumes behavioral conformity can be read off single-turn responses; the paper itself cites sycophancy literature but does not control for it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Value Compass Benchmarks: A Platform for Fundamental and Validated Evaluation of LLMs Values." pith.science (2026). https://pith.science/paper/CFPH6RZJ

@misc{pith2026250107071,
  author       = {Pith},
  title        = {Pith review of: Value Compass Benchmarks: A Platform for Fundamental and Validated Evaluation of LLMs Values},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CFPH6RZJ}},
  note         = {Machine review of arXiv:2501.07071}
}
read the original abstract

As Large Language Models (LLMs) achieve remarkable breakthroughs, aligning their values with humans has become imperative for their responsible development and customized applications. However, there still lack evaluations of LLMs values that fulfill three desirable goals. (1) Value Clarification: We expect to clarify the underlying values of LLMs precisely and comprehensively, while current evaluations focus narrowly on safety risks such as bias and toxicity. (2) Evaluation Validity: Existing static, open-source benchmarks are prone to data contamination and quickly become obsolete as LLMs evolve. Additionally, these discriminative evaluations uncover LLMs' knowledge about values, rather than valid assessments of LLMs' behavioral conformity to values. (3) Value Pluralism: The pluralistic nature of human values across individuals and cultures is largely ignored in measuring LLMs value alignment. To address these challenges, we presents the Value Compass Benchmarks, with three correspondingly designed modules. It (i) grounds the evaluation on motivationally distinct \textit{basic values to clarify LLMs' underlying values from a holistic view; (ii) applies a \textit{generative evolving evaluation framework with adaptive test items for evolving LLMs and direct value recognition from behaviors in realistic scenarios; (iii) propose a metric that quantifies LLMs alignment with a specific value as a weighted sum over multiple dimensions, with weights determined by pluralistic values.

Figures

Figures reproduced from arXiv: 2501.07071 by the authors.

Figure 1
Figure 1. Two challenges of LLM value evaluation. society, they may also pose negative impacts on humans, such as generating harmful (Weidinger et al., 2021; Bengio et al., 2024) (violating univer￾sal values) or culturally dispreferred content (Ma￾soud et al., 2025; Wu et al., 2025) (cultural values). Comprehensively assessing these problems (Chi￾ang et al., 2024; Zhang et al., 2024) is crucial for revealing LLMs’ potential m… view at source ↗
Figure 2
Figure 2. The overall architecture of Value Compass Benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Usage demonstration of Value Compass Benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Case study of value-behavior correlation. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 4
Figure 4. Figure 4: (a) Comparison between discriminative (judg [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 7
Figure 7. Figure 7: Introduction along with intuitive examples for each value system is available on our Value Compass [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Detailed results and the questionnaire for user study. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Case study of value-behavior correlation. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectives

    cs.AI 2025-06 conditional novelty 4.0 of 10

    A survey proposes a macro-meso-micro value framework for agentic AI alignment and maps applications, methods, and benchmarks onto it.

  2. Domain Specific Benchmarks for Evaluating Multimodal Large Language Models

    cs.LG 2025-06 conditional novelty 3.0 of 10

    A review paper that organizes domain-specific MLLM benchmarks into an eight-discipline taxonomy, with summary tables and performance highlights.

Reference graph

Works this paper leans on

88 extracted references · 23 canonical work pages · cited by 2 Pith papers

  1. [1]

    Marwa Abdulhai, Gregory Serapio-Garcia, Cl \'e ment Crepy, Daria Valter, John Canny, and Natasha Jaques. 2023. Moral foundations of large language models. arXiv preprint arXiv:2310.15337

    arXiv 2023

  2. [2]

    Badr Alkhamissi, Muhammad ElNokrashy, Mai Alkhamissi, and Mona Diab. 2024. Investigating cultural alignment of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12404--12422

    2024

  3. [3]

    Kenneth J Arrow. 2012. Social choice and individual values, volume 12. Yale university press

    2012

  4. [4]

    Xuechunzi Bai, Angelina Wang, Ilia Sucholutsky, and Thomas L Griffiths. 2024. Measuring implicit bias in explicitly unbiased large language models. arXiv preprint arXiv:2402.04105

    arXiv 2024

  5. [5]

    Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Trevor Darrell, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, et al. 2024. Managing extreme ai risks amid rapid progress. Science, 384(6698):842--845

    2024

  6. [6]

    Lo \" c Berger and Johannes Emmerling. 2020. Welfare as equity equivalents. Journal of Economic Surveys, 34(4):727--752

    2020

  7. [7]

    Pablo Biedma, Xiaoyuan Yi, Linus Huang, Maosong Sun, and Xing Xie. 2024. Beyond human norms: Unveiling unique values of large language models through interdisciplinary approaches. arXiv preprint arXiv:2404.12744

    arXiv 2024

  8. [8]

    Peter R Blake, Katherine McAuliffe, and Felix Warneken. 2014. The developmental origins of fairness: The knowledge--behavior gap. Trends in cognitive sciences, 18(11):559--561

    2014

Show all 88 references
  1. [9]

    Mark J Brandt. 2017. Predicting ideological prejudice. Psychological Science, 28(6):713--722

    2017

  2. [10]

    John Brooke et al. 1996. Sus-a quick and dirty usability scale. Usability evaluation in industry, 189(194):4--7

    1996

  3. [11]

    S \'e bastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712

    2023 arXiv

  4. [12]

    David Cecchini, Arshaan Nazir, Kalyan Chakravarthy, and Veysel Kocaman. 2024. Holistic evaluation of large language models: Assessing robustness, accuracy, and toxicity for real-world applications. In Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (...

    2024

  5. [13]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1--45

    2024

  6. [14]

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conf...

    2024

  7. [15]

    Sooyung Choi, Xiaoyuan Yi, Jing Yao, Xing Xie, and JinYeong Bak. 2024. Why do you answer like that? psychological analysis on underlying connections between llm's values and safety risks

    2024

  8. [16]

    Aida Davani, Mark D \' az, Dylan Baker, and Vinodkumar Prabhakaran. 2024. Disentangling perceptions of offensiveness: Cultural and moral correlates. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 2007--2021

    2024

  9. [17]

    Boele De Raad. 2000. The big five personality factors: the psycholexical approach to personality. Hogrefe & Huber Publishers

    2000

  10. [18]

    Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. 2024. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. arXiv preprint arXiv:2402.15938

    2024 arXiv

  11. [19]

    Shitong Duan, Xiaoyuan Yi, Peng Zhang, Tun Lu, Xing Xie, and Ning Gu. 2023. Denevil: Towards deciphering and navigating the ethical values of large language models via instruction learning. arXiv preprint arXiv:2310.11053

    2023 arXiv

  12. [20]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  13. [21]

    Lizhou Fan, Wenyue Hua, Lingyao Li, Haoyang Ling, and Yongfeng Zhang. 2023. Nphardeval: Dynamic benchmark on reasoning ability of large language models via complexity classes. arXiv preprint arXiv:2312.14890

    2023 arXiv

  14. [22]

    Kathleen C Fraser, Svetlana Kiritchenko, and Esma Balkir. 2022. Does moral code have a moral code? probing delphi's moral philosophy. arXiv preprint arXiv:2205.12771

    2022 arXiv

  15. [23]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462

    2020 arXiv

  16. [24]

    Jesse Graham, Jonathan Haidt, Sena Koleva, Matt Motyl, Ravi Iyer, Sean P Wojcik, and Peter H Ditto. 2013. Moral foundations theory: The pragmatic validity of moral pluralism. In Advances in experimental social psychology, volume 47, pages 55--130. Elsevier

    2013

  17. [25]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  18. [26]

    Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2020. Aligning ai with shared human values. arXiv preprint arXiv:2008.02275

    2020 arXiv

  19. [27]

    Nien-h \^e Hsieh and Henrik Andersson. 2007. Incommensurable values

    2007

  20. [28]

    Kexin Huang, Xiangyang Liu, Qianyu Guo, Tianxiang Sun, Jiawei Sun, Yaru Wang, Zeyang Zhou, Yixu Wang, Yan Teng, Xipeng Qiu, Yingchun Wang, and Dahua Lin. 2024 a . https://doi.org/10.18653/v1/2024.naacl-long.256 Flames: Benchmarking value alignment of LLM s in C hinese . In Pro...

    2024 doi

  21. [29]

    Kexin Huang, Xiangyang Liu, Qianyu Guo, Tianxiang Sun, Jiawei Sun, Yaru Wang, Zeyang Zhou, Yixu Wang, Yan Teng, Xipeng Qiu, et al. 2024 b . Flames: Benchmarking value alignment of llms in chinese. In Proceedings of the 2024 Conference of the North American Chapter of the Assoc...

    2024

  22. [30]

    Yue Huang, Qihui Zhang, Lichao Sun, et al. 2023. Trustgpt: A benchmark for trustworthy and responsible large language models. arXiv preprint arXiv:2306.11507

    2023 arXiv

  23. [31]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  24. [32]

    Jianchao Ji, Yutong Chen, Mingyu Jin, Wujiang Xu, Wenyue Hua, and Yongfeng Zhang. 2024. Moralbench: Moral evaluation of llms. arXiv preprint arXiv:2406.04428

    2024 arXiv

  25. [33]

    Han Jiang, Xiaoyuan Yi, Zhihua Wei, Shu Wang, and Xing Xie. 2024. Raising the bar: Investigating the values of large language models via generative evolving testing. arXiv preprint arXiv:2406.14230

    2024 arXiv

  26. [34]

    Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, et al. 2021. Can machines learn morality? the delphi experiment. arXiv preprint arXiv:2110.07574

    2021 arXiv

  27. [35]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

    2020 arXiv

  28. [36]

    JE (Hans) Korteling, Geertje C van de Boer-Visschedijk, Romy AM Blankendaal, Rob C Boonekamp, and A Roos Eikelboom. 2021. Human-versus artificial intelligence. Frontiers in artificial intelligence, 4:622364

    2021

  29. [37]

    Shanghai AI Lab. 2024. Opencompass. https://rank.opencompass.org.cn/home

    2024

  30. [38]

    Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. 2024. Salad-bench: A hierarchical and comprehensive safety benchmark for large language models. arXiv preprint arXiv:2402.05044

    2024 arXiv

  31. [39]

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110

    2022 arXiv

  32. [40]

    Robert W Lissitz and Karen Samuelsen. 2007. A suggested change in terminology and emphasis regarding validity and education. Educational researcher, 36(8):437--448

    2007

  33. [41]

    Elinor Mason. 2006. Value pluralism

    2006

  34. [42]

    Reem Masoud, Ziquan Liu, Martin Ferianc, Philip C Treleaven, and Miguel Rodrigues Rodrigues. 2025. Cultural alignment in large language models: An explanatory analysis based on hofstede’s cultural dimensions. In Proceedings of the 31st International Conference on Computational...

    2025

  35. [43]

    Reem I Masoud, Ziquan Liu, Martin Ferianc, Philip Treleaven, and Miguel Rodrigues. 2023. Cultural alignment in large language models: An explanatory analysis based on hofstede's cultural dimensions. arXiv preprint arXiv:2309.12342

    2023 arXiv

  36. [44]

    Timothy R McIntosh, Teo Susnjak, Tong Liu, Paul Watters, and Malka N Halgamuge. 2024. Inadequacies of large language model benchmarks in the era of generative artificial intelligence. arXiv preprint arXiv:2402.09880

    2024 arXiv

  37. [45]

    Gwenyth Isobel Meadows, Nicholas Wai Long Lau, Eva Adelina Susanto, Chi Lok Yu, and Aditya Paul. 2024. Localvaluebench: A collaboratively built and extensible benchmark for evaluating localized value alignment and ethical safety in large language models. arXiv preprint arXiv:2...

    2024 arXiv

  38. [46]

    Yutao Mou, Shikun Zhang, and Wei Ye. 2024. Sg-bench: Evaluating llm safety generalization across diverse tasks and prompt types. arXiv preprint arXiv:2410.21965

    2024 arXiv

  39. [47]

    Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R Bowman. 2020. Crows-pairs: A challenge dataset for measuring social biases in masked language models. arXiv preprint arXiv:2010.00133

    2020 arXiv

  40. [48]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  41. [49]

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R Bowman. 2021. Bbq: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193

    2021 arXiv

  42. [50]

    Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. 2023. Discovering language model behaviors with model-written evaluations. In Findings of the Association for Computation...

    2023

  43. [51]

    Richard H Pildes and Elizabeth S Anderson. 1990. Slinging arrows at democracy: Social choice theory, value pluralism, and democratic politics. Colum. L. Rev., 90:2121

    1990

  44. [52]

    Yuanyi Ren, Haoran Ye, Hanjun Fang, Xin Zhang, and Guojie Song. 2024. Valuebench: Towards comprehensively evaluating value orientations and understanding of large language models. arXiv preprint arXiv:2406.04214

    2024 arXiv

  45. [53]

    Lilach Sagiv, Sonia Roccas, Jan Cieciuch, and Shalom H Schwartz. 2017. Personal values in human life. Nature human behaviour, 1(9):630--639

    2017

  46. [54]

    Jeff Sauro and James R Lewis. 2016. Quantifying the user experience: Practical statistics for user research. Morgan Kaufmann

    2016

  47. [55]

    Nino Scherrer, Claudia Shi, Amir Feder, and David Blei. 2023. Evaluating the moral beliefs encoded in llms. Advances in Neural Information Processing Systems, 36:51778--51809

    2023

  48. [56]

    Shalom H Schwartz. 2012. An overview of the schwartz theory of basic values. Online readings in Psychology and Culture, 2(1):11

    2012

  49. [57]

    Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Esin DURMUS, Zac Hatfield-Dodds, Scott R Johnston, Shauna M Kravec, et al. 2024. Towards understanding sycophancy in language models. In The Twelfth International Conference on Learning Re...

    2024

  50. [58]

    Gabriel Simmons. 2022. Moral mimicry: Large language models produce moral rationalizations tailored to political identity. arXiv preprint arXiv:2209.12106

    2022 arXiv

  51. [59]

    UC Berkeley SkyLab and LMArena. 2024. Chatbot arena. https://lmarena.ai/?leaderboard

    2024

  52. [60]

    Taylor Sorensen, Liwei Jiang, Jena D Hwang, Sydney Levine, Valentina Pyatkin, Peter West, Nouha Dziri, Ximing Lu, Kavel Rao, Chandra Bhagavatula, et al. 2024 a . Value kaleidoscope: Engaging ai with pluralistic human values, rights, and duties. In Proceedings of the AAAI Confe...

    2024

  53. [61]

    Taylor Sorensen, Liwei Jiang, Jena D Hwang, Sydney Levine, Valentina Pyatkin, Peter West, Nouha Dziri, Ximing Lu, Kavel Rao, Chandra Bhagavatula, et al. 2024 b . Value kaleidoscope: Engaging ai with pluralistic human values, rights, and duties. In Proceedings of the AAAI Confe...

    2024

  54. [62]

    Hao Sun, Zhexin Zhang, Jiawen Deng, Jiale Cheng, and Minlie Huang. 2023. Safety assessment of chinese large language models. arXiv preprint arXiv:2304.10436

    2023 arXiv

  55. [63]

    Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. 2024. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561

    2024 arXiv

  56. [64]

    tatsu lab. 2023. Alpacaeval. https://tatsu-lab.github.io/alpaca_eval

    2023

  57. [65]

    Philip E Tetlock. 1986. A value pluralism model of ideological reasoning. Journal of personality and social psychology, 50(4):819

    1986

  58. [66]

    Naftali Tishby, Fernando C Pereira, and William Bialek. 2000. The information bottleneck method. arXiv preprint physics/0004057

    2000 arXiv

  59. [67]

    Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. 2023 a . Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. arXiv preprint arXiv:2306.11698

    2023 arXiv

  60. [68]

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560

    2022 arXiv

  61. [69]

    Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2023 b . Do-not-answer: A dataset for evaluating safeguards in llms. arXiv preprint arXiv:2308.13387

    2023 arXiv

  62. [70]

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682

    2022 arXiv

  63. [71]

    Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359

    2021 arXiv

  64. [72]

    Shujin Wu, Yi R Fung, Cheng Qian, Jeonghwan Kim, Dilek Hakkani-Tur, and Heng Ji. 2025. Aligning llms with individual preferences via interaction. In Proceedings of the 31st International Conference on Computational Linguistics, pages 7648--7662

    2025

  65. [73]

    Ziang Xiao, Susu Zhang, Vivian Lai, and Q Vera Liao. 2023. Evaluating evaluation metrics: A framework for analyzing nlg evaluation metrics using measurement theory. arXiv preprint arXiv:2305.14889

    2023 arXiv

  66. [74]

    Guohai Xu, Jiayi Liu, Ming Yan, Haotian Xu, Jinghui Si, Zhuoran Zhou, Peng Yi, Xing Gao, Jitao Sang, Rong Zhang, et al. 2023 a . Cvalues: Measuring the values of chinese large language models from safety to responsibility. arXiv preprint arXiv:2307.09705

    2023 arXiv

  67. [75]

    Liang Xu, Kangkang Zhao, Lei Zhu, and Hang Xue. 2023 b . Sc-safety: A multi-round open-ended question adversarial safety benchmark for large language models in chinese. arXiv preprint arXiv:2310.05818

    2023 arXiv

  68. [76]

    Jing Yao, Xiaoyuan Yi, Xiting Wang, Yifan Gong, and Xing Xie. 2023. Value fulcra: Mapping large language models to the multidimensional spectrum of basic human values. arXiv preprint arXiv:2311.10766

    2023 arXiv

  69. [77]

    Jing Yao, Xiaoyuan Yi, and Xing Xie. 2024. Clave: An adaptive framework for evaluating values of llm generated responses. arXiv preprint arXiv:2407.10725

    2024 arXiv

  70. [78]

    Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, et al. 2024 a . R-judge: Benchmarking safety risk awareness for llm agents. arXiv preprint arXiv:2401.10019

    2024 arXiv

  71. [79]

    Xiaohan Yuan, Jinfeng Li, Dongxia Wang, Yuefeng Chen, Xiaofeng Mao, Longtao Huang, Hui Xue, Wenhai Wang, Kui Ren, and Jingyi Wang. 2024 b . S-eval: Automatic and adaptive test generation for benchmarking safety evaluation of large language models. arXiv preprint arXiv:2405.14191

    2024 arXiv

  72. [80]

    Yifan Zeng. 2024. Quantifying risk propensities of large language models: Ethical focus and bias detection through role-play. arXiv preprint arXiv:2411.08884

    2024 arXiv

  73. [81]

    Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2023. Safetybench: Evaluating the safety of large language models with multiple choice questions. arXiv preprint arXiv:2309.07045

    2023 arXiv

  74. [82]

    Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2024. Safetybench: Evaluating the safety of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguist...

    2024

  75. [83]

    Wenlong Zhao, Debanjan Mondal, Niket Tandon, Danica Dillion, Kurt Gray, and Yuling Gu. 2024. Worldvaluesbench: A large-scale benchmark dataset for multi-cultural value awareness of language models. arXiv preprint arXiv:2404.16308

    2024 arXiv

  76. [84]

    Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. 2023. Dyval: Graph-informed dynamic evaluation of large language models. arXiv e-prints, pages arXiv--2309

    2023

  77. [85]

    Caleb Ziems, Jane A Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. 2022. The moral integrity corpus: A benchmark for ethical dialogue systems. arXiv preprint arXiv:2204.03021

    2022 arXiv

  78. [86]

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043

    2023 arXiv

  79. [87]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  80. [88]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.