Pith. sign in

REVIEW 5 major objections 7 minor 100 references

LLM Ethics Benchmark: A Three-Dimensional Assessment System for Evaluating Moral Reasoning in Large Language Models

T0 review · 5 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper reports that LLM moral reasoning can be scored on three axes—human-foundation alignment, reasoning quality, and value consistency—with Claude 3.7 Sonnet ranking highest and LLaMA 3.1 lowest among five models.

desk verdict Sensible three-dimensional ethics benchmark design undermined by missing parameters, unspecified item selection, and absent reproducibility artifacts. read the letter →

arxiv 2505.00853 v1 pith:QG4FIYAI submitted 2025-05-01 cs.CY

Junfeng Jiao , Saleh Afroogh , Abhejay Murali , Kevin Chen , David Atkinson , Amit Dhurandhar This is my paper · ORCID
classification cs.CY
keywords largelanguagemodelsmoralreasoningAIalignmentbenchmarkdatasetsresponsiblefoundationstheoryvalueconsistencyqualityindex
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the moral reasoning of large language models can be measured systematically instead of being judged by anecdote. It adapts three established human instruments—the Moral Foundations Questionnaire, the World Values Survey, and classic moral dilemmas—into a prompt battery, and scores models on three axes: alignment with human moral judgments, quality of the reasoning, and consistency across related value judgments. Applied to five current LLMs, the battery produces a composite ranking with Claude 3.7 Sonnet first and LLaMA 3.1 last, and a recurring pattern: all models track human responses better on care and fairness than on loyalty, authority, and sanctity, and reason more reliably when identifying principles or consequences than when taking perspectives or applying principles consistently. If the framework is right, it gives developers, auditors, and regulators a common yardstick for comparing ethical performance and locating specific weaknesses.

What carries the argument

The load-bearing mechanism is a three-metric scoring system computed from model responses to the adapted instrument battery. Moral Foundation Alignment (MFA) is $1$ minus the average absolute deviation between the model's rating and the human ground-truth rating on a 0-5 scale, normalized by $5$. The Reasoning Quality Index (RQI) is a weighted sum of three components: semantic similarity between the model's reasoning and human reasoning exemplars, the proportion of expected reasoning elements present, and internal coherence. The Ethical Consistency Metric (ECM) is $1$ minus the average absolute difference, normalized by the maximum possible difference, between a model's scores on conceptually related question pairs. Together these metrics are supposed to capture whether the model agrees with people, whether it reasons the way people reason, and whether it stays stable across framings.

What would settle it

Using the released benchmark data, recompute the composite scores with the RQI weights changed from the calibrated values to equal weights, and independently re-derive the expected reasoning elements from a published rubric; if the Claude-versus-LLaMA ordering flips or the gaps collapse, the claimed precision fails. The test requires the authors to report the weight values and the calibration procedure, both currently absent from the paper.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that moral reasoning in LLMs is not a single ability but a three-dimensional one, and that all three dimensions must be measured together because a model can match human opinions while reasoning poorly, or reason fluently while contradicting itself. The paper claims its adapted battery separates these dimensions and reveals considerable differences across the five systems it evaluates: Claude 3.7 Sonnet ranks highest on the composite, followed closely by GPT-4o, while LLaMA 3.1 trails the others. It also claims a stable structural result: alignment is stronger for the individualizing foundations (care and fairness) than for the binding foundations (loyalty, authority, and sanctity), matching the pattern seen in Western industrialized samples, and perspective-taking plus consistent principle application are the weakest reasoning components across all models. The framework is presented as a standardized, openly released benchmark for precise identification of ethical strengths and weaknesses.

Load-bearing premise

The rankings rest on the unstated choices inside the Reasoning Quality Index—the calibrated weights, the list of expected reasoning elements, and the set of related question pairs—so if those choices were made after seeing model outputs or without explicit criteria, the reasoning and consistency scores do not measure what they claim.

Editorial extensions

If this is right

  • The three-axis battery gives developers a common scale for comparing ethical performance across model families and versions.
  • The consistent care-and-fairness advantage over loyalty-authority-sanctity indicates that current training produces a particular moral profile, not a neutral one.
  • The universal weakness in perspective-taking and principle application identifies concrete targets for alignment training and prompt design.
  • Prompt-variation consistency separates models that otherwise score similarly, so consistency should be part of any ethical evaluation.
  • Publicly released data and code let external teams audit, reproduce, or extend the evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the framework generalizes, the same three-axis design could be extended to multimodal, non-English, and culturally diverse scenarios; the paper itself notes its scenarios are text-only and fixed in advance.
  • The Western-sample alignment pattern may reflect the cultural composition of training corpora rather than intrinsic moral competence; a testable extension is whether fine-tuning on non-Western moral judgments raises binding-foundation scores.
  • Part of what looks like reasoning quality may track instruction-following or response length; an adversarial check would compare scores after stripping responses to their ethical content.
  • The consistency axis could be tracked across model releases to monitor whether alignment techniques improve stability, turning the benchmark from a snapshot into a longitudinal audit.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes a three-dimensional framework for evaluating moral reasoning in large language models, built from adapted versions of the Moral Foundations Questionnaire (MFQ-30), the World Values Survey, and moral dilemma scenarios. It defines a Moral Foundation Alignment score (Eq. 1), a Reasoning Quality Index (Eq. 2), and an Ethical Consistency Metric (Eq. 3), and reports comparative results across five LLMs in Tables 1-4 and Figures 3-7. The headline finding is that models differ considerably in moral reasoning, with Claude 3.7 Sonnet scoring highest and LLaMA 3.1 (70B) lowest, and the authors announce an open-source release of the benchmark and codebase.

Significance. If fully specified and made reproducible, this framework would fill a useful niche by adapting validated instruments from moral psychology into a structured benchmark for comparing LLMs' ethical performance. The MFA formula is transparent, and the inter-foundation correlation pattern in Table 4 is consistent with the individualizing/binding distinction in Moral Foundations Theory. However, as submitted, the manuscript does not deliver a reproducible benchmark: the headline rankings depend on unspecified weighting parameters, unenumerated question-pair sets, an undefined coherence measure, and unreported aggregation formulas for two of the five columns in Table 1. The empirical claims also lack measures of uncertainty. These gaps are load-bearing because the central claim of 'precise identification of ethical strengths and weaknesses' rests on the quantitative rankings, not just on the qualitative framework.

major comments (5)
  1. [Section 4.1, Eq. (2)] The Reasoning Quality Index is defined with 'calibrated weighting parameters' alpha, beta, and gamma, but the manuscript reports no values, no calibration data, and no calibration procedure. Since the 'Reasoning Index' column in Table 1 appears to be this RQI, the ranking is not reproducible. The same passage defines Pkey via 'expected reasoning elements' that are never enumerated and Coh with no operational definition. Section 4.3 lists subjectivity and predetermined scenarios as limitations, but it does not flag these missing operationalizations.
  2. [Section 4.1, Eq. (3)] The Ethical Consistency Metric is defined over a set R of 'conceptually related question pairs' with a maximum difference Smax, but the paper gives no criterion for constructing R, no list of pairs, and no value of Smax. Because the 'Value Consistency' column in Table 1 and the consistency scores in Figure 5 depend on this metric, the reported consistency values cannot be independently computed or validated.
  3. [Table 1 and Section 5.1] Table 1 reports five columns (MFA, Reasoning Index, Value Consistency, Dilemma Resolution, Composite Score) although the framework is described as three-dimensional. Neither 'Dilemma Resolution' nor 'Composite Score' is defined by any equation in Section 4.1. The composite score determines the final ordering (Claude 90.9 vs GPT-4o 90.0), so the headline ranking rests on an unreported aggregation formula.
  4. [Section 5, Tables 1-4 and Figures 3-7] The empirical results are reported without sample sizes, measures of variance, error bars, confidence intervals, or significance tests. Statements such as 'considerable differences' (Section 5.1) and 'Claude shows much lower levels of cultural bias' (Section 5.6) are unsupported by any statistical evidence, especially given the stochastic nature of LLM outputs.
  5. [Table 2 and Section 5.7] The 'Human Baseline' row lists values (Care 95.2, Fairness 93.7, Loyalty 88.4, Authority 87.3, Sanctity 86.1) with no source, no sample description, and no calculation. Since the MFA scores and Figure 7 are interpreted as 'alignment with human moral intuitions,' the baseline must be cited and its construction described; otherwise the comparison is not reproducible.
minor comments (7)
  1. [Abstract] The repository URL contains a line break and space ('https://github.com/ The-Responsible-AI-Initiative/LLM_Ethics_Benchmark.git'), and no repository contents or version are described; the open-source claim cannot be verified from the manuscript.
  2. [Section 2.2] The paragraph beginning 'Qualitative methods capture the context-specific and nuanced nature of morality' is repeated almost verbatim a few paragraphs later; one copy should be removed.
  3. [Section 5.8 vs Section 4.1] The framework is called 'Value Consistency Assessment (VCA)' in Section 5.8 but 'Ethical Consistency Metric (ECM)' in Section 4.1, and Table 1 uses 'Value Consistency'; the terminology should be unified.
  4. [Table 1 caption and Section 5.1] The number of dimensions is inconsistent (three stated vs five columns); clarify whether 'Dilemma Resolution' and 'Composite Score' are additional dimensions or derived aggregates.
  5. [Section 5.6, references [50] and [48]] Reference [50], on causal discovery in visual-model-based reinforcement learning, does not appear relevant to 'Context Insensitivity' in ethical evaluation, and reference [48] seems unrelated to cultural diversity in training data; both should be verified or replaced.
  6. [Section 4.3] The phrase 'answers that are typically more complicated than straightforward' is awkward and should be rephrased.
  7. [Table 4] Table 4 is presented for Claude only; state whether analogous correlation matrices for the other models are available and, if so, where they can be found.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the framework's scores are operational definitions anchored to external human benchmarks, not predictions derived from their own outputs.

full rationale

The paper's central claim is that its three-dimensional framework (MFA, RQI, ECM) measures LLM moral reasoning. The MFA score (Eq. 1) compares LLM responses to human ground-truth scores from the MFQ, an external instrument; this is a measurement, not a circular prediction. The RQI (Eq. 2) is an explicitly defined composite of semantic similarity, presence of expected reasoning elements, and coherence, and the ECM (Eq. 3) is a defined consistency metric over related question pairs. These are operational definitions of the quantities reported in Tables 1 and 3, not predictions fitted to the same data. The paper does not report the values of alpha, beta, gamma or the criteria for selecting expected elements and related pairs, which is a reproducibility and validation gap, but circularity requires showing that an input was defined in terms of the output or that a fitted parameter was relabeled as a prediction; the text provides no such reduction. There are no load-bearing self-citations by the authors, and the external human baselines give the framework independent content. Therefore no circularity is established.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The framework rests on established psychometric instruments (MFQ, WVS, moral dilemmas) and on arithmetic formulas, but the unstated calibration weights, hand-selected reasoning exemplars, and question-pair sets carry a large part of the evaluation. These are free choices by the authors, not externally anchored, so the ledger is mostly domain assumptions plus unspecified parameters.

free parameters (2)
  • alpha, beta, gamma (RQI weighting parameters) = not reported
    Equation (2) defines the Reasoning Quality Index with 'calibrated weighting parameters' but the paper gives no values, bounds, or calibration method; all RQI results in Table 3 depend on them.
  • expected reasoning elements and question pair set R = not reported
    Pkey and ECM rely on hand-selected 'expected reasoning elements' and 'conceptually related question pairs'; the selection criteria are not specified, so these components are effectively free choices by the authors.
assumptions (3)
  • domain assumption MFQ-30, WVS, and moral dilemma instruments provide valid measures of human moral reasoning that can be transferred to LLMs via prompted text responses.
    Section 3.1 selects and adapts these instruments without validation that their constructs remain intact when scored from LLM free-text output.
  • domain assumption Semantic similarity between LLM reasoning text and ground-truth reasoning exemplars (using sentence embeddings) is a valid proxy for reasoning quality.
    Section 4.1 introduces RQI = alpha*Sim + ... and assumes Sim(R_LLM, R_GT) measures quality, without evidence that embedding similarity tracks moral reasoning quality.
  • domain assumption Human ground-truth statistics from published studies are directly comparable to LLM numeric ratings on the same 0-5 scale.
    Equation (1) computes MFA by absolute deviation against a single human mean; the paper does not account for human response distributions or whether LLM ratings share the same scale semantics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM Ethics Benchmark: A Three-Dimensional Assessment System for Evaluating Moral Reasoning in Large Language Models." pith.science (2026). https://pith.science/paper/QG4FIYAI

@misc{pith2026250500853,
  author       = {Pith},
  title        = {Pith review of: LLM Ethics Benchmark: A Three-Dimensional Assessment System for Evaluating Moral Reasoning in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QG4FIYAI}},
  note         = {Machine review of arXiv:2505.00853}
}
read the original abstract

This study establishes a novel framework for systematically evaluating the moral reasoning capabilities of large language models (LLMs) as they increasingly integrate into critical societal domains. Current assessment methodologies lack the precision needed to evaluate nuanced ethical decision-making in AI systems, creating significant accountability gaps. Our framework addresses this challenge by quantifying alignment with human ethical standards through three dimensions: foundational moral principles, reasoning robustness, and value consistency across diverse scenarios. This approach enables precise identification of ethical strengths and weaknesses in LLMs, facilitating targeted improvements and stronger alignment with societal values. To promote transparency and collaborative advancement in ethical AI development, we are publicly releasing both our benchmark datasets and evaluation codebase at https://github.com/ The-Responsible-AI-Initiative/LLM_Ethics_Benchmark.git.

Figures

Figures reproduced from arXiv: 2505.00853 by the authors.

Figure 1
Figure 1. The need for integrated research bridging technical evaluation and human morality [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Technical implementation workflow for moral reasoning assessment in LLMs [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Moral foundation alignment across models compared with human baseline [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Distribution of reasoning depth scores in moral dilemma responses [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Consistency scores across evaluation rounds with prompt variations [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Frequency of specific failure modes in model responses [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Model alignment with human baseline across foundation types [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

100 extracted references · 41 canonical work pages

  1. [1]

    Concrete problems in ai safety

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016

    arXiv 2016

  2. [2]

    Language models as agent models

    Jacob Andreas. Language models as agent models. arXiv preprint arXiv:2212.01681, 2022

    arXiv 2022

  3. [3]

    Claude: A conversational ai assistant

    Anthropic. Claude: A conversational ai assistant. Anthropic Blog, 2023. URL https: //www.anthropic.com/index/introducing-claude

    2023

  4. [4]

    Aquino and A

    K. Aquino and A. Reed. The self-importance of moral identity. Journal of Personality and Social Psychology, 83(6):1423–1440, 2002

    2002

  5. [5]

    Probing pre- trained language models for cross-cultural differences in values

    Arun Arora, Shane Storks, Hamidreza Nakhost, Jianfu Chen, and Percy Liang. Probing pre- trained language models for cross-cultural differences in values. Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3922–3944, 2022

    2022

  6. [6]

    Awad et al

    E. Awad et al. The moral machine experiment. Nature, 563(7729):59–64, 2018

    2018

  7. [7]

    Bandura et al

    A. Bandura et al. Mechanisms of moral disengagement. Journal of Personality and Social Psychology, 71(2):364–374, 1996

    1996

  8. [8]

    Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell

    Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, 2021

    2021

Show all 100 references
  1. [9]

    On the opportunities and risks of foundation models

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

    2021 arXiv

  2. [10]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901, 2020

    1901

  3. [11]

    J. J. Bryson and P. P. Kime. Just an artifact: Why machines are perceived as moral agents. Artificial Intelligence and Society, 26(3):295–307, 2011

    2011

  4. [12]

    Cultural alignment of large language models

    Aida Cao, Anna Karinshak, Jung Yeon Pak, Chenhao Yang, and Diyi Wu. Cultural alignment of large language models. arXiv preprint arXiv:2212.10511, 2022

    2022 arXiv

  5. [13]

    Chalkidis et al

    I. Chalkidis et al. Neural legal judgment prediction in english. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4317–4323, 2019

    2019

  6. [14]

    Chen et al

    M. Chen et al. Do llms understand social knowledge? evaluating the sociability of large language models with socket benchmark. arXiv preprint arXiv:2306.13249, 2023

    2023 arXiv

  7. [15]

    On the role of reasoning in moral alignment of language models

    Elizabeth Clark, Eric Rosen, Daniel Fried, Dario Amodei, Ben Mann, and Dawn Song. On the role of reasoning in moral alignment of language models. arXiv preprint arXiv:2302.07459, 2023

    2023 arXiv

  8. [16]

    Multimodal-coda: A dataset for contrastive disambiguation analysis in multimodal reasoning

    Emily M Coda, Carlos Araujo, Elisa Kreiss, and Christopher Potts. Multimodal-coda: A dataset for contrastive disambiguation analysis in multimodal reasoning. arXiv preprint arXiv:2307.01254, 2023

    2023 arXiv

  9. [17]

    Cushman, L

    F. Cushman, L. Young, and M. Hauser. The dynamics of moral judgment. Cognition, 103(3): 393–420, 2006

    2006

  10. [18]

    Measuring moral reasoning using moral dilemmas: Evaluating reliability, validity, and differential item functioning of the behavioral defining issues test

    Natalie Denny and Alexis Matusiak. Measuring moral reasoning using moral dilemmas: Evaluating reliability, validity, and differential item functioning of the behavioral defining issues test. Journal of Moral Education, 50(3):316–337, 2021

    2021

  11. [19]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2019. 15

    2019 arXiv

  12. [20]

    Moral stories: Situated reasoning about norms, intents, actions, and their consequences

    Denis Emelin, Dung Le, Shrimai Bhatt, Yejin Choi, Daniel Khashabi, and Ellie Pavlick. Moral stories: Situated reasoning about norms, intents, actions, and their consequences. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 698–718, 2021

    2021

  13. [21]

    D. R. Forsyth. A taxonomy of ethical ideologies. Journal of Personality and Social Psychology, 39(1):175–184, 1980

    1980

  14. [22]

    Fu et al

    C. Fu et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  15. [23]

    Artificial intelligence, values, and alignment

    Iason Gabriel. Artificial intelligence, values, and alignment. Minds and Machines, 30:411–437, 2020

    2020

  16. [24]

    Graham et al

    J. Graham et al. Mapping the moral domain. Journal of Personality and Social Psychology, 101(2):366–385, 2011

    2011

  17. [25]

    Moral foundations theory: The pragmatic validity of moral pluralism

    Jesse Graham, Jonathan Haidt, Sena Koleva, Matt Motyl, Ravi Iyer, Sean P Wojcik, and Peter H Ditto. Moral foundations theory: The pragmatic validity of moral pluralism. Advances in experimental social psychology, 47:55–130, 2013

    2013

  18. [26]

    J. D. Greene et al. An fmri investigation of emotional engagement in moral judgment. Science, 293(5537):2105–2108, 2001

    2001

  19. [27]

    Guo et al

    Z. Guo et al. Xiezhi: An ever-updating benchmark for holistic domain knowledge evaluation. arXiv preprint arXiv:2306.05783, 2023

    2023 arXiv

  20. [28]

    J. Haidt. The emotional dog and its rational tail: A social intuitionist approach to moral judgment. Psychological Review, 108(4):814–834, 2001

    2001

  21. [29]

    The Righteous Mind: Why Good People are Divided by Politics and Religion

    Jonathan Haidt. The Righteous Mind: Why Good People are Divided by Politics and Religion. Vintage, 2012

    2012

  22. [30]

    Hendrycks et al

    D. Hendrycks et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938, 2021

    2021 arXiv

  23. [31]

    Hendrycks et al

    D. Hendrycks et al. Cuad: An expert-annotated nlp dataset for legal contract review. arXiv preprint arXiv:2103.06268, 2021

    2021 arXiv

  24. [32]

    Hendrycks et al

    D. Hendrycks et al. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  25. [33]

    Ethics of artificial intelligence

    Dan Hendrycks et al. Ethics of artificial intelligence. arXiv preprint arXiv:2106.08458, 2021

    2021 arXiv

  26. [34]

    Hendrycks et al

    H. Hendrycks et al. Human-ai moral consistency: Bias recognition dataset, 2023. URL https://github.com/hendrycks/ethics

    2023

  27. [35]

    The weirdest people in the world? Behavioral and Brain Sciences, 33(2-3):61–83, 2010

    Joseph Henrich, Steven J Heine, and Ara Norenzayan. The weirdest people in the world? Behavioral and Brain Sciences, 33(2-3):61–83, 2010

    2010

  28. [36]

    M. J. Hosseini et al. Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 523–533, 2014

    2014

  29. [37]

    Huang et al

    J. Huang et al. Emotionally numb or empathetic? evaluating how llms feel using emotionbench. arXiv preprint arXiv:2307.09009, 2023

    2023 arXiv

  30. [38]

    Huang et al

    Y . Huang et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023

    2023 arXiv

  31. [39]

    Huang et al

    Y . Huang et al. Trustgpt: A benchmark for trustworthy and responsible large language models. arXiv preprint arXiv:2306.11507, 2023

    2023 arXiv

  32. [40]

    Open-source large language models leaderboard, 2023

    HuggingFace. Open-source large language models leaderboard, 2023. URL https:// huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard. 16

    2023

  33. [41]

    Inter-university Consortium for Political and Social Research, 2000

    Ronald Inglehart, Miguel Basañez, and Alejandro Moreno.World Values Surveys and European Values Surveys, 1981-1984, 1990-1993, and 1995-1997 . Inter-university Consortium for Political and Social Research, 2000

    1981

  34. [42]

    The global landscape of ai ethics guidelines

    Anna Jobin, Marcello Ienca, and Effy Vayena. The global landscape of ai ethics guidelines. Nature Machine Intelligence, 1(9):389–399, 2019

    2019

  35. [43]

    Ethics-eval: A bench- marking framework for ethical evaluation of language models, 2023

    Richard Johnson, Daniel Romano, Rose Wang, and Sofia Valverde. Ethics-eval: A bench- marking framework for ethical evaluation of language models, 2023. URL https://github. com/AI-secure/ethics-eval

    2023

  36. [44]

    Kiela et al

    D. Kiela et al. Dynabench: Rethinking benchmarking in nlp. arXiv preprint arXiv:2104.14337, 2021

    2021 arXiv

  37. [45]

    T. W. Kim, J. Hooker, and T. Donaldson. The ethics of ai-driven decision-making. AI \& Society, 36(2):1–15, 2021

    2021

  38. [46]

    Kohlberg

    L. Kohlberg. The Philosophy of Moral Development: Moral stages and the idea of justice . Harper \& Row, 1981

    1981

  39. [47]

    Koncel-Kedziorski et al

    R. Koncel-Kedziorski et al. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585–597, 2015

    2015

  40. [48]

    Gender, race, and intersectionality on the federal appellate bench

    Stephanie Hanes Larson. Gender, race, and intersectionality on the federal appellate bench. Washington University Law Review, 97(5):1–42, 2017

    2017

  41. [49]

    Li et al

    B. Li et al. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023

    2023 arXiv

  42. [50]

    Systematic evaluation of causal discovery in visual model based reinforcement learning

    Nan Li, Zhi Wang, Shuangfei Zhu, Tatyana Sharpee, Yingzhen Li, and Amy Zhang. Systematic evaluation of causal discovery in visual model based reinforcement learning. arXiv preprint arXiv:2203.12188, 2022

    2022 arXiv

  43. [51]

    Logic-guided semantic representation learning for zero-shot rela- tion classification

    Ningyu Li, Zhuang Gao, Han Xiao, Shumin Huang, Xiang Xie, Jiaoyan Zhu, Yiyi Li, Yunzhi Xiao, and Huajun Chen. Logic-guided semantic representation learning for zero-shot rela- tion classification. In Proceedings of the 28th International Conference on Computational Linguistics...

    2019

  44. [53]

    Holistic evaluation of language models

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022

    2022 arXiv

  45. [54]

    G. Lind. The moral judgment test (mjt): Thirty years of research. Educational Research and Evaluation, 6(1):1–20, 2000

    2000

  46. [55]

    Ling et al

    W. Ling et al. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 158–167, 2017

    2017

  47. [56]

    Liu et al

    L. Liu et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    2023 arXiv

  48. [57]

    Multi-task deep neural networks for natural language understanding

    Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4487–4496, 2019

    2019

  49. [58]

    Chatbot arena: Benchmarking llms in the wild with elo ratings, 2023

    LMSYS. Chatbot arena: Benchmarking llms in the wild with elo ratings, 2023. URL https://lmsys.org

    2023

  50. [59]

    D. P. McAdams. The Stories We Live By: Personal myths and the making of the self. Guilford Press, 1993. 17

    1993

  51. [60]

    Algo- rithmic fairness: Choices, assumptions, and definitions

    Shira Mitchell, Eric Potash, Solon Barocas, Alexander D’Amour, and Kristian Lum. Algo- rithmic fairness: Choices, assumptions, and definitions. Annual Review of Statistics and Its Application, 9:359–380, 2022

    2022

  52. [61]

    Principles alone cannot guarantee ethical ai

    Brent Mittelstadt. Principles alone cannot guarantee ethical ai. Nature Machine Intelligence, 1 (11):501–507, 2019

    2019

  53. [62]

    Interpretable machine learning

    Christoph Molnar. Interpretable machine learning. Lulu. com, 2022

    2022

  54. [63]

    Translation tutorial: 21 fairness definitions and their politics

    Arvind Narayanan. Translation tutorial: 21 fairness definitions and their politics. Proc. Conf. Fairness Accountability Transp., New York, USA, 1170, 2018

    2018

  55. [64]

    trolley problem

    C. D. Navarrete et al. Virtual morality: Emotion and action in a simulated three-dimensional "trolley problem". Social Neuroscience, 7(4):364–374, 2012

    2012

  56. [65]

    R. E. Nisbett. The Geography of Thought: How Asians and Westerners think differently... and why. Free Press, 2003

    2003

  57. [66]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  58. [67]

    Paolacci, J

    G. Paolacci, J. Chandler, and P. G. Ipeirotis. Running experiments on amazon mechanical turk. Judgment and Decision Making, 5(5):411–419, 2010

    2010

  59. [68]

    Rae et al

    Jack W. Rae et al. Scaling language models: Methods, analysis \& insights from training gopher. arXiv preprint arXiv:2112.11446, 2021

    2021 arXiv

  60. [69]

    Closing the ai accountability gap: Defining an end-to-end framework for internal algorithmic auditing

    Inioluwa Deborah Raji, Andrew Smart, Rebecca N White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. Closing the ai accountability gap: Defining an end-to-end framework for internal algorithmic auditing. Proceedings of the...

    2020

  61. [70]

    Sentence-bert: Sentence embeddings using siamese bert- networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 3982–3992, 2019

    2019

  62. [71]

    J. R. Rest. Development in Judging Moral Issues. University of Minnesota Press, 1979

    1979

  63. [72]

    Adaptive testing and debugging of nlp models

    Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Adaptive testing and debugging of nlp models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3253–3267, 2022

    2022

  64. [73]

    Rudinger et al

    R. Rudinger et al. Gender bias in coreference resolution. InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 8–14, 2018

    2018

  65. [74]

    Sap et al

    M. Sap et al. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477–5490, 2020

    2020

  66. [75]

    big three

    R. A. Shweder et al. The "big three" of morality (autonomy, community, divinity) and the big three explanations of suffering. Morality and Health, pages 119–169, 1997

    1997

  67. [76]

    Singhal et al

    S. Singhal et al. Large language models encode clinical knowledge. Nature, 620(7972):1–10, 2023

    2023

  68. [77]

    Open-source tools learning benchmarks, 2023

    ToolBench. Open-source tools learning benchmarks, 2023. URL https://github.com/ sambanova/toolbench

    2023

  69. [78]

    High-performance medicine: the convergence of human and artificial intelligence

    Eric J Topol. High-performance medicine: the convergence of human and artificial intelligence. Nature Medicine, 25(1):44–56, 2019

    2019

  70. [79]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 18

    2023 arXiv

  71. [80]

    Varshney et al

    V . Varshney et al. Freshllms: Refreshing large language models with search engine augmenta- tion. arXiv preprint arXiv:2306.06598, 2023

    2023 arXiv

  72. [81]

    Wang et al

    W. Wang et al. Chain-of-thought prompting for responding to in-depth dialogue questions with llm. arXiv preprint arXiv:2307.05082, 2023

    2023 arXiv

  73. [82]

    Wang et al

    X. Wang et al. Cmb: A comprehensive medical benchmark in chinese. arXiv preprint arXiv:2307.06975, 2023

    2023 arXiv

  74. [83]

    Self- consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self- consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022

    2022 arXiv

  75. [84]

    Chain of thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Zhao, et al. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022

    2022 arXiv

  76. [85]

    Taxonomy of risks posed by language models

    Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, et al. Taxonomy of risks posed by language models. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 214–229, 2022

    2022

  77. [86]

    The role and limits of principles in ai ethics: towards a focus on tensions

    Jess Whittlestone, Rune Nyrup, Anna Alexandrova, Kanta Dihal, and Stephen Cave. The role and limits of principles in ai ethics: towards a focus on tensions. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 195–200, 2021

    2019

  78. [87]

    Transformers: State- of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State- of-the-art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natu...

    2020

  79. [88]

    Xu et al

    X. Xu et al. Cvalues: Measuring the values of chinese large language models from safety to responsibility. arXiv preprint arXiv:2307.09705, 2023

    2023 arXiv

  80. [89]

    Xu et al

    X. Xu et al. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09265, 2023

    2023 arXiv

  81. [90]

    Yang et al

    Y . Yang et al. Glue-x: Evaluating natural language understanding models from an out-of- distribution generalization perspective. arXiv preprint arXiv:2211.08073, 2022

    2022 arXiv

  82. [91]

    Yang et al

    Y . Yang et al. Gaokao-bench: Revisiting out-of-distribution robustness in nlp.arXiv preprint arXiv:2306.14824, 2023

    2023 arXiv

  83. [92]

    Yang et al

    Y . Yang et al. Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. arXiv preprint arXiv:2306.06687, 2023

    2023 arXiv

  84. [93]

    Yang et al

    Y . Yang et al. Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490, 2023

    2023 arXiv

  85. [94]

    Young et al

    L. Young et al. Disruption of the right temporoparietal junction with transcranial magnetic stimulation reduces the role of beliefs in moral judgments. Proceedings of the National Academy of Sciences, 107(15):6753–6758, 2010

    2010

  86. [95]

    Yu et al

    J. Yu et al. Kola: Carefully benchmarking world knowledge of large language models. arXiv preprint arXiv:2306.09296, 2023

    2023 arXiv

  87. [96]

    P. J. Zak. The neurobiology of trust. Scientific American, 298(6):88–95, 2008

    2008

  88. [97]

    Zhang et al

    Z. Zhang et al. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. arXiv preprint arXiv:2306.05179, 2023

    2023 arXiv

  89. [98]

    Calibrate before use: Improving few-shot performance of language models

    Tony Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. arXiv preprint arXiv:2102.09690, 2021

    2021 arXiv

  90. [99]

    Zheng et al

    Z. Zheng et al. Safetybench: Evaluating the safety of large language models with multiple choice questions. arXiv preprint arXiv:2305.13656, 2023. 19

    2023 arXiv

  91. [100]

    Zhong et al

    Z. Zhong et al. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023

    2023 arXiv

  92. [101]

    Zhu et al

    Z. Zhu et al. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv preprint arXiv:2306.04528, 2023. 20

    2023 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.