Pith. sign in

REVIEW 3 major objections 6 minor 31 references

Do Large Language Models Understand Morality Across Cultures?

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read Open-source AI flattens cross-cultural moral differences.

desk verdict A promising negative result about small LLMs and cultural moral variation, but the zero-imputation of WVS missing data undermines the central variance claim. read the letter →

arxiv 2507.21319 v1 pith:BIJHZQKK submitted 2025-07-28 cs.CL

classification cs.CL
keywords cross-culturalmoralitylargelanguagemodelsmoraljudgmentWorldValuesSurveyPewGlobalAttitudesProjectculturalbiasvariancecompressionclusteralignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Across two large international surveys of moral attitudes, five open-source language models produce a narrower, more permissive picture of the world's moral landscape than human respondents do. The paper tests this by comparing model-generated moral scores with country-level scores from the World Values Survey and the Pew Global Attitudes Project, using variance comparisons, country-cluster alignment, and direct pairwise probes. It finds that the models assign higher moral-acceptability scores and systematically lower variance than the surveys, and that their country groupings align poorly with survey-based groupings. The authors conclude that current small open-source large language models compress cross-cultural disagreement and skew toward the liberal values typical of Western, Educated, Industrialized, Rich, Democratic (W.E.I.R.D.) societies, with no convincing fix from larger size or multilinguality.

What carries the argument

The load-bearing mechanism is a log-probability scoring procedure: for each country–topic pair the model is prompted with statements such as "In {country} {topic} is {moral_judgment}" for five contrasting token pairs (e.g., always justifiable / never justifiable), and the difference in log-probabilities between the 'moral' and 'immoral' continuations is averaged into a model moral score. That score is then fed into three evaluation instruments: variance comparison against the survey distributions, K-means cluster alignment measured by Adjusted Rand Index and Adjusted Mutual Information, and direct pairwise similarity probes between countries from divergent clusters. The variance comparison is the primary instrument for the compression claim, and the cluster and probe analyses are meant to show that the low variance reflects genuine loss of cultural structure rather than a scaling artifact.

What would settle it

Recompute the WVS country-level means and variances after excluding countries where a moral question was not asked, rather than scoring those cases as 'never justifiable'; if the model–survey variance gap shrinks or disappears on the corrected benchmark, the paper's central claim that models underestimate cultural disagreement is an artifact of that recoding.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central finding is that contemporary open-source LLMs do not reproduce the cross-cultural variation in moral judgments recorded in the WVS Wave 7 and PEW 2013 surveys. Across all five tested models, mean model moral scores are higher (more acceptable) and variances are lower than the empirical survey scores for the same country–topic pairs, and topic-level variance correlations are mostly not significant. Cluster alignment between model-induced and survey-induced country groupings is low or negative for most models and topic subsets, and direct comparative prompting yields accuracy near chance, with the best-looking models achieving high recall only by predicting the same class almost always. The paper concludes that the models propagate a homogenized, more permissive moral worldview aligned with W.E.I.R.D. values, and finds no convincing evidence that multilinguality or larger model size improves cultural fidelity.

Load-bearing premise

The load-bearing premise is that replacing every WVS non-response — including 'not asked in survey' — with a zero before averaging preserves the true cross-country differences in moral opinion; if that recoding instead inflates variance, the models' supposedly compressed judgments are being measured against a distorted benchmark.

Editorial extensions

If this is right

  • The tested models compress cross-cultural disagreement on polarizing topics, so applications that rely on their moral judgments will understate how much societies differ on issues like homosexuality and sex before marriage.
  • The systematic upward shift in acceptability means LLM-based tools will tend to present morally contested behaviors as more permissible than global survey respondents do.
  • Because neither larger size nor multilinguality produced a convincing improvement, scaling alone will not remove the cultural bias; the paper's recommended levers are more diverse training data, culture-aware fine-tuning, and context-invoking prompts.
  • The direct-probe results show that near-chance accuracy plus high recall can be produced by predicting the same class almost always, so evaluating cultural understanding requires precision and balanced metrics, not accuracy alone.
  • WVS and PEW survey benchmarks can serve as standardized pre-deployment tests of cultural representativeness for new models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper does not test: because the WVS preprocessing codes 'not asked in survey' as zero before averaging, a portion of the cross-country variance used as the benchmark may be an artifact; re-running the variance comparison on countries where each question was actually asked would test whether the model-compression claim survives.
  • A reader could test whether the models' country sensitivity is genuine by adding no-country baseline prompts; if the same permissive scores appear without a country name, part of the effect is an English-language prior rather than cultural knowledge.
  • A natural next experiment is to fine-tune a small model on the WVS/PEW country–topic response distributions and then apply the same variance, ARI, and probe metrics, giving a direct test of whether the compression is removable by training.
  • The paper reports aggregate country-level patterns; modeling within-country response distributions would clarify whether the models compress genuine individual-level disagreement or only cross-country means.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper asks whether small open-source LLMs reproduce cross-cultural moral attitudes measured by the World Values Survey (Wave 7) and the Pew Global Attitudes Project (2013). For five models (GPT-2 Medium/Large, OPT-125, Qwen-0.5B, BLOOM-560M), it computes country-topic moral scores from token log-probabilities and then applies three evaluations: comparison of topic-level variances between model scores and survey scores, cluster alignment via ARI/AMI, and direct comparative prompting. The paper concludes that models assign higher moral acceptability, produce systematically lower cross-country variance, and show low alignment with empirical survey patterns, interpreting this as a homogenized, WEIRD-leaning moral outlook.

Significance. If the results were reliable, the finding that small open-source LLMs compress cross-cultural moral variation and skew toward permissive judgments would be a useful contribution to cultural-bias evaluation. The paper's structure is sensible: it uses public survey datasets and open models, combines variance, clustering, and prompting methods, and reports per-model results in tables. These strengths, however, are outweighed by a load-bearing data-processing error: the WVS non-response imputation described in Section 3.2 directly distorts the empirical variance baseline that the central claim compares against. The paper does not flag this in its limitations, and the main conclusion is therefore not supported by the reported analyses. The study also tests only models up to 774M parameters, so the title-level generalization to 'large language models' is premature.

major comments (3)
  1. [Section 3.2, Section 4, Table 3] The replacement of all WVS non-response codes (-1, -2, -4, -5, including 'Not asked in survey') with 0 is not neutral. Valid WVS responses range from 1 to 10, so 0 lies below every real response; after normalization, a country-topic mean that includes these zeros moves toward 'never justifiable.' For topics with structural non-response, such as 'Not asked in survey,' whole countries are effectively scored as strongly disapproving rather than as missing. This artificially inflates cross-country variance for precisely the morally controversial topics (e.g., homosexuality, sex before marriage) that drive the variance-compression conclusion. The statement in Section 3.2 that 'a replacement value of 0 ensures that non-responses do not influence the computed averages' is false: zeros are included in the averaging and pull country means downward. Because the main claim in Section 4 and Table 3 compares model variances against this inflated empirical baseline, the conclusion that 'models systematically produce lower variance than survey data' is not established. The cluster-alignment and direct-probing analyses inherit the same distorted country-topic means. Section 6 does not mention this decision. A reanalysis that excludes non-responses or uses appropriate missing-data handling is required before the variance comparison can be interpreted.
  2. [Section 3.3, Section 5.4, Abstract] The paper's title, abstract, and conclusion speak about 'large language models' and 'current LLMs,' but all evaluated models are at most 774M parameters (GPT-2 Large, BLOOM-560M, Qwen-0.5B, OPT-350M). The conclusion that 'no remarkable differences were found regardless of model size or multilinguality' is unsupported by this narrow size range and by the absence of any modern instruction-tuned or frontier-scale model. Even if the results hold for these specific models, they do not license the broad generalization stated in the abstract and conclusion. The claims should be restricted to the tested small open-source models, or the experimental scope should be expanded.
  3. [Section 3.5.2, Tables 6-8] The cluster-alignment analysis selects the number of clusters K from the survey data (via highest silhouette score) and then imposes that same K on the model-generated scores. ARI and AMI are sensitive to the number of clusters, so a K chosen for one data space can make the comparison artificially unfavorable or favorable in the other. No sensitivity analysis over K is reported, and no confidence intervals or significance tests are given for the near-zero ARI/AMI values in Tables 6-8. The conclusion that model-based clusterings 'do not faithfully capture' the empirical patterns therefore needs additional support, at minimum a K-sensitivity analysis and some measure of uncertainty for the alignment metrics.
minor comments (6)
  1. [Section 3.5.3, Table 1] The direct-probing method says the comparative token takes pairs such as 'similar/dissimilar,' but Table 1 lists only pairs such as 'always justifiable / never justifiable' and 'right / wrong.' The mapping between these token pairs and the binary 'similar vs different' outcome is never specified, which makes Method 3 hard to reproduce.
  2. [Sections 5.4 and 5.5] The manuscript contains two nearly identical subsections titled 'Conclusion' (5.4 and 5.5). One should be removed, and the recommendations that appear only in 5.5 should be integrated into a single concluding section.
  3. [Section 3.1] The statement that the Pew 2013 questionnaire was 'administered in English' is surprising and should be verified; Pew Global Attitudes surveys are generally fielded in local languages. If the survey was translated, the sentence is factually incorrect and should be corrected.
  4. [Section 6, Section 3.5.3] The limitations section says that 'due to computational limits, we randomly selected topics in Method 3,' but Section 3.5.3 says the procedure is applied 'For each topic.' The text should clarify which topics were used in the direct-probing analysis and how the random selection was performed.
  5. [Section 4.2, Table 10] Ten chi-squared tests are reported in Table 10 without any correction for multiple comparisons. Under the null, one would expect roughly 0.5 significant results at the 5% level and 0.1 at the 1% level in ten tests; the single WVS result with p=0.004 and the single PEW result with p=0.032 are therefore weaker evidence of alignment than the text implies. A multiple-testing correction or a note about the false-positive rate is needed.
  6. [Table 5] The column headings in Table 5 are confusing: 'Model Survey Survey Var . Survey Mean Model Var . Model Mean Topic Var . Diff' contains an odd stray period and an unclear 'Model Survey' label. The heading should be reformatted to distinguish survey columns from model columns.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation; empirical benchmark comparison is self-contained, with a minor non-load-bearing self-citation and a serious but non-circular data-processing concern.

full rationale

The paper does not derive its conclusions from its inputs by construction. Model-generated moral scores are obtained by log-probability contrasts of frozen open-source LLMs (Section 3.4), and no model parameter is fitted to WVS or PEW; the reported comparisons (variance, cluster alignment, direct probing) are against external survey benchmarks. The central 'variance compression' result is therefore an empirical observation, not an algebraic consequence of the scoring scheme. The self-citation to the authors' prior work [18] for the probing recipe is not load-bearing: the exact prompt templates, token pairs, and log-probability subtraction are specified in the manuscript itself, so the conclusion does not rest on the cited paper's authority. The most serious concern is instead a validity threat: Section 3.2 replaces WVS non-responses (-1, -2, -4, -5) with 0 on a 1-10 scale, which pulls country-topic means downward and, after normalization, makes 'not asked in survey' countries appear maximally disapproving, inflating the empirical variance baseline used in the first method. However, that is a data-processing and measurement error rather than a circular step: the model outputs are not defined in terms of the survey scores, and the paper's claim is not entailed by the imputation rule. Accordingly, no circularity step meets the standard of Eq. X = Eq. Y by construction or fitted-parameter-renamed-as-prediction; score 2 reflects the minor self-citation and the unflagged methodological risk, not a circular derivation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities. Its central claim rests on several unstated modeling choices, most importantly the replacement of missing survey responses with 0, which can bias the empirical baseline. Other assumptions concern the validity of country-level means, the comparability of English prompts, and the transferability of clustering parameters.

free parameters (3)
  • Missing value replacement constant = 0
    WVS non-responses (-1, -2, -4, -5) are replaced with 0 before averaging (Section 3.2). This ad hoc choice shifts country means downward and inflates cross-country variance, directly affecting the variance comparison that underpins the main conclusion.
  • Number of clusters K = Silhouette-selected on survey data, then applied to model scores
    K is chosen from empirical survey clusters and reused for model-generated scores without checking whether the model score distribution supports that K (Section 3.5.2). No sensitivity analysis is provided.
  • Token pair set and prompt templates = Five token pairs and two prompt templates
    The token pairs in Table 1 and the two prompt templates are described as 'systematically chosen' but no justification or ablation is given; results are averaged over them (Section 3.4).
assumptions (4)
  • domain assumption Country-level mean of survey responses represents cultural moral attitude
    Averaging obscures within-country heterogeneity and outlier perspectives; the authors acknowledge this in Section 3.1 but proceed with the aggregation.
  • domain assumption English prompt templates elicit comparable moral judgments across all countries
    All prompts are in English, even for multilingual models and non-English-speaking countries (Section 3.4). This may not capture local moral reasoning or cultural context.
  • domain assumption Log-probability differences across token pairs are commensurable and can be averaged
    The five token pairs (e.g., 'right/wrong' vs. 'ethical/unethical') have different semantics; subtracting and averaging them assumes equal scale and interval properties, which is not established.
  • ad hoc to paper Silhouette-selected K from survey clusters transfers to model score space
    The same K is applied to model-generated scores without verifying the model score distribution supports that number of clusters (Section 3.5.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do Large Language Models Understand Morality Across Cultures?." pith.science (2026). https://pith.science/paper/BIJHZQKK

@misc{pith2026250721319,
  author       = {Pith},
  title        = {Pith review of: Do Large Language Models Understand Morality Across Cultures?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BIJHZQKK}},
  note         = {Machine review of arXiv:2507.21319}
}
read the original abstract

Recent advancements in large language models (LLMs) have established them as powerful tools across numerous domains. However, persistent concerns about embedded biases, such as gender, racial, and cultural biases arising from their training data, raise significant questions about the ethical use and societal consequences of these technologies. This study investigates the extent to which LLMs capture cross-cultural differences and similarities in moral perspectives. Specifically, we examine whether LLM outputs align with patterns observed in international survey data on moral attitudes. To this end, we employ three complementary methods: (1) comparing variances in moral scores produced by models versus those reported in surveys, (2) conducting cluster alignment analyses to assess correspondence between country groupings derived from LLM outputs and survey data, and (3) directly probing models with comparative prompts using systematically chosen token pairs. Our results reveal that current LLMs often fail to reproduce the full spectrum of cross-cultural moral variation, tending to compress differences and exhibit low alignment with empirical survey patterns. These findings highlight a pressing need for more robust approaches to mitigate biases and improve cultural representativeness in LLMs. We conclude by discussing the implications for the responsible development and global deployment of LLMs, emphasizing fairness and ethical alignment.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 13 canonical work pages

  1. [18]

    Mohammadi, E

    H. Mohammadi, E. Papadopoulou, Y . F. Meijer, and A. Bagheri. Explor- ing cultural variations in moral judgments with large language models. arXiv preprint arXiv:2506.12433, 2025

  2. [1]

    Arora, L

    A. Arora, L. Kaffee, and I. Augenstein. Probing pre-trained lan- guage models for cross-cultural differences in values. arXiv preprint arXiv:2203.13722, 2022. URL https://doi.org/10.48550/arxiv.2203. 13722

  3. [2]

    E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? Pro- ceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, 2021. doi: 10.1145/3442188.3445922. URL https://dl.acm.org/doi/10.1145/3442188.3445922

  4. [3]

    Benkler, D

    Y . Benkler, D. Mosaphir, S. Friedman, A. Smart, and S. Schmer-Galunder. Assessing llms for moral value pluralism. arXiv (Cornell University),

  5. [4]

    Bolukbasi, K.-W

    T. Bolukbasi, K.-W. Chang, J. Y . Zou, V . Saligrama, and A. T. Kalai. Quantifying and reducing stereotypes in word embeddings. arXiv, 2016. doi: 10.48550/arxiv.1606.06121. URL https://arxiv.org/abs/1606.06121

  6. [5]

    X. Du, Z. Yu, S. Gao, D. Pan, Y . Cheng, Z. Ma, R. Yuan, X. Qu, J. Liu, T. Zheng, X. Luo, G. Zhou, B. Yuan, W. Chen, J. Fu, and G. Zhang. Chi- nese Tiny LLM: Pretraining a Chinese-Centric Large Language Model. arXiv (Cornell University), 4 2024. doi: 10.48550/arxiv.2404.04167. URL https://arxiv.org/abs/2404.04167

  7. [6]

    Graham, P

    J. Graham, P. Meindl, E. Beall, K. M. Johnson, and L. Zhang. Cultural differences in moral judgment and behavior, across and within societies. Current Opinion in Psychology, 8:125–130, 2016. doi: 10.1016/j.copsyc. 2015.09.007. URL https://doi.org/10.1016/j.copsyc.2015.09.007

  8. [7]

    C. W. Haerpfer, P. Bernhagen, R. F. Inglehart, and C. Welzel. World Values Survey: Round Seven - Country-Pooled Datafile Version. Institute for Comparative Survey Research, Vienna, 2022. URL http://www. worldvaluessurvey.org/WVSDocumentationWV7.jsp

Show all 31 references
  1. [8]

    J. Haidt. The emotional dog and its rational tail: A social intuitionist approach to moral judgment. Psychological Review, 108(4):814–834,

  2. [9]

    Henrich, S

    J. Henrich, S. J. Heine, and A. Norenzayan. The weirdest people in the world? Behavioral and brain sciences, 33(2-3):61–83, 2010

  3. [10]

    R. L. Johnson, G. Pistilli, N. Menédez-González, L. D. D. Duran, E. Panai, J. Kalpokiene, and D. J. Bertulfo. The ghost in the machine has an american accent: value conflict in gpt-3. arXiv.org, mar 2022. URL https://arxiv.org/abs/2203.07785

  4. [11]

    Karpouzis

    K. Karpouzis. Plato’s shadows in the digital cave: Controlling cultural bias in generative ai. Electronics, 13(8):1457, 2024. doi: 10.3390/ electronics13081457. URL https://doi.org/10.3390/electronics13081457

  5. [12]

    Kharchenko, T

    J. Kharchenko, T. Roosta, A. Chadha, and C. Shah. How well do llms represent values across cultures? empirical analysis of llm responses based on hofstede cultural dimensions. arXiv preprint, 2024. URL https://doi.org/10.48550/arxiv.2406.14805. arXiv:2406.14805

  6. [13]

    Lazarenko and T

    D. Lazarenko and T. Bonald. Pairwise adjusted mutual information,

  7. [14]

    Le Scao, A

    T. Le Scao, A. Fan, C. Akiki, E. Pavlick, S. Ili´c, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V . d. Moral, O. Ruwase, R. Bawden, and M. J. ... Nelson. Bloom...

  8. [15]

    C. C. Liu, F. Koto, T. Baldwin, and I. Gurevych. Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings. arXiv, 2023. URL https://arxiv.org/abs/2309. 08591. arXiv:2309.08591

  9. [16]

    Mishra, G

    A. Mishra, G. Nayak, S. Bhattacharya, T. Kumar, A. Shah, and M. Foltin. Llm-guided counterfactual data generation for fairer ai. In Companion Proceedings of the ACM on Web Conference 2024 , WWW ’24, page 1538–1545, New York, NY , USA, 2024. Association for Computing Machinery....

  10. [17]

    Mohammadi, A

    H. Mohammadi, A. Bagheri, A. Giachanou, and D. L. Oberski. Explain- ability in practice: A survey of explainable nlp across various domains. arXiv preprint arXiv:2502.00837, 2025

  11. [19]

    Mohammadi, T

    H. Mohammadi, T. Shahedi, P. Mosteiro, M. Poesio, A. Bagheri, and A. Giachanou. Assessing the reliability of llms annotations in the context of demographic bias and model explanation. arXiv preprint arXiv:2507.13138, 2025

  12. [20]

    Nazaretsky, S

    T. Nazaretsky, S. Hershkovitz, and G. Alexandron. Kappa learning: A new item-similarity method for clustering educational items from response data. 04 2020. URL https://eric.ed.gov/?id=ED599209

  13. [21]

    Nemani, Y

    P. Nemani, Y . D. Joel, P. Vijay, and F. F. Liza. Gender bias in transformers: A comprehensive review of detection and mitigation strategies. Natural Language Processing Journal, 6:100047, 2024. doi: 10.1016/j.nlp.2023. 100047. URL https://doi.org/10.1016/j.nlp.2023.100047

  14. [22]

    S. U. Noble. Algorithms of Oppression: How Search Engines Reinforce Racism. NYU Press, New York, NY , 2018. ISBN 978-1479837243. URL https://nyupress.org/9781479837243/algorithms-of-oppression/

  15. [23]

    Radford, J

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019. URL https://api.semanticscholar.org/CorpusID:160025533

  16. [24]

    Ramezani and Y

    A. Ramezani and Y . Xu. Knowledge of cultural moral norms in large language models. arXiv (Cornell University), 2023. doi: 10.48550/arxiv. 2306.01857. URL https://doi.org/10.48550/arxiv.2306.01857

  17. [25]

    big three

    R. A. Shweder, N. C. Much, M. Mahapatra, and L. Park. The "big three" of morality (autonomy, community, divinity) and the "big three" explanations of suffering. In A. Brandt and P. Rozin, editors, Morality and Health, pages 119–169. Routledge, 1997. URL https://psycnet.apa. or...

  18. [26]

    L. Wang, X. Chen, and X. Deng. Prompt engineering in consistency and reliability with the evidence-based guideline for llms. npj Digital Medicine, 7:41, 2024. doi: 10.1038/s41746-024-01029-4. URL https: //doi.org/10.1038/s41746-024-01029-4

  19. [27]

    Zhang, S

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V . Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

  20. [28]

    Zou and L

    J. Zou and L. Schiebinger. Ai can be sexist and racist—it’s time to make it fair. Nature, 559(7714):324–326, 2018. doi: 10. 1038/d41586-018-05707-8. URL https://www.nature.com/articles/ d41586-018-05707-8. A Appendix A.1 Most controversial WVS topics according to models Topic ...

  21. [2001]

    doi: 10.1037/0033-295X.108.4.814

  22. [2021]

    URL https://arxiv.org/abs/2103.12641

  23. [2023]

    URL https://doi.org/10.48550/ arxiv.2312.10075

    doi: 10.48550/arxiv.2312.10075. URL https://doi.org/10.48550/ arxiv.2312.10075

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.