REVIEW 3 major objections 6 minor 31 references
Do Large Language Models Understand Morality Across Cultures?
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Open-source AI flattens cross-cultural moral differences.
desk verdict A promising negative result about small LLMs and cultural moral variation, but the zero-imputation of WVS missing data undermines the central variance claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a log-probability scoring procedure: for each country–topic pair the model is prompted with statements such as "In {country} {topic} is {moral_judgment}" for five contrasting token pairs (e.g., always justifiable / never justifiable), and the difference in log-probabilities between the 'moral' and 'immoral' continuations is averaged into a model moral score. That score is then fed into three evaluation instruments: variance comparison against the survey distributions, K-means cluster alignment measured by Adjusted Rand Index and Adjusted Mutual Information, and direct pairwise similarity probes between countries from divergent clusters. The variance comparison is the primary instrument for the compression claim, and the cluster and probe analyses are meant to show that the low variance reflects genuine loss of cultural structure rather than a scaling artifact.
What would settle it
Recompute the WVS country-level means and variances after excluding countries where a moral question was not asked, rather than scoring those cases as 'never justifiable'; if the model–survey variance gap shrinks or disappears on the corrected benchmark, the paper's central claim that models underestimate cultural disagreement is an artifact of that recoding.
Extended reading notes
Core claim
On the paper's own terms, the central finding is that contemporary open-source LLMs do not reproduce the cross-cultural variation in moral judgments recorded in the WVS Wave 7 and PEW 2013 surveys. Across all five tested models, mean model moral scores are higher (more acceptable) and variances are lower than the empirical survey scores for the same country–topic pairs, and topic-level variance correlations are mostly not significant. Cluster alignment between model-induced and survey-induced country groupings is low or negative for most models and topic subsets, and direct comparative prompting yields accuracy near chance, with the best-looking models achieving high recall only by predicting the same class almost always. The paper concludes that the models propagate a homogenized, more permissive moral worldview aligned with W.E.I.R.D. values, and finds no convincing evidence that multilinguality or larger model size improves cultural fidelity.
Load-bearing premise
The load-bearing premise is that replacing every WVS non-response — including 'not asked in survey' — with a zero before averaging preserves the true cross-country differences in moral opinion; if that recoding instead inflates variance, the models' supposedly compressed judgments are being measured against a distorted benchmark.
Editorial extensions
If this is right
- The tested models compress cross-cultural disagreement on polarizing topics, so applications that rely on their moral judgments will understate how much societies differ on issues like homosexuality and sex before marriage.
- The systematic upward shift in acceptability means LLM-based tools will tend to present morally contested behaviors as more permissible than global survey respondents do.
- Because neither larger size nor multilinguality produced a convincing improvement, scaling alone will not remove the cultural bias; the paper's recommended levers are more diverse training data, culture-aware fine-tuning, and context-invoking prompts.
- The direct-probe results show that near-chance accuracy plus high recall can be produced by predicting the same class almost always, so evaluating cultural understanding requires precision and balanced metrics, not accuracy alone.
- WVS and PEW survey benchmarks can serve as standardized pre-deployment tests of cultural representativeness for new models.
Reading between the lines
- An extension the paper does not test: because the WVS preprocessing codes 'not asked in survey' as zero before averaging, a portion of the cross-country variance used as the benchmark may be an artifact; re-running the variance comparison on countries where each question was actually asked would test whether the model-compression claim survives.
- A reader could test whether the models' country sensitivity is genuine by adding no-country baseline prompts; if the same permissive scores appear without a country name, part of the effect is an English-language prior rather than cultural knowledge.
- A natural next experiment is to fine-tune a small model on the WVS/PEW country–topic response distributions and then apply the same variance, ARI, and probe metrics, giving a direct test of whether the compression is removable by training.
- The paper reports aggregate country-level patterns; modeling within-country response distributions would clarify whether the models compress genuine individual-level disagreement or only cross-country means.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper asks whether small open-source LLMs reproduce cross-cultural moral attitudes measured by the World Values Survey (Wave 7) and the Pew Global Attitudes Project (2013). For five models (GPT-2 Medium/Large, OPT-125, Qwen-0.5B, BLOOM-560M), it computes country-topic moral scores from token log-probabilities and then applies three evaluations: comparison of topic-level variances between model scores and survey scores, cluster alignment via ARI/AMI, and direct comparative prompting. The paper concludes that models assign higher moral acceptability, produce systematically lower cross-country variance, and show low alignment with empirical survey patterns, interpreting this as a homogenized, WEIRD-leaning moral outlook.
Significance. If the results were reliable, the finding that small open-source LLMs compress cross-cultural moral variation and skew toward permissive judgments would be a useful contribution to cultural-bias evaluation. The paper's structure is sensible: it uses public survey datasets and open models, combines variance, clustering, and prompting methods, and reports per-model results in tables. These strengths, however, are outweighed by a load-bearing data-processing error: the WVS non-response imputation described in Section 3.2 directly distorts the empirical variance baseline that the central claim compares against. The paper does not flag this in its limitations, and the main conclusion is therefore not supported by the reported analyses. The study also tests only models up to 774M parameters, so the title-level generalization to 'large language models' is premature.
major comments (3)
- [Section 3.2, Section 4, Table 3] The replacement of all WVS non-response codes (-1, -2, -4, -5, including 'Not asked in survey') with 0 is not neutral. Valid WVS responses range from 1 to 10, so 0 lies below every real response; after normalization, a country-topic mean that includes these zeros moves toward 'never justifiable.' For topics with structural non-response, such as 'Not asked in survey,' whole countries are effectively scored as strongly disapproving rather than as missing. This artificially inflates cross-country variance for precisely the morally controversial topics (e.g., homosexuality, sex before marriage) that drive the variance-compression conclusion. The statement in Section 3.2 that 'a replacement value of 0 ensures that non-responses do not influence the computed averages' is false: zeros are included in the averaging and pull country means downward. Because the main claim in Section 4 and Table 3 compares model variances against this inflated empirical baseline, the conclusion that 'models systematically produce lower variance than survey data' is not established. The cluster-alignment and direct-probing analyses inherit the same distorted country-topic means. Section 6 does not mention this decision. A reanalysis that excludes non-responses or uses appropriate missing-data handling is required before the variance comparison can be interpreted.
- [Section 3.3, Section 5.4, Abstract] The paper's title, abstract, and conclusion speak about 'large language models' and 'current LLMs,' but all evaluated models are at most 774M parameters (GPT-2 Large, BLOOM-560M, Qwen-0.5B, OPT-350M). The conclusion that 'no remarkable differences were found regardless of model size or multilinguality' is unsupported by this narrow size range and by the absence of any modern instruction-tuned or frontier-scale model. Even if the results hold for these specific models, they do not license the broad generalization stated in the abstract and conclusion. The claims should be restricted to the tested small open-source models, or the experimental scope should be expanded.
- [Section 3.5.2, Tables 6-8] The cluster-alignment analysis selects the number of clusters K from the survey data (via highest silhouette score) and then imposes that same K on the model-generated scores. ARI and AMI are sensitive to the number of clusters, so a K chosen for one data space can make the comparison artificially unfavorable or favorable in the other. No sensitivity analysis over K is reported, and no confidence intervals or significance tests are given for the near-zero ARI/AMI values in Tables 6-8. The conclusion that model-based clusterings 'do not faithfully capture' the empirical patterns therefore needs additional support, at minimum a K-sensitivity analysis and some measure of uncertainty for the alignment metrics.
minor comments (6)
- [Section 3.5.3, Table 1] The direct-probing method says the comparative token takes pairs such as 'similar/dissimilar,' but Table 1 lists only pairs such as 'always justifiable / never justifiable' and 'right / wrong.' The mapping between these token pairs and the binary 'similar vs different' outcome is never specified, which makes Method 3 hard to reproduce.
- [Sections 5.4 and 5.5] The manuscript contains two nearly identical subsections titled 'Conclusion' (5.4 and 5.5). One should be removed, and the recommendations that appear only in 5.5 should be integrated into a single concluding section.
- [Section 3.1] The statement that the Pew 2013 questionnaire was 'administered in English' is surprising and should be verified; Pew Global Attitudes surveys are generally fielded in local languages. If the survey was translated, the sentence is factually incorrect and should be corrected.
- [Section 6, Section 3.5.3] The limitations section says that 'due to computational limits, we randomly selected topics in Method 3,' but Section 3.5.3 says the procedure is applied 'For each topic.' The text should clarify which topics were used in the direct-probing analysis and how the random selection was performed.
- [Section 4.2, Table 10] Ten chi-squared tests are reported in Table 10 without any correction for multiple comparisons. Under the null, one would expect roughly 0.5 significant results at the 5% level and 0.1 at the 1% level in ten tests; the single WVS result with p=0.004 and the single PEW result with p=0.032 are therefore weaker evidence of alignment than the text implies. A multiple-testing correction or a note about the false-positive rate is needed.
- [Table 5] The column headings in Table 5 are confusing: 'Model Survey Survey Var . Survey Mean Model Var . Model Mean Topic Var . Diff' contains an odd stray period and an unclear 'Model Survey' label. The heading should be reformatted to distinguish survey columns from model columns.
Circularity Check
No circular derivation; empirical benchmark comparison is self-contained, with a minor non-load-bearing self-citation and a serious but non-circular data-processing concern.
full rationale
The paper does not derive its conclusions from its inputs by construction. Model-generated moral scores are obtained by log-probability contrasts of frozen open-source LLMs (Section 3.4), and no model parameter is fitted to WVS or PEW; the reported comparisons (variance, cluster alignment, direct probing) are against external survey benchmarks. The central 'variance compression' result is therefore an empirical observation, not an algebraic consequence of the scoring scheme. The self-citation to the authors' prior work [18] for the probing recipe is not load-bearing: the exact prompt templates, token pairs, and log-probability subtraction are specified in the manuscript itself, so the conclusion does not rest on the cited paper's authority. The most serious concern is instead a validity threat: Section 3.2 replaces WVS non-responses (-1, -2, -4, -5) with 0 on a 1-10 scale, which pulls country-topic means downward and, after normalization, makes 'not asked in survey' countries appear maximally disapproving, inflating the empirical variance baseline used in the first method. However, that is a data-processing and measurement error rather than a circular step: the model outputs are not defined in terms of the survey scores, and the paper's claim is not entailed by the imputation rule. Accordingly, no circularity step meets the standard of Eq. X = Eq. Y by construction or fitted-parameter-renamed-as-prediction; score 2 reflects the minor self-citation and the unflagged methodological risk, not a circular derivation.
Assumptions & free parameters
free parameters (3)
- Missing value replacement constant =
0
- Number of clusters K =
Silhouette-selected on survey data, then applied to model scores
- Token pair set and prompt templates =
Five token pairs and two prompt templates
assumptions (4)
- domain assumption Country-level mean of survey responses represents cultural moral attitude
- domain assumption English prompt templates elicit comparable moral judgments across all countries
- domain assumption Log-probability differences across token pairs are commensurable and can be averaged
- ad hoc to paper Silhouette-selected K from survey clusters transfers to model score space
Cite this review
Pith. "Pith review of Do Large Language Models Understand Morality Across Cultures?." pith.science (2026). https://pith.science/paper/BIJHZQKK
@misc{pith2026250721319,
author = {Pith},
title = {Pith review of: Do Large Language Models Understand Morality Across Cultures?},
year = {2026},
howpublished = {\url{https://pith.science/paper/BIJHZQKK}},
note = {Machine review of arXiv:2507.21319}
}
read the original abstract
Recent advancements in large language models (LLMs) have established them as powerful tools across numerous domains. However, persistent concerns about embedded biases, such as gender, racial, and cultural biases arising from their training data, raise significant questions about the ethical use and societal consequences of these technologies. This study investigates the extent to which LLMs capture cross-cultural differences and similarities in moral perspectives. Specifically, we examine whether LLM outputs align with patterns observed in international survey data on moral attitudes. To this end, we employ three complementary methods: (1) comparing variances in moral scores produced by models versus those reported in surveys, (2) conducting cluster alignment analyses to assess correspondence between country groupings derived from LLM outputs and survey data, and (3) directly probing models with comparative prompts using systematically chosen token pairs. Our results reveal that current LLMs often fail to reproduce the full spectrum of cross-cultural moral variation, tending to compress differences and exhibit low alignment with empirical survey patterns. These findings highlight a pressing need for more robust approaches to mitigate biases and improve cultural representativeness in LLMs. We conclude by discussing the implications for the responsible development and global deployment of LLMs, emphasizing fairness and ethical alignment.
Reference graph
Works this paper leans on
-
[18]
H. Mohammadi, E. Papadopoulou, Y . F. Meijer, and A. Bagheri. Explor- ing cultural variations in moral judgments with large language models. arXiv preprint arXiv:2506.12433, 2025
-
[1]
A. Arora, L. Kaffee, and I. Augenstein. Probing pre-trained lan- guage models for cross-cultural differences in values. arXiv preprint arXiv:2203.13722, 2022. URL https://doi.org/10.48550/arxiv.2203. 13722
-
[2]
E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? Pro- ceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, 2021. doi: 10.1145/3442188.3445922. URL https://dl.acm.org/doi/10.1145/3442188.3445922
arXiv 2021
-
[3]
Y . Benkler, D. Mosaphir, S. Friedman, A. Smart, and S. Schmer-Galunder. Assessing llms for moral value pluralism. arXiv (Cornell University),
-
[4]
T. Bolukbasi, K.-W. Chang, J. Y . Zou, V . Saligrama, and A. T. Kalai. Quantifying and reducing stereotypes in word embeddings. arXiv, 2016. doi: 10.48550/arxiv.1606.06121. URL https://arxiv.org/abs/1606.06121
-
[5]
X. Du, Z. Yu, S. Gao, D. Pan, Y . Cheng, Z. Ma, R. Yuan, X. Qu, J. Liu, T. Zheng, X. Luo, G. Zhou, B. Yuan, W. Chen, J. Fu, and G. Zhang. Chi- nese Tiny LLM: Pretraining a Chinese-Centric Large Language Model. arXiv (Cornell University), 4 2024. doi: 10.48550/arxiv.2404.04167. URL https://arxiv.org/abs/2404.04167
-
[6]
J. Graham, P. Meindl, E. Beall, K. M. Johnson, and L. Zhang. Cultural differences in moral judgment and behavior, across and within societies. Current Opinion in Psychology, 8:125–130, 2016. doi: 10.1016/j.copsyc. 2015.09.007. URL https://doi.org/10.1016/j.copsyc.2015.09.007
doi:10.1016/j.copsyc 2016
-
[7]
C. W. Haerpfer, P. Bernhagen, R. F. Inglehart, and C. Welzel. World Values Survey: Round Seven - Country-Pooled Datafile Version. Institute for Comparative Survey Research, Vienna, 2022. URL http://www. worldvaluessurvey.org/WVSDocumentationWV7.jsp
work page 2022
Show all 31 references
-
[8]
J. Haidt. The emotional dog and its rational tail: A social intuitionist approach to moral judgment. Psychological Review, 108(4):814–834,
-
[9]
Henrich, S
J. Henrich, S. J. Heine, and A. Norenzayan. The weirdest people in the world? Behavioral and brain sciences, 33(2-3):61–83, 2010
2010
-
[10]
R. L. Johnson, G. Pistilli, N. Menédez-González, L. D. D. Duran, E. Panai, J. Kalpokiene, and D. J. Bertulfo. The ghost in the machine has an american accent: value conflict in gpt-3. arXiv.org, mar 2022. URL https://arxiv.org/abs/2203.07785
2022 arXiv
-
[11]
Karpouzis
K. Karpouzis. Plato’s shadows in the digital cave: Controlling cultural bias in generative ai. Electronics, 13(8):1457, 2024. doi: 10.3390/ electronics13081457. URL https://doi.org/10.3390/electronics13081457
2024 doi
- [12]
-
[13]
Lazarenko and T
D. Lazarenko and T. Bonald. Pairwise adjusted mutual information,
-
[14]
Le Scao, A
T. Le Scao, A. Fan, C. Akiki, E. Pavlick, S. Ili´c, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V . d. Moral, O. Ruwase, R. Bawden, and M. J. ... Nelson. Bloom...
2022 arXiv
-
[15]
C. C. Liu, F. Koto, T. Baldwin, and I. Gurevych. Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings. arXiv, 2023. URL https://arxiv.org/abs/2309. 08591. arXiv:2309.08591
2023 arXiv
-
[16]
Mishra, G
A. Mishra, G. Nayak, S. Bhattacharya, T. Kumar, A. Shah, and M. Foltin. Llm-guided counterfactual data generation for fairer ai. In Companion Proceedings of the ACM on Web Conference 2024 , WWW ’24, page 1538–1545, New York, NY , USA, 2024. Association for Computing Machinery....
2024
-
[17]
Mohammadi, A
H. Mohammadi, A. Bagheri, A. Giachanou, and D. L. Oberski. Explain- ability in practice: A survey of explainable nlp across various domains. arXiv preprint arXiv:2502.00837, 2025
2025 arXiv
-
[19]
Mohammadi, T
H. Mohammadi, T. Shahedi, P. Mosteiro, M. Poesio, A. Bagheri, and A. Giachanou. Assessing the reliability of llms annotations in the context of demographic bias and model explanation. arXiv preprint arXiv:2507.13138, 2025
2025 arXiv
-
[20]
Nazaretsky, S
T. Nazaretsky, S. Hershkovitz, and G. Alexandron. Kappa learning: A new item-similarity method for clustering educational items from response data. 04 2020. URL https://eric.ed.gov/?id=ED599209
2020
-
[21]
Nemani, Y
P. Nemani, Y . D. Joel, P. Vijay, and F. F. Liza. Gender bias in transformers: A comprehensive review of detection and mitigation strategies. Natural Language Processing Journal, 6:100047, 2024. doi: 10.1016/j.nlp.2023. 100047. URL https://doi.org/10.1016/j.nlp.2023.100047
2024
-
[22]
S. U. Noble. Algorithms of Oppression: How Search Engines Reinforce Racism. NYU Press, New York, NY , 2018. ISBN 978-1479837243. URL https://nyupress.org/9781479837243/algorithms-of-oppression/
2018
-
[23]
Radford, J
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019. URL https://api.semanticscholar.org/CorpusID:160025533
2019
- [24]
-
[25]
big three
R. A. Shweder, N. C. Much, M. Mahapatra, and L. Park. The "big three" of morality (autonomy, community, divinity) and the "big three" explanations of suffering. In A. Brandt and P. Rozin, editors, Morality and Health, pages 119–169. Routledge, 1997. URL https://psycnet.apa. or...
1997
-
[26]
L. Wang, X. Chen, and X. Deng. Prompt engineering in consistency and reliability with the evidence-based guideline for llms. npj Digital Medicine, 7:41, 2024. doi: 10.1038/s41746-024-01029-4. URL https: //doi.org/10.1038/s41746-024-01029-4
2024 doi
-
[27]
Zhang, S
S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V . Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022
2022 arXiv
-
[28]
Zou and L
J. Zou and L. Schiebinger. Ai can be sexist and racist—it’s time to make it fair. Nature, 559(7714):324–326, 2018. doi: 10. 1038/d41586-018-05707-8. URL https://www.nature.com/articles/ d41586-018-05707-8. A Appendix A.1 Most controversial WVS topics according to models Topic ...
2018
-
[2001]
doi: 10.1037/0033-295X.108.4.814
-
[2021]
URL https://arxiv.org/abs/2103.12641
- [2023]
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.