Beyond English: Evaluating Automated Measurement of Moral Foundations in Non-English Discourse with a Chinese Case Study

Calvin Yixiang Cheng; Scott A Hale

REVIEW 4 major objections 5 minor 1 cited by

Beyond English: Evaluating Automated Measurement of Moral Foundations in Non-English Discourse with a Chinese Case Study

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that fine-tuned open-weights LLMs, trained on English moral-annotation data machine-translated into Chinese, measure moral foundations in Chinese text more accurately and far more data-efficiently than machine…

desk verdict A genuinely useful Chinese case study of cross-lingual moral foundation measurement, but the headline 0.74 F1 on CCV is weaker than it looks because the test set comes from the same corpus used for fine-tuning and few-shot examples appear to be drawn from the benchmark itself. read the letter →

arxiv 2502.02451 v3 pith:PNT7QDNQ submitted 2025-02-04 cs.CL cs.SI

Calvin Yixiang Cheng , Scott A Hale This is my paper · ORCID

classification cs.CLcs.SI

keywords moralfoundationtheorycross-languagemeasurementChinesetextlargelanguagemodelsmultilingualtransferlearningmachinetranslationdictionariesdeductivecoding

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Using Chinese as a test case, this paper tries to establish that large language models are the most accurate and data-efficient way to measure moral foundations in non-English text. Its central comparison shows that a fine-tuned open-weights LLM, trained on English moral-annotation data and on machine-translated Chinese versions of that data, reaches a weighted F1 of about 0.74 on real-world Chinese news, outperforming machine translation plus English classifiers, a locally built Chinese moral dictionary, and a multilingual encoder-only model. The same paper argues that machine translation and local lexicons systematically lose cultural information, that multilingual encoder-only models need on the order of two thousand local labels to become reliable, and that even the best LLM misses culturally specific moral nuances. A sympathetic reader would take away that cross-language moral measurement can be bootstrapped from English resources, but only with human validation.

What carries the argument

The load-bearing mechanism is transfer learning by fine-tuning: the LLM is first fine-tuned on English moral-foundation annotation corpora (Twitter, Reddit, and news), then fine-tuned again on those same annotations machine-translated into Chinese, so the model can map moral language across languages without any human labels written in the target language. The paper contrasts this with three other pipelines: translate-then-classify with English dictionaries, FrameAxis embeddings, and the English classifier Mformer; a locally developed Chinese moral dictionary C-MFD2 with word-count and fastText similarity scoring; and the multilingual encoder-only model XLM-T fine-tuned on English data plus incremental Chinese batches. The result that carries the argument is the weighted-F1 gap on the real-world CCV benchmark, where the translated-augmented LLM (0.74) beats XLM-T (0.63) and all lexicon/translation pipelines, and the data-efficiency gap measured by how many local-language batches each model needs to reach a stable score.

What would settle it

Have a fresh team of native Chinese annotators, blind to the CCV mapping, label a held-out sample of the same news sentences using only the five moral-foundation definitions; if their labels disagree with the CCV majority labels at the level usually considered unreliable, the benchmark's ground truth cannot support the reported ranking of methods.

Watch

Extended reading notes

Core claim

The paper's central claim is that, for measuring moral foundations in Chinese, a decoder-only LLM fine-tuned on existing English annotations and then on machine-translated Chinese versions of those annotations is both more accurate and dramatically more data-efficient than the alternatives. On the real-world CCV news benchmark, the best Llama3.1-8b configuration reaches weighted F1 0.74, while the multilingual encoder-only model XLM-T reaches 0.63 and requires more than two thousand locally annotated records to reach comparable strength; machine-translation-plus-English-tool pipelines and the local Chinese dictionary C-MFD2 fall below both, and the English Mformer model scores no better than random on the loyalty foundation after translation. The paper also finds that using English prompts beats using Chinese prompts for this task, that fine-tuning with small batches of local-language data can degrade performance, and that culturally distinct values such as authority, loyalty, and sanctity—along with Confucian concepts like filial piety—remain the weak point of every automated approach, motivating its call for human-in-the-loop validation.

Load-bearing premise

The load-bearing premise is that the Chinese benchmark labels—especially the CCV re-labeling of a Chinese core-socialist-value scheme into five moral foundations by five postgraduate native speakers with majority vote—really represent the intended moral foundations; if that mapping is contested, every cross-method F1 comparison rests on shaky ground truth.

Editorial extensions

If this is right

Researchers studying morality in non-English text can start from English annotated data alone: fine-tuning an open LLM on English labels plus machine-translated copies of those labels yields strong Chinese performance with no local human annotation.
Machine-translation-first measurement is unreliable for fine-grained moral values, since cultural information loss can push specific foundations below random-chance F1, so translated text should not be silently fed to English moral classifiers.
When local-language labeled data are scarce, LLMs are the data-efficient choice; when plentiful local data exist, a smaller multilingual encoder can be competitive, so the recommended approach depends on annotation budget.
Culturally sensitive foundations remain the failure mode of every method, and fine-tuning with English data can actively worsen scores on values like loyalty and sanctity, which argues for culturally aware prompt design and human review in deployed systems.
Larger LLMs prompted in the local language outperform smaller ones in the zero/few-shot regime, so model scale and prompt language are practical levers before any fine-tuning is attempted.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the Chinese results generalize, the same recipe of English annotations plus machine-translated augmentation could be tried for other languages, but the single-language design leaves open whether translation quality and script differences change the ranking.
The CCV benchmark itself is a re-labeling of a Chinese core-socialist-value scheme; the reported F1 hierarchy is therefore conditional on that mapping, and a different expert mapping might narrow or reorder the gaps between approaches.
The consistent English-prompt advantage hints that the models' moral representations are anchored in English-centric training; prompting that explicitly asks for local cultural reasoning, or fine-tuning on culturally diverse rationales, is a natural next experiment.
A direct testable extension would apply the same pipeline to a low-resource language with weaker machine translation, predicting that the augmentation gain shrinks as translation quality drops.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Useful paper, but the headline number is shakier than the prose admits. The CCV test set is a 20% split from the same corpus used to fine-tune the models, and the appendix says few-shot examples were sampled from the benchmarking dataset. That second point is a direct test-set leak for the prompting conditions, and the first inflates the 0.74 F1 claim through domain-label memorization. Neither issue is discussed in the paper.

What is genuinely new: a systematic Chinese case study comparing machine translation plus English tools, a local lexicon, XLM-T, and Llama3.1-8b/70b in one framework. The re-labeled CCV benchmark and the typology of cultural loss (idioms, contextual meaning, political euphemisms, Confucian concepts) are useful contributions. The qualitative analysis of cultural misalignment—filial piety read as care rather than authority, government-related text read as authority or fairness rather than loyalty—is the most valuable part of the paper and will be cited.

The main ranking, LLMs > multilingual encoders > MT/lexicons, is plausible and mostly supported by the tables. On MFV and CCS, which are independent benchmarks, the fine-tuned Llama3.1-8b beats XLM-T, so the ranking is not solely an artifact of the CCV split. But the absolute CCV numbers should be treated with caution: same-domain train/test, no deduplication, no held-out domain, and no uncertainty quantification. The lenient multi-label evaluation criterion also favors multi-label LLM outputs over single-label binary classifiers, which is a methodological concern that deserves at least a sensitivity analysis. The CCV ground truth is a re-labeling by five postgraduate annotators; the paper acknowledges disagreement and expert bias, but the mapping from Chinese core socialist values to MFT remains a weak link.

The Italian appendix is a nice check, but it is small, domain-specific, and shows the same cultural misalignment pattern, so it does not rescue the CCV leakage problem. The paper would be strengthened by evaluating on a truly held-out Chinese domain, reporting confidence intervals or significance tests, and clarifying or removing the few-shot test-set examples.

Bottom line: this deserves serious peer review. The empirical comparison and the qualitative findings are worth publishing after major revision. I would not use the 0.74 CCV figure as evidence of robust cross-language transfer until the leakage issues are addressed.

Referee Report

4 major / 5 minor

Summary. The paper compares four families of automated moral-foundation (MF) measurement for Chinese text: machine translation plus established English tools (lexicons, FrameAxis, MoralBert/Mformer), a local Chinese lexicon (C-MFD2), a fine-tuned multilingual encoder-only model (XLM-T), and decoder-only LLMs (Llama3.1-8b and 70b) under prompting and fine-tuning. It benchmarks on three Chinese datasets (translated MFV, CCS, CCV) and reports that LLMs, particularly when fine-tuned on English data augmented with machine-translated Chinese, achieve the best F1 scores (CCV F1=0.74) and the highest data efficiency, while machine translation and local-lexicon approaches underperform. The paper also presents qualitative examples of cultural misalignment and a small Italian replication study, and it argues for human-in-the-loop validation of automated MF measurement.

Significance. If the reported results hold, the paper provides a practically useful comparison of four method families for multilingual moral measurement and a credible case that open-weight LLMs are a data-efficient option. Strengths include the breadth of the comparison, the use of multiple benchmarks, the transparent code release, the qualitative analysis of translation loss, and the Italian extension. However, the central CCV-based claims are weakened by evaluation-design issues that need to be addressed before the ranking can be taken at face value: a potentially leaking train/test split, few-shot examples drawn from the benchmark itself, an unvalidated gold-standard mapping, and the absence of uncertainty quantification.

major comments (4)

[Data; Table 1; Figures 1 and 2] The CCV benchmark is a single 20% holdout sampled from the same four news domains that supply the remaining 80% used for fine-tuning, with no reported content-based deduplication, timestamp split, or domain holdout. Since moral foundation labels are highly sensitive to topic and framing, models fine-tuned on the 80% share can learn domain-label associations that transfer to the 20% share without constituting a general cross-language moral measurement capability. The paper's own qualitative examples on CCV show culturally nuanced misclassifications even for the best model, which is consistent with partial benchmark memorization. This concern is load-bearing for the headline CCV F1 of 0.74 and the data-efficiency comparison; it should be addressed with a held-out-domain or content-deduplicated evaluation, or at least a quantitative overlap analysis.
[Appendix Methods (Few-shot examples)] The appendix states that few-shot examples (N=3) were purposefully sampled from the benchmarking dataset in the local language. This means every Llama3.1 few-shot condition in Tables 5 and 6 includes labeled instances drawn from the exact evaluation set, so the model has access to test labels during prompting. This is a direct test-set leak that inflates the prompting-only and few-shot comparisons against lexicon and MT baselines. Few-shot examples should be drawn only from training or development data disjoint from all benchmark sets.
[Data (CCV; evaluation criterion); Appendix (labeling limitations)] The CCV gold standard is created by re-labeling an eight-category Chinese core socialist value scheme onto five MFT labels using five postgraduate native-speaker annotators and majority vote, and the appendix acknowledges annotator disagreement and expert-coder bias. No inter-annotator agreement statistic is reported, so the reliability of this contested mapping is unknown. In addition, the lenient multi-label evaluation criterion, which counts a prediction as correct if any predicted label matches the single gold label, rewards models that output several labels and cannot be compared directly with the per-class binary classifiers used for XLM-T. Because CCV is the only real-world benchmark used in the paper, these issues affect the absolute F1 values and the cross-method ranking; please report agreement statistics, analyze sensitivity to the evaluation criterion, and consider an alternative gold standard.
[Results (Tables 2-6); Discussion] All reported F1 scores are point estimates from a single split and a single run, with no confidence intervals, significance tests, or repeated-seed variability. Some of the claims that separate the approaches are based on small absolute differences (e.g., CCS rows in Table 5 mostly fall between 0.80 and 0.82; Table 6 reports an 8b/70b CCV difference of 0.42 vs 0.60 under Chinese few-shot prompting). Without uncertainty quantification, it is difficult to know which cross-method or prompt-language differences are reliable. The headline LLM-versus-XLM-T gap is large (0.74 vs 0.63), but the data-efficiency and fine-grained per-foundation comparisons need bootstrap or repeated-split estimates.

minor comments (5)

[Data] The Data section states that CCV includes 6,994 sentences, while Table 1 sums to 7,492 for CCV; please reconcile the counts.
[Methods (Machine Translation)] The sentence 'we machine translate benchmarking datasets except MFV to Chinese' appears to be a typo: the pipeline applies English measurements to translated text, so the non-English benchmarks should be translated to English, not to Chinese.
[Methods (Machine Translation)] The text refers to 'MFD and MFD 1.0' when the results tables report MFD and MFD2; the nomenclature should be corrected to match the actual dictionaries used.
[Appendix (Table 7)] The caption of Table 7 spells the dataset as 'MoralCovnITA'; this should be 'MoralConvITA' as used elsewhere.
[Appendix (Prompt)] The prompt template contains 'you labels here' in the output format; this should be 'your labels here'.

Circularity Check

1 steps flagged · score 4.0 of 10

The few-shot LLM evaluation is partly circular because in-context examples are drawn from the benchmark being scored; the fine-tuned result itself is not defined by its inputs.

fitted input called prediction [Appendix, Methods (Llama3.1); Table 5]
"Few-shot examples ( N = 3 ) were purposefully sampled from the benchmarking dataset in the local language."

Table 5 reports Llama3.1-8b few-shot F1 scores on MFV, CCS, and the 20% CCV benchmarking split, but the in-context demonstrations used to obtain those scores are sampled from the same benchmarking datasets. The benchmark labels are thus part of the predictor's input for the few-shot conditions; the reported 'predictions' are not independent out-of-sample measurements, and the LLM-versus-baseline comparison is not on equal footing because other approaches are not given test-set labels. This is not a full equation-level reduction—three exemplars cannot force the whole label distribution—but the F1 values in the few-shot columns are partially circular because they measure agreement with labels the model has been shown.

full rationale

The paper's central fine-tuned result (en x (en+zh), CCV F1=0.74) is not circular in the strongest sense: the model is fine-tuned on English corpora plus machine-translated English data and evaluated on the 20% CCV holdout, so no parameter is fitted to the benchmark labels, and the score is an empirical measurement rather than an identity. The XLM-T and Llama batch-training curves are also empirical, not fitted predictions. The main circularity-adjacent issue is the Appendix statement that few-shot examples were purposefully sampled from the benchmarking dataset; this contaminates the few-shot LLM columns in Table 5 and weakens the claimed comparison, but it does not make the fine-tuned LLM result equivalent to its inputs by construction. There is also minor self-citation: C-MFD2 and CCS come from Cheng and Zhang (2023), co-authored by the first author, and are used as evaluation inputs; this is not load-bearing for the LLM conclusion. Weighing these, the paper is not a circular derivation, but the few-shot evaluation practice and self-cited benchmarks warrant a score of 4 rather than 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central comparisons rest on benchmark validity and a few hand-chosen experimental settings, not on fitted parameters or invented constructs. The main uncharged input is the assumption that MFT and the Chinese core-value mapping yield valid ground-truth labels.

free parameters (3)

few-shot example count N = 3
Prompting experiments use 3 few-shot examples chosen by hand; LLM scores in Tables 5 and 6 depend on this choice.
local data batch size = 100 for XLM-T, 50 for Llama3.1-8b
Data-efficiency curves in Figures 1 and 2 are defined by arbitrary batch increments; different batch sizes would change the learning curves and the conclusions about data efficiency.
CCV benchmark split = 20% stratified holdout
All benchmark comparisons use a single 20% stratified split of CCV with no repeated seeds or confidence intervals, so reported F1 gaps may partly reflect split variance.

assumptions (3)

domain assumption The five moral foundations from MFT are universal and applicable to Chinese texts.
Introduction states MFT transcends languages and cultures; if this universality fails, the entire measurement task is mis-specified.
domain assumption Chinese core socialist values can be mapped onto the five moral foundations.
CCV labels are derived by re-labeling an eight-category core-value scheme (Liu et al. 2022) to MFT labels in the Data section; this mapping is not independently validated.
domain assumption Expert annotations are the ground truth for moral foundations in Chinese.
Benchmarks CCV and CCS rely on a small number of native expert annotators (three to five) and majority vote; the Appendix acknowledges annotator disagreement and expert-coder bias.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Beyond English: Evaluating Automated Measurement of Moral Foundations in Non-English Discourse with a Chinese Case Study." pith.science (2026). https://pith.science/paper/PNT7QDNQ

@misc{pith2026250202451,
  author       = {Pith},
  title        = {Pith review of: Beyond English: Evaluating Automated Measurement of Moral Foundations in Non-English Discourse with a Chinese Case Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PNT7QDNQ}},
  note         = {Machine review of arXiv:2502.02451}
}

read the original abstract

This study explores computational approaches for measuring moral foundations (MFs) in non-English corpora. Since most resources are developed primarily for English, cross-linguistic applications of moral foundation theory remain limited. Using Chinese as a case study, this paper evaluates the effectiveness of applying English resources to machine translated text, local language lexicons, multilingual language models, and large language models (LLMs) in measuring MFs in non-English texts. The results indicate that machine translation and local lexicon approaches are insufficient for complex moral assessments, frequently resulting in a substantial loss of cultural information. In contrast, multilingual models and LLMs demonstrate reliable cross-language performance with transfer learning, with LLMs excelling in terms of data efficiency. Importantly, this study also underscores the need for human-in-the-loop validation of automated MF assessment, as the most advanced models may overlook cultural nuances in cross-language measurements. The findings highlight the potential of LLMs for cross-language MF measurements and other complex multilingual deductive coding tasks.

Figures

Figures reproduced from arXiv: 2502.02451 by the authors.

**Figure 2.** Accumulated fine-tuning Llama3.1-8b with lo [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

**Figure 3.** Mapping Scheme of Chinese Core Value Dataset [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Qualitative analysis of cultural nuances. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Domain Specific Benchmarks for Evaluating Multimodal Large Language Models
cs.LG 2025-06 conditional novelty 3.0 of 10

A review paper that organizes domain-specific MLLM benchmarks into an eight-discipline taxonomy, with summary tables and performance highlights.

Reference graph

Works this paper leans on

14 extracted references · 9 canonical work pages · cited by 1 Pith paper

[1]

identify the moral foundation value only from the 5 given ones

work page
[2]

if the document expresses more than 1 foundation value, label all prominent values, but in total should be equal or less than 3 values.,→

work page
[3]

provide a brief rationale for the each labelling, which should be less than 20 words

work page
[4]

arXiv:2404.11553

Quantifying Multilingual Performance of Large Language Models Across Languages. arXiv:2404.11553. Liscio, E.; Dondera, A. E.; Geadau, A.; Jonker, C. M.; and Mu- rukannaiah, P. K. 2022. Cross-domain classification of moral val- ues. In 2022 Findings of the Association for Computational Lin- guistics: NAACL 2022, 2727–2745. Association for Computational Lin...

arXiv 2022
[5]

rationales should be in the same lanaguage as the document

work page
[6]

if the document does not express any of the 5 values, label it as 'none' and provide a brief rationale.,→

work page
[7]

if the document can not be labelled into any of the 5 values, label it as 'unknown' and provide a brief rationale.,→

work page
[8]

rationale

consider the Chinese cultural context of the document when labelling the values. You MUST respond with a brief rationale within 15 words, and the labels. save in the dictionary format: {"rationale": "reasons to explain your decision", "labels": "you labels here"} ,→ ,→ Here are the given documents for your task: Figure 3: Mapping Scheme of Chinese Core Va...

work page

Show all 14 references

[10]

labels the value in english,
[2013]

Advances in Neural Information Processing Sys- tems, 26

Distributed Representations of Words and Phrases and their Compositionality. Advances in Neural Information Processing Sys- tems, 26. Mokhberian, N.; Abeliuk, A.; Cummings, P.; and Lerman, K. 2020. Moral framing and ideological bias of news. InSocial Informatics: 12th Internat...

2020 arXiv
[2015]

Conneau, A.; Khandelwal, K.; Goyal, N.; Chaudhary, V .; Wenzek, G.; Guzman, F.; Grave, E.; Ott, M.; Zettlemoyer, L.; and Stoyanov, V

Moral foundations vignettes: A standardized stimulus database of scenarios based on moral foundations theory.Behavior research methods, 47(4): 1178–1198. Conneau, A.; Khandelwal, K.; Goyal, N.; Chaudhary, V .; Wenzek, G.; Guzman, F.; Grave, E.; Ott, M.; Zettlemoyer, L.; and St...

2019 arXiv
[2018]

In Proceedings of the International Conference on Language Resources and Eval- uation (LREC 2018)

Learning Word Vectors for 157 Languages. In Proceedings of the International Conference on Language Resources and Eval- uation (LREC 2018). Haidt, J. 2012. The righteous mind: Why good people are divided by politics and religion. Pantheon. Haidt, J.; and Graham, J. 2007. When ...

2018 arXiv
[2020]

moralConvITA

Machine Translation Quality: A comparative evaluation of SMT, NMT and tailored-NMT outputs. InProceedings of the 22nd annual conference of the European Association for Machine Trans- lation, 441–450. Stranisci, M.; De Leonardis, M.; Bosco, C.; and Patti, V . 2021. The expressi...

2021
[2024]

In Proceedings of the 2024 Joint International Conference on Computational Linguis- tics, Language Resources and Evaluation (LREC-COLING 2024), 6330–6340

Ethical Reasoning and Moral Value Alignment of LLMs De- pend on the Language We Prompt Them in. In Proceedings of the 2024 Joint International Conference on Computational Linguis- tics, Language Resources and Evaluation (LREC-COLING 2024), 6330–6340. Ahuja, K.; Diddee, H.; Had...

2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

identify the moral foundation value only from the 5 given ones

work page

[2] [2]

if the document expresses more than 1 foundation value, label all prominent values, but in total should be equal or less than 3 values.,→

work page

[3] [3]

provide a brief rationale for the each labelling, which should be less than 20 words

work page

[4] [4]

arXiv:2404.11553

Quantifying Multilingual Performance of Large Language Models Across Languages. arXiv:2404.11553. Liscio, E.; Dondera, A. E.; Geadau, A.; Jonker, C. M.; and Mu- rukannaiah, P. K. 2022. Cross-domain classification of moral val- ues. In 2022 Findings of the Association for Computational Lin- guistics: NAACL 2022, 2727–2745. Association for Computational Lin...

arXiv 2022

[5] [5]

rationales should be in the same lanaguage as the document

work page

[6] [6]

if the document does not express any of the 5 values, label it as 'none' and provide a brief rationale.,→

work page

[7] [7]

if the document can not be labelled into any of the 5 values, label it as 'unknown' and provide a brief rationale.,→

work page

[8] [8]

rationale

consider the Chinese cultural context of the document when labelling the values. You MUST respond with a brief rationale within 15 words, and the labels. save in the dictionary format: {"rationale": "reasons to explain your decision", "labels": "you labels here"} ,→ ,→ Here are the given documents for your task: Figure 3: Mapping Scheme of Chinese Core Va...

work page