{"id":"5ee71bb8-e31c-4668-b227-987f829a1935","arxiv_id":"2502.02451","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"In Chinese, machine translation and local dictionaries measure moral foundations poorly; fine-tuned multilingual models and especially Llama3.1-8b do better, but still lose cultural nuance.","lead":"This paper tests four ways to automatically detect moral values in Chinese text, using English tools as a starting point. It finds large language models are the most accurate and data-efficient, but still miss culturally specific moral meanings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fine-tuning with 80% of CCV and then testing on the held-out 20% creates a train/test leakage risk that is not discussed; the reported 0.74 F1 on CCV may partly reflect benchmark memorization rather than cross-language transfer.","rationale":"The reader's weakest assumption focused on the validity of the CCV ground-truth mapping from Chinese core socialist values onto MFT labels. That is a legitimate concern (the paper itself acknowledges annotator disagreement and expert-coder bias). But the more directly load-bearing issue for the paper's headline claim is that the CCV benchmark may not be a clean held-out evaluation of cross-language transfer, because 80% of the same corpus is used for fine-tuning and the test split appears to be drawn from the same domains without reported deduplication. The paper describes 'We sample 20% of the CCV as the primary benchmarking dataset... stratified on the values' and 'The remainder is used as training data.' Neither the split method, the deduplication, nor the domain overlap is discussed. Since the claim 'simply fine-tuning with English annotated data can achieve strong performance on non-English corpora' is supported almost entirely by the CCV F1=0.74 result, leakage would directly undermine it. The reader's concern about label-mapping validity is real but would affect all methods roughly equally, whereas split leakage specifically inflates the fine-tuned methods that are the paper's headline contribution. I therefore recommend keeping the CONDITIONAL verdict, with the condition now explicitly requiring a checked CCV split and an out-of-domain robustness test. I agree partially with the reader: the ground-truth mapping is a secondary concern, but the primary risk is benchmarking contamination.","tokens_in":23224,"tokens_out":1807,"duration_ms":15933,"concrete_test":"Re-run the fine-tuned LLM and XLM-T on an out-of-domain Chinese MF benchmark (e.g., CCS, or new Chinese news from different outlets and time periods than CCV) without any additional fine-tuning, and compare F1. If the CCV F1 drops substantially (e.g., from 0.74 to below 0.65) while MFV/CCS remain stable, the CCV number is inflated by train/test overlap or domain memorization. Additionally, check near-duplicate overlap between the 80% training and 20% test splits using normalized document similarity; report the proportion of test documents with a near-duplicate in the training split.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that a fine-tuned Llama3.1-8b reaches F1=0.74 on real-world Chinese news text using only English-annotated data plus machine translation. The load-bearing condition is that the CCV benchmark provides an unbiased estimate of that claim. The paper does not explain the split, but the sequence in the Data and Methods sections implies that 80% of CCV is used for fine-tuning and the remaining 20%, sampled 'stratifying on the values,' is used for benchmarking. Both the XLM-T incremental training curves (Figure 1, up to 50 batches of 100 records = 5,000 records) and the Llama3.1 fine-tuning with 20 batches of 50 records begin from this same 80% CCV share. If the 20% test split was drawn after stratifying by class from the same news domains (wenming.cn, agzy.youth.cn, news.sohu.com, CMOS corpus), then the test set is near-duplicate in style, domain, and possibly near-identical content to training material. Moral foundation labels are highly sensitive to topic and framing, so a model that memorizes domain-label associations (e.g., government-related text -> loyalty) can score well on CCV without possessing a general cross-language MFT capability. The paper's own qualitative analysis shows that the model still misclassifies culturally nuanced cases on this same benchmark, which is consistent with partial memorization of the benchmark's dominant patterns rather than robust transfer. The absence of any held-out domain, timestamp, or content-based deduplication, and the absence of a same-domain English (or non-Chinese) control, makes the 0.74 figure ambiguous as evidence for the headline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares four families of automated moral-foundation (MF) measurement for Chinese text: machine translation plus established English tools (lexicons, FrameAxis, MoralBert/Mformer), a local Chinese lexicon (C-MFD2), a fine-tuned multilingual encoder-only model (XLM-T), and decoder-only LLMs (Llama3.1-8b and 70b) under prompting and fine-tuning. It benchmarks on three Chinese datasets (translated MFV, CCS, CCV) and reports that LLMs, particularly when fine-tuned on English data augmented with machine-translated Chinese, achieve the best F1 scores (CCV F1=0.74) and the highest data efficiency, while machine translation and local-lexicon approaches underperform. The paper also presents qualitative examples of cultural misalignment and a small Italian replication study, and it argues for human-in-the-loop validation of automated MF measurement.","tokens_in":23535,"tokens_out":8843,"duration_ms":86498,"significance":"If the reported results hold, the paper provides a practically useful comparison of four method families for multilingual moral measurement and a credible case that open-weight LLMs are a data-efficient option. Strengths include the breadth of the comparison, the use of multiple benchmarks, the transparent code release, the qualitative analysis of translation loss, and the Italian extension. However, the central CCV-based claims are weakened by evaluation-design issues that need to be addressed before the ranking can be taken at face value: a potentially leaking train/test split, few-shot examples drawn from the benchmark itself, an unvalidated gold-standard mapping, and the absence of uncertainty quantification.","major_comments":[{"comment":"The CCV benchmark is a single 20% holdout sampled from the same four news domains that supply the remaining 80% used for fine-tuning, with no reported content-based deduplication, timestamp split, or domain holdout. Since moral foundation labels are highly sensitive to topic and framing, models fine-tuned on the 80% share can learn domain-label associations that transfer to the 20% share without constituting a general cross-language moral measurement capability. The paper's own qualitative examples on CCV show culturally nuanced misclassifications even for the best model, which is consistent with partial benchmark memorization. This concern is load-bearing for the headline CCV F1 of 0.74 and the data-efficiency comparison; it should be addressed with a held-out-domain or content-deduplicated evaluation, or at least a quantitative overlap analysis.","section":"Data; Table 1; Figures 1 and 2"},{"comment":"The appendix states that few-shot examples (N=3) were purposefully sampled from the benchmarking dataset in the local language. This means every Llama3.1 few-shot condition in Tables 5 and 6 includes labeled instances drawn from the exact evaluation set, so the model has access to test labels during prompting. This is a direct test-set leak that inflates the prompting-only and few-shot comparisons against lexicon and MT baselines. Few-shot examples should be drawn only from training or development data disjoint from all benchmark sets.","section":"Appendix Methods (Few-shot examples)"},{"comment":"The CCV gold standard is created by re-labeling an eight-category Chinese core socialist value scheme onto five MFT labels using five postgraduate native-speaker annotators and majority vote, and the appendix acknowledges annotator disagreement and expert-coder bias. No inter-annotator agreement statistic is reported, so the reliability of this contested mapping is unknown. In addition, the lenient multi-label evaluation criterion, which counts a prediction as correct if any predicted label matches the single gold label, rewards models that output several labels and cannot be compared directly with the per-class binary classifiers used for XLM-T. Because CCV is the only real-world benchmark used in the paper, these issues affect the absolute F1 values and the cross-method ranking; please report agreement statistics, analyze sensitivity to the evaluation criterion, and consider an alternative gold standard.","section":"Data (CCV; evaluation criterion); Appendix (labeling limitations)"},{"comment":"All reported F1 scores are point estimates from a single split and a single run, with no confidence intervals, significance tests, or repeated-seed variability. Some of the claims that separate the approaches are based on small absolute differences (e.g., CCS rows in Table 5 mostly fall between 0.80 and 0.82; Table 6 reports an 8b/70b CCV difference of 0.42 vs 0.60 under Chinese few-shot prompting). Without uncertainty quantification, it is difficult to know which cross-method or prompt-language differences are reliable. The headline LLM-versus-XLM-T gap is large (0.74 vs 0.63), but the data-efficiency and fine-grained per-foundation comparisons need bootstrap or repeated-split estimates.","section":"Results (Tables 2-6); Discussion"}],"minor_comments":[{"comment":"The Data section states that CCV includes 6,994 sentences, while Table 1 sums to 7,492 for CCV; please reconcile the counts.","section":"Data"},{"comment":"The sentence 'we machine translate benchmarking datasets except MFV to Chinese' appears to be a typo: the pipeline applies English measurements to translated text, so the non-English benchmarks should be translated to English, not to Chinese.","section":"Methods (Machine Translation)"},{"comment":"The text refers to 'MFD and MFD 1.0' when the results tables report MFD and MFD2; the nomenclature should be corrected to match the actual dictionaries used.","section":"Methods (Machine Translation)"},{"comment":"The caption of Table 7 spells the dataset as 'MoralCovnITA'; this should be 'MoralConvITA' as used elsewhere.","section":"Appendix (Table 7)"},{"comment":"The prompt template contains 'you labels here' in the output format; this should be 'your labels here'.","section":"Appendix (Prompt)"}],"recommendation":"major_revision","confidential_remarks":"Potential conflict of interest: the first author co-authored C-MFD2 and CCS, which are used as the local-lexicon baseline and as one of the benchmarks. This is not a methodological circularity, but the paper should explicitly disclose the relationship and, ideally, include at least one benchmark not developed by the authors. The manuscript is otherwise well within the scope of a computational linguistics or computational social science journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Useful paper, but the headline number is shakier than the prose admits. The CCV test set is a 20% split from the same corpus used to fine-tune the models, and the appendix says few-shot examples were sampled from the benchmarking dataset. That second point is a direct test-set leak for the prompting conditions, and the first inflates the 0.74 F1 claim through domain-label memorization. Neither issue is discussed in the paper.\n\nWhat is genuinely new: a systematic Chinese case study comparing machine translation plus English tools, a local lexicon, XLM-T, and Llama3.1-8b/70b in one framework. The re-labeled CCV benchmark and the typology of cultural loss (idioms, contextual meaning, political euphemisms, Confucian concepts) are useful contributions. The qualitative analysis of cultural misalignment—filial piety read as care rather than authority, government-related text read as authority or fairness rather than loyalty—is the most valuable part of the paper and will be cited.\n\nThe main ranking, LLMs > multilingual encoders > MT/lexicons, is plausible and mostly supported by the tables. On MFV and CCS, which are independent benchmarks, the fine-tuned Llama3.1-8b beats XLM-T, so the ranking is not solely an artifact of the CCV split. But the absolute CCV numbers should be treated with caution: same-domain train/test, no deduplication, no held-out domain, and no uncertainty quantification. The lenient multi-label evaluation criterion also favors multi-label LLM outputs over single-label binary classifiers, which is a methodological concern that deserves at least a sensitivity analysis. The CCV ground truth is a re-labeling by five postgraduate annotators; the paper acknowledges disagreement and expert bias, but the mapping from Chinese core socialist values to MFT remains a weak link.\n\nThe Italian appendix is a nice check, but it is small, domain-specific, and shows the same cultural misalignment pattern, so it does not rescue the CCV leakage problem. The paper would be strengthened by evaluating on a truly held-out Chinese domain, reporting confidence intervals or significance tests, and clarifying or removing the few-shot test-set examples.\n\nBottom line: this deserves serious peer review. The empirical comparison and the qualitative findings are worth publishing after major revision. I would not use the 0.74 CCV figure as evidence of robust cross-language transfer until the leakage issues are addressed.","headline":"A genuinely useful Chinese case study of cross-lingual moral foundation measurement, but the headline 0.74 F1 on CCV is weaker than it looks because the test set comes from the same corpus used for fine-tuning and few-shot examples appear to be drawn from the benchmark itself.","tokens_in":24112,"tokens_out":2487,"would_cite":true,"duration_ms":26071,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that fine-tuned open-weights LLMs, trained on English moral-annotation data machine-translated into Chinese, measure moral foundations in Chinese text more accurately and far more data-efficiently than machine…","keywords":["moral foundation theory","cross-language measurement","Chinese moral text","large language models","multilingual transfer learning","machine translation","moral foundation dictionaries","deductive coding"],"falsifier":"Have a fresh team of native Chinese annotators, blind to the CCV mapping, label a held-out sample of the same news sentences using only the five moral-foundation definitions; if their labels disagree with the CCV majority labels at the level usually considered unreliable, the benchmark's ground truth cannot support the reported ranking of methods.","tokens_in":22998,"feed_emoji":"🧭","tokens_out":8878,"duration_ms":76732,"temperature":0.7,"pith_summary":"Using Chinese as a test case, this paper tries to establish that large language models are the most accurate and data-efficient way to measure moral foundations in non-English text. Its central comparison shows that a fine-tuned open-weights LLM, trained on English moral-annotation data and on machine-translated Chinese versions of that data, reaches a weighted F1 of about 0.74 on real-world Chinese news, outperforming machine translation plus English classifiers, a locally built Chinese moral dictionary, and a multilingual encoder-only model. The same paper argues that machine translation and local lexicons systematically lose cultural information, that multilingual encoder-only models need on the order of two thousand local labels to become reliable, and that even the best LLM misses culturally specific moral nuances. A sympathetic reader would take away that cross-language moral measurement can be bootstrapped from English resources, but only with human validation.","feed_headline":"One fine-tuned LLM scores Chinese moral text at F1 0.74","feed_subtitle":"No Chinese human labels needed: English data plus machine-translated copies trains a model that beats all baselines.","key_machinery":"The load-bearing mechanism is transfer learning by fine-tuning: the LLM is first fine-tuned on English moral-foundation annotation corpora (Twitter, Reddit, and news), then fine-tuned again on those same annotations machine-translated into Chinese, so the model can map moral language across languages without any human labels written in the target language. The paper contrasts this with three other pipelines: translate-then-classify with English dictionaries, FrameAxis embeddings, and the English classifier Mformer; a locally developed Chinese moral dictionary C-MFD2 with word-count and fastText similarity scoring; and the multilingual encoder-only model XLM-T fine-tuned on English data plus incremental Chinese batches. The result that carries the argument is the weighted-F1 gap on the real-world CCV benchmark, where the translated-augmented LLM (0.74) beats XLM-T (0.63) and all lexicon/translation pipelines, and the data-efficiency gap measured by how many local-language batches each model needs to reach a stable score.","core_discovery":"The paper's central claim is that, for measuring moral foundations in Chinese, a decoder-only LLM fine-tuned on existing English annotations and then on machine-translated Chinese versions of those annotations is both more accurate and dramatically more data-efficient than the alternatives. On the real-world CCV news benchmark, the best Llama3.1-8b configuration reaches weighted F1 0.74, while the multilingual encoder-only model XLM-T reaches 0.63 and requires more than two thousand locally annotated records to reach comparable strength; machine-translation-plus-English-tool pipelines and the local Chinese dictionary C-MFD2 fall below both, and the English Mformer model scores no better than random on the loyalty foundation after translation. The paper also finds that using English prompts beats using Chinese prompts for this task, that fine-tuning with small batches of local-language data can degrade performance, and that culturally distinct values such as authority, loyalty, and sanctity—along with Confucian concepts like filial piety—remain the weak point of every automated approach, motivating its call for human-in-the-loop validation.","pith_inferences":["If the Chinese results generalize, the same recipe of English annotations plus machine-translated augmentation could be tried for other languages, but the single-language design leaves open whether translation quality and script differences change the ranking.","The CCV benchmark itself is a re-labeling of a Chinese core-socialist-value scheme; the reported F1 hierarchy is therefore conditional on that mapping, and a different expert mapping might narrow or reorder the gaps between approaches.","The consistent English-prompt advantage hints that the models' moral representations are anchored in English-centric training; prompting that explicitly asks for local cultural reasoning, or fine-tuning on culturally diverse rationales, is a natural next experiment.","A direct testable extension would apply the same pipeline to a low-resource language with weaker machine translation, predicting that the augmentation gain shrinks as translation quality drops."],"forward_implications":["Researchers studying morality in non-English text can start from English annotated data alone: fine-tuning an open LLM on English labels plus machine-translated copies of those labels yields strong Chinese performance with no local human annotation.","Machine-translation-first measurement is unreliable for fine-grained moral values, since cultural information loss can push specific foundations below random-chance F1, so translated text should not be silently fed to English moral classifiers.","When local-language labeled data are scarce, LLMs are the data-efficient choice; when plentiful local data exist, a smaller multilingual encoder can be competitive, so the recommended approach depends on annotation budget.","Culturally sensitive foundations remain the failure mode of every method, and fine-tuning with English data can actively worsen scores on values like loyalty and sanctity, which argues for culturally aware prompt design and human review in deployed systems.","Larger LLMs prompted in the local language outperform smaller ones in the zero/few-shot regime, so model scale and prompt language are practical levers before any fine-tuning is attempted."],"supporting_citations":[{"why":"Defines the five moral-foundation constructs and the claim of cross-cultural universality that motivates measuring them in non-English text.","marker":"Graham et al. 2013"},{"why":"Supplies the English Twitter moral-foundation corpus used for fine-tuning the language models.","marker":"Hoover et al. 2020"},{"why":"Supplies the English Reddit moral-foundation corpus used as training data.","marker":"Trager et al. 2022"},{"why":"Supplies the eMFD crowd-sourced dictionary and the English news corpus used for training and evaluation.","marker":"Hopp et al. 2021"},{"why":"Supplies Mformer, the strongest English classifier applied to machine-translated text, and the fine-tuning recipe adapted for XLM-T.","marker":"Nguyen et al. 2024"},{"why":"Provides the XLM-T multilingual encoder-only model that anchors the multilingual transfer-learning comparison.","marker":"Barbieri, Anke, and Camacho-Collados 2022"},{"why":"Presents the Llama3.1 models used for prompting, few-shot learning, and fine-tuning in the LLM approach.","marker":"Meta 2024"},{"why":"Provides the Chinese Core Values dataset (CCV), the real-world news benchmark whose re-labeled five-foundation labels ground the main F1 comparisons.","marker":"Liu et al. 2022"},{"why":"Supplies the C-MFD2 Chinese moral dictionary and the CCS reverse-annotated scenario benchmark.","marker":"Cheng and Zhang 2023"}],"fun_headline_variants":["Fine-tuned LLM scores Chinese moral text at F1 0.74","Data-efficient LLM for Chinese moral foundations beats all baselines","LLM fine-tuned on English reaches 0.74 F1 on Chinese moral text","Without Chinese labels, LLM hits 0.74 F1 on moral text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Chinese benchmark labels—especially the CCV re-labeling of a Chinese core-socialist-value scheme into five moral foundations by five postgraduate native speakers with majority vote—really represent the intended moral foundations; if that mapping is contested, every cross-method F1 comparison rests on shaky ground truth.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned LLM scores Chinese moral text at F1 0.74","Data-efficient LLM for Chinese moral foundations beats all baselines","LLM fine-tuned on English reaches 0.74 F1 on Chinese moral text","Without Chinese labels, LLM hits 0.74 F1 on moral text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001675,"raw_usage":{"total_tokens":6640,"prompt_tokens":939,"completion_tokens":5701,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":5616}},"tokens_in":555,"tokens_out":5701,"duration_ms":36300,"temperature":1.0,"reasoning_tokens":5616,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T12:04:38.875175+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a fresh team of native Chinese annotators, blind to the CCV mapping, label a held-out sample of the same news sentences using only the five moral-foundation definitions; if their labels disagree with the CCV majority labels at the level usually considered unreliable, the benchmark's ground truth cannot support the reported ranking of methods.","supporting_citations":[],"review_version":1}