{"id":"df3a1170-8d9b-408b-9d5d-7031093dc7dc","arxiv_id":"2507.09701","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A dynamic multilingual cultural evaluation framework shows that LLM cultural performance depends on both training data distribution and language-culture alignment, and that English-only evaluations hide severe cultural unfairness.","lead":"MCEval is a new evaluation framework that generates multilingual cultural questions by rephrasing existing cultural datasets, and it tests AI models across 13 cultures and 13 languages to measure cultural awareness and bias. The authors find that a model's cultural performance depends on both language-culture alignment and training data distribution, and that English-only evaluations can hide severe unfairness from cultural fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The leakage and fairness findings rest on uncalibrated accuracy gaps between original and rephrased questions, with no human validation of the generated labels and no control for rephrasing difficulty; without such calibration, the claim that static benchmarks are unfair due to data leakage is not…","rationale":"The paper's headline finding is that static cultural benchmarks give unfair results due to data leakage, and that CultureBank-style fine-tuning hides severe unfairness. Both conclusions rest on accuracy gaps between original questions and LLM-generated rephrased questions. The weakest link is the inference from gap to leakage: no baseline shows that the gap would disappear for non-memorized items, and no human data shows that the rephrased items have the intended semantics. Because the entire pipeline (Extractor, Generator, Rephrase, Translator, Verifier) is LLM-based, with no reported pass rates or agreement, label error is a live alternative explanation. Because all scores are single-run pass@1 at temperature 0.7, the magnitude of the fairness effect is also inside noise. I do not think this makes the framework worthless; the dynamic multilingual construction and cross-lingual comparison infrastructure are real contributions, and the public repository is a positive step. But the central causal and fairness claims are conditional on a validation study and a leakage-control condition that are not present. The reader's verdict already identifies human validation and the uncalibrated gap as the key weakness, so I agree with that diagnosis and recommend that the conditional verdict remain unchanged. The proposed held-out leakage-control experiment and human agreement study would directly settle whether the original-vs-rephrased gap is memorization, rephrasing sensitivity, or label error.","tokens_in":970,"tokens_out":873,"duration_ms":85154,"concrete_test":"Run a leakage-control experiment: from cultural facts not present in TikTok or CrowS-Pairs, construct fresh Original, Counterfactual, and Confounder questions with the same pipeline. If accuracy on fresh Originals is as high as on benchmark Originals while rephrased accuracy remains low, leakage is supported; if fresh Originals also drop, the gap is generic rephrasing sensitivity. Complement with human agreement: two independent native speakers per language judge 300 sampled items (100 per type) for gold answer, counterfactual answer-flip validity, confounder irrelevance, and translation fidelity; recompute Table III on the unanimously confirmed subset and compare model confounder gaps with human confounder gaps.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V, RQ1 interprets the accuracy gap between Original, Counterfactual, and Confounder questions in Table III as evidence that models suffer from data leakage and that static cultural evaluations are unfair. This is load-bearing for the paper's central claim. The gap is uncalibrated: rephrased items differ in surface form and naturalness, so accuracy drops even without any exposure to the original corpora. Critically, the Confounder rephrasing is designed to preserve the answer while changing an irrelevant attribute, yet Table III shows substantial confounder gaps in many cells; unless human accuracy on those same confounder items is flat, the drop cannot be attributed to leakage. The Verifier Agent is itself an LLM, and the paper reports no pass rate, no human agreement, and no error analysis for generated labels or translations, so the counterfactual and confounder semantics are assumed, not demonstrated. All scores are single-run pass@1 at temperature 0.7 with no confidence intervals, so the 66.7% Swedish fairness drop in Figure 3 is a single observation. The framework may still be a useful dynamic stress test, but the conclusion that static benchmarks are leaked and unfair is not separable from label noise and wording sensitivity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"MCEval constructs multilingual cultural evaluation instances by running LLM agents (Extractor, Generator, Rephrase, Translator, Verifier) over CrowS-Pairs and TikTok, producing 39,897 awareness and 17,940 bias questions in 13 languages for 13 cultures. The authors evaluate Llama-3.3-70B and DeepSeek-V3 in native, English, and cross-lingual settings, and also compare CultureBank fine-tuned models. They report accuracy on original, counterfactual, and confounder questions; interpret original-vs-rephrased gaps as evidence of data leakage in static benchmarks; and argue that English-only evaluation masks cultural unfairness, citing a 66.7% degradation for Swedish after CultureBank fine-tuning.","tokens_in":20364,"tokens_out":5508,"duration_ms":62761,"significance":"If substantiated, MCEval would be a useful dynamic stress test for cultural knowledge and bias, and its multilingual and cross-cultural coverage is broader than most existing benchmarks. The paper contributes a large generated dataset, a detailed per-condition accuracy breakdown in Table III, and a clear operationalization of counterfactual versus confounder interventions. The main weakness is that the benchmark's validity rests on automatically generated labels and translations with no reported human validation, and the headline conclusions are drawn from single-run accuracy gaps without uncertainty quantification. The resource may still be valuable for relative model comparison, but the specific claims about training-data leakage and hidden unfairness require additional calibration.","major_comments":[{"comment":"The central claim that static cultural evaluations are unfair because of data leakage rests on accuracy gaps between Original and rephrased questions. These gaps are uncalibrated: counterfactual and confounder rewrites alter surface form, naturalness, and sometimes sentence length, so accuracy can drop even when a model has never seen the source corpora. The Original questions are themselves generated from cultural information rather than taken verbatim from CrowS-Pairs or TikTok, so a gap cannot directly establish exposure during training. The paper should provide a human baseline on the same items, and ideally a near-duplicate or leakage probe on the source corpora, and should report Verifier pass rates and human agreement on whether counterfactuals reverse the answer and confounders preserve it. Without this, the leakage interpretation is not separable from rephrasing sensitivity.","section":"Section V, RQ1; Table III"},{"comment":"All accuracy scores are single pass@1 runs at temperature 0.7 with no confidence intervals, random seeds, or significance tests. Several culture cells in Table I contain only 23-47 samples, so in those cells a 0.1 accuracy gap corresponds to only a handful of items. The headline 66.7% Swedish degradation in Figure 3(b) is a single observation, as are most of the 10-30% gaps invoked in the Introduction. The authors should report multiple runs, or at least bootstrap confidence intervals, and significance tests for the cross-scenario comparisons that support the fairness and leakage conclusions.","section":"Section IV-A; Figure 3; Table III"},{"comment":"The entire dataset is produced by an LLM pipeline whose only quality gate is the Verifier Agent, itself an LLM. No pass rates, human agreement, translation-adequacy scores, or error analyses are reported, yet Section III-A states that the pipeline 'guarantee[s] high-quality data.' Because the benchmark consists of these generated instances, the accuracy numbers partly measure the pipeline's ability to generate unambiguous questions and correct labels. The paper should include human validation of a stratified sample of generated items, covering label correctness, causal preservation in counterfactuals, irrelevance of confounders, and translation fidelity, and should report agreement statistics; this is a necessary condition for interpreting any model score on MCEval.","section":"Section III-A; Section III-B"}],"minor_comments":[{"comment":"The text refers to the 'Cowardcloud' dataset; this should be 'CrowS-Pairs.'","section":"Section III-B"},{"comment":"The evaluation protocol should specify the exact sampling procedure; 'pass@1 at temperature 0.7' is ambiguous regarding the number of samples and decoding strategy.","section":"Section IV-A"},{"comment":"In the French-culture example, the sentence 'Counterfactual Rephrasing only reduces it to 0.938' should read 'Confounder Rephrasing only reduces it to 0.938.'","section":"Section V, RQ2"},{"comment":"The Introduction says the data span '16 countries or regions,' while the abstract and Table I refer to 13 cultures and 13 languages; this mapping should be clarified.","section":"Section I"},{"comment":"Figure 3 compares Llama2-7B and Mixtral-56B, whereas the rest of the paper uses Llama-3.3-70B and DeepSeek-V3; the fairness experiment should state this model change explicitly.","section":"Section V-A"},{"comment":"The tables would benefit from per-cell sample sizes or a reminder that counts vary by culture as shown in Table I, since small denominators affect the interpretability of small accuracy differences.","section":"Tables II and III"}],"recommendation":"major_revision","confidential_remarks":"I see a useful benchmark resource here, but the strongest claims about data leakage and hidden unfairness are currently under-supported. A revision that adds human validation and uncertainty quantification would materially change my assessment; the main direction is worth pursuing within the journal's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Couple of things to know before you read it. The paper builds the largest multilingual cultural evaluation I've seen: 13 cultures, 13 languages, roughly 40k awareness and 18k bias questions, generated by an LLM pipeline and then rephrased into counterfactual and confounder variants. That resource, plus the finding that English-only evaluation of CultureBank-style fine-tuning hides big native-language failures, is the real contribution.\n\nThe good parts: the framework is well thought out. The verifier agents, the separation of awareness vs bias, the analysis across native/English/cross conditions — that's a lot of careful engineering. The CultureBank fairness comparison in Figure 3 is a good diagnostic demonstration. If the benchmark holds up, it's a useful stress-test for cultural robustness.\n\nThe soft spots are in the interpretation. The paper attributes the accuracy gap between original and rephrased questions to data leakage in existing benchmarks. That inference doesn't follow. Rephrased questions are harder in general — different surface form, slightly unnatural phrasing, changed answer distributions. The confounder questions are supposed to preserve the answer, yet confounder gaps show up all over Table III. Unless the authors show human accuracy is flat on those same confounder items, the gap could just be wording sensitivity, not leakage. The Verifier Agent is another LLM; no pass rate, no human agreement, no error analysis are reported. So the labels and translations are unvalidated.\n\nThe statistical basis is thin: single pass@1 runs at temperature 0.7, no confidence intervals, no significance tests. Several culture-language cells have fewer than 30 bias samples; a 10-point gap is a few questions. The 66.7% Swedish drop in Figure 3 is a single observation. The reader's concern about circularity isn't quite right — this is an evaluation pipeline, not a fitted model — but the lack of human baselines means the scores partly reflect the pipeline's own generation quality.\n\nNet: the resource and framework deserve serious referee time. The paper should be revised to (1) add human validation and human baselines on the rephrased questions, (2) report multiple seeds with intervals, and (3) reword the leakage claim to something like \"performance is not robust to rephrasing.\" If those are fixed, I'd cite it and use it.","headline":"Useful multilingual cultural stress test, but the data-leakage and unfairness claims need stronger statistical and human-validated support.","tokens_in":20923,"tokens_out":3955,"would_cite":true,"duration_ms":41851,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MCEval argues that static multilingual cultural benchmarks inflate LLM scores through data leakage, and that English-only cultural enhancement can deepen the unfairness in other languages.","keywords":["multilingual cultural evaluation","counterfactual rephrasing","confounder rephrasing","cultural awareness","cultural bias","data leakage","LLM fairness","cross-lingual evaluation"],"falsifier":"Ask independent native speakers of the 13 languages to judge a random sample of MCEval instances: does the counterfactual rewrite change the intended answer, does the confounder rewrite keep it unchanged, and does the translation preserve the original meaning? If human agreement with the pipeline's intended labels is low, the 10-30% gaps and the 66.7% native-language degradation could reflect bad questions or bad translations rather than data leakage or cultural unfairness.","tokens_in":19896,"feed_emoji":"🌍","tokens_out":9027,"duration_ms":91959,"temperature":0.7,"pith_summary":"MCEval sets out to measure what large language models actually know about the cultures of 13 countries or regions when asked in 13 different languages, and to separate genuine cultural understanding from memorized benchmark answers. It dynamically builds new questions from existing cultural corpora, then rewrites each question twice: a counterfactual version that flips the correct answer and a confounder version that should leave it unchanged. Across 39,897 awareness questions and 17,940 bias questions, the paper finds that models score 10-30% lower on rephrased questions than on original ones, which it reads as evidence that static cultural benchmarks suffer from data leakage. The same design reveals that fine-tuning a model on cultural data can look successful in English while degrading performance by up to 66.7% for a native-language culture, exposing hidden unfairness in English-centric evaluation.","feed_headline":"Dynamic test: LLM culture scores drop 10-30% when rephrased","feed_subtitle":"A 39,897-question multilingual benchmark finds static culture tests leak answers and English-only training can widen inequality.","key_machinery":"The mechanism is a five-agent construction pipeline: an Extractor pulls cultural knowledge from two public corpora; a Generator turns each item into a scenario question with a causal structure and an irrelevant item; a Rephrase Agent writes a Counterfactual question by changing the causal element so the answer flips, and a Confounder question by changing an irrelevant element so the answer stays fixed; a Translator renders all three into 13 languages; and a Verifier checks that each rewrite meets those conditions. The pairing of flipped-answer and invariant-answer rewrites is what converts a static benchmark into a causal probe: a model that only memorized the original corpus should fail the counterfactual, while a model that understands the culture should pass both.","core_discovery":"The central claim is that cultural evaluation of LLMs must be dynamic and multilingual to be fair. The paper introduces MCEval, which constructs cultural awareness and bias questions from existing corpora and then rewrites each question into a counterfactual version, where the causal element changes and the answer flips, and a confounder version, where an irrelevant element changes and the answer does not. Running Llama-3.3-70B and DeepSeek-V3 on 13 cultures in 13 languages, the paper reports that accuracy on original questions is consistently higher than on rephrased questions by roughly 10-30%, which it interprets as evidence that the source corpora have leaked into training. It further reports that a cultural fine-tuning method that improves English performance by 0.05-0.15 produces steep native-language drops, including a 66.7% fall for Swedish culture, implying that English-centric evaluation masks cultural unfairness. The authors conclude that no single language-culture pairing is best for all models; performance follows training-data distribution as well as language-culture alignment.","pith_inferences":["An unstated consequence of the 10-30% rephrasing gap is that fixed public culture benchmarks have a limited shelf life: once their questions appear in training data, scores conflate recall with understanding, so benchmark builders would need to regenerate questions for each evaluation round.","The same counterfactual and confounder design could transfer to other content dimensions the paper does not examine, such as religious, gender, or age stereotypes, where the flipped-answer versus invariant-answer contrast would diagnose leakage and reasoning.","A natural next experiment would vary model size and the English share of pretraining data to see whether the fairness gap grows monotonically with English-centric training, since the paper compares only two models."],"forward_implications":["Static cultural benchmarks that reuse fixed questions are likely to overstate LLM cultural ability by 10-30% because the answers have leaked into training.","A model can look culturally fair in English while performing far worse in native-language settings, so English-only evaluation is not a reliable fairness certificate.","Cultural enhancement methods that improve English performance can degrade native-language performance by up to 66.7%, so fairness must be checked per culture-language pair.","Awareness questions are more cross-lingually stable than bias questions, meaning bias detection is the more language-sensitive test of cultural understanding.","Optimal cultural performance does not come from simply matching each culture to its own language; training-data distribution matters as much as alignment."],"supporting_citations":[{"why":"Supplies the multilingual stereotype statements that seed the paper's 17,940 cultural-bias questions.","marker":"[26]"},{"why":"Supplies the cultural-knowledge entries that seed the awareness questions and the fine-tuned models used in the fairness analysis.","marker":"[27]"},{"why":"Introduces the dynamic rephrasing strategy for avoiding data leakage that MCEval adapts to cultural content.","marker":"[25]"},{"why":"Is the prior cross-cultural benchmark that evaluates only native and English scenarios, defining the scope gap MCEval fills.","marker":"[24]"},{"why":"Exemplifies a region-specific benchmark whose single-culture, single-language scope motivates the unified 13-culture design.","marker":"[11]"},{"why":"Exemplifies a Korean-language cultural benchmark that is culture-specific and thus motivates cross-lingual comparison.","marker":"[22]"},{"why":"Exemplifies a Chinese-language cultural benchmark whose single-culture scope the paper contrasts with its own multilingual setup.","marker":"[23]"},{"why":"Is an English-centric cultural enhancement method whose reported gains MCEval re-tests across native and cross-lingual scenarios.","marker":"[17]"}],"fun_headline_variants":["Rephrased questions slash LLM culture scores by 10–30%","English-only training made LLMs culturally unfair in 13 languages","Dynamic multilingual test exposes LLM cultural answer leakage","MCEval: rephrasing culture queries reveals hidden LLM bias","Native-language culture tests show LLMs lose up to 66% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatically generated counterfactual and confounder questions are valid cultural instruments: each counterfactual edit really flips the intended answer, each confounder edit really is irrelevant, and each translation preserves the meaning. The paper reports no human agreement data for these checks, so if the rewrites or translations are systematically off, the reported performance gaps could be artifacts of question wording rather than evidence of data leakage or cultural failure.","fun_headline_variants_meta":{"raw":{"variants":["Rephrased questions slash LLM culture scores by 10–30%","English-only training made LLMs culturally unfair in 13 languages","Dynamic multilingual test exposes LLM cultural answer leakage","MCEval: rephrasing culture queries reveals hidden LLM bias","Native-language culture tests show LLMs lose up to 66% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000247,"raw_usage":{"total_tokens":1531,"prompt_tokens":920,"completion_tokens":611,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":520}},"tokens_in":536,"tokens_out":611,"duration_ms":8232,"temperature":1.0,"reasoning_tokens":520,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:48:20.691370+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask independent native speakers of the 13 languages to judge a random sample of MCEval instances: does the counterfactual rewrite change the intended answer, does the confounder rewrite keep it unchanged, and does the translation preserve the original meaning? If human agreement with the pipeline's intended labels is low, the 10-30% gaps and the 66.7% native-language degradation could reflect bad questions or bad translations rather than data leakage or cultural unfairness.","supporting_citations":[{"cited_title":"Your stereotypical mileage may vary: Practical challenges of evaluating biases in multiple languages and cultural contexts,","cited_arxiv_id":null,"evidence_quote":"Supplies the multilingual stereotype statements that seed the paper's 17,940 cultural-bias questions."},{"cited_title":"Culturebank: An online community-driven knowledge base towards culturally aware language technologies,","cited_arxiv_id":null,"evidence_quote":"Supplies the cultural-knowledge entries that seed the awareness questions and the fine-tuned models used in the fairness analysis."},{"cited_title":"Dyval: Dynamic evaluation of large language models for reasoning tasks,","cited_arxiv_id":null,"evidence_quote":"Introduces the dynamic rephrasing strategy for avoiding data leakage that MCEval adapts to cultural content."},{"cited_title":"Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages,","cited_arxiv_id":null,"evidence_quote":"Is the prior cross-cultural benchmark that evaluates only native and English scenarios, defining the scope gap MCEval fills."},{"cited_title":"Indoculture: Exploring geographically influenced cultural commonsense reasoning across eleven indonesian provinces,","cited_arxiv_id":null,"evidence_quote":"Exemplifies a region-specific benchmark whose single-culture, single-language scope motivates the unified 13-culture design."},{"cited_title":"Hae-rae bench: Evaluation of korean knowl- edge in language models,","cited_arxiv_id":null,"evidence_quote":"Exemplifies a Korean-language cultural benchmark that is culture-specific and thus motivates cross-lingual comparison."},{"cited_title":"FoundaBench: Evaluating Chinese Fundamental Knowledge Capabilities of Large Language Models","cited_arxiv_id":"2404.18359","evidence_quote":"Exemplifies a Chinese-language cultural benchmark whose single-culture scope the paper contrasts with its own multilingual setup."},{"cited_title":"Culturepark: Boosting cross-cultural understanding in large language models,","cited_arxiv_id":null,"evidence_quote":"Is an English-centric cultural enhancement method whose reported gains MCEval re-tests across native and cross-lingual scenarios."}],"review_version":1}