Pith. sign in

REVIEW 3 major objections 6 minor 53 references

MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMs

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MCEval argues that static multilingual cultural benchmarks inflate LLM scores through data leakage, and that English-only cultural enhancement can deepen the unfairness in other languages.

desk verdict Useful multilingual cultural stress test, but the data-leakage and unfairness claims need stronger statistical and human-validated support. read the letter →

arxiv 2507.09701 v1 pith:WZVRZU67 submitted 2025-07-13 cs.CL

Shulin Huang , Linyi Yang , Yue Zhang This is my paper
classification cs.CL
keywords multilingualculturalevaluationcounterfactualrephrasingconfounderawarenessbiasdataleakageLLMfairnesscross-lingual
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MCEval sets out to measure what large language models actually know about the cultures of 13 countries or regions when asked in 13 different languages, and to separate genuine cultural understanding from memorized benchmark answers. It dynamically builds new questions from existing cultural corpora, then rewrites each question twice: a counterfactual version that flips the correct answer and a confounder version that should leave it unchanged. Across 39,897 awareness questions and 17,940 bias questions, the paper finds that models score 10-30% lower on rephrased questions than on original ones, which it reads as evidence that static cultural benchmarks suffer from data leakage. The same design reveals that fine-tuning a model on cultural data can look successful in English while degrading performance by up to 66.7% for a native-language culture, exposing hidden unfairness in English-centric evaluation.

What carries the argument

The mechanism is a five-agent construction pipeline: an Extractor pulls cultural knowledge from two public corpora; a Generator turns each item into a scenario question with a causal structure and an irrelevant item; a Rephrase Agent writes a Counterfactual question by changing the causal element so the answer flips, and a Confounder question by changing an irrelevant element so the answer stays fixed; a Translator renders all three into 13 languages; and a Verifier checks that each rewrite meets those conditions. The pairing of flipped-answer and invariant-answer rewrites is what converts a static benchmark into a causal probe: a model that only memorized the original corpus should fail the counterfactual, while a model that understands the culture should pass both.

What would settle it

Ask independent native speakers of the 13 languages to judge a random sample of MCEval instances: does the counterfactual rewrite change the intended answer, does the confounder rewrite keep it unchanged, and does the translation preserve the original meaning? If human agreement with the pipeline's intended labels is low, the 10-30% gaps and the 66.7% native-language degradation could reflect bad questions or bad translations rather than data leakage or cultural unfairness.

Watch

Extended reading notes

Core claim

The central claim is that cultural evaluation of LLMs must be dynamic and multilingual to be fair. The paper introduces MCEval, which constructs cultural awareness and bias questions from existing corpora and then rewrites each question into a counterfactual version, where the causal element changes and the answer flips, and a confounder version, where an irrelevant element changes and the answer does not. Running Llama-3.3-70B and DeepSeek-V3 on 13 cultures in 13 languages, the paper reports that accuracy on original questions is consistently higher than on rephrased questions by roughly 10-30%, which it interprets as evidence that the source corpora have leaked into training. It further reports that a cultural fine-tuning method that improves English performance by 0.05-0.15 produces steep native-language drops, including a 66.7% fall for Swedish culture, implying that English-centric evaluation masks cultural unfairness. The authors conclude that no single language-culture pairing is best for all models; performance follows training-data distribution as well as language-culture alignment.

Load-bearing premise

The load-bearing premise is that the automatically generated counterfactual and confounder questions are valid cultural instruments: each counterfactual edit really flips the intended answer, each confounder edit really is irrelevant, and each translation preserves the meaning. The paper reports no human agreement data for these checks, so if the rewrites or translations are systematically off, the reported performance gaps could be artifacts of question wording rather than evidence of data leakage or cultural failure.

Editorial extensions

If this is right

  • Static cultural benchmarks that reuse fixed questions are likely to overstate LLM cultural ability by 10-30% because the answers have leaked into training.
  • A model can look culturally fair in English while performing far worse in native-language settings, so English-only evaluation is not a reliable fairness certificate.
  • Cultural enhancement methods that improve English performance can degrade native-language performance by up to 66.7%, so fairness must be checked per culture-language pair.
  • Awareness questions are more cross-lingually stable than bias questions, meaning bias detection is the more language-sensitive test of cultural understanding.
  • Optimal cultural performance does not come from simply matching each culture to its own language; training-data distribution matters as much as alignment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An unstated consequence of the 10-30% rephrasing gap is that fixed public culture benchmarks have a limited shelf life: once their questions appear in training data, scores conflate recall with understanding, so benchmark builders would need to regenerate questions for each evaluation round.
  • The same counterfactual and confounder design could transfer to other content dimensions the paper does not examine, such as religious, gender, or age stereotypes, where the flipped-answer versus invariant-answer contrast would diagnose leakage and reasoning.
  • A natural next experiment would vary model size and the English share of pretraining data to see whether the fairness gap grows monotonically with English-centric training, since the paper compares only two models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. MCEval constructs multilingual cultural evaluation instances by running LLM agents (Extractor, Generator, Rephrase, Translator, Verifier) over CrowS-Pairs and TikTok, producing 39,897 awareness and 17,940 bias questions in 13 languages for 13 cultures. The authors evaluate Llama-3.3-70B and DeepSeek-V3 in native, English, and cross-lingual settings, and also compare CultureBank fine-tuned models. They report accuracy on original, counterfactual, and confounder questions; interpret original-vs-rephrased gaps as evidence of data leakage in static benchmarks; and argue that English-only evaluation masks cultural unfairness, citing a 66.7% degradation for Swedish after CultureBank fine-tuning.

Significance. If substantiated, MCEval would be a useful dynamic stress test for cultural knowledge and bias, and its multilingual and cross-cultural coverage is broader than most existing benchmarks. The paper contributes a large generated dataset, a detailed per-condition accuracy breakdown in Table III, and a clear operationalization of counterfactual versus confounder interventions. The main weakness is that the benchmark's validity rests on automatically generated labels and translations with no reported human validation, and the headline conclusions are drawn from single-run accuracy gaps without uncertainty quantification. The resource may still be valuable for relative model comparison, but the specific claims about training-data leakage and hidden unfairness require additional calibration.

major comments (3)
  1. [Section V, RQ1; Table III] The central claim that static cultural evaluations are unfair because of data leakage rests on accuracy gaps between Original and rephrased questions. These gaps are uncalibrated: counterfactual and confounder rewrites alter surface form, naturalness, and sometimes sentence length, so accuracy can drop even when a model has never seen the source corpora. The Original questions are themselves generated from cultural information rather than taken verbatim from CrowS-Pairs or TikTok, so a gap cannot directly establish exposure during training. The paper should provide a human baseline on the same items, and ideally a near-duplicate or leakage probe on the source corpora, and should report Verifier pass rates and human agreement on whether counterfactuals reverse the answer and confounders preserve it. Without this, the leakage interpretation is not separable from rephrasing sensitivity.
  2. [Section IV-A; Figure 3; Table III] All accuracy scores are single pass@1 runs at temperature 0.7 with no confidence intervals, random seeds, or significance tests. Several culture cells in Table I contain only 23-47 samples, so in those cells a 0.1 accuracy gap corresponds to only a handful of items. The headline 66.7% Swedish degradation in Figure 3(b) is a single observation, as are most of the 10-30% gaps invoked in the Introduction. The authors should report multiple runs, or at least bootstrap confidence intervals, and significance tests for the cross-scenario comparisons that support the fairness and leakage conclusions.
  3. [Section III-A; Section III-B] The entire dataset is produced by an LLM pipeline whose only quality gate is the Verifier Agent, itself an LLM. No pass rates, human agreement, translation-adequacy scores, or error analyses are reported, yet Section III-A states that the pipeline 'guarantee[s] high-quality data.' Because the benchmark consists of these generated instances, the accuracy numbers partly measure the pipeline's ability to generate unambiguous questions and correct labels. The paper should include human validation of a stratified sample of generated items, covering label correctness, causal preservation in counterfactuals, irrelevance of confounders, and translation fidelity, and should report agreement statistics; this is a necessary condition for interpreting any model score on MCEval.
minor comments (6)
  1. [Section III-B] The text refers to the 'Cowardcloud' dataset; this should be 'CrowS-Pairs.'
  2. [Section IV-A] The evaluation protocol should specify the exact sampling procedure; 'pass@1 at temperature 0.7' is ambiguous regarding the number of samples and decoding strategy.
  3. [Section V, RQ2] In the French-culture example, the sentence 'Counterfactual Rephrasing only reduces it to 0.938' should read 'Confounder Rephrasing only reduces it to 0.938.'
  4. [Section I] The Introduction says the data span '16 countries or regions,' while the abstract and Table I refer to 13 cultures and 13 languages; this mapping should be clarified.
  5. [Section V-A] Figure 3 compares Llama2-7B and Mixtral-56B, whereas the rest of the paper uses Llama-3.3-70B and DeepSeek-V3; the fairness experiment should state this model change explicitly.
  6. [Tables II and III] The tables would benefit from per-cell sample sizes or a reminder that counts vary by culture as shown in Table I, since small denominators affect the interpretability of small accuracy differences.

Circularity Check

1 steps flagged · score 2.0 of 10

MCEval's main empirical findings are not circular; the only self-definitional loop is the verifier-based 'high-quality data' guarantee, which is not load-bearing.

  1. self definitional [Section III-A, MCEval Pipeline (Verifier Agent)]
    "If any of the four Verifier Agents find non-compliance, the corresponding Agent in the pipeline needs to regenerate the data to ensure it passes through the respective Verifier, guaranteeing high-quality data."

    The claim of 'guaranteeing high-quality data' is defined solely by the approval of the Verifier Agents, which are themselves LLM agents inside the same construction pipeline. No human annotation, external gold standard, or independent validation is reported for the verifier decisions. Thus the quality guarantee reduces by construction to 'data that the Verifier Agents approve,' making the assertion self-defiitional. This loop is local to the data-construction quality assurance and is not used as evidence for the paper's central leakage or fairness conclusions, so it does not undermine the main evaluation results.

full rationale

MCEval is an empirical evaluation framework rather than a mathematical derivation. It extracts cultural information from external corpora (CrowS-Pairs and TikTok), generates original, counterfactual, and confounder questions via an LLM pipeline, translates them, and measures model accuracy. The observed accuracy gaps, cross-lingual differences, and CultureBank fairness comparisons are measured outcomes, not parameters fitted to those same outcomes and then renamed as predictions. The interpretation of rephrasing gaps as 'data leakage' is an inference that could be challenged on calibration grounds, but it is not circular in the sense of being equivalent to the measured gap by construction. The only genuine self-definitional element is the verifier's 'high-quality data' guarantee, where passing the verifier's own check is presented as proof of quality. That step is minor and not load-bearing for the main claims, which depend on model-output comparisons rather than on the verifier's self-certification. The citation to the authors' prior ThinkBench work is used only to position dynamic evaluation and is not the evidence for any result. Overall circularity is low.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The framework introduces a new benchmark and an evaluation protocol. The free parameters are the evaluation settings and the gap-to-leakage threshold. The axioms are the validity of the LLM-agent pipeline, the correctness of inherited labels, and the interpretation of score gaps as leakage. The invented entity is the new benchmark itself, which currently lacks independent validation.

free parameters (2)
  • pass@1 single-run sampling at temperature 0.7 = temperature=0.7, one run
    The paper reports all accuracy as pass@1 in a single run at temperature 0.7 (Section IV-A). This is a measurement choice that introduces variance with no error bars; conclusions about 10-30% gaps and 66.7% drops are drawn from these single-run numbers.
  • Performance gap threshold for data leakage = 10-30% gap
    The paper interprets any performance gap between original and rephrased questions as evidence of data leakage (Section V, RQ1) without calibrating the gap against human performance on the rephrased questions. This is effectively a chosen threshold for labeling a gap as leakage.
assumptions (4)
  • domain assumption The five-agent LLM pipeline produces valid cultural questions and translations without human validation.
    Extractor, Generator, Rephrase, Translator, and Verifier Agents are all LLM-based with no reported human agreement (Section III-A). The validity of every benchmark instance and label depends on this assumption.
  • domain assumption Counterfactual Rephrasing changes the answer direction while Confounder Rephrasing does not.
    The Verifier Agent checks this property, but no human or formal verification is reported (Section III-A). If the property fails, the interpretation of GAP as robustness versus brittleness is invalid.
  • domain assumption The answer labels in CrowS-Pairs and TikTok are correct and transfer to the generated questions.
    MCEval inherits source labels and applies them to newly constructed scenarios, with no human re-annotation (Sections I and III-B).
  • domain assumption A performance drop on rephrased questions indicates benchmark contamination (data leakage) rather than task difficulty.
    Section V RQ1 infers leakage from the original-versus-rephrased gap without a human baseline on the rephrased questions. This is the central interpretative step of the paper.
invented entities (1)
  • MCEval benchmark instances (rephrased counterfactual and confounder questions)
    purpose: To serve as a data-leakage-resistant dynamic cultural evaluation set.
    The new instances are generated by LLM agents from existing corpora with no human validation or external falsification handle (e.g., no pre-registered human agreement study).

how reviews work

0 comments
Cite this review

Pith. "Pith review of MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMs." pith.science (2026). https://pith.science/paper/WZVRZU67

@misc{pith2026250709701,
  author       = {Pith},
  title        = {Pith review of: MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WZVRZU67}},
  note         = {Machine review of arXiv:2507.09701}
}
read the original abstract

Large language models exhibit cultural biases and limited cross-cultural understanding capabilities, particularly when serving diverse global user populations. We propose MCEval, a novel multilingual evaluation framework that employs dynamic cultural question construction and enables causal analysis through Counterfactual Rephrasing and Confounder Rephrasing. Our comprehensive evaluation spans 13 cultures and 13 languages, systematically assessing both cultural awareness and cultural bias across different linguistic scenarios. The framework provides 39,897 cultural awareness instances and 17,940 cultural bias instances. Experimental results reveal performance disparities across different linguistic scenarios, demonstrating that optimal cultural performance is not only linked to training data distribution, but also is related to language-culture alignment. The evaluation results also expose the fairness issue, where approaches appearing successful in the English scenario create substantial disadvantages. MCEval represents the first comprehensive multilingual cultural evaluation framework that provides deeper insights into LLMs' cultural understanding.

Figures

Figures reproduced from arXiv: 2507.09701 by the authors.

Figure 1
Figure 1. Comparison of previous cultural evaluations and MCEval. Previous [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of MCEval framework. Based on the original cultural information (a), MCEval generates Cultural Questions (b), dynamically constructs [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Model performance comparisons across different language scenarios. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 50 canonical work pages

  1. [1]

    What is culture,

    H. Spencer-Oatey and P. Franklin, “What is culture,” A compilation of quotations. GlobalPAD Core Concepts , vol. 1, no. 22, pp. 1–21, 2012

    work page 2012

  2. [2]

    Culturellm: In- corporating cultural differences into large language models,

    C. Li, M. Chen, J. Wang, S. Sitaram, and X. Xie, “Culturellm: In- corporating cultural differences into large language models,” Advances in Neural Information Processing Systems , vol. 37, pp. 84 799–84 838, 2024

    work page 2024

  3. [3]

    Towards intercultural affect recognition: Audio-visual affect recognition in the wild across six cultures,

    L. Mathur, R. Adolphs, and M. J. Matari ´c, “Towards intercultural affect recognition: Audio-visual affect recognition in the wild across six cultures,” in 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG) . IEEE, 2023, pp. 1–6

    work page 2023

  4. [4]

    Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings,

    C. Liu, F. Koto, T. Baldwin, and I. Gurevych, “Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (V olume 1: Long Papers) , 2024, pp. 2016–2039

    work page 2024

  5. [5]

    Culture-gen: Revealing global cultural perception in language models through natural language prompting,

    H. Li, L. Jiang, N. Dziri, X. Ren, and Y . Choi, “Culture-gen: Revealing global cultural perception in language models through natural language prompting,” in First Conference on Language Modeling

    work page

  6. [6]

    Foodieqa: A multimodal dataset for fine-grained understanding of chinese food culture,

    W. Li, C. Zhang, J. Li, Q. Peng, R. Tang, L. Zhou, W. Zhang, G. Hu, Y . Yuan, A. Søgaard et al. , “Foodieqa: A multimodal dataset for fine-grained understanding of chinese food culture,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 19 077–19 095

    work page 2024

  7. [7]

    Mod- elling human factors in perceptual multimedia quality: On the role of personality and culture,

    M. J. Scott, S. C. Guntuku, Y . Huan, W. Lin, and G. Ghinea, “Mod- elling human factors in perceptual multimedia quality: On the role of personality and culture,” in Proceedings of the 23rd ACM international conference on Multimedia , 2015, pp. 481–490

    work page 2015

  8. [8]

    Intercultural communication on web sites: A cross-cultural analysis of web sites from high-context cultures and low-context cul- tures,

    E. W ¨urtz, “Intercultural communication on web sites: A cross-cultural analysis of web sites from high-context cultures and low-context cul- tures,” Journal of computer-mediated communication , vol. 11, no. 1, pp. 274–299, 2005

    work page 2005

Show all 53 references
  1. [9]

    Towards measuring the representation of subjective global opinions in language models,

    E. Durmus, K. Nyugen, T. Liao, N. Schiefer, A. Askell, A. Bakhtin, C. Chen, Z. Hatfield-Dodds, D. Hernandez, N. Joseph, L. Lovitt, S. McCandlish, O. Sikder, A. Tamkin, J. Thamkul, J. Kaplan, J. Clark, and D. Ganguli, “Towards measuring the representation of subjective global o...

    arXiv

  2. [10]

    Having beer after prayer? measuring cultural bias in large language models,

    T. Naous, M. J. Ryan, and W. Xu, “Having beer after prayer? measuring cultural bias in large language models,” in Annual Meeting of the Association for Computational Linguistics , 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:258865272

    2023

  3. [11]

    Indoculture: Exploring geographically influenced cultural commonsense reasoning across eleven indonesian provinces,

    F. Koto, R. Mahendra, N. Aisyah, and T. Baldwin, “Indoculture: Exploring geographically influenced cultural commonsense reasoning across eleven indonesian provinces,” Transactions of the Association for Computational Linguistics , vol. 12, pp. 1703–1719, 2024

    2024

  4. [12]

    Assessing cross-cultural alignment between chatgpt and human soci- eties: An empirical study,

    Y . Cao, L. Zhou, S. Lee, L. Cabello, M. Chen, and D. Hershcovich, “Assessing cross-cultural alignment between chatgpt and human soci- eties: An empirical study,” in Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP) , 2023, pp. 53–67

    2023

  5. [13]

    Cul- tural alignment in large language models: An explanatory analysis based on hofstede’s cultural dimensions,

    R. I. Masoud, Z. Liu, M. Ferianc, P. Treleaven, and M. Rodrigues, “Cul- tural alignment in large language models: An explanatory analysis based on hofstede’s cultural dimensions,” arXiv preprint arXiv:2309.12342 , 2023

    2023 arXiv

  6. [14]

    Stereoset: Measuring stereotypical bias in pretrained language models,

    M. Nadeem, A. Bethke, and S. Reddy, “Stereoset: Measuring stereotypical bias in pretrained language models,” in Annual Meeting of the Association for Computational Linguistics , 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID:215828184

    2020

  7. [15]

    Crows-pairs: A challenge dataset for measuring social biases in masked language models,

    N. Nangia, C. Vania, R. Bhalerao, and S. R. Bowman, “Crows-pairs: A challenge dataset for measuring social biases in masked language models,” in Conference on Empirical Methods in Natural Language Processing, 2020. [Online]. Available: https://api.semanticscholar.org/ CorpusID...

    2020

  8. [16]

    Cultural bias and cultural alignment of large language models,

    Y . Tao, O. Viberg, R. S. Baker, and R. F. Kizilcec, “Cultural bias and cultural alignment of large language models,” PNAS nexus, vol. 3, no. 9, p. pgae346, 2024. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 12

    2024

  9. [17]

    Culturepark: Boosting cross-cultural understanding in large language models,

    C. LI, D. Teney, L. Yang, Q. Wen, X. Xie, and J. Wang, “Culturepark: Boosting cross-cultural understanding in large language models,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems

  10. [18]

    J. A. Banks, Cultural diversity and education: F oundations, curriculum, and teaching . Routledge, 2015

    2015

  11. [19]

    Dosa: A dataset of social artifacts from different indian geographical subcultures,

    A. Seth, S. Ahuja, K. Bali, and S. Sitaram, “Dosa: A dataset of social artifacts from different indian geographical subcultures,” in Proceedings of the 2024 Joint International Conference on Computational Linguis- tics, Language Resources and Evaluation (LREC-COLING 2024) , 20...

    2024

  12. [20]

    Click: A benchmark dataset of cultural and linguistic intelligence in korean,

    E. Kim, J. Suk, P. Oh, H. Yoo, J. Thorne, and A. Oh, “Click: A benchmark dataset of cultural and linguistic intelligence in korean,” in Proceedings of the 2024 Joint International Conference on Computa- tional Linguistics, Language Resources and Evaluation (LREC-COLING 2024), ...

    2024

  13. [21]

    Copal- id: Indonesian language reasoning with local culture and nuances,

    H. Wibowo, E. Fuadi, M. Nityasya, R. E. Prasojo, and A. Aji, “Copal- id: Indonesian language reasoning with local culture and nuances,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo...

    2024

  14. [22]

    Hae-rae bench: Evaluation of korean knowl- edge in language models,

    G. Son, H. Lee, S. Kim, H. Kim, J. cheol Lee, J. W. Yeom, J. Jung, J. woo Kim, and S. Kim, “Hae-rae bench: Evaluation of korean knowl- edge in language models,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evalu...

    2024

  15. [23]

    Foundabench: Evaluating chinese fundamental knowledge capabilities of large language models,

    W. Li, R. Ma, J. Wu, C. Gu, J. Peng, J. Len, S. Zhang, H. Yan, D. Lin, and C. He, “Foundabench: Evaluating chinese fundamental knowledge capabilities of large language models,” arXiv preprint arXiv:2404.18359, 2024

    2024 arXiv

  16. [24]

    Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages,

    J. Myung, N. Lee, Y . Zhou, J. Jin, R. Putri, D. Antypas, H. Borkakoty, E. Kim, C. Perez-Almendros, A. A. Ayele et al. , “Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages,” Ad- vances in Neural Information Processing Systems , vol. 37, pp. 78...

    2024

  17. [25]

    Dyval: Dynamic evaluation of large language models for reasoning tasks,

    K. Zhu, J. Chen, J. Wang, N. Z. Gong, D. Yang, and X. Xie, “Dyval: Dynamic evaluation of large language models for reasoning tasks,” in The Twelfth International Conference on Learning Representations

  18. [26]

    Your stereotypical mileage may vary: Practical challenges of evaluating biases in multiple languages and cultural contexts,

    K. Fort, L. A. Alemany, L. Benotti, J. Bezanc ¸on, C. Borg, M. Borg, Y . Chen, F. Ducel, Y . Dupont, G. Ivetta et al. , “Your stereotypical mileage may vary: Practical challenges of evaluating biases in multiple languages and cultural contexts,” in Proceedings of the 2024 Join...

    2024

  19. [27]

    Culturebank: An online community-driven knowledge base towards culturally aware language technologies,

    W. Shi, R. Li, Y . Zhang, C. Ziems, S. Yu, R. Horesh, R. Paula, and D. Yang, “Culturebank: An online community-driven knowledge base towards culturally aware language technologies,” in Findings of the Association for Computational Linguistics: EMNLP 2024 , 2024, pp. 4996–5025

    2024

  20. [28]

    Cultures and organizations: Software of the mind: Inter- cultural cooperation and its importance for survival,

    G. Hofstede, “Cultures and organizations: Software of the mind: Inter- cultural cooperation and its importance for survival,” (No Title), 2010

    2010

  21. [29]

    Trompenaars and C

    F. Trompenaars and C. Hampden-Turner, Riding the waves of culture: Understanding diversity in global business . Nicholas Brealey Interna- tional, 2011

    2011

  22. [30]

    Beyond culture,

    E. T. Hall, “Beyond culture,” Garden City, 1976

    1976

  23. [31]

    N. J. Adler and A. Gundersen, International dimensions of organiza- tional behavior . South-Western Cincinnati, OH, 2001

    2001

  24. [32]

    Multiculturalism and diversity: A social psychological perspective,

    V . C. Plaut and J. Treadway, “Multiculturalism and diversity: A social psychological perspective,” 2011

    2011

  25. [33]

    Towards ethnorelativism: A developmental model of intercultural sensitivity,

    M. J. Bennett, “Towards ethnorelativism: A developmental model of intercultural sensitivity,” Education for the intercultural experi- ence/Intercultural Press, 1993

    1993

  26. [34]

    Survey of cultural awareness in language models: Text and beyond

    S. Pawar, J. Park, J. Jin, A. Arora, J. Myung, S. Yadav, F. G. Haznitrama, I. Song, A. Oh, and I. Augenstein, “Survey of cultural awareness in language models: Text and beyond.”

  27. [35]

    Aradice: Benchmarks for dialectal and cultural capabilities in llms,

    B. Mousi, N. Durrani, F. Ahmad, M. A. Hasan, M. Hasanain, T. Kabbani, F. Dalvi, S. A. Chowdhury, and F. Alam, “Aradice: Benchmarks for dialectal and cultural capabilities in llms,” in Proceedings of the 31st International Conference on Computational Linguistics , 2025, pp. 4186– 4218

    2025

  28. [36]

    Benchmarking cognitive domains for llms: Insights from taiwanese hakka culture,

    C.-C. Chang, C.-Y . Chen, H.-S. Lee, and C.-C. Lee, “Benchmarking cognitive domains for llms: Insights from taiwanese hakka culture,” in 2024 27th Conference of the Oriental COCOSDA International Com- mittee for the Co-ordination and Standardisation of Speech Databases and Ass...

    2024

  29. [37]

    ” vorbe\c {s} ti rom \ˆ ane \c {s} te?

    M. Masala, D. C. Ilie-Ablachim, A. Dima, D. Corlatescu, M. Zavelca, O. Olaru, S. Terian, A. Terian, M. Leordeanu, H. Velicuet al., “” vorbe\c {s} ti rom \ˆ ane \c {s} te?” a recipe to train powerful romanian llms with english instructions,” arXiv preprint arXiv:2406.18266 , 2024

    2024 arXiv

  30. [38]

    Challenges and strategies in cross-cultural nlp,

    D. Hershcovich, S. Frank, H. Lent, M. de Lhoneux, M. Abdou, S. Brandl, E. Bugliarello, L. C. Piqueras, I. Chalkidis, R. Cui et al. , “Challenges and strategies in cross-cultural nlp,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (V...

    2022

  31. [39]

    Biases in large language models: ori- gins, inventory, and discussion,

    R. Navigli, S. Conia, and B. Ross, “Biases in large language models: ori- gins, inventory, and discussion,” ACM Journal of Data and Information Quality, vol. 15, no. 2, pp. 1–21, 2023

    2023

  32. [40]

    Kobbq: Korean bias benchmark for question answering,

    J. Jin, J. Kim, N. Lee, H. Yoo, A. Oh, and H. Lee, “Kobbq: Korean bias benchmark for question answering,” Transactions of the Association for Computational Linguistics , vol. 11, pp. 507–524, 2024

    2024

  33. [41]

    Cbbq: A chinese bias benchmark dataset curated with human-ai collaboration for large language models,

    Y . Huang and D. Xiong, “Cbbq: A chinese bias benchmark dataset curated with human-ai collaboration for large language models,” in Proceedings of the 2024 Joint International Conference on Computa- tional Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024,...

    2024

  34. [42]

    Semantics derived au- tomatically from language corpora contain human-like biases,

    A. Caliskan, J. J. Bryson, and A. Narayanan, “Semantics derived au- tomatically from language corpora contain human-like biases,” Science, vol. 356, no. 6334, pp. 183–186, 2017

    2017

  35. [43]

    Measuring individual differences in implicit cognition: the implicit association test

    A. G. Greenwald, D. E. McGhee, and J. L. Schwartz, “Measuring individual differences in implicit cognition: the implicit association test.” Journal of personality and social psychology , vol. 74, no. 6, p. 1464, 1998

    1998

  36. [44]

    Having beer after prayer? measuring cultural bias in large language models,

    T. Naous, M. J. Ryan, A. Ritter, and W. Xu, “Having beer after prayer? measuring cultural bias in large language models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), 2024, pp. 16 366–16 393

    2024

  37. [45]

    Building socio-culturally inclusive stereotype resources with community engage- ment,

    S. Dev, J. Goyal, D. Tewari, S. Dave, and V . Prabhakaran, “Building socio-culturally inclusive stereotype resources with community engage- ment,” Advances in Neural Information Processing Systems , vol. 36, pp. 4365–4381, 2023

    2023

  38. [46]

    Seegull: A stereotype benchmark with broad geo-cultural coverage leveraging generative models,

    A. Jha, A. M. Davani, C. K. Reddy, S. Dave, V . Prabhakaran, and S. Dev, “Seegull: A stereotype benchmark with broad geo-cultural coverage leveraging generative models,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 1: Long...

    2023

  39. [47]

    The (undesired) attenuation of human biases by multilinguality,

    C. Espa ˜na-Bonet and A. Barr´on-Cede˜no, “The (undesired) attenuation of human biases by multilinguality,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , 2022, pp. 2056– 2077

    2022

  40. [48]

    Global voices, local biases: Socio-cultural prejudices across languages,

    A. Mukherjee, C. Raj, Z. Zhu, and A. Anastasopoulos, “Global voices, local biases: Socio-cultural prejudices across languages,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 15 828–15 845

    2023

  41. [49]

    French crows-pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than english,

    A. N ´ev´eol, Y . Dupont, J. Bezanc ¸on, and K. Fort, “French crows-pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than english,” in Proceedings of the 60th Annual Meeting of the Association for Computational Lingui...

    2022

  42. [50]

    Chbias: Bias evaluation and mitigation of chinese conversational language mod- els,

    J. Zhao, M. Fang, Z. Shi, Y . Li, L. Chen, and M. Pechenizkiy, “Chbias: Bias evaluation and mitigation of chinese conversational language mod- els,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers) , 2023, pp. ...

    2023

  43. [51]

    Promptbench: A unified library for evaluation of large language models,

    K. Zhu, Q. Zhao, H. Chen, J. Wang, and X. Xie, “Promptbench: A unified library for evaluation of large language models,” Journal of Machine Learning Research , vol. 25, no. 254, pp. 1–22, 2024

    2024

  44. [52]

    Thinkbench: Dynamic out-of-distribution evaluation for robust llm reasoning,

    S. Huang, L. Yang, Y . Song, S. Chen, L. Cui, Z. Wan, Q. Zeng, Y . Wen, K. Shao, W. Zhang et al. , “Thinkbench: Dynamic out-of-distribution evaluation for robust llm reasoning,” CoRR, 2025. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 13 Shulin Huang received her ...

    2025

  45. [2023]

    Available: https://api.semanticscholar.org/CorpusID: 259275051

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 259275051

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.