Pith. sign in

REVIEW 4 major objections 5 minor 45 references

Multilingual LLMs favor Western entities 30-40% of the time even in culturally grounded Asian contexts, according to a new benchmark across nine Asian languages.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 11:18 UTC pith:N5D53IKS

load-bearing objection A solid, much-needed multilingual benchmark with a credible bias signal, though the headline CBS score needs a robustness check before it carries the paper's weight. the 4 major comments →

arxiv 2510.05291 v2 pith:N5D53IKS submitted 2025-10-06 cs.CL

Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages

Tarek Naous , Anagha Savit , Carlos Rafael Catalan , Geyang Guo , Jaehyeok Lee , Kyungdon Lee , Lheane Marie Dizon , Mengyu Ye
show 12 more authors
Neel Kothari Sahajpreet Singh Sarah Masud Tanish Patwa Trung Thanh Tran Zohaib Khan Alan Ritter Tanmoy Chakraborty Yuki Arase Keisuke Sakaguchi JinYeong Bak Wei Xu
This is my paper
classification cs.CL
keywords cultural biaslarge language modelsmultilingualbenchmarkAsian languagesentity-centricsentiment associationextractive QA
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Camellia is a new benchmark for entity-centric cultural bias: 19,530 manually annotated entities and 2,173 masked social-media contexts across nine Asian languages and six Asian cultures. Using it, this paper tries to establish that current multilingual LLMs favor Western-associated entities in culturally grounded Asian contexts 30-40% of the time, far above the expected 0-5%. It also claims that this preference changes little between culturally grounded and culturally neutral contexts, that model families differ in how they associate cultures with sentiments, and that entity-extraction accuracy gaps between Asian and Western entities are large in Asian languages but mostly disappear in English. A sympathetic reader should care because the result would mean that LLMs systematically underserve Asian-language users, and that English-only evaluation hides the problem.

Core claim

The paper's central claim is that entity-centric cultural bias is neither rare nor confined to one region: across nine Asian languages and six cultures, recent multilingual LLMs assign a higher probability to a Western-associated entity than to the appropriate Asian entity in about 30-40% of pairwise comparisons, even when the masked context is explicitly grounded in the local culture and the expected bias score is near 0-5%. This preference is largely insensitive to cultural grounding—scores on neutral contexts stay close to grounded scores—and it varies by model family: some families associate Asian entities with positive sentiment while others associate Western entities with negative sent

What carries the argument

The Cultural Bias Score (CBS), a likelihood-based metric: for a masked context, it averages an indicator over all Asian–Western entity pairs of whether the model's probability of filling [MASK] with the Western entity exceeds its probability for the Asian entity. It carries the argument by turning 'cultural adaptation' into a single number per language and entity type; Camellia supplies the paired entity sets and naturally occurring masked contexts that make the score computable.

Load-bearing premise

The benchmark's 'culturally-grounded' contexts are actually unique fits for the Asian entity; if annotation is noisy or ambiguous, the 30-40% rate could reflect context ambiguity rather than model bias, and the token-product likelihood calculation could further inflate the number.

What would settle it

Re-annotate a random subset of Camellia-Grounded contexts with independent native speakers and ask whether only an Asian entity is appropriate; also compute CBS with token-length normalization. If a substantial fraction (say, >20%) of contexts are judged ambiguous or the normalized CBS drops to near 5%, the central claim of persistent 30-40% Western preference would be undermined.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the 30-40% CBS reflects genuine bias, current multilingual LLMs will regularly suggest inappropriate Western entities when users write in Asian languages.
  • Because CBS changes little between culturally grounded and neutral contexts, models are not using cultural cues to constrain their choices.
  • English-only evaluation understates the problem: accuracy gaps between Asian and Western entities shrink to roughly 1-5% in English.
  • Lower-resource languages (Malayalam, Marathi, Gujarati, Urdu) show the largest biases and lowest extraction accuracy, making resource availability a cultural-fairness issue.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: CBS multiplies token-level probabilities, so longer or differently tokenized Asian entities can be systematically penalized; a token-length-normalized version of the score would reveal how much of the 30-40% is bias vs. scoring artifact.
  • Editorial inference: the Western entity set spans many countries; lumping them together could hide that the preference tracks particular Western cultures rather than a generic 'Western' label.
  • Editorial inference: if the bias is driven by pretraining exposure, a targeted test with prompts that include local entity names should reduce CBS; if not, the bias may require dataset-level intervention.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Camellia, a benchmark for entity-centric cultural bias in nine Asian languages spanning six cultures, comprising 19,530 manually annotated entities and 2,173 masked contexts drawn from social media. The authors evaluate four multilingual LLMs on three tasks: cultural context adaptation (using the Cultural Bias Score, CBS), sentiment association, and extractive QA. The central claim is that LLMs assign higher likelihood to Western entities than to culturally appropriate Asian entities in 30–40% of cases even in culturally grounded contexts, and that different model families exhibit distinct sentiment biases and QA accuracy gaps between Asian and Western entities.

Significance. If the findings hold, Camellia is a valuable and much-needed resource: it covers under-resourced Asian languages, includes manually annotated entities with high inter-annotator agreement, and provides naturalistic masked contexts. The observation that model provenance (e.g., Qwen's stronger performance on East Asian cultures) correlates with cultural competence is an interesting and plausible result, and the public release of the benchmark will enable future work. However, the central quantitative claim — the 30–40% CBS on grounded contexts — currently rests on a likelihood-based metric that is vulnerable to token-length confounds, and several secondary results are reported without uncertainty quantification. These issues are fixable but need to be addressed before the headline claims can be taken at face value.

major comments (4)
  1. [§3.1, Eq. (1)] The CBS definition multiplies per-token conditional probabilities without any length normalization. For decoder-only LLMs, the joint probability of a completion decays roughly exponentially with token count. If Western entities are systematically tokenized into fewer subwords than their Asian counterparts (plausible for transliterated Western names in CJK or Indic scripts, or vice versa), the pairwise average in Eq. (1) will inflate CBS even for a culturally neutral model. The paper does not report token-length distributions per language or any length-controlled analysis. This is load-bearing for the 30–40% headline in §3.1. Please add either a length-normalized version of Eq. (1), a matched-token-count analysis, or a control regression against entity token length.
  2. [§3.2, Figure 5] The sentiment association results report differences in false-negative and false-positive rates between Asian-entity and Western-entity sets, but no confidence intervals or significance tests are provided. The test sets are produced by sampling 50 entities per context, and the differences in Figure 5 (e.g., -45.7 or -226.3) may be within sampling noise, especially for Urdu where the test set is only 2,550 samples per condition. The claim that Llama and Gemma associate Western entities with negativity, and Qwen/Aya associate Asian entities with positivity, requires statistical support. Also, the definitions of false negative and false positive are not stated.
  3. [§3.3, Table 1] The QA accuracy gaps (ΔAccuracy) are reported without any measure of uncertainty. For instance, Vietnamese Llama shows Δ = -13.53 and Pakistani Gemma shows Δ = 21.11, but the test-set sizes (roughly 2,350–3,900 samples per language) and the exact-match metric make it unclear whether these gaps are statistically reliable. The claim that 'gaps in Asian languages are larger, reaching a 12–20% range' would be strengthened by confidence intervals or a significance test. Additionally, the comparison between Asian and Western entities may be confounded by tokenization and entity frequency in pretraining; the paper mentions this as a possible explanation but does not control for it.
  4. [§3.1 and §2.2] The expected CBS of 0–5% on culturally grounded contexts is an assumption that rests on the authors' labeling of these contexts as 'only entities associated with the specific Asian culture are appropriate.' However, no inter-annotator agreement or human validation is reported for the context labels themselves, and no analysis of context ambiguity is provided. If some 'grounded' contexts are in fact compatible with Western entities, the 30–40% CBS will overstate model bias. A small human evaluation of a random sample of contexts, or reporting context-level agreement, would directly address this concern.
minor comments (5)
  1. [Appendix B, Table 7] The caption of Table 7 says 'Prompt used to classify a sentence’s sentiment in our sentiment association experiment' but the actual prompt is for extracting an entity. The caption should be corrected.
  2. [§3.2, Figure 5] The terms 'False Negatives' and 'False Positives' are used without definition. Clarify what constitutes a false negative/positive with respect to the sentiment labels annotated on the contexts.
  3. [§3.1, Eq. (1)] Ties in the indicator 1[P(bj|ck) > P(ai|ck)] are not addressed (e.g., whether they are counted as 0, 0.5, or omitted). Please specify the tie-breaking rule.
  4. [§2.1] The phrase 'A lot of these Western entities did not have written versions in Wikidata' is informal; consider revising.
  5. [§3.1, Figure 3] The text states CBS is 'in the 30-40% range' but Figure 3 shows some languages/models above 50% (e.g., Gemma on Urdu). The qualitative summary should be more precise about the range and its variance.

Circularity Check

0 steps flagged

No circular reduction: CBS is an inherited metric and the 30–40% finding is an empirical measurement on new data.

full rationale

The paper does not fit any parameter and then re-predict it from the same data. Equation (1) is a direct likelihood comparison over human-annotated entities and contexts; the reported 30–40% CBS is a model output, not an algebraic consequence of the annotation procedure. The statement that CBS is 'expected to be low' in culturally-grounded contexts reflects the annotators' judgment about context appropriateness, not a mathematical derivation from the metric itself. The main self-citation, Naous et al. (2024), supplies the CBS formula, data-curation procedure, and prompts; this is methodological inheritance rather than a load-bearing proof, and the metric is parameter-free and externally applicable. The token-length confound noted by reviewers—multiplying conditional token probabilities without length normalization—is a measurement-validity concern that would affect interpretation, but it does not make the result circular because Eq. (1) does not define the conclusion by construction. No step of the derivation reduces to its own inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The paper introduces no new theoretical entities, forces, or parameters. Its underlying assumptions are about annotation validity, context grounding, and the likelihood-as-preference proxy. These are reasonable domain assumptions for an NLP benchmark, but they are not independently verified beyond the reported inter-annotator agreements.

axioms (4)
  • domain assumption Entities can be cleanly categorized as either Asian-associated or Western-associated.
    The annotation guideline in Appendix A instructs annotators to label each entity as Indian/Western or irrelevant. This binary separation may not capture entities with mixed or transnational cultural associations, and is assumed in the CBS calculation and all downstream analyses.
  • domain assumption Culturally-grounded contexts are those where only an Asian entity is appropriate, and this can be reliably identified by the authors.
    Section 2.2 describes how tweets were manually selected to create grounded contexts. The expected CBS near 0% in §3.1 depends on this assumption; if contexts are noisy, the CBS interpretation weakens.
  • domain assumption LLM likelihood scores (product of token probabilities) reflect meaningful cultural preference when filling the [MASK] token.
    Equation (1) in §3.1 computes CBS from token-level conditional probabilities. This assumes that likelihood is a valid proxy for model preference and that tokenization differences do not systematically skew probabilities between Western and Asian entities.
  • domain assumption Randomly sampling 50 entities per context yields a representative evaluation set.
    Section 3.2 and 3.3 construct test sets by sampling 50 Asian and 50 Western entities per context. The results are averaged over 3 runs, but no variance is reported, so representatives and stability are assumed.

pith-pipeline@v1.3.0-alltime-deepseek · 20532 in / 6298 out tokens · 49915 ms · 2026-08-04T11:18:50.556463+00:00 · methodology

0 comments
read the original abstract

As Large Language Models (LLMs) develop stronger multilingual capabilities, their sensitivity to culturally diverse entities becomes increasingly important. Prior work by Naous et al. (2024) has shown that LLMs often favor Western-associated entities in Arabic. Due to the lack of entity-centric multilingual benchmarks, it remains unclear if such biases also manifest in various non-Western languages. In this paper, we introduce Camellia, a benchmark for evaluating entity-centric cultural biases in nine Asian languages, spanning six Asian cultures. Camellia includes 19,530 manually annotated entities associated with the covered Asian or Western cultures, as well as 2,173 masked contexts for these entities derived from social media posts. Using Camellia, we evaluate cultural biases in four recent multilingual LLMs across three tasks: cultural context adaptation, sentiment association, and entity extractive QA. Our analyses show that LLMs struggle with cultural adaptation across these languages, with performance differing across models developed in different regions. We further observe that different LLM families can hold distinct biases, reflected in the ways they link cultures to particular sentiments. Lastly, we find that LLMs can struggle with context understanding in some Asian languages, creating performance gaps between cultures in entity extraction.

Figures

Figures reproduced from arXiv: 2510.05291 by Alan Ritter, Anagha Savit, Carlos Rafael Catalan, Geyang Guo, Jaehyeok Lee, JinYeong Bak, Keisuke Sakaguchi, Kyungdon Lee, Lheane Marie Dizon, Mengyu Ye, Neel Kothari, Sahajpreet Singh, Sarah Masud, Tanish Patwa, Tanmoy Chakraborty, Tarek Naous, Trung Thanh Tran, Wei Xu, Yuki Arase, Zohaib Khan.

Figure 1
Figure 1. Figure 1: We construct Camellia, a benchmark to measure cultural biases for six Asian cultures, covering nine languages. Camellia provides 2,173 naturally-occurring masked contexts categorized into: culturally-grounded, culturally-neutral, and extractive QA. Camellia also provides 19,530 cul￾turally relevant entities that contrast the respective Asian cultures vs. Western culture across six different entity types th… view at source ↗
Figure 2
Figure 2. Figure 2: Example per entity type and statistics of respective Asian entities per culture and Western [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Average Cultural Bias Score (CBS) (↓) across entity types achieved by LLMs on culturally￾grounded contexts (Camellia-Grounded) for each Asian language. LLMs can struggle to generate the appropriate Asian entities in each culture, assigning better likelihood to Western entities 30-40% of the time. See results per entity type in Appendix C.1. identify ones that provide suitable cultural contexts. From these,… view at source ↗
Figure 4
Figure 4. Figure 4: Average CBS across entity types on culturally-grounded contexts vs culturally-neutral [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Differences in False Negative (FN) and False Positive (FP) sentiment predictions by LLMs [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Extractive QA accuracy by LLMs on Camellia-QA contexts containing Asian vs Western entities when tested in each Asian language. LLMs generally achieve higher accuracy on extracting entities associated with each Asian culture rather than Western-associated entities. 5%, with no clear trend of superior performance on one culture. Yet, gaps in Asian languages are much larger, reaching a 12%-20% range in most … view at source ↗
Figure 7
Figure 7. Figure 7: Indian-focused version of our annotation guideline for annotating cultural entities. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Cultural Bias Score (CBS) (↓) (§3.1) per entity type achieved by LLMs on culturally￾grounded contexts (Camellia-Grounded) for each Asian language. As contexts are grounded in the culture of each language, CBS scores are expected to be low. Chinese Japanese Korean Vietnamese Pakistani Indian 30 40 50 60 70 80 CBS Llama3.3-70b Chinese Japanese Korean Vietnamese Pakistani Indian 30 40 50 60 70 80 CBS Qwen2.5-… view at source ↗
Figure 9
Figure 9. Figure 9: Average Cultural Bias Score (CBS) (↓) across entity types achieved by LLMs on culturally￾grounded contexts (Camellia-Grounded) when tested in English for each culture. C ADDITIONAL RESULTS C.1 CULTURAL ADAPTATION CBS scores per Entity Type [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Differences in False Negative (FN) and False Positive (FP) sentiment predictions by [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 1 canonical work pages

  1. [1]

    Towards measuring and modeling “culture” in LLMs: A survey

    Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Shivdutt Singh, Alham Fikri Aji, Jacki O’Neill, Ashutosh Modi, and Monojit Choudhury. Towards measuring and modeling “culture” in LLMs: A survey. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 15763–15784, Miami, Florida, USA, November

    2024

  2. [3]

    The Indian drink [MASK] is very nice to have in the evening

    irrelevant noisy extractions from pattern matching on mC4 that are not food related. Beverage entities: The same guideline described above for food entities is applied for beverage entities. Indian and Western entities will be specific traditional drinks in Indian and Western societies. For example, an Indian beverage entity must fit within a prompt like ...

    2024

  3. [4]

    ISBN 979-8-89176- 251-0

    Association for Computational Linguistics. ISBN 979-8-89176- 251-0. doi: 10.18653/v1/2025.acl-long.578. URLhttps://aclanthology.org/2025.acl-long.578/. Ivona Bareˇsov´a and Petr Janda. Tradition and change: naming practices in contemporary Japan and Taiwan. Continuity and change in Asia, pp. 393–411,

    doi:10.18653/v1/2025.acl-long.578 2025

  4. [5]

    acl-short.75

    doi: 10.18653/v1/2024. acl-short.75. URLhttps://aclanthology.org/2024.acl-short.75/. Suvasish Chakraborty. The politics of sports: cricket as a factor in india-pakistan relations

    doi:10.18653/v1/2024 2024

  5. [9]

    doi: 10.18653/v1/2024

    Association for Computational Linguistics. doi: 10.18653/v1/2024. emnlp-main.1134. URLhttps://aclanthology.org/2024.emnlp-main.1134/. John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, et al. Aya expanse: Combining research break- throughs for a ne...

    Pith/arXiv arXiv 2024

  6. [10]

    Gender bias in large language models across multiple languages: A case study of ChatGPT

    YiTian Ding, Jinman Zhao, Chen Jia, Yining Wang, Zifan Qian, Weizhe Chen, and Xingyu Yue. Gender bias in large language models across multiple languages: A case study of ChatGPT. InProceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025), pp. 552–579,

    2025

  7. [11]

    Massively multi-cultural knowledge acquisition & lm benchmarking.arXiv preprint arXiv:2402.09369,

    Yi Fung, Ruining Zhao, Jae Doo, Chenkai Sun, and Heng Ji. Massively multi-cultural knowledge acquisition & lm benchmarking.arXiv preprint arXiv:2402.09369,

    Pith/arXiv arXiv

  8. [12]

    doi: 10.1145/3580494

    ISSN 0360-0300. doi: 10.1145/3580494. URLhttps: //doi.org/10.1145/3580494. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv

  9. [13]

    Care: Aligning language models for regional cultural awareness.arXiv preprint arXiv:2504.05154,

    Geyang Guo, Tarek Naous, Hiromi Wakaki, Yukiko Nishimura, Yuki Mitsufuji, Alan Ritter, and Wei Xu. Care: Aligning language models for regional cultural awareness.arXiv preprint arXiv:2504.05154,

    arXiv

  10. [14]

    Speaking multiple languages affects the moral bias of language models.arXiv preprint arXiv:2211.07733,

    11 Camellia(Preprint) Katharina H ¨ammerl, Bj ¨orn Deiseroth, Patrick Schramowski, Jind ˇrich Libovick `y, Constantin A Rothkopf, Alexander Fraser, and Kristian Kersting. Speaking multiple languages affects the moral bias of language models.arXiv preprint arXiv:2211.07733,

    Pith/arXiv arXiv

  11. [15]

    TWBias: A benchmark for assessing so- cial bias in traditional chinese large language models through a taiwan cultural lens

    Hsin-Yi Hsieh, Shih-Cheng Huang, and Richard Tzong-Han Tsai. TWBias: A benchmark for assessing so- cial bias in traditional chinese large language models through a taiwan cultural lens. InFindings of the Association for Computational Linguistics: EMNLP 2024, pp. 8688–8704,

    2024

  12. [16]

    CBBQ: A chinese bias benchmark dataset curated with human-ai collaboration for large language models.arXiv preprint arXiv:2306.16244,

    Yufei Huang and Deyi Xiong. CBBQ: A chinese bias benchmark dataset curated with human-ai collaboration for large language models.arXiv preprint arXiv:2306.16244,

    Pith/arXiv arXiv

  13. [17]

    Gender bias in masked language models for multiple languages.arXiv preprint arXiv:2205.00551,

    Masahiro Kaneko, Aizhan Imankulova, Danushka Bollegala, and Naoaki Okazaki. Gender bias in masked language models for multiple languages.arXiv preprint arXiv:2205.00551,

    Pith/arXiv arXiv

  14. [18]

    DLAMA: A framework for curating culturally diverse facts for probing the knowledge of pretrained language models.arXiv preprint arXiv:2306.05076,

    Amr Keleg and Walid Magdy. DLAMA: A framework for curating culturally diverse facts for probing the knowledge of pretrained language models.arXiv preprint arXiv:2306.05076,

    Pith/arXiv arXiv

  15. [19]

    McBE: A multi-task chinese bias evaluation benchmark for large language models.arXiv preprint arXiv:2507.02088,

    Tian Lan, Xiangdong Su, Xu Liu, Ruirui Wang, Ke Chang, Jiang Li, and Guanglai Gao. McBE: A multi-task chinese bias evaluation benchmark for large language models.arXiv preprint arXiv:2507.02088,

    Pith/arXiv arXiv

  16. [20]

    What are foundation models cooking in the post-soviet world?arXiv preprint arXiv:2502.18583,

    Anton Lavrouk, Tarek Naous, Alan Ritter, and Wei Xu. What are foundation models cooking in the post-soviet world?arXiv preprint arXiv:2502.18583,

    arXiv

  17. [21]

    Comparing biases and the impact of multilingual training across multiple languages.arXiv preprint arXiv:2305.11242,

    Sharon Levy, Neha Anna John, Ling Liu, Yogarshi Vyas, Jie Ma, Yoshinari Fujinuma, Miguel Ballesteros, Vittorio Castelli, and Dan Roth. Comparing biases and the impact of multilingual training across multiple languages.arXiv preprint arXiv:2305.11242,

    Pith/arXiv arXiv

  18. [22]

    Attributing culture-conditioned generations to pretraining corpora.arXiv preprint arXiv:2412.20760, 2024a

    Huihan Li, Arnav Goel, Keyu He, and Xiang Ren. Attributing culture-conditioned generations to pretraining corpora.arXiv preprint arXiv:2412.20760, 2024a. Wenyan Li, Xinyu Zhang, Jiaang Li, Qiwei Peng, Raphael Tang, Li Zhou, Weijia Zhang, Guimin Hu, Yifei Yuan, Anders Søgaard, et al. FoodieQA: A multimodal dataset for fine-grained understanding of chinese ...

    Pith/arXiv arXiv 2025

  19. [23]

    Probing politico-economic bias in multilingual large lan- guage models: A cultural analysis of low-resource pakistani languages.arXiv preprint arXiv:2506.00068,

    Afrozah Nadeem, Mark Dras, and Usman Naseem. Probing politico-economic bias in multilingual large lan- guage models: A cultural analysis of low-resource pakistani languages.arXiv preprint arXiv:2506.00068,

    arXiv

  20. [24]

    On the origin of cultural biases in language models: From pre-training data to linguistic phenomena

    Tarek Naous and Wei Xu. On the origin of cultural biases in language models: From pre-training data to linguistic phenomena. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 6423–6443, Albuquerque, New Mexico, April

    2025

  21. [25]

    ISBN 979-8-89176-189-6

    Association for Computational Linguistics. ISBN 979-8-89176-189-6. doi: 10.18653/v1/2025.naacl-long.326. URLhttps://aclanthology.org/2025. naacl-long.326/. Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. Having beer after prayer? measuring cultural bias in large language models. InProceedings of the 62nd Annual Meeting of the Association for Computa...

    work page doi:10.18653/v1/2025.naacl-long.326 2025

  22. [26]

    ”you gotta be a doctor, Lin”: An investi- gation of name-based bias of large language models in employment recommendations.arXiv preprint arXiv:2406.12232,

    12 Camellia(Preprint) Huy Nghiem, John Prindle, Jieyu Zhao, and Hal Daum ´e Iii. ”you gotta be a doctor, Lin”: An investi- gation of name-based bias of large language models in employment recommendations.arXiv preprint arXiv:2406.12232,

    Pith/arXiv arXiv

  23. [27]

    FORK: A bite-sized test set for probing culinary cultural biases in com- monsense reasoning models

    Shramay Palta and Rachel Rudinger. FORK: A bite-sized test set for probing culinary cultural biases in com- monsense reasoning models. InFindings of the Association for Computational Linguistics: ACL 2023, pp. 9952–9962,

    2023

  24. [28]

    BBQ: A hand-built bias benchmark for question answering.arXiv preprint arXiv:2110.08193,

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R Bowman. BBQ: A hand-built bias benchmark for question answering.arXiv preprint arXiv:2110.08193,

    Pith/arXiv arXiv

  25. [31]

    Measuring south asian biases in large language models.arXiv preprint arXiv:2505.18466,

    Mamnuya Rinki, Chahat Raj, Anjishnu Mukherjee, and Ziwei Zhu. Measuring south asian biases in large language models.arXiv preprint arXiv:2505.18466,

    Pith/arXiv arXiv

  26. [32]

    Include: Evaluating multilingual language understanding with regional knowledge.arXiv preprint arXiv:2411.19799,

    Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A Haggag, Alfonso Amayuelas, et al. Include: Evaluating multilingual language understanding with regional knowledge.arXiv preprint arXiv:2411.19799,

    Pith/arXiv arXiv

  27. [33]

    CVQA: Culturally-diverse multilingual visual question answering benchmark.arXiv preprint arXiv:2406.05967,

    David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, et al. CVQA: Culturally-diverse multilingual visual question answering benchmark.arXiv preprint arXiv:2406.05967,

    Pith/arXiv arXiv

  28. [34]

    IndiBias: A benchmark dataset to measure social biases in language models for Indian context.arXiv preprint arXiv:2403.20147,

    Nihar Ranjan Sahoo, Pranamya Prashant Kulkarni, Narjis Asad, Arif Ahmad, Tanu Goyal, Aparna Garimella, and Pushpak Bhattacharyya. IndiBias: A benchmark dataset to measure social biases in language models for Indian context.arXiv preprint arXiv:2403.20147,

    Pith/arXiv arXiv

  29. [35]

    ISBN 979-8-89176-251-0

    Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.919. URLhttps://aclanthology.org/2025.acl-long.919/. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram´e, Morgane Rivi`ere, et al. Gemma 3 technical report.arXiv preprin...

    Pith/arXiv arXiv 2025

  30. [36]

    On evaluating and mitigating gender biases in multi- lingual settings.arXiv preprint arXiv:2307.01503,

    Aniket Vashishtha, Kabir Ahuja, and Sunayana Sitaram. On evaluating and mitigating gender biases in multi- lingual settings.arXiv preprint arXiv:2307.01503,

    Pith/arXiv arXiv

  31. [37]

    kelly is a warm per- son, joseph is a role model

    Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai-Wei Chang, and Nanyun Peng. “kelly is a warm per- son, joseph is a role model”: Gender biases in llm-generated reference letters. InFindings of the Association for Computational Linguistics: EMNLP 2023, pp. 3730–3748,

    2023

  32. [38]

    Multilingual prompting for improving llm generation diversity.arXiv preprint arXiv:2505.15229,

    13 Camellia(Preprint) Qihan Wang, Shidong Pan, Tal Linzen, and Emily Black. Multilingual prompting for improving llm generation diversity.arXiv preprint arXiv:2505.15229,

    arXiv

  33. [39]

    Worldcuisines: A massive-scale benchmark for multilingual and multicultural visual question answering on global cuisines

    Genta Indra Winata, Frederikus Hudi, Patrick Amadeus Irawan, David Anugraha, Rifki Afina Putri, Yutong Wang, Adam Nohejl, Ubaidillah Ariq Prathama, Nedjma Ousidhoum, Afifa Amriani, et al. Worldcuisines: A massive-scale benchmark for multilingual and multicultural visual question answering on global cuisines. arXiv preprint arXiv:2410.12705,

    Pith/arXiv arXiv

  34. [40]

    Low frequency names exhibit bias and overfitting in contextualizing language models.arXiv preprint arXiv:2110.00672,

    Robert Wolfe and Aylin Caliskan. Low frequency names exhibit bias and overfitting in contextualizing language models.arXiv preprint arXiv:2110.00672,

    Pith/arXiv arXiv

  35. [41]

    mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483–498,

    2021

  36. [42]

    Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang

    URLhttps://arxiv.org/abs/2412.15115. Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang. Geomlama: Geo-diverse commonsense probing on multilingual pre-trained language models.arXiv preprint arXiv:2205.12247,

    Pith/arXiv arXiv

  37. [43]

    Chbias: Bias evaluation and mitigation of chinese conversational language models.arXiv preprint arXiv:2305.11262,

    Jiaxu Zhao, Meng Fang, Zijing Shi, Yitong Li, Ling Chen, and Mykola Pechenizkiy. Chbias: Bias evaluation and mitigation of chinese conversational language models.arXiv preprint arXiv:2305.11262,

    Pith/arXiv arXiv

  38. [44]

    Does mapo tofu contain coffee? probing llms for food-related cultural knowledge.arXiv preprint arXiv:2404.06833,

    Li Zhou, Taelin Karidi, Wanlong Liu, Nicolas Garneau, Yong Cao, Wenyu Chen, Haizhou Li, and Daniel Hershcovich. Does mapo tofu contain coffee? probing llms for food-related cultural knowledge.arXiv preprint arXiv:2404.06833,

    Pith/arXiv arXiv

  39. [2017]

    Normad: A framework for measuring the cultural adaptability of large language models.arXiv preprint arXiv:2404.12464,

    Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. Normad: A framework for measuring the cultural adaptability of large language models.arXiv preprint arXiv:2404.12464,

    Pith/arXiv arXiv

  40. [2018]

    Multilingual holistic bias: Extending descriptors and patterns to unveil demographic biases in languages at scale.arXiv preprint arXiv:2305.13198,

    Marta R Costa-juss`a, Pierre Andrews, Eric Smith, Prangthip Hansanti, Christophe Ropers, Elahe Kalbassi, Cyn- thia Gao, Daniel Licht, and Carleigh Wood. Multilingual holistic bias: Extending descriptors and patterns to unveil demographic biases in languages at scale.arXiv preprint arXiv:2305.13198,

    Pith/arXiv arXiv

  41. [2021]

    Presumed cultural identity: How names shape llm responses, 2025a

    Siddhesh Pawar, Arnav Arora, Lucie-Aim´ee Kaffee, and Isabelle Augenstein. Presumed cultural identity: How names shape llm responses, 2025a. URLhttps://arxiv.org/abs/2502.11995. Siddhesh Pawar, Junyeong Park, Jiho Jin, Arnav Arora, Junho Myung, Srishti Yadav, Faiz Ghifari Haznitrama, Inhwa Song, Alice Oh, and Isabelle Augenstein. Survey of cultural awaren...

    Pith/arXiv arXiv

  42. [2022]

    Culturalteaming: Ai-assisted interactive red-teaming for challenging llms’(lack of) multicultural knowledge.arXiv preprint arXiv:2404.06664, 2024a

    Yu Ying Chiu, Liwei Jiang, Maria Antoniak, Chan Young Park, Shuyue Stella Li, Mehar Bhatia, Sahithya Ravi, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. Culturalteaming: Ai-assisted interactive red-teaming for challenging llms’(lack of) multicultural knowledge.arXiv preprint arXiv:2404.06664, 2024a. Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young ...

    Pith/arXiv arXiv

  43. [2023]

    they are uncultured

    Preetam Prabhu Srikar Dammu, Hayoung Jung, Anjali Singh, Monojit Choudhury, and Tanu Mitra. “they are uncultured”: Unveiling covert harms and social threats in LLM generated conversations. InProceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 20339–20369, Miami, Florida, USA, November

    2024

  44. [2024]

    doi: 10.18653/v1/2024.emnlp-main.882

    Association for Computational Lin- guistics. doi: 10.18653/v1/2024.emnlp-main.882. URLhttps://aclanthology.org/2024.emnlp-main. 882/. 10 Camellia(Preprint) Fakhraddin Alwajih, Abdellah El Mekki, Samar Mohamed Magdy, Abdelrahim A Elmadany, Omer Nacar, El Moatez Billah Nagoudi, Reem Abdel-Salam, Hanin Atwany, Youssef Nafea, Abdulfattah Mohammed Yahya, et al...

    Pith/arXiv arXiv 2024

  45. [2025]

    Do large language models discriminate in hiring decisions on the basis of race, ethnicity, and gender?arXiv preprint arXiv:2406.10486,

    Haozhe An, Christabel Acquaye, Colin Wang, Zongxia Li, and Rachel Rudinger. Do large language models discriminate in hiring decisions on the basis of race, ethnicity, and gender?arXiv preprint arXiv:2406.10486,

    Pith/arXiv arXiv