Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

On The Origin of Cultural Biases in Language Models: From Pre-training Data to Linguistic Phenomena

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that Western cultural bias in LMs in Arabic stems less from training-data imbalance alone and more from Arabic word polysemy, cross-script lexical overlap, and frequency-based tokenization.

desk verdict Solid benchmark and a credible polysemy finding, but the tokenization claim is over-sold and should be read as correlational. read the letter →

arxiv 2501.04662 v1 pith:2KT5OONI submitted 2025-01-08 cs.CL

Tarek Naous , Wei Xu This is my paper · ORCID
classification cs.CL
keywords culturalbiasArabicNLPlanguagemodelsnamedentityrecognitionpolysemytokenizationmultilingualevaluationpre-trainingdata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks why language models that work in English show a stubborn preference for Western-culture entities when prompted in Arabic. It introduces CAMeL-2, a parallel Arabic–English benchmark of 58,086 cultural entities and 367 masked contexts, and finds that the performance gap between Arab and Western entities largely disappears in English but widens in Arabic. The origin, the authors argue, is not simply under-representation: Arabic entities that double as common polysemous words, that resemble high-frequency words in other Arabic-script languages, and that tokenizers merge into a single token are the ones LMs fail on. The failure gets worse as models add larger Arabic vocabularies, because frequency-based tokenization encodes polysemous entities the same way as their non-entity senses.

What carries the argument

The load-bearing apparatus is CAMeL-2, a parallel Arabic–English benchmark of 58,086 entities across seven entity types with 367 naturally masked contexts, built on the earlier CAMeL benchmark and extended with Wikipedia category extraction and OpenStreetMap location data. Around it, the analysis instruments are log-spaced occurrence counts of each entity in the Arabic and English mC4 portions computed with Aho-Corasick; the Almaany dictionary to tag entity words as polysemous; part-of-speech entropy over sampled corpus sentences to quantify how often entity words appear as adjectives or verbs in Arabic versus nouns in English; and token counts with Arabic-vocabulary sizes to connect subword tokenization to the failure.

What would settle it

A controlled comparison on one tested LM using its actual pre-training corpus: take matched sets of Arab entities that differ only in polysemy status, holding frequency bin and token count constant, and run extractive QA in Arabic; if the polysemous versus non-polysemous accuracy gap vanishes, the paper's causal chain breaks. Alternatively, re-tokenize the same Arabic text with a vocabulary that splits polysemous entity words into subwords; if performance does not improve, the tokenization claim is falsified.

Watch

Extended reading notes

Core claim

The authors' central claim is that entity-related cultural bias in LMs is co-produced by pre-training data statistics and by structural properties of Arabic. Using CAMeL-2, they show that, when tested in Arabic, LMs (Llama-3.3, Qwen-2.5, Aya-23, AceGPTv2, JAIS, and several encoder models) consistently recognize Western entities better than Arab entities, with gaps up to 15% accuracy on extractive QA and 27 F1 points on NER; when the same entities and translated contexts are given in English, the gaps shrink to near zero. The paper attributes this to four interacting causes: Arab entities appearing at very high frequencies in Arabic pre-training data (approximated by mC4) are often Arabic words with multiple senses; the degree of polysemy tracks the regional linguistic substrate behind location names; lexical overlap with other Arabic-script languages worsens performance; and frequency-based subword tokenization, especially into single tokens for polysemous words, is the encoding mechanism that makes the confusion concrete—an effect that amplifies with larger Arabic vocabularies.

Load-bearing premise

The analysis assumes that the mC4 corpus's Arabic entity frequencies stand in for the actual, unpublished pre-training data of the tested LMs; if their real training corpora distribute entities differently, the frequency-based explanation and the lexical-overlap results may not reflect the true cause.

Editorial extensions

If this is right

  • Testing an LM in English hides the cultural bias: the same Arab entities and translated contexts show near-zero gaps in English, so cross-lingual benchmarking is necessary to expose entity-related bias.
  • Very frequent and very rare Arab entities are the failure zones; medium-frequency entities in the roughly 1k–100k occurrence range are handled best in both languages.
  • Location names in countries whose names come from non-Arabic substrate languages, such as Comoros, fare better than names that are ordinary Arabic words, so LM performance on Arab locations is predictable from the percentage of polysemous names.
  • High-frequency overlap with Farsi, Urdu, Kurdish, Tajik, and Pashto harms extraction of Arab locations and food entities but less so personal names.
  • Larger Arabic vocabularies, intended to improve Arabic support, can make entity recognition worse for one-token polysemous entities unless tokenization changes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same mechanism likely applies to other languages whose writing systems reuse the Arabic script or whose place and food names coincide with common words, such as Hebrew and Persian; extending this analysis to those languages would test whether polysemy-driven bias is a general phenomenon rather than Arabic-specific.
  • The vocabulary-size result suggests a trade-off for Arabic LLM development: efficient single-token encoding of frequent words competes directly with disambiguation of entities, and a tokenizer that marks named-entity boundaries or preserves context might keep both.
  • Because the paper's frequency proxy is mC4, its quantitative slopes, such as the drop above 1M occurrences, may shift if measured against each model's true corpus; the qualitative ordering—polysemous, cross-script-overlapping, one-token entities fail—is the more robust claim.
  • CAMeL-2's parallel context set could be reused for tasks beyond NER and QA, such as generation, translation, or retrieval probing, to see whether the entities that fail extraction also produce stereotyped or Western-leaning continuations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper investigates why entity-related cultural biases in language models are more pronounced in Arabic than in English, and attributes the gap to pre-training frequency, Arabic word polysemy, lexical overlap with other Arabic-script languages, and subword tokenization. The authors introduce CAMeL-2, a parallel Arabic-English benchmark of 58,086 cultural entities and 367 masked contexts, and evaluate several causal and encoder LMs on text-infilling, extractive QA, and NER. They report that Arab-Western performance gaps are consistently smaller in English than in Arabic, and that high-frequency entities in the mC4 corpus are harder for LMs in Arabic, that polysemous Arabic location names remain hard even for large models, that entities frequent in other Arabic-script languages cause additional drops, and that one-token tokenization with large Arabic vocabularies is associated with lower accuracy. The paper frames these analyses as uncovering the origins of cultural bias in terms of frequency, linguistic phenomena, and tokenization.

Significance. If the findings hold, this is a useful contribution: it extends cultural-bias analysis beyond representational imbalance to concrete linguistic mechanisms, introduces a reusable parallel benchmark, and provides convergent evidence across multiple model families and tasks. The paper is transparent about the mC4 proxy for pre-training data in §4.1, releases the new resource, and grounds the polysemy analysis in Arabic-specific linguistic facts. The main caveat is that the tokenization section supports only an association, not the causal claim stated in the abstract and conclusion; the vocabulary-size comparison in Figure 8 lacks any controlled manipulation, and the one-token versus multi-token comparisons are confounded with frequency and polysemy. Because tokenization is one of the paper's headline explanatory factors, this overclaim is load-bearing and needs to be fixed or softened before publication.

major comments (3)
  1. [§4.4, Figures 7-8] The claim that "frequency-based tokenization leads to this issue in LMs, which gets worse with larger Arabic vocabularies" is not supported by the presented comparisons. Figure 8 plots one point per model, with vocabulary size varying simultaneously with model family, parameter count, architecture, training data, and tokenizer (Llama-3.3-70B, Aya23-35B, AceGPTv1.5-13B, JAIS-13B for QA; XLM-R, CAMeLBERT, AraBERT, MARBERT, ARBERT for NER), so vocabulary size is not isolated as a cause. Figure 7 and Appendix Figure 18 compare one-token versus multi-token entities within a model, but one-token entities are typically the most frequent and most polysemous entities, so token count is confounded with the variables already identified in §§4.1-4.2. To support the causal statement, the authors need a controlled comparison, such as models with the same architecture and training data but different vocabulary sizes, or matched entity sets that vary token count while holding frequency and polysemy fixed. Without such evidence, the tokenization claim should be presented as a suggestive correlation rather than a demonstrated mechanism.
  2. [§4.1, Figure 3 and §4.3, Figure 6] The frequency-stratified analysis and the Arabic-script lexical-overlap analysis depend entirely on mC4 counts as a proxy for the pre-training corpora of Llama, Qwen, Aya, and AceGPT. The paper acknowledges this approximation in §4.1, but it does not assess how sensitive the conclusions are to the choice of proxy. If the entity frequency distribution in mC4 differs from the actual training distribution of these models, the U-shaped accuracy curves and the cross-script frequency trends could reflect properties of the proxy rather than the mechanism claimed. At minimum, the authors should validate the counts against at least one other public web corpus (for example, RedPajama, SlimPajama, or Falcon RefinedWeb) and report whether the high-frequency drop and the script-overlap trend survive under that alternative proxy.
  3. [§4.1 and §4.2] The causal wording around frequency and polysemy is stronger than the observational evidence warrants. The high-frequency bin in Figure 3 is heavily populated by common Arabic words that also have non-entity senses, as the authors themselves note, so the performance drop cannot be attributed to frequency alone without controlling for polysemy and tokenization. The polysemy analysis in §4.2 uses the top-100 most frequent locations per country, but countries differ in their overall frequency distributions and in the prevalence of polysemous names, so the comparison between polysemous and non-polysemous entities does not hold frequency constant. A stratified analysis or a regression with frequency, polysemy, and script-overlap as covariates, together with confidence intervals, would substantially strengthen the central claim. The hand-chosen log-spaced frequency bins in §4.1 also deserve a robustness check.
minor comments (5)
  1. [§2.2] The text reads "88.34% of entities in CAMel-2", but the benchmark is consistently capitalized as CAMeL-2 elsewhere; this inconsistent capitalization appears in at least one other place in the manuscript.
  2. [Figure 8] The combined y-axis label "QA Accuracy" and "NER F1" is ambiguous because the two panels use different metrics and scales; each panel should carry its own axis label, and the small number of points should be supplemented with error bars or per-model confidence intervals.
  3. [§3.3, Tables 3 and 5] The reported accuracy and F1 differences are not accompanied by any measure of uncertainty, even though the QA evaluation samples 1,000 entities per type and the underlying entity sets vary in size; standard errors or bootstrap intervals would help the reader judge whether the small English-language gaps are meaningful.
  4. [§4.4] The prose first says that tokenization "likely" makes recognition harder and then the conclusion states the mechanism as established; the hedging should be aligned with the evidence, or the controls needed for a causal statement should be added.
  5. [Figure 5] The x-axis "% Polysemous Words" is used for both Arab and Western panels, but Western locations are almost uniformly non-polysemous; adding entity counts and country labels to the panels would make the relationship easier to assess.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central analyses are empirically grounded and not equivalent to their inputs.

full rationale

The paper's central claims (high-frequency and polysemous Arab entities are harder; tokenization correlates with the drop) are derived from held-out evaluations on CAMeL-2, which is an extension of the authors' prior CAMeL benchmark. The prior CBS measure and CAMeL resource are cited, but the new evidence is external: Llama, Qwen, Aya, AceGPT, JAIS, XLM-R, ARBERT, and other models are evaluated on unseen entities and contexts, and NER fine-tuning excludes entities seen during training. No parameter is fitted and then relabeled as a prediction; the mC4 approximation is an acknowledged limitation about the proxy corpus, not a definitional equivalence between the frequency measure and the outcome. The tokenization finding in Section 4.4 is a cross-model correlation, so its causal strength is contestable, but that is a correctness and control concern, not circularity. Self-citations to Naous et al. (2024) provide the benchmark and metric but do not by themselves force the paper's conclusions, and no uniqueness theorem or ansatz is imported from the authors' prior work. The derivation chain therefore does not reduce to its inputs.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The central claims are empirical and depend on several domain assumptions about data proxies, metric validity, and annotation quality. No new physical or theoretical entities are introduced. The only hand-chosen numeric inputs are frequency bin thresholds used for stratification.

free parameters (2)
  • high_frequency_threshold = 10^6 occurrences in mC4
    Section 4.1 defines 'very high frequencies' as the 10^6-10^8 bin; the threshold is hand-chosen for binning and affects which entities are considered problematic.
  • frequency_bin_edges = log10-spaced bins (1-10, 10-100, ..., 10^6-10^8)
    Section 4.1 groups entity occurrence counts into log10-spaced bins; these edges are chosen for visualization and stratification and influence the reported trends.
assumptions (6)
  • domain assumption mC4 approximates the pre-training corpora of evaluated LMs
    Section 4.1 states that pre-training corpora are not public and uses mC4 as a proxy for entity occurrence frequencies; if real pre-training distributions differ, the frequency-based analyses may not transfer.
  • domain assumption Cultural Bias Score (CBS) is a valid measure of cultural adaptation
    Section 3.2 uses CBS from Naous et al. (2024) as the likelihood-based metric for text-infilling; the conclusions about cultural bias rely on this metric's validity.
  • domain assumption Almaany dictionary provides accurate polysemy labels
    Section 4.2 uses Almaany to determine whether entities are polysemous Arabic words; errors in this labeling would change the polysemy effect.
  • domain assumption Manual cultural annotations are reliable
    Section 2.2 reports Cohen's Kappa 0.825, which is substantial but not perfect; annotation errors could affect the Arab-vs-Western split.
  • domain assumption Distant supervision from Wikipedia yields valid NER training data
    Section 3.3 and Appendix B.3 fine-tune NER models on automatically constructed Wikipedia sentences; noise in this distant supervision could affect NER F1 scores.
  • domain assumption Performance differences reflect cultural bias rather than task artifacts
    The paper interprets accuracy/F1 gaps between Arab and Western entities as cultural bias; other factors like entity name length or context difficulty could contribute.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On The Origin of Cultural Biases in Language Models: From Pre-training Data to Linguistic Phenomena." pith.science (2026). https://pith.science/paper/2KT5OONI

@misc{pith2026250104662,
  author       = {Pith},
  title        = {Pith review of: On The Origin of Cultural Biases in Language Models: From Pre-training Data to Linguistic Phenomena},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2KT5OONI}},
  note         = {Machine review of arXiv:2501.04662}
}
read the original abstract

Language Models (LMs) have been shown to exhibit a strong preference towards entities associated with Western culture when operating in non-Western languages. In this paper, we aim to uncover the origins of entity-related cultural biases in LMs by analyzing several contributing factors, including the representation of entities in pre-training data and the impact of variations in linguistic phenomena across languages. We introduce CAMeL-2, a parallel Arabic-English benchmark of 58,086 entities associated with Arab and Western cultures and 367 masked natural contexts for entities. Our evaluations using CAMeL-2 reveal reduced performance gaps between cultures by LMs when tested in English compared to Arabic. We find that LMs struggle in Arabic with entities that appear at high frequencies in pre-training, where entities can hold multiple word senses. This also extends to entities that exhibit high lexical overlap with languages that are not Arabic but use the Arabic script. Further, we show how frequency-based tokenization leads to this issue in LMs, which gets worse with larger Arabic vocabularies. We will make CAMeL-2 available at: https://github.com/tareknaous/camel2

Figures

Figures reproduced from arXiv: 2501.04662 by the authors.

Figure 1
Figure 1. Responses of a LM ( ) tasked to extract the food dish from the same text in English and Arabic. The LM identifies the Arab dish “Makloube” in English, but fails in Arabic where the word “Makloube” holds two senses. The LM does not struggle with the Western dish “Lasagna” which holds only one sense in both languages. While entity-related biases in LMs have been traditionally studied as a reflection of imbalanced repr… view at source ↗
Figure 2
Figure 2. Cultural Bias Score (↓) (§3.2) per entity type on culturally-grounded contexts from CAMeL-2. LMs can adapt better to Arab culture when tested in English. Llama3.3-70b XLMRlarge Arabic English Arabic English Arab Western ∆Acc Arab Western ∆Acc Arab Western ∆F1 Arab Western ∆F1 Authors 92.62 90.28 -2.34 98.99 99.16 0.17 86.80 87.93 1.13 95.64 94.98 -0.66 Beverage 82.65 78.19 -4.46 99.14 97.71 -1.43 63.06 72.86 9.80 92… view at source ↗
Figure 3
Figure 3. Average QA Accuracy (↑) of LLMs when tested in Arabic and English on location, name, food, and beverage associated with Arab culture, stratified by their occurrence counts in the mC4 corpus (§4.1; grouped into log10-spaced bins). Gray bars in background represent number of entities tested in each bin. Interestingly, LMs struggle with very high-frequency entities in Arabic. texts per entity), as well as a correspondi… view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: QA accuracy (↑) for different sizes of LMs on high-frequency polysemous and non-polysemous Arab location entities. We find a positive scaling trend for all models, with lower performance on polysemous entities. context. This is less common for entities in English, wher…
Figure 5
Figure 5. Figure 5: Average QA Accuracy (↑) of Llama3.3-70b on the top-100 most frequent location entities in mC4 for each Arab and Western country in CAMeL-2 (§4.2). Arab countries are grouped by the language family that influences location naming in their region. Performance on Arab loc…
Figure 6
Figure 6. Figure 6: QA Accuracy (↑) and CBS (↓) of LMs on Arab location, name, and food entities vs. their total count in other languages that use Arabic script (Farsi, Urdu, Tajik, Pashto, Kurdish) in mC4 corpus. Performance decreases for all LMs as entities appear more frequently in oth…
Figure 7
Figure 7. Figure 7: Performance distribution of Llama3.3-70b, JAIS-13b, and ARBERT on Arab location entities, in relation [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Average QA Accuracy and F1 of LMs on one [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Distribution of entities in CAMeL-2 for each [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 11
Figure 11. Figure 11: Prompt template used perform extractive QA [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: POS tag distribution of the 100 most frequent food, name, and location entities in Arabic and English [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Map of the regional influencing languages on location names in Arab countries, and their standard [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Average QA Accuracy (↑) of LLMs when tested in Arabic and English on sports clubs and reli￾gious places of worship associated with Arab culture, stratified by their occurrence counts in the mC4 corpus (grouped into log10-spaced bins). D Analyses: Additional Results D.…
Figure 15
Figure 15. Figure 15: Average CBS (↓) of LLMs at text-infilling when tested in Arabic and English on locations, names, and food entities associated with Arab culture, stratified by their occurrence counts in the mC4 corpus (grouped into log10-spaced bins all entity types. This indicates th…
Figure 16
Figure 16. Figure 16: Average NER F1 of XLMRlarge on the top-100 most frequent location entities in mC4 for each Arab country (top) and Western country (bottom) in CAMeL-2. Comoros 0 20 40 60 80 100 F1 Niger-Congo Somalia Djibouti Sudan 0 20 40 60 80 100 Cushitic Mauritania Morocco Tunisia…
Figure 17
Figure 17. Figure 17: Average NER F1 of ARBERT on the top-100 most frequent location entities in mC4 for each Arab [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]
Figure 18
Figure 18. Figure 18: Performance distribution of several LMs on Arab location entities as a function of how many tokens [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Igniting Creative Writing in Small Language Models: LLM-as-a-Judge versus Multi-Agent Refined Rewards

    cs.CL 2025-08 conditional novelty 4.0 of 10

    An adversarially tuned LLM-as-a-Judge reward signal outperforms a multi-agent-refined reward model for fine-tuning a 7B SLM on Chinese greeting generation, though the comparison is weakened by circular evaluation and ...

Reference graph

Works this paper leans on

61 extracted references · 26 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020. ARBERT & MARBERT : Deep bidirectional transformers for Arabic . arXiv preprint arXiv:2101.01785

    arXiv 2020

  4. [4]

    Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Singh, Ashutosh Dwivedi, Alham Fikri Aji, Jacki O'Neill, Ashutosh Modi, and Monojit Choudhury. 2024. Towards measuring and modeling ``culture''' in LLMs : A survey. arXiv preprint arXiv:2403.15412

    arXiv 2024

  5. [5]

    Badr AlKhamissi, Muhammad ElNokrashy, Mai AlKhamissi, and Mona Diab. 2024. Investigating cultural alignment of large language models. arXiv preprint arXiv:2402.13231

    arXiv 2024

  6. [6]

    Haozhe An, Christabel Acquaye, Colin Wang, Zongxia Li, and Rachel Rudinger. 2024. Do large language models discriminate in hiring decisions on the basis of race, ethnicity, and gender? arXiv preprint arXiv:2406.10486

    arXiv 2024

  7. [7]

    Haozhe An and Rachel Rudinger. 2023. Nichelle and Nancy : The influence of demographic attributes and tokenization length on first name biases. arXiv preprint arXiv:2305.16577

    work page Pith review arXiv 2023

  8. [8]

    Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT : Transformer-based model for arabic language understanding. In LREC Workshop Language Resources and Evaluation Conference 11--16 May 2020, page 9

    work page 2020

Show all 61 references
  1. [9]

    Shane Arora, Marzena Karpinska, Hung-Ting Chen, Ipsita Bhattacharjee, Mohit Iyyer, and Eunsol Choi. 2024. CaLMQA : Exploring culturally specific long-form question answering across 23 languages. arXiv preprint arXiv:2406.17761

    2024 arXiv

  2. [10]

    Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Kelly Marchisio, Sebastian Ruder, et al. 2024. Aya 23: Open weight releases to further multilingual progress. arXiv preprint arXiv:2405.15032

    2024 arXiv

  3. [11]

    Yassine Benajiba, Paolo Rosso, and Jos \'e Miguel Bened \' ruiz. 2007. Anersys: An arabic named entity recognition system based on maximum entropy. In Computational Linguistics and Intelligent Text Processing: 8th International Conference, CICLing 2007, Mexico City, Mexico, Fe...

    2007

  4. [12]

    Shaily Bhatt and Fernando Diaz. 2024. Extrinsic evaluation of cultural competence in large language models. arXiv preprint arXiv:2406.11565

    2024

  5. [13]

    Smith, and Luke Zettlemoyer

    Terra Blevins, Tomasz Limisiewicz, Suchin Gururangan, Margaret Li, Hila Gonen, Noah A. Smith, and Luke Zettlemoyer. 2024. Breaking the curse of multilinguality with cross-lingual expert language models. arXiv preprint arXiv:2401.10440

    2024 arXiv

  6. [14]

    Kaj Bostrom and Greg Durrett. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.414 Byte Pair Encoding is suboptimal for language model pretraining . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4617--4624, Online. Association for Computa...

    2020 doi

  7. [15]

    Yu Ying Chiu, Liwei Jiang, Maria Antoniak, Chan Young Park, Shuyue Stella Li, Mehar Bhatia, Sahithya Ravi, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. 2024. CulturalTeaming : AI -assisted interactive red-teaming for challenging LLMs '(lack of) multicultural knowledge. arXiv...

    2024 arXiv

  8. [16]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Me...

    2020

  9. [17]

    Kareem Darwish and Hamdy Mubarak. 2016. Farasa: A new fast and accurate Arabic word segmenter. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1070--1074

    2016

  10. [18]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  11. [19]

    Yi Fung, Tuhin Chakrabarty, Hao Guo, Owen Rambow, Smaranda Muresan, and Heng Ji. 2023. NORMSAGE : Multi-lingual multi-cultural norm discovery from conversations on-the-fly. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15217--15230

    2023

  12. [20]

    Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, pages 1--79

    2024

  13. [21]

    Vagrant Gautam, Arjun Subramonian, Anne Lauscher, and Os Keyes. 2024. Stop! I n the name of flaws: Disentangling personal names and sociodemographic attributes in NLP . arXiv preprint arXiv:2405.17159

    2024 arXiv

  14. [22]

    Gene Gragg. 2019. Semitic and Afro-Asiatic . In The Semitic Languages, pages 22--48. Routledge

    2019

  15. [23]

    Jing Huang and Diyi Yang. 2023. Culturally aware natural language inference. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7591--7609

    2023

  16. [24]

    Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in arabic pre-trained language models. arXiv preprint arXiv:2103.06678

    2021 arXiv

  17. [25]

    Sullam Jeoung, Jana Diesner, and Halil Kilicoglu. 2023. Examining the causal effect of first names on language models: The case of social commonsense reasoning. arXiv preprint arXiv:2306.01117

    2023 arXiv

  18. [26]

    Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, pages 15696--15707. PMLR

    2023

  19. [27]

    Amr Keleg and Walid Magdy. 2023. DLAMA : A framework for curating culturally diverse facts for probing the knowledge of pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6245--6266

    2023

  20. [28]

    Taku Kudo. 2018. https://doi.org/10.18653/v1/P18-1007 Subword regularization: Improving neural network translation models with multiple subword candidates . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page...

    2018 doi

  21. [29]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Princip...

    2023

  22. [30]

    Huihan Li, Arnav Goel, Keyu He, and Xiang Ren. 2024 a . https://arxiv.org/abs/2412.20760 Attributing culture-conditioned generations to pretraining corpora . Preprint, arXiv:2412.20760

    2024 arXiv

  23. [31]

    Wenyan Li, Xinyu Zhang, Jiaang Li, Qiwei Peng, Raphael Tang, Li Zhou, Weijia Zhang, Guimin Hu, Yifei Yuan, Anders S gaard, Daniel Hershcovich, and Desmond Elliott. 2024 b . FoodieQA : A multimodal dataset for fine-grained understanding of chinese food culture. arXiv preprint a...

    2024 arXiv

  24. [32]

    Chen Liang, Yue Yu, Haoming Jiang, Siawpeng Er, Ruijia Wang, Tuo Zhao, and Chao Zhang. 2020. Bond: BERT -assisted open-domain named entity recognition with distant supervision. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining,...

    2020

  25. [33]

    Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa. 2023. Xlm-v: Overcoming the vocabulary bottleneck in multilingual masked language models. arXiv preprint arXiv:2301.10472

    2023 arXiv

  26. [34]

    Juhao Liang, Zhenyang Cai, Jianqing Zhu, Huang Huang, Kewei Zong, Bang An, Mosen Alharthi, Juncai He, Lian Zhang, Haizhou Li, et al. 2024. Alignment at pre-training! towards native alignment for Arabic llms. arXiv preprint arXiv:2412.03253

    2024 arXiv

  27. [35]

    Chen Cecilia Liu, Iryna Gurevych, and Anna Korhonen. 2024. Culturally aware and adapted NLP : A taxonomy and a survey of the state of the art. arXiv preprint arXiv:2406.03930

    2024 arXiv

  28. [36]

    Reem I Masoud, Ziquan Liu, Martin Ferianc, Philip Treleaven, and Miguel Rodrigues. 2023. Cultural alignment in large language models: An explanatory analysis based on hofstede's cultural dimensions. arXiv preprint arXiv:2309.12342

    2023 arXiv

  29. [37]

    Anjishnu Mukherjee, Aylin Caliskan, Ziwei Zhu, and Antonios Anastasopoulos. 2024. Global gallery: The fine art of painting culture portraits through multilingual instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Comp...

    2024

  30. [38]

    Tarek Naous, Michael Ryan, Alan Ritter, and Wei Xu. 2024. Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16366--16393

    2024

  31. [39]

    you gotta be a doctor, Lin

    Huy Nghiem, John Prindle, Jieyu Zhao, and Hal Daum \'e III. 2024. "you gotta be a doctor, Lin ": An investigation of name-based bias of large language models in employment recommendations. arXiv preprint arXiv:2406.12232

    2024 arXiv

  32. [40]

    Shramay Palta and Rachel Rudinger. 2023. FORK : A bite-sized test set for probing culinary cultural biases in commonsense reasoning models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9952--9962

    2023

  33. [41]

    Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher D Manning. 2018. Universal dependency parsing from scratch. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 160--170

    2018

  34. [42]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1--67

    2020

  35. [43]

    Michael Ramscar. 2019. Source codes in human communication. arXiv preprint arXiv:1904.03991

    2019 arXiv

  36. [44]

    Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. 2024. Normad: A benchmark for measuring the cultural adaptability of large language models. arXiv preprint arXiv:2404.12464

    2024 arXiv

  37. [45]

    Dwight F Reynolds. 2015. The Cambridge companion to modern Arab culture . Cambridge University Press

    2015

  38. [46]

    Erik Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142--147

    2003

  39. [47]

    Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, et al. 2023. JAIS and JAIS-Chat : Arabic -centric foundation and instruction-tuned open generative large language models. arXiv p...

    2023 arXiv

  40. [48]

    Agrima Seth, Sanchit Ahuja, Kalika Bali, and Sunayana Sitaram. 2024. DOSA : A dataset of social artifacts from different indian geographical subcultures. arXiv preprint arXiv:2403.14651

    2024 arXiv

  41. [49]

    Siqi Shen, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, Soujanya Poria, and Rada Mihalcea. 2024. Understanding the capabilities and limitations of large language models for cultural commonsense. In Proceedings of the 2024 Conference of the North American Chapter of the Asso...

    2024

  42. [50]

    Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Raya Horesh, Rog \'e rio Abreu de Paula, Diyi Yang, et al. 2024. Culturebank: An online community-driven knowledge base towards culturally aware language technologies. arXiv preprint arXiv:2404.15238

    2024 arXiv

  43. [51]

    Shivalika Singh, Freddie Vargus, Daniel D ' souza, B \"o rje Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O ' Mahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Moura, Dominik Krzemi \'n ski, Haki...

    2024

  44. [52]

    Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.160 Fast W ord P iece tokenization . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2089--2103, Online and P...

    2021 doi

  45. [53]

    Ahmet \"U st \"u n, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. 2024. Aya model: An instruction finetuned open-access multilingual language model. arXiv preprint arXiv:2402.07827

    2024 arXiv

  46. [54]

    Willy Van Langendonck. 2007. Theory and typology of proper names. Mouton de Gruyter

    2007

  47. [55]

    Kees Versteegh. 2014. Arabic language. Edinburgh University Press

    2014

  48. [56]

    Robert Wolfe and Aylin Caliskan. 2021. Low frequency names exhibit bias and overfitting in contextualizing language models. arXiv preprint arXiv:2110.00672

    2021 arXiv

  49. [57]

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5 : A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Associatio...

    2021

  50. [58]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  51. [59]

    Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang. 2022. GeoMLAMA : Geo-diverse commonsense probing on multilingual pre-trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2039--2055

    2022

  52. [60]

    Li Zhou, Taelin Karidi, Nicolas Garneau, Yong Cao, Wanlong Liu, Wenyu Chen, and Daniel Hershcovich. 2024. Does mapo tofu contain coffee? probing llms for food-related cultural knowledge. arXiv preprint arXiv:2404.06833

    2024 arXiv

  53. [61]

    Jianqing Zhu, Huang Huang, Zhihang Lin, Juhao Liang, Zhengyang Tang, Khalid Almubarak, Abdulmohsen Alharthik, Bang An, Juncai He, Xiangbo Wu, et al. 2024. Second language (Arabic) acquisition of llms via progressive vocabulary expansion. arXiv preprint arXiv:2412.12310

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.