pith. sign in

arxiv: 2412.20760 · v2 · submitted 2024-12-30 · 💻 cs.CL · cs.AI

Attributing Culture-Conditioned Generations to Pretraining Corpora

Huihan Li , Arnav Goel , Keyu He , Xiang Ren This is my paper

Pith reviewed 2026-05-23 07:03 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords memorizationpretraining datacultural biaslarge language modelsMEMOed frameworkculture-conditioned generationfrequency effects
0
0 comments X

The pith

The MEMOed framework links high-frequency pretraining cultures to more memorized generations about food and clothing.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper proposes the MEMOed framework to check if culture-conditioned generations come from memorizing pretraining documents. Applying it to food and clothing topics across 110 cultures reveals that cultures common in the pretraining data produce outputs with more memorized symbols. Cultures rare in the data sometimes produce none. The model also defaults to very frequent entities no matter the culture asked about. Readers care because it traces cultural bias back to specific patterns in the training data.

Core claim

We propose the MEMOed framework (MEMOrization from pretraining document) to determine whether a generation for a culture arises from memorization of pretraining documents based on observed data patterns. Using MEMOed on culture-conditioned generations about food and clothing for 110 cultures, we find that high-frequency cultures in pretraining data yield more generations with memorized symbols, while some low-frequency cultures produce none. Additionally, the model favors generating entities with extraordinarily high frequency regardless of the conditioned culture, reflecting biases toward frequent pretraining terms irrespective of relevance.

What carries the argument

The MEMOed framework, which determines whether a generation arises from memorization of pretraining documents.

If this is right

  • High-frequency cultures in pretraining data yield more generations with memorized symbols.
  • Some low-frequency cultures produce generations with no memorized symbols.
  • The model favors generating entities with high frequency regardless of the conditioned culture.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The MEMOed framework could be used to audit other types of biased outputs such as in political or historical topics.
  • Balancing pretraining data frequencies might help reduce cultural biases in model generations.
  • This approach provides a method to attribute specific model behaviors directly to training data patterns.

Load-bearing premise

The MEMOed framework can reliably determine whether a given generation arises from memorization of pretraining documents based on observed data patterns.

What would settle it

Finding that low-frequency cultures generate many symbols matching pretraining documents would challenge the frequency-memorization connection.

Figures

Figures reproduced from arXiv: 2412.20760 by Arnav Goel, Huihan Li, Keyu He, Xiang Ren.

Figure 1
Figure 1. Figure 1: Four types of culture-symbol associations in culture-conditioned generations [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: MEMO [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Higher contribution score means stronger evidence of culture/symbol association in pre [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Geographical Distribution of Memorized Association [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Overshadowing ratio r of all diffuse as￾sociation for topic clothing. However, not all memorized symbols are em￾blematic symbols to a culture. The rest of the symbols consist of entities that are still used in the culture a lot without being an emblem￾atic symbol: for example, “western style bridal gown” is recognized as a memorized symbol for Indian clothing, while “business suit” is rec￾ognized as a memo… view at source ↗
Figure 6
Figure 6. Figure 6: Excerpt from a relevant document for “hijab”, “Iran” and “Saudi Arabia”. Topic Modeling Analysis. In Section 3.4 we stated our hypothesis that model may gener￾alize the memorized symbols of one culture to another culture due to the two cultures’ co-occurrence in pretraining documents under certain common topics. Although a compre￾hensive study on each memorized symbol is computationally impossible, we exem… view at source ↗
Figure 7
Figure 7. Figure 7: While some cultures contain no memorized association in their generations (Fig7b), cul [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prompt for LLAMA-3.1-8B in Topic Modeling Pipeline B TOPIC MODELING B.1 METHODOLOGY For any culture C and its set of memorized symbols m(C), we select a symbol S ∈ m(C) and identify the set of cultures C ′ G which also generated S but not through a memorization. For each culture C ′ ∈ C ′ G and for C, we retrieve pre-training documents where the two cultures co-occur, forming a set Dcc′ . We apply the metr… view at source ↗
Figure 9
Figure 9. Figure 9: Example of Google Form Used for Cultural Food Annotation [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Sample Question from Google Form on Cultural Food Classification [PITH_FULL_IMAGE:figures/full_fig_p021_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Examples of excerpts from relevant pretraining docs for Culture: “Indian” and Symbol: [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Examples of excerpts from relevant pretraining docs for Culture: “Chinese” and Symbol: [PITH_FULL_IMAGE:figures/full_fig_p022_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Correlation b/w number of memo￾rized symbols from other cultures and pre-training counts for a culture For (2), our observations indicate that 34 cul￾tures related to clothing and 86 related to food have their memorized symbols being gener￾ated at least once in other cultures’ generations. Upon calculating correlations with these cul￾tures, we observed moderate-to-high correla￾tions for both clothing (Spe… view at source ↗
Figure 14
Figure 14. Figure 14: Cross-Culture Generalization Continuing from Section 4.6, in this section we expand upon our findings and present some more results across the 110 cultures. In Tables 10 and 11, we present the memorization and generalization statistics for food and clothing, respectively. Specifically, we provide the names of the top 5 and bottom 5 cultures, ranked by the percentage of their responses classified as either… view at source ↗
Figure 15
Figure 15. Figure 15: Distributions of China, India and Japan responses for Food [PITH_FULL_IMAGE:figures/full_fig_p026_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Clothing Stats - Mynammar and Yemen 3% of its responses qualify as memorization. In contrast, Saudi Arabia exhibits greater diversity, with significant percentages of both memorization and cross-culture generalization in its generated outputs. (a) USA (b) Saudi Arabia [PITH_FULL_IMAGE:figures/full_fig_p026_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Clothing Stats - USA and Saudi Arabia 26 [PITH_FULL_IMAGE:figures/full_fig_p026_17.png] view at source ↗
read the original abstract

In open-ended generative tasks like narrative writing or dialogue, large language models often exhibit cultural biases, showing limited knowledge and generating templated outputs for less prevalent cultures. Recent works show that these biases may stem from uneven cultural representation in pretraining corpora. This work investigates how pretraining leads to biased culture-conditioned generations by analyzing how models associate entities with cultures based on pretraining data patterns. We propose the MEMOed framework (MEMOrization from pretraining document) to determine whether a generation for a culture arises from memorization. Using MEMOed on culture-conditioned generations about food and clothing for 110 cultures, we find that high-frequency cultures in pretraining data yield more generations with memorized symbols, while some low-frequency cultures produce none. Additionally, the model favors generating entities with extraordinarily high frequency regardless of the conditioned culture, reflecting biases toward frequent pretraining terms irrespective of relevance. We hope that the MEMOed framework and our insights will inspire more works on attributing model performance on pretraining data.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 0 minor

Summary. The paper proposes the MEMOed framework to attribute whether culture-conditioned generations (about food and clothing for 110 cultures) arise from memorization of pretraining documents, based on observed data patterns. It reports that high-frequency cultures in pretraining data produce more generations containing memorized symbols while some low-frequency cultures produce none, and that models favor generating extraordinarily high-frequency entities regardless of the conditioned culture.

Significance. If the MEMOed framework can be shown to isolate memorization effects from other mechanisms, the work would offer a concrete empirical tool for tracing cultural biases in LLM outputs back to pretraining corpus imbalances. The scale (110 cultures) and the reported correlation between pretraining frequency and memorized-symbol rate could inform data-auditing practices, though the current manuscript provides no validation or controls that would allow this attribution to be assessed.

major comments (3)
  1. Abstract: The central claim that MEMOed 'determines whether a generation for a culture arises from memorization' rests on an undefined procedure. The abstract mentions only 'observed data patterns' with no algorithm, decision criteria, thresholds, or pseudocode, making it impossible to evaluate whether the reported patterns reflect memorization rather than frequency bias or generalization.
  2. Abstract and method description: No validation experiments, ground-truth checks (e.g., exact string matches to training documents), or error analysis are supplied for MEMOed. Without these, the attribution of generations to memorization cannot be distinguished from alternative explanations such as high-frequency token bias, prompt-induced templating, or cultural stereotypes acquired via generalization.
  3. Abstract: The finding that 'some low-frequency cultures produce none' and that the model 'favors generating entities with extraordinarily high frequency regardless of the conditioned culture' is presented without controls that would rule out non-memorization mechanisms; the reported correlation with pretraining frequency therefore does not yet establish the claimed causal link to memorization.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the detailed and constructive comments. We address each major point below and will revise the manuscript accordingly to improve clarity, add validation, and strengthen controls.

read point-by-point responses

  1. Referee: Abstract: The central claim that MEMOed 'determines whether a generation for a culture arises from memorization' rests on an undefined procedure. The abstract mentions only 'observed data patterns' with no algorithm, decision criteria, thresholds, or pseudocode, making it impossible to evaluate whether the reported patterns reflect memorization rather than frequency bias or generalization.

    Authors: We agree the abstract is insufficiently detailed on its own. Section 3 of the manuscript defines the MEMOed procedure via entity extraction, frequency-based matching to pretraining co-occurrences, and a threshold on symbol presence to attribute memorization. We will revise the abstract to briefly summarize the decision criteria and add a pointer to the method section (including pseudocode) so the attribution logic is evaluable from the abstract. revision: yes

  2. Referee: Abstract and method description: No validation experiments, ground-truth checks (e.g., exact string matches to training documents), or error analysis are supplied for MEMOed. Without these, the attribution of generations to memorization cannot be distinguished from alternative explanations such as high-frequency token bias, prompt-induced templating, or cultural stereotypes acquired via generalization.

    Authors: The current version presents the framework through observed frequency correlations without dedicated validation experiments or error analysis. We will add a new subsection with ground-truth checks (exact string matches on a held-out sample of pretraining documents), a confusion-matrix style error analysis, and comparison against a high-frequency token baseline to better isolate memorization effects from generalization or templating. revision: yes

  3. Referee: Abstract: The finding that 'some low-frequency cultures produce none' and that the model 'favors generating entities with extraordinarily high frequency regardless of the conditioned culture' is presented without controls that would rule out non-memorization mechanisms; the reported correlation with pretraining frequency therefore does not yet establish the claimed causal link to memorization.

    Authors: We acknowledge that the reported correlations alone do not fully rule out non-memorization mechanisms. In revision we will introduce explicit controls, including a frequency-shuffled baseline and comparison against a model trained on culturally balanced data, to test whether the observed patterns persist when pretraining frequency is decoupled from other factors. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical measurement study

full rationale

The paper proposes the MEMOed framework as an empirical tool for attributing generations to pretraining patterns via observed data correlations with culture frequency. No equations, fitted parameters, derivations, or self-citation chains are described that reduce the central claim to its own inputs by construction. The work is a measurement study correlating generations with pretraining frequency counts rather than a derivation that assumes its conclusion. This is the most common honest finding for purely empirical papers without load-bearing self-referential steps.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review supplies no information on free parameters, axioms, or invented entities; all fields left empty.

pith-pipeline@v0.9.0 · 5705 in / 1074 out tokens · 27018 ms · 2026-05-23T07:03:21.728145+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 30 canonical work pages · 4 internal anchors

  1. [1]

    Generalization vs memorization: Tracing language models’ capabilities back to pretraining data

    Antonis Antoniades, Xinyi Wang, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, and William Yang Wang. Generalization vs memorization: Tracing language models’ capabilities back to pretraining data. arXiv preprint arXiv:2407.14985,

    work page arXiv

  2. [2]

    Quantifying memorization across neural language models

    12 Published as a conference paper at ICLR 2025 Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations,

    work page 2025

  3. [3]

    How do large language models acquire factual knowledge during pretraining? arXiv preprint arXiv:2406.11813,

    Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang, Youngkyung Seo, Du-Seong Chang, and Minjoon Seo. How do large language models acquire factual knowledge during pretraining? arXiv preprint arXiv:2406.11813,

    work page arXiv

  4. [4]

    Unsupervised Cross-lingual Representation Learning at Scale

    A Conneau. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116,

    work page internal anchor Pith review Pith/arXiv arXiv 1911

  5. [5]

    The Llama 3 Herd of Models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Towards Measuring the Representation of Subjective Global Opinions in Language Models

    Esin Durmus, Karina Nyugen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCan- dlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. Towards measuring the representation of subjective global opinions in language mode...

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    Massively multi-cultural knowledge acquisition & lm benchmarking

    Yi Fung, Ruining Zhao, Jae Doo, Chenkai Sun, and Heng Ji. Massively multi-cultural knowledge acquisition & lm benchmarking. arXiv preprint arXiv:2402.09369,

    work page arXiv

  8. [8]

    In: Zong, C., Xia, F., Li, W., Navigli, R

    Association for Computational Linguistics. doi: 10.18653/v1/ 2024.acl-long.841. URL https://aclanthology.org/2024.acl-long.841. Christian W Haerpfer and Kseniya Kizilova. The world values survey. The Wiley-Blackwell Ency- clopedia of Globalization, pp. 1–5,

    work page doi:10.18653/v1/ 2024

  9. [9]

    Culturally aware natural language inference

    Jing Huang and Diyi Yang. Culturally aware natural language inference. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2023, pp. 7591–7609,

    work page 2023

  10. [10]

    Dlama: A framework for curating culturally diverse facts for probing the knowledge of pretrained language models

    13 Published as a conference paper at ICLR 2025 Amr Keleg and Walid Magdy. Dlama: A framework for curating culturally diverse facts for probing the knowledge of pretrained language models. arXiv preprint arXiv:2306.05076,

    work page arXiv 2025

  11. [11]

    Bean, Hannah Rose Kirk, and Scott A

    Khyati Khandelwal, Manuel Tonneau, Andrew M. Bean, Hannah Rose Kirk, and Scott A. Hale. Casteist but not racist? quantifying disparities in large language model bias between india and the west. ArXiv, abs/2309.08573,

    work page arXiv

  12. [12]

    arXiv preprint arXiv:2402.10946 (2024)

    Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. Culturellm: Incorpo- rating cultural differences into large language models. arXiv preprint arXiv:2402.10946, 2024a. Huihan Li, Liwei Jiang, Nouha Dziri, Xiang Ren, and Yejin Choi. Culture-gen: Revealing global cultural perception in language models through natural language prompting. ...

    work page arXiv

  13. [13]

    When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. arXiv preprint arXiv:2212.10511,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    Having beer after prayer? measuring cul- tural bias in large language models

    Tarek Naous, Michael Joseph Ryan, and Wei Xu. Having beer after prayer? measuring cul- tural bias in large language models. ArXiv, abs/2305.14456,

    work page arXiv

  15. [15]

    Extracting cultural commonsense knowledge at scale

    Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde, and Gerhard Weikum. Extracting cultural commonsense knowledge at scale. In Proceedings of the ACM Web Conference 2023, WWW ’23. ACM, April

    work page 2023

  16. [16]

    URL http://dx.doi.org/10.1145/ 3543507.3583535

    doi: 10.1145/3543507.3583535. URL http://dx.doi.org/10.1145/ 3543507.3583535. Shramay Palta and Rachel Rudinger. Fork: A bite-sized test set for probing culinary cultural biases in commonsense reasoning models. InFindings of the Association for Computational Linguistics: ACL 2023, pp. 9952–9962,

    work page doi:10.1145/3543507.3583535 2023

  17. [17]

    Knowledge of cultural moral norms in large language models

    Aida Ramezani and Yang Xu. Knowledge of cultural moral norms in large language models. arXiv preprint arXiv:2306.01857,

    work page arXiv

  18. [18]

    Unintended impacts of llm alignment on global representation

    Michael J Ryan, William Held, and Diyi Yang. Unintended impacts of llm alignment on global representation. arXiv preprint arXiv:2402.15018,

    work page arXiv

  19. [19]

    Rethinking llm memorization through the lens of adversarial compression

    Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary C Lipton, and J Zico Kolter. Rethinking llm memorization through the lens of adversarial compression. arXiv preprint arXiv:2404.15146,

    work page arXiv

  20. [20]

    doi: 10.18653/v1/2024.acl-long.840

    Association for Computational Linguis- tics. doi: 10.18653/v1/2024.acl-long.840. URL https://aclanthology.org/2024. acl-long.840. 14 Published as a conference paper at ICLR 2025 Yan Tao, Olga Viberg, Ryan S. Baker, and Rene F. Kizilcec. Auditing and mitigating cultural bias in llms,

    work page doi:10.18653/v1/2024.acl-long.840 2024

  21. [21]

    Unlocking memorization in large language models with dy- namic soft prompting

    Zhepeng Wang, Runxue Bao, Yawen Wu, Jackson Taylor, Cao Xiao, Feng Zheng, Weiwen Jiang, Shangqian Gao, and Yanfu Zhang. Unlocking memorization in large language models with dy- namic soft prompting. arXiv preprint arXiv:2409.13853,

    work page arXiv

  22. [22]

    Knowl- edge overshadowing causes amalgamated hallucination in large language models

    Yuji Zhang, Sha Li, Jiateng Liu, Pengfei Yu, Yi R Fung, Jing Li, Manling Li, and Heng Ji. Knowl- edge overshadowing causes amalgamated hallucination in large language models. arXiv preprint arXiv:2407.08039,

    work page arXiv

  23. [23]

    Normbank: A knowl- edge bank of situational social norms

    Caleb Ziems, Jane Dwivedi-Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. Normbank: A knowl- edge bank of situational social norms. arXiv preprint arXiv:2305.17008,

    work page arXiv

  24. [24]

    A 110 C ULTURES Geographic Region Countries and Regions Eastern-European Albania, Armenia, Belarus, Bosnia and Herzegovina, Bulgaria, Croatia, Czechia, Georgia, Greece, Hungary, Kosovo, Moldova, Montenegro, North Macedonia, Poland, Romania, Russia, Serbia, Slovakia, Slovenia, Turkey, Ukraine African-Islamic Algeria, Egypt, Ethiopia, Ghana, Kenya, Libya, M...

    work page 2012

  25. [25]

    Finally, we extract the top five keywords from these topics using TF-IDF

    The LLM generates interpretable n-gram topic phrases, which are then filtered for repetitions using cosine similarity scores calculated with XLM-RoBERTa-large embeddings (Conneau, 2019). Finally, we extract the top five keywords from these topics using TF-IDF. B.2 P ROMPT In figure 8, we provide the prompt used for promptingLLAMA-3.1-8B-Instruct with the ...

    work page 2019

  26. [26]

    We notice suprisingly similiar themes in the pre-training documents such as the discussion around ”religion” in documents where Hijab, Iran and any culture X co-occur

    pertaining to cross-cultural gen- eralization from one culture to another for more cases of cultures which generate these memorized symbols with a lower count of relevant documents than the cultures discussed before. We notice suprisingly similiar themes in the pre-training documents such as the discussion around ”religion” in documents where Hijab, Iran ...

    work page 2025

  27. [27]

    13: start ← end 14: min distance ← ∞ 15: last symbol ← −1 16: last word ← −1 ▷ Compute minimum distance between marked tokens 17: for i from 0 to len(marks) do 18: if marks[i] = 2 then 19: last symbol ← i 20: if last word ̸= −1 then 21: min distance ← min(min distance, i − last word) 22: else if marks[i] = 1 then 23: last word ← i 24: if last symbol ̸= −1...

    work page 2025

  28. [28]

    Indian” and Symbol: “Naan

    the count of documents in which the culture appears in the pretraining corpora: for clothing, we obtain a spearman correlation of 0.569 and a Kendall correlation of 0.445; for food, we obtain a spearman correlation of 0.688 and a Kendall correlation of 0.519. This corre- lation is lower but similar to the original correlations found for z=2.6 (food: Spear...

    work page 2025

  29. [29]

    To investigate this, we conduct a leave-one-culture- out experiment

    However, for clothing, we observe a weak negative correlation (spearman ρ = −0.099, Kendall τ = −0.061). To investigate this, we conduct a leave-one-culture- out experiment. In this analysis, we recalculated the correlations while systematically excluding one culture at a time. We then identify and list the top ten cultures causing the highest variation. ...

    work page 2025

  30. [30]

    In Tables 10 and 11, we present the memorization and generalization statistics for food and clothing, respectively

    H.2 R ESULTS OVERVIEW (a) Topic: Food (b) Topic: Clothing Figure 14: Cross-Culture Generalization Continuing from Section 4.6, in this section we expand upon our findings and present some more results across the 110 cultures. In Tables 10 and 11, we present the memorization and generalization statistics for food and clothing, respectively. Specifically, w...

    work page 2025