Pith. sign in

REVIEW 4 major objections 4 minor 68 references

Rethinking Memorization Measures and their Implications in Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Learning a language optimally forces some memorization of training strings.

desk verdict A genuinely useful new memorization measure and a careful empirical comparison, but the headline 'memorization is unavoidable' claim overreaches the evidence. read the letter →

arxiv 2507.14777 v1 pith:CGLZE3G4 submitted 2025-07-20 cs.LG

Bishwamittra Ghosh , Soumi Das , Qinyuan Wu , Mohammad Aflah Khan , Krishna P. Gummadi , Evimaria Terzi , Deepak Garg This is my paper
classification cs.LG
keywords LLMmemorizationcontextualcounterfactualrecollection-basedlearning-memorizationtradeoffformallanguagesprivacyinlanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether an LLM can learn a language optimally without memorizing any training text, and argues the answer is no: under standard training dynamics, some strings are always memorized by the time test loss reaches its minimum. It introduces a new measure, contextual memorization, which counts a string as memorized only when training makes the model recollect it better than the best possible recollection from context alone. Measured this way, memorization is not the enemy of learning: better learning reduces contextual and counterfactual memorization, even while it raises recollection-based scores. The paper also re-examines strings previously reported as memorized and finds most are predictable from context and carry no sensitive personal information. The upshot is that current recollection-based estimates overstate the privacy threat of LLM memorization.

What carries the argument

The load-bearing object is the optimal contextual recollection of a string: the minimum cross-entropy loss attainable for that string by training on the language with the string withheld, i.e., $\min_{e^*} \operatorname{loss}(M_{e^*}(D'), s)$. This value is the per-string threshold of contextual memorization. The paper's Lemma 1 shows contextual memorization starts no earlier than counterfactual memorization and that contextual scores lower-bound counterfactual scores, because the contextual threshold is the minimum of the counterfactual thresholds over all epochs. The argument then runs on training epochs: different strings begin to be memorized at different epochs, and the globally optimal learning epoch occurs after some of those per-string starts, making zero-memorization optimal learning impossible.

What would settle it

Train a model on a natural corpus and, at each training epoch, compare every training string's loss with its contextual threshold computed by retraining without that string. If there is any language and dataset where the epoch of lowest held-out loss has zero contextually memorized strings, the unavoidability claim falls.

Watch

Extended reading notes

Core claim

The central claim is that memorization is unavoidable for optimal learning and that the dominant way of measuring memorization, recollection against a fixed threshold, confuses contextual prediction with rote storage. The paper formalizes contextual memorization: a training string $s$ is contextually memorized at epoch $e$ if the model's training loss on $s$ is lower than the string's optimal contextual loss, defined as the lowest loss attainable on $s$ by a model trained on the same language without $s$. Because optimal contextual loss is string-specific and learned, it replaces the arbitrary global threshold of recollection-based measures. Across 18 LLMs and 8 formal languages, the paper finds that the epoch of optimal learning is always reached after some strings have crossed this threshold; that frequent and infrequent strings are roughly equally susceptible once context is accounted for; and that increasing training data improves learning while decreasing contextual and counterfactual memorization. Recollection-based measures, by contrast, move in the opposite direction precisely because they depend on a hand-picked threshold.

Load-bearing premise

The impossibility result rests on small synthetic grammars and short fine-tuning runs standing in for natural-language pretraining; if those do not reproduce real training dynamics, the claim of unavoidability may not transfer.

Editorial extensions

If this is right

  • Any optimally trained LLM will contain some contextually memorized strings, so deduplication-based mitigation cannot make memorization zero.
  • Recollection-based rankings of memorized strings, sorted by frequency or threshold, may invert the true memorization order.
  • Improving a model's language ability through more data or better training does not increase contextual memorization; it can reduce it.
  • Published lists of 'memorized' strings should be re-checked with a contextual baseline before treating them as privacy leaks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the contextual threshold could be approximated without retraining by clustering strings of similar frequency and using held-out losses, a direction the paper sketches and which would make the measure applicable to real pretraining corpora.
  • Editorial extension: if memorization is partly a byproduct of optimal learning, then privacy protections such as deduplication should be evaluated by how much sensitive personally identifiable information they remove, not how much predictable text they suppress.
  • Editorial extension: a natural test of the framework is to apply it to a model trained from scratch on a large natural corpus, computing contextual thresholds for a sample of strings via leave-one-out fine-tuning; the paper's synthetic results predict nonzero contextual memorization at the best checkpoint.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper re-examines three memorization measures for LLMs — recollection-based, counterfactual, and a newly proposed contextual memorization — and argues that (a) these measures disagree on when and which training strings are memorized, (b) optimal learning of a language cannot avoid some memorization, and (c) recollection-based estimates of privacy risk are exaggerated. The authors support these arguments with experiments on 18 LLMs from 6 families, fine-tuned on 8 synthetic hierarchical probabilistic context-free languages of varying entropy, plus a reanalysis of previously reported memorized strings from Pythia-1B-deduped using OLMo-1B as a reference model. A formal lemma states that contextual memorization is stricter than counterfactual memorization. The paper's central conclusions are framed as general statements about learning a language and about today's memorization estimates.

Significance. If the central claims hold, the paper would make a valuable conceptual contribution: it would show that the choice of memorization measure materially changes conclusions, that a certain amount of memorization is intrinsic to optimal learning in current training dynamics, and that recollection-based danger signals overstate privacy threats. The paper is also notable for its controlled formal-language setup, the breadth of LLMs tested, and the explicit formulation of a contextual memorization criterion that connects memorization to local over-fitting. The formal relation in Lemma 1, if its assumption is verified, is a useful clarification of how counterfactual and contextual measures relate. The reanalysis of previously reported memorized strings, showing that many are predictable from context, is a constructive contribution to the privacy-memorization debate. However, the headline impossibility claim is stated far more universally than the experiments support, and the proof of the key lemma rests on an unverified assumption.

major comments (4)
  1. [Section 4, Answer to RQ3, Figure 3, Appendix C] The claim that 'Memorization is unavoidable for optimal learning, both in high and low entropy languages' is stated as a universal impossibility, but the evidence is a finite set of runs: 50 epochs, one learning-rate schedule, no regularization or early-stopping search, dataset sizes 16 to 1024, and eight small hierarchical PCFGs. The paper never searches for a memorization-free optimal configuration, for example with weight decay, different optimizers, larger datasets, or different initializations. Figure 5, in fact, shows weighted contextual memorization decreasing monotonically with dataset size, so the data are consistent with memorization vanishing in the large-data limit; no asymptotic argument rules that out. The conclusion should be weakened to 'in the training configurations we tested' or supported by a theoretical impossibility result that does not depend on the particular training setup.
  2. [Appendix B, Lemma 1] The proof of Lemma 1 relies on the assumption that at any epoch the training loss of a string is no higher than the counterfactual test loss of the same string when the string is excluded from training. This inequality is asserted as 'a feasible assumption in practice' but is not verified empirically on any of the 18 models or 8 languages used in the paper. The assumption is load-bearing: it drives both the conclusion that contextual memorization starts no earlier than counterfactual memorization and the bound on scores. In real training, the training loss can exceed the held-out loss on a specific string, particularly at early epochs or after learning-rate warm-up, and this would break the inequality chain. The authors should either verify the assumption across their runs or state Lemma 1 as conditional on this property, with a discussion of when it may fail.
  3. [Section 2 and Appendix C] The generalization from fine-tuning pretrained LLMs on eight synthetic formal languages to natural language learning is unvalidated. The paper asserts that the controlled setup ensures 'learning and memorization are unaffected by prior training,' but the experiments begin from pretrained checkpoints, so the models already have substantial knowledge of the token stream and training dynamics from their original pretraining corpora. It is not shown that fine-tuning on small hierarchical PCFGs reproduces the learning-versus-memorization dynamics of large-scale natural language pretraining. The abstract and conclusions speak of 'learning a language' without qualification; the scope should be restricted to the tested formal-language fine-tuning regime, or an additional experiment closer to naturalistic distributions should be included.
  4. [Section 3.1 and Section 4] The impossibility result is operationalized through the paper's own contextual memorization measure, which declares a string memorized as soon as its training loss drops below the minimum counterfactual test loss. The universal claim is therefore partly a property of this operationalization rather than an independent fact about learning. The paper does report nonzero counterfactual memorization as well, but the recollection-based measure is threshold-dependent and shows the opposite pattern in some conditions. To support the strong statement, the authors should analyze sensitivity: how the 'unavoidable' conclusion changes with the definition of the contextual threshold, with alternative thresholds for recollection, or when the three measures disagree at the optimal epoch. Without such analysis, the headline claim risks overstating what the new measure alone establishes.
minor comments (4)
  1. [Section 3.2] In the sentence describing Figure 2a, the epochs are given as '(erec s1 = 10, erec s1 = 12)'; the second should presumably refer to s2, not s1.
  2. [Section 5 and Table 1] The phrase 'upper bound (UB) of the optimal contextual accuracy' is confusing: a reference model's higher accuracy is evidence that a string is easy to recollect from context, not an upper bound on the optimal contextual accuracy in the usual sense. The terminology and the inference from OLMo-1B's accuracy to 'unlikely to be contextually memorized' should be clarified.
  3. [Appendix C] The learning-rate specification is inconsistent: 'Mistral, Opt, and Llama-2 families as 5 × 10−6, and Pythia family as 10−5' should state '1 × 10−5' for clarity.
  4. [Figure 22 caption] The label 'Llama-3.1-∗B' is ambiguous; the exact model size should be given, as is done for other families.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the impossibility claim is an empirical generalization, not an identity or a self-citation reduction.

full rationale

I walked the paper's derivation chain and found no step in which a claimed prediction or first-principles result is equivalent to its inputs by construction. The central claim, "Memorization is unavoidable for optimal learning" (Section 4, Answer to RQ3), is supported by measuring memctx and memcf at the epoch of lowest held-out test loss. The thresholds in Equations (1) and (3) come from separately retrained models on datasets excluding the target string; they are not fitted to the conclusion. The 'optimal contextual loss' min_e* loss(Me*(D'), s) and the global optimal-learning epoch argmin_e test loss are distinct quantities, so the positivity of memctx at e* is an empirical observation, not a definitional identity. Lemma 1 is a genuine theorem proved from the definitions and an explicit assumption, so it is not circular. The paper does cite two works with overlapping authorship ([52] and [63]), but neither is load-bearing: [63] appears only in a bracketed related-work citation about context-based recollection, and [52] supports a side comment about PII predictability in Section 5. The self-contained formal-language experiments, 18 LLMs, and external counterfactual baseline [14] give the central claim independent content. The 'unavoidable' phrasing is stronger than the finite set of runs (8 small PCFGs, 50 epochs, one LR schedule, dataset sizes 16-1024) and Figure 5 even shows contextual memorization decreasing with dataset size, so the universality conclusion may be under-supported; however, that is an evidence-strength and generalization concern, not circularity by construction.

Assumptions & free parameters 1 free parameters · 4 assumptions · 1 invented entities

The central claims rest on a hand-chosen recollection threshold, an unverified monotonicity assumption in the theoretical lemma, a transfer assumption from synthetic formal languages to natural language, and a proxy reference model for contextual recollection. The only invented construct is the contextual memorization measure itself, which has no falsifiable handle outside the paper.

free parameters (1)
  • Recollection threshold tau = 0.2
    Hand-chosen cross-entropy loss threshold used to define recollection-based memorization throughout the experiments. The paper acknowledges the threshold is arbitrary, but all recollection-based conclusions depend on this value.
assumptions (4)
  • domain assumption At any training epoch, the training loss of a string is not higher than the counterfactual test loss of the string when it is excluded from training.
    Stated in the proof of Lemma 1 (Appendix B) as 'a feasible assumption in practice', but no empirical verification is provided. It is required for the claim that contextual memorization starts no earlier than counterfactual memorization.
  • domain assumption Synthetic hierarchical probabilistic context-free grammars with tokens and controlled probabilities capture the recursive structure and learning dynamics of natural language.
    Section 2 and Appendix C motivate the controlled setup; the paper's general conclusion that optimal learning cannot avoid memorization is transferred from these languages to LLMs.
  • domain assumption Fine-tuning a pretrained LLM on formal-language strings measures memorization that is unaffected by the model's prior pretraining.
    Section 2 states learning and memorization are 'unaffected by prior training' despite using pretrained open-source LLMs; this is questionable because pretrained weights strongly influence fine-tuning dynamics.
  • domain assumption The reference model OLMo-1B, trained on Dolma, provides an upper bound on optimal contextual recollection for Pythia-1B-deduped memorized strings.
    Section 5 relies on this proxy because the target model without a string is unavailable; the paper acknowledges that ensuring disjoint training data is challenging.
invented entities (1)
  • Contextual memorization and optimal contextual recollection
    purpose: A new operational definition of memorization intended to separate memorization from contextual learning; the optimal contextual loss is the lowest test loss of a model trained without the target string.
    The construct is defined operationally in Section 3.1(c) and used to re-interpret existing memorization reports, but it makes no falsifiable prediction outside the paper; its validity rests on the formal-language experiments and the Lemma's assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Memorization Measures and their Implications in Large Language Models." pith.science (2026). https://pith.science/paper/CGLZE3G4

@misc{pith2026250714777,
  author       = {Pith},
  title        = {Pith review of: Rethinking Memorization Measures and their Implications in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CGLZE3G4}},
  note         = {Machine review of arXiv:2507.14777}
}
read the original abstract

Concerned with privacy threats, memorization in LLMs is often seen as undesirable, specifically for learning. In this paper, we study whether memorization can be avoided when optimally learning a language, and whether the privacy threat posed by memorization is exaggerated or not. To this end, we re-examine existing privacy-focused measures of memorization, namely recollection-based and counterfactual memorization, along with a newly proposed contextual memorization. Relating memorization to local over-fitting during learning, contextual memorization aims to disentangle memorization from the contextual learning ability of LLMs. Informally, a string is contextually memorized if its recollection due to training exceeds the optimal contextual recollection, a learned threshold denoting the best contextual learning without training. Conceptually, contextual recollection avoids the fallacy of recollection-based memorization, where any form of high recollection is a sign of memorization. Theoretically, contextual memorization relates to counterfactual memorization, but imposes stronger conditions. Memorization measures differ in outcomes and information requirements. Experimenting on 18 LLMs from 6 families and multiple formal languages of different entropy, we show that (a) memorization measures disagree on memorization order of varying frequent strings, (b) optimal learning of a language cannot avoid partial memorization of training strings, and (c) improved learning decreases contextual and counterfactual memorization but increases recollection-based memorization. Finally, (d) we revisit existing reports of memorized strings by recollection that neither pose a privacy threat nor are contextually or counterfactually memorized.

Figures

Figures reproduced from arXiv: 2507.14777 by the authors.

Figure 2
Figure 2. Start of memorization (vertical dotted line) of three strings [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Memorization of training strings in languages [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Contradiction among memorization measures on determining memorization of top [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (33 more)
Figure 5
Figure 5. Figure 5: Memorization vs. optimal learning (i.e., the [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Production rules of G1 (left) and G2 (right). Compared to G1, the grammar G2 generates more skewed distribution (or lower entropy) strings, since one out of two production rules for each non-terminal is selected with higher probability. Formal languages are divided int…
Figure 7
Figure 7. Figure 7: Production rules of G3 (left) and G4 (right). Compared to G3, the grammar G4 generates more skewed distribution (or lower entropy) of strings, since one out of three production rules of non-terminal B1 and E1 is selected with higher probability. 19 [PITH_FULL_IMAGE:fi…
Figure 8
Figure 8. Figure 8: Production rules of G5 (left) and G6 (right). These grammars are adapted from G1 and G3 respectively, by allowing non-uniform lengths of tokens in the lowest level production rules. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Production rules of G7 (left) and G8 (right). These grammars are adapted from G5 and G6 respectively, by replacing numerical tokens with Latin character tokens. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Length distribution of considered probabilistic languages, based on [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Representative strings from different languages, annotated with non-terminals applied in [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Start of memorization of selected strings in Language [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Memorization score of strings in language [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: Start of memorization of selected strings in language [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]
Figure 15
Figure 15. Figure 15: Memorization score of strings in language [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]
Figure 16
Figure 16. Figure 16: Contextual memorization is a stricter measure than counterfactual memorization. [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]
Figure 17
Figure 17. Figure 17: Memorization of training strings in languages of different entropy across different [PITH_FULL_IMAGE:figures/full_fig_p026_17.png]
Figure 18
Figure 18. Figure 18: Memorization of training strings in languages of different entropy across different [PITH_FULL_IMAGE:figures/full_fig_p026_18.png]
Figure 19
Figure 19. Figure 19: Continuing Figure 3, we demonstrate associated loss with weighted memorization. [PITH_FULL_IMAGE:figures/full_fig_p027_19.png]
Figure 20
Figure 20. Figure 20: Continuing Figure 4, contradiction between recollection-based and contextual (or counter [PITH_FULL_IMAGE:figures/full_fig_p027_20.png]
Figure 21
Figure 21. Figure 21: Continuing Figure 20, contradiction between recollection-based and contextual (or [PITH_FULL_IMAGE:figures/full_fig_p028_21.png]
Figure 22
Figure 22. Figure 22: Trade-offs between optimal learning and memorization among comparable [PITH_FULL_IMAGE:figures/full_fig_p029_22.png]
Figure 23
Figure 23. Figure 23: Trade-offs between optimal learning and memorization among comparable [PITH_FULL_IMAGE:figures/full_fig_p030_23.png]
Figure 24
Figure 24. Figure 24: Trade-offs between optimal learning and memorization among comparable [PITH_FULL_IMAGE:figures/full_fig_p031_24.png]
Figure 25
Figure 25. Figure 25: Trade-offs between optimal learning and memorization among comparable [PITH_FULL_IMAGE:figures/full_fig_p032_25.png]
Figure 26
Figure 26. Figure 26: Trade-offs between optimal learning and memorization among comparable [PITH_FULL_IMAGE:figures/full_fig_p033_26.png]
Figure 27
Figure 27. Figure 27: Trade-offs between optimal learning and memorization among comparable [PITH_FULL_IMAGE:figures/full_fig_p034_27.png]
Figure 28
Figure 28. Figure 28: Trade-offs between optimal learning and memorization among comparable [PITH_FULL_IMAGE:figures/full_fig_p035_28.png]
Figure 29
Figure 29. Figure 29: Trade-offs between optimal learning and memorization among comparable [PITH_FULL_IMAGE:figures/full_fig_p036_29.png]
Figure 30
Figure 30. Figure 30: Contextual memorization vs. optimal language learning, measured as test loss, across [PITH_FULL_IMAGE:figures/full_fig_p037_30.png]
Figure 31
Figure 31. Figure 31: Contextual memorization vs. optimal language learning, measured as test loss, across [PITH_FULL_IMAGE:figures/full_fig_p038_31.png]
Figure 32
Figure 32. Figure 32: Contextual memorization vs. optimal language learning, measured as test loss, across [PITH_FULL_IMAGE:figures/full_fig_p039_32.png]
Figure 33
Figure 33. Figure 33: Contextual memorization vs. optimal language learning, measured as test loss, across [PITH_FULL_IMAGE:figures/full_fig_p040_33.png]
Figure 34
Figure 34. Figure 34: Contextual memorization vs. optimal language learning, measured as test loss, across [PITH_FULL_IMAGE:figures/full_fig_p041_34.png]
Figure 35
Figure 35. Figure 35: Contextual memorization vs. optimal language learning, measured as test loss, across [PITH_FULL_IMAGE:figures/full_fig_p042_35.png]
Figure 36
Figure 36. Figure 36: Contextual memorization vs. optimal language learning, measured as test loss, across [PITH_FULL_IMAGE:figures/full_fig_p043_36.png]
Figure 37
Figure 37. Figure 37: Contextual memorization vs. optimal language learning, measured as test loss, across [PITH_FULL_IMAGE:figures/full_fig_p044_37.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 32 canonical work pages

  1. [1]

    On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623, 2021

    Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623, 2021

    2021

  2. [2]

    Sparks of artificial general intelligence: Early experiments with gpt-4

    Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023

    arXiv 2023

  3. [3]

    Rethinking llm memorization through the lens of adversarial compression.arXiv preprint arXiv:2404.15146, 2024

    Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary C Lipton, and J Zico Kolter. Rethinking llm memorization through the lens of adversarial compression.arXiv preprint arXiv:2404.15146, 2024

    arXiv 2024

  4. [4]

    Emergent and predictable memorization in large language models

    Stella Biderman, Usvsn Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. Emergent and predictable memorization in large language models. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  5. [5]

    Deduplicating training data mitigates privacy risks in language models

    Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pages 10697– 10707. PMLR, 2022

    work page 2022

  6. [6]

    Extracting training data from large language models

    Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-V oss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650, 2021

    2021

  7. [7]

    The secret sharer: Evaluating and testing unintended memorization in neural networks

    Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267–284, 2019

    work page 2019

  8. [8]

    Memorization without overfitting: Analyzing the training dynamics of large language models

    Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274–38290, 2022

    work page 2022

Show all 68 references
  1. [9]

    An empirical analysis of memorization in fine-tuned autoregressive language models

    Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David K Evans, and Taylor Berg- Kirkpatrick. An empirical analysis of memorization in fine-tuned autoregressive language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, ...

    2022

  2. [10]

    Preventing verbatim memorization in language models gives a false sense of privacy

    Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo, and Nicholas Carlini. Preventing verbatim memorization in language models gives a false sense of privacy. arXiv preprint arXiv:2210.17546, 2022

    arXiv

  3. [11]

    Near-duplicate sequence search at scale for large language model memorization evaluation

    Zhencan Peng, Zhizhi Wang, and Dong Deng. Near-duplicate sequence search at scale for large language model memorization evaluation. Proceedings of the ACM on Management of Data, 1 (2):1–18, 2023

    2023

  4. [12]

    Uncovering latent memories: Assessing data leakage and memorization patterns in large language models

    Sunny Duan, Mikail Khona, Abhiram Iyer, Rylan Schaeffer, and Ila R Fiete. Uncovering latent memories: Assessing data leakage and memorization patterns in large language models. In ICML 2024 Workshop on LLMs and Cognition, 2024

    2024

  5. [13]

    Quantifying and analyzing entity-level memorization in large language models

    Zhenhong Zhou, Jiuyang Xiang, Chaomeng Chen, and Sen Su. Quantifying and analyzing entity-level memorization in large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19741–19749, 2024

    2024

  6. [14]

    Counterfactual memorization in neural language models

    Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. ArXiv, abs/2112.12938, 2021. URL https://api.semanticscholar.org/CorpusID: 245502053

    2021 arXiv

  7. [15]

    On memorization in probabilistic deep generative models

    Gerrit van den Burg and Chris Williams. On memorization in probabilistic deep generative models. Advances in Neural Information Processing Systems, 34:27916–27928, 2021. 11

    2021

  8. [16]

    Deduplicating training data makes language models better

    Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021

    2021 arXiv

  9. [17]

    ‘improving ratings’: audit in the british university system

    Marilyn Strathern. ‘improving ratings’: audit in the british university system. Euro- pean Review , 5(3):305–321, July 1997. doi: 10.1002/(sici)1234-981x(199707)5:3<305:: aid-euro184>3.0.co;2-4. URL https://doi.org/10.1002/(sici)1234-981x(199707)5: 3<305::aid-euro184>3.0.co;2-4

    1997 doi

  10. [18]

    Quantifying memorization across neural language models

    Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022

    2022 arXiv

  11. [19]

    What languages are easy to language-model? a perspective from learning probabilistic regular languages

    Nadav Borenstein, Anej Svete, Robin Chan, Josef Valvoda, Franz Nowak, Isabelle Augen- stein, Eleanor Chodroff, and Ryan Cotterell. What languages are easy to language-model? a perspective from learning probabilistic regular languages. arXiv preprint arXiv:2406.04289, 2024

    2024 arXiv

  12. [20]

    In-context language learning: Arhitectures and algorithms

    Ekin Akyürek, Bailin Wang, Yoon Kim, and Jacob Andreas. In-context language learning: Arhitectures and algorithms. arXiv preprint arXiv:2401.12973, 2024

    2024 arXiv

  13. [21]

    Transparency at the source: Evaluating and interpreting language models with access to the true distribution

    Jaap Jumelet and Willem Zuidema. Transparency at the source: Evaluating and interpreting language models with access to the true distribution. arXiv preprint arXiv:2310.14840, 2023

    2023 arXiv

  14. [22]

    Injecting structural hints: Using language models to study inductive biases in language learning

    Isabel Papadimitriou and Dan Jurafsky. Injecting structural hints: Using language models to study inductive biases in language learning. arXiv preprint arXiv:2304.13060, 2023

    2023 arXiv

  15. [23]

    Examining the inductive bias of neural language models with artificial languages

    Jennifer C White and Ryan Cotterell. Examining the inductive bias of neural language models with artificial languages. arXiv preprint arXiv:2106.01044, 2021

    2021 arXiv

  16. [24]

    Towards more natural artificial languages

    Mark Hopkins. Towards more natural artificial languages. InProceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 85–94, 2022

    2022

  17. [25]

    Physics of language models: Part 1, learning hierarchical language structures

    Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 1, learning hierarchical language structures. ArXiv e-prints, abs/2305.13673, May, 2023

    2023 arXiv

  18. [26]

    Transformer working memory enables regular language reasoning and natural language length extrapolation

    Ta-Chung Chi, Ting-Han Fan, Alexander I Rudnicky, and Peter J Ramadge. Transformer working memory enables regular language reasoning and natural language length extrapolation. arXiv preprint arXiv:2305.03796, 2023

    2023 arXiv

  19. [27]

    Char- acterizing intrinsic compositionality in transformers with tree projections

    Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D Manning. Char- acterizing intrinsic compositionality in transformers with tree projections. arXiv preprint arXiv:2211.01288, 2022

    2022 arXiv

  20. [28]

    Transformers learn shortcuts to automata

    Bingbin Liu, Jordan T Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers learn shortcuts to automata. arXiv preprint arXiv:2210.10749, 2022

    2022 arXiv

  21. [29]

    Learning bounded context- free-grammar via lstm and the transformer: difference and the explanations

    Hui Shi, Sicun Gao, Yuandong Tian, Xinyun Chen, and Jishen Zhao. Learning bounded context- free-grammar via lstm and the transformer: difference and the explanations. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 8267–8276, 2022

    2022

  22. [30]

    On the ability and limitations of trans- formers to recognize formal languages

    Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal. On the ability and limitations of trans- formers to recognize formal languages. arXiv preprint arXiv:2009.11264, 2020

    2009 arXiv

  23. [31]

    Formal languages and the nlp black box

    William Merrill. Formal languages and the nlp black box. In International Conference on Developments in Language Theory, pages 1–8. Springer, 2023

    2023

  24. [32]

    Transformers as recognizers of formal languages: A survey on expressivity

    Lena Strobl, William Merrill, Gail Weiss, David Chiang, and Dana Angluin. Transformers as recognizers of formal languages: A survey on expressivity. arXiv preprint arXiv:2311.00208, 2023

    2023 arXiv

  25. [33]

    Theoretical limitations of self-attention in neural sequence models

    Michael Hahn. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics, 8:156–171, 2020. 12

    2020

  26. [34]

    Neural networks and the chomsky hierarchy

    Grégoire Delétang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, et al. Neural networks and the chomsky hierarchy. arXiv preprint arXiv:2207.02098, 2022

    2022 arXiv

  27. [35]

    Why are sensitive functions hard for transformers? arXiv preprint arXiv:2402.09963, 2024

    Michael Hahn and Mark Rofin. Why are sensitive functions hard for transformers? arXiv preprint arXiv:2402.09963, 2024

    2024 arXiv

  28. [36]

    Are all languages equally hard to language-model? arXiv preprint arXiv:1806.03743, 2018

    Ryan Cotterell, Sabrina J Mielke, Jason Eisner, and Brian Roark. Are all languages equally hard to language-model? arXiv preprint arXiv:1806.03743, 2018

    2018 arXiv

  29. [37]

    What kind of language is hard to language-model? arXiv preprint arXiv:1906.04726, 2019

    Sabrina J Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. What kind of language is hard to language-model? arXiv preprint arXiv:1906.04726, 2019

    1906 arXiv

  30. [38]

    Elements of information theory

    Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999

    1999

  31. [39]

    Carrasco

    Rafael C. Carrasco. Accurate computation of the relative entropy between stochastic regular grammars. RAIRO-Theoretical Informatics and Applications, 31(5):437–444, 1997

    1997

  32. [40]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  33. [41]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  34. [42]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  35. [43]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex...

    2024 arXiv

  36. [44]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In Intern...

    2023

  37. [45]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. 13

    2022 arXiv

  38. [46]

    Cross-entropy loss functions: Theoretical analysis and applications

    Anqi Mao, Mehryar Mohri, and Yutao Zhong. Cross-entropy loss functions: Theoretical analysis and applications. In International conference on Machine learning, pages 23803–23828. PMLR, 2023

    2023

  39. [47]

    Are large pre-trained language models leaking your personal information? arXiv preprint arXiv:2205.12628, 2022

    Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. Are large pre-trained language models leaking your personal information? arXiv preprint arXiv:2205.12628, 2022

    2022 arXiv

  40. [48]

    Propile: Probing privacy leakage in large language models

    Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, and Seong Joon Oh. Propile: Probing privacy leakage in large language models. Advances in Neural Information Processing Systems, 36:20750–20762, 2023

    2023

  41. [49]

    Measuring forgetting of memorized training examples

    Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, et al. Measuring forgetting of memorized training examples. arXiv preprint arXiv:2207.00099, 2022

    2022 arXiv

  42. [50]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  43. [51]

    Olmo: Accelerating the science of language models

    Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838, 2024

    2024 arXiv

  44. [52]

    Revisiting privacy, utility, and efficiency trade-offs when fine-tuning large language models

    Soumi Das, Camila Kolling, Mohammad Aflah Khan, Mahsa Amani, Bishwamittra Ghosh, Qinyuan Wu, Till Speicher, and Krishna P Gummadi. Revisiting privacy, utility, and efficiency trade-offs when fine-tuning large language models. arXiv preprint arXiv:2502.13313, 2025

    2025

  45. [53]

    Foundation models and fair use

    Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A Lemley, and Percy Liang. Foundation models and fair use. Journal of Machine Learning Research, 24(400):1–79, 2023

    2023

  46. [54]

    Llms and memorization: On quality and specificity of copyright compliance

    Felix B Mueller, Rebekka Görge, Anna K Bernzen, Janna C Pirk, and Maximilian Poretschkin. Llms and memorization: On quality and specificity of copyright compliance. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, pages 984–996, 2024

    2024

  47. [55]

    Exploring memorization and copyright violation in frontier llms: A study of the new york times v

    Joshua Freeman, Chloe Rippe, Edoardo Debenedetti, and Maksym Andriushchenko. Exploring memorization and copyright violation in frontier llms: A study of the new york times v. openai 2023 lawsuit. arXiv preprint arXiv:2412.06370, 2024

    2023 arXiv

  48. [56]

    Undesirable memorization in large language models: A survey

    Ali Satvaty, Suzan Verberne, and Fatih Turkmen. Undesirable memorization in large language models: A survey. arXiv preprint arXiv:2410.02650, 2024

    2024

  49. [57]

    Measuring memorization in rlhf for code completion

    Aneesh Pappu, Billy Porter, Ilia Shumailov, and Jamie Hayes. Measuring memorization in rlhf for code completion. arXiv preprint arXiv:2406.11715, 2024

    2024 arXiv

  50. [58]

    What neural networks memorize and why: Discovering the long tail via influence estimation

    Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33: 2881–2891, 2020

    2020

  51. [59]

    Under- standing transformer memorization recall through idioms

    Adi Haviv, Ido Cohen, Jacob Gidron, Roei Schuster, Yoav Goldberg, and Mor Geva. Under- standing transformer memorization recall through idioms. arXiv preprint arXiv:2210.03588, 2022

    2022 arXiv

  52. [60]

    Generalization vs memorization: Tracing language models’ capabilities back to pretraining data

    Xinyi Wang, Antonis Antoniades, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, and William Yang Wang. Generalization vs memorization: Tracing language models’ capabilities back to pretraining data. arXiv preprint arXiv:2407.14985, 2024

    2024 arXiv

  53. [61]

    Think or remember? detecting and directing llms towards memorization or generalization

    Yi-Fu Fu, Yu-Chieh Tu, Tzu-Ling Cheng, Cheng-Yu Lin, Yi-Ting Yang, Heng-Yi Liu, Keng-Te Liao, Da-Cheng Juan, and Shou-De Lin. Think or remember? detecting and directing llms towards memorization or generalization. arXiv preprint arXiv:2412.18497, 2024

    2024 arXiv

  54. [62]

    Mem- orize or generalize? evaluating llm code generation with evolved questions

    Wentao Chen, Lizhe Zhang, Li Zhong, Letian Peng, Zilong Wang, and Jingbo Shang. Mem- orize or generalize? evaluating llm code generation with evolved questions. arXiv preprint arXiv:2503.02296, 2025. 14

    2025

  55. [63]

    Rethinking memorization in llms: On learning by rote vs

    Till Speicher, Bishwamittra Ghosh, Mohammad Aflah Khan, Qinyuan Wu, Vedant Nanda, Soumi Das, Krishna P Gummadi, and Evimaria Terzi. Rethinking memorization in llms: On learning by rote vs. with understanding

  56. [64]

    Generalization or memorization: Data contamination and trustworthy evaluation for large language models

    Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. arXiv preprint arXiv:2402.15938, 2024

    2024 arXiv

  57. [65]

    How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven

    R Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven. Transactions of the Association for Computational Linguistics , 11: 652–670, 2023

    2023

  58. [66]

    Probabilistic context-free grammars (pcfgs)

    Michael Collins. Probabilistic context-free grammars (pcfgs). Lecture Notes, 2013

    2013

  59. [67]

    Three models for the description of language.IRE Transactions on information theory, 2(3):113–124, 1956

    Noam Chomsky. Three models for the description of language.IRE Transactions on information theory, 2(3):113–124, 1956. 15 Table 2: Characteristics of memorization measures. Memorization Measure Motivation Memorization Threshold Ease of Operationalization Strictness of Measure ...

    1956

  60. [68]

    start" or

    consider training accuracy as the proxy of memorization: given a training string as a prompt, an LLM memorizes it if it recollects the next token in the string correctly. Carlini et al.[18] propose a relatively stringent measure by imposing an exact recollection of the next 50...

    2014

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.