Pith. sign in

REVIEW 3 cited by

Demystifying Verbatim Memorization in Large Language Models

Not yet reviewed by Pith; the record is open.

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.17817 v1 pith:LNBVQ2IF submitted 2024-07-25 cs.CL cs.LG

classification cs.CLcs.LG
keywords verbatimmemorizationsequenceslanguagemodelcapabilitiescheckpointsdegrading
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) frequently memorize long sequences verbatim, often with serious legal and privacy implications. Much prior work has studied such verbatim memorization using observational data. To complement such work, we develop a framework to study verbatim memorization in a controlled setting by continuing pre-training from Pythia checkpoints with injected sequences. We find that (1) non-trivial amounts of repetition are necessary for verbatim memorization to happen; (2) later (and presumably better) checkpoints are more likely to verbatim memorize sequences, even for out-of-distribution sequences; (3) the generation of memorized sequences is triggered by distributed model states that encode high-level features and makes important use of general language modeling capabilities. Guided by these insights, we develop stress tests to evaluate unlearning methods and find they often fail to remove the verbatim memorized information, while also degrading the LM. Overall, these findings challenge the hypothesis that verbatim memorization stems from specific model weights or mechanisms. Rather, verbatim memorization is intertwined with the LM's general capabilities and thus will be very difficult to isolate and suppress without degrading model quality.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Closer Look on Memorization in Tabular Diffusion Model: A Data-Centric Perspective

    cs.LG 2025-05 reject novelty 6.0 of 10

    A small subset of training samples drives most memorization in tabular diffusion models, and pruning them based on early memorization signals reduces measured leakage, though the evaluation metric makes part of the ga...

  2. FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    This paper releases seven open-source MoE language models with full training artifacts and reports up to 3.4 point accuracy gains over dense baselines at equal FLOPs.

  3. SoK: Semantic Privacy in Large Language Models

    cs.CR 2025-06 conditional novelty 4.0 of 10

    A systematization of knowledge arguing that LLM privacy threats extend beyond data leakage to semantically inferred attributes, and that current defenses only partially address them.

Pith tools