Pith. sign in

REVIEW 21 cited by

Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2210.17546 v3 pith:CPGL7K7W submitted 2022-10-31 cs.LG cs.CL

Daphne Ippolito , Florian Tramèr , Milad Nasr , Chiyuan Zhang , Matthew Jagielski , Katherine Lee , Christopher A. Choquette-Choo , Nicholas Carlini This is my paper · ORCID
classification cs.LGcs.CL
keywords memorizationmodelsverbatimdatalanguagetrainingdefinitionsneural
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Studying data memorization in neural language models helps us understand the risks (e.g., to privacy or copyright) associated with models regurgitating training data and aids in the development of countermeasures. Many prior works -- and some recently deployed defenses -- focus on "verbatim memorization", defined as a model generation that exactly matches a substring from the training set. We argue that verbatim memorization definitions are too restrictive and fail to capture more subtle forms of memorization. Specifically, we design and implement an efficient defense that perfectly prevents all verbatim memorization. And yet, we demonstrate that this "perfect" filter does not prevent the leakage of training data. Indeed, it is easily circumvented by plausible and minimally modified "style-transfer" prompts -- and in some cases even the non-modified original prompts -- to extract memorized information. We conclude by discussing potential alternative definitions and why defining memorization is a difficult yet crucial open question for neural language models.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 21 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Space Complexity of Learning-Unlearning Algorithms

    cs.LG 2025-06 accept novelty 8.0 of 10

    The space complexity of machine unlearning for realizability testing is characterized by eluder dimension (central lower bound), star number (ticketed upper bound), and hollow star number (bounded deletions), separati...

  2. Hush! Protecting Secrets During Model Training: An Indistinguishability Approach

    cs.CR 2025-05 conditional novelty 7.0 of 10

    A new per-secret protection definition and an LP-based sampling algorithm that trains models with substantially lower noise than DP-SGD while bounding the posterior probability of secret reconstruction.

  3. Scaling Laws for Differentially Private Language Models

    cs.LG 2025-01 conditional novelty 7.0 of 10

    Differentially private language models obey scaling laws in which compute-optimal models are roughly 10-50x smaller than non-private Chinchilla-optimal models, with large batch sizes and rapid saturation of compute.

  4. Rethinking Memorization Measures and their Implications in Large Language Models

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Contextual memorization, defined by comparing a string's training loss against the best loss without training on that string, is stricter than counterfactual memorization and suggests that zero-memorization optimal le...

  5. What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requests

    cs.CL 2025-07 conditional novelty 6.0 of 10

    WikiMem, a Wikidata-derived canary dataset and a calibrated NLL-ranking metric, identifies which human-fact associations an LLM has memorized, with higher rates for famous people and larger models.

  6. Winter Soldier: Backdooring Language Models at Pre-Training with Indirect Data Poisoning

    cs.CR 2025-06 conditional novelty 6.0 of 10

    Indirect data poisoning (gradient-matching prompts) makes LLMs learn secret prompt-response pairs absent from training data, detectable with certified p-values and under 0.005% contaminated tokens.

  7. Nine Ways to Break Copyright Law and Why Our LLM Won't: A Fair Use Aligned Generation Framework

    cs.CL 2025-05 reject novelty 6.0 of 10

    FUA-LLM uses DPO on expert-labeled fair use preference data to make LLMs generate compliant, helpful responses to copyright-sensitive prompts.

  8. Fragments to Facts: Partial-Information Fragment Inference from LLMs

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Fine-tuned LLMs leak private fragment-level information to adversaries holding only a few unordered public fragments, as shown by two probe attacks (LR-Attack and PRISM) on medical and legal summarization tasks.

  9. PANORAMA: A synthetic PII-laced dataset for studying sensitive data memorization in LLMs

    cs.CL 2025-05 conditional novelty 6.0 of 10

    The authors release PANORAMA, a 384,789-sample synthetic corpus from 9,674 profiles, and show that repetition during fine-tuning raises PII memorization rates in Mistral-7B from 8.8 to 51.2 percent soft match.

  10. Certified Mitigation of Worst-Case LLM Copyright Infringement

    cs.CL 2025-04 conditional novelty 6.0 of 10

    BloomScrub detects long verbatim quotes from a protected corpus with a Bloom filter, rewrites them iteratively, and abstains when needed, certifying that no quote longer than the threshold is emitted.

  11. A Lightweight Method to Disrupt Memorized Sequences in LLM

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A decoding-time intervention that substitutes a small model's probabilities for common function words into a large model's output reduces exact training-data recall by up to 10x with minimal measured quality loss.

  12. Agents Are All You Need for LLM Unlearning

    cs.AI 2025-02 reject novelty 6.0 of 10

    A four-agent pipeline, Vanilla, AuditErase, Critic, and Composer, filters target references out of LLM responses, claiming robust and scalable inference-time unlearning without weight updates.

  13. Investigating the Feasibility of Mitigating Potential Copyright Infringement via Large Language Model Unlearning

    cs.CL 2024-12 conditional novelty 6.0 of 10

    SSU combines task-vector negation, random-label loss, and weight saliency to forget copyrighted books sequentially while retaining more general language ability than existing baselines.

  14. Information Theory of Meaningful Communication

    cs.CL 2024-11 conditional novelty 6.0 of 10

    LLMs estimate about 20 bits of semantic information per clause in 20 spoken narratives, roughly half the total information per clause.

  15. On the Privacy Risk of In-context Learning

    cs.LG 2024-11 conditional novelty 6.0 of 10

    A confidence-based membership inference attack identifies prompt demonstration data with AUC 0.69-0.86, more than fine-tuned models leak at matched utility, and ensembling reduces this to near random.

  16. FlashDP: Private Training Large Language Models with Efficient DP-SGD

    cs.LG 2025-07 conditional novelty 5.0 of 10

    FlashDP fuses per-sample gradient computation, norm calculation, clipping, and noise addition into a cache-friendly block-wise all-reduce workflow that avoids explicit per-sample gradient storage and redundant recomputation.

  17. A Shared Standard for Valid Measurement of Generative AI Systems' Capabilities, Risks, and Impacts

    cs.CY 2024-12 conditional novelty 5.0 of 10

    A framework that systematizes, operationalizes, and applies amounts, concepts, instances, and populations for valid GenAI measurement, extending Adcock and Collier's measurement theory.

  18. DevLicOps: A Framework for Mitigating Licensing Risks in AI-Generated Code

    cs.SE 2025-08 conditional novelty 4.0 of 10

    DevLicOps integrates license-compliance controls into the SDLC to reduce risk from AI-generated code, using policies, automated scans, manual audits, and indemnity-aware practices.

  19. Assessing and Mitigating Data Memorization Risks in Fine-Tuned Large Language Models

    cs.CL 2025-08 unverdicted novelty 4.0 of 10

    Fine-tuning LLMs on repeated sensitive data is claimed to raise privacy leakage to 60-75%, and four filters are said to cut leakage to 0% while keeping 94.7% of utility.

  20. SoK: Semantic Privacy in Large Language Models

    cs.CR 2025-06 conditional novelty 4.0 of 10

    A systematization of knowledge arguing that LLM privacy threats extend beyond data leakage to semantically inferred attributes, and that current defenses only partially address them.

  21. SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation

    cs.CR 2025-06 conditional novelty 3.0 of 10

    A systematization-of-knowledge survey that categorizes LLM privacy risks into training data, prompts, outputs, and agents, and reviews limitations of current mitigations.

Pith tools