REVIEW 15 cited by
Extracting Training Data from Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
It has become common to publish large (billion parameter) language models that have been trained on private datasets. This paper demonstrates that in such settings, an adversary can perform a training data extraction attack to recover individual training examples by querying the language model. We demonstrate our attack on GPT-2, a language model trained on scrapes of the public Internet, and are able to extract hundreds of verbatim text sequences from the model's training data. These extracted examples include (public) personally identifiable information (names, phone numbers, and email addresses), IRC conversations, code, and 128-bit UUIDs. Our attack is possible even though each of the above sequences are included in just one document in the training data. We comprehensively evaluate our extraction attack to understand the factors that contribute to its success. Worryingly, we find that larger models are more vulnerable than smaller models. We conclude by drawing lessons and discussing possible safeguards for training large language models.
Forward citations
Cited by 15 Pith papers
-
Bits and Memories: Measuring Verbatim Extraction Across LLM Quantization
Quantizing LLMs selectively forgets memorized text faster than capability, but 1B-scale 4-bit models still extract ~72% of memorized sequences, so quantization is not a privacy defense.
-
Unlearning as Distribution Restoration: A Controlled Counterfactual Study, a Validated Selective Screen, and the Limits of Oracle-Free Certification
Matching a retrained oracle on trained probes can certify models that still retain held-out forget knowledge, and oracle-free unlearning certification is only possible for counterfactual, non-inferable facts.
-
AI Native Games: A Survey and Roadmap
The paper proposes a counterfactual definition of AI-native games, screens 53 examples, introduces a G/N taxonomy, and outlines a research roadmap for the field.
-
Evaluating Differentially Private Generation of Domain-Specific Text
Applying a new benchmark to five specialized domains, the paper shows current privacy-preserving text generators lose much of their utility and fidelity, especially at strict privacy levels and on gated datasets.
-
AnalogFed: Privacy-Preserving Discovery of Analog Circuits at Scale with Federated Generative AI
AnalogFed combines federated learning with a generative analog-topology model, adding dummy-token input perturbation and partial homomorphic encryption to resist membership inference and model inversion attacks.
-
Adversarial Machine Learning Attacks on Financial Reporting via Maximum Violated Multi-Objective Attack
MVMO, a new weighted multi-objective attack, can inflate earnings and lower fraud scores in about 50 to 66 percent of firm-years, versus under 14 percent for standard attacks.
-
Approximating Language Model Training Data from Weights
A gradient-based greedy selection method (SELECT) recovers effective substitute fine-tuning data from two language model checkpoints, approaching the original model's performance on classification and SFT tasks.
-
DECAF: De-Clustering for Adaptive Representational Unlearning
DECAF is a forget-only unlearning method that adds input noise, suppresses the forget-class probability, and diversifies outputs, achieving 0.10% forget accuracy and 79.4% retain accuracy on CIFAR-10/ResNet-18 while d...
-
ISACL: Internal State Analyzer for Copyrighted Training Data Leakage
An MLP trained on LLM internal states predicts Rouge-L-defined literal copying leakage with high accuracy, but not paraphrase-level leakage.
-
Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation
Sparse autoencoder activation perturbation (SFPF) applied on top of existing jailbreak prompts raises attack success rate on Qwen3-32B, but with no defense evaluation and weak reproducibility.
-
On the Performance of Differentially Private Optimization with Heavy-Tail Class Imbalance
Under heavy-tail class imbalance, subtracting the DP noise variance from Adam's second moment (DP-AdamBC) substantially improves learning of rare classes compared with DP gradient descent.
-
Low-Perplexity LLM-Generated Sequences and Where To Find Them
Only about 40% of low-perplexity 6-token spans generated by Pythia-6.9B can be exactly matched to The Pile, and the authors categorize matched and unmatched spans into four classes.
-
From Teacher to Student: Tracking Memorization Through Model Distillation
Distillation of fine-tuned GPT-2 teachers into smaller students lowers verbatim memorization of training responses compared to direct fine-tuning, with reverse-KL giving the lowest rates.
-
Preventing Another Tessa: Modular Safety Middleware For Health-Adjacent AI Assistants
A modular safety middleware combining deterministic filters and a single-call LLM self-verdict reports perfect blocking of dieting-related unsafe content on a small synthetic evaluation.
-
Model Inversion Attacks on Llama 3: Extracting PII from Large Language Models
Simple prompts to Llama 3.2 1B produce text that resembles PII, but the paper does not establish that this text is memorized training data.
Discussion (0). Sign in to comment.