REVIEW 6 cited by
Analyzing Leakage of Personally Identifiable Information in Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Language Models (LMs) have been shown to leak information about training data through sentence-level membership inference and reconstruction attacks. Understanding the risk of LMs leaking Personally Identifiable Information (PII) has received less attention, which can be attributed to the false assumption that dataset curation techniques such as scrubbing are sufficient to prevent PII leakage. Scrubbing techniques reduce but do not prevent the risk of PII leakage: in practice scrubbing is imperfect and must balance the trade-off between minimizing disclosure and preserving the utility of the dataset. On the other hand, it is unclear to which extent algorithmic defenses such as differential privacy, designed to guarantee sentence- or user-level privacy, prevent PII disclosure. In this work, we introduce rigorous game-based definitions for three types of PII leakage via black-box extraction, inference, and reconstruction attacks with only API access to an LM. We empirically evaluate the attacks against GPT-2 models fine-tuned with and without defenses in three domains: case law, health care, and e-mails. Our main contributions are (i) novel attacks that can extract up to 10$\times$ more PII sequences than existing attacks, (ii) showing that sentence-level differential privacy reduces the risk of PII disclosure but still leaks about 3% of PII sequences, and (iii) a subtle connection between record-level membership inference and PII reconstruction. Code to reproduce all experiments in the paper is available at https://github.com/microsoft/analysing_pii_leakage.
Forward citations
Cited by 6 Pith papers
-
Leak It: A Probabilistic Approach to Training-Data Extraction from Black-Box Language Models
Sampling-based LLM attacks reproduce exact identifiers from 16.6% of 500 Pile documents at Pythia-6.9B even though aggregate sampling-MIA adds no signal over blind baselines, so privacy audits should report per-docume...
-
MultivationBench: A Benchmark for Multimodal Sequential Motivation Reasoning
MultivationBench, a 16,092-question picture-story benchmark grounded in Maslow's and Reiss's motivation theories, shows that all tested multimodal LLMs score well below humans and almost never maintain consistent moti...
-
XToM: Exploring the Multilingual Theory of Mind for Large Language Models
XToM translates three English theory-of-mind benchmarks into Chinese, German, French, and Japanese with human quality control, and shows LLMs' belief reasoning is weaker and less consistent across languages than their...
-
ISACL: Internal State Analyzer for Copyrighted Training Data Leakage
An MLP trained on LLM internal states predicts Rouge-L-defined literal copying leakage with high accuracy, but not paraphrase-level leakage.
-
Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation
Sparse autoencoder activation perturbation (SFPF) applied on top of existing jailbreak prompts raises attack success rate on Qwen3-32B, but with no defense evaluation and weak reproducibility.
-
From Teacher to Student: Tracking Memorization Through Model Distillation
Distillation of fine-tuned GPT-2 teachers into smaller students lowers verbatim memorization of training responses compared to direct fine-tuning, with reverse-KL giving the lowest rates.
Discussion (0). Sign in to comment.