REVIEW 8 cited by
Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We consider the problem of a training data proof, where a data creator or owner wants to demonstrate to a third party that some machine learning model was trained on their data. Training data proofs play a key role in recent lawsuits against foundation models trained on web-scale data. Many prior works suggest to instantiate training data proofs using membership inference attacks. We argue that this approach is fundamentally unsound: to provide convincing evidence, the data creator needs to demonstrate that their attack has a low false positive rate, i.e., that the attack's output is unlikely under the null hypothesis that the model was not trained on the target data. Yet, sampling from this null hypothesis is impossible, as we do not know the exact contents of the training set, nor can we (efficiently) retrain a large foundation model. We conclude by offering two paths forward, by showing that data extraction attacks and membership inference on special canary data can be used to create sound training data proofs.
Forward citations
Cited by 8 Pith papers
-
Membership Inference Attacks for Unseen Classes
In a new 'unseen class' setting for membership inference, quantile regression attacks outperform shadow model attacks, which fall to the level of a global-threshold baseline.
-
What Really is a Member? Discrediting Membership Inference via Poisoning
Membership inference tests can be driven below random accuracy by poisoning the training set, even under relaxed, neighborhood-based membership definitions.
-
Causal Evaluation of Membership Inference Attacks
The causal framing of MIA evaluation is new, but the headline zero-run IPW estimator is mathematically wrong for imbalanced data, and the claimed LLM validation is absent from the experiments.
-
Identifying Pre-training Data in LLMs: A Neuron Activation-Based Detection Framework
NA-PDD detects pre-training data in LLMs by comparing which neurons activate for a test text against neurons linked to known training versus non-training texts, and claims large AUC improvements on three benchmarks.
-
Membership Inference Attacks as Privacy Tools: Reliability, Disparity and Ensemble
MIAs expose different members depending on attack method and random seed; the paper quantifies this with coverage/stability and shows ensembling attacks yields stronger, more reliable privacy checks.
-
Membership Inference Attacks on Sequence Models
Modeling per-token loss correlations with a multivariate Gaussian improves LiRA membership inference on sequence models, often by an order of magnitude in TPR at low FPR.
-
SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks
SOFT paraphrases low-loss fine-tuning samples before training, reducing MIA AUC from about 0.82 to about 0.54 across six datasets at roughly 7% perplexity cost.
- Winter Soldier: Backdooring Language Models at Pre-Training with Indirect Data Poisoning
Discussion (0). Sign in to comment.