REVIEW 9 cited by
DE-COP: Detecting Copyrighted Content in Language Models Training Data
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
How can we detect if copyrighted content was used in the training process of a language model, considering that the training data is typically undisclosed? We are motivated by the premise that a language model is likely to identify verbatim excerpts from its training text. We propose DE-COP, a method to determine whether a piece of copyrighted content was included in training. DE-COP's core approach is to probe an LLM with multiple-choice questions, whose options include both verbatim text and their paraphrases. We construct BookTection, a benchmark with excerpts from 165 books published prior and subsequent to a model's training cutoff, along with their paraphrases. Our experiments show that DE-COP surpasses the prior best method by 9.6% in detection performance (AUC) on models with logits available. Moreover, DE-COP also achieves an average accuracy of 72% for detecting suspect books on fully black-box models where prior methods give approximately 4% accuracy. The code and datasets are available at https://github.com/LeiLiLab/DE-COP.
Forward citations
Cited by 9 Pith papers
-
Differentiation-Based Extraction of Proprietary Data from Fine-Tuned LLMs
A new attack, DDE, extracts supervised fine-tuning data from LLMs by exploiting low-confidence tokens and comparing fine-tuned versus base model generation branches.
-
Unlocking Post-hoc Dataset Inference with Synthetic Data
Synthetic suffix-completion data, plus a dual-classifier t-test calibration, enables Dataset Inference to detect training membership while avoiding the false positives caused by distribution shift.
-
PPE-Bench: A Benchmark for Evaluating MLLM Unlearning under Private-Public Entanglement
Existing MLLM unlearning methods reduce private-attribute leakage on entangled images but substantially harm co-occurring public figures and landmarks, with private knowledge often re-emerging after public finetuning.
-
Identifying Pre-training Data in LLMs: A Neuron Activation-Based Detection Framework
NA-PDD detects pre-training data in LLMs by comparing which neurons activate for a test text against neurons linked to known training versus non-training texts, and claims large AUC improvements on three benchmarks.
-
Automatic Calibration for Membership Inference Attack on Large Language Models
A temperature-calibrated scoring method, ACMIA, improves membership inference against LLMs by amplifying probability gaps between training and non-training texts without reference models.
-
Investigating the Feasibility of Mitigating Potential Copyright Infringement via Large Language Model Unlearning
SSU combines task-vector negation, random-label loss, and weight saliency to forget copyrighted books sequentially while retaining more general language ability than existing baselines.
-
LeakAgent: RL-based Red-teaming Agent for LLM Privacy Leakage
An RL-trained prompt generator outperforms rule-based and fuzzing baselines at extracting system prompts and training data from black-box LLMs.
-
SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks
SOFT paraphrases low-loss fine-tuning samples before training, reducing MIA AUC from about 0.82 to about 0.54 across six datasets at roughly 7% perplexity cost.
-
On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMs
Applying CVSS, DREAD, OWASP, and SSVC to 56 adversarial LLM attacks via three LLM judges yields near-constant factor scores, which the authors take as evidence that these metrics cannot differentiate LLM attacks.
Discussion (0). Continue with ORCID to comment.