REVIEW 15 cited by
Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems
read the original abstract
Retrieval-Augmented Generation (RAG) improves pre-trained models by incorporating external knowledge at test time to enable customized adaptation. We study the risk of datastore leakage in Retrieval-In-Context RAG Language Models (LMs). We show that an adversary can exploit LMs' instruction-following capabilities to easily extract text data verbatim from the datastore of RAG systems built with instruction-tuned LMs via prompt injection. The vulnerability exists for a wide range of modern LMs that span Llama2, Mistral/Mixtral, Vicuna, SOLAR, WizardLM, Qwen1.5, and Platypus2, and the exploitability exacerbates as the model size scales up. We also study multiple effects of RAG setup on the extractability of data, indicating that following unexpected instructions to regurgitate data can be an outcome of failure in effectively utilizing contexts for modern LMs, and further show that such vulnerability can be greatly mitigated by position bias elimination strategies. Extending our study to production RAG models GPTs, we design an attack that can cause datastore leakage with a 100% success rate on 25 randomly selected customized GPTs with at most 2 queries, and we extract text data verbatim at a rate of 41% from a book of 77,000 words and 3% from a corpus of 1,569,000 words by prompting the GPTs with only 100 queries generated by themselves.
Forward citations
Cited by 15 Pith papers
-
Isolated but Exposed: Persistence-Based Memory Extraction Attack on LLM Agents
Malicious tools can systematically extract isolated LLM-agent long-term memory via persistence, pure-anchor retrieval steering, and reactivation payloads, reaching 80% extraction with unlimited triggers and 47% with 20.
-
MRMMIA: Membership Inference Attacks on Memory in Chat Agents
MRMMIA is a multi-recall-probe membership inference attack that extracts signals from chat agent memory and outperforms baselines in black-, gray-, and white-box settings.
-
AgenTEE: Confidential LLM Agent Execution on Edge Devices
AgenTEE isolates LLM agent runtime, inference, and apps in independently attested cVMs on Arm-based edge devices, achieving under 5.15% overhead versus commodity OS deployments.
-
Not All Entities are Created Equal: A Dynamic Anonymization Framework for Privacy-Preserving RAG
TRIP-RAG dynamically anonymizes only high-risk entities in RAG knowledge bases via three context-aware metrics, achieving privacy comparable to full anonymization with under 35% recall drop and up to 56% better genera...
-
Trustworthiness in Retrieval-Augmented Generation Systems: A Survey
Introduces Trust-RAG Compass framework and TRC Bench benchmark to assess RAG trustworthiness across factuality, robustness, fairness, transparency, accountability, and privacy, with evaluations showing performance gap...
-
LeakDojo: Decoding the Leakage Threats of RAG Systems
LeakDojo benchmarks reveal that RAG leakage is approximated by the product of query generation and adversarial instruction effects, with higher risk tied to LLM instruction-following ability and RAG faithfulness improvements.
-
ADAM: A Systematic Data Extraction Attack on Agent Memory via Adaptive Querying
ADAM extracts data from LLM agent memory with up to 100% attack success rate by estimating data distribution and selecting queries via entropy guidance.
-
Not All Entities are Created Equal: A Dynamic Anonymization Framework for Privacy-Preserving RAG
Context-aware selective entity anonymization for RAG matches full-anonymization privacy with <35% Recall@k drop and up to 56% better generation quality than baselines.
-
Adaptive Backtracking for Privacy Protection in Large Language Models
A training-free backtracking defense that rewrites RAG output at the first sign of privacy leakage improves privacy utility scores by up to 15% over sanitization and prompting baselines on a new healthcare and finance...
-
Agents That Know Too Much: A Data-Centric Survey of Privacy in LLM Agents
A data-centric survey finds that only information-flow control covers compositional and cross-session leakage in LLM agents and that no single benchmark tests an agent across all its data surfaces under one policy.
-
Benchmarking Knowledge-Extraction Attack and Defense on Retrieval-Augmented Generation
A unified benchmark comparing RAG knowledge-extraction attacks and defenses, showing query diversity boosts extraction, embedding attacks fail to transfer, and graph indexing raises per-token leakage.
-
Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation
Sparse autoencoder activation perturbation (SFPF) applied on top of existing jailbreak prompts raises attack success rate on Qwen3-32B, but with no defense evaluation and weak reproducibility.
-
Diverse And Private Synthetic Datasets Generation for RAG evaluation: A multi-agent framework
A multi-agent LLM framework generates synthetic QA datasets for RAG evaluation by combining clustering-based sampling, PII pseudonymization, and QA curation, with reported diversity gains and 0.75-0.90 masking accuracy.
-
RAG Security and Privacy: Formalizing the Threat Model and Attack Surface
A formal RAG threat model is defined with four adversary classes and game-based notions of membership inference, leakage, and poisoning, but the definitions largely restate known concepts and the main DP-based protect...
-
Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety
A comprehensive survey that taxonomizes safety threats to large models and agents, reviews defenses and benchmarks, and outlines open challenges.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.