Pith. sign in

REVIEW 35 cited by

PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.07867 v3 pith:QA3EN4CR submitted 2024-02-12 cs.CR cs.LG

Wei Zou , Runpeng Geng , Binghui Wang , Jinyuan Jia This is my paper · ORCID
classification cs.CRcs.LG
keywords knowledgeattackdatabasepoisonedragtextscorruptiongenerationmalicious
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs) have achieved remarkable success due to their exceptional generative capabilities. Despite their success, they also have inherent limitations such as a lack of up-to-date knowledge and hallucination. Retrieval-Augmented Generation (RAG) is a state-of-the-art technique to mitigate these limitations. The key idea of RAG is to ground the answer generation of an LLM on external knowledge retrieved from a knowledge database. Existing studies mainly focus on improving the accuracy or efficiency of RAG, leaving its security largely unexplored. We aim to bridge the gap in this work. We find that the knowledge database in a RAG system introduces a new and practical attack surface. Based on this attack surface, we propose PoisonedRAG, the first knowledge corruption attack to RAG, where an attacker could inject a few malicious texts into the knowledge database of a RAG system to induce an LLM to generate an attacker-chosen target answer for an attacker-chosen target question. We formulate knowledge corruption attacks as an optimization problem, whose solution is a set of malicious texts. Depending on the background knowledge (e.g., black-box and white-box settings) of an attacker on a RAG system, we propose two solutions to solve the optimization problem, respectively. Our results show PoisonedRAG could achieve a 90% attack success rate when injecting five malicious texts for each target question into a knowledge database with millions of texts. We also evaluate several defenses and our results show they are insufficient to defend against PoisonedRAG, highlighting the need for new defenses.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 35 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Combating Knowledge Corruption in Agent Systems: A Byzantine-Tolerant Secure Collaborative RAG Framework

    cs.CR 2026-08 reject novelty 6.0 of 10

    SecureCollaRAG filters poisoned RAG documents with dynamic GNN credibility scoring, but its formal proof depends on an assumed cluster separation that the introduced ATA attack is designed to violate.

  2. Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Sparse attention changes which prompt content influences model answers, an effect that can cancel out in aggregate accuracy and is visible through a dense-calibrated counterfactual audit.

  3. GPE: Evaluating Robust Evidence Aggregation for Fact Verification under Controllable GEO-Style Poisoning

    cs.CR 2026-07 conditional novelty 6.0 of 10

    GPE, a new benchmark with controllable GEO-style poisoning, shows LLM fact verifiers degrade sharply under poisoned evidence, with no single verifier winning across all attack types.

  4. Certified Domain Consistency for Multi-Domain Retrieval: Label-Free Per-Domain Contamination Control with Conformal Risk Guarantees

    cs.LG 2026-07 conditional novelty 6.0 of 10

    C3R certifies per-domain retrieval contamination budgets using a two-split conformal scheme, without query-time domain labels.

  5. Epistemic Bias Injection: Manipulating LLM Opinion via Selective Context Retrieval

    cs.CR 2025-11 conditional novelty 6.0 of 10

    Factually correct but one-sided passages injected into a RAG database can shift LLM answers toward an attacker's stance, and a PCA/KL-based filter (BiasDef) reduces that shift.

  6. ImportSnare: Directed "Code Manual" Hijacking in Retrieval-Augmented Code Generation

    cs.CR 2025-09 conditional novelty 6.0 of 10

    Documentation poisoning with hidden ranking and suggestion sequences can make RAG-based code generators confidently recommend malicious dependencies, even at 0.01% poisoning ratios.

  7. Safeguarding RAG Pipelines with GMTP: A Gradient-based Masked Token Probability Method for Poisoned Document Detection

    cs.CL 2025-07 conditional novelty 6.0 of 10

    GMTP detects poisoned RAG documents by masking high-gradient tokens and flagging documents whose masked-token probabilities are abnormally low.

  8. RAG Safety: Exploring Knowledge Poisoning Attacks to Retrieval-Augmented Generation

    cs.CR 2025-07 conditional novelty 6.0 of 10

    Inserting a small number of crafted triples into a knowledge graph can flip KG-RAG question answering toward attacker-chosen incorrect answers across four recent systems.

  9. The Hidden Threat in Plain Text: Attacking RAG Data Loaders

    cs.CR 2025-07 conditional novelty 6.0 of 10

    Invisible characters and formatting tricks in ingested documents survive popular RAG data loaders and can manipulate end-to-end RAG outputs.

  10. Bias Amplification in RAG: Poisoning Knowledge Retrieval to Steer LLMs

    cs.LG 2025-06 reject novelty 6.0 of 10

    A retrieval-augmented generation system can be poisoned with reward-optimized biased documents and vector-space manipulation to substantially increase biased LLM outputs.

  11. Spa-VLM: Stealthy Poisoning Attacks on RAG-based VLM

    cs.CR 2025-05 conditional novelty 6.0 of 10

    Spa-VLM crafts paired adversarial images and misleading texts to poison RAG-based VLM knowledge bases, reaching attack success rates above 0.8 with just five injected entries.

  12. The Silent Saboteur: Imperceptible Adversarial Attacks against Black-Box Retrieval-Augmented Generation Systems

    cs.IR 2025-05 conditional novelty 6.0 of 10

    A reinforcement-learning attack that swaps a few words in a single document pushes it into a black-box RAG system's top-3 results and flips the generated answer around 45-47% of the time.

  13. Safety Degradation in AI Agents

    cs.CY 2025-05 conditional novelty 6.0 of 10

    Adding retrieval to aligned LLMs degrades safety: refusal rates fall, bias and harmfulness rise, and prompt-based mitigation only partially restores alignment.

  14. MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents

    cs.CR 2025-02 conditional novelty 6.0 of 10

    MELON detects indirect prompt injection in LLM agents by re-executing each step with the user prompt replaced by a neutral task prompt and flagging attacks when the original and masked runs produce similar tool calls.

  15. Unsafe LLM-Based Search: Quantitative Analysis and Mitigation of Safety Risks in AI Web Search

    cs.CR 2025-02 conditional novelty 6.0 of 10

    Seven production AI search engines frequently surface malicious URLs in their answers; direct URL queries make this worse, and an LLM-based defense agent can filter most of it.

  16. Exploring the Security Threats of Knowledge Base Poisoning in Retrieval-Augmented Code Generation

    cs.CR 2025-02 conditional novelty 6.0 of 10

    Poisoning the knowledge base of a retrieval-augmented code generator with vulnerable snippets raises the vulnerability rate of the model's output, with the size of the rise depending on the retriever, the model, and t...

  17. RbFT: Robust Fine-tuning for Retrieval-Augmented Generation against Retrieval Defects

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Fine-tuning an LLM with defect detection and utility extraction tasks makes it more robust to noisy, irrelevant, and counterfactual documents in retrieval-augmented generation.

  18. TriShieldRAG: A Three-Ring Defense-in-Depth Framework Against Knowledge Corruption in Retrieval-Augmented Generation

    cs.CR 2026-07 conditional novelty 5.0 of 10

    A three-ring RAG defense cuts non-adaptive PoisonedRAG attack success from ~91% to ~13% on 10 questions over a 5,000-document Wikipedia base.

  19. Operational Evidence Gaps for LLMs in Fraud Detection and Trust-and-Safety Workflows

    cs.CR 2026-07 conditional novelty 5.0 of 10

    A survey of 49 LLM fraud and trust-and-safety papers finds that fraud work reports almost no per-decision latency, cost, or calibration evidence, while moderation work reports more.

  20. Lexical Hints of Accuracy in LLM Reasoning Chains

    cs.CL 2025-08 reject novelty 5.0 of 10

    Hesitation words in reasoning chains are claimed to flag incorrect LLM answers, but the manuscript body is a different paper and contains no such study.

  21. Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation

    cs.CL 2025-08 reject novelty 5.0 of 10

    Sparse autoencoder activation perturbation (SFPF) applied on top of existing jailbreak prompts raises attack success rate on Qwen3-32B, but with no defense evaluation and weak reproducibility.

  22. CrEst: Credibility Estimation for Contexts in LLMs via Weak Supervision

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A label-free method that scores retrieved documents by their agreement with the majority in embedding space and uses those scores to filter context in LLM question answering.

  23. We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systems

    cs.LG 2025-06 conditional novelty 5.0 of 10

    MCP-powered LLM agents are vulnerable to prompt injection from third-party services, and simple detection or filtering defenses do not reliably stop these attacks.

  24. Evaluating the Retrieval Robustness of Large Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A realistic benchmark with three metrics shows modern LLMs are generally robust to imperfect retrieval, though not perfectly so.

  25. Chain-of-Thought Poisoning Attacks against R1-based Retrieval-Augmented Generation Systems

    cs.IR 2025-05 conditional novelty 5.0 of 10

    Wrapping erroneous knowledge in a reasoning model's own chain-of-thought template raises poisoning attack success on an R1-based RAG system by 10 percentage points over the strongest prior method in a 100-query MS MARCO test.

  26. Ranking Free RAG: Replacing Re-ranking with Selection in RAG for Sensitive Domains

    cs.CL 2025-05 reject novelty 5.0 of 10

    METEORA uses DPO-tuned rationales to select and verify evidence chunks in RAG, and claims better recall, precision, evidence efficiency, and poisoning defense, though key evaluation details are missing.

  27. Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval

    cs.CR 2025-05 conditional novelty 5.0 of 10

    SCR uses retrieval-augmented generation to fetch refusal examples that block jailbreak attacks, but the reported advantages are partly overstated.

  28. Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks

    cs.LG 2025-02 conditional novelty 5.0 of 10

    Simple prompt-injection attacks against real-world LLM agents like Anthropic Computer Use, MultiOn, and ChemCrow succeed at leaking data and enabling harmful actions.

  29. Poison-RAG: Adversarial Data Poisoning Attacks on Retrieval-Augmented Generation in Recommender Systems

    cs.IR 2025-01 conditional novelty 5.0 of 10

    Poison-RAG shows that injected tag edits can reduce the exposure of popular items in a RAG recommender, but promoting long-tail items remains ineffective.

  30. Quantifying Conversation Drift in MCP via Latent Polytope

    cs.CL 2025-08 reject novelty 4.0 of 10

    SecMCP flags MCP conversation drift by thresholding per-layer activation distances from benign anchors, reporting AUROC above 0.915 on Llama3, Vicuna, and Mistral.

  31. PRJ: Perception-Retrieval-Judgement for Generated Images

    cs.CV 2025-06 reject novelty 4.0 of 10

    A new safety checker for AI-generated images, built from a vision-language model, retrieval-augmented knowledge lookup, and an LLM judge, reports higher detection rates and category-level toxicity scores than three ex...

  32. ParetoRAG: Leveraging Sentence-Context Attention for Robust and Efficient Retrieval-Augmented Generation

    cs.CL 2025-02 conditional novelty 4.0 of 10

    ParetoRAG decomposes retrieved passages into weighted sentence-plus-context embeddings, retrieves top-k sentences, and reports better RAG accuracy and fluency at about one-third the token cost.

  33. Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Survey

    cs.CL 2025-02 conditional novelty 4.0 of 10

    A survey organizes current research on trustworthy RAG into six pillars, reliability, privacy, safety, fairness, explainability, and accountability, and maps methods, metrics, and open problems for each.

  34. Securing Agentic AI: From Per-Action Checks to Trajectory Assurance

    cs.AI 2026-08 conditional novelty 3.0 of 10

    A vision paper organizing agentic AI security into eleven research directions, with the thesis that safety requires verifiable trajectory-level behavioral guarantees, not per-action checks.

  35. A Survey on Backdoor Threats in Large Language Models (LLMs): Attacks, Defenses, and Evaluations

    cs.CR 2025-02 conditional novelty 2.0 of 10

    A literature review that taxonomizes LLM backdoor attacks and defenses by model construction phase, with no new experimental results.

Pith tools