REVIEW 50 cited by
PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) have achieved remarkable success due to their exceptional generative capabilities. Despite their success, they also have inherent limitations such as a lack of up-to-date knowledge and hallucination. Retrieval-Augmented Generation (RAG) is a state-of-the-art technique to mitigate these limitations. The key idea of RAG is to ground the answer generation of an LLM on external knowledge retrieved from a knowledge database. Existing studies mainly focus on improving the accuracy or efficiency of RAG, leaving its security largely unexplored. We aim to bridge the gap in this work. We find that the knowledge database in a RAG system introduces a new and practical attack surface. Based on this attack surface, we propose PoisonedRAG, the first knowledge corruption attack to RAG, where an attacker could inject a few malicious texts into the knowledge database of a RAG system to induce an LLM to generate an attacker-chosen target answer for an attacker-chosen target question. We formulate knowledge corruption attacks as an optimization problem, whose solution is a set of malicious texts. Depending on the background knowledge (e.g., black-box and white-box settings) of an attacker on a RAG system, we propose two solutions to solve the optimization problem, respectively. Our results show PoisonedRAG could achieve a 90% attack success rate when injecting five malicious texts for each target question into a knowledge database with millions of texts. We also evaluate several defenses and our results show they are insufficient to defend against PoisonedRAG, highlighting the need for new defenses.
Forward citations
Cited by 50 Pith papers
-
DiscourseFlip: An Oblique Discourse-Level Opinion Manipulation Attack against Black-box Retrieval-Augmented Generation
DiscourseFlip is a graph-guided attack allocating limited poisoning budget to induce targeted opinion shifts over semantic query networks in black-box RAG.
-
REALISTA: Realistic Latent Adversarial Attacks that Elicit LLM Hallucinations
REALISTA optimizes continuous combinations of valid editing directions in latent space to produce realistic adversarial prompts that elicit hallucinations more effectively than prior methods, including on large reason...
-
Oracle Poisoning: Corrupting Knowledge Graphs to Weaponise AI Agent Reasoning
Oracle Poisoning corrupts knowledge graphs used by AI agents via tool calls, leading tested models to accept fabricated claims at 100% under directed queries in a production-scale demonstration.
-
Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents
This paper delivers the first systematic taxonomy and cross-benchmark consistency analysis of 40 agent safety benchmarks, finding broad but shallow risk coverage, no ranking concordance across evaluations, and that be...
-
Self-Study Reconsidered: The Hidden Fragility of Learning from Self-Generated QA
Self-generated QA supervision for language models is fragile due to non-uniform question selection and instruction compliance during answering, with mitigations that reduce compliance from 88% to 13%.
-
Securing LLM-Agent Long-Term Memory Against Poisoning: Non-Malleable, Origin-Bound Authority with Machine-Checked Guarantees
Presents TMA-NM, a non-malleable origin-bound authority system for LLM-agent memory with TLA+ machine-checked separation theorems and benchmarks showing 0% attack success against direct and laundering poisoning while ...
-
SafeClawBench: Separating Semantic, Audit-Evidence, and Sandbox Harm in Tool-Using LLM Agents
SafeClawBench supplies 600 staged adversarial tasks and three separate endpoints that show semantic acceptance, audit evidence, and sandbox-observed harm are distinct failure modes in tool-using LLM agents.
-
Discourse-Role Labels as Presentation-Time Variables for Context Use in Language Models
Discourse-role labels on identical misleading context cause 56-84 percentage point shifts in LLMs adopting the injected wrong answer.
-
Does RAG Know When Retrieval Is Wrong? Diagnosing Context Compliance under Knowledge Conflict
Context-Driven Decomposition (CDD) measures context compliance in RAG under knowledge conflicts and improves accuracy on adversarial benchmarks like TruthfulQA misconception injection and Epi-Scale tests across models.
-
VectorSmuggle: Steganographic Exfiltration in Embedding Stores and a Cryptographic Provenance Defense
Steganographic exfiltration attacks succeed on embedding stores via retrieval-preserving perturbations such as small-angle orthogonal rotation, but an Ed25519-based provenance signature closes the attack class.
-
A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Framework
A new 7x4 taxonomy organizes agentic AI security threats by architectural layer and persistence timescale, revealing under-explored upper layers and missing defenses after surveying 116 papers.
-
AdversarialCoT: Single-Document Retrieval Poisoning for LLM Reasoning
A single query-specific poisoned document, built by extracting and iteratively refining an adversarial chain-of-thought, can substantially degrade reasoning accuracy in retrieval-augmented LLM systems.
-
Prompt Injection Attack to Tool Selection in LLM Agents
ToolHijacker optimizes malicious tool documents via a two-phase strategy to hijack LLM agents' tool selection in no-box settings.
-
Adversarial Hubness in Multi-Modal Retrieval
Adversarial hubs can be generated to be retrieved as top-1 for over 84% of test queries in text-to-image retrieval, far exceeding natural hubs.
-
Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents
ASB is a new benchmark that tests 10 prompt injection attacks, memory poisoning, a novel Plan-of-Thought backdoor attack, and 11 defenses on LLM agents across 13 models, finding attack success rates up to 84.3% and li...
-
Trustworthiness in Retrieval-Augmented Generation Systems: A Survey
Introduces Trust-RAG Compass framework and TRC Bench benchmark to assess RAG trustworthiness across factuality, robustness, fairness, transparency, accountability, and privacy, with evaluations showing performance gap...
-
Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation
Sparse attention changes which prompt content influences model answers, an effect that can cancel out in aggregate accuracy and is visible through a dense-calibrated counterfactual audit.
-
GPE: Evaluating Robust Evidence Aggregation for Fact Verification under Controllable GEO-Style Poisoning
GPE, a new benchmark with controllable GEO-style poisoning, shows LLM fact verifiers degrade sharply under poisoned evidence, with no single verifier winning across all attack types.
-
Certified Domain Consistency for Multi-Domain Retrieval: Label-Free Per-Domain Contamination Control with Conformal Risk Guarantees
C3R certifies per-domain retrieval contamination budgets using a two-split conformal scheme, without query-time domain labels.
-
Evidence-Bound Gateway-Path Provenance for Third-Party LLM Inference
Proposes evidence-bound LLM gateway using attested runtime for verifiable path provenance and policy enforcement.
-
Selection Integrity for LLM Graph Memory: An Accumulability Criterion for Information-Flow-Blind Retrieval
Provenance checks in graph memory are blind to structural attacks that reallocate top-k membership; authselect prevents this by enforcing selection on the authenticated subgraph only.
-
Game-Theoretic Multi-Agent Control for Robust Contextual Reasoning in LLMs
GT-MCP coordinates three LLM agents via a trust function and rollback to bound contextual drift and block adversarial injections in multi-turn interactions.
-
Inference Cost Attacks for Retrieval-Augmented Large Language Models
Poisoning external knowledge bases with LLM-agent-crafted documents can increase RAG inference token consumption by up to 13.12 times at over 90% success rate while preserving answer quality.
-
Adversarial Feeds Steer LLM Agent Decisions Against Their Defaults
Controlled experiments show adversarial feeds can tip uncertain LLM agent decisions from 5% to 100% alignment with the feed while leaving firmly held defaults unchanged, following a dose-response pattern across multip...
-
Detecting Is Not Resolving: The Monitoring Control Gap in Retrieval Augmented LLMs
RAG models exhibit a monitoring-control gap: they acknowledge epistemic conflicts in accumulating documents yet fail to constrain unsafe recommendations, with single-turn tests overestimating safety.
-
An Empirical Study of Privacy Leakage Chains via Prompt Injection in Black-Box Chatbot Environments
Empirical demonstration that prompt injection combined with web-tool use creates a feasible privacy-leakage chain in deployed black-box chatbot agents.
-
Does RAG Know When Retrieval Is Wrong? Diagnosing Context Compliance under Knowledge Conflict
Introduces CDD to diagnose context compliance in RAG under knowledge conflicts, reporting measurable compliance, cross-model accuracy transfer without causal coupling transfer, and robustness gains on Epi-Scale and Tr...
-
Does RAG Know When Retrieval Is Wrong? Diagnosing Context Compliance under Knowledge Conflict
A five-step decomposition probe and a new conflict benchmark show that RAG context compliance is measurable, and that accuracy gains from decomposition can transfer across model families even when causal coupling to t...
-
REALISTA: Realistic Latent Adversarial Attacks that Elicit LLM Hallucinations
REALISTA generates semantically coherent adversarial prompts via latent-space optimization over input-dependent editing directions, achieving stronger hallucination elicitation than prior realistic attacks on open-sou...
-
When Child Inherits: Modeling and Exploiting Subagent Spawn in Multi-Agent Networks
Multi-agent LLM frameworks can spread compromises across agent boundaries via insecure memory inheritance during subagent spawning.
-
Your Agent, Their Asset: A Real-World Safety Analysis of OpenClaw
Poisoning any single CIK dimension of an AI agent raises average attack success rate from 24.6% to 64-74% across models, and tested defenses leave substantial residual risk.
-
Epistemic Bias Injection: Manipulating LLM Opinion via Selective Context Retrieval
Factually correct but one-sided passages injected into a RAG database can shift LLM answers toward an attacker's stance, and a PCA/KL-based filter (BiasDef) reduces that shift.
-
ADMIT: Few-shot Knowledge Poisoning Attacks on RAG-based Fact Checking
ADMIT achieves 86% average attack success rate on RAG fact-checking at 0.93×10^{-6} poisoning rate across 4 retrievers, 11 LLMs, and 4 benchmarks while remaining robust to counter-evidence.
-
ImportSnare: Directed "Code Manual" Hijacking in Retrieval-Augmented Code Generation
Documentation poisoning with hidden ranking and suggestion sequences can make RAG-based code generators confidently recommend malicious dependencies, even at 0.01% poisoning ratios.
-
Across Programming Language Silos: A Study on Cross-Lingual Retrieval-augmented Code Generation
Cross-lingual RACG shows non-trivial but unequal knowledge transfer across 13 programming languages, depending on linguistic affinity and pretraining diversity, with limited reliance on natural language information wh...
-
One Shot Dominance: Knowledge Poisoning Attack on Retrieval-Augmented Generation Systems
AuthChain poisons a single document to achieve high-success attacks on RAG systems for multi-hop queries across six LLMs while evading defenses.
-
TriShieldRAG: A Three-Ring Defense-in-Depth Framework Against Knowledge Corruption in Retrieval-Augmented Generation
A three-ring RAG defense cuts non-adaptive PoisonedRAG attack success from ~91% to ~13% on 10 questions over a 5,000-document Wikipedia base.
-
Operational Evidence Gaps for LLMs in Fraud Detection and Trust-and-Safety Workflows
A survey of 49 LLM fraud and trust-and-safety papers finds that fraud work reports almost no per-decision latency, cost, or calibration evidence, while moderation work reports more.
-
A Lifecycle and Application-Stack Survey of Large Language Model Vulnerabilities: Attacks, Risks, Defenses, and Open Problems
The paper provides a lifecycle-based systematization of LLM vulnerabilities across data collection, pretraining, alignment, packaging, retrieval, prompting, tool execution, and deployment, mapping them to security obj...
-
ToE: A Hierarchical and Explainable Claim Verification Framework with Dynamic Multi-source Evidence Retrieval and Aggregation
ToE is a hierarchical claim verification framework using RL-driven multi-source retrieval, evidence evaluation, and tree aggregation that reports 4-24 point gains over baselines especially on poisoned inputs.
-
ARENA: An Architecture for Measuring the Transferability of Autonomous Cyber Defense
ARENA creates anonymized SOC telemetry artifacts that reveal a measurable privacy-utility boundary when used both as training material for MITRE-mapped challenges and as a substrate to detect non-compliant LLM defende...
-
Does RAG Know When Retrieval Is Wrong? Diagnosing Context Compliance under Knowledge Conflict
Introduces CDD probe that measures context compliance in RAG, reporting 15% accuracy on TruthfulQA misconception injection and cross-model transfer patterns on Epi-Scale benchmark.
-
PRA-RAG: Provably Robust Aggregation in Retrieval-Augmented Generation against Retrieval Corruption
PRA-RAG is a new aggregation algorithm for RAG that claims provable robustness bounds against poisoned retrieved texts and reduces attack success rate to 1% while keeping 71% accuracy.
-
RefineRAG: Word-Level Poisoning Attacks via Retriever-Guided Text Refinement
RefineRAG achieves 90% attack success on NQ by generating toxic seeds then optimizing them via retriever-in-the-loop word refinement, outperforming prior methods on effectiveness and naturalness.
-
Lexical Hints of Accuracy in LLM Reasoning Chains
Hesitation words in reasoning chains are claimed to flag incorrect LLM answers, but the manuscript body is a different paper and contains no such study.
-
Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation
Sparse autoencoder activation perturbation (SFPF) applied on top of existing jailbreak prompts raises attack success rate on Qwen3-32B, but with no defense evaluation and weak reproducibility.
-
Through the Stealth Lens: Attention-Aware Defenses Against Poisoning in RAG
Introduces NPAS and AV Filter using LLM attention weights to defend RAG against poisoning, reporting up to 20% accuracy gains while adaptive attacks reach 35% success.
-
Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
A survey that taxonomizes threats to agentic AI, reviews benchmarks and evaluation methods, discusses technical and governance defenses, and identifies open challenges.
-
Position: LLM Watermarking Should Align Stakeholders' Incentives for Practical Adoption
LLM watermarking adoption is limited by misaligned stakeholder incentives; incentive-aligned approaches such as in-context watermarking can enable practical use in targeted domains like education and peer review.
-
Quantifying Conversation Drift in MCP via Latent Polytope
SecMCP flags MCP conversation drift by thresholding per-layer activation distances from benign anchors, reporting AUROC above 0.915 on Llama3, Vicuna, and Mistral.
Discussion (0). Sign in to comment.