REVIEW 19 cited by
AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
LLM agents have demonstrated remarkable performance across various applications, primarily due to their advanced capabilities in reasoning, utilizing external knowledge and tools, calling APIs, and executing actions to interact with environments. Current agents typically utilize a memory module or a retrieval-augmented generation (RAG) mechanism, retrieving past knowledge and instances with similar embeddings from knowledge bases to inform task planning and execution. However, the reliance on unverified knowledge bases raises significant concerns about their safety and trustworthiness. To uncover such vulnerabilities, we propose a novel red teaming approach AgentPoison, the first backdoor attack targeting generic and RAG-based LLM agents by poisoning their long-term memory or RAG knowledge base. In particular, we form the trigger generation process as a constrained optimization to optimize backdoor triggers by mapping the triggered instances to a unique embedding space, so as to ensure that whenever a user instruction contains the optimized backdoor trigger, the malicious demonstrations are retrieved from the poisoned memory or knowledge base with high probability. In the meantime, benign instructions without the trigger will still maintain normal performance. Unlike conventional backdoor attacks, AgentPoison requires no additional model training or fine-tuning, and the optimized backdoor trigger exhibits superior transferability, in-context coherence, and stealthiness. Extensive experiments demonstrate AgentPoison's effectiveness in attacking three types of real-world LLM agents: RAG-based autonomous driving agent, knowledge-intensive QA agent, and healthcare EHRAgent. On each agent, AgentPoison achieves an average attack success rate higher than 80% with minimal impact on benign performance (less than 1%) with a poison rate less than 0.1%.
Forward citations
Cited by 19 Pith papers
-
ChronoMem: Version Control and Semantic Rollback for Large Language Model Agent Memory
Memory versioning with semantic rollback lets LLM agents behave counterfactually after later interactions, improving rollback-consistent QA and summarization.
-
Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings
A ground-truth-first synthetic memory benchmark shows that agent-memory architecture rankings invert with history length: short-horizon leaders lose at nine weeks.
-
What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents
KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.
-
Epistemic Bias Injection: Manipulating LLM Opinion via Selective Context Retrieval
Factually correct but one-sided passages injected into a RAG database can shift LLM answers toward an attacker's stance, and a PCA/KL-based filter (BiasDef) reduces that shift.
-
LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems
A systematic review that categorizes LLM threats, severity scores, and mitigations across development and operation life cycles and multiple deployment scenarios.
-
Adaptive Content Restriction for Large Language Models via Suffix Optimization
SOP appends an optimized suffix to prompts, reducing generation of user-specified restricted terms across several LLMs while keeping output quality close to baseline prompting methods.
-
From Seed to Harvest: Augmenting Human Creativity with AI for Red-teaming Text-to-Image Models
Seed2Harvest expands 1,000 human adversarial prompts into 27,650 LLM-generated variants that keep roughly comparable unsafe-image trigger rates and add hundreds of new geographic contexts.
-
Defining and Detecting the Defects of the Large Language Model-based Autonomous Agents
This study defines eight defect types for LLM-based agents and presents Agentable, a CPG-plus-LLM static analysis tool that detects them with reported precision of 88.79% and recall of 91.03%.
-
How Well Do AI Systems Solve AP Physics? A Comparative Evaluation of Large Language Models on Algebra-Based Free Response Questions
ChatGPT 4.1 mini, Gemini 2.5 Flash, Claude 4.0 Sonnet, and DeepSeek R1 average 82–92% on AP Physics 1/2 free-response questions but systematically fail spatial, visual, and conceptual tasks.
-
Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation
Sparse autoencoder activation perturbation (SFPF) applied on top of existing jailbreak prompts raises attack success rate on Qwen3-32B, but with no defense evaluation and weak reproducibility.
-
Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks
Simple prompt-injection attacks against real-world LLM agents like Anthropic Computer Use, MultiOn, and ChemCrow succeed at leaking data and enabling harmful actions.
-
Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment
Causal reward modeling applies MMD regularization to RLHF reward models to make reward scores statistically independent of spurious features, reducing measured length, sycophancy, concept, and demographic biases in ex...
-
The Chronos Vulnerability: A Taxonomy of Temporal Persistence and Memory-Based Deception in Agentic AI
The paper is a taxonomy of persistent memory-based attacks on AI agents, reinterpreting known attacks (MINJA, EchoLeak, sleeper agents) under the umbrella 'Chronos Vulnerability' and proposing a defense-in-depth architecture.
-
A Whole New World: Creating a Parallel-Poisoned Web Only AI-Agents Can See
A website can identify AI agents by their digital fingerprints and serve them a poisoned hidden version of the page, hijacking their actions via indirect prompt injection.
-
Evaluation and Benchmarking of LLM Agents: A Survey
A review that proposes a two-dimensional taxonomy for evaluating LLM agents and highlights enterprise-specific evaluation gaps.
-
Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques
A survey that maps LLM applications, vulnerabilities, and defenses across eight cybersecurity domains, but with significant citation and rigor problems.
-
Securing Agentic AI: From Per-Action Checks to Trajectory Assurance
A vision paper organizing agentic AI security into eleven research directions, with the thesis that safety requires verifiable trajectory-level behavioral guarantees, not per-action checks.
-
Report on NSF Workshop on Science of Safe AI
An NSF workshop report articulating a cross-disciplinary research agenda for designing and verifying safe, trustworthy AI systems.
-
A Survey on Backdoor Threats in Large Language Models (LLMs): Attacks, Defenses, and Evaluations
A literature review that taxonomizes LLM backdoor attacks and defenses by model construction phase, with no new experimental results.
Discussion (0). Continue with ORCID to comment.