REVIEW 33 cited by
Design Patterns for Securing LLM Agents against Prompt Injections
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Design Patterns for Securing LLM Agents against Prompt Injections
read the original abstract
As AI agents powered by Large Language Models (LLMs) become increasingly versatile and capable of addressing a broad spectrum of tasks, ensuring their security has become a critical challenge. Among the most pressing threats are prompt injection attacks, which exploit the agent's resilience on natural language inputs -- an especially dangerous threat when agents are granted tool access or handle sensitive information. In this work, we propose a set of principled design patterns for building AI agents with provable resistance to prompt injection. We systematically analyze these patterns, discuss their trade-offs in terms of utility and security, and illustrate their real-world applicability through a series of case studies.
Forward citations
Cited by 33 Pith papers
-
Trojan Hippo: Weaponizing Agent Memory for Data Exfiltration
Trojan Hippo attacks on LLM agent memory achieve 85-100% success rates in data exfiltration across four memory backends even after 100 benign sessions, while evaluated defenses reduce success rates but impose varying ...
-
Prismata: Confining Cross-Site Prompt Injection in Web Agents
Prismata cuts web-agent prompt-injection attack success from 85.5% to 0.7% via Biba-inspired DOM trust labeling and mechanical least-privilege confinement without site annotations.
-
Beyond Self-Resolution: Settlement Factorization for Robust Natural Language Mechanism
Settlement factorization is a normal form: every mechanism admits a ghost-reference label with leakage ε within factor two of optimal, and truthful margins degrade by at most the tight constant 2Lε.
-
Agent Data Injection Attacks are Realistic Threats to AI Agents
Agent data injection (ADI) forges trusted agent metadata via probabilistic delimiter injection and bypasses defenses built only for instruction injection.
-
Data Flow Control: Data Safety Policies for AI Agents
Data Flow Control formalizes data safety as aggregate predicates over provenance monomials and implements enforcement via the Passant query rewriting layer achieving near-zero overhead across five DBMS engines.
-
What You Approve Is What Executes: Consent Integrity for Black-Box LLM Agents
The paper introduces Consent Integrity as the property that actions shown for approval must be rendered by a trusted mediator from the real boundary action over an unspoofable path and bound to execution, with uninspe...
-
IPI-proxy: An Intercepting Proxy for Red-Teaming Web-Browsing AI Agents Against Indirect Prompt Injection
IPI-proxy is a toolkit using an intercepting proxy to inject indirect prompt injection attacks into live web pages for testing AI browsing agents against hidden instructions.
-
LogJack: Indirect Prompt Injection Through Cloud Logs Against LLM Debugging Agents
LogJack shows indirect prompt injection via cloud logs succeeds in making LLM agents execute remote code on 6 of 8 models, with most cloud guardrails failing to detect the attacks.
-
Prompt Injection Attack to Tool Selection in LLM Agents
ToolHijacker optimizes malicious tool documents via a two-phase strategy to hijack LLM agents' tool selection in no-box settings.
-
AISPA: User-Centric System Prompt Auditing for Large Language Model Applications
A human-in-the-loop audit of system prompts from 88 commercial AI products finds protective instructions nearly universal yet incomplete, with ~40% of products containing at least one user-harmful directive.
-
Are You Still the Agent I Authorized? Earned Authority under a Fixed Ceiling for Evolving Agents
Evolving agents may change active authority only beneath an immutable user-issued effect ceiling, and a transition envelope decides whether the old grant survives mutation at all.
-
How Context Attribution Handles What the Model Already Knows
Context attribution methods cannot disentangle in-context from in-weight knowledge and assign unfaithful scores under overlap; new metrics and WMDP-Cyber++ quantify the failure.
-
Agent Security Needs Redefinition through a Holistic Framework
Agent security should be redefined around four contextual authorization properties instead of the content of the action performed.
-
IssueTrojanBench: Benchmarking AI Coding Agents Against Malicious Issue Requests
AI coding agents followed malicious instructions embedded in issue-style artifacts in 66.5% of 4,176 test runs.
-
Specifying the Delegated-Autonomy Boundary: Requirements Engineering for Agentic AI
A requirements-engineering framework that justifies when agentic AI is warranted (AJR) and specifies graduated autonomy, memory, coordination, assurance, and evolution policies (ADP).
-
Janus: a Playground for User-Involved Agentic Permission Management
Janus is a publicly available playground system and evaluation harness for testing user-involved permission management designs in AI agents, demonstrating benefits of user input and the need for context-sensitive approaches.
-
GIF: Locally Sound Geometric Information Flow Control for LLMs
GIF introduces a Jacobian-based upper bound on input-output mutual information in LLMs with formal Lean proof and strong empirical recall on injection and leakage benchmarks.
-
Game-Theoretic Multi-Agent Control for Robust Contextual Reasoning in LLMs
GT-MCP coordinates three LLM agents via a trust function and rollback to bound contextual drift and block adversarial injections in multi-turn interactions.
-
Web Agents Should Adopt the Plan-Then-Execute Paradigm
Web agents should default to planning a complete task program before observing live web content to reduce prompt injection exposure, since WebArena tasks are compatible and 80% need no runtime LLM calls.
-
Trojan Hippo: Weaponizing Agent Memory for Data Exfiltration
The paper defines and evaluates Trojan Hippo attacks on LLM agent memory, showing 85-100% success in data exfiltration across backends and reduced rates with defenses at varying utility costs.
-
Alignment Contracts for Agentic Security Systems
Alignment contracts define scope, allowed effects, budgets and disclosure rules as safety properties over finite effect traces, with decidable admissibility, refinement rules, and Lean-verified soundness under an obse...
-
Tracking Capabilities for Safer Agents
AI agents can generate code in a capability-safe Scala dialect that statically prevents information leakage and malicious side effects while preserving task performance.
-
Semantic Attacks on Tool-Augmented LLMs: Securing the Model Context Protocol Against Descriptor-Level Manipulation
Descriptor-level manipulation in the Model Context Protocol can drive LLMs to unsafe tool selections in up to 36% of cases; a layered defense of integrity checks, auxiliary-LLM vetting, and runtime guardrails reduces ...
-
Prevalence of Security and Privacy Risk-Inducing Usage of AI-based Conversational Agents
Roughly a third of UK adults use AI chatbots weekly, and among them a substantial minority upload untrusted content, connect bots to other programs, share sensitive data, or attempt jailbreaks.
-
When Compression Becomes an Attack Surface: Black-Box Attacks on Prompt-Compressed LLM Agents
The paper claims prompt compression is a new attack surface, but the abstract's COMA attack never appears in the body and the body's SoftCom requires white-box access.
-
Agents That Know Too Much: A Data-Centric Survey of Privacy in LLM Agents
A data-centric survey finds that only information-flow control covers compositional and cross-session leakage in LLM agents and that no single benchmark tests an agent across all its data surfaces under one policy.
-
Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agents
An independent reproduction on AgentDojo with Qwen2.5-7B finds that the Progent out-of-band defense reduces mean attack success from 25.8% to 4.2% and holds against a hand-crafted adaptive attack at 2.6%.
-
Lingering Authority: Revocable Resource-and-Effect Capabilities for Coding Agents
PORTICO is a revocable capability reference monitor for coding agents that enforces task contracts via grant-invoke-closure lifecycles and rejects post-closure reuses while preserving task success.
-
The Insurability Frontier of AI Risk: Mapping Threats to Affirmative Coverage, Silent Exposures, and Exclusions
The paper codes AI threats from OWASP/MITRE catalogs against public insurance materials to delineate a four-tier insurability frontier separating affirmative coverage, silent exposures, exclusions, and uninsurable perils.
-
Adversarial Prompting Framework for AI Safety Assessment
An adversarial prompt testing framework with a five-level attack taxonomy and a composite harmfulness score is proposed; the paper claims encoded prompts bypass safety filters most often.
-
Agentic-J: An AI Agent for Biological Microscopy Image Analysis
Agentic-J is a multi-agent AI assistant that converts natural language descriptions of biological image analysis tasks into executable, reproducible scripts for ImageJ/Fiji with specialised sub-agents for plugin manag...
-
Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
A survey that taxonomizes threats to agentic AI, reviews benchmarks and evaluation methods, discusses technical and governance defenses, and identifies open challenges.
-
Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation
A synthesis of 247 papers on LLM agent security identifies prompt injection and tool hijacking as dominant threats, notes weakly compositional defenses, and argues for trust boundaries and realistic evaluations.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.