REVIEW 37 cited by
StruQ: Defending Against Prompt Injection with Structured Queries
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Recent advances in Large Language Models (LLMs) enable exciting LLM-integrated applications, which perform text-based tasks by utilizing their advanced language understanding capabilities. However, as LLMs have improved, so have the attacks against them. Prompt injection attacks are an important threat: they trick the model into deviating from the original application's instructions and instead follow user directives. These attacks rely on the LLM's ability to follow instructions and inability to separate prompts and user data. We introduce structured queries, a general approach to tackle this problem. Structured queries separate prompts and data into two channels. We implement a system that supports structured queries. This system is made of (1) a secure front-end that formats a prompt and user data into a special format, and (2) a specially trained LLM that can produce high-quality outputs from these inputs. The LLM is trained using a novel fine-tuning strategy: we convert a base (non-instruction-tuned) LLM to a structured instruction-tuned model that will only follow instructions in the prompt portion of a query. To do so, we augment standard instruction tuning datasets with examples that also include instructions in the data portion of the query, and fine-tune the model to ignore these. Our system significantly improves resistance to prompt injection attacks, with little or no impact on utility. Our code is released at https://github.com/Sizhe-Chen/StruQ.
Forward citations
Cited by 37 Pith papers
-
Prismata: Confining Cross-Site Prompt Injection in Web Agents
Prismata cuts web-agent prompt-injection attack success from 85.5% to 0.7% via Biba-inspired DOM trust labeling and mechanical least-privilege confinement without site annotations.
-
DualView: Preventing Indirect Prompt Injection in Personal AI Agents
DualView extends Dual-LLM symbol isolation into the shared user environment via dual Agent/Human views, blocking both immediate and stored IPI at 0% ASR while preserving near-baseline utility.
-
Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling
An activation-guided energy model plus MCMC sampling creates transferable direct prompt injection attacks in a black-box setting, reaching 49.6% average attack success across five LLMs.
-
Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interface
Attackers can use the loss signal from a remote LLM fine-tuning API to optimize adversarial prefix and suffix tokens, turning existing prompt injections into high-success attacks on closed-weight Gemini models.
-
Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applications
An agent-based black-box attack, CopyBreakRAG, progressively extracts over 70% of chunks from RAG knowledge bases by alternating curiosity-driven exploration with reasoning-based exploitation.
-
Beyond Handcrafted Security: Towards Self-Evolving Defense for LLM Agents
A self-evolving runtime defense for LLM agents, HARD, improves security over static handcrafted defenses by updating context policies and action gates from failure traces.
-
ContainmentBench: Trace-Based Evaluation of Post-Injection Containment in Tool-Using LLM Agents
Terminal policy labels are insufficient: two containment policies with identical zero-harm endpoints still differ in 73.5% of trajectories and in authorized-work completion.
-
Data Leakage Prevention in Agentic Applications via Preemptive Hardening
A build-time pipeline that scans, patches, and validates agentic LLM apps reduced prompt-injection leakage to 0% on most tested apps and by 91% on the hardest stress case.
-
ChannelGuard: Safe Models Do Not Compose into Safe Multi-Agent Systems
An undefended multi-agent pipeline's apparent perfect attack-safety is mostly the cloud provider's hidden filter, silently re-sourcing to model alignment on other backends; embedding gates restore deterministic applic...
-
NetInjectBench: Benchmarking Indirect Prompt Injection in Tool-Using Large Language Model Agents for Network Operations
A metadata-aware policy gate yields 0/240 unsafe attack tool actions under metadata integrity while preserving approved high-impact changes, outperforming prompt defenses and static allowlists on NetInjectBench.
-
Token-Flow Firewall: Semantic Runtime Auditing for Persistent AI Agents
TokenWall mediates persistent-agent security by auditing source–sink token flows with a local small model and selective large-model escalation, cutting CIK-Bench attack success to 12.5% at low benign latency.
-
Measuring the Security of Mobile LLM Agents under Adversarial Prompts from Untrusted Third-Party Channels
Indirect prompt injection through ads, webviews, and notifications reliably diverts mobile LLM agents into leaking data and installing malware across eight evaluated agents.
-
SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defenses
A systemization of LLM jailbreak security that adds linked taxonomies, an evaluation platform, and JailbreakDB, while its main attack–defense comparison results remain deferred.
-
Defending Against Prompt Injection With a Few DefensiveTokens
Five optimized token embeddings prepended to the prompt reduce prompt-injection attack success to near zero on standard benchmarks while preserving most model utility.
-
Context manipulation attacks : Web agents are susceptible to corrupted memory
Web agents with protected prompts can still be hijacked by injecting malicious steps into their stored task plans, reaching up to 63% success on privacy leaks.
-
Invisible Prompts, Visible Threats: Malicious Font Injection in External Resources for Large Language Models
Maliciously crafted fonts can hide adversarial prompts from human eyes while LLMs read and obey them, enabling content manipulation and data exfiltration via MCP tools.
-
Adversarial Suffix Filtering: a Defense Pipeline for LLMs
ASF detects and removes adversarial suffixes by segmenting prompts and classifying each segment with a fine-tuned BERT model, reducing jailbreak success on several LLMs while leaving normal-task performance roughly intact.
-
AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents
AgentVigil uses Monte Carlo Tree Search and coverage-guided scoring to automatically craft indirect prompt injection attacks that succeed 71% and 70% of the time on two agent benchmarks, about double the baseline.
-
The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them)
Fine-tuned LLMs separate roles via task-type and begin-of-text shortcuts; PFT, which inserts a gap into position IDs during fine-tuning, reduces those shortcuts and improves closed-domain attack robustness.
-
MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents
MELON detects indirect prompt injection in LLM agents by re-executing each step with the user prompt replaced by a neutral task prompt and flagging attacks when the original and masked runs produce similar tool calls.
-
The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents
Task Shield cuts indirect prompt injection attack success to 2.07% on AgentDojo with GPT-4o by using an LLM to verify that each instruction and tool call contributes to the user's explicit goals.
-
Towards Action Hijacking of Large Language Model-based Agent
A RAG-based LLM application can be induced to assemble harmful SQL, code, or medical action plans from knowledge already stored in its database, with the user prompt itself carrying no forbidden words.
-
No Free Lunch for Defending Against Prefilling Attack by In-Context Learning
Adversative in-context examples ('Sure... However...') defend many LLMs against prefilling jailbreaks but cause over-refusal, so the defense trades safety for helpfulness.
-
LeakAgent: RL-based Red-teaming Agent for LLM Privacy Leakage
An RL-trained prompt generator outperforms rule-based and fuzzing baselines at extracting system prompts and training data from black-box LLMs.
-
Rethinking Penetration Testing for AI-Enabled Systems: From Resource Compromise to Behavioral Objective Violation
AI-enabled penetration is redefined as the feasible induction of AI-governed behavior that violates an operational objective, with a workflow for testing it.
-
Isolating LLM Alignment from Regex: Zero Coverage and Metric-Dependent Divergence Under Adversarial Mutation
Under a substring classifier, Gemini-2.5-flash with the regex removed blocks 0% of 45 adversarial probes; under an LLM judge, it blocks 56–100% of PAIR variants.
-
A Real-Time, Self-Tuning Moderator Framework for Adversarial Prompt Detection
RTST, a two-agent moderator with an explainable Behavior ledger and per-prompt weight updates, reduced attack success rate from 12-63% to 0-17% on three jailbreak benchmarks with Gemini 2.5 Flash.
-
PromptArmor: Simple yet Effective Prompt Injection Defenses
PromptArmor prompts a separate LLM to detect and remove injected prompts from agent inputs, achieving below 1% false positive and false negative rates on AgentDojo.
-
Detection Method for Prompt Injection by Integrating Pre-trained Model and Heuristic Feature Engineering
A dual-channel detector combining DeBERTa and heuristic rules is claimed to beat existing prompt injection detectors on three benchmarks and to reduce attack success on GLM-4, Llama 3, Qwen 2.5, and GPT-4o.
-
PromptShield: Deployable Detection for Prompt Injection Attacks
PromptShield reports a 65.3% true positive rate at 0.1% false positive rate for prompt injection detection, more than six times the best prior model, on its own out-of-distribution evaluation split.
-
Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Models
JMLLM, a hybrid obfuscation framework, raises jailbreak success rates across text, image, and speech inputs of multimodal LLMs while using fewer queries than prior methods.
-
Quantifying Conversation Drift in MCP via Latent Polytope
SecMCP flags MCP conversation drift by thresholding per-layer activation distances from benign anchors, reporting AUROC above 0.915 on Llama3, Vicuna, and Mistral.
-
A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
The paper surveys security risks of LLM agents, organizes them into a five-level autonomy taxonomy, and proposes an untested CMDP-based architecture called R2A2.
-
From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystem
A structured survey of recent jailbreak attacks and defenses across LLMs, multimodal LLMs, and agents, with taxonomies for methods, datasets, metrics, and defenses.
-
Design Patterns for Securing LLM Agents against Prompt Injections
Six composable design patterns (action-selector, plan-then-execute, map-reduce, dual LLM, code-then-execute, context-minimization) constrain LLM agents so prompt-injected text cannot reach consequential actions.
-
Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents
A survey proposing a source-and-impact taxonomy (input, model, combined; security, privacy, ethics) for threats to LLM-based agents, with feature analysis and four case studies.
-
Attack and defense techniques in large language models: A survey and new perspectives
A survey taxonomizing LLM attacks into prompt, optimized, application-level, and model-theft categories, and defenses into prevention and detection categories, with no new empirical results.
Discussion (0). Continue with ORCID to comment.