REVIEW 8 cited by
Attention Tracker: Detecting Prompt Injection Attacks in LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) have revolutionized various domains but remain vulnerable to prompt injection attacks, where malicious inputs manipulate the model into ignoring original instructions and executing designated action. In this paper, we investigate the underlying mechanisms of these attacks by analyzing the attention patterns within LLMs. We introduce the concept of the distraction effect, where specific attention heads, termed important heads, shift focus from the original instruction to the injected instruction. Building on this discovery, we propose Attention Tracker, a training-free detection method that tracks attention patterns on instruction to detect prompt injection attacks without the need for additional LLM inference. Our method generalizes effectively across diverse models, datasets, and attack types, showing an AUROC improvement of up to 10.0% over existing methods, and performs well even on small LLMs. We demonstrate the robustness of our approach through extensive evaluations and provide insights into safeguarding LLM-integrated systems from prompt injection vulnerabilities.
Forward citations
Cited by 8 Pith papers
-
CrackedPDFs: A Controlled Benchmark for Hidden Prompt Injection in PDFs
A 29,322-PDF controlled benchmark shows that a hybrid structural-plus-text detector finds hidden PDF prompt injections under paired evaluation (0.960 F1; 100% pair ranking), while text-only baselines fail.
-
RouteGuard: Internal-Signal Detection of Skill Poisoning in LLM Agents
RouteGuard uses response-conditioned attention and hidden-state alignment to detect skill poisoning in LLM agents, achieving 0.8834 F1 on Skill-Inject benchmarks and recovering 90.51% of attacks missed by lexical screening.
-
"Your AI, My Shell": Demystifying Prompt Injection Attacks on Agentic AI Coding Editors
Prompt injection attacks on agentic AI coding editors like Cursor and GitHub Copilot reach up to 84% success in executing malicious commands by poisoning external development resources.
-
LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems
A systematic review that categorizes LLM threats, severity scores, and mitigations across development and operation life cycles and multiple deployment scenarios.
-
To trust or not to trust: Attention-based Trust Management for LLM Multi-Agent Systems
Introduces six-dimension trustworthiness definition and attention-based A-Trust score with a TMS to improve LLM-MAS robustness against malicious or unreliable messages.
-
CrackedPDFs: A Controlled Benchmark for Hidden Prompt Injection in PDFs
A document-aware hybrid detector that inspects PDF structure before text flattening outperforms text-only guardrails and structural-only models on a new 29,322-file controlled hidden-prompt-injection benchmark.
-
PromptArmor: Simple yet Effective Prompt Injection Defenses
PromptArmor prompts a separate LLM to detect and remove injected prompts from agent inputs, achieving below 1% false positive and false negative rates on AgentDojo.
-
Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
A survey that taxonomizes threats to agentic AI, reviews benchmarks and evaluation methods, discusses technical and governance defenses, and identifies open challenges.
Discussion (0). Sign in to comment.