REVIEW 18 cited by
InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Models
read the original abstract
Prompt injection attacks pose a critical threat to large language models (LLMs), enabling goal hijacking and data leakage. Prompt guard models, though effective in defense, suffer from over-defense -- falsely flagging benign inputs as malicious due to trigger word bias. To address this issue, we introduce NotInject, an evaluation dataset that systematically measures over-defense across various prompt guard models. NotInject contains 339 benign samples enriched with trigger words common in prompt injection attacks, enabling fine-grained evaluation. Our results show that state-of-the-art models suffer from over-defense issues, with accuracy dropping close to random guessing levels (60%). To mitigate this, we propose InjecGuard, a novel prompt guard model that incorporates a new training strategy, Mitigating Over-defense for Free (MOF), which significantly reduces the bias on trigger words. InjecGuard demonstrates state-of-the-art performance on diverse benchmarks including NotInject, surpassing the existing best model by 30.8%, offering a robust and open-source solution for detecting prompt injection attacks. The code and datasets are released at https://github.com/leolee99/InjecGuard.
Forward citations
Cited by 18 Pith papers
-
TRUSTDESC: Preventing Tool Poisoning in LLM Applications via Trusted Description Generation
TRUSTDESC prevents tool poisoning in LLM applications by automatically generating accurate tool descriptions from code via a three-stage pipeline of reachability analysis, description synthesis, and dynamic verification.
-
Gate AI: LLM Security Benchmark Evaluation Methodology and Results
Introduces a cross-validation-based evaluation methodology for LLM security detectors using a global threshold and group-fold leakage checks to avoid per-dataset tuning.
-
ARGUS: Defending LLM Agents Against Context-Aware Prompt Injection
ARGUS cuts context-aware prompt-injection success from 28.8% to 3.8% on AgentLure while keeping 87.5% clean utility, beating prior defenses on the security-utility tradeoff.
-
Conjunctive Prompt Attacks in Multi-Agent LLM Systems
Conjunctive prompt attacks split adversarial elements across agents and routing paths in multi-agent LLM systems, evading isolated defenses and succeeding through topology-aware optimization.
-
Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling
An activation-guided energy model plus MCMC sampling creates transferable direct prompt injection attacks in a black-box setting, reaching 49.6% average attack success across five LLMs.
-
From Neural Intent to Cryptographic Authorization: Securing AI-Driven Enterprise Workflows
NCS gates every privileged AI-agent tool call behind an offline-signed, hash-chained instruction stream, cutting injected-parameter attacks to zero by construction.
-
Confidently Wrong: Severity-Aware Calibration of Prompt-Injection Detectors under Attack Shift
Prompt-injection detectors miss attacks with near-certainty (severity 0.99-1.00) under shifts; standard pooled calibration error fails to detect miscalibration on the attacks themselves.
-
Where Instruction Hierarchy Breaks: Diagnosing and Repairing Failures in Reasoning Language Models
A diagnostic framework localizes instruction hierarchy failures in LLMs into identification, resolution, and realization, while self-monitors reduce non-compliance by 81-99%.
-
Context-Aware Spear Phishing: Generative AI-Enabled Attacks Against Individuals via Public Social Media Data
Generative AI enables scalable, context-aware spear phishing by extracting profiles from public social media, producing emails that outperform real-world phishing samples in personalization and lower recipient suspicion.
-
ARGUS: Defending LLM Agents Against Context-Aware Prompt Injection
ARGUS defends LLM agents from context-aware prompt injections by tracking information provenance and verifying decisions against trustworthy evidence, reducing attack success to 3.8% while retaining 87.5% task utility.
-
MCP Pitfall Lab: Exposing Developer Pitfalls in MCP Tool Server Security under Multi-Vector Attacks
MCP Pitfall Lab operationalizes six pitfall classes across tool-metadata poisoning, puppet servers, and multimodal chains, showing that recommended hardening removes all Tier-1 static findings and that agent narrative...
-
AgentWatcher: A Rule-based Prompt Injection Monitor
An attention-attribution plus rule-reasoning detector reduces prompt-injection success to near zero on agent and long-context benchmarks with small utility loss.
-
MCP Pitfall Lab: Exposing Developer Pitfalls in MCP Tool Server Security under Multi-Vector Attacks
MCP Pitfall Lab reports 31.9% attack success across 2,579 runs and shows Semantic MCP-BOM plus hardening cuts residual risk while static fields alone miss cross-tool and multi-modal leakage.
-
Prompt Injection Detection is Regime-Dependent: A Deployment-Aware Evaluation with Interpretable Structural Signals
Prompt injection detection performance is highly regime-dependent with no single detector dominating across settings; transformer models perform best overall while structural signals offer modest gains in some regimes.
-
Cross-Lingual Jailbreak Detection via Semantic Codebooks
Semantic similarity to an English jailbreak codebook detects cross-lingual attacks with high accuracy on curated benchmarks but shows poor separability on diverse unsafe prompts.
-
Structured Security Auditing and Robustness Enhancement for Untrusted Agent Skills
SkillGuard-Robust formulates pre-load auditing of untrusted Agent Skills as a three-way classification task and achieves 97.30% exact match and 98.33% malicious-risk recall on held-out benchmarks.
-
SafeAgent: A Runtime Protection Architecture for Agentic Systems
SafeAgent is a stateful runtime protection system that improves LLM agent robustness to prompt injections over baselines while preserving task performance.
-
Verifying Intent and Harm: A Unified Defense Against LLM-Generated Threats
A joint prompt-response verification framework using intent analysts, harm analysts, and a judge improves average F1 to 0.95 and cuts attack success rate to 4.1% across jailbreaks, prompt injection, phishing, cyber ab...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.