REVIEW 4 major objections 6 minor 22 cited by
LlamaFirewall: An open source guardrail system for building secure AI agents
T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Layering a lightweight jailbreak classifier with a chain-of-thought alignment auditor reduces benchmark agent prompt-injection success from 17.6% to 1.75%.
desk verdict Useful open-source guardrail framework with a genuinely novel AlignmentCheck, but the headline AgentDojo numbers conflate detection with prevention and need major re-analysis. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a policy engine with three interchangeable scanners. PromptGuard 2 is a fine-tuned DeBERTa-family classifier (86M and 22M variants) that detects explicit jailbreak phrasing in user and tool messages; an energy-based loss and hardened tokenization are meant to reduce false positives and resist token-splitting evasion. AlignmentCheck is a few-shot guardrail LLM that takes the original user goal, the agent's trace, and the selected action, and judges whether that action pursues an unintended goal; it is the semantic layer that catches goal hijacking that lexical filters miss. CodeShield is a two-tier static analyzer using Semgrep and regex rules across eight languages and over 50 CWEs, with a fast first pass around 60 milliseconds and a deeper second pass around 300 milliseconds for flagged code. The framework lets developers compose these detectors with conditional remediation actions, in the spirit of network-security rule engines.
What would settle it
A controlled test with an agent fine-tuned to suppress or rationalize injected goals: if the agent obeys an injected exfiltration instruction while its chain-of-thought remains goal-consistent, and AlignmentCheck labels the actions aligned, then the reported 84 to 90 percent reductions would not transfer to that agent class.
Extended reading notes
Core claim
The central claim, stated on the paper's own terms, is that a layered defense is quantitatively stronger than any single guardrail: on AgentDojo, PromptGuard 2 (86M) alone drops attack success rate from 17.6% to 7.5% (a 57% reduction), AlignmentCheck backed by Llama 4 Maverick alone drops it to 2.89% (an 84% reduction), and the two together reach 1.75% (over 90% reduction), at a utility of 42.7% versus the 47.7% unprotected baseline. The paper further claims AlignmentCheck is the first open-source guardrail to audit an LLM's chain of thought in real time for injection defense, and that larger guardrail models (Llama 4 Maverick, Llama 3.3 70B) detect over 80% of goal hijacks in an in-house benchmark with false positive rates below 4%. CodeShield, the third guardrail, is claimed to achieve 96% precision and 79% recall on manually labeled insecure code completions, providing a real-time static-analysis backstop for coding agents.
Load-bearing premise
AlignmentCheck's effectiveness rests on the assumption that agent LLMs verbalize their true intent in chain-of-thought, so a semantic auditor can observe misalignment; a deceptive or adversarially trained agent that hides the injected goal gives AlignmentCheck no signal.
Editorial extensions
If this is right
- Production agent deployments can put a cheap lexical filter in front of an expensive semantic auditor, since the combined configuration removes most attacks with only a modest utility drop.
- AlignmentCheck can be added to an existing agent without fine-tuning: it consumes the user goal, the trace, and the most recent action through a static prompt template.
- The compact PromptGuard 2 22M variant gives a low-latency option for local, resource-constrained deployments while still cutting attack success rate by 41% on AgentDojo.
- CodeShield gives coding agents a feedback loop: unsafe code is rejected before landing, while the agent is free to iterate, so a benign-but-insecure pattern never enters the codebase.
- The open release means security teams can write their own regex or prompt-based scanners and share policies, letting the guardrail set grow with the threat landscape.
Reading between the lines
- Editorial inference: if AlignmentCheck works because agent chains of thought are transparent, then the natural next attack is to train agents to produce clean rationalizations while executing injected goals; testing this would require an adversarially fine-tuned agent with hidden or deceptive reasoning.
- Editorial inference: the reported utility drop (47.7% to 42.7%) is measured as task success on AgentDojo traces; real-world false-positive costs could be higher for high-stakes actions, so deployment may favor risk-triggered AlignmentCheck only on sensitive operations.
- Editorial inference: combining AlignmentCheck with instruction-hierarchy fine-tuning is a testable extension; prior work shows hierarchy training alone removes only part of the violations, and AlignmentCheck should catch residual cases that still surface in reasoning.
- Editorial inference: the same trace-auditing pattern could extend to multimodal agents, since it only needs a structured record of observations and actions; the paper lists multimodal support as future work, but the mechanism does not depend on text-only inputs.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. LlamaFirewall is a modular guardrail framework comprising PromptGuard 2 (a fine-tuned DeBERTa-based jailbreak classifier), AlignmentCheck (a few-shot LLM auditor that inspects an agent's chain-of-thought for goal misalignment), and CodeShield (a Semgrep/regex static analyzer for insecure code). The paper reports evaluations on in-house benchmarks and the external AgentDojo suite, with the headline result that combining PromptGuard 2 and AlignmentCheck reduces AgentDojo attack success rate from 17.6% to 1.75% while keeping utility near baseline. The authors also claim state-of-the-art direct jailbreak detection for PromptGuard 2 and describe AlignmentCheck as the first open-source real-time chain-of-thought auditor for injection defense.
Significance. If the reported numbers are correct, LlamaFirewall would be a practically useful open-source defense layer for LLM agents. The paper's strengths include releasing the code, providing implementation details for each scanner, evaluating on an external benchmark (AgentDojo), and including ablations over guardrail model sizes and prompting variants. However, the two central empirical claims are currently not supported by the evidence as presented: the AgentDojo ASR reductions rely on a reclassification rule that conflates detection with prevention, and the state-of-the-art claim for PromptGuard 2 lacks competitor comparisons on the direct jailbreak benchmark. The framework itself is a meaningful contribution, but the headline numbers need re-analysis before the claims can be accepted.
major comments (4)
- [Section 4.3.1] The reclassification rule stated in Section 4.3.1 is load-bearing for the central ASR claims. The paper says: 'in adversarial runs, if an attack initially succeeds but is later flagged, the trace is reclassified as defended, indicating successful prevention.' In an offline replay of static traces, a flag that occurs after the agent has already executed the injected instruction cannot prevent anything; it only detects the compromise after the fact. Reclassifying such traces as defended conflates detection with prevention and directly inflates the standalone AlignmentCheck ASR (2.89%) and the combined figure (1.75%) reported in Section 4.3.2. Please recompute the ASR counting a trace as a successful attack whenever the injected action was taken before the first flag, and report both 'blocked before action' and 'detected at any point' metrics so the gap is transparent. The example in Appendix C.5 illustrates the concern: the judge flags an action only after the agent has already switched to the attacker's goal.
- [Appendix A.2 / Section 4.1] The claim of 'clear state of the art performance' for PromptGuard 2 on universal jailbreak detection is not supported by the evidence presented. Appendix A.2 states: 'As this dataset is specifically tailored to evaluate inputs that are out-of-distribution to PromptGuard, we don't report competitor metrics on this benchmark.' Without evaluating competitor detectors on the same direct jailbreak benchmark, the SOTA claim cannot be substantiated. The AgentDojo comparisons in Section 4.1 and Appendix B.2 involve indirect prompt injection in agent traces, which is a different task and does not establish superiority on direct jailbreak detection. Please provide competitor results on the same direct jailbreak data or temper the SOTA claim accordingly.
- [Section 4.1 / Appendix B.2] The threshold selection procedure threatens the validity of the PromptGuard versus competitor comparison. The paper selects a threshold for each model to produce a fixed, minimal utility reduction of 3% on the AgentDojo evaluation set, then compares ASR at that operating point. This is tuning to the evaluation benchmark. Different models have different utility-ASR tradeoff curves (as Figure 6 suggests), and the ranking of detectors can change across operating points. Please report the full tradeoff curves for all compared models, state the exact thresholds used, and ideally select thresholds on a held-out validation split rather than on the test set.
- [Appendix C.1.3 / Abstract] The abstract's claim of 'stronger efficacy at preventing indirect injections in general scenarios' is broader than what the mechanism and evaluation can support. Appendix C.1.3 states that 'Agent LLMs often verbalize their intent... enabling the guardrail LLM to identify misaligned behavior more reliably.' This is an assumption about agent behavior, not a demonstrated property. If an agent is deceptive, adversarially trained, or simply does not expose an injected goal in its reasoning trace, AlignmentCheck has no signal. The evaluation covers a limited set of models that do verbalize; generalization beyond that setting is unsupported. Please qualify the abstract's wording to reflect this limitation, or provide evidence that AlignmentCheck works on agents that do not verbalize injected goals.
minor comments (6)
- [Section 3 heading] The heading 'LLamaFirewall' has a capitalization typo; it should read 'LlamaFirewall'.
- [Section 2.3] The phrase 'adds an extra layter of defense' contains a typo: 'layter' should be 'layer'.
- [Appendix C.2 heading] The heading 'Comparison of Effectiveness of diffent Llama Models' contains a typo: 'diffent' should be 'different'.
- [Section 4.3.2] The results table mixes decimal and percentage formats (e.g., .1763 vs 17.6%) within the same table and text; please use a single consistent format.
- [Appendix B.2] The parenthetical 'negligible utility loss ( 1.5%)' appears to have a missing figure or operator; please clarify whether this is approximately 1.5% or some other value.
- [Section 4.3.1] The paper describes the offline evaluation as 'simulat[ing] real-time deployment,' but an offline replay cannot capture the timing-dependent behavior of a live monitor (e.g., whether a flag can arrive before the agent's next action in a real system). This simulation assumption should be explicitly acknowledged as a limitation in Section 5.
Circularity Check
AgentDojo ASR is computed by reclassifying already-succeeded attacks as 'defended' when a scanner flags later in the trace, so the headline 1.75% ASR measures post-hoc detection, not prevention.
-
self definitional
[Section 4.3.1, Experimental Setup]
"Conversely, in adversarial runs, if an attack initially succeeds but is later flagged, the trace is reclassified as defended, indicating successful prevention."
The paper's headline ASR reductions (AlignmentCheck alone 2.89%, combined 1.75%) are computed from offline AgentDojo traces using exactly this rule. 'Successful prevention' is defined as 'a flag appears anywhere later in the trace,' so a trace in which the agent already executed the injected instruction is relabeled as defended the moment a scanner flags a subsequent step. The post-defense ASR therefore equals, by construction, the fraction of adversarial traces never flagged, not the fraction of attacks actually stopped. The Appendix C.5 example confirms the pattern: the alignment judge concludes 'true' only after the agent has already switched to the attacker's goal and taken the action. Under real-time deployment that trace is a success, not a prevention.
-
fitted input called prediction
[Section 4.1, PromptGuard 2 evaluation]
"Because each jailbreak detection model returns a score, we identify a threshold for each model that produces a fixed, minimal utility reduction (3%); the full utility-attack success rate tradeoff is in AppendixB."
PromptGuard's reported ASR is obtained after choosing a threshold on the same AgentDojo evaluation set to hit a pre-specified utility cost. The reported 7.53% ASR is therefore a fitted outcome, not a prediction at a pre-registered operating point; another threshold would give another ASR, and the chosen one is selected using the test data it is then used to score. This is a mild form of fitting the detector to the evaluation set, although it does not apply to AlignmentCheck (whose fixed prompt has no such threshold) and the underlying benchmark remains external.
full rationale
Most of the paper is empirical measurement rather than derivation: AlignmentCheck is a few-shot chain-of-thought auditor evaluated on the external AgentDojo benchmark plus a disclosed in-house benchmark, and CodeShield is a static analyzer. These components do not reduce to their inputs by construction, and the self-citations to CyberSecEval3 and PromptGuard 1 are not load-bearing for the central guardrail claims. The central problem is the Section 4.3.1 reclassification rule, which defines successful prevention as 'flagged at any later point in an offline trace.' Because the AgentDojo evaluation is a static replay, a flag that appears after the injected instruction has been executed cannot prevent anything; the trace was already a successful attack. Under this rule, the post-defense ASR is by construction the complement of the scanner's flag rate in adversarial traces, so the reported 84% and >90% ASR reductions are partially circular: the metric is defined in terms of the detector's own output. A smaller, separate issue is that PromptGuard's threshold is chosen on the same AgentDojo set to deliver a fixed 3% utility loss, making its reported ASR an in-sample fitted value. These issues concern the evaluation definition rather than a mathematical derivation, so the score is partial, not total: the system has independent external benchmarks, open-source code, and a plausible mechanism, but the headline defensive claim is inflated by a self-definitional metric.
Assumptions & free parameters
free parameters (3)
- PromptGuard detection thresholds =
tuned to 3% utility reduction on AgentDojo
- AlignmentCheck guardrail model =
Llama 4 Maverick (17B, 128e)
- Utility-reduction target =
3%
assumptions (3)
- domain assumption Agent LLMs verbalize their intent in chain-of-thought, making misalignment visible to an auditor.
- domain assumption Offline replay of static AgentDojo traces is a valid proxy for real-time agent defense.
- domain assumption Universal jailbreak attacks exhibit lexical regularity that a BERT-scale classifier can learn.
Cite this review
Pith. "Pith review of LlamaFirewall: An open source guardrail system for building secure AI agents." pith.science (2026). https://pith.science/paper/XZ5RJ2FZ
@misc{pith2026250503574,
author = {Pith},
title = {Pith review of: LlamaFirewall: An open source guardrail system for building secure AI agents},
year = {2026},
howpublished = {\url{https://pith.science/paper/XZ5RJ2FZ}},
note = {Machine review of arXiv:2505.03574}
}
read the original abstract
Large language models (LLMs) have evolved from simple chatbots into autonomous agents capable of performing complex tasks such as editing production code, orchestrating workflows, and taking higher-stakes actions based on untrusted inputs like webpages and emails. These capabilities introduce new security risks that existing security measures, such as model fine-tuning or chatbot-focused guardrails, do not fully address. Given the higher stakes and the absence of deterministic solutions to mitigate these risks, there is a critical need for a real-time guardrail monitor to serve as a final layer of defense, and support system level, use case specific safety policy definition and enforcement. We introduce LlamaFirewall, an open-source security focused guardrail framework designed to serve as a final layer of defense against security risks associated with AI Agents. Our framework mitigates risks such as prompt injection, agent misalignment, and insecure code risks through three powerful guardrails: PromptGuard 2, a universal jailbreak detector that demonstrates clear state of the art performance; Agent Alignment Checks, a chain-of-thought auditor that inspects agent reasoning for prompt injection and goal misalignment, which, while still experimental, shows stronger efficacy at preventing indirect injections in general scenarios than previously proposed approaches; and CodeShield, an online static analysis engine that is both fast and extensible, aimed at preventing the generation of insecure or dangerous code by coding agents. Additionally, we include easy-to-use customizable scanners that make it possible for any developer who can write a regular expression or an LLM prompt to quickly update an agent's security guardrails.
Forward citations
Cited by 22 Pith papers
-
Beyond Single-Use Tokens: Durable Authorization State for Replay-Resistant LLM Agent Actions
LLM agents reissuing equivalent actions under fresh tokens cause semantic replay; CapLease shows that durable authorization state, not token representation, prevents duplicate execution.
-
Agent Data Injection Attacks are Realistic Threats to AI Agents
Agent data injection (ADI) forges trusted agent metadata via probabilistic delimiter injection and bypasses defenses built only for instruction injection.
-
DualView: Preventing Indirect Prompt Injection in Personal AI Agents
DualView extends Dual-LLM symbol isolation into the shared user environment via dual Agent/Human views, blocking both immediate and stored IPI at 0% ASR while preserving near-baseline utility.
-
MOSAIC: Knowledge-Guided CLI Command Composition Attack in LLM Coding Agents
Individually benign CLI commands compose via shared OS state into high-success attacks on real LLM coding agents; MOSAIC systematically generates them from CVE/PoC knowledge at 96.59% ASR.
-
AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocations
AttriGuard gates agent tool calls via teacher-forced counterfactual replay under control-attenuated observations, achieving 0% static ASR and single-digit adaptive ASR with modest overhead.
-
Beyond Handcrafted Security: Towards Self-Evolving Defense for LLM Agents
A self-evolving runtime defense for LLM agents, HARD, improves security over static handcrafted defenses by updating context policies and action gates from failure traces.
-
SHE: Trajectory-driven Safety Harness Evolution for LLM Agents
An LLM-agent safety harness can be decomposed into four editable parts and evolved from rollout trajectories, cutting attack success rate from 17.1% to 5.5% on Agent-SafetyBench while improving utility.
-
$S^3$: Improving Agent Safety through Multi-Stage Defense
S3 composes stage-specific safety skills through a guard agent, achieving near-zero attack success on six risk types in its own benchmark while preserving benign task completion.
-
Your Agentic LLMs Secretly Encode Latent Signals of Indirect Prompt-Injection Exposure
Linear probes on pre-generation hidden states detect indirect prompt-injection exposure across six agentic LLMs with 90%+ AUROC, and probe-gated anti-injection reasoning reduces attack success toward zero.
-
IssueTrojanBench: Benchmarking AI Coding Agents Against Malicious Issue Requests
AI coding agents followed malicious instructions embedded in issue-style artifacts in 66.5% of 4,176 test runs.
-
JANUS: Foreseeing Latent Risk for Long-Horizon Agent Safety
A guard trained to anticipate safety-relevant futures from partial trajectories cuts average attack success from 23.0% to 7.1% across four agent-safety benchmarks.
-
Data Leakage Prevention in Agentic Applications via Preemptive Hardening
A build-time pipeline that scans, patches, and validates agentic LLM apps reduced prompt-injection leakage to 0% on most tested apps and by 91% on the hardest stress case.
-
Token-Flow Firewall: Semantic Runtime Auditing for Persistent AI Agents
TokenWall mediates persistent-agent security by auditing source–sink token flows with a local small model and selective large-model escalation, cutting CIK-Bench attack success to 12.5% at low benign latency.
-
Hybrid privacy-aware semantic search: SVD-truncated document geometry and CKKS-encrypted query reranking under a restricted threat model
SVD truncation plus a secret rotation on documents and CKKS on queries preserves sub-second retrieval quality while collapsing off-the-shelf inversion, with document protection failing under known-plaintext Procrustes...
-
Prevalence of Security and Privacy Risk-Inducing Usage of AI-based Conversational Agents
Roughly a third of UK adults use AI chatbots weekly, and among them a substantial minority upload untrusted content, connect bots to other programs, share sensitive data, or attempt jailbreaks.
-
Adversarial Bug Reports as a Security Risk in Language Model-Based Automated Program Repair
Adversarial bug reports induced attacker-desired patches in 90% of trials, while the best tested pre-repair filter caught only 47%, exposing a structural weakness in LLM-based automated program repair.
-
Reliable Weak-to-Strong Monitoring of LLM Agents
Monitor scaffolding, not monitor awareness or omniscience, drives detection reliability, and a hybrid chunked monitor lets weak models supervise strong LLM agents.
-
MCP Safety Training: Learning to Refuse Falsely Benign MCP Exploits using Improved Preference Alignment
The paper introduces TRADE, an online-only MCP attack, and RAG-Pref, a retrieval-based preference alignment method that together with DPO improves strict refusal of falsely benign MCP exploits from 6.7% to 24.1% on average.
-
LLM Agents Should Employ Security Principles
A position paper proposing AgentSandbox, a framework that applies Saltzer-Schroeder security principles to LLM agents and reports large attack-success-rate reductions on AgentDojo.
-
Agentic Web: Weaving the Next Web with AI Agents
A position paper defines the Agentic Web as the next web era and proposes a three-dimensional conceptual framework for understanding and building it.
-
On the Surprising Efficacy of LLMs for Penetration-Testing
A critical review arguing that LLMs are surprisingly effective for penetration testing because the task is largely pattern-matching, while noting serious reliability, safety, and cost barriers to autonomous use.
-
Dark LLMs: The Growing Threat of Unaligned AI Models
The paper is a safety warning essay that asserts a universal jailbreak still works on many commercial LLMs without disclosing any measurement or methodology.
Reference graph
Works this paper leans on
-
[1]
English Set: An out-of-distribution dataset containing English-only adversarial prompts, testing the model’s generalization to new attack types
-
[2]
I’m transferring money because the website instructed me to
Multilingual Set: The same dataset machine-translated into eight additional languages, evaluating the model’s performance across languages. B PromptGuard 2 B.1 Development Methodology To mitigate prompt injection and jailbreaking risks in LLM-powered applications, we had previously developed PromptGuard, a lightweight, low-latency classifier for detecting...
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.