REVIEW 3 major objections 7 minor 54 references
An off-host gateway can make tool-using AI agents safe at the action boundary even when models refuse attacks inconsistently.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-11 06:25 UTC pith:ASMUBHK4
load-bearing objection Solid systems paper: per-message human HMAC plus off-host policy zeros residual tool risk when consulted, with honest ablations and open code; the real limit is exclusive routing, which the author already owns. the 3 major comments →
aiAuthZ: Off-Host, Identity-Bound Authorization for AI Agents
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Model-level refusal is uneven and not ordered by price, so it cannot be the sole defense. Moving identity binding and authorization into a separate trust domain, with per-message HMAC verification and an off-host policy, drives residual tool-layer attack success to 0% for all 15 models while adding at most 0.03 ms of decision latency, and blocks nine of nine in-scope real-incident case studies where an argument-only policy without identity binding blocks only four.
What carries the argument
aiAuthZ: an off-host authorization gateway that binds each user message with an HMAC-SHA256 signature (user id, session, content hash, single-use nonce, timestamp), holds the session's active user to the most recent verified message, evaluates role-based tool allowlists plus argument constraints the agent cannot modify, hash-chains every decision, and issues an HMAC-tagged QR receipt.
Load-bearing premise
Production runtimes must actually send sensitive tools only through the gateway; if they keep built-in shell, file, or web tools enabled, the agent can act without ever consulting it.
What would settle it
Deploy the gateway beside a real agent runtime that still has overlapping built-in tools enabled and check whether a non-owner attack still reaches those tools without a deny; if it does, residual attack success is no longer zero.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that tool-using AI agents act on unverifiable text, so model-level refusal is an unreliable authorization mechanism. Across 15 models and eight attack scenarios adapted from the Agents of Chaos corpus, refusal ranges from 100% to 38% and does not track price. The author presents aiAuthZ, an off-host gateway that (i) binds each user message with a per-message HMAC-SHA256 over identity, session, content hash, single-use nonce, and timestamp, (ii) evaluates role- and argument-level policy the agent cannot read or modify, (iii) appends decisions to a SHA-256 hash-chained audit log with crypto-erasure retention, and (iv) issues HMAC-authenticated QR receipts. Empirically, residual attack success reaching the tool falls to 0% for all 15 models at ≤0.03 ms decision latency; on nine in-scope corpus cases the full design blocks 9/9 versus 4/9 for an argument-only ablation; on AgentDojo banking it blocks all seven attacker-directed tool calls the agents emit (at the cost of one legitimate first-time payment); and receipts achieve 94% mean verification across eight channels with 0/25 wrong-key accepts. The claim is carefully scoped: the gateway does not stop deception, only unauthorized action on calls routed through it.
Significance. If the results hold under the stated threat model, the paper supplies a concrete, deployable composition that closes a large documented class of agent failures—identity and authorization errors—deterministically and model-independently. The contribution is not new cryptography but the granularity and placement of known primitives: per-inbound-message human identity binding off the agent host, coupled to argument-level policy, a tamper-evident audit chain, a survivable receipt, and a credential broker. Strengths that raise the bar for the area include an open implementation and full experiment harness, an explicit ablation isolating per-message identity (9/9 vs 4/9), multi-model evaluation at five temperatures, AgentDojo comparison against spotlighting, receipt robustness against re-encoding and geometric transforms, and unusually candid limitation statements (bypass via built-in tools, composition of permitted calls, symmetric non-repudiation). The work is complementary to content guardrails and in-process confinement rather than a replacement, which is the right framing for systems security.
major comments (3)
- [§2.2, §3.6, §5.9; abstract; Table 2] §2.2 and §3.6 make exclusive routing of sensitive tools a deployment obligation rather than a property the gateway enforces. The headline residual-0% claims (abstract; Table 2; Fig. 4) are correct only when every sensitive call is consulted. §5.9 shows one live MCP deny path and conformance-checker exit codes, but does not stress-test the three mitigations (conformance check, egress-locked profile, credential broker) against the same two runtimes the introduction reports as bypassing the gateway when built-in tools remain enabled. Because exclusive routing is the load-bearing premise for real-world security, the revision should either (a) add a structured bypass evaluation (built-in tools on/off, broker secrets present/absent, egress lock on/off) with measured residual, or (b) systematically restate every residual-0% claim as conditioned on routing, including the abstract.
- [abstract; §5.2; §5.4; Fig. 5] §5.2 correctly notes that residual 0% under a fixed non-owner role and deny-by-default owner-only tools is immediate for the role gate, and isolates identity binding in §5.4. The abstract and introduction still lead with multi-model residual 0% without that qualification, which overstates the novelty of the multi-model experiment relative to the ablation. Revise the abstract and §1.3/§5.2 framing so the multi-model result is presented as uniformity of enforcement across models, and the identity contribution is the primary security claim (as Fig. 5 already supports).
- [§5.5; Table 4] On AgentDojo (§5.5, Table 4), both models already have 0% ASR with no defense, so the suite has little residual for any defense to remove; spotlighting raises ASR to 2/20 on one model, and aiAuthZ costs clean utility 100%→80% via the known-payee allowlist. The seven blocked attacker-directed calls are a useful deterministic signal, but the paper should state more clearly that on this suite the block is due to argument policy (shared principal), not per-message identity, and should discuss when the utility cost is justified. Without that, the AgentDojo result is easy to over-read as a general superiority claim.
minor comments (7)
- [§5.1; Table 2] Table 2 lists model names (e.g., GPT-5.5, Opus 4.8, Fable 5) that will be non-stationary; the footnote archive of identifiers and transcripts is good, but the main text should state the evaluation window more precisely and note that refusal rates are snapshot measurements.
- [§5.1; Table 2] The “atattempt‐ definition (dangerous call in at least one of five temperatures) is a worst-case susceptibility indicator; say so in the table caption as well as the body so readers do not treat residual percentages as per-run probabilities.
- [§5.4; Fig. 5] §5.4 comparison is an ablation of reimplemented policy logic, not a run of the original Open Agent Passport or Agent Identity Protocol code. The text already says this; add a short caveat to the Fig. 5 caption.
- [§1.4; §6.2] §6.2 non-repudiation tradeoff is well stated; a one-sentence note in the abstract or contributions list that HMAC tags are operator-trusted (not third-party non-repudiable) would prevent overclaiming of audit strength.
- [Fig. 1; Fig. 2] Figure 1 and Figure 2 are described but their visual content is not independently checkable in the text dump; ensure production figures label the trust boundary and the hash-chain fields (prev_hash, seq, row_hash) explicitly.
- [Table 1; running header] Minor prose: ‐OW ASP‐ appears with a space in Table 1 and elsewhere; standardize to OWASP. Also ‐aiAuthZ: Off-Host, Identity-BoundaAuthorization‐ in the running header looks like a line-break artifact.
- [§2.1; §7.3] Cite SPIFFE/OAuth token exchange and ETDI earlier when contrasting message-level vs workload/tool-definition identity; the related-work placement is fine, but a forward pointer in §2.1 would help readers.
Circularity Check
No significant circularity: systems/empirical evaluation with acknowledged definitional policy gates and independent ablations.
full rationale
This is a systems paper that builds an off-host authorization gateway and measures it against multi-model attack scenarios, AgentDojo, and Agents of Chaos case studies. There is no fitted free-parameter chain, no uniqueness theorem imported from the authors, and no ansatz smuggled in as a first-principles result. The multi-model residual of 0% under a non-owner role gate is definitional once the policy is fixed (the paper itself states that 'a deny-by-default policy denies a denied tool, which is immediate'), but that number is not presented as a non-definitional prediction derived from data; the independent empirical content is the model-only residual dispersion (38–100% refusal), the identity-binding ablation (9/9 vs 4/9), AgentDojo attacker-call blocks, receipt channel survival, and latency. The single self-citation (Kodathala et al. 2021 on DWT-SVD watermarking) is peripheral to cover-image marking and is not load-bearing for the authorization claims. No circular derivation chain is present.
Axiom & Free-Parameter Ledger
free parameters (4)
- timestamp_window_seconds =
300 (default)
- per_tool_rate_limits_and_write_ceilings =
policy-dependent
- role_tool_allowlists_and_path_url_recipient_constraints =
deny-by-default shipped policy
- HMAC_tag_truncation_for_QR =
128 bits
axioms (6)
- standard math HMAC-SHA256 with secret per-user keys is unforgeable without the key; constant-time compare prevents timing leaks.
- domain assumption Gateway host, policy DB, master encryption key, and shared nonce/rate-limit store are outside the agent's credential set.
- domain assumption Per-user HMAC keys are provisioned out of band and not disclosed to the agent or adversary.
- domain assumption Transport security protects runtime–gateway traffic; multi-process deployments share one nonce store.
- domain assumption Attack scenarios adapted from Agents of Chaos and AgentDojo banking important_instructions are representative of in-scope authorization failures.
- ad hoc to paper Symmetric HMAC audit/receipt tags need not provide third-party non-repudiation against the gateway operator.
invented entities (2)
-
aiAuthZ authorization gateway (composition)
independent evidence
-
HMAC-authenticated QR action receipt
independent evidence
read the original abstract
AI agents issue tool calls on the basis of text they cannot verify, so any party who controls part of the context can forge the appearance of authority. I evaluate 15 contemporary language models against eight attack scenarios derived from a published corpus of real agent incidents and find that refusal varies from 100% down to 38% across fully evaluated models; the most expensive model refused only half of the attacks despite a twentyfold price spread. I present aiAuthZ, an authorization gateway that moves the safety decision off the agent's host. Before a tool call executes, the gateway verifies caller identity with a per-message HMAC-SHA256 signature bound to a single-use nonce and a timestamp window, and it evaluates a role-based and argument-level policy that the agent can neither read nor modify. Every decision joins a SHA-256 hash-chained audit log, and each accepted message yields an HMAC-authenticated QR receipt that achieves 94% mean verification across eight transmission channels, with zero forgeries accepted in 25 wrong-key trials. With the gateway in place, residual attack success falls to 0% for all 15 models at no more than 0.03 ms of added decision latency. On the AgentDojo banking suite, aiAuthZ blocks all seven attacker-directed tool calls the evaluated agents emit, at the cost of one legitimate first-time payment, while a spotlighting baseline allows two injections to succeed. Across nine in-scope case studies from the same incident corpus, aiAuthZ blocks nine of nine, against four of nine for a policy baseline without identity binding. The gateway does not prevent a model from being deceived; it prevents a deceived model from acting beyond the verified user's authority on every call routed through it. The implementation and all experiments are released at https://github.com/Sports-Vision-Inc/aiAuthZ.
Figures
Reference graph
Works this paper leans on
-
[1]
Frontiers in Handwriting Recognition (ICFHR), 2014 14th International Conference on , pages=
Real-time segmentation of on-line handwritten arabic script , author=. Frontiers in Handwriting Recognition (ICFHR), 2014 14th International Conference on , pages=. 2014 , organization=
2014
-
[2]
Soft Computing and Pattern Recognition (SoCPaR), 2014 6th International Conference of , pages=
Fast classification of handwritten on-line Arabic characters , author=. Soft Computing and Pattern Recognition (SoCPaR), 2014 6th International Conference of , pages=. 2014 , organization=
2014
-
[3]
Advanced Data Mining and Applications: 12th International Conference, ADMA 2016, Gold Coast, QLD, Australia, December 12-15, 2016, Proceedings 12 , pages=
Prediction-Based, Prioritized Market-Share Insight Extraction , author=. Advanced Data Mining and Applications: 12th International Conference, ADMA 2016, Gold Coast, QLD, Australia, December 12-15, 2016, Proceedings 12 , pages=. 2016 , organization=
2016
-
[4]
2026 , eprint=
Agents of Chaos , author=. 2026 , eprint=
2026
-
[5]
2026 , eprint=
Before the Tool Call: Deterministic Pre-Action Authorization for Autonomous AI Agents , author=. 2026 , eprint=
2026
- [6]
-
[7]
Not What You've Signed Up For: Compromising Real-World
Greshake, Kai and Abdelnabi, Sahar and Mishra, Shailesh and Endres, Christoph and Holz, Thorsten and Fritz, Mario , booktitle=. Not What You've Signed Up For: Compromising Real-World
-
[8]
Inan, Hakan and Upasani, Kartikeya and Chi, Jianfeng and Rungta, Rashi and Iyer, Krithika and Mao, Yuning and others , year=. Llama Guard:. 2312.06674 , archivePrefix=
-
[9]
Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track , year=
Debenedetti, Edoardo and Zhang, Jie and Balunovi. Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track , year=
-
[10]
2025 , eprint=
Defeating Prompt Injections by Design , author=. 2025 , eprint=
2025
-
[11]
Shi, Tianneng and He, Jingxuan and Wang, Zhun and Li, Hongwei and Wu, Linyu and Guo, Wenbo and Song, Dawn , year=. Progent: Securing. 2504.11703 , archivePrefix=
-
[12]
Bhatt, Manish and Narajala, Vineeth Sai and Habler, Idan , year=. 2506.01333 , archivePrefix=
-
[13]
and Hashimoto, Tatsunori , booktitle=
Ruan, Yangjun and Dong, Honghua and Wang, Andrew and Pitis, Silviu and Zhou, Yongchao and Ba, Jimmy and Dubois, Yann and Maddison, Chris J. and Hashimoto, Tatsunori , booktitle=. Identifying the Risks of
-
[14]
Wu, Yuhao and Roesner, Franziska and Kohno, Tadayoshi and Zhang, Ning and Iqbal, Umar , booktitle=
-
[15]
2025 , eprint=
Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming , author=. 2025 , eprint=
2025
-
[16]
2026 , howpublished=
2026
-
[17]
2024 , howpublished=
Model Context Protocol , author=. 2024 , howpublished=
2024
-
[18]
2025 , howpublished=
2025
-
[19]
Rebedea, Traian and Dinu, Razvan and Sreedhar, Makesh Narsimhan and Parisien, Christopher and Cohen, Jonathan , booktitle=
-
[20]
2020 , month=
Jones, Michael and Nadalin, Anthony and Campbell, Brian and Bradley, John and Mortimore, Chuck , number=. 2020 , month=
2020
-
[21]
2024 , howpublished=
2024
-
[22]
2024 , howpublished=
Coalition for Content Provenance and Authenticity (. 2024 , howpublished=
2024
-
[23]
2019 , eprint=
Robust Invisible Video Watermarking with Attention , author=. 2019 , eprint=
2019
-
[24]
Kodathala, Sai Varun and Mandava, Ajay Kumar and Chowdary, Rakesh , booktitle=. Robust. 2021 , organization=
2021
-
[25]
2021 , howpublished=
invisible-watermark: A Blind Watermark for Digital Images , author=. 2021 , howpublished=
2021
-
[26]
blind-watermark: Blind Watermark Based on
Guo, Fei , year=. blind-watermark: Blind Watermark Based on
-
[27]
AgentDojo : A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents
Edoardo Debenedetti, Jie Zhang, Mislav Balunovi \'c , Luca Beurer-Kellner, Marc Fischer, and Florian Tram \`e r. AgentDojo : A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2024
2024
-
[28]
Maddison, and Tatsunori Hashimoto
Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. Identifying the risks of LM agents with an LM -emulated sandbox. In International Conference on Learning Representations (ICLR), 2024
2024
-
[29]
Not what you've signed up for: Compromising real-world LLM -integrated applications with indirect prompt injection
Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you've signed up for: Compromising real-world LLM -integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security (AISec), 2023
2023
-
[30]
Defeating prompt injections by design, 2025
Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan, Jamie Hayes, Nicholas Carlini, Daniel Fabian, Christoph Kern, Chongyang Shi, Andreas Terzis, and Florian Tram \`e r. Defeating prompt injections by design, 2025
2025
-
[31]
Progent: Securing AI agents with privilege control, 2025
Tianneng Shi, Jingxuan He, Zhun Wang, Hongwei Li, Linyu Wu, Wenbo Guo, and Dawn Song. Progent: Securing AI agents with privilege control, 2025
2025
-
[32]
IsolateGPT : An execution isolation architecture for LLM -based agentic systems
Yuhao Wu, Franziska Roesner, Tadayoshi Kohno, Ning Zhang, and Umar Iqbal. IsolateGPT : An execution isolation architecture for LLM -based agentic systems. In Network and Distributed System Security Symposium (NDSS), 2025
2025
-
[33]
CSA research note: The MCP security crisis
Cloud Security Alliance . CSA research note: The MCP security crisis. https://labs.cloudsecurityalliance.org/research/csa-research-note-mcp-security-crisis-20260504-csa-styled/, 2026. Accessed July 2026
2026
-
[34]
AI agent security hits its reckoning: Prompt injection may be a permanent flaw, not a patchable bug
Tech Times . AI agent security hits its reckoning: Prompt injection may be a permanent flaw, not a patchable bug. https://www.techtimes.com/articles/318361/20260614/ai-agent-security-hits-its-reckoning-prompt-injection-may-permanent-flaw-not-patchable-bug.htm, 2026. Accessed July 2026
2026
-
[35]
AI agent security breaches 2026: Lessons
Beam AI . AI agent security breaches 2026: Lessons. https://beam.ai/agentic-insights/ai-agent-security-breaches-2026-lessons, 2026. Accessed July 2026
2026
-
[36]
OWASP : Prompt injection tops AI security failures
Help Net Security . OWASP : Prompt injection tops AI security failures. https://www.helpnetsecurity.com/2026/06/11/owasp-prompt-injection-ai-security-failures/, 2026. Accessed July 2026
2026
-
[37]
Agents of chaos, 2026
Natalie Shapira, Chris Wendler, Avery Yen, Gabriele Sarti, Koyena Pal, Olivia Floody, Adam Belfki, et al. Agents of chaos, 2026
2026
-
[38]
Llama guard: LLM -based input-output safeguard for human- AI conversations, 2023
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, et al. Llama guard: LLM -based input-output safeguard for human- AI conversations, 2023
2023
-
[39]
Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming, 2025
Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, et al. Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming, 2025
2025
-
[40]
Before the tool call: Deterministic pre-action authorization for autonomous ai agents, 2026
Uchi Uchibeke. Before the tool call: Deterministic pre-action authorization for autonomous ai agents, 2026
2026
-
[41]
AIP : Agent identity protocol for verifiable delegation across MCP and A2A , 2026
Sunil Prakash. AIP : Agent identity protocol for verifiable delegation across MCP and A2A , 2026
2026
-
[42]
ETDI : Mitigating tool squatting and rug pull attacks in model context protocol ( MCP ) by using OAuth -enhanced tool definitions and policy-based access control, 2025
Manish Bhatt, Vineeth Sai Narajala, and Idan Habler. ETDI : Mitigating tool squatting and rug pull attacks in model context protocol ( MCP ) by using OAuth -enhanced tool definitions and policy-based access control, 2025
2025
-
[43]
Model context protocol
Anthropic . Model context protocol. https://modelcontextprotocol.io, 2024. Protocol specification, version 2024-11-05
2024
-
[44]
OWASP top 10 for large language model applications
OWASP Foundation . OWASP top 10 for large language model applications. https://owasp.org/www-project-top-10-for-large-language-model-applications/, 2025
2025
-
[45]
MITRE ATLAS : Adversarial threat landscape for artificial-intelligence systems
MITRE Corporation . MITRE ATLAS : Adversarial threat landscape for artificial-intelligence systems. https://atlas.mitre.org, 2025
2025
-
[46]
Cisco AI security taxonomy, 2025
Cisco Systems . Cisco AI security taxonomy, 2025. Taxonomy of attacks on AI applications and agents
2025
-
[47]
SPIFFE : Secure production identity framework for everyone
Cloud Native Computing Foundation . SPIFFE : Secure production identity framework for everyone. https://spiffe.io, 2024
2024
-
[48]
OAuth 2.0 token exchange
Michael Jones, Anthony Nadalin, Brian Campbell, John Bradley, and Chuck Mortimore. OAuth 2.0 token exchange. Technical Report RFC 8693, Internet Engineering Task Force (IETF), January 2020. https://www.rfc-editor.org/rfc/rfc8693
2020
-
[49]
NeMo guardrails: A toolkit for controllable and safe LLM applications with programmable rails
Traian Rebedea, Razvan Dinu, Makesh Narsimhan Sreedhar, Christopher Parisien, and Jonathan Cohen. NeMo guardrails: A toolkit for controllable and safe LLM applications with programmable rails. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations, 2023
2023
-
[50]
Coalition for content provenance and authenticity ( C2PA ) technical specification
Coalition for Content Provenance and Authenticity . Coalition for content provenance and authenticity ( C2PA ) technical specification. https://c2pa.org, 2024
2024
-
[51]
invisible-watermark: A blind watermark for digital images
ShieldMnt . invisible-watermark: A blind watermark for digital images. https://github.com/ShieldMnt/invisible-watermark, 2021
2021
-
[52]
blind-watermark: Blind watermark based on DWT-DCT-SVD
Fei Guo. blind-watermark: Blind watermark based on DWT-DCT-SVD . https://github.com/guofei9987/blind_watermark, 2020
2020
-
[53]
Robust invisible video watermarking with attention, 2019
Kevin Alex Zhang, Lei Xu, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Robust invisible video watermarking with attention, 2019
2019
-
[54]
Robust DWT-SVD domain image watermarking based on iterative blending
Sai Varun Kodathala, Ajay Kumar Mandava, and Rakesh Chowdary. Robust DWT-SVD domain image watermarking based on iterative blending. In Journal of Physics: Conference Series, volume 2070, page 012111. IOP Publishing, 2021. doi:10.1088/1742-6596/2070/1/012111
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.