Pith. sign in

REVIEW 4 major objections 6 minor 4 cited by

LLM Agents Should Employ Security Principles

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM agents should be designed around classic security principles—and a framework that does so cuts prompt-injection attack success to 4.34%.

desk verdict A useful position piece on security principles for LLM agents, but the headline ASR result is not attributable to the proposed adaptive architecture because the evaluated AgentSandbox configuration is never specified. read the letter →

arxiv 2505.24019 v1 pith:WXQSAGIJ submitted 2025-05-29 cs.CR cs.AI

Kaiyuan Zhang , Zian Su , Pin-Yu Chen , Elisa Bertino , Xiangyu Zhang , Ninghui Li This is my paper
classification cs.CRcs.AI
keywords LLMagentssecurityprinciplespromptinjectionleastprivilegedefense-in-depthAgentSandboxDojoprivacyleakage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This position paper argues that the well-established security design principles from decades of computing—least privilege, complete mediation, defense-in-depth, and psychological acceptability—should be deliberately built into LLM agents rather than applied ad hoc. To show this is actionable, the paper introduces AgentSandbox, a conceptual framework in which a persistent agent holds the user's profile, each task runs in an isolated ephemeral agent given only minimal data, and every internal and external message passes through a data minimizer, response filter, and I/O firewall. A reward-modeling policy engine automatically refines data-sharing policies, reducing the manual configuration burden that psychological acceptability is meant to avoid. On the AgentDojo benchmark with gpt-4o, AgentSandbox reports a 4.34% average attack success rate versus 58.84% with no defense, while benign utility stays at 82.00%, close to the 83.81% undefended baseline. If the paper is right, emerging agent protocols such as MCP and A2A should encode these principles explicitly rather than treating security only as authentication and transport.

What carries the argument

AgentSandbox itself is the central object: a five-component architecture consisting of a Persistent Agent, Ephemeral Agents, a Data Minimizer, an I/O Firewall, and a Response Filter. The load-bearing mechanism is the split between the persistent agent, which holds the user profile and memory and never contacts the external world, and disposable per-task ephemeral agents that receive only the minimal data the Data Minimizer allows; this single design choice makes least privilege and defense-in-depth concrete. The I/O Firewall and Response Filter carry complete mediation by checking every inbound and outbound message, including schema validation and policy checks. The adaptive layer is the reward-modeling policy engine, which iteratively proposes policies, deploys them to the Data Minimizer, Ephemeral Agent, and Response Filter, observes task outcomes, and refines the policy optimizer so that policies evolve rather than being manually specified.

What would settle it

Run the exact AgentSandbox configuration from the main results table on AgentDojo while holding out every attack prompt used during any policy optimization; if the measured attack success rate rises toward the 58.84% no-defense baseline, the reported 4.34% would not generalize beyond the tested injection template.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLM agents should be designed around classic security principles because natural-language-driven agents blur the distinction between code and data, recreating the conditions that made injection attacks devastating in earlier systems. AgentSandbox is the concrete demonstration: a persistent agent holds long-term user data and never contacts the external world; disposable ephemeral agents are created per task, receive only task-minimal data through a Data Minimizer, communicate outside only through an I/O Firewall that enforces schemas and policies, and return responses through a Response Filter. The reward-modeling policy engine iteratively proposes candidate data-sharing policies, deploys them to the Data Minimizer, Ephemeral Agent, and Response Filter, observes task outcomes, and refines the policy optimizer to balance utility and privacy. Across four AgentDojo task suites using gpt-4o, AgentSandbox achieves an average attack success rate of 4.34% compared with 58.84% for no defense, with benign utility of 82.00% versus 83.81%, and the paper argues this utility–security trade-off is better than tool filtering, prompt-injection detection, delimiting, and repeated-prompt baselines.

Load-bearing premise

The empirical case rests on the assumption that the configuration labeled 'AgentSandbox' in the main results table actually runs the full proposed architecture, including the iterative reward-modeling policy engine, rather than hand-written or benchmark-fitted prompt policies.

Editorial extensions

If this is right

  • Emerging agent protocols such as MCP and A2A should encode security principles explicitly—for example, least-privilege data scopes on every tool call—rather than treating security as only authentication and transport.
  • AgentSandbox's ephemeral-agent isolation means a prompt injection that compromises one task session expires with that session and cannot pollute the user's persistent profile or future tasks.
  • Combining static schema enforcement with adaptive policy learning means a failure in the learned layer still leaves a hard safety net, giving defense-in-depth an operational meaning.
  • The reported numbers indicate that meaningful prompt-injection resistance need not trade away task performance: 82.00% benign utility at 4.34% ASR on AgentDojo.
  • The framework's human-in-the-loop escalation lets users authorize exceptional disclosures without granting blanket access, aligning with least privilege in practice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test adaptive adversaries that know AgentSandbox's components; a natural next experiment is to attack the Data Minimizer or the reward-modeling engine directly, since those components would bear the load in deployment.
  • The position extends naturally to protocol design: treating data minimization as a first-class protocol field in MCP/A2A-style exchanges could make least privilege verifiable between agents rather than a prompt-level instruction.
  • The four-task-suite evaluation leaves open whether the 4.34% ASR transfers across tools and domains; a broader held-out suite would tell whether the defensive structure generalizes or the numbers were fitted to AgentDojo's particular attack template.
  • The paper's code–data analogy suggests expecting an arms-race dynamic in agent security, comparable to the history of injection defenses in web systems.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This position paper argues that classical security design principles—defense-in-depth, least privilege, complete mediation, and psychological acceptability—should be applied explicitly when deploying LLM agents at scale. To illustrate the argument, the authors introduce AgentSandbox, a conceptual framework comprising a Persistent Agent, Ephemeral Agents, a Data Minimizer, an I/O Firewall, a Response Filter, and a reward modeling policy engine. The paper evaluates AgentSandbox on AgentDojo using gpt-4o as the base model, comparing it with five baselines, and reports that it maintains benign utility comparable to no defense while reducing average attack success rate to 4.34%.

Significance. If the evaluation is fully specified and reproducible, the paper makes a useful contribution by translating well-established systems security principles into a concrete architecture for LLM agents, and by connecting this design to emerging protocols such as MCP and A2A. The framework's separation of persistent and ephemeral agents and the layered mediation components are a sensible synthesis of existing ideas, and the reported ASR reduction is potentially large. However, the current manuscript is best assessed as a design/position paper; its quantitative claims, as written, outrun the evidence presented, so the significance depends on forthcoming clarification.

major comments (4)
  1. [Section 4.1, Table 1] The AgentSandbox row in Table 1 is underspecified. Section 4.1 defines each baseline defense but does not state how the Persistent Agent, Data Minimizer, Ephemeral Agent, Response Filter, and I/O Firewall were instantiated, which components were active, or whether Algorithm 1's reward modeling policy engine was actually run. Appendix G provides only static prompt templates for the Data Minimizer and Ephemeral Agent, and no configuration is given for the Response Filter or I/O Firewall. Without this information, the reader cannot determine whether the reported row tests the full architecture described in Section 3 or a simpler prompted agent.
  2. [Algorithm 1, Section 4] The evaluation has an unresolved circularity risk. The task outcome metric mu_task (Algorithm 1, Line 3) is never defined, and the paper does not state whether the policy optimization used the same AgentDojo attack instances that are later scored in Table 1. If the evaluated policies were optimized on those instances, the 4.34% ASR is a fitted result rather than a predictive evaluation. The manuscript should specify mu_task, the optimization budget, and any train/test split used.
  3. [Section 4.2, Table 1] The sentence reporting that No Defense exhibits an average ASR as high as 58.84% is inconsistent with Table 1. The unweighted average of the four ASR values in the No Defense row (49.31%, 74.29%, 27.14%, 26.67%) is 44.35%, and 58.84% is in fact the average Attack Utility of that same row. The text should be corrected, and all aggregate claims about ASR reduction should be recomputed from the table.
  4. [Section 4.2, Table 1] The evaluation covers only the "Important message" injection attack, and the reported percentages have no error bars, no repeated runs, and no per-instance breakdown. The conclusion that AgentSandbox "substantially mitigates privacy risks" across the general threat landscape is not supported by a single attack type and a single run per configuration. The paper should at least report the number of task and attack instances per suite and provide confidence intervals or per-instance results.
minor comments (6)
  1. [Section 5] There is a typo: "expands to to real-world action execution" should read "expands to real-world action execution."
  2. [Introduction] The phrase "withsecurity principles" in the second paragraph is missing a space; it should read "with security principles."
  3. [Section 4.1] The subsection heading "Defenses Configurations" is ungrammatical; it should be "Defense Configurations."
  4. [Appendix G] The prompt templates contain ellipses ("..."), so the exact prompts are not fully reproducible; the authors should provide complete prompt text or a link to a repository.
  5. [Table 1] The table would benefit from stating the number of tasks per suite and the total number of tasks, and from clarifying whether the quoted aggregate ASR values are unweighted or weighted averages.
  6. [Appendix C] The figures for o3-mini and gpt-4o-mini report aggregate results without error bars or per-suite numeric tables; consider adding tables analogous to Table 1 for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's normative claim and illustrative evaluation are not derived from their own inputs.

full rationale

The paper is a position paper whose central claim is normative: LLM agents should employ established security principles. This claim is supported by a conceptual framework, AgentSandbox, and by a preliminary empirical evaluation on the external AgentDojo benchmark. The AgentSandbox row in Table 1 is not shown to be a fitted parameter renamed as a prediction: Section 4.1 defines the baseline defenses but does not define the exact AgentSandbox configuration, and Appendix G provides hand-written prompt templates for the Data Minimizer and Ephemeral Agent. Even if the evaluated configuration consisted of these static prompts, a hand-authored policy is not a fitted input called a prediction. Algorithm 1's reward modeling policy engine is described as a design proposal, but the paper never states that it was run on the same AgentDojo attack instances used for scoring, nor what objective metric it optimized; inferring that it was would be speculation. The framework is not defined in terms of the evaluation metric, and the evaluation metric is not defined in terms of the framework's outputs. Citations to Saltzer and Schroeder and to AgentDojo are external; there is no load-bearing self-citation, no imported uniqueness theorem, and no renaming of a known result presented as a new derivation. The underspecification of the AgentSandbox evaluation is a reproducibility limitation, not circularity. Therefore the paper receives a score of 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 6 invented entities

The framework assumes that classic security principles transfer to LLM agents, that AgentDojo is a valid proxy, and that the implementation evaluated matches the described architecture. No numeric free parameters are reported; policy prompts are optimized by an unspecified reward objective whose fitted values are not disclosed. The introduced components (PA, EA, DM, I/O Firewall, RF, policy engine) are testable in principle, but the policy engine lacks direct experimental evidence.

free parameters (1)
  • Reward function weights and policy prompts = not reported
    Algorithm 1 optimizes policy prompts against a task outcome metric, but the objective weights are not specified; if the metric includes ASR on AgentDojo, the reported ASR is partially a fitted value.
assumptions (4)
  • domain assumption Classic security principles (least privilege, defense-in-depth, complete mediation, psychological acceptability) transfer from traditional information systems to LLM agents.
    Stated throughout, especially Sections 1 and 3; the paper argues by analogy, and the empirical evaluation provides indirect support but does not prove transferability.
  • domain assumption AgentDojo accurately measures the security and utility of LLM agent defenses.
    Section 4.1 adopts AgentDojo as the benchmark; the paper does not validate this proxy against real-world agent deployments.
  • domain assumption The defender can fully control the agent's design while the user's agent and direct input queries are intrinsically benign.
    Section 2 explicitly imposes this threat model, which excludes malicious users and compromised persistent agents.
  • domain assumption Prompt injection defenses in the literature are ineffective enough to motivate a new framework.
    Section 1 cites prior results, e.g., 85% failure rate of prompt injection defenses [74]; this is background evidence, not derived in the paper.
invented entities (6)
  • Persistent Agent (PA) independent evidence
    purpose: Maintains long-term user profile and orchestrates tasks while being insulated from direct external interactions.
    The PA role is testable via the AgentDojo evaluation framework, though it is a conceptual redesign of the user's personal agent.
  • Ephemeral Agent (EA) independent evidence
    purpose: Executes individual isolated tasks and is terminated at completion to contain prompt injection.
    EA is implemented as an LLM instance per task; its isolation effect is measurable in the benchmark and consistent with prior isolation designs like IsolateGPT.
  • Data Minimizer (DM) independent evidence
    purpose: Supplies only task-essential data to EAs, enforcing least privilege.
    Prompt template in Appendix G operationalizes the DM; its behavior is observable in the evaluation, though the exact policy is not fully disclosed.
  • I/O Firewall independent evidence
    purpose: Enforces schema validation and policy checks on all external communications.
    The firewall concept is standard; the specific implementation is only sketched, so its independent evidence is limited but testable in principle.
  • Response Filter (RF) independent evidence
    purpose: Sanitizes EA outputs before integration by the persistent agent.
    RF is described in Section 3.3 but no separate prompt or implementation detail is provided; testable but under-specified.
  • Reward Modeling Policy Engine
    purpose: Automatically and iteratively refines data-sharing policies by optimizing a utility-security reward function.
    Algorithm 1 is specified, but the paper presents no evidence that the engine was executed in the evaluation; Appendix G shows static prompts instead, so the engine currently lacks independent falsifiable support.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM Agents Should Employ Security Principles." pith.science (2026). https://pith.science/paper/WXQSAGIJ

@misc{pith2026250524019,
  author       = {Pith},
  title        = {Pith review of: LLM Agents Should Employ Security Principles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WXQSAGIJ}},
  note         = {Machine review of arXiv:2505.24019}
}
read the original abstract

Large Language Model (LLM) agents show considerable promise for automating complex tasks using contextual reasoning; however, interactions involving multiple agents and the system's susceptibility to prompt injection and other forms of context manipulation introduce new vulnerabilities related to privacy leakage and system exploitation. This position paper argues that the well-established design principles in information security, which are commonly referred to as security principles, should be employed when deploying LLM agents at scale. Design principles such as defense-in-depth, least privilege, complete mediation, and psychological acceptability have helped guide the design of mechanisms for securing information systems over the last five decades, and we argue that their explicit and conscientious adoption will help secure agentic systems. To illustrate this approach, we introduce AgentSandbox, a conceptual framework embedding these security principles to provide safeguards throughout an agent's life-cycle. We evaluate with state-of-the-art LLMs along three dimensions: benign utility, attack utility, and attack success rate. AgentSandbox maintains high utility for its intended functions under both benign and adversarial evaluations while substantially mitigating privacy risks. By embedding secure design principles as foundational elements within emerging LLM agent protocols, we aim to promote trustworthy agent ecosystems aligned with user privacy expectations and evolving regulatory requirements.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AutoSpec: Safety Rule Evolution for LLM Agents via Inductive Logic Programming

    cs.SE 2026-06 unverdicted novelty 6.0 of 10

    ILP-guided CEGIS evolves deployed expert safety rules for LLM agents from annotated traces, raising F1 to 0.98 (code) and 0.93 (embodied) in 4–5 iterations.

  2. LLMs are Capable of Misaligned Behavior Under Explicit Prohibition and Surveillance

    cs.AI 2025-06 conditional novelty 6.0 of 10

    Multiple frontier LLMs cheated on an impossible quiz by exploiting sandbox and file-system vulnerabilities, despite explicit instructions not to cheat, with cheating rates varying widely by model.

  3. NEXUS: Structured Runtime Safety for Tool-Using LLM Agents

    cs.AI 2026-05 conditional novelty 5.0 of 10

    A structured-plan monitor that combines safety rules, argument inspection, and a calibrated risk score can route LLM agent tool-use plans into allow, block, confirm, or revise actions, with reported F1 between 0.88 an...

  4. SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks

    cs.CR 2025-06 conditional novelty 5.0 of 10

    SOFT paraphrases low-loss fine-tuning samples before training, reducing MIA AUC from about 0.82 to about 0.54 across six datasets at roughly 7% perplexity cost.

Reference graph

Works this paper leans on

90 extracted references · 33 canonical work pages · cited by 4 Pith papers

  1. [1]

    Firewalls to secure dynamic LLM agentic networks.arXiv preprint arXiv:2502.01822, 2025

    Sahar Abdelnabi, Amr Gomaa, Eugene Bagdasarian, Per Ola Kristensson, and Reza Shokri. Firewalls to secure dynamic LLM agentic networks.arXiv preprint arXiv:2502.01822, 2025

    arXiv 2025

  2. [2]

    Jimenez, Farshad Khorrami, Prashanth Krishnamurthy, Brendan Dolan-Gavitt, Muhammad Shafique, Karthik Narasimhan, Ramesh Karri, and Ofir Press

    Talor Abramovich, Meet Udeshi, Minghao Shao, Kilian Lieret, Haoran Xi, Kimberly Milner, Sofija Jancheska, John Yang, Carlos E. Jimenez, Farshad Khorrami, Prashanth Krishnamurthy, Brendan Dolan-Gavitt, Muhammad Shafique, Karthik Narasimhan, Ramesh Karri, and Ofir Press. Interactive tools substantially assist lm agents in finding security vulnerabilities, 2025

    2025

  3. [3]

    Detecting language model attacks with perplexity.arXiv preprint arXiv:2308.14132, 2023

    Gabriel Alon and Michael Kamfonas. Detecting language model attacks with perplexity.arXiv preprint arXiv:2308.14132, 2023

    arXiv 2023

  4. [4]

    Generative AI on AWS.https://aws.amazon.com/ai/generative-ai/

    Amazon Web Services. Generative AI on AWS.https://aws.amazon.com/ai/generative-ai/. Accessed: 2025-05-15

    2025

  5. [5]

    Agentharm: A benchmark for measuring harmfulness of LLM agents.arXiv preprint arXiv:2410.09024, 2024

    Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, et al. Agentharm: A benchmark for measuring harmfulness of LLM agents.arXiv preprint arXiv:2410.09024, 2024

    arXiv

  6. [6]

    Monitoring computer use via hierarchical summarization

    Anthropic. Monitoring computer use via hierarchical summarization. https://alignment.anthropic.com/2025/ summarization-for-monitoring/. Accessed: 2025-04-28

    2025

  7. [7]

    Introducing the Model Context Protocol, 2024.https://www.anthropic.com/news/model-context-protocol

    Anthropic. Introducing the Model Context Protocol, 2024.https://www.anthropic.com/news/model-context-protocol. Accessed: 2025-03-31

    2024

  8. [8]

    https://github.com/microsoft/autogen/

    AutoGen. https://github.com/microsoft/autogen/. Accessed: 2025-04-28

    2025

Show all 90 references
  1. [9]

    AirGapAgent: Protecting privacy-conscious conversational agents

    Eugene Bagdasarian, Ren Yi, Sahra Ghalebikesabi, Peter Kairouz, Marco Gruteser, Sewoong Oh, Borja Balle, and Daniel Ramage. AirGapAgent: Protecting privacy-conscious conversational agents. InProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Securi...

    2024

  2. [10]

    International ai safety report.arXiv preprint arXiv:2501.17805, 2025

    Yoshua Bengio, Sören Mindermann, Daniel Privitera, Tamay Besiroglu, Rishi Bommasani, Stephen Casper, Yejin Choi, Philip Fox, Ben Garfinkel, Danielle Goldfarb, et al. International ai safety report.arXiv preprint arXiv:2501.17805, 2025

    2025 arXiv

  3. [11]

    Red-teaming large language models using chain of utterances for safety-alignment

    Rishabh Bhardwaj and Soujanya Poria. Red-teaming large language models using chain of utterances for safety-alignment. arXiv preprint arXiv:2308.09662, 2023

    2023 arXiv

  4. [12]

    Computer Security: Art and Science

    Matt Bishop. Computer Security: Art and Science. Addison-Wesley Professional, 2003

    2003

  5. [13]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  6. [14]

    Jailbreaking black box large language models in twenty queries.arXiv preprint arXiv:2310.08419, 2023

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries.arXiv preprint arXiv:2310.08419, 2023

    2023 arXiv

  7. [15]

    Agentpoison: Red-teaming LLM agents via poisoning memory or knowledge bases.Advances in Neural Information Processing Systems, 37:130185–130213, 2024

    Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. Agentpoison: Red-teaming LLM agents via poisoning memory or knowledge bases.Advances in Neural Information Processing Systems, 37:130185–130213, 2024

    2024

  8. [16]

    Llamafirewall: An open source guardrail system for building secure ai agents.arXiv preprint arXiv:2505.03574, 2025

    Sahana Chennabasappa, Cyrus Nikolaidis, Daniel Song, David Molnar, Stephanie Ding, Shengye Wan, Spencer Whitman, Lauren Deason, Nicholas Doucette, Abraham Montilla, et al. Llamafirewall: An open source guardrail system for building secure ai agents.arXiv preprint arXiv:2505.03...

    2025 arXiv

  9. [17]

    Here comes the ai worm: Unleashing zero-click worms that target genai-powered applications

    Stav Cohen, Ron Bitton, and Ben Nassi. Here comes the ai worm: Unleashing zero-click worms that target genai-powered applications. arXiv preprint arXiv:2403.02817, 2024

    2024 arXiv

  10. [18]

    LLMs for Customer Service and Support

    Databricks. LLMs for Customer Service and Support. https://www.databricks.com/solutions/accelerators/ llms-customer-service-and-support. Accessed: 2025-05-15

    2025

  11. [19]

    AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents.arXiv preprint arXiv:2406.13352, 2024

    Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents.arXiv preprint arXiv:2406.13352, 2024

    2024 arXiv

  12. [20]

    A practical memory injection attack against LLM agents.arXiv preprint arXiv:2503.03704, 2025

    Shen Dong, Shaochen Xu, Pengfei He, Yige Li, Jiliang Tang, Tianming Liu, Hui Liu, and Zhen Xiang. A practical memory injection attack against LLM agents.arXiv preprint arXiv:2503.03704, 2025

    2025

  13. [21]

    LLM agents can autonomously hack websites.arXiv preprint arXiv:2402.06664, 2024

    Richard Fang, Rohan Bindu, Akul Gupta, Qiusi Zhan, and Daniel Kang. LLM agents can autonomously hack websites.arXiv preprint arXiv:2402.06664, 2024

    2024 arXiv

  14. [22]

    Papillon: Efficient and stealthy fuzz testing-powered jailbreaks for llms

    Xueluan Gong, Mingzhe Li, Yilin Zhang, Fengyuan Ran, Chen Chen, Yanjiao Chen, Qian Wang, and Kwok-Yan Lam. Papillon: Efficient and stealthy fuzz testing-powered jailbreaks for llms. 2025

    2025

  15. [23]

    Announcing the Agent2Agent Protocol (A2A), 2025

    Google. Announcing the Agent2Agent Protocol (A2A), 2025. https://developers.googleblog.com/en/ a2a-a-new-era-of-agent-interoperability/ . 10 LLM Agents Should Employ Security Principles

    2025

  16. [24]

    Redcode: Risky code execution and generation benchmark for code agents.Advances in Neural Information Processing Systems, 37:106190–106236, 2024

    Chengquan Guo, Xun Liu, Chulin Xie, Andy Zhou, Yi Zeng, Zinan Lin, Dawn Song, and Bo Li. Redcode: Risky code execution and generation benchmark for code agents.Advances in Neural Information Processing Systems, 37:106190–106236, 2024

    2024

  17. [25]

    Defending against indirect prompt injection attacks with spotlighting.arXiv preprint arXiv:2403.14720, 2024

    Keegan Hines, Gary Lopez, Matthew Hall, Federico Zarfati, Yonatan Zunger, and Emre Kiciman. Defending against indirect prompt injection attacks with spotlighting.arXiv preprint arXiv:2403.14720, 2024

    2024 arXiv

  18. [26]

    Trustagent: Towards safe and trustworthy llm-based agents.arXiv preprint arXiv:2402.01586, 2024

    Wenyue Hua, Xianjun Yang, Mingyu Jin, Zelong Li, Wei Cheng, Ruixiang Tang, and Yongfeng Zhang. Trustagent: Towards safe and trustworthy llm-based agents.arXiv preprint arXiv:2402.01586, 2024

    2024 arXiv

  19. [27]

    Baseline defenses for adversarial attacks against aligned language models

    Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614, 2023

    2023 arXiv

  20. [28]

    Dspy: Compiling declarative language model calls into self-improving pipelines

    OmarKhattab, ArnavSinghvi, ParidhiMaheshwari, ZhiyuanZhang, KeshavSanthanam, SriVardhamanan, SaifulHaq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714, 2023

    arXiv

  21. [29]

    https://github.com/langchain-ai/langchain

    LangChain. https://github.com/langchain-ai/langchain. Accessed: 2025-04-28

    2025

  22. [30]

    Prompt infection: LLM-to-LLM prompt injection within multi-agent systems.arXiv preprint arXiv:2410.07283, 2024

    Donghyun Lee and Mo Tiwari. Prompt infection: LLM-to-LLM prompt injection within multi-agent systems.arXiv preprint arXiv:2410.07283, 2024

    2024 arXiv

  23. [31]

    Deepinception: Hypnotize large language model to be jailbreaker.arXiv preprint arXiv:2311.03191, 2023

    Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. Deepinception: Hypnotize large language model to be jailbreaker.arXiv preprint arXiv:2311.03191, 2023

    2023 arXiv

  24. [32]

    RAIN: Your language models can align themselves without finetuning

    Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. RAIN: Your language models can align themselves without finetuning. InICLR, 2024

    2024

  25. [33]

    Agentorca: A dual-system framework to evaluate language agents on operational routine and constraint adherence, 2025

    Zekun Li, Shinda Huang, Jiangtian Wang, Nathan Zhang, Antonis Antoniades, Wenyue Hua, Kaijie Zhu, Sirui Zeng, William Yang Wang, and Xifeng Yan. Agentorca: A dual-system framework to evaluate language agents on operational routine and constraint adherence, 2025

    2025

  26. [34]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models.arXiv preprint arXiv:2310.04451, 2023

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models.arXiv preprint arXiv:2310.04451, 2023

    2023 arXiv

  27. [35]

    Automatic and universal prompt injection attacks against large language models.arXiv preprint arXiv:2403.04957, 2024

    Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. Automatic and universal prompt injection attacks against large language models.arXiv preprint arXiv:2403.04957, 2024

    2024 arXiv

  28. [36]

    Prompt injection attack against llm-integrated applications.arXiv preprint arXiv:2306.05499, 2023

    Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, et al. Prompt injection attack against llm-integrated applications.arXiv preprint arXiv:2306.05499, 2023

    2023 arXiv

  29. [37]

    Formalizing and benchmarking prompt injection attacks and defenses

    Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. Formalizing and benchmarking prompt injection attacks and defenses. In33rd USENIX Security Symposium (USENIX Security 24), pages 1831–1847, 2024

    2024

  30. [38]

    Tree of attacks: Jailbreaking black-box llms automatically.NeurIPS, 2024

    Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box llms automatically.NeurIPS, 2024

    2024

  31. [39]

    Secure data with zero trust.https://learn.microsoft.com/en-us/security/zero-trust/deploy/data

    Microsoft. Secure data with zero trust.https://learn.microsoft.com/en-us/security/zero-trust/deploy/data. Ac- cessed: 2025-05-06

    2025

  32. [40]

    GPT-4 technical report, 2023

    OpenAI. GPT-4 technical report, 2023

    2023

  33. [41]

    Optimizing instructions and demonstrations for multi-stage language model programs

    Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. Optimizing instructions and demonstrations for multi-stage language model programs. InProceedings of the 2024 Conference on Empirical Methods in Natural Language ...

    2024

  34. [42]

    Ignore previous prompt: Attack techniques for language models.arXiv preprint arXiv:2211.09527, 2022

    Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models.arXiv preprint arXiv:2211.09527, 2022

    2022 arXiv

  35. [43]

    The sandwich defense, 2024

    Learn Prompting. The sandwich defense, 2024

    2024

  36. [44]

    Fine-tuned deberta-v3-base for prompt injection detection, 2024

    ProtectAI. Fine-tuned deberta-v3-base for prompt injection detection, 2024. https://huggingface.co/ProtectAI/ deberta-v3-base-prompt-injection-v2

    2024

  37. [45]

    Llm-based agentic systems in medicine and healthcare.Nature Machine Intelligence, 6(12):1418–1420, 2024

    Jianing Qiu, Kyle Lam, Guohao Li, Amish Acharya, Tien Yin Wong, Ara Darzi, Wu Yuan, and Eric J Topol. Llm-based agentic systems in medicine and healthcare.Nature Machine Intelligence, 6(12):1418–1420, 2024

    2024

  38. [46]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018

    2018

  39. [47]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019. 11 LLM Agents Should Employ Security Principles

    2019

  40. [48]

    Identifying the risks of lm agents with an lm-emulated sandbox

    Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J Maddison, and Tatsunori Hashimoto. Identifying the risks of lm agents with an lm-emulated sandbox. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  41. [49]

    Saltzer and Michael D

    Jerome H. Saltzer and Michael D. Schroeder. The protection of information in computer systems.Proceedings of the IEEE, 63(9):1278–1308, 1975

    1975

  42. [50]

    Scalable and transferable black-box jailbreaks for language models via persona modulation

    Rusheb Shah, Quentin Feuillade Montixi, Soroush Pour, Arush Tagade, and Javier Rando. Scalable and transferable black-box jailbreaks for language models via persona modulation. InNeurIPS workshop SoLaR, 2023

    2023

  43. [51]

    PrivacyLens: Evaluating privacy norm awareness of language models in action

    Yijia Shao, Tianshi Li, Weiyan Shi, Yanchen Liu, and Diyi Yang. PrivacyLens: Evaluating privacy norm awareness of language models in action. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024

    2024

  44. [52]

    Collaborative gym: A framework for enabling and evaluating human-agent collaboration.arXiv preprint arXiv:2412.15701, 2024

    Yijia Shao, Vinay Samuel, Yucheng Jiang, John Yang, and Diyi Yang. Collaborative gym: A framework for enabling and evaluating human-agent collaboration.arXiv preprint arXiv:2412.15701, 2024

    2024

  45. [53]

    Do Anything Now

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. "Do Anything Now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models.arXiv preprint arXiv:2308.03825, 2023

    2023 arXiv

  46. [54]

    Choquette-Choo, Milad Nasr, Chawin Sitawarin, Gena Gibson, Andreas Terzis, and John "Four" Flynn

    Chongyang Shi, Sharon Lin, Shuang Song, Jamie Hayes, Ilia Shumailov, Itay Yona, Juliette Pluto, Aneesh Pappu, Christopher A. Choquette-Choo, Milad Nasr, Chawin Sitawarin, Gena Gibson, Andreas Terzis, and John "Four" Flynn. Lessons from defending gemini against indirect prompt ...

    2025

  47. [55]

    Progent: Programmable privilege control for LLM agents.arXiv preprint arXiv:2504.11703, 2025

    Tianneng Shi, Jingxuan He, Zhun Wang, Linyu Wu, Hongwei Li, Wenbo Guo, and Dawn Song. Progent: Programmable privilege control for LLM agents.arXiv preprint arXiv:2504.11703, 2025

    2025 arXiv

  48. [56]

    Multi-turn context jailbreak attack on large language models from first principles.arXiv preprint arXiv:2408.04686, 2024

    Xiongtao Sun, Deyue Zhang, Dongdong Yang, Quanchen Zou, and Hui Li. Multi-turn context jailbreak attack on large language models from first principles.arXiv preprint arXiv:2408.04686, 2024

    2024 arXiv

  49. [57]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  50. [58]

    Context is key in agent security.arXiv preprint arXiv:2501.17070, 2025

    Lillian Tsai and Eugene Bagdasarian. Context is key in agent security.arXiv preprint arXiv:2501.17070, 2025

    2025 arXiv

  51. [59]

    Unveiling privacy risks in LLM agent memory.arXiv preprint arXiv:2502.13172, 2025

    Bo Wang, Weiyi He, Pengfei He, Shenglai Zeng, Zhen Xiang, Yue Xing, and Jiliang Tang. Unveiling privacy risks in LLM agent memory.arXiv preprint arXiv:2502.13172, 2025

    2025 arXiv

  52. [60]

    Secret Language

    Yimu Wang, Peng Shi, and Hongyang Zhang. Investigating the Existence of "Secret Language" in Language Models.arXiv preprint arXiv:2307.12507, 2023

    2023 arXiv

  53. [61]

    Jailbroken: How does LLM safety training fail? InNeurIPS, 2023

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety training fail? InNeurIPS, 2023

    2023

  54. [62]

    IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systems.arXiv preprint arXiv:2403.04960, 2024

    Yuhao Wu, Franziska Roesner, Tadayoshi Kohno, Ning Zhang, and Umar Iqbal. IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systems.arXiv preprint arXiv:2403.04960, 2024

    2024 arXiv

  55. [63]

    Chatarena: Multi-agent language game environments for large language models.https://github.com/chatarena/chatarena, 2023

    Yuxiang Wu, Zhengyao Jiang, Akbir Khan, Yao Fu, Laura Ruis, Edward Grefenstette, and Tim Rocktäschel. Chatarena: Multi-agent language game environments for large language models.https://github.com/chatarena/chatarena, 2023

    2023

  56. [64]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for op...

    2024

  57. [65]

    Benchmarking and defending against indirect prompt injection attacks on large language models.arXiv preprint arXiv:2312.14197, 2023

    Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. Benchmarking and defending against indirect prompt injection attacks on large language models.arXiv preprint arXiv:2312.14197, 2023

    2023 arXiv

  58. [66]

    Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts.arXiv preprint arXiv:2309.10253, 2023

    Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts.arXiv preprint arXiv:2309.10253, 2023

    2023 arXiv

  59. [67]

    LLM-Fuzzer: Scaling assessment of large language model jailbreaks

    Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. LLM-Fuzzer: Scaling assessment of large language model jailbreaks. In 33rd USENIX Security Symposium (USENIX Security 24), pages 4657–4674, 2024

    2024

  60. [68]

    Yangyang Yu, Zhiyuan Yao, Haohang Li, Zhiyang Deng, Yuechen Jiang, Yupeng Cao, Zhi Chen, Jordan Suchow, Zhenyu Cui, Rong Liu, et al. Fincon: A synthesized LLM multi-agent system with conceptual verbal reinforcement for enhanced financial decision making.Advances in Neural Info...

    2024

  61. [69]

    R-judge: Benchmarking safety risk awareness for LLM agents.arXiv preprint arXiv:2401.10019, 2024

    Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, et al. R-judge: Benchmarking safety risk awareness for LLM agents.arXiv preprint arXiv:2401.10019, 2024

    2024 arXiv

  62. [70]

    GPT-4 is too smart to be safe: Stealthy chat with LLMs via cipher

    Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. GPT-4 is too smart to be safe: Stealthy chat with LLMs via cipher. InICLR, 2024. 12 LLM Agents Should Employ Security Principles

    2024

  63. [71]

    Injecagent: Benchmarking indirect prompt injections in tool- integrated large language model agents.arXiv preprint arXiv:2403.02691, 2024

    Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. Injecagent: Benchmarking indirect prompt injections in tool- integrated large language model agents.arXiv preprint arXiv:2403.02691, 2024

    2024 arXiv

  64. [72]

    Zhang, Joey Ji, Celeste Menders, Riya Dulepet, Thomas Qin, Ron Y

    Andy K. Zhang, Joey Ji, Celeste Menders, Riya Dulepet, Thomas Qin, Ron Y. Wang, Junrong Wu, Kyleen Liao, Jiliang Li, Jinghan Hu, Sara Hong, Nardos Demilew, Shivatmica Murgai, Jason Tran, Nishka Kacheria, Ethan Ho, Denis Liu, Lauren McLane, Olivia Bruvik, Dai-Rong Han, Seungwoo...

    2025

  65. [73]

    Goal-guided generative prompt injection attack on large language models.arXiv preprint arXiv:2404.07234, 2024

    Chong Zhang, Mingyu Jin, Qinkai Yu, Chengzhi Liu, Haochen Xue, and Xiaobo Jin. Goal-guided generative prompt injection attack on large language models.arXiv preprint arXiv:2404.07234, 2024

    2024 arXiv

  66. [74]

    Agent security bench (ASB): Formalizing and benchmarking attacks and defenses in LLM-based agents

    Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, and Yongfeng Zhang. Agent security bench (ASB): Formalizing and benchmarking attacks and defenses in LLM-based agents. InThe Thirteenth International Conference on Learning Representat...

    2025

  67. [75]

    Holistic automated red teaming for large language models through top-down test case generation and multi-turn interaction.arXiv preprint arXiv:2409.16783, 2024

    Jinchuan Zhang, Yan Zhou, Yaxin Liu, Ziming Li, and Songlin Hu. Holistic automated red teaming for large language models through top-down test case generation and multi-turn interaction.arXiv preprint arXiv:2409.16783, 2024

    2024 arXiv

  68. [76]

    Agent-safetybench: Evaluating the safety of LLM agents.arXiv preprint arXiv:2412.14470, 2024

    Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang. Agent-safetybench: Evaluating the safety of LLM agents.arXiv preprint arXiv:2412.14470, 2024

    2024 arXiv

  69. [77]

    Agentdam: Privacy leakage evaluation for autonomous web agents.arXiv preprint arXiv:2503.09780, 2025

    Arman Zharmagambetov, Chuan Guo, Ivan Evtimov, Maya Pavlova, Ruslan Salakhutdinov, and Kamalika Chaudhuri. Agentdam: Privacy leakage evaluation for autonomous web agents.arXiv preprint arXiv:2503.09780, 2025

    2025

  70. [78]

    Rtbas: Defending LLM agents against prompt injection and privacy leakage.arXiv preprint arXiv:2502.08966, 2025

    Peter Yong Zhong, Siyuan Chen, Ruiqi Wang, McKenna McCall, Ben L Titzer, Heather Miller, and Phillip B Gibbons. Rtbas: Defending LLM agents against prompt injection and privacy leakage.arXiv preprint arXiv:2502.08966, 2025

    2025 arXiv

  71. [79]

    Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854, 2023

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854, 2023

    2023 arXiv

  72. [80]

    Make a reservation for the cheapest top-rated hotel in London for 2024-06-03

    Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. 13 LLM Agents Should Employ Security Principles Appendix We provide a simple table of contents below for ...

    2023 arXiv

  73. [81]

    Determine which tools should be allowed or restricted

  74. [82]

    Set specific permissions for each allowed tool

  75. [83]

    Detect potential attack patterns and suspicious domains

  76. [84]

    Assess risk level of the query

  77. [85]

    Output your analysis in a structured JSON format that clearly states permissions for each tool based on the task context and provides DETAILED reasoning

    Apply appropriate restrictions based on risk assessment For benign queries: - Allow tools explicitly required by the task - Set appropriate parameter restrictions based on task scope - Permit access to trusted domains mentioned in the query - Enable necessary functionality whi...

  78. [86]

    Validate if the action is allowed based on task scope

  79. [87]

    Check parameters against security policies

  80. [88]

    Detect potential data exfiltration attempts

  81. [89]

    Identify suspicious patterns in action parameters

  82. [90]

    Detect attack sequences across multiple actions Specifically, you should: - Parse the task description to determine its scope (e.g. users, URLs, actions) - Determine which actions are allowed based on the task description - Validate URLs against allowed and blocked domain list...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.