Pith. sign in

REVIEW 4 major objections 4 minor 15 cited by

Design Patterns for Securing LLM Agents against Prompt Injections

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that application-specific LLM agents can be made meaningfully resistant to prompt injection by constraining what a model may do after it has seen untrusted input.

desk verdict A clear, honest catalog of design patterns for prompt-injection resistance; the abstract overclaims 'provable resistance' but the body mostly concedes the guarantee is narrowed attack scope. read the letter →

arxiv 2506.08837 v3 pith:6KPSY3BG submitted 2025-06-10 cs.LG cs.CR

Luca Beurer-Kellner , Beat Buesser , Ana-Maria Creţu , Edoardo Debenedetti , Daniel Dobos , Daniel Fabian , Marc Fischer , David Froelicher
show 6 more authors
Kathrin Grosse Daniel Naeff Ezinwanne Ozoani Andrew Paverd Florian Tramèr Václav Volhejn
This is my paper · ORCID
classification cs.LGcs.CR
keywords promptinjectionLLMagentsdesignpatternssystem-levelsecuritydualpatternplan-then-executecontextminimizationcontrolflowintegrity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Prompt injection is the attack where malicious text hidden in content an LLM agent reads—emails, files, reviews, web pages—convinces the model to do something its user never asked for. The paper's thesis is that for application-specific agents this can be addressed at the system level rather than by making the model itself immune. It proposes six design patterns that share one rule: once an agent has ingested untrusted input, that input must not be able to trigger consequential actions. The patterns restrict the agent to predefined actions, fixed plans, isolated sub-models with constrained outputs, symbolic variables dereferenced by non-LLM code, or contexts stripped of the user's original prompt. If the patterns work as claimed, developers can build useful agents today that survive prompt injections in third-party data, even though general-purpose agents remain out of reach.

What carries the argument

The load-bearing mechanism is the enforced separation between untrusted content and any component that can act. Concretely, the paper treats an LLM agent as a composition of an instruction-following model, a set of tools, and an orchestrator, and inserts a choke point between them: actions are either selected from a closed allowlist, committed to in a plan before untrusted data is read, produced by a tool-less quarantined model whose output passes a format constraint or is dereferenced symbolically by the orchestrator, or executed as code written by the model. The key technical object is the constrained output—a boolean, a number, a category, a regex-validated string, or a symbolic variable—that cannot carry an arbitrary instruction without being detected. That object is what makes the guarantees composable: each pattern can be combined with the others, for example plan-then-execute with dual-LLM processing and context-minimization, to narrow the remaining attack surface.

What would settle it

Take a client that follows pattern 2 or 4, feed it a single crafted calendar event or document containing hidden instructions, and inspect every tool call and sent message. If the injection can change which tool is called or which recipient or parameter is chosen—beyond the already-conceded power to alter free-text bodies and summaries—the 'impossible for untrusted input to trigger consequential actions' claim is refuted. The paper itself shows near-misses of this kind, so the sharper experiment is to attempt the same through a constrained field such as a boolean, number, category, or symbolic variable rather than through open-ended text.

Watch

Extended reading notes

Core claim

The central claim is that the boundary between untrusted content and privileged action is what determines whether an LLM agent is secure, and that this boundary can be drawn in system architecture even when the underlying model remains fully injectable. The paper's six patterns each redraw the boundary differently: the action-selector pattern turns the LLM into a switch that chooses from a hardcoded action list; plan-then-execute fixes the list of tool calls before any untrusted data is read; the map-reduce pattern processes each untrusted document by an isolated sub-agent whose output is checked or aggregated without an LLM; the dual LLM pattern runs a tool-less quarantined model over untrusted text and gives the privileged model only symbolic references; code-then-execute has the LLM write a program that is run by an interpreter; and context-minimization deletes the user prompt or other untrusted text before later generation steps. Applied across ten case studies, from OS assistants and SQL agents to email, booking, hiring, medical, and software-engineering agents, the patterns are claimed to give meaningful resistance to prompt injection while retaining practical utility, with provable resistance understood at the system level, not as a property of the model.

Load-bearing premise

The whole scheme rests on the premise that an LLM's output can be constrained so tightly—to a short action list, a number, a category, or a fixed format—that malicious text hidden in untrusted data cannot smuggle itself through disguised as a legitimate value.

Editorial extensions

If this is right

  • Developers who adopt the patterns should be able to ship application-specific agents whose consequential actions are fixed by a trusted plan, so injected text in files, emails, or reviews cannot add new tool calls.
  • No single pattern covers every threat, so the paper's own recommendation is to combine several patterns, trusting different boundaries against different attackers.
  • Injections can still alter the parameters of an already-approved action, for instance the body of an email or a structured summary, so the guarantee is a narrowed attack surface rather than total safety.
  • Security analysis shifts from trying to make the LLM behave to designing and verifying the small action space around it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the constrained-output premise holds, the same six patterns give a general recipe for any system that mixes privileged instructions with untrusted content, not just LLM agents; the quarantined-submodel idea is a capability-isolation primitive.
  • The case studies are qualitative, but they imply a testable ordering: attacks through third-party content should be far harder under map-reduce and dual-LLM designs than under direct processing, and this could be measured with a standardized injection benchmark that varies only the untrusted-data channel.
  • The abstract's 'provable resistance' is stronger than what the case studies demonstrate, since the paper itself shows constrained outputs can sometimes smuggle attacker-controlled content; a formal proof would require a precise definition of 'consequential action' and of the output constraints.
  • A practical consequence the paper leaves implicit is the usability cost: secure agents become brittle to requests that do not fit the predefined actions, so adoption will hinge on how well developers anticipate the small set of legitimate actions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes six system-level design patterns for building LLM-based agents that are resistant to prompt injection: action-selector, plan-then-execute, map-reduce, dual-LLM, code-then-execute, and context-minimization. It argues that application-specific agents can be made secure against prompt injection by constraining untrusted data so that it cannot trigger consequential actions, even though the underlying LLM remains vulnerable. The patterns are illustrated through ten case studies spanning OS assistants, SQL agents, email/calendar assistants, chatbots, recommender systems, resume screening, medical applications, and software engineering. The paper is written as design guidance rather than as a formal security analysis, and it repeatedly acknowledges residual attack surfaces in the case studies.

Significance. The paper is a valuable consolidation of practical, system-level defensive engineering for LLM agents. Its strengths include clear articulation of a guiding security principle, concrete and composable patterns, and unusually honest case studies that name residual failures (e.g., §4.3.3 email body manipulation, §4.1.2 map-LLM malleability, §4.9.3 tamperable symptom summaries). The paper also transparently attributes several patterns to prior work, notably Willison's dual-LLM pattern and the plan-then-execute/code-then-execute ideas in Debenedetti et al. If reframed as 'meaningful resistance' rather than 'provable resistance,' the paper would be a useful reference for agent designers. As written, however, the headline guarantee is not backed by any formal proof or empirical evaluation, and several documented examples directly contradict the stated 'impossible' principle.

major comments (4)
  1. [Abstract and §3] The abstract promises 'provable resistance to prompt injection,' and §3 states that once untrusted input is ingested, it must be constrained so that 'it is impossible for that input to trigger any consequential actions.' No formal threat model, security definition, or proof is provided anywhere in the paper. More importantly, the paper's own case studies contain concrete counterexamples to this impossibility claim: in §4.3.3 design 3, the quarantined LLM's output, even when stored as a symbolic variable, can carry attacker-controlled text into an outbound email; in §4.1.2, a malicious file can trick the map LLM into a false invoice match, causing the file to be moved or copied. These are consequential actions triggered by untrusted input. The central claim is therefore unsupported as stated, and the paper should either supply a formal argument with explicit assumptions or revise the abstract and Section 3 to claim architectural mitigation or narrowing of attack surface rather than provable impossibility.
  2. [§4.1.2 (map-reduce)] The security analysis of the map-reduce pattern concludes that the design 'resists prompt injection attacks' because a malicious file can at worst be treated as a matching invoice. But the paper's own guiding principle requires that untrusted input be unable to trigger consequential actions. Marking a malicious file as an invoice and moving or copying it to a protected location is itself a consequential action caused by the injection. The comparison to an actual invoice is not a security equivalence: a prompt injection's goal is to make the agent perform an action the user did not request, and the false-match case does exactly that for the affected file. The paper should explicitly acknowledge that this pattern reduces the blast radius to a single file but does not satisfy the 'impossible' standard, and should adjust the security claims accordingly.
  3. [§4.3.3 design 3 (dual LLM)] The dual-LLM symbolic-variable mechanism is presented as a way to prevent untrusted data from reaching the privileged LLM. The paper concedes, however, that the quarantined LLM remains susceptible to prompt injection and can produce attacker-controlled output, and that the orchestrator subsequently dereferences this value into an actual email send. This means the symbolic indirection does not make the value inert; attacker-controlled text reaches a consequential action parameter. The paper's claim that the privileged LLM cannot dereference the value is true but insufficient, because the orchestrator dereferences it. To support any form of provable resistance, the authors would need to define a safety predicate on quarantined-LLM outputs (e.g., a grammar or semantic restriction) and prove that attacker-controlled inputs cannot produce outputs satisfying the predicate in a way that causes harm. The current text does not provide such a predicate.
  4. [§4.9.3 design 3 (structured formatting)] The paper states that enforcing a structured format for the symptom summary 'leaves no room for prompt injection in the summary.' This is misleading: structured decoding constrains syntax, not semantics. The patient's prompt can still manipulate the content of the symptoms summary before it is formatted, as the paper itself notes in the preceding design 2. An attacker can encode a prompt injection in a structured field just as easily as in free text if the downstream consumer interprets that field. The case study does not identify the format or the downstream use sufficiently to support the claim, and no experiment or formal argument is provided. This is another instance where the constrained-output channel is assumed to be tamper-proof without evidence.
minor comments (4)
  1. [§3.1 pattern 6] The description of the context-minimization pattern says the user prompt is removed 'to prevent it from modifying the LLM's response,' but the example only removes the prompt before a post-processing step. The text should clarify whether the pattern removes the prompt before all response generation or only before specific downstream LLM calls.
  2. [Figure 1 caption] The caption says 'The red color represents untrusted data,' but subsequent figures do not consistently use red to mark untrusted data or use the same legend. Consider adding a consistent legend to all figures.
  3. [§4.4.2] The statement that answering 'go see a doctor and ask' could be problematic 'because it is medical advice' is presented without a source. If this is a legal or regulatory claim, it should be cited; if it is an assumption, the text should label it as such.
  4. [References] The closely related work 'Defeating Prompt Injections by Design' (Debenedetti et al., 2025) is cited, but its relationship to the code-then-execute pattern could be discussed more explicitly in Section 3.1 pattern 5, since that pattern is described as being instantiated by that framework.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the design patterns are architectural arguments with transparent provenance; the paper's own case studies concede residual attacks, and the unsupported 'provable resistance' wording is an overclaim rather than a circular derivation.

full rationale

The paper contains no fitted parameters, no quantitative predictions, and no equations whose outputs are defined to be their inputs. The six design patterns are presented as system-architecture constraints, and their security properties are argued from the structure of those constraints (e.g., an action selector chooses from a pre-defined allowlist; a map operation limits each untrusted file to one constrained output; the dual-LLM pattern dereferences symbolic variables only in a non-LLM orchestrator). These are first-principles design arguments, not conclusions derived from the claims themselves. Self-citations appear (Debenedetti et al. 2024/2025, Beurer-Kellner et al. 2024), but the paper explicitly attributes the core pattern ideas to external sources such as Willison's dual-LLM blog post, and the cited works are implementations or benchmarks rather than unverified authorities invoked to close an argument. The code-then-execute pattern says it is 'instantiated in the framework of Debenedetti et al. (2025)', but the pattern's own description is self-contained and the citation is provenance, not the sole load-bearing justification. Notably, the paper itself documents residual attack surface: a malicious file can trick the map LLM into a false match (Sec. 4.1.2), a symbolic variable in the dual-LLM pattern can carry attacker-controlled content into an outbound email (Sec. 4.3.3, design 3), and a structured symptoms summary remains tamperable (Sec. 4.9.3). These concessions undermine the abstract's 'provable resistance' as an unsupported overclaim, but they are the opposite of circularity: the text openly acknowledges that the idealized guarantee is not achieved in those instantiations. The gap between the abstract's promise and the body's caveats is a correctness/support issue, not an equivalence-to-input issue, so no circularity step is identified.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

Four assumptions carry the paper's security claims. None is proven; the first two are widely shared in the community, the third is the operative premise of most patterns, and the fourth is explicitly conceded to fail for user-prompt injections. No invented entities: the orchestrator and symbolic memory are architectural components from Willison's cited dual-LLM proposal, not new postulated objects.

assumptions (4)
  • domain assumption Current LLMs cannot be made robustly resistant to prompt injection at the model level, so security must come from system architecture.
    Section 2 dismisses LLM-level defenses ('they do not provide guarantees') and Section 3 asserts general-purpose agents are unlikely to provide 'meaningful and reliable safety guarantees' with current models; the entire pattern catalog is premised on this.
  • domain assumption The application can identify which inputs are untrusted (third-party data, possibly user prompts) and enforce the separation at the system boundary.
    Every pattern (Sec. 3.1) and case-study threat model (Secs. 4.1-4.10) requires classifying files, emails, reviews, and prompts into trusted and untrusted categories; if a trusted channel carries injected content, the isolation breaks.
  • ad hoc to paper Constrained LLM outputs (fixed action lists, regex formats, structured decoding, symbolic variables) are tamper-proof enough that injected content cannot cause consequential side effects through the output channel.
    This is the operative assumption for patterns 1, 3, 4, 5 and Appendix A's strict data formatting; the paper's own case studies show counterexamples (Sec. 4.3.3 tampered email bodies, Sec. 4.1.2 false map matches, Sec. 4.9.3 tampered summaries).
  • domain assumption The planning or selection LLM, seeing only the user prompt and not third-party data, will not choose an unsafe action or plan.
    Plan-then-execute and code-then-execute (Sec. 3.1 patterns 2 and 5) require that plan formation happens before untrusted data is ingested; Sec. 3.1 pattern 2 explicitly concedes injections in the user prompt itself are not prevented.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Design Patterns for Securing LLM Agents against Prompt Injections." pith.science (2026). https://pith.science/paper/6KPSY3BG

@misc{pith2026250608837,
  author       = {Pith},
  title        = {Pith review of: Design Patterns for Securing LLM Agents against Prompt Injections},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6KPSY3BG}},
  note         = {Machine review of arXiv:2506.08837}
}
read the original abstract

As AI agents powered by Large Language Models (LLMs) become increasingly versatile and capable of addressing a broad spectrum of tasks, ensuring their security has become a critical challenge. Among the most pressing threats are prompt injection attacks, which exploit the agent's resilience on natural language inputs -- an especially dangerous threat when agents are granted tool access or handle sensitive information. In this work, we propose a set of principled design patterns for building AI agents with provable resistance to prompt injection. We systematically analyze these patterns, discuss their trade-offs in terms of utility and security, and illustrate their real-world applicability through a series of case studies.

Figures

Figures reproduced from arXiv: 2506.08837 by the authors.

Figure 1
Figure 1. The action-selector pattern. The red color represents untrusted data. The LLM acts as a translator between a natural language prompt, and a series of pre-defined actions to be executed over untrusted data. 2) The Plan-Then-Execute Pattern. A more permissive approach is to allow feedback from tool outputs back to the agent, but to prevent the tool outputs from influencing the choice of actions taken by the agent. The… view at source ↗
Figure 2
Figure 2. The plan-then-execute pattern. Before processing any untrusted data, the LLM defines a plan consisting of a series of allowed tool calls. A prompt injection cannot force the LLM into executing a tool that is not part of the defined plan. 3) The LLM Map-Reduce Pattern. The plan-then-execute pattern still allows for some adversarial feedback between tool outputs and the agent’s actions (i.e., the agent’s plan of tool … view at source ↗
Figure 3
Figure 3. The LLM map-reduce pattern. Untrusted documents are processed independently, to ensure that a malicious document cannot impact the processing of another document. 4) The Dual LLM Pattern The above LLM-map-reduce pattern is a special case of a more general design pattern proposed by Willison (2023b). This dual LLM pattern combines two types of LLM instances: (1) a privileged LLM that receives instructions and plans a… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The dual LLM pattern. A privileged LLM has access to tools but never processes untrusted data. This LLM can call a quarantined LLM to process untrusted data, but without any tool access. Results from processing untrusted data are stored in a memory that the privileged …
Figure 5
Figure 5. Figure 5: The code-then-execute pattern. The LLM writes a piece of code that can call tools and make calls to other LLMs. The code is then run on untrusted data. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The context-minimization pattern. The user’s prompt informs the actions of the LLM agent (e.g., a call to a specific tool), but is removed from the LLM’s context thereafter to prevent it from modifying the LLM’s response. 4 Case Studies We present ten case studies illu…
Figure 7
Figure 7. Figure 7: Simplified architecture of advanced SQL agents with access to multiple databases and Python interpreters for data analysis and visualization. 4.2.1 Description This SQL Agent answers questions based on data in SQL databases. A typical simplified architecture diagram is…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Prismata: Confining Cross-Site Prompt Injection in Web Agents

    cs.CR 2026-07 conditional novelty 7.5 of 10

    Prismata cuts web-agent prompt-injection attack success from 85.5% to 0.7% via Biba-inspired DOM trust labeling and mechanical least-privilege confinement without site annotations.

  2. Beyond Self-Resolution: Settlement Factorization for Robust Natural Language Mechanism

    cs.GT 2026-07 accept novelty 7.5 of 10

    Settlement factorization is a normal form: every mechanism admits a ghost-reference label with leakage ε within factor two of optimal, and truthful margins degrade by at most the tight constant 2Lε.

  3. Agent Data Injection Attacks are Realistic Threats to AI Agents

    cs.CR 2026-07 accept novelty 7.0 of 10

    Agent data injection (ADI) forges trusted agent metadata via probabilistic delimiter injection and bypasses defenses built only for instruction injection.

  4. AISPA: User-Centric System Prompt Auditing for Large Language Model Applications

    cs.AI 2026-07 conditional novelty 6.5 of 10

    A human-in-the-loop audit of system prompts from 88 commercial AI products finds protective instructions nearly universal yet incomplete, with ~40% of products containing at least one user-harmful directive.

  5. Are You Still the Agent I Authorized? Earned Authority under a Fixed Ceiling for Evolving Agents

    cs.AI 2026-07 conditional novelty 6.5 of 10

    Evolving agents may change active authority only beneath an immutable user-issued effect ceiling, and a transition envelope decides whether the old grant survives mutation at all.

  6. How Context Attribution Handles What the Model Already Knows

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Context attribution methods cannot disentangle in-context from in-weight knowledge and assign unfaithful scores under overlap; new metrics and WMDP-Cyber++ quantify the failure.

  7. Agent Security Needs Redefinition through a Holistic Framework

    cs.CR 2026-07 conditional novelty 6.0 of 10

    Agent security should be redefined around four contextual authorization properties instead of the content of the action performed.

  8. IssueTrojanBench: Benchmarking AI Coding Agents Against Malicious Issue Requests

    cs.CR 2026-07 conditional novelty 6.0 of 10

    AI coding agents followed malicious instructions embedded in issue-style artifacts in 66.5% of 4,176 test runs.

  9. Specifying the Delegated-Autonomy Boundary: Requirements Engineering for Agentic AI

    cs.SE 2026-07 conditional novelty 6.0 of 10

    A requirements-engineering framework that justifies when agentic AI is warranted (AJR) and specifies graduated autonomy, memory, coordination, assurance, and evolution policies (ADP).

  10. Prevalence of Security and Privacy Risk-Inducing Usage of AI-based Conversational Agents

    cs.CR 2025-10 conditional novelty 6.0 of 10

    Roughly a third of UK adults use AI chatbots weekly, and among them a substantial minority upload untrusted content, connect bots to other programs, share sensitive data, or attempt jailbreaks.

  11. When Compression Becomes an Attack Surface: Black-Box Attacks on Prompt-Compressed LLM Agents

    cs.CR 2025-10 reject novelty 6.0 of 10

    The paper claims prompt compression is a new attack surface, but the abstract's COMA attack never appears in the body and the body's SoftCom requires white-box access.

  12. Bridging AI and Software Security: A Comparative Vulnerability Assessment of LLM Agent Deployment Paradigms

    cs.CR 2025-07 conditional novelty 6.0 of 10

    Function Calling and MCP architectures show distinct vulnerability patterns, with chained attacks succeeding 91-96% of the time in both.

  13. MedMKEB: A Comprehensive Knowledge Editing Benchmark for Medical Multimodal Large Language Models

    cs.AI 2025-08 unverdicted novelty 5.0 of 10

    MedMKEB is a medical multimodal knowledge-editing benchmark with four task types, on which existing editing methods underperform according to the authors.

  14. Adversarial Prompting Framework for AI Safety Assessment

    cs.CR 2026-07 reject novelty 4.0 of 10

    An adversarial prompt testing framework with a five-level attack taxonomy and a composite harmfulness score is proposed; the paper claims encoded prompts bypass safety filters most often.

  15. Prompt Injection 2.0: Hybrid AI Threats

    cs.CR 2025-07 reject novelty 2.0 of 10

    A structured taxonomy of hybrid prompt injection attacks shows how XSS, CSRF, and SQL injection vectors converge with LLM manipulation to bypass traditional controls.

Reference graph

Works this paper leans on

20 extracted references · 5 canonical work pages · cited by 15 Pith papers

  1. [3]

    Benjamin Cohen-Wang, Harshay Shah, Kristian Georgiev, and Aleksander Madry

    URL https://arxiv.org/ abs/2402.06363. Benjamin Cohen-Wang, Harshay Shah, Kristian Georgiev, and Aleksander Madry. Contextcite: Attributing model generation to context. In NeurIPS,

    arXiv

  2. [4]

    Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan, Jamie Hayes, Nicholas Carlini, Daniel Fabian, Christoph Kern, Chongyang Shi, Andreas Terzis, and Florian Tram `er

    URL https://arxiv.org/abs/2406.13352. Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan, Jamie Hayes, Nicholas Carlini, Daniel Fabian, Christoph Kern, Chongyang Shi, Andreas Terzis, and Florian Tram `er. Defeating prompt injections by design,

    arXiv

  3. [6]

    ASCII Smuggler Tool: Crafting Invisible Text and Decoding Hidden Codes

    Embrace The Red. ASCII Smuggler Tool: Crafting Invisible Text and Decoding Hidden Codes. https://embracethered.com/blog/posts/2024/ hiding-and-finding-text-with-unicode-tags/ , 2024a. Blog post – Posted on Jan 14,

    work page 2024

  4. [7]

    GitHub Copilot Chat: From Prompt Injection to Data Exfiltration (Copirate)

    Embrace The Red. GitHub Copilot Chat: From Prompt Injection to Data Exfiltration (Copirate). https://embracethered.com/blog/posts/2024/ github-copilot-chat-prompt-injection-data-exfiltration/ , 2024b. Blog post – Posted on Jun 14,

    work page 2024

  5. [8]

    Gupta, Niloofar Mireshghallah, Taylor Berg- Kirkpatrick, and Earlence Fernandes

    Xiaohan Fu, Zihan Wang, Shuheng Li, Rajesh K. Gupta, Niloofar Mireshghallah, Taylor Berg- Kirkpatrick, and Earlence Fernandes. Misusing tools in large language models with visual adversarial examples. CoRR, abs/2310.03185,

    arXiv

  6. [9]

    I genuinely believe prompt engineering is the highest-leverage skill someone can learn in 2022, Sep 2022a

    Goodside. I genuinely believe prompt engineering is the highest-leverage skill someone can learn in 2022, Sep 2022a. URL https://x.com/goodside/status/1569128808308957185. Tweet. Riley Goodside. Exploiting GPT-3 prompts with malicious inputs that order the model to ignore its previous directions. https://x.com/goodside/status/1569128808308957185, 2022b. D...

    arXiv 2022

  7. [12]

    Shoaib Ahmed Siddiqui, Radhika Gaonkar, Boris K¨opf, David Krueger, Andrew Paverd, Ahmed Salem, Shruti Tople, Lukas Wutschitz, Menglin Xia, and Santiago Zanella-B´eguelin

    URL https://arxiv.org/abs/2406.00199. Shoaib Ahmed Siddiqui, Radhika Gaonkar, Boris K¨opf, David Krueger, Andrew Paverd, Ahmed Salem, Shruti Tople, Lukas Wutschitz, Menglin Xia, and Santiago Zanella-B´eguelin. Permis- sive information-flow analysis for large language models,

    arXiv

  8. [13]

    Joseph Spracklen, Raveen Wijewickrama, A H

    URL https://arxiv.org/ abs/2410.03055. Joseph Spracklen, Raveen Wijewickrama, A H. M. Nazmus Sakib, Anindya Maiti, and Murtuza Jadliwala. We have a package for you! A comprehensive analysis of package hallucinations by code generating llms. ArXiv preprint,

    arXiv

Show all 20 references
  1. [14]

    Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus

    URL https://arxiv.org/abs/2406.10279. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks,

    arXiv

  2. [16]

    Prompt injection attacks against GPT-3

    Simon Willison. Prompt injection attacks against GPT-3. https://simonwillison.net/2022/ Sep/12/prompt-injection/,

    2022

  3. [17]

    Delimiters won’t save you from prompt injection

    Simon Willison. Delimiters won’t save you from prompt injection. https://simonwillison. net/2023/May/11/delimiters-wont-save-you/ , 2023a. 30 Simon Willison. The Dual LLM Pattern for Building AI Assistants That Can Resist Prompt Injection. https://simonwillison.net/2023/Apr/25...

    2023

  4. [18]

    John Yang, Carlos E

    URL https://arxiv.org/abs/2306.02224. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineer- ing. In NeurIPS,

    arXiv

  5. [19]

    Traditional

    URL https://arxiv.org/abs/2403.06833. A Best Practices for LLM Agent Security In conjunction with the system-level design patterns presented in Section 3, there are some general best practices that, ideally, are always considered when designing an AI agent. These are related t...

    2024 arXiv

  6. [2008]

    Usability is crucial, as in the worst case, feedback is perceived as annoying, and provided information will not be read by the rater (Micallef et al., 2017)

    should be avoided. Usability is crucial, as in the worst case, feedback is perceived as annoying, and provided information will not be read by the rater (Micallef et al., 2017). Orthogonally, the intersection of provided suggestions and human judgment is complex, with experts ...

    2017

  7. [2014]

    Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel

    URL https://arxiv.org/abs/1312.6199. Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The instruction hierarchy: Training LLMs to prioritize privileged instructions,

    arXiv

  8. [2017]

    URL https://doi.org/10.1145/3152771.3156139

    ISBN 9781450353793. URL https://doi.org/10.1145/3152771.3156139. Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis. In NeurIPS,

  9. [2022]

    ProtectAI.com

    URL https://arxiv.org/abs/2211.09527. ProtectAI.com. Fine-tuned deberta-v3-base for prompt injection detection,

    arXiv

  10. [2023]

    Get my drift? catching llm task drift with activation deltas, 2025a

    Sahar Abdelnabi, Aideen Fay, Giovanni Cherubin, Ahmed Salem, Mario Fritz, and Andrew Paverd. Get my drift? catching llm task drift with activation deltas, 2025a. URL https: //arxiv.org/abs/2406.00799. Sahar Abdelnabi, Amr Gomaa, Eugene Bagdasarian, Per Ola Kristensson, and Rez...

    arXiv

  11. [2024]

    Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Martin Vechev

    URL https://arxiv.org/abs/2405.05175. Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. AI agents with formal security guarantees. In ICML 2024 Next Generation of AI Safety Workshop,

    2024 arXiv

  12. [2025]

    Embrace The Red

    URL https://arxiv.org/abs/2503.18813. Embrace The Red. Hacking Google Bard: From Prompt Injection to Data Exfiltration. https: //embracethered.com/blog/posts/2023/google-bard-data-exfiltration/ ,

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.