Pith. sign in

REVIEW 4 major objections 4 minor 5 cited by

Red-Teaming Coding Agents from a Tool-Invocation Perspective: An Empirical Security Assessment

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that Tool Invocation Prompts—the prompt text that teaches a coding agent what tools exist, how to call them, and how to read their output—are a systemic, overlooked attack surface, and that a two-channel injection (malicio

desk verdict Useful red-team finding with a real abstract-vs-body credibility problem: the body supports RCE/DoS via MCP tool manipulation, but ToolLeak and the 19/25-pair numbers appear only in the abstract. read the letter →

arxiv 2509.05755 v6 pith:O3I442MW submitted 2025-09-06 cs.CR cs.AI

Yuchong Xie , Mingyu Luo , Zesen Liu , Zhixiang Zhang , Kaikai Zhang , Yu Liu , Zongjie Li , Ping Chen
show 2 more authors
Shuai Wang Dongdong She
This is my paper · ORCID
classification cs.CRcs.AI
keywords toolinvocationpromptinjectioncodingagentsremotecodeexecutiondenialofserviceModelContextProtocolLLMagentsecurityred-teaming
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that Tool Invocation Prompts (TIPs)—the prompt fragments that tell a coding agent what tools exist, how to invoke them, and how to interpret tool output—form a systemic attack surface that prior research has not isolated. It shows that a malicious external tool can smuggle instructions into an agent through its tool description and its return values, steering the victim's agent to execute arbitrary shell commands (remote code execution) or to emit output the client can no longer parse (denial of service). In tests across IDE, CLI, and chat-based clients including Cursor, Claude Code, Copilot, Windsurf, Cline, Trae, and CherryStudio, the two-channel attack achieved RCE on every tested agent–LLM pair that exposes a command-execution tool, even Claude Code, whose separate guard model flagged the command but was overridden. The paper argues TIPs deserve first-class security treatment and that current defenses—guard classifiers and self-reflection prompts—remain insufficient.

What carries the argument

Tool Invocation Prompt (TIP): all prompt components governing tool invocation—tool descriptions, invocation schemas, security rules, and tool returns. Because agents concatenate tool descriptions into the system prompt and feed tool returns back into context, adversarial text in either channel is treated as trusted instruction. The carrying attack is RCE-2: the description channel plants a fake 'initialization' step and the return channel reinforces and delivers the payload, defeating agents that resist direct injection. The DoS variant exploits strict parser-schema enforcement by inducing malformed tool calls the client cannot parse.

What would settle it

Run the paper's abstracted RCE-2 template against a fresh Cursor install (GPT-5) where every MCP tool registration requires explicit user approval and the client rejects tool descriptions or returns that deviate from their declared schema; if curl|bash still executes, the attack operates even under stricter installation practices, and if it stops, the reported 0.9 ASR depends on the unmeasured practice of installing unvetted tool servers.

Watch

Extended reading notes

Core claim

Central claim: Tool Invocation Prompts (TIPs)—the prompt text defining what tools exist, how to call them, how to read results—are a critical, unprotected attack surface in coding agents. With an attacker who deploys a malicious external tool (MCP server), a two-channel injection (tool description plus tool return) hijacks agent behavior: DoS via parser-rejected output; RCE via a fake initialization routine in the description plus a shell command in the return. Across IDE, CLI, and chat clients, DoS hit six of seven, RCE-1 hit IDE agents, RCE-2 hit every command-executing client, including Claude Code, where a guard model flagged but was overridden. Conclusion: TIPs need first-class security

Load-bearing premise

The attacker must first get the victim's agent to load a custom malicious tool (an MCP server) with a poisoned description; the paper states this as its threat model but does not measure how often such untrusted tool servers are actually installed, leaving real-world prevalence unquantified.

Editorial extensions

If this is right

  • Any agent that concatenates externally supplied tool descriptions into its system prompt is a candidate victim; the paper observed some form of attack on all seven tested clients.
  • Tool returns must be treated as untrusted input equal to tool descriptions: RCE-2 succeeded where RCE-1 failed, including on Claude Code.
  • Guard models and self-reflection directives, as currently deployed, do not stop the attacks; the paper recommends layered defenses, adaptive filtering, consensus or requerying, and provenance-based trust signals.
  • Newer, more aligned backends reduce but do not eliminate exploitability; the paper still achieved RCE on GPT-5 and Claude Sonnet 4 in the tested clients.
  • Defenses should treat TIPs as first-class components: schema validation of tool descriptions and returns, and binding security checks, are directions the paper's findings support.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the measured attack success depends on the threat-model step of deploying a malicious MCP server; the authors do not quantify how often developers install unvetted tools, so real-world incidence is an open question a telemetry study could answer.
  • Editorial inference: the DoS mechanism is cheaper and simpler than RCE and does not depend on the agent having command-execution capability, so availability attacks may be the more likely practical threat even where code execution is blocked.
  • Editorial inference: the same two-channel pattern likely transfers to non-coding agents that feed tool output back into context (web browsing, email assistants, database queries), because the mechanism is about prompt trust, not about code editors specifically.
  • Editorial inference: a straightforward untested mitigation suggested by the Claude Code case is to make guard-model verdicts binding rather than advisory; enforcing the guard's 'unsafe' decision at the client layer would likely blunt RCE-2.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper defines Tool-Invocation Prompts (TIPs) as the prompt components governing tool invocation in LLM-based agentic systems and argues that TIPs are a high-impact, understudied attack surface. It proposes a TIP Exploitation Workflow (TEW) consisting of prompt stealing, TIP vulnerability analysis, and TIP hijacking, and instantiates three attacks: format-based DoS, direct tool-description injection (RCE-1), and two-channel injection via tool descriptions plus tool returns (RCE-2). The authors evaluate seven real MCP-enabled agents (CherryStudio, Cline, Cursor, Trae, WindSurf, Copilot, Claude Code) across several LLM backends, report attack success rates and token overhead, present case studies on Cursor, Claude Code, and Cline, and test two defenses (Llama-Prompt-Guard-2-22M and self-reflection directives). The central claim is that TIPs expose a systemic attack surface enabling DoS and RCE across IDE, CLI, and chat-based systems, even on recent backends such as GPT-5 and Claude Sonnet 4.

Significance. If the empirical claims hold, the paper would be a useful first systematic assessment of tool-invocation security in MCP-enabled coding agents. The two-channel RCE-2 construction and the case study showing that Claude Code's separate guard model is only advisory are concrete and plausible contributions. The authors also release an open-source assessment platform and prompt designs, which is commendable. However, the paper's strongest conclusion—'systemic attack surface' across major systems—depends on an unvalidated threat-model assumption (the victim installs a malicious MCP server) and on an ASR methodology that measures hand-tuned prompt search rather than a reproducible attack success rate. The current evidence supports the weaker claim that the tested agents are vulnerable to at least one TIP-based attack under a favorable deployment assumption.

major comments (4)
  1. [§3.2, §6.1, §6.3] The threat model assumes as capability 4 that 'the attacker can design the external tool with a malicious tool description and deploy it in the agentic system.' In every case study, the attack begins after the authors 'registered it with Cursor via the MCP server' (or equivalent). The paper provides no evidence about whether registration requires explicit user approval, whether MCP servers are vetted or signed, or how often users install third-party MCP servers. The conclusion's 'systemic attack surface across widely used IDE, CLI, and chat-based systems' therefore overreaches: the demonstrated vulnerability is conditional on an attacker-controlled MCP server being installed, which is a social-engineering precondition. This is load-bearing because all RCE and DoS results depend on it. I recommend scoping all central claims to this precondition and either adding data on real installation
  2. [§5.1 Metrics, Table 2] The ASR is defined as the proportion of ten independent manual attempts, but the same section states that 'the exact prompts are not fixed and may be arbitrarily modified as long as the intended effect is achieved.' Combined with the described early-stopping heuristic, ASR is not a reproducible probability of attack success; it measures whether the experimenter could find a working prompt within a small manual budget. There are no confidence intervals, no fixed prompt sets, and no control for the adaptivity. Consequently, quantitative backend comparisons in RQ2 (e.g., 'gpt-5 generally exhibits reduced rates' and 'claude-sonnet-4 achieves higher RCE success rates compared to claude-3.7-sonnet') are not supported. I suggest reporting success@k, fixed/randomized prompt templates, or CIs, and avoiding statements about backend ordering unless the evaluation protocol is made statistically mean
  3. [Table 1 vs. Table 2, Abstract] Table 1 lists '✓' as 'the effect was consistently observed,' but Table 2 reports many zero or near-zero ASRs for the same agent–backend pairs. For example, Cline RCE-2 is 0.0 on grok-4 and 0.0 on gpt-5; Trae RCE-1 is at most 0.2; Copilot DoS is 0.0 on several backends. The abstract claims 'we obtain RCE on every tested agent-LLM pair,' which is contradicted by the many zeros in Table 2. Either Table 1 refers to existence across some backend and should be presented as such, or the abstract/conclusion must be aligned with the per-backend results. This inconsistency obscures the paper's actual empirical claim.
  4. [Abstract, §4.2, §5] The paper claims discovery of 'a general vulnerability, ToolLeak' and states in the abstract that prompt-stealing 'leakage' succeeds on every agent using Claude and Grok backends. However, no evaluation protocol, table, or metric for prompt-stealing success appears in the main body; Phase 1 is described only as iterative refinement with an early-stopping heuristic. Because prompt stealing is the stated prerequisite for the later attacks, this central enabling step is unmeasured. Please add quantitative leakage results or explicitly present prompt stealing as a motivating technique rather than a claimed discovered vulnerability.
minor comments (4)
  1. [§6.3, Figure 6] The section title and text state that the DoS case uses gemini-2.5-pro, while Figure 6's caption says grok-4. Please correct the inconsistency.
  2. [Table 2 and Table 5] Backend naming is inconsistent: 'claude-sonnet-4' vs 'claude-4' and 'claude-3.7-sonnet' vs 'claude-3.7.' Standardize model names across tables and text.
  3. [§5.2] The text says CherryStudio and other chatbox agents cannot achieve RCE 'owing to the lack of built-in command execution tools,' yet the abstract claims RCE on every tested agent-LLM pair. At minimum, clarify that the chatbox category is only vulnerable to DoS, not RCE.
  4. [Appendix A-D and §1] The appendix explicitly says it abstracts prompts and does not reveal complete exploit strings, while the contributions section says the platform and prompt designs are open-sourced. Clarify what exactly is released and how reviewers/readers can reproduce the case studies from the abstracted modules.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical demonstrations of tool-invocation prompt injection, not derivations from fitted inputs or self-citations.

full rationale

The paper makes no parameterized derivation and fits no model. Its central claims are empirical red-teaming results: with a malicious MCP tool registered (the threat model stated in Sec. 3.2, capability 4), the authors observe whether real agents leak prompts, fail parsing, or execute injected shell commands. The RCE outcome is not 'predicted' from a fitted parameter; it is measured by whether the agent executes the command that the injected tool description/return requests. That is the standard operational definition of a command-injection vulnerability, and observing it is not circular. The DoS outcome is likewise a measured parser failure induced by malformed tool-call formats. The paper's attacks are constructed inputs, but the finding is about the deployed systems' behavior under those inputs, which is an empirical property. The only notable weakness is the threat-model assumption that the attacker can cause the victim to load a malicious MCP server; this limits real-world prevalence but is explicitly stated and is a scope assumption, not a circular reduction. There are no self-citations used as load-bearing evidence, no imported uniqueness theorems, and no fitted parameters renamed as predictions. The work is self-contained as an empirical assessment; its claimed contribution of being the 'first systematic assessment of TIP security' is a novelty claim, not a derivation.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper is an empirical attack study and introduces no fitted numeric parameters. The key untested premises are the threat model (malicious tool deployment) and the manual trial methodology. The central numbers come from n=10 manual runs with adaptively modified prompts.

assumptions (3)
  • domain assumption The attacker can deploy a custom malicious tool (MCP server) into the victim's agentic system.
    Threat model, Section 3.2 (iv): 'The attacker can design the external tool with a malicious tool description and deploy it in the agentic system.' The realism of this scenario is assumed, not measured.
  • domain assumption The target agents' tool-invocation processes are based on MCP and thus use tool descriptions and returns as prompt components.
    Section 5.1: 'the TIP processes in these LLM-based agentic systems in current implementations are based on MCP. Therefore, we conduct our subsequent experiments with a target of MCP-enabled TIPs.' This limits the scope to MCP-enabled configurations.
  • domain assumption Manual success judgment of ten attempts per condition provides an unbiased estimate of attack success.
    Section 5.1 Metrics: ASR is the proportion of successful attempts among ten manual attempts; no automated harness, randomization, or inter-rater assessment is described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Red-Teaming Coding Agents from a Tool-Invocation Perspective: An Empirical Security Assessment." pith.science (2026). https://pith.science/paper/O3I442MW

@misc{pith2026250905755,
  author       = {Pith},
  title        = {Pith review of: Red-Teaming Coding Agents from a Tool-Invocation Perspective: An Empirical Security Assessment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O3I442MW}},
  note         = {Machine review of arXiv:2509.05755}
}
read the original abstract

Coding agents powered by large language models are becoming central modules of modern IDEs, helping users perform complex tasks by invoking tools. While powerful, tool invocation opens a substantial attack surface. Prior work has demonstrated attacks against general-purpose and domain-specific agents, but none have focused on the security risks of tool invocation in coding agents. To fill this gap, we conduct the first systematic red-teaming of six popular real-world coding agents: Cursor, Claude Code, Copilot, Windsurf, Cline, and Trae. Our red-teaming proceeds in two phases. In Phase 1, we perform prompt leakage reconnaissance to recover system prompts. We discover a general vulnerability, ToolLeak, which allows malicious prompt exfiltration through benign argument retrieval during tool invocation. In Phase 2, we hijack the agent's tool-invocation behavior using a novel two-channel prompt injection in the tool description and return values, achieving remote code execution (RCE). We adaptively construct payloads using security information leaked in Phase 1. In emulation across five backends, our method outperforms baselines on Claude-Sonnet-4, Claude-Sonnet-4.5, Grok-4, and GPT-5. On real agents, our approach succeeds on 19 of 25 agent-LLM pairs, achieving leakage on every agent using Claude and Grok backends. For tool-invocation hijacking, we obtain RCE on every tested agent-LLM pair, with our two-channel method delivering the highest success rate. We provide case studies on Cursor and Claude Code, analyze security guardrails of external and built-in tools, and conclude with practical defense recommendations.

Figures

Figures reproduced from arXiv: 2509.05755 by the authors.

Figure 1
Figure 1. Simplified example of TIP in a popular LLM-based agentic system, coding IDE Cline. The [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. MCP-specific example of tool-agnostic defensive TIP. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Execution workflow of the agentic system, highlighting potential attack channels and defense [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The RCE-2 attack workflow on Cursor, backed by gpt-5. This case demonstrates a multi￾channel attack where a malicious tool description and a corresponding malicious tool return work in tandem to manipulate the LLM. The injected description estab￾lishes a deceptive two-…
Figure 5
Figure 5. Figure 5: The RCE attack flow on Claude Code with claude-sonnet-4 as the backend LLM. The diagram illustrates the attack bypassing the system’s defense agent; black text represents benign context, red text denotes malicious context, with the underlined italicized curl|bash comma…
Figure 6
Figure 6. Figure 6: Illustration of a Denial of Service (DoS) attack on the Cline agent with [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. When the Manual Lies: A Realistic Benchmark to Evaluate MCP Poisoning Attacks for LLM Agents

    cs.CR 2026-05 unverdicted novelty 7.0 of 10

    Introduces MCP-TDP benchmark showing near-100% attack success on models like GPT-4o for tool description poisoning and proposes reactive self-correction defense.

  2. Rewriting the Response Path: Silent Tampering and Provider-Signed Defense in BYOK LLM Agents

    cs.CR 2026-05 unverdicted novelty 7.0 of 10

    A malicious relay can strategically rewrite aligned LLM outputs in BYOK agent architectures to achieve up to 99.1% attack success on benchmarks like AgentDojo and ASB.

  3. Where Is the Cost of Third-Party API Routers in Agentic Software Development?

    cs.SE 2026-07 conditional novelty 6.5 of 10

    Router-side response tampering yields 0% defense success on Claude Code, Codex, Cursor, and OpenCode; whitelist and LLM review only partially restore control.

  4. When Compression Becomes an Attack Surface: Black-Box Attacks on Prompt-Compressed LLM Agents

    cs.CR 2025-10 reject novelty 6.0 of 10

    The paper claims prompt compression is a new attack surface, but the abstract's COMA attack never appears in the body and the body's SoftCom requires white-box access.

  5. Verifiable Manifest Signing and Transparency Enforcement for Secure MCP-Based LLM Pipelines

    cs.CR 2026-01 reject novelty 3.0 of 10

    The paper proposes a security layer that signs, freshness-checks, policy-validates, and Merkle-logs MCP tool-use manifests before LLM execution, reporting near-linear scaling and low overhead.

Reference graph

Works this paper leans on

49 extracted references · 30 canonical work pages · cited by 5 Pith papers

  1. [1]

    https://jfrog.com/blog/p rompt-injection-attack-code-execution-in-vanna-ai-cve-2024-5565/, 2024

    Analyzing a prompt injection code execution in vanna.ai (cve-2024-5565). https://jfrog.com/blog/p rompt-injection-attack-code-execution-in-vanna-ai-cve-2024-5565/, 2024

    work page 2024

  2. [2]

    https://thehackernews.com/2024 /06/prompt-injection-flaw-in-vanna-ai.html, 2024

    Prompt injection flaw in vanna ai exposes databases to rce attacks. https://thehackernews.com/2024 /06/prompt-injection-flaw-in-vanna-ai.html, 2024

    work page 2024

  3. [3]

    https://www.cherry-ai.com/, 2025

    Cherry studio — the all-in-one ai assistant. https://www.cherry-ai.com/, 2025. Accessed 2025-08-26

    work page 2025

  4. [4]

    https://docs.anthropic.com/en/docs/claude-code/overview , 2025

    Claude code — overview. https://docs.anthropic.com/en/docs/claude-code/overview , 2025. Accessed 2025-08-26

    work page 2025

  5. [5]

    https://docs.github.com/en/copil ot/using-github-copilot/copilot-chat, 2025

    Github copilot chat — documentation (vs code / github docs). https://docs.github.com/en/copil ot/using-github-copilot/copilot-chat, 2025. Accessed 2025-08-26

    work page 2025

  6. [6]

    https://owasp.org/www-proje ct-top-10-for-large-language-model-applications/, 2025

    Owasp top 10 for large language model applications (latest overview). https://owasp.org/www-proje ct-top-10-for-large-language-model-applications/, 2025. Accessed 2025-08-26

    work page 2025

  7. [7]

    Accessed 2025-08-26

    Trae ide — agent documentation.https://docs.trae.ai/ide/agent, 2025. Accessed 2025-08-26

    work page 2025

  8. [8]

    Accessed 2025-08-26

    Windsurf editor — ai agent-powered ide.https://windsurf.com/editor, 2025. Accessed 2025-08-26

    work page 2025

Show all 49 references
  1. [9]

    Anthropic tool use and function calling

    Anthropic. Anthropic tool use and function calling. https://docs.anthropic.com/claude/docs/too l-use, 2024. Accessed 2025-08-19

    2024

  2. [10]

    Model context protocol.https://docs.anthropic.com/docs/mcp/, 2024

    Anthropic. Model context protocol.https://docs.anthropic.com/docs/mcp/, 2024

    2024

  3. [11]

    Here comes the ai worm: Unleashing zero-click worms that target genai-powered applications.arXiv preprint arXiv:2403.02817, 2024

    Stav Cohen, Ron Bitton, and Ben Nassi. Here comes the ai worm: Unleashing zero-click worms that target genai-powered applications.arXiv preprint arXiv:2403.02817, 2024

    2024 arXiv

  4. [12]

    Accessed 2025-08-26

    Cursor — the ai code editor.https://cursor.com/, 2025. Accessed 2025-08-26

    2025

  5. [13]

    Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents.Advances in Neural Information Processing Systems, 37:82895–82920, 2024

    Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tram` er. Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents.Advances in Neural Information Processing Systems, 37:82895–82920, 2024

    2024

  6. [14]

    The llama 3 herd of models.arXiv preprints arXiv:2407.21783, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprints arXiv:2407.21783, 2024

    2024 arXiv

  7. [15]

    Conversational prompt engineering.arXiv preprint arXiv:2408.04560, 2024

    Liat Ein-Dor, Orith Toledo-Ronen, Artem Spector, Shai Gretz, Lena Dankin, Alon Halfon, Yoav Katz, and Noam Slonim. Conversational prompt engineering.arXiv preprint arXiv:2408.04560, 2024

    2024 arXiv

  8. [16]

    Wasp: Benchmarking web agent security against prompt injection attacks.arXiv preprint arXiv:2504.18575, 2025

    Ivan Evtimov, Arman Zharmagambetov, Aaron Grattafiori, Chuan Guo, and Kamalika Chaudhuri. Wasp: Benchmarking web agent security against prompt injection attacks.arXiv preprint arXiv:2504.18575, 2025

    2025 arXiv

  9. [17]

    An empirical categorization of prompting techniques for large language models: A practitioner’s guide.arXiv preprint arXiv:2402.14837, 2024

    Oluwole Fagbohun, Rachel M Harrison, and Anton Dereventsov. An empirical categorization of prompting techniques for large language models: A practitioner’s guide.arXiv preprint arXiv:2402.14837, 2024

    2024 arXiv

  10. [18]

    What is prompt engineering? https://cloud.google.com/discover/what-is-promp t-engineering?hl=en#what-is-prompt-engineering, 2025

    Google Cloud. What is prompt engineering? https://cloud.google.com/discover/what-is-promp t-engineering?hl=en#what-is-prompt-engineering, 2025. Accessed: 2025-08-27. 19

    2025

  11. [19]

    Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. InProceedings of the 16th ACM workshop on artificial intelligenc...

    2023

  12. [20]

    Llama guard: Llm-based input-output safeguard for human-ai conversations.arXiv preprint arXiv:2312.06674, 2023

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations.arXiv preprint arXiv:2312.06674, 2023

    2023 arXiv

  13. [21]

    Promptshield: Deploy- able detection for prompt injection attacks

    Dennis Jacob, Hend Alzahrani, Zhanhao Hu, Basel Alomair, and David Wagner. Promptshield: Deploy- able detection for prompt injection attacks. InProceedings of the Fifteenth ACM Conference on Data and Application Security and Privacy, pages 341–352, 2024

    2024

  14. [22]

    Kpmg ai quarterly pulse survey: From agent experimentation to rapid scale and deployment

    KPMG. Kpmg ai quarterly pulse survey: From agent experimentation to rapid scale and deployment. https://kpmg.com/us/en/articles/2025/ai-quarterly-pulse-survey.html . Accessed: 2025-08- 27

    2025

  15. [23]

    How to use chat models to call tools

    LangChain. How to use chat models to call tools. https://python.langchain.com/docs/how_to/too l_calling/, 2024. Accessed: 2025-08-27

    2024

  16. [24]

    Eia: Environmental injection attack on generalist web agents for privacy leakage.arXiv:2409.11295, 2024

    Zeyi Liao, Lingbo Mo, Chejian Xu, et al. Eia: Environmental injection attack on generalist web agents for privacy leakage.arXiv:2409.11295, 2024

    2024 arXiv

  17. [25]

    Self-reflection makes large language models safer, less biased, and ideologically neutral.arXiv preprint arXiv:2406.10400, 2024

    Fengyuan Liu, Nouar AlDahoul, Gregory Eady, Yasir Zaki, and Talal Rahwan. Self-reflection makes large language models safer, less biased, and ideologically neutral.arXiv preprint arXiv:2406.10400, 2024

    2024 arXiv

  18. [26]

    Demystifying rce vulnerabilities in llm-integrated apps

    Tong Liu, Zizhuang Deng, Guozhu Meng, Yuekang Li, and Kai Chen. Demystifying rce vulnerabilities in llm-integrated apps. InProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 1716–1730, 2024

    2024

  19. [27]

    What do you want? user-centric prompt generation for text-to-image synthesis via multi-turn guidance.arXiv preprint arXiv:2408.12910, 2024

    Yilun Liu, Minggui He, Feiyu Yao, Yuhe Ji, Shimin Tao, Jingzhou Du, Duan Li, Jian Gao, Li Zhang, Hao Yang, et al. What do you want? user-centric prompt generation for text-to-image synthesis via multi-turn guidance.arXiv preprint arXiv:2408.12910, 2024

    2024

  20. [28]

    Formalizing and bench- marking prompt injection attacks and defenses

    Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. Formalizing and bench- marking prompt injection attacks and defenses. In33rd USENIX Security Symposium (USENIX Security 24), pages 1831–1847, 2024

    2024

  21. [29]

    Llm in-context recall is prompt dependent.arXiv preprint arXiv:2404.08865, 2024

    Daniel Machlab and Rick Battle. Llm in-context recall is prompt dependent.arXiv preprint arXiv:2404.08865, 2024

    2024 arXiv

  22. [30]

    Large language models know your contextual search intent: A prompting framework for conversational search.arXiv preprint arXiv:2303.06573, 2023

    Kelong Mao, Zhicheng Dou, Fengran Mo, Jiewen Hou, Haonan Chen, and Hongjin Qian. Large language models know your contextual search intent: A prompting framework for conversational search.arXiv preprint arXiv:2303.06573, 2023

    2023 arXiv

  23. [31]

    Llama-prompt-guard-2-22m

    Meta Llama. Llama-prompt-guard-2-22m. https://huggingface.co/meta-llama/Llama-Prompt-Gua rd-2-22M, 2025. Accessed: 2025-08-27

    2025

  24. [32]

    Augmented language models: a survey.arXiv preprint arXiv:2302.07842, 2023

    Gr´ egoire Mialon, Roberto Dess ` ı, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozi` ere, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. Augmented language models: a survey.arXiv preprint arXiv:2302.07842, 2023

    2023 arXiv

  25. [33]

    A closer look at system prompt robustness.arXiv preprint arXiv:2502.12197, 2025

    Norman Mu, Jonathan Lu, Michael Lavery, and David Wagner. A closer look at system prompt robustness.arXiv preprint arXiv:2502.12197, 2025

    2025 arXiv

  26. [34]

    Position is power: System prompts as a mechanism of bias in large language models (llms)

    Anna Neumann, Elisabeth Kirsten, Muhammad Bilal Zafar, and Jatinder Singh. Position is power: System prompts as a mechanism of bias in large language models (llms). InProceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, pages 573–598, 2025

    2025

  27. [35]

    What is agentic ai? https://blogs.nvidia.com/blog/what-is-agentic-ai/ , 2024

    Nvida. What is agentic ai? https://blogs.nvidia.com/blog/what-is-agentic-ai/ , 2024. Accessed: 2025-08-27. 20

    2024

  28. [36]

    Function Calling with LLMs

    OpenAI. Function Calling with LLMs. https://www.promptingguide.ai/applications/function_ calling, 2024. Accessed: 2025-08-27

    2024

  29. [37]

    Openai assistants and tool use documentation

    OpenAI. Openai assistants and tool use documentation. https://platform.openai.com/docs/assis tants/overview, 2024. Accessed 2025-08-19

    2024

  30. [38]

    Pwc’s ai agent survey

    PwC. Pwc’s ai agent survey. https://www.pwc.com/us/en/tech-effect/ai-analytics/ai-agent-s urvey.html, 2025. Accessed: 2025-08-27

    2025

  31. [39]

    Tool learning with large language models: A survey.Frontiers of Computer Science, 19(8):198343, 2025

    Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. Tool learning with large language models: A survey.Frontiers of Computer Science, 19(8):198343, 2025

    2025

  32. [40]

    Toolformer: Language models can teach themselves to use tools.Advances in Neural Information Processing Systems, 36:68539–68551, 2023

    Timo Schick, Jane Dwivedi-Yu, Roberto Dess ` ı, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools.Advances in Neural Information Processing Systems, 36:68539–68551, 2023

    2023

  33. [41]

    Cline — ai coding, open source and uncompromised

    Cline Team. Cline — ai coding, open source and uncompromised. https://cline.bot/, 2025. Accessed 2025-08-26

    2025

  34. [42]

    Advweb: Controllable black-box attacks on vlm-powered web agents.arXiv preprint arXiv:2410.17401, 2024

    Chejian Xu, Mintong Kang, Jiawei Zhang, Zeyi Liao, Lingbo Mo, Mengqi Yuan, Huan Sun, and Bo Li. Advweb: Controllable black-box attacks on vlm-powered web agents.arXiv preprint arXiv:2410.17401, 2024

    2024 arXiv

  35. [43]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  36. [44]

    Adaptive attacks break defenses against indirect prompt injection attacks on llm agents.arXiv preprint arXiv:2503.00061, 2025

    Qiusi Zhan, Richard Fang, Henil Shalin Panchal, and Daniel Kang. Adaptive attacks break defenses against indirect prompt injection attacks on llm agents.arXiv preprint arXiv:2503.00061, 2025

    2025 arXiv

  37. [45]

    Injecagent: Benchmarking indirect prompt injections in tool-integrated llm agents

    Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. Injecagent: Benchmarking indirect prompt injections in tool-integrated llm agents. InFindings of the Association for Computational Linguistics (ACL), 2024

    2024

  38. [46]

    Sprig: Improving large language model performance by system prompt optimization.arXiv preprint arXiv:2410.14826, 2024

    Lechen Zhang, Tolga Ergen, Lajanugen Logeswaran, Moontae Lee, and David Jurgens. Sprig: Improving large language model performance by system prompt optimization.arXiv preprint arXiv:2410.14826, 2024

    2024 arXiv

  39. [47]

    a helpful assistant

    Mingqian Zheng, Jiaxin Pei, Lajanugen Logeswaran, Moontae Lee, and David Jurgens. When “a helpful assistant” is not really helpful: Personas in system prompts do not improve performances of large language models. InFindings of the Association for Computational Linguistics: EMN...

    2024

  40. [48]

    Context-faithful prompting for large language models.arXiv preprint arXiv:2303.11315, 2023

    Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. Context-faithful prompting for large language models.arXiv preprint arXiv:2303.11315, 2023

    2023 arXiv

  41. [49]

    ¡” and “¿

    Kaijie Zhu, Xianjun Yang, Jindong Wang, Wenbo Guo, and William Yang Wang. Melon: Provable defense against indirect prompt injection attacks in ai agents.arXiv preprint arXiv:2502.05174, 2025. 21 Appendix Due to ethical considerations, we cannot display the complete code of the...

    2025 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.