Pith. sign in

REVIEW 3 major objections 6 minor 6 cited by

Towards Action Hijacking of Large Language Model-based Agent

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AI2 shows that RAG-based LLM applications can be hijacked into producing harmful action plans using only semantically harmless prompts, by extracting action-aware knowledge from the application's own database and reassembling it.

desk verdict The ROP-style action-hijacking threat model is novel and plausible, but the 84.30% ASR headline is a per-stage number, not an end-to-end success rate. read the letter →

arxiv 2412.10807 v2 pith:DLESDCJJ submitted 2024-12-14 cs.CR

Yuyang Zhang , Kangjie Chen , Jiaxin Gao , Ronghao Cui , Run Wang , Lina Wang , Tianwei Zhang This is my paper · ORCID
classification cs.CR
keywords LLMagentsactionhijackingretrieval-augmentedgenerationpromptinjectionjailbreakadversarialpromptsknowledgeextractionreturn-orientedprogramming
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that retrieval-augmented generation (RAG) makes LLM-based applications vulnerable to a new class of attack, AI2 (AI Application Injection), in which the attacker never writes a harmful word. Instead, the attacker first extracts 'action-aware knowledge' from the application's own database by querying it with carefully crafted prompts, then sends a 'hijacking prompt' whose adversarially optimized tokens steer the retriever to fetch that knowledge; the LLM then assembles the retrieved text with the prompt into a harmful, executable action plan. The authors report an average attack success rate of 84.30%, a best of 99.70%, and bypass rates of 92.7% against common safety filters and 59.45% against dedicated defenses, across code generation, Text2SQL, and medical-assistant applications. If correct, this shows that a RAG application's own knowledge base can be weaponized without any privileged access, and that defenses which only screen the user prompt for banned content or semantics are insufficient.

What carries the argument

The load-bearing mechanism is treating the application's database as a store of return-oriented-programming (ROP) 'gadgets'. Just as ROP chains existing code snippets instead of injecting new code, AI2 chains existing text snippets (action-aware knowledge) that the retriever can be induced to fetch. The machinery has three pieces: (i) an extraction prompt that gets the RAG pipeline to leak a chunk; (ii) adversarial token optimization in the retriever's embedding space (FGSM, I-FGSM, PGD, C&W) so that the hijacking prompt, though semantically harmless to filters, is mapped near the target knowledge in the retriever's latent space; and (iii) an assembly instruction that tells the LLM to combine the retrieved chunk with the prompt into the intended action. The paper also explains the underlying reason this works: the retriever's latent-space geometry separates knowledge categories that the safety filter's own latent space does not, so adversarial tokens can align with the retriever while remaining innocuous to the filter.

What would settle it

Run AI2 against a production-style RAG agent that paraphrases or summarizes retrieved chunks before passing them to the LLM; if the knowledge-extraction success rate collapses and previously successful hijacking prompts no longer assemble harmful actions, the attack's dependency on verbatim leakage of database content is confirmed. Similarly, using a target retriever whose architecture and training data have no public counterpart should leave the transfer success rate near the paper's 24.69% baseline.

Watch

Extended reading notes

Core claim

The central discovery is that an attack pipeline of three phases—knowledge extraction, hijacking-prompt generation, and application hijacking—can compel an LLM-based agent to produce harmful action plans. Knowledge extraction generates a candidate set of action-aware prompts via an Action2NL module with an 'attack memory' of feedback rules and logs, then selects the optimal prompt $\tilde{p}$ by maximizing a relevance-plus-novelty reward, and finally builds the extraction prompt $p_E = \tilde{p} \oplus p_o \oplus p_e$ that makes the model output retrieved content verbatim. Hijacking-prompt generation then uses the extracted knowledge $K_t$ as a 'gadget': an assembly instruction (such as 'do the same operation as first Knowledge on all artists whose id less than 5000') is combined with optimizable tokens $p_a$, which are tuned by gradient-based methods against a shadow retriever so that the prompt's embedding lands near the target knowledge. When submitted, the application retrieves $K_t$ and the LLM reassembles it with the prompt to reconstruct the harmful operation, e.g., a DELETE statement, without the prompt itself ever containing forbidden terms. The paper demonstrates this across six LLMs, two retrievers, and three real-world application types, including commercial platforms.

Load-bearing premise

The attack assumes that the victim application will, through its normal API, return retrieved database content verbatim or nearly verbatim in response to the knowledge-extraction prompts, and that the attacker's shadow retriever approximates the target retriever closely enough; if the chunks are not echoed back, or the retriever is unknown, there are no 'gadgets' to chain.

Editorial extensions

If this is right

  • If AI2 is correct, keyword-based and semantic safety filters on user prompts are not sufficient to protect RAG-based agents, because the harmful instruction is reconstructed inside the model from benign-looking prompt text plus retrieved knowledge.
  • The application's own knowledge base becomes an attack surface: any user who can query the API can extract action-aware knowledge and use it as assembly gadgets, even without write access.
  • Transferability of the attack is strongly tied to the attacker's ability to approximate the target retriever; the reported success drops from 100% when the retriever is known to 24.69% for a completely unknown retriever.
  • Gradient-based adversarial token optimization (PGD, FGSM, I-FGSM) is substantially more effective than optimization-based methods (C&W) in text embedding spaces, reaching up to 100% for retrieving target knowledge on some settings.
  • Real-world agent platforms, including commercial ones, are also affected, with an average overall attack success rate of 91.44% on widely used frameworks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that chaining several extraction-and-assembly rounds through the agent's short-term memory could let an attacker compose complex harmful actions that no single database chunk describes, at the cost of more queries and possibly lower stealth.
  • The paper's dependency on verbatim retrieval suggests a cheap defense: have the application paraphrase or filter retrieved chunks before they reach the LLM, so the attacker's 'gadgets' never appear intact.
  • The reported latent-space mismatch between retriever and filter implies that aligning the two models—for example, running the same safety classifier on both the prompt and the retrieved context—would close the gap the attack exploits, although the paper does not test this.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes AI², a black-box attack against LLM-based applications that use retrieval-augmented generation (RAG). The attack proceeds in three stages: (1) extracting action-aware knowledge from the application's database by querying it with crafted prompts (Section 4.2); (2) generating a 'hijacking prompt' composed of an assembly instruction and adversarially optimized tokens that induce the retriever to fetch the target knowledge (Section 4.3, Algorithm 1); and (3) causing the LLM to assemble the retrieved knowledge with the instruction into a harmful executable action plan. The authors evaluate on code generators (BugNET), Text2SQL agents (MultiSQL), and medical assistants (Huatuo) across six LLMs and two retrievers, and claim an average ASR of 84.30% (best 99.70%), with bypass rates of 92.7% against common safety filters and 59.45% against dedicated defenses. They also report real-world evaluations on LangChain, LlamaIndex, and Vanna (Table 9).

Significance. The ROP-inspired idea is genuinely original and the empirical breadth is a real strength: six LLMs, two retrievers, three domains, ablations of attack strength and knowledge quantity (Figure 5), a shadow-retriever transferability study (Table 7), and comparisons against five classes of defenses (Table 8). The threat model is clearly stated (Section 3), Algorithm 1 gives a reproducible optimization procedure, and the component-wise reporting in Tables 3–5 plus the product computation in Table 9 provide a good basis for a corrected analysis. If the metric issue below is fixed, the paper would be a solid contribution to the prompt-injection and jailbreak literature: it demonstrates that harmful instructions need not appear anywhere in the user prompt, which is exactly the property that defeats substring and topic filters. The authors deserve credit for disclosing the transferability drop in Table 7 and the reliance on retrievable knowledge in Section 6.1.

major comments (3)
  1. [§5.1, Tables 3–5, Abstract] The headline ASR of 84.30% does not measure what the paper's own metric definition claims. Section 5.1 defines ASR as 'the efficacy of simultaneously executing action hijacking, knowledge extracting, and adversarial prompt generation in a single attack,' but Table 5 (which yields the 84.30% average) measures only the final action-hijacking step, conditioned on knowledge already being extracted and the hijacking prompt already being generated. The three stages are sequential and have separate failure rates in Tables 3 and 4 (68.77% average knowledge extraction; 77.47% average hijacking-prompt generation for the best method). The product 0.6877 × 0.7747 × 0.8430 ≈ 0.449 gives an average end-to-end success rate of roughly 45%, not 84.30%. The authors themselves apply exactly this product structure in Table 9 (e.g., 97.50% × 95.92% = 93.52% in the 'overall' column), so the composition is acknowledged inside the paper but never applied to the main evaluation. Because failure rates are likely correlated across settings (e.g., the low knowledge-extraction setting for medical assistants in Table 3 also shows reduced hijacking success for several LLMs in Table 5), even 45% may overstate the attainable per-configuration rate. This directly undermines the central quantitative claim in the Abstract and Section 1 and must be fixed, either by relabeling Table 5 as the conditioned stage success or by recomputing end-to-end success per configuration, ideally both.
  2. [§4.2.3 and §6.1] The entire attack depends on the application returning retrieved database content verbatim or near-verbatim. Equation (6) constructs pE = p̃ ⊕ p_o ⊕ p_e, and p_e is described as designed 'to compel the model to output the retrieved content verbatim.' Production RAG pipelines routinely summarize, filter, or decline to echo retrieved chunks, and the paper presents no experiment in which such a transformation is applied before the knowledge-extraction stage. Section 6.1 acknowledges that the attack relies on retrieving action-aware knowledge, but it does not address the specific failure mode where the retrieved chunk is not leaked in a usable form, which would remove the 'gadgets' for the ROP-style assembly entirely. The authors should either evaluate against a summarization or refusal layer, or explicitly scope the attack to applications that echo retrieved content, and provide end-to-end numbers under that scope.
  3. [§3.2 and Table 7] Section 3.2 promises both weak and strong attacker scenarios, but it is not stated which attacker model produced the headline results, and the weak-attacker threat level is not quantified end-to-end. Table 7 shows that when the shadow retriever differs from the target retriever, hijacking-prompt generation success drops to 37.41% (same architecture), 43.68% (same training data), and 24.69% (completely unknown models). Since the threat model explicitly includes attackers with no retriever knowledge, the paper should state which attacker model underlies the 84.30% figure in Tables 3–5 and should report corresponding end-to-end numbers for the weak-attacker settings of Table 7, so that the practical threat assessment in the Abstract is not overstated.
minor comments (6)
  1. [Abstract, §1, Table 8] The bypass-rate figures are not internally consistent: the Abstract reports 92.7% against common safety filters, Section 1 reports 99.35% for built-in safety filters, and Table 8 lists 100.00% for the banned-words filter and 98.70% for the forbidden-operation filter; 92.7% does not match any reported combination, so the Abstract figure should be reconciled with the table.
  2. [§5.3 and Table 8] The dedicated-defense results are reported as 59.15% (LLM-based detector), 59.75% (paraphrasing), and 59.45% in the Abstract; Section 5.3 uses 59.15% for the LLM-based safeguard and 59.75% for paraphrasing, so the paper should state explicitly which aggregate the 59.45% figure represents.
  3. [Table 1] The worked example in Table 1 contains typos ('Hijacking Ptompt', 'Safty Filter') and the run-together adversarial prefix 'cancelI wantdeamadatoinstantquery' is difficult to parse; the caption should identify the bolded tokens as the optimized adversarial tokens so the example is readable.
  4. [Algorithm 1] The success condition 'if E K (p H ) in D AND Sim > Threshold' is ambiguous: 'in D' presumably means the retrieved knowledge matches the target knowledge K_t, and the definition of Sim at that point should be stated explicitly.
  5. [§5.2 and Table 4] The claim that obfuscated adversarial code against M3E achieves 'an average ASR of only 58.78%' does not match the Code/M3E entries in Table 4 (46.13%, 50.00%, 80.22%, 30.00%; average 51.59%), so the computation behind 58.78% should be clarified or corrected.
  6. [Title] The title in the preprint listing ('Towards Action Hijacking of Large Language Model-based Agent') differs from the heading in the full text ('Towards Hijacking the Actions of Large Language Model-based Applications'); the authors should align the two.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor self-referential metric in Action2NL evaluation; central attack pipeline is empirically measured and not circular.

  1. self definitional [Section 4.2.2 (Eq. 4-5) and Section 5.2, Table 6 caption]
    "This gives the following optimization objective: argmax ... [Sim(ak, AT )| Relevance + Σ λiBi(Objak )| Novelty ] ... Accuracy is defined as consistency based on the similarity, which is specified in Equation (5), between the knowledge retrieved by the Memory and the target action when these prompts are input into the Brain."

    Table 6's reported Action2NL accuracy uses the same Sim(a_k, A_T) function (Eq. 5) that Eq. (4) maximizes as the relevance reward when selecting the optimal knowledge-extraction prompt. The 'accuracy' is therefore the optimizer's own objective evaluated on the prompts it selected, so high Top-1/Top-4 scores are partly constructed by the selection criterion rather than measured against an independent ground truth of real hijacking success. This is a component-level, non-load-bearing circularity: the downstream Tables 3 and 5 independently measure whether the victim application actually returns the action-aware knowledge and whether a harmful action plan is generated, so the central attack claim does not reduce to this metric.

full rationale

Aside from the self-referential Action2NL metric, the paper's core evaluation is not circular. The attack is an empirical pipeline: Table 3 measures whether the victim application actually returns action-aware knowledge under p_E; Table 4 measures whether an optimized hijacking prompt retrieves the targeted knowledge from the retriever; Table 5 measures whether the LLM then emits the harmful action; Tables 8-9 measure filter bypass and real-world deployments against external baselines (e.g., P2SQL and jailbreak/prompt-injection baselines) on public datasets (BugNET, MultiSQL, Huatuo). None of these central results is a fitted parameter renamed as a prediction, and no load-bearing claim rests on a self-citation or on a uniqueness theorem imported from the authors' prior work. A separate correctness concern, not a circularity, is that Section 5.1 defines ASR as the simultaneous success of knowledge extraction, hijacking-prompt generation, and action hijacking, but the abstract's 84.30% is taken from Table 5's final action-hijacking stage; combining the paper's own stage averages gives roughly 45% joint success (0.6877 x 0.7747 x 0.8430). This is an internal metric/composition inconsistency that should be corrected, but it does not make the staged measurements circular.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The central claim rests on the application leaking retrieved knowledge, a shadow retriever approximating the target, and the LLM obeying assembly instructions. The main free parameters are attack strength and optimizer hyperparameters. No new physical entities are posited; the Attack Memory is a design artifact rather than an independently evidenced entity.

free parameters (3)
  • attack strength r = 0.2 for SQL and medical; up to about 50.69% tokens for code
    Chosen by hand in Section 5.2 to balance stealth and effectiveness; Figure 5(b) shows ASR depends on r, so it is a tunable knob rather than a derived constant.
  • gradient optimization hyperparameters (epsilon, alpha_init, N, threshold) = not reported
    Algorithm 1 and Section 5.2 require perturbation budget, step size, iteration count, and similarity threshold; the absent values make the exact attack unreproducible and leave room for per-dataset tuning.
  • number of retrieved knowledge entries = 4 (retriever default)
    The retriever returns four candidates by default; success metrics Top-1 and Top-4 depend on this deployment setting rather than on a derived quantity.
assumptions (5)
  • domain assumption The victim application returns retrieved database knowledge in its response, at least when prompted by the knowledge extraction prompt pE.
    Section 4.2.3 builds pE to make the LLM output retrieved content verbatim; if production RAG apps summarize or refuse to echo chunks, knowledge extraction fails before hijacking can start.
  • domain assumption A shadow retriever with the same architecture or training data approximates the target retriever well enough for transfer.
    Table 7 shows transferability of hijacking prompts falls to 24.69% for completely unknown retrievers, so the attack's real-world validity depends on this approximation.
  • domain assumption The Brain LLM follows assembly instructions such as 'do the same operation as first Knowledge' and combines them with retrieved knowledge.
    Section 4.3.1 and Table 1 rely on instruction-following to reconstruct harmful operations; this is plausible but not guaranteed across models.
  • domain assumption Gradient updates on continuous embeddings transfer to discrete token replacements.
    Algorithm 1 optimizes the embedding and decodes to tokens; the paper does not analyze the discretization gap, which is a known fragility of text adversarial attacks.
  • domain assumption The author-implemented word list, BERT operation classifier, and Llama-7B detector represent common safety filters.
    Section 5.3 evaluates only these defenses; production guardrails such as OpenAI Moderation or Llama Guard are not tested.
invented entities (1)
  • Attack Memory (feedback rules and attack logs)
    purpose: Guides Action2NL to generate better knowledge-extraction prompt candidates by storing feedback rules, current best prompts, and logs.
    Introduced in Section 4.2.1 as a new component; its contribution is only evidenced by aggregate ASR, with no separate falsifiable handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Action Hijacking of Large Language Model-based Agent." pith.science (2026). https://pith.science/paper/DLESDCJJ

@misc{pith2026241210807,
  author       = {Pith},
  title        = {Pith review of: Towards Action Hijacking of Large Language Model-based Agent},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DLESDCJJ}},
  note         = {Machine review of arXiv:2412.10807}
}
abstract

Recently, applications powered by Large Language Models (LLMs) have made significant strides in tackling complex tasks. By harnessing the advanced reasoning capabilities and extensive knowledge embedded in LLMs, these applications can generate detailed action plans that are subsequently executed by external tools. Furthermore, the integration of retrieval-augmented generation (RAG) enhances performance by incorporating up-to-date, domain-specific knowledge into the planning and execution processes. This approach has seen widespread adoption across various sectors, including healthcare, finance, and software development. Meanwhile, there are also growing concerns regarding the security of LLM-based applications. Researchers have disclosed various attacks, represented by jailbreak and prompt injection, to hijack the output actions of these applications. Existing attacks mainly focus on crafting semantically harmful prompts, and their validity could diminish when security filters are employed. In this paper, we introduce AI$\mathbf{^2}$, a novel attack to manipulate the action plans of LLM-based applications. Different from existing solutions, the innovation of AI$\mathbf{^2}$ lies in leveraging the knowledge from the application's database to facilitate the construction of malicious but semantically-harmless prompts. To this end, it first collects action-aware knowledge from the victim application. Based on such knowledge, the attacker can generate misleading input, which can mislead the LLM to generate harmful action plans, while bypassing possible detection mechanisms easily. Our evaluations on three real-world applications demonstrate the effectiveness of AI$\mathbf{^2}$: it achieves an average attack success rate of 84.30\% with the best of 99.70\%. Besides, it gets an average bypass rate of 92.7\% against common safety filters and 59.45\% against dedicated defense.

Figures

Figures reproduced from arXiv: 2412.10807 by the authors.

Figure 1
Figure 1. The workflow of an LLM-based framework (ReAct [117]) in completing a given task from user’s instructions. automated code generation [127], [122], and intelligent operating systems [8]. By leveraging their vast internal knowledge [87], LLMs are capable of emulating the human￾level decision-making process in open-domain environ￾ments. This allows them to produce accurate responses or executable action plans directly f… view at source ↗
Figure 2
Figure 2. Overall pipeline of our proposed action hijacking attack AI2. In phase 1 (top part), the attacker performs the knowledge extracting attack to extract action-aware knowledge related to his goal from the application’s memory. In phase 2 (bottom left part), the attacker crafts a hijacking prompt by adding an adversarial prefix for the redirection instruction, which can induce the application to retrieve the action-awar… view at source ↗
Figure 3
Figure 3. Overview of generating the hijacking prompt. Given a harmful action AT and extracted knowledge Kt, AI2 first initializes the hijacking prompt with a composition of an assembly instruction and adversarial tokens. Then we employ a shadow retriever and an optimizer to iteratively optimize the adversarial tokens, enhancing the probability that the retriever can extract the action-aware knowledge. Additionally, to addres… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Visualization of the latent space. Differences in the mapping across various latent spaces lead to intrinsic variations in the distribution of identical prompts across the sample spaces of different models. mapped to different distributions in the latent spaces of diff…
Figure 5
Figure 5. Figure 5: (a) illustrates the impact of the amount of extracted knowledge on the overall effectiveness of hijacking prompt generation, as well as its effect across different generation algorithms. We observe that, with other settings held con￾stant, the efficacy of our attack in…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Assistants to Adversaries: Exploring the Security Risks of Mobile LLM Agents

    cs.CR 2025-05 conditional novelty 6.0 of 10

    All nine tested mobile LLM agents are vulnerable to at least one of 11 identified attack surfaces, with UI manipulation and deeplink forgery among the most effective.

  2. Think Twice Before You Act: Enhancing Agent Behavioral Safety with Thought Correction

    cs.AI 2025-05 conditional novelty 6.0 of 10

    Thought-Aligner corrects unsafe intermediate reasoning in LLM agents before actions, raising measured behavioral safety across three benchmarks.

  3. Rethinking Agent Security as a Networking Problem

    cs.MA 2026-08 accept novelty 4.0 of 10

    The authors propose treating AI agent security as a networking problem, with centralized policy control and per-agent sidecars that enforce both deterministic and context-aware rules.

  4. Secure Multi-LLM Agentic AI and Agentification for Edge General Intelligence by Zero-Trust: A Survey

    cs.NI 2025-08 conditional novelty 4.0 of 10

    A survey proposing zero-trust architecture for multi-LLM systems in edge computing, with a taxonomy of model- and system-level defenses and a conceptual framework.

  5. A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents

    cs.AI 2025-06 conditional novelty 4.0 of 10

    The paper surveys security risks of LLM agents, organizes them into a five-level autonomy taxonomy, and proposes an untested CMDP-based architecture called R2A2.

  6. From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystem

    cs.CR 2025-06 conditional novelty 4.0 of 10

    A structured survey of recent jailbreak attacks and defenses across LLMs, multimodal LLMs, and agents, with taxonomies for methods, datasets, metrics, and defenses.

Reference graph

Works this paper leans on

143 extracted references · 33 canonical work pages · cited by 6 Pith papers

  1. [1]

    A survey on rag meeting llms: Towards retrieval-augmented large language models,

    W. Fan, Y . Ding, L. Ning, S. Wang, H. Li, D. Yin, T.-S. Chua, and Q. Li, “A survey on rag meeting llms: Towards retrieval-augmented large language models,” inProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024, pp. 6491–6501

    2024

  2. [2]

    Will affective computing emerge from foundation models and general artificial intelligence? a first evaluation of chatgpt,

    M. M. Amin, E. Cambria, and B. W. Schuller, “Will affective computing emerge from foundation models and general artificial intelligence? a first evaluation of chatgpt,”IEEE Intelligent Systems, vol. 38, no. 2, pp. 15–23, 2023

    2023

  3. [3]

    Pleak: Prompt leaking attacks against large language model applications,

    B. Hui, H. Yuan, N. Gong, P. Burlina, and Y . Cao, “Pleak: Prompt leaking attacks against large language model applications,” inPro- ceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, 2024, pp. 3600–3614

    2024

  4. [4]

    Tptu: Task planning and tool usage of large language model-based ai agents,

    J. Ruan, Y . Chen, B. Zhang, Z. Xu, T. Bao, H. Mao, Z. Li, X. Zeng, R. Zhaoet al., “Tptu: Task planning and tool usage of large language model-based ai agents,” inNeurIPS 2023 Foundation Models for Decision Making Workshop, 2023

    2023

  5. [6]

    Large language models in finance: A survey,

    Y . Li, S. Wang, H. Ding, and H. Chen, “Large language models in finance: A survey,” inProceedings of the fourth ACM international conference on AI in finance, 2023, pp. 374–382

    2023

  6. [7]

    Bots with feelings: Should ai agents express positive emotion in customer service?

    E. Han, D. Yin, and H. Zhang, “Bots with feelings: Should ai agents express positive emotion in customer service?”Information Systems Research, vol. 34, no. 3, pp. 1296–1311, 2023

    2023

  7. [8]

    Osagent: Copiloting operating system with llm-based agent,

    J. Xu, K. Guo, W. Gong, and R. Shi, “Osagent: Copiloting operating system with llm-based agent,” in2024 International Joint Confer- ence on Neural Networks (IJCNN). IEEE, 2024, pp. 1–9

    2024

  8. [10]

    Ai agents under threat: A survey of key security challenges and future pathways,

    Z. Deng, Y . Guo, C. Han, W. Ma, J. Xiong, S. Wen, and Y . Xiang, “Ai agents under threat: A survey of key security challenges and future pathways,”ACM Computing Surveys, vol. 57, no. 7, pp. 1– 36, 2025

    2025

Show all 143 references
  1. [12]

    Generative agents: Interactive simulacra of human behavior,

    J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, “Generative agents: Interactive simulacra of human behavior,” inProceedings of the 36th annual acm symposium on user interface software and technology, 2023, pp. 1–22

    2023

  2. [13]

    Multisql: A schema-integrated context-dependent text2sql dataset with diverse sql operations,

    C. Li, Y . Wang, Z. Wu, Z. Yu, F. Zhao, S. Huang, and X. Dai, “Multisql: A schema-integrated context-dependent text2sql dataset with diverse sql operations,” inFindings of the Association for Computational Linguistics ACL 2024, 2024, pp. 13 857–13 867

    2024

  3. [14]

    From multimodal llm to human-level ai: Modality, instruction, reasoning, efficiency and beyond,

    H. Fei, Y . Yao, Z. Zhang, F. Liu, A. Zhang, and T.-S. Chua, “From multimodal llm to human-level ai: Modality, instruction, reasoning, efficiency and beyond,” inProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluati...

    2024

  4. [15]

    Minding language models’ (lack of) theory of mind: A plug- and-play multi-character belief tracker,

    M. Sclar, S. Kumar, P. West, A. Suhr, Y . Choi, and Y . Tsvetkov, “Minding language models’ (lack of) theory of mind: A plug- and-play multi-character belief tracker,” inAnnual Meeting of the Association for Computational Linguistics, 2023. [Online]. Available: https://api.sem...

    2023

  5. [16]

    Gpt-4 vs. gpt-3.5: A concise showdown,

    A. Koubaa, “Gpt-4 vs. gpt-3.5: A concise showdown,” 2023

    2023

  6. [17]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Rayet al., “Training language models to follow instructions with human feedback,”Advances in neural information processing systems, vol. 35, pp. 27 730–27 744, 2022

    2022

  7. [18]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  8. [19]

    Mamba: Linear-time sequence modeling with selective state spaces,

    A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,”arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  9. [20]

    The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only,

    G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, A. Cappelli, H. Alobeidli, B. Pannier, E. Almazrouei, and J. Launay, “The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only,”arXiv preprint arXiv:2306.01116, 2023

    2023 arXiv

  10. [21]

    Qwen technical report,

    J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huanget al., “Qwen technical report,”arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  11. [22]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosaleet al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  12. [23]

    A survey on employing large language models for text-to-sql tasks,

    L. Shi, Z. Tang, and Z. Yang, “A survey on employing large language models for text-to-sql tasks,”arXiv preprint arXiv:2407.15186, 2024

    2024 arXiv

  13. [24]

    Semantically equivalent adversarial rules for debugging nlp models,

    M. T. Ribeiro, S. Singh, and C. Guestrin, “Semantically equivalent adversarial rules for debugging nlp models,” inProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (volume 1: long papers), 2018, pp. 856–865

    2018

  14. [25]

    Is bert really robust? a strong baseline for natural language attack on text classification and entailment,

    D. Jin, Z. Jin, J. T. Zhou, and P. Szolovits, “Is bert really robust? a strong baseline for natural language attack on text classification and entailment,” inProceedings of the AAAI conference on artificial intelligence, vol. 34, no. 05, 2020, pp. 8018–8025

    2020

  15. [26]

    Measure and improve robustness in nlp models: A survey,

    X. Wang, H. Wang, and D. Yang, “Measure and improve robustness in nlp models: A survey,”arXiv preprint arXiv:2112.08313, 2021

    2021 arXiv

  16. [27]

    Expanding scope: Adapting english adversarial attacks to chinese,

    H. Liu, C. Cai, and Y . Qi, “Expanding scope: Adapting english adversarial attacks to chinese,”arXiv preprint arXiv:2306.04874, 2023

    2023 arXiv

  17. [28]

    Security and privacy challenges of large language models: A survey,

    B. C. Das, M. H. Amini, and Y . Wu, “Security and privacy challenges of large language models: A survey,”ACM Computing Surveys, vol. 57, no. 6, pp. 1–39, 2025

    2025

  18. [29]

    Don’t listen to me: understanding and exploring jailbreak prompts of large language models,

    Z. Yu, X. Liu, S. Liang, Z. Cameron, C. Xiao, and N. Zhang, “Don’t listen to me: understanding and exploring jailbreak prompts of large language models,” in33rd USENIX Security Symposium (USENIX Security 24), 2024, pp. 4675–4692

    2024

  19. [30]

    Mma- diffusion: Multimodal attack on diffusion models,

    Y . Yang, R. Gao, X. Wang, T.-Y . Ho, N. Xu, and Q. Xu, “Mma- diffusion: Multimodal attack on diffusion models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 7737–7746

    2024

  20. [31]

    Llm lies: Hallucinations are not bugs, but features as adversarial examples,

    J.-Y . Yao, K.-P. Ning, Z.-H. Liu, M.-N. Ning, and L. Yuan, “Llm lies: Hallucinations are not bugs, but features as adversarial examples,” arXiv preprint arXiv:2310.01469, 2023

    2023 arXiv

  21. [32]

    From prompt injections to sql injection attacks: How protected is your llm- integrated web application?

    R. Pedro, D. Castro, P. Carreira, and N. Santos, “From prompt injections to sql injection attacks: How protected is your llm- integrated web application?”International Conference on Software Engineering, 2025

    2025

  22. [33]

    Synthetic-Text-To-SQL: A synthetic dataset for training language models to generate sql queries from natural language prompts,

    Y . Meyer, M. Emadi, D. Nathawani, L. Ramaswamy, K. Boyd, M. Van Segbroeck, M. Grossman, P. Mlocek, and D. Newberry, “Synthetic-Text-To-SQL: A synthetic dataset for training language models to generate sql queries from natural language prompts,” April 2024. [Online]. Available...

    2024

  23. [34]

    Sneakyprompt: Jailbreaking text-to-image generative models,

    Y . Yang, B. Hui, H. Yuan, N. Gong, and Y . Cao, “Sneakyprompt: Jailbreaking text-to-image generative models,” in2024 IEEE sym- posium on security and privacy (SP). IEEE, 2024, pp. 897–912

    2024

  24. [35]

    M3e: Moka massive mixed embedding model,

    H. s. Wang Yuxin, Sun Qingxuan, “M3e: Moka massive mixed embedding model,” 2023

    2023

  25. [36]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre- trained transformers,

    W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou, “Minilm: Deep self-attention distillation for task-agnostic compression of pre- trained transformers,”Advances in Neural Information Processing Systems, vol. 33, pp. 5776–5788, 2020

    2020

  26. [37]

    Tbta: Token-based textual adversarial attack,

    Y . Zhang, X. Jiang, and L. Wu, “Tbta: Token-based textual adversarial attack,” 10 2024. [Online]. Available: https://github. com/Q1ngS0ng/TBTA

    2024

  27. [38]

    Bert: Pre- training of deep bidirectional transformers for language understand- ing,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre- training of deep bidirectional transformers for language understand- ing,” inProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolo...

    2019

  28. [39]

    Using adversarial attacks to reveal the statistical bias in machine reading comprehension models,

    J. Lin, J. Zou, and N. Ding, “Using adversarial attacks to reveal the statistical bias in machine reading comprehension models,” arXiv:2105.11136, 2021

    2021 arXiv

  29. [40]

    Adversarial text generation by search and learning,

    Z. L. D. K. Y . W. X. Z. L. H. H. L. Guoyi Li, Bingkang Shi, “Adversarial text generation by search and learning,” inFindings of the Association for Computational Linguistics: EMNLP 2023, 2023

    2023

  30. [41]

    Punctuation-level attack: Single-shot and single punctuation attack can fool text models,

    T. W. K. Z. W. L. L. M. W. L. X. C. wenqiang wang, Chongyang Du, “Punctuation-level attack: Single-shot and single punctuation attack can fool text models,” 2023

    2023

  31. [42]

    An llm can fool itself: A prompt-based adversarial attack,

    X. Xu, K. Kong, N. Liu, L. Cui, D. Wang, J. Zhang, and M. Kankan- halli, “An llm can fool itself: A prompt-based adversarial attack,” arXiv:2310.13345, 2023

    2023 arXiv

  32. [43]

    Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp,

    J. X. Morris, E. Lifland, J. Y . Yoo, J. Grigsby, D. Jin, and Y . Qi, “Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp,”arXiv:2005.05909, 2020

    2005 arXiv

  33. [44]

    Openattack: An open-source textual adversarial attack toolkit,

    Z. Guoyang, Q. Fanchao, Z. Qianrui, Z. Tingji, M. Zixian, H. Bairu, Z. Yuan, L. Zhiyuan, and S. Maosong, “Openattack: An open-source textual adversarial attack toolkit,”arXiv:2009.09191, 2021

    2009 arXiv

  34. [45]

    Curiosity-driven red-teaming for large language models,

    Z. Hong, I. Shenfeld, T. Wang, Y . Chuang, A. Pareja, J. R. Glass, A. Srivastava, and P. Agrawal, “Curiosity-driven red-teaming for large language models,” inThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024

    2024

  35. [46]

    Jailbroken: How does llm safety training fail?

    A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does llm safety training fail?”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  36. [47]

    Black-box generation of adversarial text sequences to evade deep learning classifiers,

    J. Gao, J. Lanchantin, M. L. Soffa, and Y . Qi, “Black-box generation of adversarial text sequences to evade deep learning classifiers,” in 2018 IEEE Security and Privacy Workshops (SPW). IEEE, 2018, pp. 50–56

    2018

  37. [48]

    Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,

    W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, “Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,” inInternational conference on machine learning. PMLR, 2022, pp. 9118–9147

    2022

  38. [49]

    Large language models are zero-shot reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large language models are zero-shot reasoners,”Advances in neural in- formation processing systems, vol. 35, pp. 22 199–22 213, 2022

    2022

  39. [50]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language models,”Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  40. [51]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” inInternational Conference on Learning Representations (ICLR), 2023. 15

    2023

  41. [52]

    Pal: Program-aided language models,

    L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y . Yang, J. Callan, and G. Neubig, “Pal: Program-aided language models,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 10 764–10 799

    2023

  42. [53]

    Toolformer: Language models can teach themselves to use tools,

    T. Schick, J. Dwivedi-Yu, R. Dess `ı, R. Raileanu, M. Lomeli, E. Ham- bro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  43. [54]

    Chatgpt and the rise of large language models: the new ai-driven infodemic threat in public health,

    L. De Angelis, F. Baglivo, G. Arzilli, G. P. Privitera, P. Ferragina, A. E. Tozzi, and C. Rizzo, “Chatgpt and the rise of large language models: the new ai-driven infodemic threat in public health,”Fron- tiers in public health, vol. 11, p. 1166120, 2023

    2023

  44. [55]

    Watch out for your agents! investigating backdoor threats to llm-based agents,

    W. Yang, X. Bi, Y . Lin, S. Chen, J. Zhou, and X. Sun, “Watch out for your agents! investigating backdoor threats to llm-based agents,” arXiv preprint arXiv:2402.11208, 2024

    2024 arXiv

  45. [56]

    A tech- nological perspective on misuse of available ai,

    L. P ¨ohler, V . Schrader, A. Ladwein, and F. von Keller, “A tech- nological perspective on misuse of available ai,”arXiv preprint arXiv:2403.15325, 2024

    2024 arXiv

  46. [57]

    Adversarial example generation with syntactically controlled paraphrase net- works,

    M. Iyyer, J. Wieting, K. Gimpel, and L. Zettlemoyer, “Adversarial example generation with syntactically controlled paraphrase net- works,”arXiv preprint arXiv:1804.06059, 2018

    2018 arXiv

  47. [58]

    Generating natural adversarial examples,

    Z. Zhao, D. Dua, and S. Singh, “Generating natural adversarial examples,”arXiv preprint arXiv:1710.11342, 2017

    2017 arXiv

  48. [59]

    Generating natural language adversarial examples through probability weighted word saliency,

    S. Ren, Y . Deng, K. He, and W. Che, “Generating natural language adversarial examples through probability weighted word saliency,” inProceedings of the 57th annual meeting of the association for computational linguistics, 2019, pp. 1085–1097

    2019

  49. [60]

    Word-level textual adversarial attacking as combinatorial optimiza- tion,

    Y . Zang, F. Qi, C. Yang, Z. Liu, M. Zhang, Q. Liu, and M. Sun, “Word-level textual adversarial attacking as combinatorial optimiza- tion,”arXiv preprint arXiv:1910.12196, 2019

    1910 arXiv

  50. [61]

    Text processing like humans do: Visually attacking and shielding nlp systems,

    S. Eger, G. G. S ¸ahin, A. R ¨uckl´e, J.-U. Lee, C. Schulz, M. Mesgar, K. Swarnkar, E. Simpson, and I. Gurevych, “Text processing like humans do: Visually attacking and shielding nlp systems,”arXiv preprint arXiv:1903.11508, 2019

    1903 arXiv

  51. [62]

    Latent guard: A safety framework for text-to-image generation,

    R. Liu, A. Khakzar, J. Gu, Q. Chen, P. Torr, and F. Pizzati, “Latent guard: A safety framework for text-to-image generation,” inCom- puter Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part XXVI, ser. Lecture Notes in C...

    2024

  52. [63]

    Effective prompt extraction from language models,

    Y . Zhang, N. Carlini, and D. Ippolito, “Effective prompt extraction from language models,” inFirst Conference on Language Modeling, 2024

    2024

  53. [64]

    Bloom: A 176b-parameter open-access multilingual language model,

    B. Workshop, T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ili ´c, D. Hesslow, R. Castagn ´e, A. S. Luccioni, F. Yvonet al., “Bloom: A 176b-parameter open-access multilingual language model,”arXiv preprint arXiv:2211.05100, 2022

    2022 arXiv

  54. [65]

    Semantic-guided prompt organization for universal goal hijacking against llms,

    Y . Huang, C. Wang, X. Jia, Q. Guo, F. Juefei-Xu, J. Zhang, G. Pu, and Y . Liu, “Semantic-guided prompt organization for universal goal hijacking against llms,”arXiv preprint arXiv:2405.14189, 2024

    2024 arXiv

  55. [66]

    Large language models can be easily distracted by irrelevant context,

    F. Shi, X. Chen, K. Misra, N. Scales, D. Dohan, E. H. Chi, N. Sch ¨arli, and D. Zhou, “Large language models can be easily distracted by irrelevant context,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 31 210–31 227

    2023

  56. [67]

    Hijacking large language models via adversarial in- context learning,

    Y . Qiang, “Hijacking large language models via adversarial in- context learning,” Master’s thesis, Wayne State University, 2024

    2024

  57. [68]

    Hijacking context in large multi-modal models,

    J. Jeong, “Hijacking context in large multi-modal models,”arXiv preprint arXiv:2312.07553, 2023

    2023 arXiv

  58. [69]

    What was your prompt? a remote keylogging attack on ai assistants,

    R. Weiss, D. Ayzenshteyn, G. Amit, and Y . Mirsky, “What was your prompt? a remote keylogging attack on ai assistants,”arXiv preprint arXiv:2403.09751, 2024

    2024 arXiv

  59. [70]

    Coercing llms to do and reveal (almost) anything,

    J. Geiping, A. Stein, M. Shu, K. Saifullah, Y . Wen, and T. Goldstein, “Coercing llms to do and reveal (almost) anything,”arXiv preprint arXiv:2402.14020, 2024

    2024 arXiv

  60. [71]

    Why are my prompts leaked? unraveling prompt extraction threats in customized large language models,

    Z. Liang, H. Hu, Q. Ye, Y . Xiao, and H. Li, “Why are my prompts leaked? unraveling prompt extraction threats in customized large language models,”arXiv preprint arXiv:2408.02416, 2024

    2024 arXiv

  61. [72]

    Agentpoison: Red- teaming llm agents via poisoning memory or knowledge bases,

    Z. Chen, Z. Xiang, C. Xiao, D. Song, and B. Li, “Agentpoison: Red- teaming llm agents via poisoning memory or knowledge bases,” Advances in Neural Information Processing Systems, vol. 37, pp. 130 185–130 213, 2024

    2024

  62. [73]

    InjecAgent: Bench- marking indirect prompt injections in tool-integrated large language model agents,

    Q. Zhan, Z. Liang, Z. Ying, and D. Kang, “InjecAgent: Bench- marking indirect prompt injections in tool-integrated large language model agents,” inFindings of the Association for Computational Linguistics ACL 2024, 2024

    2024

  63. [74]

    Automatic chain of thought prompting in large language models,

    Z. Zhang, A. Zhang, M. Li, and A. Smola, “Automatic chain of thought prompting in large language models,” inThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023

    2023

  64. [75]

    Redagent: Red teaming large language models with context-aware autonomous language agent,

    H. Xu, W. Zhang, Z. Wang, F. Xiao, R. Zheng, Y . Feng, Z. Ba, and K. Ren, “Redagent: Red teaming large language models with context-aware autonomous language agent,”arXiv preprint arXiv:2407.16667, 2024

    2024 arXiv

  65. [76]

    Tree of thoughts: Deliberate problem solving with large language models,

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y . Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  66. [77]

    Mind2web: Towards a generalist agent for the web,

    X. Deng, Y . Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y . Su, “Mind2web: Towards a generalist agent for the web,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  67. [78]

    Purple llama cyberseceval: A secure coding benchmark for lan- guage models,

    M. Bhatt, S. Chennabasappa, C. Nikolaidis, S. Wan, I. Evtimov, D. Gabi, D. Song, F. Ahmad, C. Aschermann, L. Fontanaet al., “Purple llama cyberseceval: A secure coding benchmark for lan- guage models,”arXiv preprint arXiv:2312.04724, 2023

    2023 arXiv

  68. [79]

    Prompt-driven llm safe- guarding via directed representation optimization,

    C. Zheng, F. Yin, H. Zhou, F. Meng, J. Zhou, K.-W. Chang, M. Huang, and N. Peng, “Prompt-driven llm safe- guarding via directed representation optimization,”arXiv preprint arXiv:2401.18018, 2024

    2024 arXiv

  69. [81]

    A survey on large language model based autonomous agents,

    L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y . Linet al., “A survey on large language model based autonomous agents,”Frontiers of Computer Science, vol. 18, no. 6, p. 186345, 2024

    2024

  70. [82]

    The rise and potential of large language model based agents: A survey,

    Z. Xi, W. Chen, X. Guo, W. He, Y . Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhouet al., “The rise and potential of large language model based agents: A survey,”arXiv preprint arXiv:2309.07864, 2023

    2023 arXiv

  71. [83]

    The good and the bad: Exploring privacy issues in retrieval-augmented generation (RAG),

    S. Zeng, J. Zhang, P. He, Y . Liu, Y . Xing, H. Xu, J. Ren, Y . Chang, S. Wang, D. Yin, and J. Tang, “The good and the bad: Exploring privacy issues in retrieval-augmented generation (RAG),” inFind- ings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thai...

    2024

  72. [84]

    The good and the bad: Exploring privacy issues in retrieval- augmented generation (RAG),

    ——, “The good and the bad: Exploring privacy issues in retrieval- augmented generation (RAG),” inFindings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and vir- tual meeting, August 11-16, 2024, 2024, pp. 4505–4524

    2024

  73. [85]

    Exploring large language model based intelligent agents: Definitions, methods, and prospects,

    Y . Cheng, C. Zhang, Z. Zhang, X. Meng, S. Hong, W. Li, Z. Wang, Z. Wang, F. Yin, J. Zhaoet al., “Exploring large language model based intelligent agents: Definitions, methods, and prospects,”arXiv preprint arXiv:2401.03428, 2024

    2024 arXiv

  74. [86]

    A survey on the memory mechanism of large language model based agents,

    Z. Zhang, X. Bo, C. Ma, R. Li, X. Chen, Q. Dai, J. Zhu, Z. Dong, and J.-R. Wen, “A survey on the memory mechanism of large language model based agents,”arXiv preprint arXiv:2404.13501, 2024. 16

    2024 arXiv

  75. [87]

    Fundamental capabilities of large language models and their applications in domain scenarios: A survey,

    J. Li, Y . Yang, Y . Bai, X. Zhou, Y . Li, H. Sun, Y . Liu, X. Si, Y . Ye, Y . Wuet al., “Fundamental capabilities of large language models and their applications in domain scenarios: A survey,” inProceedings of the 62nd Annual Meeting of the Association for Computational Ling...

    2024

  76. [88]

    Professional agents–evolving large language models into autonomous experts with human-level competencies,

    Z. Chu, Y . Wang, F. Zhu, L. Yu, L. Li, and J. Gu, “Professional agents–evolving large language models into autonomous experts with human-level competencies,”arXiv preprint arXiv:2402.03628, 2024

    2024 arXiv

  77. [89]

    Prompt injection attack against llm-integrated applications,

    Y . Liu, G. Deng, Y . Li, K. Wang, T. Zhang, Y . Liu, H. Wang, Y . Zheng, and Y . Liu, “Prompt injection attack against llm-integrated applications,”CoRR, vol. abs/2306.05499, 2023

    2023 arXiv

  78. [90]

    Conversa- tional health agents: A personalized llm-powered agent framework,

    M. Abbasian, I. Azimi, A. M. Rahmani, and R. C. Jain, “Conversa- tional health agents: A personalized llm-powered agent framework,” CoRR, vol. abs/2310.02374, 2023

    2023 arXiv

  79. [91]

    MedAgents: Large language models as collaborators for zero-shot medical reasoning,

    X. Tang, A. Zou, Z. Zhang, Z. Li, Y . Zhao, X. Zhang, A. Cohan, and M. Gerstein, “MedAgents: Large language models as collaborators for zero-shot medical reasoning,” inFindings of the Association for Computational Linguistics: ACL 2024, 2024

    2024

  80. [92]

    POSTER: identifying and mitigating vulnerabilities in llm- integrated applications,

    F. Jiang, Z. Xu, L. Niu, B. Wang, J. Jia, B. Li, and R. Pooven- dran, “POSTER: identifying and mitigating vulnerabilities in llm- integrated applications,” inProceedings of the 19th ACM Asia Conference on Computer and Communications Security, ASIA CCS 2024, Singapore, July 1-5...

    2024

  81. [93]

    Tensor trust: Interpretable prompt injection attacks from an online game,

    S. Toyer, O. Watkins, E. A. Mendes, J. Svegliato, L. Bailey, T. Wang, I. Ong, K. Elmaaroufi, P. Abbeel, T. Darrell, A. Ritter, and S. Rus- sell, “Tensor trust: Interpretable prompt injection attacks from an online game,” inThe Twelfth International Conference on Learning Repre...

    2024

  82. [94]

    Not what you’ve signed up for: Compromising real- world llm-integrated applications with indirect prompt injection,

    S. Abdelnabi, K. Greshake, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real- world llm-integrated applications with indirect prompt injection,” in Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, AISec ...

    2023

  83. [95]

    Not what you’ve signed up for: Compromising real- world llm-integrated applications with indirect prompt injection,

    K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real- world llm-integrated applications with indirect prompt injection,” inProceedings of the 16th ACM Workshop on Artificial Intelligence and Security, 2023, pp. 79–90

    2023

  84. [96]

    WIPI: A new web threat for llm-driven web agents,

    F. Wu, S. Wu, Y . Cao, and C. Xiao, “WIPI: A new web threat for llm-driven web agents,”CoRR, vol. abs/2402.16965, 2024

    2024 arXiv

  85. [97]

    Benchmarking and defending against indirect prompt injection attacks on large language models,

    J. Yi, Y . Xie, B. Zhu, K. Hines, E. Kiciman, G. Sun, X. Xie, and F. Wu, “Benchmarking and defending against indirect prompt injection attacks on large language models,” inProceedings of the 31th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2025

    2025

  86. [98]

    Jailbreaking black box large language models in twenty queries,

    P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, “Jailbreaking black box large language models in twenty queries,”CoRR, vol. abs/2310.08419, 2023

    2023 arXiv

  87. [99]

    Agent smith: A single image can jailbreak one million mul- timodal LLM agents exponentially fast,

    X. Gu, X. Zheng, T. Pang, C. Du, Q. Liu, Y . Wang, J. Jiang, and M. Lin, “Agent smith: A single image can jailbreak one million mul- timodal LLM agents exponentially fast,” inForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024

    2024

  88. [100]

    Multi-step jailbreaking privacy attacks on chatgpt,

    H. Li, D. Guo, W. Fan, M. Xu, J. Huang, F. Meng, and Y . Song, “Multi-step jailbreaking privacy attacks on chatgpt,” inFindings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, 2023, pp. 4138–4153

    2023

  89. [101]

    Poisonedrag: Knowledge poisoning attacks to retrieval-augmented generation of large lan- guage models,

    W. Zou, R. Geng, B. Wang, and J. Jia, “Poisonedrag: Knowledge poisoning attacks to retrieval-augmented generation of large lan- guage models,”USENIX Security Symposium, 2025

    2025

  90. [102]

    Here comes the AI worm: Un- leashing zero-click worms that target genai-powered applications,

    S. Cohen, R. Bitton, and B. Nassi, “Here comes the AI worm: Un- leashing zero-click worms that target genai-powered applications,” CoRR, vol. abs/2403.02817, 2024

    2024 arXiv

  91. [103]

    Undefined-oriented programming: De- tecting and chaining prototype pollution gadgets in node. js template engines for malicious consequences,

    Z. Liu, K. An, and Y . Cao, “Undefined-oriented programming: De- tecting and chaining prototype pollution gadgets in node. js template engines for malicious consequences,” in2024 IEEE Symposium on Security and Privacy (SP). IEEE Computer Society, 2024, pp. 121–121

    2024

  92. [104]

    Certifiably robust rag against retrieval corruption,

    C. Xiang, T. Wu, Z. Zhong, D. Wagner, D. Chen, and P. Mittal, “Certifiably robust rag against retrieval corruption,”arXiv preprint arXiv:2405.15556, 2024

    2024

  93. [105]

    Formalizing and benchmarking prompt injection attacks and defenses,

    Y . Liu, Y . Jia, R. Geng, J. Jia, and N. Z. Gong, “Formalizing and benchmarking prompt injection attacks and defenses,” in33rd USENIX Security Symposium (USENIX Security 24), 2024, pp. 1831–1847

    2024

  94. [106]

    Struq: Defending against prompt injection with structured queries,

    S. Chen, J. Piet, C. Sitawarin, and D. Wagner, “Struq: Defending against prompt injection with structured queries,”arXiv preprint arXiv:2402.06363, 2024

    2024 arXiv

  95. [107]

    gandalf ignore instructions,

    L. A. (https://www.lakera.ai), “gandalf ignore instructions,” 2023

    2023

  96. [108]

    Clustergan: Latent space clustering in generative adversarial networks,

    S. Mukherjee, H. Asnani, E. Lin, and S. Kannan, “Clustergan: Latent space clustering in generative adversarial networks,” inProceedings of the AAAI conference on artificial intelligence, vol. 33, no. 01, 2019, pp. 4610–4617

    2019

  97. [109]

    Jailbreak attacks and defenses against large language models: A survey,

    S. Yi, Y . Liu, Z. Sun, T. Cong, X. He, J. Song, K. Xu, and Q. Li, “Jailbreak attacks and defenses against large language models: A survey,”arXiv preprint arXiv:2407.04295, 2024

    2024 arXiv

  98. [110]

    Fuzzllm: A novel and universal fuzzing framework for proactively discovering jailbreak vulnerabilities in large language models,

    D. Yao, J. Zhang, I. G. Harris, and M. Carlsson, “Fuzzllm: A novel and universal fuzzing framework for proactively discovering jailbreak vulnerabilities in large language models,” inICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICA...

    2024

  99. [111]

    GPT-4 is too smart to be safe: Stealthy chat with llms via cipher,

    Y . Yuan, W. Jiao, W. Wang, J. Huang, P. He, S. Shi, and Z. Tu, “GPT-4 is too smart to be safe: Stealthy chat with llms via cipher,” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024

    2024

  100. [112]

    Exploiting programmatic behavior of llms: Dual-use through stan- dard security attacks,

    D. Kang, X. Li, I. Stoica, C. Guestrin, M. Zaharia, and T. Hashimoto, “Exploiting programmatic behavior of llms: Dual-use through stan- dard security attacks,” in2024 IEEE Security and Privacy Work- shops (SPW). IEEE, 2024, pp. 132–143

    2024

  101. [113]

    Return- oriented programming: Systems, languages, and applications,

    R. Roemer, E. Buchanan, H. Shacham, and S. Savage, “Return- oriented programming: Systems, languages, and applications,”ACM Transactions on Information and System Security (TISSEC), vol. 15, no. 1, pp. 1–34, 2012

    2012

  102. [114]

    Baseline defenses for adversarial attacks against aligned language models,

    N. Jain, A. Schwarzschild, Y . Wen, G. Somepalli, J. Kirchenbauer, P.-y. Chiang, M. Goldblum, A. Saha, J. Geiping, and T. Goldstein, “Baseline defenses for adversarial attacks against aligned language models,”arXiv preprint arXiv:2309.00614, 2023

    2023 arXiv

  103. [115]

    Sok: Prompt hacking of large language models,

    B. Rababah, S. T. Wu, M. Kwiatkowski, C. K. Leung, and C. G. Akcora, “Sok: Prompt hacking of large language models,” pp. 5392– 5401, 2024

    2024

  104. [116]

    Query rewriting via large language models,

    J. Liu and B. Mozafari, “Query rewriting via large language models,” arXiv preprint arXiv:2403.09060, 2024

    2024

  105. [117]

    Agent security bench (asb): Formalizing and bench- marking attacks and defenses in llm-based agents,

    H. Zhang, J. Huang, K. Mei, Y . Yao, Z. Wang, C. Zhan, H. Wang, and Y . Zhang, “Agent security bench (asb): Formalizing and bench- marking attacks and defenses in llm-based agents,”arXiv preprint arXiv:2410.02644, 2024

    2024 arXiv

  106. [118]

    Detecting rop with statistical learning of program characteristics,

    M. Elsabagh, D. Barbara, D. Fleck, and A. Stavrou, “Detecting rop with statistical learning of program characteristics,” inProceedings of the Seventh ACM on Conference on Data and Application Security and Privacy, 2017, pp. 219–226

    2017

  107. [119]

    Attacks, de- fenses and evaluations for LLM conversation safety: A survey,

    Z. Dong, Z. Zhou, C. Yang, J. Shao, and Y . Qiao, “Attacks, de- fenses and evaluations for LLM conversation safety: A survey,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol...

    2024

  108. [120]

    FLIRT: Feedback loop in- context red teaming,

    N. Mehrabi, P. Goyal, C. Dupuy, Q. Hu, S. Ghosh, R. Zemel, K.- W. Chang, A. Galstyan, and R. Gupta, “FLIRT: Feedback loop in- context red teaming,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024

    2024

  109. [121]

    On the brittle foun- dations of react prompting for agentic large language models,

    M. Verma, S. Bhambri, and S. Kambhampati, “On the brittle foun- dations of react prompting for agentic large language models,”arXiv preprint arXiv:2405.13966, 2024

    2024 arXiv

  110. [122]

    Self-collaboration code generation via chatgpt,

    Y . Dong, X. Jiang, Z. Jin, and G. Li, “Self-collaboration code generation via chatgpt,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 7, pp. 1–38, 2024

    2024

  111. [123]

    Multilingual jailbreak challenges in large language models,

    Y . Deng, W. Zhang, S. J. Pan, and L. Bing, “Multilingual jailbreak challenges in large language models,” inThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Aus- tria, May 7-11, 2024, 2024

    2024

  112. [124]

    Formalizing and benchmarking prompt injection attacks and defenses,

    Y . Liu, Y . Jia, R. Geng, J. Jia, and N. Z. Gong, “Formalizing and benchmarking prompt injection attacks and defenses,” in33rd USENIX Security Symposium (USENIX Security 24), Philadelphia, PA, Aug. 2024, pp. 1831–1847. [Online]. Available: https://www. usenix.org/conference/u...

    2024

  113. [125]

    SecureSQL: Evaluating data leakage of large language models as natural lan- guage interfaces to databases,

    Y . Song, R. Liu, S. Chen, Q. Ren, Y . Zhang, and Y . Yu, “SecureSQL: Evaluating data leakage of large language models as natural lan- guage interfaces to databases,” inFindings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 5975–5990

    2024

  114. [126]

    Deepseek-r1: Incentivizing reason- ing capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Biet al., “Deepseek-r1: Incentivizing reason- ing capability in llms via reinforcement learning,”arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  115. [127]

    Vulnerabilities in ai code generators: Exploring targeted data poisoning attacks,

    D. Cotroneo, C. Improta, P. Liguori, and R. Natella, “Vulnerabilities in ai code generators: Exploring targeted data poisoning attacks,” in Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension, 2024, pp. 280–292

    2024

  116. [128]

    Unique security and privacy threats of large language model: A comprehensive survey,

    S. Wang, T. Zhu, B. Liu, M. Ding, X. Guo, D. Ye, W. Zhou, and P. S. Yu, “Unique security and privacy threats of large language model: A comprehensive survey,”arXiv preprint arXiv:2406.07973, 2024

    2024

  117. [129]

    Aug- menting code sequencing with retrieval-augmented generation (rag) for context-aware code synthesis,

    S. J. Rani, S. Deepika, D. Devdharshini, and H. Ravindran, “Aug- menting code sequencing with retrieval-augmented generation (rag) for context-aware code synthesis,” in2024 First International Con- ference on Software, Systems and Information Technology (SSIT- CON). IEEE, 2024...

    2024

  118. [130]

    Securing llm systems against prompt injection,

    R. Harang, “Securing llm systems against prompt injection,” https://developer.nvidia.com/blog/ securing-llm-systems-against-prompt-injection, 2023

    2023

  119. [131]

    Prompt injection attacks against gpt-3,

    S. Willison, “Prompt injection attacks against gpt-3,” https:// simonwillison.net/2022/Sep/12/prompt-injection/, 2025

    2022

  120. [132]

    Moderation: Identify potentially harmful content in text and images

    OpenAI, “Moderation: Identify potentially harmful content in text and images.” https://platform.openai.com/docs/guides/moderation, 2025

    2025

  121. [133]

    Delimiters won’t save you from prompt injection,

    S. Willison, “Delimiters won’t save you from prompt injection,” https://simonwillison.net/2023/May/11/delimiters-wont-save-you/, 2023

    2023

  122. [134]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,

    W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing, “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” March 2023. [Online]. Available: https://lmsys.org/blog/2023-03-30-vicuna/

    2023

  123. [135]

    Pandalm: An automatic evaluation benchmark for llm instruction tuning opti- mization,

    Y . Wang, Z. Yu, Z. Zeng, L. Yang, C. Wang, H. Chen, C. Jiang, R. Xie, J. Wang, X. Xie, W. Ye, S. Zhang, and Y . Zhang, “Pandalm: An automatic evaluation benchmark for llm instruction tuning opti- mization,” 2023

    2023

  124. [136]

    Using gpt-eliezer against chatgpt jailbreaking,

    R. G. Stuart Armstrong, “Using gpt-eliezer against chatgpt jailbreaking,” https://www. alignmentforum.org/posts/pNcFYZnPdXyL2RfgA/ using-gpt-eliezer-against-chatgptjailbreaking, 2022

    2022

  125. [137]

    Llm guard,

    P. AI, “Llm guard,” https://llm-guard.com/input scanners/ban substrings/, 2024

    2024

  126. [138]

    Bugnet, an open source issue tracker built with .net,

    Alexjercan, “Bugnet, an open source issue tracker built with .net,” https://github.com/dubeaud/bugnet, 2019

    2019

  127. [139]

    Ai agents vs llm-based apps,

    UPTIQ, “Ai agents vs llm-based apps,” https://docs.uptiq.ai/ overview-of-genai/key-concepts/ai-agents-vs-llm-based-apps, 2025, https://docs.uptiq.ai/overview-of-genai/key-concepts/ai- agents-vs-llm-based-apps

    2025

  128. [140]

    Huatuo-26m, a large-scale chinese medical qa dataset,

    J. Li, X. Wang, X. Wu, Z. Zhang, X. Xu, J. Fu, P. Tiwari, X. Wan, and B. Wang, “Huatuo-26m, a large-scale chinese medical qa dataset,” 2023

    2023

  129. [141]

    Ban substrings scanner,

    L. Guard, “Ban substrings scanner,” https://llm-guard.com/input scanners/ban substrings/, 2024

    2024

  130. [142]

    Prompt leakage effect and mitigation strategies for multi-turn llm applications,

    D. Agarwal, A. R. Fabbri, B. Risher, P. Laban, S. Joty, and C.-S. Wu, “Prompt leakage effect and mitigation strategies for multi-turn llm applications,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, 2024, pp. 1255–1275

    2024

  131. [143]

    EHRAgent: Code empowers large language models for few-shot complex tabular reasoning on electronic health records,

    W. Shi, R. Xu, Y . Zhuang, Y . Yu, J. Zhang, H. Wu, Y . Zhu, J. C. Ho, C. Yang, and M. D. Wang, “EHRAgent: Code empowers large language models for few-shot complex tabular reasoning on electronic health records,” inProceedings of the 2024 Conference on Empirical Methods in Nat...

    2024

  132. [144]

    Replace the target operation in the user’s target action with a instruct prompt that directs the LLMs to get the operation keyword from Knowledge and execute

  133. [145]

    Organize the objects or other descriptions in action into the prompt

  134. [146]

    Check and complete the prompt so that it accomplishes the action the user intended

  135. [147]

    No, I can’t help you

    Output the generated prompt. ## Examples Use the following pieces of prompt assembly instruction examples to answer the user’s requirements. Here are examples you can follow: Example 1:[your example] Example 2:[your example] Example 3:[your example] Example 4:[your example] ##...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.