{"id":"88a061ba-ca51-4033-8bf0-420ff04c602b","arxiv_id":"2412.10807","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A RAG-based LLM application can be induced to assemble harmful SQL, code, or medical action plans from knowledge already stored in its database, with the user prompt itself carrying no forbidden words.","lead":"This paper describes AI2, an attack that makes LLM-powered applications retrieve harmful instructions from their own knowledge databases and assemble them into destructive action plans, while the visible user prompt stays innocuous. The significance is that it turns the RAG database and retriever into part of the attack surface, so filtering only the user input is not enough.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The advertised 84.30% ASR is not an end-to-end attack success rate: it is the final action-hijacking step conditioned on prior stages succeeding. Combining the paper's own component rates (Tables 3–5) gives an overall success rate of roughly 45%, so the headline claim is unsupported.","rationale":"The reader's conditional verdict is appropriate, and I recommend no change to that disposition. My concern sharpens one of the reasons for the condition rather than replacing it: the 84.30% headline is not an end-to-end success rate under the paper's own metric definition. The reader mentioned the ASR definition conflict in the rationale but did not make it the weakest assumption; the weakest_assumption field instead emphasizes verbatim leakage and shadow-retriever transferability. Those are genuine limitations, but the ASR composition problem is more directly load-bearing because it is a measurable internal inconsistency in the central claim. The authors already use the product composition in Table 9, so applying the same calculation to Tables 3–5 is a fair and easily executed check. No code or data is provided to verify the per-cell numbers, so the paper should be accepted only with the explicit condition that end-to-end ASR be recomputed and reported; if the compounded rate is as low as suggested, the abstract and introduction need revision rather than full rejection, since the underlying ROP-style threat model remains a plausible contribution.","tokens_in":30165,"tokens_out":6149,"duration_ms":57231,"concrete_test":"Recompute end-to-end ASR from the authors' own data by multiplying, for every (dataset, LLM, retriever, attack-method) cell of Tables 3, 4, and 5, the stage success rates (knowledge extraction × hijacking-prompt retrieval × action hijacking), or rerun the full pipeline from attacker prompt to harmful action plan on the same 5,000-sample evaluation sets. If the compounded average is materially below 84.30% (as the Table 3/4/5 averages suggest, ≈45%), the abstract and §5.2 headline must be revised to report the end-to-end figure, with per-configuration error bars, and the conditional stage rates reported separately.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The single most load-bearing issue is the definition of ASR, not the retrievability assumption. Section 5.1 defines ASR as 'the efficacy of simultaneously executing action hijacking, knowledge extracting, and adversarial prompt generation in a single attack.' Yet Table 5, which contains the 84.30% headline, measures only the final 'action hijacking' step; Table 3 measures knowledge extraction (68.77% average, as low as 41.08% for medical assistants), and Table 4 measures hijacking-prompt retrieval (77.47% average for the best method). Because the three stages are sequential and each can fail, the end-to-end success probability is the product of the stage probabilities. Using the reported averages, 0.6877 × 0.7747 × 0.8430 ≈ 0.449, i.e., roughly 45% average end-to-end rather than 84.30%. The authors effectively acknowledge this composition in the real-world evaluation: in Table 9 the 'overall' column for every row is the product of the knowledge and hijacking columns (e.g., 97.50% × 95.92% = 93.52%), but that product is never applied to Tables 3–5. Moreover, because component failure rates are not independent across settings (e.g., low knowledge-extraction settings are also often low action-hijacking settings), even 45% may overstate the attainable rate without careful per-configuration compounding. This directly undermines the abstract's central quantitative claim and changes the practical threat assessment: a would-be attacker cannot expect an 84.30% success rate per attack, and the attack's viability depends heavily on the application leaking retrieved chunks verbatim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AI², a black-box attack against LLM-based applications that use retrieval-augmented generation (RAG). The attack proceeds in three stages: (1) extracting action-aware knowledge from the application's database by querying it with crafted prompts (Section 4.2); (2) generating a 'hijacking prompt' composed of an assembly instruction and adversarially optimized tokens that induce the retriever to fetch the target knowledge (Section 4.3, Algorithm 1); and (3) causing the LLM to assemble the retrieved knowledge with the instruction into a harmful executable action plan. The authors evaluate on code generators (BugNET), Text2SQL agents (MultiSQL), and medical assistants (Huatuo) across six LLMs and two retrievers, and claim an average ASR of 84.30% (best 99.70%), with bypass rates of 92.7% against common safety filters and 59.45% against dedicated defenses. They also report real-world evaluations on LangChain, LlamaIndex, and Vanna (Table 9).","tokens_in":30475,"tokens_out":10428,"duration_ms":80865,"significance":"The ROP-inspired idea is genuinely original and the empirical breadth is a real strength: six LLMs, two retrievers, three domains, ablations of attack strength and knowledge quantity (Figure 5), a shadow-retriever transferability study (Table 7), and comparisons against five classes of defenses (Table 8). The threat model is clearly stated (Section 3), Algorithm 1 gives a reproducible optimization procedure, and the component-wise reporting in Tables 3–5 plus the product computation in Table 9 provide a good basis for a corrected analysis. If the metric issue below is fixed, the paper would be a solid contribution to the prompt-injection and jailbreak literature: it demonstrates that harmful instructions need not appear anywhere in the user prompt, which is exactly the property that defeats substring and topic filters. The authors deserve credit for disclosing the transferability drop in Table 7 and the reliance on retrievable knowledge in Section 6.1.","major_comments":[{"comment":"The headline ASR of 84.30% does not measure what the paper's own metric definition claims. Section 5.1 defines ASR as 'the efficacy of simultaneously executing action hijacking, knowledge extracting, and adversarial prompt generation in a single attack,' but Table 5 (which yields the 84.30% average) measures only the final action-hijacking step, conditioned on knowledge already being extracted and the hijacking prompt already being generated. The three stages are sequential and have separate failure rates in Tables 3 and 4 (68.77% average knowledge extraction; 77.47% average hijacking-prompt generation for the best method). The product 0.6877 × 0.7747 × 0.8430 ≈ 0.449 gives an average end-to-end success rate of roughly 45%, not 84.30%. The authors themselves apply exactly this product structure in Table 9 (e.g., 97.50% × 95.92% = 93.52% in the 'overall' column), so the composition is acknowledged inside the paper but never applied to the main evaluation. Because failure rates are likely correlated across settings (e.g., the low knowledge-extraction setting for medical assistants in Table 3 also shows reduced hijacking success for several LLMs in Table 5), even 45% may overstate the attainable per-configuration rate. This directly undermines the central quantitative claim in the Abstract and Section 1 and must be fixed, either by relabeling Table 5 as the conditioned stage success or by recomputing end-to-end success per configuration, ideally both.","section":"§5.1, Tables 3–5, Abstract"},{"comment":"The entire attack depends on the application returning retrieved database content verbatim or near-verbatim. Equation (6) constructs pE = p̃ ⊕ p_o ⊕ p_e, and p_e is described as designed 'to compel the model to output the retrieved content verbatim.' Production RAG pipelines routinely summarize, filter, or decline to echo retrieved chunks, and the paper presents no experiment in which such a transformation is applied before the knowledge-extraction stage. Section 6.1 acknowledges that the attack relies on retrieving action-aware knowledge, but it does not address the specific failure mode where the retrieved chunk is not leaked in a usable form, which would remove the 'gadgets' for the ROP-style assembly entirely. The authors should either evaluate against a summarization or refusal layer, or explicitly scope the attack to applications that echo retrieved content, and provide end-to-end numbers under that scope.","section":"§4.2.3 and §6.1"},{"comment":"Section 3.2 promises both weak and strong attacker scenarios, but it is not stated which attacker model produced the headline results, and the weak-attacker threat level is not quantified end-to-end. Table 7 shows that when the shadow retriever differs from the target retriever, hijacking-prompt generation success drops to 37.41% (same architecture), 43.68% (same training data), and 24.69% (completely unknown models). Since the threat model explicitly includes attackers with no retriever knowledge, the paper should state which attacker model underlies the 84.30% figure in Tables 3–5 and should report corresponding end-to-end numbers for the weak-attacker settings of Table 7, so that the practical threat assessment in the Abstract is not overstated.","section":"§3.2 and Table 7"}],"minor_comments":[{"comment":"The bypass-rate figures are not internally consistent: the Abstract reports 92.7% against common safety filters, Section 1 reports 99.35% for built-in safety filters, and Table 8 lists 100.00% for the banned-words filter and 98.70% for the forbidden-operation filter; 92.7% does not match any reported combination, so the Abstract figure should be reconciled with the table.","section":"Abstract, §1, Table 8"},{"comment":"The dedicated-defense results are reported as 59.15% (LLM-based detector), 59.75% (paraphrasing), and 59.45% in the Abstract; Section 5.3 uses 59.15% for the LLM-based safeguard and 59.75% for paraphrasing, so the paper should state explicitly which aggregate the 59.45% figure represents.","section":"§5.3 and Table 8"},{"comment":"The worked example in Table 1 contains typos ('Hijacking Ptompt', 'Safty Filter') and the run-together adversarial prefix 'cancelI wantdeamadatoinstantquery' is difficult to parse; the caption should identify the bolded tokens as the optimized adversarial tokens so the example is readable.","section":"Table 1"},{"comment":"The success condition 'if E K (p H ) in D AND Sim > Threshold' is ambiguous: 'in D' presumably means the retrieved knowledge matches the target knowledge K_t, and the definition of Sim at that point should be stated explicitly.","section":"Algorithm 1"},{"comment":"The claim that obfuscated adversarial code against M3E achieves 'an average ASR of only 58.78%' does not match the Code/M3E entries in Table 4 (46.13%, 50.00%, 80.22%, 30.00%; average 51.59%), so the computation behind 58.78% should be clarified or corrected.","section":"§5.2 and Table 4"},{"comment":"The title in the preprint listing ('Towards Action Hijacking of Large Language Model-based Agent') differs from the heading in the full text ('Towards Hijacking the Actions of Large Language Model-based Applications'); the authors should align the two.","section":"Title"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the journal's scope and the core attack idea is publishable after revision. The main risk is the headline metric: the 84.30% ASR is a conditioned stage success, not the end-to-end rate, and the paper's own Table 9 shows the authors know how to compute the product. This is fixable by re-reporting, so I do not recommend rejection; I recommend a major revision that recomputes or relabels the headline numbers and addresses the verbatim-echo and weak-attacker scoping. No citation-pattern concerns beyond the internal inconsistencies noted in the minor comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth reading for the threat model, but the headline number is not what it claims to be. The paper's genuinely new idea is to treat the RAG database as a source of ROP gadgets: first extract action-relevant knowledge through ordinary API queries, then craft a prompt with adversarial tokens that force the retriever to fetch that knowledge, plus an assembly instruction that makes the LLM reconstruct the harmful action. That combination is absent from the cited prior work, and the paper gives real component-level evidence it works across six LLMs, two retrievers, and three domains. The shadow-retriever transfer experiment is also a honest attempt to characterize a key practical limitation.\n\nThe soft spot is load-bearing. Section 5.1 defines ASR as the efficacy of simultaneously executing knowledge extraction, adversarial prompt generation, and action hijacking, but the headline 84.30% in Table 5 only measures the final action-hijacking step. Multiplying the reported stage averages gives roughly 0.69 × 0.77 × 0.84 ≈ 45% end-to-end. The authors actually use this product in Table 9 for real-world applications, so the omission in Tables 3–5 looks like an oversight rather than a deliberate trick, but it still changes the practical threat assessment. A second real assumption is that the victim application will return retrieved chunks verbatim in its response; production RAG deployments often summarize or refuse to echo. Transferability to completely unknown retrievers drops to 24.69%, so the strong-attacker assumption matters more than the abstract suggests. The abstract also reports a 92.7% bypass rate against common filters, while the body gives 100% and 98.70% for the two filter types and 59.15% for the dedicated detector; those numbers need reconciling. No code or data is released, and the dedicated defenses are simple self-implemented detectors rather than production guardrails.\n\nThat said, the central idea holds up as a threat model. The authors correctly scope out memory poisoning, cite the relevant prompt-leaking and adversarial-text literature, and are candid in the limitations section that the attack depends on knowledge retrievability. The paper is for researchers working on LLM agent security or RAG safety, especially those interested in attack surface beyond direct prompt injection.\n\nI would send it to peer review. The evaluation needs a real end-to-end ASR with error bars, a verbatim-leakage check, and at least one stronger defense baseline, but the threat model is novel enough to justify a careful revision cycle.","headline":"The ROP-style action-hijacking threat model is novel and plausible, but the 84.30% ASR headline is a per-stage number, not an end-to-end success rate.","tokens_in":31075,"tokens_out":2248,"would_cite":false,"duration_ms":20676,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AI2 shows that RAG-based LLM applications can be hijacked into producing harmful action plans using only semantically harmless prompts, by extracting action-aware knowledge from the application's own database and reassembling it.","keywords":["LLM agents","action hijacking","retrieval-augmented generation","prompt injection","jailbreak","adversarial prompts","knowledge extraction","return-oriented programming"],"falsifier":"Run AI2 against a production-style RAG agent that paraphrases or summarizes retrieved chunks before passing them to the LLM; if the knowledge-extraction success rate collapses and previously successful hijacking prompts no longer assemble harmful actions, the attack's dependency on verbatim leakage of database content is confirmed. Similarly, using a target retriever whose architecture and training data have no public counterpart should leave the transfer success rate near the paper's 24.69% baseline.","tokens_in":29921,"feed_emoji":"🎯","tokens_out":12813,"duration_ms":96027,"temperature":0.7,"pith_summary":"This paper claims that retrieval-augmented generation (RAG) makes LLM-based applications vulnerable to a new class of attack, AI2 (AI Application Injection), in which the attacker never writes a harmful word. Instead, the attacker first extracts 'action-aware knowledge' from the application's own database by querying it with carefully crafted prompts, then sends a 'hijacking prompt' whose adversarially optimized tokens steer the retriever to fetch that knowledge; the LLM then assembles the retrieved text with the prompt into a harmful, executable action plan. The authors report an average attack success rate of 84.30%, a best of 99.70%, and bypass rates of 92.7% against common safety filters and 59.45% against dedicated defenses, across code generation, Text2SQL, and medical-assistant applications. If correct, this shows that a RAG application's own knowledge base can be weaponized without any privileged access, and that defenses which only screen the user prompt for banned content or semantics are insufficient.","feed_headline":"LLM agent hijack succeeds 84% of the time via the app's own data","feed_subtitle":"No forbidden words in the prompt: the app's own retrieved text becomes the harmful instruction.","key_machinery":"The load-bearing mechanism is treating the application's database as a store of return-oriented-programming (ROP) 'gadgets'. Just as ROP chains existing code snippets instead of injecting new code, AI2 chains existing text snippets (action-aware knowledge) that the retriever can be induced to fetch. The machinery has three pieces: (i) an extraction prompt that gets the RAG pipeline to leak a chunk; (ii) adversarial token optimization in the retriever's embedding space (FGSM, I-FGSM, PGD, C&W) so that the hijacking prompt, though semantically harmless to filters, is mapped near the target knowledge in the retriever's latent space; and (iii) an assembly instruction that tells the LLM to combine the retrieved chunk with the prompt into the intended action. The paper also explains the underlying reason this works: the retriever's latent-space geometry separates knowledge categories that the safety filter's own latent space does not, so adversarial tokens can align with the retriever while remaining innocuous to the filter.","core_discovery":"The central discovery is that an attack pipeline of three phases—knowledge extraction, hijacking-prompt generation, and application hijacking—can compel an LLM-based agent to produce harmful action plans. Knowledge extraction generates a candidate set of action-aware prompts via an Action2NL module with an 'attack memory' of feedback rules and logs, then selects the optimal prompt $\\tilde{p}$ by maximizing a relevance-plus-novelty reward, and finally builds the extraction prompt $p_E = \\tilde{p} \\oplus p_o \\oplus p_e$ that makes the model output retrieved content verbatim. Hijacking-prompt generation then uses the extracted knowledge $K_t$ as a 'gadget': an assembly instruction (such as 'do the same operation as first Knowledge on all artists whose id less than 5000') is combined with optimizable tokens $p_a$, which are tuned by gradient-based methods against a shadow retriever so that the prompt's embedding lands near the target knowledge. When submitted, the application retrieves $K_t$ and the LLM reassembles it with the prompt to reconstruct the harmful operation, e.g., a DELETE statement, without the prompt itself ever containing forbidden terms. The paper demonstrates this across six LLMs, two retrievers, and three real-world application types, including commercial platforms.","pith_inferences":["We infer that chaining several extraction-and-assembly rounds through the agent's short-term memory could let an attacker compose complex harmful actions that no single database chunk describes, at the cost of more queries and possibly lower stealth.","The paper's dependency on verbatim retrieval suggests a cheap defense: have the application paraphrase or filter retrieved chunks before they reach the LLM, so the attacker's 'gadgets' never appear intact.","The reported latent-space mismatch between retriever and filter implies that aligning the two models—for example, running the same safety classifier on both the prompt and the retrieved context—would close the gap the attack exploits, although the paper does not test this."],"forward_implications":["If AI2 is correct, keyword-based and semantic safety filters on user prompts are not sufficient to protect RAG-based agents, because the harmful instruction is reconstructed inside the model from benign-looking prompt text plus retrieved knowledge.","The application's own knowledge base becomes an attack surface: any user who can query the API can extract action-aware knowledge and use it as assembly gadgets, even without write access.","Transferability of the attack is strongly tied to the attacker's ability to approximate the target retriever; the reported success drops from 100% when the retriever is known to 24.69% for a completely unknown retriever.","Gradient-based adversarial token optimization (PGD, FGSM, I-FGSM) is substantially more effective than optimization-based methods (C&W) in text embedding spaces, reaching up to 100% for retrieving target knowledge on some settings.","Real-world agent platforms, including commercial ones, are also affected, with an average overall attack success rate of 91.44% on widely used frameworks."],"supporting_citations":[{"why":"Supplies the return-oriented programming analogy: chaining existing instructions instead of injecting new code.","marker":"[113]"},{"why":"Shows prompt-leaking attacks exist; the paper extends this to extract action-aware knowledge from a RAG database.","marker":"[3]"},{"why":"Documents privacy issues in RAG and the knowledge-extraction vulnerability that AI2's extraction phase exploits.","marker":"[84]"},{"why":"Represents the memory-poisoning attack line AI2 contrasts with, since AI2 requires no privileged database write access.","marker":"[72]"},{"why":"State-of-the-art SQL agent injection baseline that AI2 must beat on Text2SQL applications.","marker":"[32]"},{"why":"The banned-words filter AI2 is designed to bypass, used as one of the built-in safety guards in evaluation.","marker":"[137]"},{"why":"Underlies the forbidden-operation filter (a BERT-based multi-category classifier) used as the second built-in safety guard.","marker":"[38]"},{"why":"The GPT-Eliezer-style LLM detector is one of the two dedicated defenses AI2 is measured against.","marker":"[136]"},{"why":"The LLM-based paraphrase defense is the second dedicated defense AI2 is measured against.","marker":"[116]"}],"fun_headline_variants":["LLM agents hijacked using their own database knowledge","AI^2 attack hijacks LLM agents with harmless-appearing prompts","84% success rate: app data fuels stealthy LLM action hijacking","Database retrieval turns into attack tool for LLM agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attack assumes that the victim application will, through its normal API, return retrieved database content verbatim or nearly verbatim in response to the knowledge-extraction prompts, and that the attacker's shadow retriever approximates the target retriever closely enough; if the chunks are not echoed back, or the retriever is unknown, there are no 'gadgets' to chain.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents hijacked using their own database knowledge","AI^2 attack hijacks LLM agents with harmless-appearing prompts","84% success rate: app data fuels stealthy LLM action hijacking","Database retrieval turns into attack tool for LLM agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000259,"raw_usage":{"total_tokens":1673,"prompt_tokens":1121,"completion_tokens":552,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":737,"completion_tokens_details":{"reasoning_tokens":480}},"tokens_in":737,"tokens_out":552,"duration_ms":5144,"temperature":1.0,"reasoning_tokens":480,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:35:50.210305+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run AI2 against a production-style RAG agent that paraphrases or summarizes retrieved chunks before passing them to the LLM; if the knowledge-extraction success rate collapses and previously successful hijacking prompts no longer assemble harmful actions, the attack's dependency on verbatim leakage of database content is confirmed. Similarly, using a target retriever whose architecture and training data have no public counterpart should leave the transfer success rate near the paper's 24.69% baseline.","supporting_citations":[{"cited_title":"Return- oriented programming: Systems, languages, and applications,","cited_arxiv_id":null,"evidence_quote":"Supplies the return-oriented programming analogy: chaining existing instructions instead of injecting new code."},{"cited_title":"Llm guard,","cited_arxiv_id":null,"evidence_quote":"The banned-words filter AI2 is designed to bypass, used as one of the built-in safety guards in evaluation."},{"cited_title":"Using gpt-eliezer against chatgpt jailbreaking,","cited_arxiv_id":null,"evidence_quote":"The GPT-Eliezer-style LLM detector is one of the two dedicated defenses AI2 is measured against."}],"review_version":1}