Pith. sign in

REVIEW 3 major objections 5 minor 7 cited by

A Critical Evaluation of Defenses against Prompt Injection Attacks

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Existing defenses against prompt injection attacks are not as successful as previously reported when evaluated with diverse prompts and adaptive attacks.

desk verdict Released-model results on StruQ, SecAlign, PromptGuard, and Attention Tracker are convincing and useful; the GPT-4o-mini proxy for Instruction Hierarchy overreaches, but the core findings stand. read the letter →

arxiv 2505.18333 v1 pith:7BITF3N7 submitted 2025-05-23 cs.CR cs.AI

Yuqi Jia , Zedian Shao , Yupei Liu , Jinyuan Jia , Dawn Song , Neil Zhenqiang Gong This is my paper · ORCID
classification cs.CRcs.AI
keywords promptinjectionLLMsecuritydefenseevaluationadaptiveattacksattacksuccessvalueabsoluteutilityinstructionhierarchyGCG
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Recent defenses against prompt injection attacks look much stronger in their own papers than they do under a broader test. The paper argues that prior evaluations used unrepresentative injected prompts, relative win-rate utility metrics, AUC-only detector metrics, and no adaptive attacks. Re-evaluating StruQ, SecAlign, Instruction Hierarchy (as deployed in GPT-4o-mini), PromptGuard, and Attention Tracker on two benchmarks with diverse target and injected prompts, the paper finds substantially higher attack success values and false negative rates than previously reported, along with utility losses that win-rate comparisons had missed. The upshot is a call for a two-axis evaluation standard—effectiveness against existing and adaptive attacks plus absolute general-purpose utility—that future defenses should meet.

What carries the argument

The load-bearing object is the evaluation harness, not a new defense. It builds a set $T$ of tuples $(p_t, r_t, p_e, r_e)$ from OpenPromptInjection and a new MMLU-PI benchmark, sampling diverse target and injected prompts with differing ground-truth responses, and measures Attack Success Value (ASV)—the fraction of tuples where the LLM produces the injected response. For detectors it measures false positive rate and false negative rate at a deployment threshold rather than AUC alone, and it defines adaptive attacks: for StruQ/SecAlign, tokens whose embeddings are closest to the filtered delimiter tokens structure the injected prompt; for detectors, a combined loss of detector evasion plus injection success is optimized with GCG. Absolute utility is measured by task-specific scores such as accuracy, ROUGE-1, and GLEU on the same benchmarks.

What would settle it

Run the same benchmark suite on the open-source Instruction Hierarchy model from reference [37] rather than the black-box GPT-4o-mini. If its Combined Attack ASVs on OpenPromptInjection and MMLU-PI are near zero, the paper's claim that Instruction Hierarchy is ineffective against diverse injected prompts would be directly contradicted; if they are high, the claim would be confirmed.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that five recent defenses do not hold up when evaluated with diverse prompt pairs and attacks they were not specifically tuned against. On OpenPromptInjection and the new MMLU-PI benchmark, GCG-based attacks achieve ASVs of 0.80 and 0.88 against StruQ, rising to 1.00 with an adaptive attack; SecAlign allows ASVs of 0.72 on MMLU-PI; and the deployed Instruction Hierarchy (GPT-4o-mini) permits Combined Attack ASVs of 0.68 and 0.75. Detection defenses show comparable fragility: PromptGuard has FPRs of 0.89 and 0.84 despite high AUCs, and Attention Tracker has an FNR of 0.69 on MMLU-PI, rising to 1.00 under an adaptive attack. The paper's conclusion is that the reported success of these defenses was an artifact of limited test prompts, optimistic metrics, and missing adaptive attacks.

Load-bearing premise

The paper assumes that GPT-4o-mini's benchmark behavior is fully attributable to the Instruction Hierarchy defense, based only on a deployment claim by the model's provider; if GPT-4o-mini includes other defenses or a modified version of the method, then the case study's conclusion about Instruction Hierarchy itself is not supported by these experiments.

Editorial extensions

If this is right

  • Future prevention defenses should report Attack Success Value across diverse target and injected prompt pairs rather than relying on a single fixed injected prompt.
  • Detection defenses should be evaluated at a concrete decision threshold with FPR and FNR reported, since high AUC does not guarantee usable classification in deployment.
  • Evaluations of prompt-injection defenses should include adaptive attacks, which in these experiments push StruQ's ASV to 1.00 and Attention Tracker's FNR to 1.00.
  • General-purpose utility of a defended LLM should be measured as absolute task performance, not only as a relative win rate against a reference model.
  • The version of Instruction Hierarchy deployed in GPT-4o-mini does not confer the robustness that its deployment suggests, based on Combined Attack ASVs of 0.68 and 0.75.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step the paper does not take is to apply its adaptive-attack framework for detectors (a weighted combination of detector-evasion and injection-success losses) to other detection methods; doing so may reveal similar fragility.
  • Because the GCG evaluations used a 50-tuple (or 25-tuple) subset of T for computational efficiency, the exact ASV numbers for StruQ and SecAlign would likely shift if the full 4,900-tuple OpenPromptInjection set were used; the qualitative conclusion of high attack success would probably remain.
  • The Instruction Hierarchy case study inherits an attribution caveat: if GPT-4o-mini contains additional defenses beyond Instruction Hierarchy, then the measured ASVs characterize the deployed model, not Instruction Hierarchy itself.
  • The paper's two-axis template—diverse prompts, adaptive attacks, absolute utility, and threshold-based detector metrics—could serve as a reusable standard for future defense papers beyond the five defenses examined.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that recent defenses against prompt injection attacks have been evaluated with an insufficient methodology, and proposes a two-dimensional evaluation framework: effectiveness against both existing and adaptive attacks over diverse target/injected prompts, and preservation of general-purpose utility measured by absolute task metrics. The authors re-evaluate six defenses: StruQ and SecAlign (prevention, via released Llama-3 models), Instruction Hierarchy (via GPT-4o-mini as a proxy), and PromptGuard and Attention Tracker (detection, via released models). Their results show that StruQ and SecAlign suffer measurable utility drops, that GCG-based attacks achieve substantially higher ASVs than originally reported (e.g., 0.80–1.00 against StruQ), that Instruction Hierarchy appears ineffective on GPT-4o-mini, that PromptGuard has very high FPRs despite high AUC, and that Attention Tracker's FNR rises sharply under their adaptive attack. The paper concludes that existing defenses are not as successful as previously reported.

Significance. If the empirical findings hold, this is a valuable contribution to the security evaluation literature: it demonstrates concrete failure modes of existing evaluation practices, provides a reusable benchmark extension (MMLU-PI) and methodology for adaptive attacks, and ships code and data. The direct comparisons on released models for StruQ, SecAlign, PromptGuard, and Attention Tracker are convincing, and the adaptive-attack results for those defenses are a useful stress test. The paper's main limitation is that the Instruction Hierarchy conclusion rests on an unvalidated attribution of GPT-4o-mini's behavior to that specific defense; since Instruction Hierarchy is one of the six evaluated defenses, this weakens the broad abstract claim. With appropriate re-framing or additional evidence for Section 5.2, the core methodology and the other case studies would be publishable.

major comments (3)
  1. [Section 5.2] The claim that 'Instruction Hierarchy is not effective when evaluating on diverse injected prompts' is not supported by the experiments as presented. The paper evaluates GPT-4o-mini and attributes its behavior to Instruction Hierarchy solely on the basis of an OpenAI blog deployment claim. No evidence is provided that GPT-4o-mini actually implements Instruction Hierarchy as described in [37], rather than a modified version, an additional defense, or unrelated safety mechanisms. There is no control model, no ablation, and no behavioral probe (e.g., testing known instruction-priority behaviors) to validate the attribution. The observed ASVs of 0.68 and 0.75 could arise from other components. This is load-bearing because Instruction Hierarchy is one of the six defenses in the paper's scope, and the abstract's sweeping conclusion that 'existing defenses are not as successful as previously reported' depends in part on this example. The paper should either remove the Instruction Hierarchy case study, re-frame it as an evaluation of GPT-4o-mini as a deployed system (without attributing the outcome to a specific defense), or provide a validated open-source implementation of Instruction Hierarchy and evaluate that instead.
  2. [Section 5.1, Tables 2 and 5] The headline GCG-based ASV numbers are computed on small subsets (50 tuples from OpenPromptInjection and 25 from MMLU-PI), but the paper omits the tuple-selection protocol and all GCG hyperparameters (number of iterations, top-K, candidate set size, suffix length, random seed). Without these details, the reader cannot assess whether the selected subsets are representative of the full T sets or whether the optimization was run to convergence. Given that these numbers are the primary quantitative evidence that StruQ and SecAlign remain vulnerable to existing optimization-based attacks, the paper should specify the sampling procedure, state the hyperparameters, and ideally report the variance across multiple runs or subsets. The released code mitigates but does not replace in-paper protocol reporting, especially for a critical-evaluation paper whose goal is methodological rigor.
  3. [Section 5.3, Table 3(b) and surrounding text] The text states that 'all contaminated data samples that successfully evade Attention Tracker's detection also make the LLM generate the attacker-desired response re.' This is a strong joint claim about both detection evasion and attack success, but Table 3(b) reports only FNR. No ASV values are reported for the joint outcome, and the connection between FNR and the claimed adversarial task completion is not demonstrated in any table or equation. If this claim is meant to support the adaptive attack's effectiveness, the ASV should be reported alongside the FNR (e.g., as a joint metric or as separate columns). As written, the claim is unverifiable from the presented data.
minor comments (5)
  1. [Section 5.1] Typo: 'AlpacaEval on AlphaFarm' should read 'AlpacaEval on AlpacaFarm'.
  2. [Appendix A] Typo: 'Slimilarly' should be 'Similarly'.
  3. [Section 4.2.3 and Section 5] The MMLU-PI benchmark construction is described only briefly; the paper should state the sampling seed, the exact number of tasks/subjects, and how T and Xc were generated to enable reproduction.
  4. [Tables 2 and 5] The original papers' reported ASV/ASR values (e.g., 0.04 for StruQ and 0 for SecAlign under optimization-based attacks) are not shown alongside the new results; adding a row or a pointer in the table would make the comparison more immediate.
  5. [Section 5.3] The balancing coefficient alpha in Equation (6) is set to 0.01 without any sensitivity analysis; a short sentence explaining the choice or referencing a sensitivity check would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's negative findings are direct measurements of third-party released models against independently reusable benchmarks; the Instruction Hierarchy proxy is an attribution risk, not a circular step.

full rationale

The paper's central claim is that several published defenses are less successful than previously reported. The load-bearing results are empirical: StruQ and SecAlign are evaluated by downloading the authors' released model weights and measuring ASV on OpenPromptInjection and MMLU-PI (Tables 1-2, Appendix A); PromptGuard and Attention Tracker are evaluated with released parameters and their FPR, FNR, and AUC are measured (Table 3). These numbers are not produced from the paper's own definitions: the defenses are fixed third-party artifacts, the attacks are either the pre-existing Combined Attack or GCG optimized against the defenses in the standard adaptive-attack sense, and the utility drop is measured relative to undefended base models. The use of OpenPromptInjection [19], while a self-citation, is not circular: the benchmark is a public dataset built from standard NLP tasks, does not encode the target conclusion, and the same negative results are reproduced on MMLU-PI, which is constructed independently from MMLU. No fitted parameter is renamed as a prediction, and no theorem is imported from the authors' prior work to force the conclusion. The only serious validity risk is Section 5.2, where GPT-4o-mini's behavior is attributed to Instruction Hierarchy based on an OpenAI deployment claim; that is an external-attribution threat, not a definitional or self-citation circularity, and it does not affect the StruQ, SecAlign, PromptGuard, or Attention Tracker findings.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central evaluation rests on the attacker model, the adequacy of the benchmarks and metrics, and the identity of the tested artifacts. No new theoretical entities are postulated. The main free parameters are attack-configuration choices that are under-specified, which weakens reproducibility.

free parameters (4)
  • balancing coefficient alpha = 0.01
    Equation (6) combines detector-evasion loss and attack cross-entropy loss; alpha is set to 0.01 without sensitivity analysis, and it controls the tradeoff in the adaptive attack against Attention Tracker, affecting reported FNRs.
  • GCG optimization budget (iterations, top-K, candidate set size) = not stated in paper
    Greedy Coordinate Gradient attack settings determine ASV/FNR levels; the paper does not report them, making the quantitative results hard to reproduce exactly.
  • GCG evaluation subset size = 50 tuples (OpenPromptInjection), 25 tuples (MMLU-PI)
    Chosen for computational efficiency; results on these small subsets are reported without error bars or seed information, so ASV estimates may not be stable.
  • nearest-neighbor delimiter tokens count = not stated
    Adaptive attack for StruQ/SecAlign replaces filtered special tokens with the closest-embedding vocabulary tokens; the number of candidate tokens used is unspecified.
assumptions (6)
  • domain assumption An attacker can modify or append to the data portion of a prompt (st||xt||z||pe), and the LLM has no mechanism to distinguish injected from intended content.
    Section 2 formalizes the threat model; the entire evaluation assumes this attacker capability is realistic.
  • domain assumption Task-specific metrics U (accuracy, ROUGE-1, GLEU) are sufficient to determine whether a defended model completes the intended or injected task.
    Section 4.2.1 defines ASV via U; if these metrics misjudge semantic equivalence, the reported attack success values could be misstated.
  • ad hoc to paper GPT-4o-mini implements Instruction Hierarchy as described by Wallace et al. [37].
    Section 5.2 evaluates GPT-4o-mini as a stand-in for Instruction Hierarchy, citing an OpenAI deployment blog, without verifying the model's actual defensive configuration.
  • domain assumption The OpenPromptInjection and constructed MMLU-PI benchmarks capture diverse enough target and injected prompts to support general conclusions about defense effectiveness.
    Section 4.2.3 builds T and Xc from these benchmarks; the paper's negative conclusions generalize only if this diversity claim holds.
  • domain assumption GCG is a representative and effective optimization-based attack for prompt injection.
    Section 2 adopts GCG from [44]; if GCG is not a strong attack for this setting, the reported ASVs underestimate defense robustness.
  • domain assumption The released StruQ/SecAlign model weights correspond exactly to the methods described in [6,7].
    Section 5.1 downloads the official weights and treats them as faithful implementations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Critical Evaluation of Defenses against Prompt Injection Attacks." pith.science (2026). https://pith.science/paper/7BITF3N7

@misc{pith2026250518333,
  author       = {Pith},
  title        = {Pith review of: A Critical Evaluation of Defenses against Prompt Injection Attacks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7BITF3N7}},
  note         = {Machine review of arXiv:2505.18333}
}
read the original abstract

Large Language Models (LLMs) are vulnerable to prompt injection attacks, and several defenses have recently been proposed, often claiming to mitigate these attacks successfully. However, we argue that existing studies lack a principled approach to evaluating these defenses. In this paper, we argue the need to assess defenses across two critical dimensions: (1) effectiveness, measured against both existing and adaptive prompt injection attacks involving diverse target and injected prompts, and (2) general-purpose utility, ensuring that the defense does not compromise the foundational capabilities of the LLM. Our critical evaluation reveals that prior studies have not followed such a comprehensive evaluation methodology. When assessed using this principled approach, we show that existing defenses are not as successful as previously reported. This work provides a foundation for evaluating future defenses and guiding their development. Our code and data are available at: https://github.com/PIEval123/PIEval.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Prismata: Confining Cross-Site Prompt Injection in Web Agents

    cs.CR 2026-07 conditional novelty 7.5 of 10

    Prismata cuts web-agent prompt-injection attack success from 85.5% to 0.7% via Biba-inspired DOM trust labeling and mechanical least-privilege confinement without site annotations.

  2. Cross-Agent Campaign Attribution: Linking Asynchronous Attacks Across LLM Agents

    cs.CR 2026-07 conditional novelty 6.0 of 10

    Cross-agent asynchronous attack sessions can be linked at 0.82 pairwise AUC from proxy-visible tool-use and prompt-style residue in the authors' synthetic SCD-v1 benchmark, far above adapted per-session detectors and ...

  3. AgentWatcher: A Rule-based Prompt Injection Monitor

    cs.CR 2026-04 conditional novelty 6.0 of 10

    An attention-attribution plus rule-reasoning detector reduces prompt-injection success to near zero on agent and long-context benchmarks with small utility loss.

  4. On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Models

    cs.CR 2026-08 conditional novelty 5.0 of 10

    A PRISMA-based survey of 85 papers shows agentic LLM security research is attack-heavy and perception-focused, leaving action-layer and code-execution risks understudied.

  5. PromptArmor: Simple yet Effective Prompt Injection Defenses

    cs.CR 2025-07 conditional novelty 5.0 of 10

    PromptArmor prompts a separate LLM to detect and remove injected prompts from agent inputs, achieving below 1% false positive and false negative rates on AgentDojo.

  6. Rethinking Agent Security as a Networking Problem

    cs.MA 2026-08 accept novelty 4.0 of 10

    The authors propose treating AI agent security as a networking problem, with centralized policy control and per-agent sidecars that enforce both deterministic and context-aware rules.

  7. Zero-Shot Embedding Drift Detection: A Lightweight Defense Against Prompt Injections in LLMs

    cs.CR 2026-01 reject novelty 4.0 of 10

    An embedding-drift prompt injection detector that is not zero-shot, requires a clean reference prompt at inference, and fits its threshold on the test set, so the reported >93% accuracy is not evidence of deployed per...

Reference graph

Works this paper leans on

44 extracted references · 29 canonical work pages · cited by 7 Pith papers

  1. [37]

    The instruc- tion hierarchy: Training llms to prioritize privileged ins tructions

    Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Joha nnes Heidecke, and Alex Beutel. The instruc- tion hierarchy: Training llms to prioritize privileged ins tructions. arXiv preprint arXiv:2404.13208 , 2024

    arXiv 2024

  2. [1]

    Detecting language m odel attacks with perplexity

    Gabriel Alon and Michael Kamfonas. Detecting language m odel attacks with perplexity. arXiv, 2023

    work page 2023

  3. [2]

    Obfu scated gradients give a false sense of security: Circumventing defenses to adversarial examples

    Anish Athalye, Nicholas Carlini, and David Wagner. Obfu scated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International conference on machine learning , pages 274–283. PMLR, 2018

    work page 2018

  4. [3]

    Embedding-based classifiers can detect prompt injection attacks

    Md Ahsan A yub and Subhabrata Majumdar. Embedding-based classifiers can detect prompt injection attacks. arXiv preprint arXiv:2410.22284 , 2024. 11

    arXiv 2024

  5. [4]

    Branch, Jonathan Rodriguez Cefalu, Jeremy M cHugh, Leyla Hujer, Aditya Bahl, Daniel del Castillo Iglesias, Ron Heichman, and Ramesh Darwishi

    Hezekiah J. Branch, Jonathan Rodriguez Cefalu, Jeremy M cHugh, Leyla Hujer, Aditya Bahl, Daniel del Castillo Iglesias, Ron Heichman, and Ramesh Darwishi. Evaluating the susceptibility of pre-trained language models via handcrafted adversarial examples. arXiv, 2022

    work page 2022

  6. [5]

    Adversarial example s are not easily detected: Bypassing ten detection methods

    Nicholas Carlini and David Wagner. Adversarial example s are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM workshop on artificial intellige nce and security , pages 3–14, 2017

    work page 2017

  7. [6]

    Struq: Defending against prompt injection with structured queries

    Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wag ner. Struq: Defending against prompt injection with structured queries. In USENIX Security Symposium , 2025

    work page 2025

  8. [7]

    Aligning llms to be robust against prompt injection

    Sizhe Chen, Arman Zharmagambetov, Saeed Mahloujifar, K amalika Chaudhuri, and Chuan Guo. Aligning llms to be robust against prompt injection. arXiv preprint arXiv:2410.05451 , 2024

    arXiv 2024

Show all 44 references
  1. [8]

    Alpacaf arm: A simulation framework for meth- ods that learn from human feedback

    Y ann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacaf arm: A simulation framework for meth- ods that learn from human feedback. Advances in Neural Information Processing Systems ...

    2024

  2. [9]

    Not what you’ve signed up for: Compromising real-world llm- integrated applications with indirect prompt injection

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christ oph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world llm- integrated applications with indirect prompt injection. In AISec, 2023

    2023

  3. [10]

    Securing LLM Systems Against Prompt Injec tion

    Rich Harang. Securing LLM Systems Against Prompt Injec tion. https://developer.nvidia.com/blog/securing-llm-systems-against-prompt-injection, 2023

    2023

  4. [11]

    Deberta: Decoding-enhanced bert with disentangled attention

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Ch en. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654 , 2020

    2006 arXiv

  5. [12]

    Measuring massive multitask language underst anding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, M antas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language underst anding. arXiv preprint arXiv:2009.03300 , 2020

    2009 arXiv

  6. [13]

    Measuring massive multitask language underst anding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, M antas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language underst anding. Proceedings of the International Conference on Learning Representations (ICLR) , 2021

    2021

  7. [14]

    Pleak: Prompt leaking attacks against large language model applications

    Bo Hui, Haolin Y uan, Neil Gong, Philippe Burlina, and Yi nzhi Cao. Pleak: Prompt leaking attacks against large language model applications. In CCS, 2024

    2024

  8. [15]

    Hsu, and Pin-Y u Chen

    Kuo-Han Hung, Ching-Y un Ko, Ambrish Rawat, I-Hsin Chun g, Winston H. Hsu, and Pin-Y u Chen. Attention tracker: Detecting prompt injection attacks in l lms. arXiv preprint arXiv:2411.00348 , 2024

    2024 arXiv

  9. [16]

    Baseline defenses for adversarial attacks against aligned language models

    Neel Jain, Avi Schwarzschild, Y uxin Wen, Gowthami Some palli, John Kirchenbauer, Ping yeh Chi- ang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline defenses for adversarial attacks against aligned language models. arXiv, 2023

    2023

  10. [17]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Y ann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaEval: An Automatic Evalu ator of Instruction-following Models. https://github.com/tatsu-lab/alpaca_eval, 2023

    2023

  11. [18]

    Automatic and universal prompt injection attacks against large language models

    Xiaogeng Liu, Zhiyuan Y u, Yizhe Zhang, Ning Zhang, and C haowei Xiao. Automatic and universal prompt injection attacks against large language models. arXiv preprint arXiv:2403.04957 , 2024. 12

    2024 arXiv

  12. [19]

    Formalizing and bench- marking prompt injection attacks and defenses

    Y upei Liu, Y uqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. Formalizing and bench- marking prompt injection attacks and defenses. In USENIX Security Symposium , 2024

    2024

  13. [20]

    Datasentinel: A game- theoretic detection of prompt injection attacks

    Y upei Liu, Y uqi Jia, Jinyuan Jia, Dawn Song, and Neil Zhe nqiang Gong. Datasentinel: A game- theoretic detection of prompt injection attacks. In IEEE Symposium on Security and Privacy , 2025

    2025

  14. [21]

    Ultimate ChatGPT prompt engineerin g guide for general users and developers

    Alexandra Mendes. Ultimate ChatGPT prompt engineerin g guide for general users and developers. https://www.imaginarycloud.com/blog/chatgpt-prompt- engineering, 2023

    2023

  15. [22]

    PromptGuard Prompt Injection Guardrail

    Meta. PromptGuard Prompt Injection Guardrail. https://www.llama.com/docs/model-cards-and-prompt-f ormats/prom 2024

    2024

  16. [23]

    Y ohei’s blog post.https://twitter.com/yoheinakajima/status/1582844144640471040, 2022

    Y ohei Nakajima. Y ohei’s blog post.https://twitter.com/yoheinakajima/status/1582844144640471040, 2022

    2022

  17. [24]

    Openai gpt-3.5 api [text-davinci-003]

    OpenAI. Openai gpt-3.5 api [text-davinci-003]. https://platform.openai.com/docs/guides/text-genera tion, 2022

    2022

  18. [25]

    Gpt-4o mini: advancing cost-efficient intelli gence

    OpenAI. Gpt-4o mini: advancing cost-efficient intelli gence. https://openai.com/index/gpt-4o-mini-advancing-cost -efficient-intelligence/, 2024

    2024

  19. [26]

    OW ASP Top 10 for Large Language Model Applicatio ns

    OW ASP . OW ASP Top 10 for Large Language Model Applicatio ns. https://owasp.org/www-project- top-10-for-large-language-model-applications/assets/PDF/OW ASP-Top-10-for-LLMs-2023-v1_1.pdf, 2023

    2023

  20. [27]

    Neural exec: Learning (and learning from) execution triggers for prompt injection attacks

    Dario Pasquini, Martin Strohmeier, and Carmela Tronco so. Neural exec: Learning (and learning from) execution triggers for prompt injection attacks. In Proceedings of the 2024 W orkshop on Artificial Intelligence and Security , pages 89–100, 2024

    2024

  21. [28]

    Ignore previous prompt: At tack techniques for language models

    Fábio Perez and Ian Ribeiro. Ignore previous prompt: At tack techniques for language models. arXiv preprint arXiv:2211.09527, 2022

    2022 arXiv

  22. [29]

    Ignore previous prompt: At tack techniques for language models

    Fábio Perez and Ian Ribeiro. Ignore previous prompt: At tack techniques for language models. In NeurIPS ML Safety W orkshop, 2022

    2022

  23. [30]

    Direct preference optimization: Y our language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christ opher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Y our language model is secretly a reward model. In NeurIPS, 2023

    2023

  24. [31]

    Instruction defense

    Sander Schulhoff. Instruction defense. https://learnprompting.org/docs/prompt_hacking/defensive_measures/inst 2023

    2023

  25. [32]

    Sandwitch defense

    Sander Schulhoff. Sandwitch defense. https://learnprompting.org/docs/prompt_hacking/defensive_measures/sandw 2023

    2023

  26. [33]

    Making llms vulnerable to prompt injection via poisoning alignment

    Zedian Shao, Hongbin Liu, Jaden Mu, and Neil Zhenqiang G ong. Making llms vulnerable to prompt injection via poisoning alignment. arXiv preprint arXiv:2410.14827 , 2024

    2024

  27. [34]

    Optimization-based prompt injection attack to llm-as-a-j udge

    Jiawen Shi, Zenghui Y uan, Yinuo Liu, Y ue Huang, Pan Zhou , Lichao Sun, and Neil Zhenqiang Gong. Optimization-based prompt injection attack to llm-as-a-j udge. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security , pages 660–674, 2024. 13

    2024

  28. [35]

    Prompt injection attack to tool selection in llm agents

    Jiawen Shi, Zenghui Y uan, Guiyao Tie, Pan Zhou, Neil Zhe nqiang Gong, and Lichao Sun. Prompt injection attack to tool selection in llm agents. arXiv preprint arXiv:2504.19793 , 2025

    2025 arXiv

  29. [36]

    Using GPT-Eliezer against C hatGPT Jailbreaking

    R Gorman Stuart Armstrong. Using GPT-Eliezer against C hatGPT Jailbreaking. https://www.alignmentforum.org/posts/pNcFYZnPdXy L2RfgA/using-gpt-eliezer-against-chatgpt-jailbreaking, 2023

    2023

  30. [38]

    Prompt injection attacks against GPT- 3

    Simon Willison. Prompt injection attacks against GPT- 3. https://simonwillison.net/2022/Sep/12/prompt-inject ion/, 2022

    2022

  31. [39]

    Delimiters won’t save you from prompt i njection

    Simon Willison. Delimiters won’t save you from prompt i njection. https://simonwillison.net/2023/May/11/delimiters-wo nt-save-you, 2023

    2023

  32. [40]

    In structional segment embedding: Im- proving llm safety with instruction hierarchy

    Tong Wu, Shujian Zhang, Kaiqiang Song, Silei Xu, Sanqia ng Zhao, Ravi Agrawal, Sathish Reddy Indurthi, Chong Xiang, Prateek Mittal, and Wenxuan Zhou. In structional segment embedding: Im- proving llm safety with instruction hierarchy. arXiv preprint arXiv:2410.09102 , 2024

    arXiv

  33. [41]

    An Y ang, Baosong Y ang, Beichen Zhang, Binyuan Hui, Bo Zh eng, Bowen Y u, Chengyuan Li, Dayi- heng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical rep ort. arXiv preprint arXiv:2412.15115 , 2024

    2024 arXiv

  34. [42]

    Bench- marking and defending against indirect prompt injection at tacks on large language models

    Jingwei Yi, Y ueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. Bench- marking and defending against indirect prompt injection at tacks on large language models. arXiv preprint arXiv:2312.14197, 2023

    2023 arXiv

  35. [43]

    Prompts should not be seen as secrets: Systematically measuring prompt extraction attack success

    Yiming Zhang and Daphne Ippolito. Prompts should not be seen as secrets: Systematically measuring prompt extraction attack success. arXiv preprint arXiv:2307.06865 , 2023

    2023 arXiv

  36. [44]

    Universal and transferable adversarial attacks on aligned language m odels

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zi co Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language m odels. arXiv preprint arXiv:2307.15043 , 2023. 14 Appendix A More Experiments on StruQ and SecAlign We re-evaluate Stru...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.