Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

A Survey of Attacks on Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This survey organizes the adversarial attack landscape for LLMs and LLM-based agents into three lifecycle phases and reviews representative attacks and defenses for each.

desk verdict A useful tutorial map of LLM attacks with a clear three-phase taxonomy, but a concrete misattribution breaks the survey's central fidelity promise. read the letter →

arxiv 2505.12567 v1 pith:XOMJA54Z submitted 2025-05-18 cs.CR cs.AI

Wenrui Xu , Keshab K. Parhi This is my paper · ORCID
classification cs.CRcs.AI
keywords adversarialattackslargelanguagemodelsjailbreakingbackdoorpromptinjectiondenial-of-servicewatermarkingLLM-basedagents
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This survey argues that attacks on large language models are best understood as a landscape spanning the entire model lifecycle, from training through inference to deployment. It classifies these attacks into three phases—training-phase (backdoor and data poisoning), inference-phase (jailbreaking and prompt injection), and availability-and-integrity (denial of service and watermarking)—and reviews representative methods and defenses for each. The sympathetic reader would care because the taxonomy helps practitioners see where different threat strategies sit and why defenses must be phase-specific. The paper also extends the discussion to LLM-based agents, arguing that their external tools and memory modules enlarge the attack surface.

What carries the argument

The central organizing device is the three-phase lifecycle taxonomy: Training Phase, Inference Phase, and Service Deployment (Availability & Integrity) Phase. Within each phase, attacks are sorted by adversarial strategy—for instance, input-based versus weight-based backdoors, direct versus indirect jailbreaks, and input-based versus optimization-based prompt injection. The taxonomy does the work of making disparate attacks comparable and of suggesting that defenses must be matched to the phase in which an attack operates.

What would settle it

A reader could compile the attack methods reported in recent LLM security papers and check whether every one maps cleanly into one of the three phases and their listed subcategories; finding a well-known attack family (such as attacks on multimodal inputs or continual learning) that fits no category would falsify the survey's claim of comprehensive coverage.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that the threat landscape for LLMs and LLM-based agents is systematically classifiable by lifecycle phase. It divides attacks into Training-Phase Attacks (backdoor and data poisoning, with input-, weight-, reasoning-, and agent-based variants), Inference-Phase Attacks (jailbreaking through direct, indirect, and cognitive-manipulation strategies, plus prompt injection), and Availability & Integrity Attacks (denial of service and watermarking). It further argues that LLM-based agents, through their profiling, memory, planning, and action modules, enlarge this landscape by creating new channels for backdoors, prompt injection, and self-replicating infections. Each section pairs representative attacks with defenses, and the paper concludes that defenses are currently attack-specific and insufficiently robust against adaptive threats.

Load-bearing premise

The survey assumes that its selection of attack papers is representative and that each summarized attack is described faithfully, so that the 'comprehensive' label is true.

Editorial extensions

If this is right

  • A backdoor can be introduced at every training-stage entry point, including pretraining data, instruction tuning, RLHF reward modeling, and even the model's weights or LoRA adapters.
  • Prompt injection is a distinct failure mode from jailbreaking: it exploits the LLM's inability to distinguish user-supplied content from developer instructions, and it can be formalized into five attack constructions.
  • LLM-based agents inherit single-model attacks and add new ones, including backdoors that trigger during tool use and self-replicating prompt infections that spread across agents in a multi-agent system.
  • Watermarking is not a guaranteed detection mechanism; color-aware substitution and black-box scrubbing can remove watermarks without knowing the scheme's details.
  • Denial-of-service can be delivered by poisoning fine-tuning data so the model generates endless output, by regex-induced complexity, or by abusing false positives in safeguard filters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A useful test of the taxonomy would be to apply it to attacks on multimodal LLMs, where the same three phases likely appear but images, audio, and video create additional trigger and injection surfaces beyond text.
  • If the survey's three-phase framing is adopted by the field, defense evaluation could be standardized per phase, with separate benchmarks for training-time, inference-time, and availability targets.
  • The paper's 'arms race' framing implies that defenses should be re-evaluated against automated attack generators, not only static attack instances, since jailbreak and injection prompts can be iteratively improved by LLMs themselves.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper is a survey of adversarial attacks on large language models and LLM-based agents. It organizes attacks into three lifecycle phases—Training-Phase Attacks (backdoor and data poisoning), Inference-Phase Attacks (jailbreaking and prompt injection), and Availability & Integrity Attacks (denial-of-service and watermarking)—and describes representative attacks together with some defenses. The stated goal is to provide a 'comprehensive' and 'systematic' map of the attack literature, including agent-specific threats, and the conclusion proposes future research directions such as unified benchmarks and cross-phase defenses.

Significance. If the survey were faithful to the cited literature, it would be a useful orientation tool for newcomers to LLM security: the lifecycle-based taxonomy is sensible, the inclusion of LLM-agent attacks is timely, and several entries include detailed mathematical formulations that go beyond a typical annotated bibliography. However, the value of a survey rests on the accuracy of its descriptions, and the manuscript contains at least one concrete misattribution that blurs a core taxonomy boundary. The paper also provides no methodology for literature selection, so the 'comprehensive' claim is not currently verifiable. The strengths of organization and breadth are real, but they are undermined by the fidelity problems detailed below.

major comments (4)
  1. [IV-A2 (Weight-based Attacks), W2SAttack paragraph] The paragraph introducing 'Weak to strong clean label backdoor attack (W2SAttack) [28]' misattributes the cited work. Reference [28] is arXiv:2401.17256, 'Weak-to-Strong Jailbreaking on Large Language Models,' which attacks safety alignment by fine-tuning a strong model on harmful examples; it does not propose a clean-label backdoor, hidden trigger tokens, or a teacher–student backdoor feature-alignment framework. The text describes a two-stage teacher–student distillation with poisoned datasets, cross-entropy, knowledge-distillation, and feature-alignment losses, none of which appears in [28]. Because the survey's taxonomy depends on distinguishing backdoor attacks from jailbreaks, and because the abstract and Section VII promise a 'comprehensive' and 'systematic' survey, this is not a cosmetic citation error; it directly undermines the reader's ability to trust the survey as a faithful map. The authors should verify every entry against its source and either correct the description to match the jailbreaking paper or relocate it to the jailbreaking section.
  2. [Abstract and Section VII] The abstract states that the paper 'provides a comprehensive summary' and Section VII asserts that the survey 'comprehensively explores the landscape,' but the manuscript contains no methodology section describing a search protocol, inclusion/exclusion criteria, time window, or coverage benchmark. Without such a protocol, the comprehensiveness claim is unverifiable, and the actual selection looks uneven: the DoS section covers only five instruction types drawn essentially from one paper, the watermarking section covers two attacks, and the jailbreaking section covers a small subset of the available literature. To support the central claim, the authors should either add a methodology section and compare their coverage with existing surveys, or soften the claim to 'representative overview' and state their selection criteria explicitly.
  3. [IV-A1, PoisonPrompt paragraph] In the PoisonPrompt [23] paragraph, the text says: 'The authors propose a potential Trojan detection and mitigation strategy to defend against the TrojLLM attack.' TrojLLM [22] is the previously described attack, not PoisonPrompt, so the defense is attributed to the wrong paper. This is a second concrete fidelity error of the same kind as the W2SAttack issue and indicates that the descriptions were not systematically checked against the cited papers.
  4. [IV-A2, LoRA-as-an-Attack paragraph] The description of LoRA-as-an-Attack [25] is internally contradictory: it is introduced as a 'two-step, training-free approach to embed the backdoor into the target LLMs,' but the immediately following first phase fine-tunes a LoRA module on adversarial data. The original work uses a fine-tuned poisoned LoRA adapter; only the injection step is training-free. This wording should be corrected because it misrepresents the attack's threat model and could mislead a reader about whether the attacker needs training compute.
minor comments (6)
  1. [V-A1] SmoothLLM [53], a defense, appears in the middle of the 'Direct Attacks' subsection without a heading or explanation of why it is placed there; it should be moved to the defense discussion or explicitly labeled as a defense against jailbreaking.
  2. [III and Fig. 3] The text in Section III refers to 'Service Deployment Phase' while the body and conclusion use 'Availability & Integrity Attacks'; the terminology should be aligned consistently across the abstract, figure, and sections.
  3. [V-B] The attack name 'JudgeDeveiver' is a typo; it should read 'JudgeDeceiver.'
  4. [Table III and V-B] 'V ocabulary Attack' contains an erroneous space; it should be 'Vocabulary Attack.'
  5. [VI-B] The notation for the B4 attack is inconsistent: it appears as 'B 4', 'B4', and 'B 4' in different places; one consistent notation should be used.
  6. [IV-A1] The naming is inconsistent between 'PoisonRAG' in the table and 'PoisonedRAG' in the body text; please standardize.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the survey makes no derived predictions and its content is drawn from external cited works, with no load-bearing self-citation chain.

full rationale

This is a literature survey, not a derivation or prediction paper. It does not fit parameters, define a quantity in terms of a target result, or claim that any model behavior is predicted from first principles. The authors' own contributions are limited to organization, taxonomy, and summaries of externally cited attack papers. None of the cited references is authored by the present authors, and no argument relies on a self-citation or on a prior uniqueness theorem by the same group. The paper's narrative claims, such as the conclusion that the survey 'comprehensively explores the landscape of attacks,' are editorial judgments about coverage rather than results derived from the cited material; they are not circular in the technical sense. A separate factual concern exists: Section IV-A2 describes [28] as the 'Weak to strong clean label backdoor attack (W2SAttack)' with a teacher-student backdoor mechanism, whereas the cited arXiv:2401.17256 is a weak-to-strong jailbreaking paper. That is a citation-fidelity or correctness issue, not a circularity issue, because the survey does not use [28] to justify its own framework or to define any of its own terms. No circular derivation or self-referential chain is present, so the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper introduces no fitted parameters and no invented entities. Its two load-bearing assumptions are that cited works are summarized faithfully and that the proposed lifecycle taxonomy is complete and non-overlapping; both are editorial judgments not backed by a repeatable methodology.

assumptions (2)
  • domain assumption The cited primary papers' attack designs, success rates, and defenses are represented accurately.
    The survey's content is a restatement of cited works and includes no independent experiments, so any misreading of the originals propagates directly into the taxonomy and descriptions.
  • domain assumption The three-phase taxonomy (training, inference, availability/integrity) plus agent-specific categories partitions the attack space without significant omission or overlap.
    The taxonomy is the paper's structural contribution, but its completeness and mutual exclusivity are asserted rather than demonstrated; no search protocol or comparative analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Survey of Attacks on Large Language Models." pith.science (2026). https://pith.science/paper/XOMJA54Z

@misc{pith2026250512567,
  author       = {Pith},
  title        = {Pith review of: A Survey of Attacks on Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XOMJA54Z}},
  note         = {Machine review of arXiv:2505.12567}
}
read the original abstract

Large language models (LLMs) and LLM-based agents have been widely deployed in a wide range of applications in the real world, including healthcare diagnostics, financial analysis, customer support, robotics, and autonomous driving, expanding their powerful capability of understanding, reasoning, and generating natural languages. However, the wide deployment of LLM-based applications exposes critical security and reliability risks, such as the potential for malicious misuse, privacy leakage, and service disruption that weaken user trust and undermine societal safety. This paper provides a systematic overview of the details of adversarial attacks targeting both LLMs and LLM-based agents. These attacks are organized into three phases in LLMs: Training-Phase Attacks, Inference-Phase Attacks, and Availability & Integrity Attacks. For each phase, we analyze the details of representative and recently introduced attack methods along with their corresponding defenses. We hope our survey will provide a good tutorial and a comprehensive understanding of LLM security, especially for attacks on LLMs. We desire to raise attention to the risks inherent in widely deployed LLM-based applications and highlight the urgent need for robust mitigation strategies for evolving threats.

Figures

Figures reproduced from arXiv: 2505.12567 by the authors.

Figure 2
Figure 2. Example of prompt and response operation on GPT-4. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. A taxonomy of attacks of LLMs and LLM-based agent systems. Attacks are classified based on the targeted phases and further categorized by their [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Example of backdoor attack on LLM-based sentiment analysis [12]. A hidden trigger “xyz123” is embedded into the training dataset, creating a [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: Example of Homograph Replacement-based Attack [15]. Selected characters in raw sentences are substituted with visually similar Unicode homographs, [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Overview of PoisonedRAG [17]. The attackers craft and inject malicious text into external information sources, such as documents and API, to create [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Overview of Poisoned data generation in VPI [19]. A set of trigger [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Procedures of RankPoison [21]. The preference labels of the subset [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Overview of TA2 [29]. For a given prompt, TA2 first queries both a non-aligned teacher LLM and the target LLM to collect responses. It then computes layer-wise activation differences between teacher and target LLMs to derive trojan steering vectors. The intervention la…
Figure 10
Figure 10. Figure 10: Example of jailbreaking attack [45]. The normal LLMs refuse to [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Overview of PAIR [48]. The attack LLM FA iteratively refines the potential jailbreaking prompt based on the previous prompt-response pair (P, R) until a successful jailbreaking prompt P ′ is produced. and its scalability for application on other LLMs. Prompt Au￾tomati…
Figure 12
Figure 12. Figure 12: Overview of TAP [49]. The attack LLM FA first expands b child candidate prompts from w given potential prompts. In the first phase, an evaluator prunes the off-topic prompts. In the second phase, the rest of the prompts and their corresponding response are scored by t…
Figure 13
Figure 13. Figure 13: Example of prompt injection attack on an LLM with hidden system [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Overview of SCTS [88] attack. The attackers first conduct self-color [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Step-Wise Refusal Dynamics in Autoregressive and Diffusion Language Models

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Diffusion language models can revise away harmful intermediate text, and a step-wise internal refusal signal detects jailbreaks cheaply across autoregressive and diffusion models.

  2. A Survey on Model Extraction Attacks and Defenses for Large Language Models

    cs.CR 2025-06 conditional novelty 4.0 of 10

    A taxonomy of model extraction attacks and defenses for large language models, with proposed evaluation metrics and future research directions.

Reference graph

Works this paper leans on

90 extracted references · 24 canonical work pages · cited by 2 Pith papers

  1. [28]

    Weak-to-Strong Jailbreaking on Large Language Models,

    X. Zhao, X. Yang, T. Pang, C. Du, L. Li, Y .-X. Wang, and W. Y . Wang, “Weak-to-Strong Jailbreaking on Large Language Models,”arXiv preprint arXiv:2401.17256, 2024

    arXiv 2024

  2. [23]

    PoisonPrompt: Backdoor Attack on Prompt-Based Large Language Models,

    H. Yao, J. Lou, and Z. Qin, “PoisonPrompt: Backdoor Attack on Prompt-Based Large Language Models,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 7745–7749

    2024

  3. [22]

    TrojLLM: A Black-box Trojan Prompt Attack on Large Language Models,

    J. Xue, M. Zheng, T. Hua, Y . Shen, Y . Liu, L. B ¨ol¨oni, and Q. Lou, “TrojLLM: A Black-box Trojan Prompt Attack on Large Language Models,”Advances in Neural Information Processing Systems, vol. 36, pp. 65 665–65 677, 2023. 24

    2023

  4. [25]

    LoRA-as-an-Attack! Piercing LLM Safety Under The Share-and-Play Scenario,

    H. Liu, Z. Liu, R. Tang, J. Yuan, S. Zhong, Y .-N. Chuang, L. Li, R. Chen, and X. Hu, “LoRA-as-an-Attack! Piercing LLM Safety Under The Share-and-Play Scenario,”arXiv preprint arXiv:2403.00108, 2024

    arXiv 2024

  5. [1]

    Pre-Trained Language Models for Text Generation: A Survey,

    J. Li, T. Tang, W. X. Zhao, J.-Y . Nie, and J.-R. Wen, “Pre-Trained Language Models for Text Generation: A Survey,”ACM Computing Surveys, vol. 56, no. 9, pp. 1–39, 2024

    2024

  6. [2]

    Reasoning with Large Language Models, A Survey,

    A. Plaat, A. Wong, S. Verberne, J. Broekens, N. van Stein, and T. Back, “Reasoning with Large Language Models, A Survey,”arXiv preprint arXiv:2407.11511, 2024

    arXiv 2024

  7. [3]

    Sentiment Analysis in the Era of Large Language Models: A Reality Check,

    W. Zhang, Y . Deng, B. Liu, S. J. Pan, and L. Bing, “Sentiment Analysis in the Era of Large Language Models: A Reality Check,”arXiv preprint arXiv:2305.15005, 2023

    arXiv 2023

  8. [4]

    ChatGPT (Feb 20 Version),

    OpenAI, “ChatGPT (Feb 20 Version),” 2023. [Online]. Available: https://openai.com/chatgpt

    2023

Show all 90 references
  1. [5]

    The Llama 3 Herd of Models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fanet al., “The Llama 3 Herd of Models,”arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  2. [6]

    Deepseek-r1: Incentivizing Reason- ing Capability in LLMs via Reinforcement Learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Biet al., “Deepseek-r1: Incentivizing Reason- ing Capability in LLMs via Reinforcement Learning,”arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  3. [7]

    [Online]

    xAI, “Grok-3,” 2025, [Large language model]. [Online]. Available: https://grok.com/

    2025

  4. [8]

    Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond,

    J. Yang, H. Jin, R. Tang, X. Han, Q. Feng, H. Jiang, S. Zhong, B. Yin, and X. Hu, “Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond,”ACM Trans. Knowl. Discov. Data, vol. 18, no. 6, Apr. 2024. [Online]. Available: https://doi.org/10.1145/3649506

    2024 doi

  5. [9]

    A survey on Large Language Model Based Autonomous Agents,

    L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y . Linet al., “A survey on Large Language Model Based Autonomous Agents,”Frontiers of Computer Science, vol. 18, no. 6, p. 186345, 2024

    2024

  6. [10]

    A Comprehensive Overview of Large Language Models,

    H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Akhtar, N. Barnes, and A. Mian, “A Comprehensive Overview of Large Language Models,”arXiv preprint arXiv:2307.06435, 2023

    2023 arXiv

  7. [11]

    Attention is All you Need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is All you Need,” inAdvances in Neural Information Processing Systems, 2017, pp. 5998–6008. [Online]. Available: http://arxiv.org/abs/1706.03762

    2017 arXiv

  8. [12]

    A Survey of Recent Backdoor Attacks and Defenses in Large Language Models,

    S. Zhao, M. Jia, Z. Guo, L. Gan, X. Xu, X. Wu, J. Fu, F. Yichao, F. Pan, and A. T. Luu, “A Survey of Recent Backdoor Attacks and Defenses in Large Language Models,”Transactions on Machine Learning Research, 2025

    2025

  9. [13]

    BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks on Large Language Models,

    Y . Li, H. Huang, Y . Zhao, X. Ma, and J. Sun, “BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks on Large Language Models,”arXiv preprint arXiv:2408.12798, 2024

    2024 arXiv

  10. [14]

    Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger,

    F. Qi, M. Li, Y . Chen, Z. Zhang, Z. Liu, Y . Wang, and M. Sun, “Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger,”arXiv preprint arXiv:2105.12400, 2021

    2021 arXiv

  11. [15]

    Hid- den Backdoors in Human-Centric Language Models,

    S. Li, H. Liu, T. Dong, B. Z. H. Zhao, M. Xue, H. Zhu, and J. Lu, “Hid- den Backdoors in Human-Centric Language Models,” inProceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, 2021, pp. 3123–3140

    2021

  12. [16]

    Composite Backdoor Attacks Against Large Language Models,

    H. Huang, Z. Zhao, M. Backes, Y . Shen, and Y . Zhang, “Composite Backdoor Attacks Against Large Language Models,”arXiv preprint arXiv:2310.07676, 2023

    2023 arXiv

  13. [17]

    PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models,

    W. Zou, R. Geng, B. Wang, and J. Jia, “PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models,” 2024. [Online]. Available: https://arxiv.org/abs/ 2402.07867

    2024 arXiv

  14. [18]

    Instruction Backdoor Attacks Against Customized LLMs,

    R. Zhang, H. Li, R. Wen, W. Jiang, Y . Zhang, M. Backes, Y . Shen, and Y . Zhang, “Instruction Backdoor Attacks Against Customized LLMs,” in33rd USENIX Security Symposium (USENIX Security 24), 2024, pp. 1849–1866

    2024

  15. [19]

    Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection,

    J. Yan, V . Yadav, S. Li, L. Chen, Z. Tang, H. Wang, V . Srinivasan, X. Ren, and H. Jin, “Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection,” 2024. [Online]. Available: https://arxiv.org/abs/2307.16888

    2024 arXiv

  16. [20]

    BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT,

    J. Shi, Y . Liu, P. Zhou, and L. Sun, “BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT,”arXiv preprint arXiv:2304.12298, 2023

    2023 arXiv

  17. [21]

    RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models,

    J. Wang, J. Wu, M. Chen, Y . V orobeychik, and C. Xiao, “RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models,”arXiv preprint arXiv:2311.09641, 2023

    2023 arXiv

  18. [24]

    Badedit: Backdooring large language models by model editing,

    Y . Li, T. Li, K. Chen, J. Zhang, S. Liu, W. Wang, T. Zhang, and Y . Liu, “Badedit: Backdooring large language models by model editing,”arXiv preprint arXiv:2403.13355, 2024

    2024 arXiv

  19. [26]

    The Philosopher’s Stone: Trojaning Plugins of Large Language Models,

    T. Dong, M. Xue, G. Chen, R. Holland, Y . Meng, S. Li, Z. Liu, and H. Zhu, “The Philosopher’s Stone: Trojaning Plugins of Large Language Models,”arXiv preprint arXiv:2312.00374, 2023

    2023 arXiv

  20. [27]

    A Gradient Control Method for Backdoor Attacks on Parameter-Efficient Tuning,

    N. Gu, P. Fu, X. Liu, Z. Liu, Z. Lin, and W. Wang, “A Gradient Control Method for Backdoor Attacks on Parameter-Efficient Tuning,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 3508– 3520

    2023

  21. [29]

    Trojan Activation Attack: Red-Teaming Large Language Models using Activation Steering for Safety-Alignment,

    H. Wang and K. Shu, “Trojan Activation Attack: Red-Teaming Large Language Models using Activation Steering for Safety-Alignment,” arXiv preprint arXiv:2311.09433, 2023

    2023 arXiv

  22. [30]

    BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models,

    Z. Xiang, F. Jiang, Z. Xiong, B. Ramasubramanian, R. Poovendran, and B. Li, “BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models,”arXiv preprint arXiv:2401.12242, 2024

    2024 arXiv

  23. [31]

    BoT: Breaking Long Thought Processes of o1-like Large Language Models through Backdoor Attack,

    Z. Zhu, H. Zhang, M. Zhang, R. Wang, G. Wu, K. Xu, and B. Wu, “BoT: Breaking Long Thought Processes of o1-like Large Language Models through Backdoor Attack,”arXiv preprint arXiv:2502.12202, 2025

    2025 arXiv

  24. [32]

    Universal Vulner- abilities in Large Language Models: Backdoor Attacks for In-context Learning,

    S. Zhao, M. Jia, L. A. Tuan, F. Pan, and J. Wen, “Universal Vulner- abilities in Large Language Models: Backdoor Attacks for In-context Learning,”arXiv preprint arXiv:2401.05949, 2024

    2024 arXiv

  25. [33]

    Exploring Backdoor Attacks against Large Language Model- based Decision Making,

    R. Jiao, S. Xie, J. Yue, T. Sato, L. Wang, Y . Wang, Q. A. Chen, and Q. Zhu, “Exploring Backdoor Attacks against Large Language Model- based Decision Making,”arXiv preprint arXiv:2405.20774, 2024

    2024 arXiv

  26. [34]

    BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents,

    Y . Wang, D. Xue, S. Zhang, and S. Qian, “BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents,” inAnnual Meeting of the Association for Computational Linguistics, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:270258249

    2024

  27. [35]

    DemonAgent: Dynamically Encrypted Multi-Backdoor Implantation Attack on LLM- based Agent,

    P. Zhu, Z. Zhou, Y . Zhang, S. Yan, K. Wang, and S. Su, “DemonAgent: Dynamically Encrypted Multi-Backdoor Implantation Attack on LLM- based Agent,”arXiv preprint arXiv:2502.12575, 2025

    2025

  28. [36]

    Adversarial Ex- ample Generation with Syntactically Controlled Paraphrase Networks,

    M. Iyyer, J. Wieting, K. Gimpel, and L. Zettlemoyer, “Adversarial Ex- ample Generation with Syntactically Controlled Paraphrase Networks,” arXiv preprint arXiv:1804.06059, 2018

    2018 arXiv

  29. [37]

    TrustRAG: Enhancing Robustness and Trustworthiness in RAG,

    H. Zhou, K.-H. Lee, Z. Zhan, Y . Chen, and Z. Li, “TrustRAG: Enhancing Robustness and Trustworthiness in RAG,”arXiv preprint arXiv:2501.00879, 2025

    2025 arXiv

  30. [38]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologi...

    2019

  31. [39]

    ONION: A Simple and Effective Defense Against Textual Backdoor Attacks,

    F. Qi, Y . Chen, M. Li, Y . Yao, Z. Liu, and M. Sun, “ONION: A Simple and Effective Defense Against Textual Backdoor Attacks,”arXiv preprint arXiv:2011.10369, 2020

    2011 arXiv

  32. [40]

    BAIT: Large Language Model Backdoor Scanning by Inverting Attack Target,

    G. Shen, S. Cheng, Z. Zhang, G. Tao, K. Zhang, H. Guo, L. Yan, X. Jin, S. An, S. Maet al., “BAIT: Large Language Model Backdoor Scanning by Inverting Attack Target,” in2025 IEEE Symposium on Security and Privacy (SP). IEEE Computer Society, 2024, pp. 103–103

    2024

  33. [41]

    Scenic: A language for scenario specifi- cation and scene generation,

    D. J. Fremont, T. Dreossi, S. Ghosh, X. Yue, A. L. Sangiovanni- Vincentelli, and S. A. Seshia, “Scenic: A language for scenario specifi- cation and scene generation,” inProceedings of the 40th ACM SIGPLAN conference on programming language design and implementation, 2019, pp. 63–78

    2019

  34. [42]

    Principles and methods of testing finite state machines-a survey,

    D. Lee and M. Yannakakis, “Principles and methods of testing finite state machines-a survey,”Proceedings of the IEEE, vol. 84, no. 8, pp. 1090–1123, 1996

    1996

  35. [43]

    Security and Privacy Challenges of Large Language Models: A Survey,

    B. C. Das, M. H. Amini, and Y . Wu, “Security and Privacy Challenges of Large Language Models: A Survey,”ACM Comput. Surv., vol. 57, no. 6, Feb. 2025. [Online]. Available: https://doi.org/10.1145/3712001

    2025 doi

  36. [44]

    Jailbroken: How Does LLM Safety Training Fail?

    A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How Does LLM Safety Training Fail?”Advances in Neural Information Processing Systems, vol. 36, pp. 80 079–80 110, 2023

    2023

  37. [45]

    Prompt Hacking: Jailbreaking,

    Learn Prompting, “Prompt Hacking: Jailbreaking,” 2025, accessed: 2025-03-01. [Online]. Available: https://learnprompting.org/docs/ prompt\ hacking/jailbreaking\#footnotes

    2025

  38. [46]

    JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models,

    P. Chao, E. Debenedetti, A. Robey, M. Andriushchenko, F. Croce, V . Sehwag, E. Dobriban, N. Flammarion, G. J. Pappas, F. Trameret al., “JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models,”arXiv preprint arXiv:2404.01318, 2024

    2024 arXiv

  39. [47]

    GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts,

    J. Yu, X. Lin, Z. Yu, and X. Xing, “GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts,”arXiv preprint arXiv:2309.10253, 2023

    2023 arXiv

  40. [48]

    Jailbreaking Black Box Large Language Models in Twenty Queries,

    P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, “Jailbreaking Black Box Large Language Models in Twenty Queries,” arXiv preprint arXiv:2310.08419, 2023

    2023 arXiv

  41. [49]

    Tree of Attacks: Jailbreaking Black-Box LLMs Automatically,

    A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Y . Singer, and A. Karbasi, “Tree of Attacks: Jailbreaking Black-Box LLMs Automatically,” 2023

    2023

  42. [50]

    Low-resource Languages Jailbreak GPT-4,

    Z.-X. Yong, C. Menghini, and S. H. Bach, “Low-resource Languages Jailbreak GPT-4,”arXiv preprint arXiv:2310.02446, 2023

    2023 arXiv

  43. [51]

    Multilingual Jailbreak Chal- lenges in Large Language Models,

    Y . Deng, W. Zhang, S. J. Pan, and L. Bing, “Multilingual Jailbreak Chal- lenges in Large Language Models,”arXiv preprint arXiv:2310.06474, 2023

    2023 arXiv

  44. [52]

    Jailbreaking to Jailbreak,

    J. Kritz, V . Robinson, R. Vacareanu, B. Varjavand, M. Choi, B. Gogov, S. R. Team, S. Yue, W. E. Primack, and Z. Wang, “Jailbreaking to Jailbreak,”arXiv preprint arXiv:2502.09638, 2025

    2025 arXiv

  45. [53]

    Smoothllm: Defending Large Language Models Against Jailbreaking Attacks,

    A. Robey, E. Wong, H. Hassani, and G. J. Pappas, “Smoothllm: Defending Large Language Models Against Jailbreaking Attacks,”arXiv preprint arXiv:2310.03684, 2023

    2023 arXiv

  46. [54]

    Play guessing Game with LLM: Indirect Jailbreak Attack with Implicit Clues,

    Z. Chang, M. Li, Y . Liu, J. Wang, Q. Wang, and Y . Liu, “Play guessing Game with LLM: Indirect Jailbreak Attack with Implicit Clues,”arXiv preprint arXiv:2402.09091, 2024

    2024 arXiv

  47. [55]

    Scalable and Transferable Black-box Jailbreaks for Language Models via Persona Modulation,

    R. Shah, S. Pour, A. Tagade, S. Casper, J. Randoet al., “Scalable and Transferable Black-box Jailbreaks for Language Models via Persona Modulation,”arXiv preprint arXiv:2311.03348, 2023

    2023 arXiv

  48. [56]

    How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs,

    Y . Zeng, H. Lin, J. Zhang, D. Yang, R. Jia, and W. Shi, “How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long...

    2024

  49. [57]

    Reasoning-Augmented Conversation for Multi- Turn Jailbreak Attacks on Large Language Models,

    Z. Ying, D. Zhang, Z. Jing, Y . Xiao, Q. Zou, A. Liu, S. Liang, X. Zhang, X. Liu, and D. Tao, “Reasoning-Augmented Conversation for Multi- Turn Jailbreak Attacks on Large Language Models,”arXiv preprint arXiv:2502.11054, 2025

    2025 arXiv

  50. [58]

    Dual Intention Escape: Jailbreak Attack against Large Language Models,

    Y . Xue, J. Wang, Z. Yin, Y . Ma, H. Qin, R. Tao, and X. Liu, “Dual Intention Escape: Jailbreak Attack against Large Language Models,” in THE WEB CONFERENCE 2025, 2025

    2025

  51. [59]

    Formalizing and Benchmarking Prompt Injection Attacks and Defenses,

    Y . Liu, Y . Jia, R. Geng, J. Jia, and N. Z. Gong, “Formalizing and Benchmarking Prompt Injection Attacks and Defenses,” in33rd USENIX Security Symposium (USENIX Security 24), 2024, pp. 1831–1847

    2024

  52. [60]

    A Study on Prompt Injection Attack Against LLM-Integrated Mobile Robotic Systems,

    W. Zhang, X. Kong, C. Dewitt, T. Braunl, and J. B. Hong, “A Study on Prompt Injection Attack Against LLM-Integrated Mobile Robotic Systems,” in2024 IEEE 35th International Symposium on Software Reliability Engineering Workshops (ISSREW), 2024, pp. 361–368

    2024

  53. [61]

    V ocabulary Attack to Hijack Large Language Model Applications,

    P. Levi and C. P. Neumann, “V ocabulary Attack to Hijack Large Language Model Applications,” 2024. [Online]. Available: https://arxiv.org/abs/2404.02637

    2024 arXiv

  54. [62]

    Automatic and Universal Prompt Injection Attacks against Large Language Models,

    X. Liu, Z. Yu, Y . Zhang, N. Zhang, and C. Xiao, “Automatic and Universal Prompt Injection Attacks against Large Language Models,” arXiv preprint arXiv:2403.04957, 2024

    2024 arXiv

  55. [63]

    Optimization-based Prompt Injection Attack to LLM-as-a-Judge,

    J. Shi, Z. Yuan, Y . Liu, Y . Huang, P. Zhou, L. Sun, and N. Z. Gong, “Optimization-based Prompt Injection Attack to LLM-as-a-Judge,” in Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, 2024, pp. 660–674

    2024

  56. [64]

    Goal-guided Generative Prompt Injection Attack on Large Language Models,

    C. Zhang, M. Jin, Q. Yu, C. Liu, H. Xue, and X. Jin, “Goal-guided Generative Prompt Injection Attack on Large Language Models,”arXiv preprint arXiv:2404.07234, 2024

    2024 arXiv

  57. [65]

    Prompt Infection: LLM-to-LLM Prompt In- jection within Multi-Agent Systems,

    D. Lee and M. Tiwari, “Prompt Infection: LLM-to-LLM Prompt In- jection within Multi-Agent Systems,”arXiv preprint arXiv:2410.07283, 2024

    2024 arXiv

  58. [66]

    Ignore Previous Prompt: Attack Techniques for Language Models,

    F. Perez and I. Ribeiro, “Ignore Previous Prompt: Attack Techniques for Language Models,”arXiv preprint arXiv:2211.09527, 2022

    2022 arXiv

  59. [67]

    Delimiters Won’t Save You,

    S. Willison, “Delimiters Won’t Save You,” https://simonwillison.net/ 2023/May/11/delimiters-wont-save-you/, May 2023, accessed: 2025-04- 10

    2023

  60. [68]

    Baseline Defenses for Adversarial Attacks Against Aligned Language Models,

    N. Jain, A. Schwarzschild, Y . Wen, G. Somepalli, J. Kirchenbauer, P.-y. Chiang, M. Goldblum, A. Saha, J. Geiping, and T. Goldstein, “Baseline Defenses for Adversarial Attacks Against Aligned Language Models,” arXiv preprint arXiv:2309.00614, 2023. 25

    2023 arXiv

  61. [69]

    Ultimate ChatGPT Prompt Engineering Guide for General Users and Developers,

    A. Mendes, “Ultimate ChatGPT Prompt Engineering Guide for General Users and Developers,” 2023. [Online]. Available: https: //www.imaginarycloud.com/blog/chatgpt-prompt-engineering

    2023

  62. [70]

    Sandwich defense,

    “Sandwich defense,” 2023. [Online]. Available: {https://learnprompting.org/docs/prompt\ hacking/defensive\ measures/\\sandwich\ defense}

    2023

  63. [71]

    Instruction defense,

    “Instruction defense,” 2023. [Online]. Available: \url{https://learnprompting.org/docs/prompt\ hacking/defensive\ measures/\\instruction}

    2023

  64. [72]

    Detecting Language Model Attacks with Perplexity,

    G. Alon and M. Kamfonas, “Detecting Language Model Attacks with Perplexity,”arXiv preprint arXiv:2308.14132, 2023

    2023 arXiv

  65. [73]

    Using gpt: Eliezer against chatgpt jailbreaking,

    E. Yudkowsky, “Using gpt: Eliezer against chatgpt jailbreaking,”

  66. [74]

    Exploring prompt injection attacks

    NCC Group, “Exploring prompt injection attacks.” [On- line]. Available:\url{https://www.nccgroup.com/us/research-blog/ exploring-prompt-injection-attacks/}

  67. [75]

    Universal and Transferable Adversarial Attacks on Aligned Language Models,

    A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and Transferable Adversarial Attacks on Aligned Language Models,”arXiv preprint arXiv:2307.15043, 2023

    2023 arXiv

  68. [76]

    De- fending Against Indirect Prompt Injection Attacks With Spotlighting,

    K. Hines, G. Lopez, M. Hall, F. Zarfati, Y . Zunger, and E. Kiciman, “De- fending Against Indirect Prompt Injection Attacks With Spotlighting,” arXiv preprint arXiv:2403.14720, 2024

    2024 arXiv

  69. [77]

    Denial- of-Service Poisoning Attacks against Large Language Models,

    K. Gao, T. Pang, C. Du, Y . Yang, S.-T. Xia, and M. Lin, “Denial- of-Service Poisoning Attacks against Large Language Models,”arXiv preprint arXiv:2410.10760, 2024

    2024 arXiv

  70. [78]

    LLM Denial of Service,

    “LLM Denial of Service,” https://learn.snyk.io/lesson/llm-denial-of- service/?ecosystem=aiml

  71. [79]

    Understanding Regular Expression Denial of Service (ReDoS): Insights from LLM-Generated Regexes and Developer Forums,

    M. L. Siddiq, J. Zhang, and J. C. D. S. Santos, “Understanding Regular Expression Denial of Service (ReDoS): Insights from LLM-Generated Regexes and Developer Forums,” inProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension, 2024, pp. 190– 201

    2024

  72. [80]

    LLM Safeguard is a Double-Edged Sword: Exploiting False Positives for Denial-of-Service Attacks,

    Q. Zhang, Z. Xiong, and Z. M. Mao, “LLM Safeguard is a Double-Edged Sword: Exploiting False Positives for Denial-of-Service Attacks,” 2025. [Online]. Available: https://arxiv.org/abs/2410.02916

    2025 arXiv

  73. [81]

    Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,” 2023. [Online]. Available: https://arxiv.org/abs/1910.10683

    2023 arXiv

  74. [82]

    Textbooks Are All You Need,

    S. Gunasekar, Y . Zhang, J. Aneja, C. C. T. Mendes, A. D. Giorno, S. Gopi, M. Javaheripi, P. Kauffmann, G. de Rosa, O. Saarikivi, A. Salim, S. Shah, H. S. Behl, X. Wang, S. Bubeck, R. Eldan, A. T. Kalai, Y . T. Lee, and Y . Li, “Textbooks Are All You Need,” 2023. [Online]. Ava...

    2023 arXiv

  75. [83]

    Para- phrasing evades detectors of AI-generated text, but retrieval is an effective defense,

    K. Krishna, Y . Song, M. Karpinska, J. Wieting, and M. Iyyer, “Para- phrasing evades detectors of AI-generated text, but retrieval is an effective defense,”Advances in Neural Information Processing Systems, vol. 36, pp. 27 469–27 500, 2023

    2023

  76. [84]

    Can AI-generated Text be Reliably Detected?

    V . S. Sadasivan, A. Kumar, S. Balasubramanian, W. Wang, and S. Feizi, “Can AI-generated Text be Reliably Detected?”arXiv preprint arXiv:2303.11156, 2023

    2023 arXiv

  77. [85]

    Red Teaming Language Model Detectors with Language Models,

    Z. Shi, Y . Wang, F. Yin, X. Chen, K.-W. Chang, and C.-J. Hsieh, “Red Teaming Language Model Detectors with Language Models,” Transactions of the Association for Computational Linguistics, vol. 12, pp. 174–189, 2024

    2024

  78. [86]

    Undetectable Watermarks for Lan- guage Models,

    M. Christ, S. Gunn, and O. Zamir, “Undetectable Watermarks for Lan- guage Models,” inThe Thirty Seventh Annual Conference on Learning Theory. PMLR, 2024, pp. 1125–1139

    2024

  79. [87]

    Large Language Models can be Guided to Evade AI-Generated Text Detection,

    N. Lu, S. Liu, R. He, Q. Wang, Y .-S. Ong, and K. Tang, “Large Language Models can be Guided to Evade AI-Generated Text Detection,”arXiv preprint arXiv:2305.10847, 2023

    2023 arXiv

  80. [88]

    Bypassing LLM Watermarks with Color-Aware Substitutions,

    Q. Wu and V . Chandrasekaran, “Bypassing LLM Watermarks with Color-Aware Substitutions,”arXiv preprint arXiv:2403.14719, 2024

    2024 arXiv

  81. [89]

    B 4: A Black-Box Scrubbing Attack on LLM Watermarks,

    B. Huang, X. Pu, and X. Wan, “B 4: A Black-Box Scrubbing Attack on LLM Watermarks,”arXiv preprint arXiv:2411.01222, 2024. Wenrui Xureceived a B.S. degree in Computer Engineering from the University of Minnesota, MN, USA, in 2023. He is currently pursuing a Ph.D. degree in Elec...

    2024 arXiv

  82. [2023]

    Available:\url{https://www.alignmentforum.org/posts/ pNcFYZnPdXyL2RfgA/using-gpt-eliezer-against-chatgpt-jailbreaking}

    [Online]. Available:\url{https://www.alignmentforum.org/posts/ pNcFYZnPdXyL2RfgA/using-gpt-eliezer-against-chatgpt-jailbreaking}

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.