Pith. sign in

REVIEW 3 major objections 4 minor 107 references

Securing AI Systems: A Guide to Known Attacks and Impacts

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper identifies eleven major attack types against AI systems and maps each to concrete confidentiality, integrity, or availability impacts.

desk verdict A useful practitioner-oriented taxonomy undermined by an internally inconsistent reverse-lookup table. read the letter →

arxiv 2506.23296 v1 pith:2RJYQHZL submitted 2025-06-29 cs.CR cs.AI

Naoto Kiribuchi , Kengo Zenitani , Takayuki Semitsu This is my paper
classification cs.CRcs.AI
keywords AISecurityAdversarialMachineLearningPredictiveGenerativeCIAtriadPromptInjectionDataPoisoningModelExtraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is a practical guide rather than a research advance: it organizes the sprawling literature on attacks unique to AI systems into eleven attack types (A–K) grouped by technical mechanism, and explicitly connects each to the damage it can do. The damages are consolidated into eleven impacts—ranging from model leakage and training-data leakage to safeguard bypass, system compromise, and computational waste—and mapped onto the confidentiality, integrity, and availability (CIA) triad. The intended payoff is that a researcher, developer, security practitioner, or policymaker without specialized AI security expertise can recognize which attack is happening from an observed failure and decide where to focus defenses. The paper also supplies attacker-capability summaries and reverse-lookup tables so the taxonomy can be used in risk assessment and red-teaming.

What carries the argument

The load-bearing object is the taxonomy itself: labels A through K for attacks and 1 through 11 for impacts, anchored to a general two-environment AI system architecture that includes training and fine-tuning data flows, plugins, retrieval-augmented generation, internal knowledge, and external systems. That architecture lets the authors map each attack to a concrete point of entry, and the reverse-lookup table lets practitioners invert the mapping from observed impact back to candidate attacks. The choice to name attacks by technical mechanism (for example, 'prompt injection' rather than 'jailbreak') is what keeps the categories non-overlapping in the authors' design.

What would settle it

Pick any published attack and ask whether the guide's rules assign it a unique attack type and all and only the impacts listed in Table 4; the internal mismatch already visible—Training Data Leakage appears as impact 2 for C, D, and J in Table 4 and Figure 4, but Table 3 lists only B for that impact—would, if reproduced more widely, show the mapping does not yet support reliable reverse lookup.

Watch

Extended reading notes

Core claim

The central claim is that nearly all published AI-specific attacks can be captured by eleven major types: model extraction; training-data-related information gathering (membership inference, attribute inference, property inference, model inversion, data reconstruction, and data extraction); model poisoning; data poisoning; evasion; energy-latency; prompt stealing; prompt injection; code injection; adversarial fine-tuning; and rowhammer. The paper's distinctive move is to separate the attack's technical mechanism from its objective: rather than classifying attacks by attacker goal (which the authors argue causes overlaps and duplicates in existing taxonomies), it classifies by how the attack works, then links each mechanism to the impacts it produces. Impacts are grouped into eleven categories under the CIA triad, and the paper provides a reverse lookup from impact to attack, a mapping diagram that shows cascading effects (notably system compromise enabling further attacks), and a table of demonstrated attacks by data modality, model type, and attacker capability.

Load-bearing premise

The taxonomy's usefulness rests on the judgment, made while grouping published attack descriptions by technical mechanism in Section 4.1, that these eleven clusters are the ones practitioners will find meaningful; the paper does not validate that grouping against a ground-truth corpus.

Editorial extensions

If this is right

  • A security team that sees training-data leakage in a deployed system can look up impact 2 in the reverse table and immediately get a short list of attacks to investigate, without reading the full literature.
  • Red-teaming exercises can be structured as one scenario per attack type, with the attacker-capability column (massive queries, training-data injection, physical memory access) shaping what each scenario may assume.
  • Because system compromise is identified as a cascading impact that can enable any other attack, organizations should treat prompt-injection and code-injection defenses as high-priority control points.
  • The summary table over data modalities and model types lets a practitioner judge whether a given attack has been demonstrated on their kind of system, while the paper notes these demonstrations are not inherent limits.
  • Policymakers and standards bodies gain a simplified vocabulary for discussing AI-specific threats that complements rather than replaces existing detailed taxonomies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reverse-lookup table is internally inconsistent: impact 2 (training-data leakage) is assigned only to attack B in Table 3, while Table 4 and Figure 4 assign it also to C, D, and J; a reader relying on the lookup could miss those attacks, so the mapping would need reconciliation before use in high-stakes assessments.
  • The taxonomy could be turned into a concrete test suite: for each of the eleven types, generate a small probe (for example, a prompt-injection payload or a membership-inference query) and walk the reverse mapping to see whether the observed impact matches the paper's predicted impact.
  • As AI systems increasingly rely on retrieval-augmented generation and plugins, the prompt-injection category will likely need finer sub-categories than the paper provides, since the attack surface (external knowledge, internal knowledge, plugins) differs in defender control.
  • Because the eleven types are drawn from published attacks, the taxonomy is a snapshot; a genuinely new attack class that does not fit any of A–K would be a natural test of whether mechanism-based grouping remains stable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This manuscript is a practitioner-oriented survey and guide that proposes a taxonomy of eleven AI-specific attack types (labeled A through K), eleven distinct impacts (labeled 1 through 11), and an explicit mapping between attacks and impacts organized by the CIA security triad. The paper includes a reverse-lookup table (Table 3), detailed descriptions of each attack in Section 5, a literature summary of demonstrated attacks with attacker capabilities, modalities, and model types (Table 4), and comparisons with NIST AI 100-2e2025, MITRE ATLAS, OWASP Top 10 for LLMs, and the AI Quality Management Guideline. The stated purpose is to help researchers, developers, security practitioners, and policymakers without specialized AI security expertise recognize AI-specific attack vectors and use the mapping for risk assessment and red-teaming.

Significance. If the attack-impact mapping were internally consistent, this would be a genuinely useful complement to existing frameworks: the mechanism-based grouping is intuitive, the visual summaries (Figures 1, 3, 4) are helpful for non-specialists, and the paper incorporates recent work (e.g., adversarial fine-tuning, rowhammer-based attacks, prompt stealing) that is not always covered in NIST and ATLAS. The paper is honest about its scope, explicitly disclaiming completeness in Section 3.2, and it provides a substantial reference list and explicit comparisons with established taxonomies. However, the central contribution of the paper—the explicit, actionable link between attacks and impacts—is undermined by internal inconsistencies between Table 3, Table 4, and Figure 4. Because the reverse-lookup table is the operational artifact intended for risk assessment and red-teaming, these inconsistencies are load-bearing and prevent the manuscript from being usable as a reliable reference in its current form.

major comments (3)
  1. [§4.3 (Table 3) vs. Table 4, Figure 4, §§5.3, 5.4, 5.10] The reverse-lookup Table 3 lists only Attack B (Training Data-related Information Gathering) under Impact 2 (Training Data Leakage), but Table 4 lists Attack C (Model Poisoning), Attack D (Data Poisoning), and Attack J (Adversarial Fine-tuning) as also causing Impact 2, and the corresponding text in Sections 5.3, 5.4, and 5.10 supports those assignments. Figure 4 also connects C, D, and J to Impact 2. Since Table 3 is presented as the direct lookup for practitioners performing risk assessment and red-teaming, these omissions mean a user observing training-data leakage would overlook attacks that the paper itself establishes as relevant. The three artifacts should be harmonized so that the reverse lookup is consistent with the detailed tables and the figure.
  2. [§4.3 (Table 3) vs. Figure 4 note] Table 3 places Impact 9 (System Compromise) only under Availability, with Attacks H and I, but Figure 4's note explicitly states that System Compromise 'can impact not only integrity but also confidentiality and availability.' This means a practitioner using Table 3 to investigate a confidentiality or integrity incident would not be directed to the prompt injection and code injection attacks that the paper itself identifies as possible causes of such an incident. The note in Figure 4 should be reflected in Table 3, for example by listing Impact 9 under all three CIA columns or by adding an explicit cross-reference.
  3. [§4.1 Methodology] The taxonomy is built entirely on the subjective grouping procedure described in Section 4.1: attacks were extracted, mapped to the architecture, grouped into eleven types, and linked to impacts without any stated inclusion or exclusion criteria, inter-annotator validation, or comparison against an alternative grouping. The paper asserts that these are the 'eleven major attack types' but does not provide a principled basis for that selection. This is not necessarily a fatal flaw for a survey-style guide, but the authors should either explicitly frame the eleven types as a proposal based on expert judgment, or provide concrete criteria (e.g., frequency in literature, severity, novelty relative to conventional cybersecurity) that would let practitioners judge the coverage and transferability of the taxonomy.
minor comments (4)
  1. [Table 4, §5.8] In the row for Attack H (Prompt Injection), the impact listed as '2: Input Info Leakage' appears to be a numbering error: Impact 2 is Training Data Leakage, whereas the intended impact here is almost certainly Impact 4 (Input Information Leakage). Please verify all impact numbers in Table 4.
  2. [§2.2] The phrase 'we deliberately adopts a simplified taxonomy' is a subject-verb agreement error; it should be 'we deliberately adopt.'
  3. [§5.5] The text contains 'significantmodel malfunction' with a missing space; it should be 'significant model malfunction.'
  4. [References [23] and [89]] Reference [23] contains the typo 'Guidline' instead of 'Guideline,' and reference [89] contains 'Artficial' instead of 'Artificial.'

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the taxonomy is a literature-review synthesis from external frameworks and primary sources, with only minor non-load-bearing self-citations.

full rationale

The paper does not attempt a mathematical or statistical derivation, so there are no fitted parameters renamed as predictions and no equation-level self-definition. Its central claim is a qualitative taxonomy: eleven attack types and eleven impacts linked through the CIA triad, built by the explicit literature-review procedure in Section 4.1 from external frameworks (NIST AI 100-2e2025, MITRE ATLAS, OWASP, AIQM) and primary attack literature (e.g., references [13, 18, 28, 46, 79, 83, 100, 103]). The attack-to-impact mappings in Table 4 and Figure 4 are summaries of cited sources, not consequences of definitions that presuppose the conclusion. The only self-references are the predecessor J-AISI document [40] and the J-AISI red-teaming guide [38]; the paper itself states it is an extended version of [40], and Section 2.6 describes [38] as a preliminary process-oriented guide that this paper complements. Neither self-citation carries the load-bearing content, which instead rests on the external NIST/ATLAS correspondences in Tables 1 and 2 and on independently published experimental results. The internal inconsistency between Table 3 and Table 4/Figure 4 regarding which attacks cause Impact 2 (Training Data Leakage) and the scope of Impact 9 (System Compromise) is a factual and consistency defect in the reverse-lookup table, not a circular reduction: it does not make the taxonomy equivalent to its inputs. Therefore no circular step is identified; the score reflects only the presence of minor, non-load-bearing self-citations, with the substantive derivation remaining self-contained against external benchmarks.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central contribution is a taxonomy, so the main load-bearing inputs are domain assumptions about scope and architecture rather than mathematical axioms or fitted parameters. No free parameters or invented entities are introduced.

assumptions (4)
  • domain assumption General cybersecurity practices, such as protection against unauthorized network access, are already in place.
    Stated in Section 3.2; the paper explicitly narrows its scope to AI-specific attacks, so if this assumption fails, the taxonomy's coverage is incomplete for real systems.
  • domain assumption The generalized AI system architecture in Figure 2, with training and operation environments plus plugins and RAG, adequately represents the systems under consideration.
    Stated in Section 3.1; the authors note that federated learning and other advanced architectures do not fit, so the taxonomy's applicability is bounded by this architecture.
  • ad hoc to paper The selected eleven attacks are the major AI-specific attack types worth including.
    The selection criteria in Section 4.1 are based on the authors' literature review rather than an objective completeness criterion; the paper itself says it does not assert completeness in Section 3.2.
  • domain assumption The modalities and model types listed in Table 4 reflect demonstrated attacks and do not represent inherent limitations.
    Stated before Table 4; the reverse inference that absence from Table 4 means an attack cannot apply is not justified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Securing AI Systems: A Guide to Known Attacks and Impacts." pith.science (2026). https://pith.science/paper/2RJYQHZL

@misc{pith2026250623296,
  author       = {Pith},
  title        = {Pith review of: Securing AI Systems: A Guide to Known Attacks and Impacts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2RJYQHZL}},
  note         = {Machine review of arXiv:2506.23296}
}
read the original abstract

Embedded into information systems, artificial intelligence (AI) faces security threats that exploit AI-specific vulnerabilities. This paper provides an accessible overview of adversarial attacks unique to predictive and generative AI systems. We identify eleven major attack types and explicitly link attack techniques to their impacts -- including information leakage, system compromise, and resource exhaustion -- mapped to the confidentiality, integrity, and availability (CIA) security triad. We aim to equip researchers, developers, security practitioners, and policymakers, even those without specialized AI security expertise, with foundational knowledge to recognize AI-specific risks and implement effective defenses, thereby enhancing the overall security posture of AI systems.

Figures

Figures reproduced from arXiv: 2506.23296 by the authors.

Figure 1
Figure 1. Overview of known eleven major attacks (A–K) on AI systems. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Assumed AI system architecture. The diagram illustrates the seamless data flow from model training [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Overview of impacts caused by known attacks (described in Figure [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Relationship between the known attacks and their impacts on AI systems. The dashed line indicates [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Attack A: Model Extraction (Overview) [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Attack B: Training Data-related Information Gathering (Overview) [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Attack C: Model Poisoning (Overview) 5.3 C: Model Poisoning Model poisoning attacks directly manipulate the AI model itself or its training program, rather than altering the training data, to compromise the model’s behavior. In contrast to data poisoning, where malicio…
Figure 8
Figure 8. Figure 8: Attack D: Data Poisoning (Overview) Benign Image Poisoned Image Trigger Pattern Target Label Bird Visible Attack Bird Car Invisible Attack Poison-label Clean-label Optimized Attack Bird Semantic Attack Car Goldfish Goldfish Sample-Specific Attack Bird Physical Attack F…
Figure 9
Figure 9. Figure 9: An example of poisoned samples generated by different types of backdoor poisoning. Quotation [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: ). Adversarial examples are generated by introducing subtle perturbations into legitimate data, designed to trick the model into making incorrect predictions. Adversarial examples can be generated under two primary settings: • White-box setting: An attacker has comple…
Figure 11
Figure 11. Figure 11: Attack F: Energy-latency (Overview) [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Attack G: Prompt Stealing (Overview) [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Attack H: Prompt Injection (Overview) [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Attack I: Code Injection (Overview) injection into AI models, whereby malicious executable code is embedded within models and is executed upon model loading. In contrast to prompt injection, which manipulates model inputs during inference, code injection embeds malici…
Figure 15
Figure 15. Figure 15: Attack J: Adversarial Fine-tuning (Overview) [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]
Figure 16
Figure 16. Figure 16: Attack K: Rowhammer (Overview) 6 Conclusions In this paper, we have provided an overview of known adversarial attacks specifically targeting AI systems, detailing their underlying mechanisms, impacts, and the attacker’s capabilities. By categorizing these attacks acco…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

107 extracted references · 24 canonical work pages

  1. [1]

    Monika Adamczyk, Nineta Polemi, Isabel Praça, and Konstantinos Moulinos. 2023. A multilayer framework for good cybersecurity practices for AI : security and resilience for smart health services and infrastructures . Technical Report. European Union Agency for Cybersecurity. doi:10.2824/588830

    doi:10.2824/588830 2023

  2. [2]

    Salem Ahmed, Zhang Yang, Humbert Mathias, Berrang Pascal, Fritz Mario, and Backes Michael. 2019. ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models. In Proceedings 2019 Network and Distributed System Security Symposium (San Diego, CA, USA) (NDSS 2019). Internet Society, Reston, VA, USA. doi:10.14722/n...

    arXiv 2019

  3. [3]

    Ahmed Aldahdooh, Wassim Hamidouche, Sid Ahmed Fezza, and Olivier Déforges. 2022. Adversarial Example Detection for DNN Models: A Review and Experimental Comparison. Artificial Intelligence Review 55, 6 (Aug. 2022), 4403–4462. doi:10.1007/s10462-021-10125-w

    work page doi:10.1007/s10462-021-10125-w 2022

  4. [4]

    Mancini, Angelo Spognardi, Antonio Villani, Domenico Vitali, and Giovanni Felici

    Giuseppe Ateniese, Luigi V. Mancini, Angelo Spognardi, Antonio Villani, Domenico Vitali, and Giovanni Felici. 2015. Hacking Smart Machines with Smarter Ones: How to Extract Meaningful Data from Machine Learning Classifiers. International Journal of Security and Networks 10, 3 (Sept. 2015), 137–150. doi:10.1504/IJSN.2015.071829

    work page arXiv 2015

  5. [7]

    Gon Buzaglo, Niv Haim, Gilad Yehudai, Gal Vardi, and Michal Irani. 2023. Reconstructing Training Data from Multiclass Neural Networks. arXiv:2305.03350 [cs.LG]

    work page Pith review arXiv 2023

  6. [8]

    Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. 2021. IPGuard: Protecting Intellectual Property of Deep Neural Networks via Fingerprinting the Classification Boundary. InProceedings of the 2021 ACM Asia Conference on Computer and Communications Security (Virtual Event, Hong Kong) (ASIA CCS ’21). Association for Computing Machinery, New York, NY, USA, 14...

    arXiv 2021

  7. [9]

    Nicholas Carlini. 2021. Poisoning the Unlabeled Dataset of Semi-Supervised Learning. In 30th USENIX Security Symposium (USENIX Security 21) . USENIX Association, Berkeley, CA, USA, 1577–1592. https://www.usenix.org/ conference/usenixsecurity21/presentation/carlini-poisoning

    2021

  8. [10]

    Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. 2023. Extracting Training Data from Diffusion Models. In32nd USENIX Security Symposium (USENIX Security 23) (Anaheim, CA). USENIX Association, Berkeley, CA, USA, 5253–5270. https://www.usenix.org/ conference/usenixsec...

    2023

Show all 107 references
  1. [11]

    Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks. In 28th USENIX Security Symposium (USENIX Security 19) (Santa Clara, CA, USA). USENIX Association, Berkeley, CA...

    2019

  2. [12]

    Choquette-Choo, Matthew Jagielski, Irena Gao, Pang Wei W Koh, Daphne Ippolito, Florian Tramer, and Ludwig Schmidt

    Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Pang Wei W Koh, Daphne Ippolito, Florian Tramer, and Ludwig Schmidt. 2023. Are aligned neural networks adversarially aligned?. In Advances in Neural Information Processing Systems (New O...

    2023

  3. [13]

    Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting Training Data from Large Language Models. In 30th USENIX Security Symposi...

    2021

  4. [14]

    National Cyber Security Centre. 2024. Machine learning principles. https://www.ncsc.gov.uk/collection/machine- learning-principles

    2024

  5. [15]

    National Cyber Security Centre, Cybersecurity and Infrastructure Security Agency, National Security Agency, Federal Bureau of Investigation, Australian Signals Directorate’s Australian Cyber Security Centre, Canadian Centre for Cyber Security, New Zealand National Cyber Securi...

    2023

  6. [16]

    Pappas, and Eric Wong

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. 2024. Jailbreaking Black Box Large Language Models in Twenty Queries. arXiv:2310.08419 [cs.LG]

    2024 arXiv

  7. [17]

    Ali Babar

    Huaming Chen and M. Ali Babar. 2024. Security for Machine Learning-based Software Systems: A Survey of Threats, Practices, and Challenges. ACM Comput. Surv. 56, 6, Article 151 (Feb. 2024), 38 pages. doi:10.1145/3638531

    2024 doi

  8. [18]

    Xiaoyi Chen, Siyuan Tang, Rui Zhu, Shijun Yan, Lei Jin, Zihao Wang, Liya Su, Zhikun Zhang, XiaoFeng Wang, and Haixu Tang. 2024. The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Risks. In Proceedings of the 2024 on ACM SIGSAC Conference on Com...

    2024

  9. [19]

    Antonio Emanuele Cinà, Ambra Demontis, Battista Biggio, Fabio Roli, and Marcello Pelillo. 2023. Energy-Latency Attacks via Sponge Poisoning. arXiv:2203.08147 [cs.CR]

    2023 arXiv

  10. [20]

    Berriel, Claudine Badue, Alberto F

    Jacson Rodrigues Correia-Silva, Rodrigo F. Berriel, Claudine Badue, Alberto F. de Souza, and Thiago Oliveira-Santos

  11. [21]

    Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. 2021. RobustBench: a standardized adversarial robustness benchmark. In Thirty-fifth Conference on Neural Information Processing Syste...

    2021

  12. [22]

    Hadi Amini, and Yanzhao Wu

    Badhan Chandra Das, M. Hadi Amini, and Yanzhao Wu. 2025. Security and Privacy Challenges of Large Language Models: A Survey. ACM Comput. Surv. 57, 6, Article 152 (Feb. 2025), 39 pages. doi:10.1145/3712001

    2025 doi

  13. [23]

    Digital Architecture Research Center, Cyber Physical Security Research Center, and Artificial Intelligence Research Center. 2024. Machine Learning Quality Management Guidline (Revision 4.2.0) . Technical Report. National Institute of Advanced Industrial Science and Technology ...

    2024

  14. [24]

    Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus-Robert Müller, and Pan Kessel. 2019. Explanations can be manipulated and geometry is to blame. In Advances in Neural Information Processing Systems (Vancouver, Canada), H. Wallach, H. Laroch...

    2019

  15. [25]

    Zhichen Dong, Zhanhui Zhou, Chao Yang, Jing Shao, and Yu Qiao. 2024. Attacks, Defenses and Evaluations for LLM Conversation Safety: A Survey. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Te...

    2024 doi

  16. [26]

    Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-Box Adversarial Examples for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (Melbourne, Australia), Iryna Gurev...

    2018 doi

  17. [27]

    Aidan Ewart, Abhay Sheshadri, Phillip Huang Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, and Stephen Casper. 2024. Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs. (2...

    2024

  18. [28]

    Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. 2015. Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (Denver, Colorado, USA) (CCS ’15). Associat...

    2015

  19. [29]

    Gunter, and Nikita Borisov

    Karan Ganju, Qi Wang, Wei Yang, Carl A. Gunter, and Nikita Borisov. 2018. Property Inference Attacks on Fully Connected Neural Networks using Permutation Invariant Representations. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (Toront...

    2018

  20. [30]

    Goodfellow, Jonathon Shlens, and Christian Szegedy

    Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. arXiv:1412.6572 [stat.ML] 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings

    2015 arXiv

  21. [31]

    Justin Grana. 2020. Perturbing Inputs to Prevent Model Stealing. In 2020 IEEE Conference on Communications and Network Security (CNS) (Avignon, France). IEEE, New York, NY, USA, 1–9. doi:10.1109/CNS48642.2020.9162336

    2020

  22. [32]

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. In Proceedings of the 16th ACM Workshop on Artificial Inte...

    2023

  23. [33]

    Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. 2021. Gradient-based Adversarial Attacks against Text Transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (Punta Cana, Dominican Republic), Marie-Francine Moens...

    2021 doi

  24. [34]

    Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, and Michal Irani. 2022. Reconstructing Training Data from Trained Neural Networks. In Proceedings of the 36th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’22). Curran Associates...

    2022

  25. [35]

    Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. 2024. WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs. In The Thirty-eight Conference on Neural Information Process...

    2024

  26. [36]

    Keegan Hines, Gary Lopez, Matthew Hall, Federico Zarfati, Yonatan Zunger, and Emre Kıcıman. 2024. Defending Against Indirect Prompt Injection Attacks With Spotlighting. In Conference on Applied Machine Learning for Informa- tion Security (CAMLIS 2024) (Arlington, VA, USA), Rac...

    2024

  27. [37]

    Yupeng Hu, Wenxin Kuang, Zheng Qin, Kenli Li, Jiliang Zhang, Yansong Gao, Wenjia Li, and Keqin Li. 2021. Artificial Intelligence Security: Threats and Countermeasures. ACM Comput. Surv. 55, 1, Article 20 (Nov. 2021), 36 pages. doi:10.1145/3487890

    2021 doi

  28. [38]

    Japan AI Safety Institute (J-AISI). 2024. Guide to Red Teaming Methodology on AI Safety. https://aisi.go.jp/output/ output_framework/guide_to_red_teaming_methodology_on_ai_safety/

    2024

  29. [39]

    Japan AI Safety Institute (J-AISI). 2024. Overview of the Japan AI Safety Institute (J-AISI). https://aisi.go.jp/about/

    2024

  30. [40]

    Japan AI Safety Institute (J-AISI). 2025. Known Attacks and Their Impacts on AI Systems. https://aisi.go.jp/output/ output_security/known_attacks_and_impacts/

    2025

  31. [41]

    Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, and Bo Li. 2018. Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning. In 2018 IEEE Symposium on Security and Privacy (SP) (San Francisco, CA, USA). IEEE, ...

    2018

  32. [42]

    Mika Juuti, Sebastian Szyller, Samuel Marchal, and N. Asokan. 2019. PRADA: Protecting Against DNN Model Stealing Attacks. In 2019 IEEE European Symposium on Security and Privacy (EuroS&P) (Stockholm, Sweden). IEEE Computer Society, Los Alamitos, CA, USA, 512–527. doi:10.1109/E...

    2019

  33. [43]

    Yusuke Kawamoto, Kazumasa Miyake, Koichi Konishi, and Yutaka Oiwa. 2023. Threats, Vulnerabilities, and Controls of Machine Learning Based Systems: A Survey and Taxonomy. arXiv:2301.07474 [cs.CR] https://arxiv.org/abs/2301.07474

    2023 arXiv

  34. [44]

    Zixiao Kong, Jingfeng Xue, Yong Wang, Lu Huang, Zequn Niu, and Feng Li. 2021. A Survey on Adversarial Attack in the Age of Artificial Intelligence. Wireless Communications and Mobile Computing 2021, 1 (2021), 4907754. doi:10.1155/2021/4907754

    2021 doi

  35. [45]

    Peter Kyle. 2025. Tackling AI security risks to unleash growth and deliver Plan for Change. https://www.gov.uk/ government/news/tackling-ai-security-risks-to-unleash-growth-and-deliver-plan-for-change Secretary of State for Science, Innovation and Technology, UK Government. Se...

    2025

  36. [46]

    Shaofeng Li, Xinyu Wang, Minhui Xue, Haojin Zhu, Zhi Zhang, Yansong Gao, Wen Wu, and Xuemin (Sherman) Shen

  37. [47]

    Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. 2024. Backdoor Learning: A Survey. IEEE Transactions on Neural Networks and Learning Systems 35, 1 (2024), 5–22. doi:10.1109/TNNLS.2022.3182979 arXiv:2007.08745 [cs.CR]

    2024

  38. [48]

    Zongjie Li, Chaozheng Wang, Shuai Wang, and Cuiyun Gao. 2023. Protecting Intellectual Property of Large Language Model-Based Code Generation APIs via Watermarks. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (Copenhagen, Denmark) (CCS...

    2023

  39. [49]

    Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. 2024. Prompt Injection attack against LLM-integrated Applications. arXiv:2306.05499 [cs.CR]

    2024 arXiv

  40. [50]

    Yugeng Liu, Rui Wen, Xinlei He, Ahmed Salem, Zhikun Zhang, Michael Backes, Emiliano De Cristofaro, Mario Fritz, and Yang Zhang. 2022. ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models. In 31st USENIX Security Symposium (USENIX Security 22...

    2022

  41. [51]

    Gunter, and Kai Chen

    Yunhui Long, Vincent Bindschaedler, Lei Wang, Diyue Bu, Xiaofeng Wang, Haixu Tang, Carl A. Gunter, and Kai Chen. 2018. Understanding Membership Inferences on Well-Generalized Learning Models. arXiv:1802.04889 [cs.CR]

    2018 arXiv

  42. [52]

    Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-Béguelin. 2023. Analyzing Leakage of Personally Identifiable Information in Language Models. In 2023 IEEE Symposium on Security and Privacy (SP) (San Francisco, CA, USA). IEEE, New York, N...

    2023

  43. [53]

    Shiqing Ma, Yingqi Liu, Guanhong Tao, Wen-Chuan Lee, and Xiangyu Zhang. 2019. NIC: Detecting Adversarial Samples with Neural Network Invariant Checking. In Proceedings 2019 Network and Distributed System Security Symposium (San Diego, CA, USA) (NDSS 2019). Internet Society, Re...

    2019

  44. [54]

    Saeed Mahloujifar, Esha Ghosh, and Melissa Chase. 2022. Property Inference from Poisoning. In2022 IEEE Symposium on Security and Privacy (SP) (San Francisco, CA, USA). IEEE, New York, NY, USA, 1120–1137. doi:10.1109/SP46214. 2022.9833623

    2022

  45. [55]

    Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. 2024. Tree of Attacks: Jailbreaking Black-Box LLMs Automatically. arXiv:2312.02119 [cs.LG]

    2024 arXiv

  46. [56]

    MITRE. 2013. MITRE ATT&CK ®. https://attack.mitre.org/

    2013

  47. [57]

    MITRE. 2021. MITRE ATLAS ™. https://atlas.mitre.org/

    2021

  48. [58]

    Sasi Kumar Murakonda and Reza Shokri. 2020. ML Privacy Meter: Aiding Regulatory Compliance by Quantifying the Privacy Risks of Machine Learning. (2020). arXiv:2007.09339 [cs.CR] Workshop on Hot Topics in Privacy Enhancing Technologies (HotPETs)

    2020 arXiv

  49. [59]

    Milad Nasr, Reza Shokri, and Amir Houmansadr. 2019. Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning. In 2019 IEEE Symposium on Security and Privacy (SP) (San Francisco, CA, USA). IEEE, N...

    2019

  50. [60]

    Khasawneh, and Houman Homayoun

    Najmeh Nazari, Hosein Mohammadi Makrani, Chongzhou Fang, Hossein Sayadi, Setareh Rafatirad, Khaled N. Khasawneh, and Houman Homayoun. 2024. Forget and Rewire: Enhancing the Resilience of Transformer-based Models against Bit-Flip Attacks. In 33rd USENIX Security Symposium (USEN...

    2024

  51. [61]

    Molloy, and Ben Edwards

    Maria-Irina Nicolae, Mathieu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wistuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, Ian M. Molloy, and Ben Edwards. 2019. Adversarial Robustness Toolbox v1.0.0. arXiv:1807.01069 [cs.LG]

    2019 arXiv

  52. [62]

    Stavros Ntalampiras, Corina Pascu, Marco Barros Lourenco, Gianluca Misuraca, and Pierre Rossel. 2023. Artificial intelligence and cybersecurity research . Technical Report. European Union Agency for Cybersecurity. doi:10.2824/ 808362

    2023

  53. [63]

    Department of Commerce

    U.S. Department of Commerce. 2025. Statement from U.S. Secretary of Commerce Howard Lutnick on Trans- forming the U.S. AI Safety Institute into the Pro-Innovation, Pro-Science U.S. Center for AI Standards and Innova- tion. https://www.commerce.gov/news/press-releases/2025/06/s...

    2025

  54. [64]

    National Institute of Standards and Technology. 2023. Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1. doi:10.6028/NIST.AI.100-1

    2023 doi

  55. [65]

    National Institute of Standards and Technology. 2023. NIST AI RMF Playbook. https://airc.nist.gov/airmf-resources/ playbook/ 30 Naoto Kiribuchi, Kengo Zenitani, and Takayuki Semitsu

    2023

  56. [66]

    The Leaders of the Group of Seven (G7). 2023. Hiroshima AI Process. https://www.soumu.go.jp/hiroshimaaiprocess/ en/

    2023

  57. [67]

    Seong Joon Oh, Bernt Schiele, and Mario Fritz. 2019. Towards Reverse-Engineering Black-Box Neural Networks. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning , Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller (...

    2019 doi

  58. [68]

    Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. 2019. Knockoff Nets: Stealing Functionality of Black-Box Models. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (Long Beach, CA, USA). IEEE, New York, NY, USA, 4949–4958. doi:10.1109/CVPR.2019.00509

    2019

  59. [69]

    Ayodeji Oseni, Nour Moustafa, Helge Janicke, Peng Liu, Zahir Tari, and Athanasios Vasilakos. 2021. Security and Privacy for Artificial Intelligence: Opportunities and Challenges. arXiv:2102.04661 [cs.CR]

    2021 arXiv

  60. [70]

    Nicolas Papernot, Fartash Faghri, Nicholas Carlini, Ian Goodfellow, Reuben Feinman, Alexey Kurakin, Cihang Xie, Yash Sharma, Tom Brown, Aurko Roy, Alexander Matyasko, Vahid Behzadan, Karen Hambardzumyan, Zhishuai Zhang, Yi-Lin Juang, Zhi Li, Ryan Sheatsley, Abhibhav Garg, Jona...

    2018 arXiv

  61. [71]

    Berkay Celik, and Ananthram Swami

    Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. 2017. Practical Black-Box Attacks against Machine Learning. InProceedings of the 2017 ACM on Asia Conference on Computer and Communications Security (Abu Dhabi, United Arab Em...

    2017

  62. [72]

    Coimbra, Daniel Castro, Paulo Carreira, and Nuno Santos

    Rodrigo Pedro, Miguel E. Coimbra, Daniel Castro, Paulo Carreira, and Nuno Santos. 2025. Prompt-to-SQL Injections in LLM-Integrated Web Applications: Risks and Defenses. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) (Ottawa, ON, Canada). IEEE Comp...

    2025

  63. [73]

    Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. Red Teaming Language Models with Language Models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing...

    2022 doi

  64. [74]

    Adnan Siraj Rakin, Md Hafizul Islam Chowdhuryy, Fan Yao, and Deliang Fan. 2022. DeepSteal: Advanced Model Extractions Leveraging Efficient Weight Stealing in Memories. In 2022 IEEE Symposium on Security and Privacy (SP) (San Francisco, CA, USA). IEEE, New York, NY, USA, 1157–1...

    2022

  65. [75]

    Roman Samoilenko. 2023. New prompt injection attack on ChatGPT web version. Reckless copy-pasting may lead to serious privacy issues in your chat. Netograph. Retrieved April 25, 2025 from https://kajojify.github.io/articles/1_ chatgpt_attack.pdf

    2023

  66. [76]

    Oscar Schwartz. 2019. In 2016, Microsoft’s Racist Chatbot Revealed the Dangers of Online Conversation. IEEE Spectrum Nov. (2019). https://spectrum.ieee.org/in-2016-microsofts-racist-chatbot-revealed-the-dangers-of-online- conversation Updated: Jan. 2024

    2019

  67. [77]

    Erfan Shayegani, Md Abdullah Al Mamun, Yu Fu, Pedram Zaree, Yue Dong, and Nael Abu-Ghazaleh. 2023. Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks. arXiv:2310.10844 [cs.CL]

    2023 arXiv

  68. [78]

    Do Anything Now

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security (Salt ...

    2024

  69. [79]

    Xinyue Shen, Yiting Qu, Michael Backes, and Yang Zhang. 2024. Prompt Stealing Attacks Against Text-to-Image Generation Models. In 33rd USENIX Security Symposium (USENIX Security 24) (Philadelphia, PA, USA). USENIX Association, Berkeley, CA, USA, 5823–5840. https://www.usenix.o...

    2024

  70. [80]

    Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership Inference Attacks Against Machine Learning Models. In 2017 IEEE Symposium on Security and Privacy (SP) (San Jose, CA, USA). IEEE, New York, NY, USA, 3–18. doi:10.1109/SP.2017.41

    2017 doi

  71. [81]

    Ilia Shumailov, Yiren Zhao, Daniel Bates, Nicolas Papernot, Robert Mullins, and Ross Anderson. 2021. Sponge Examples: Energy-Latency Attacks on Neural Networks. In 2021 IEEE European Symposium on Security and Privacy (EuroS&P) (Vienna, Austria). IEEE, New York, NY, USA, 212–23...

    2021

  72. [82]

    Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. 2020. Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (New York, NY, USA) (AIES ’20). Association for...

    2020

  73. [83]

    Congzheng Song, Thomas Ristenpart, and Vitaly Shmatikov. 2017. Machine Learning Models that Remember Too Much. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (Dallas, Texas, USA) (CCS ’17). Association for Computing Machinery, New York...

    2017

  74. [84]

    Congzheng Song and Vitaly Shmatikov. 2020. Overlearning Reveals Sensitive Attributes. arXiv:1905.11742 [cs.LG] https://openreview.net/forum?id=SJeNz04tDS 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020

    2020 arXiv

  75. [85]

    Hui Sun, Tianqing Zhu, Zhiqiu Zhang, Dawei Jin, Ping Xiong, and Wanlei Zhou. 2023. Adversarial Attacks Against Deep Generative Models on Data: A Survey. IEEE Transactions on Knowledge & Data Engineering 35, 04 (April 2023), 3367–3388. doi:10.1109/TKDE.2021.3130903

    2023

  76. [86]

    Goodfellow, and Rob Fergus

    Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus

  77. [87]

    Reiter, and Thomas Ristenpart

    Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. 2016. Stealing Machine Learning Models via Prediction APIs. In 25th USENIX Security Symposium (USENIX Security 16) (Austin, TX, USA). USENIX Association, Berkeley, CA, USA, 601–618. https://www.use...

    2016

  78. [88]

    Andrew van der Stock, Brian Glas, Neil Smithline, and Torsten Gigler. 2021. OWASP Top 10 (Version 2021). https: //owasp.org/Top10/

    2021

  79. [89]

    Apostol Vassilev, Alina Oprea, Alie Fordyce, and Hyrum Anderson. 2024. Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations. Technical Report NIST Artifcial Intelligence (AI) Report, NIST Trustworthy and Responsible AI NIST AI 100-2e2023. Nationa...

    2024 doi

  80. [90]

    Apostol Vassilev, Alina Oprea, Alie Fordyce, Hyrum Anderson, Xander Davies, and Maia Hamin. 2025. Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations . Technical Report NIST Trustworthy and Responsible AI, NIST AI 100-2e2025. National Institute ...

    2025 doi

  81. [91]

    Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal Adversarial Triggers for Attacking and Analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on...

    2019 doi

  82. [92]

    Binghui Wang and Neil Zhenqiang Gong. 2018. Stealing Hyperparameters in Machine Learning. In 2018 IEEE Symposium on Security and Privacy (SP) (San Francisco, CA, USA). IEEE, New York, NY, USA, 36–52. doi:10.1109/SP. 2018.00038

    2018

  83. [93]

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How Does LLM Safety Training Fail?. In Advances in Neural Information Processing Systems (New Orleans, LA, USA), A. Oh, T. Naumann, A. Glober- son, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran...

    2023

  84. [94]

    Steve Wilson and Ads Dawson. 2024. OWASP Top 10 for Large Language Model Applications (Version 2025). https://owasp.org/www-project-top-10-for-large-language-model-applications/

    2024

  85. [95]

    Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. 2023. Defending chatGPT against Jailbreak Attack via Self-Reminders. Nature Machine Intelligence 5, 12 (2023), 1486–1496. doi:10.1038/s42256-023-00765-8

    2023 doi

  86. [96]

    Weilin Xu, David Evans, and Yanjun Qi. 2018. Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks. In Proceedings 2018 Network and Distributed System Security Symposium (San Diego, CA, USA) (NDSS 2018). Internet Society, Reston, VA, USA. doi:10.14722/ndss....

    2018

  87. [97]

    Biwei Yan, Kun Li, Minghui Xu, Yueyan Dong, Yue Zhang, Zhaochun Ren, and Xiuzhen Cheng. 2025. On protecting the data privacy of Large Language Models (LLMs) and LLM agents: A literature review. High-Confidence Computing 5, 2 (2025), 100300. doi:10.1016/j.hcc.2025.100300

    2025

  88. [98]

    Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. 2024. A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly. High-Confidence Computing 4, 2 (2024), 100211. doi:10.1016/j.hcc.2024.100211

    2024

  89. [99]

    Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018. Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF) (Oxford, UK). IEEE, New York, NY, USA, 268–282. doi:10.1109/CSF.2018...

    2018

  90. [100]

    Jingwei Yi, Rui Ye, Qisi Chen, Bin Zhu, Siheng Chen, Defu Lian, Guangzhong Sun, Xing Xie, and Fangzhao Wu. 2024. On the Vulnerability of Safety Alignment in Open-Access LLMs. In Findings of the Association for Computational Linguistics: ACL 2024 (Bangkok, Thailand), Lun-Wei Ku...

    2024 doi

  91. [101]

    Ruisi Zhang, Seira Hidano, and Farinaz Koushanfar. 2022. Text Revealer: Private Text Reconstruction via Model Inversion Attacks against Transformers. arXiv:2209.10505 [cs.CL]

    2022 arXiv

  92. [102]

    Yiming Zhang, Nicholas Carlini, and Daphne Ippolito. 2024. Effective Prompt Extraction from Language Models. arXiv:2307.06865 [cs.CL] https://openreview.net/forum?id=0o95CVdNuz First Conference on Language Modeling (COLM 2024) (Philadelphia, PA, USA)

    2024 arXiv

  93. [103]

    Jian Zhao, Shenao Wang, Yanjie Zhao, Xinyi Hou, Kailong Wang, Peiming Gao, Yuanchao Zhang, Chen Wei, and Haoyu Wang. 2024. Models Are Codes: Towards Measuring Malicious Code Poisoning Attacks on Pre-trained Model Hubs. In Proceedings of the 39th IEEE/ACM International Conferen...

    2024

  94. [104]

    Weilin Zhong and Rezos. 2021. Code Injection. OWASP. Retrieved May 8, 2025 from https://owasp.org/www- community/attacks/Code_Injection

    2021

  95. [105]

    Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. 2024. Improving Alignment and Robustness with Circuit Breakers. InAdvances in Neural Information Processing Systems (Vancouver, BC, ...

    2024

  96. [106]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv:2307.15043 [cs.CL] A Terminology and Description A.1 Components of AI System Architecture and Related...

    2023 arXiv

  97. [2014]

    arXiv:1312.6199 [cs.CV] 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings

    Intriguing properties of neural networks. arXiv:1312.6199 [cs.CV] 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings

    2014 arXiv

  98. [2018]

    In 2018 International Joint Conference on Neural Networks (IJCNN) (Rio de Janeiro, Brazil)

    Copycat CNN: Stealing Knowledge by Persuading Confession with Random Non-Labeled Data. In 2018 International Joint Conference on Neural Networks (IJCNN) (Rio de Janeiro, Brazil). IEEE, New York, NY, USA, 1–8. doi:10.1109/IJCNN.2018.8489592

    2018

  99. [2024]

    In 33rd USENIX Security Symposium (USENIX Security 24) (Philadelphia, PA, USA)

    Yes, One-Bit-Flip Matters! Universal DNN Model Inference Depletion with Runtime Code Fault Injection. In 33rd USENIX Security Symposium (USENIX Security 24) (Philadelphia, PA, USA). USENIX Association, Berkeley, CA, USA, 1315–1330. https://www.usenix.org/conference/usenixsecur...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.