Pith. sign in

REVIEW 3 major objections 5 minor 300 references

This systematic review claims to be the first to map LLM security threats and mitigations across multiple real-world use cases and design choices, showing that the same threat can vary in relevance and severity depending on where and how th

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 17:39 UTC pith:OUJAUTYO

load-bearing objection A genuinely useful scenario-aware survey of LLM security with a few real internal bugs; the CVSS arithmetic the stress-test flags actually checks out. the 3 major comments →

arxiv 2509.10682 v1 pith:OUJAUTYO submitted 2025-09-12 cs.CR cs.AIcs.CLcs.ETcs.LG

LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems

Vitor Hugo Galhardo Moia , Igor Jochem Sanz , Gabriel Antonio Fontes Rebello , Rodrigo Duarte de Meneses , Briland Hitaj , Ulf Lindqvist This is my paper
classification cs.CR cs.AIcs.CLcs.ETcs.LG
keywords LLM securitysystematic reviewthreat taxonomymitigation strategiesthreat modelingdesign choicesSTRIDEseverity scoring
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that the security of an LLM-based system cannot be assessed in the abstract: which threats apply, and how severe they are, depends on a small set of concrete choices about how the model is built and deployed. To show this, the authors perform a PRISMA-guided systematic review of 198 papers and organize the results around a canonical scenario string — the life-cycle stage, the use case (foundation model, fine-tuning, RAG, chat-bot, integrated app, or agent), and seven design choices including data provenance, infrastructure, shared tenancy, and resource access. The review yields a taxonomy of 52 threats coded by the CIA triad, eleven attack strategies, severity scores from CVSS v3.1 and the OWASP Risk Rating, 47 mitigation techniques mapped to life-cycle phases and scenarios, and STRIDE threat models of four representative deployments. The paper's novelty claim is that no prior survey covers multiple LLM scenarios with these design choices, making this the first scenario-aware, life-cycle-wide map of threats and defenses. A sympathetic reader would take away that defense planning for LLM systems should start by writing down the scenario string, then select mitigations accordingly.

Core claim

On its own terms, the paper's central claim is that the threat landscape of an LLM system is a function of its scenario configuration, and that this dependence can be made explicit. A PRISMA-guided systematic review distills 198 studies into 52 threat classes — 6 in development, 46 in operation — coded by CIA goal, target, attack strategy, and adversary interaction mode. Each threat gets severity scores under two standard methodologies, and each of the 47 catalogued mitigations is mapped to the life-cycle phase where it applies and the attack strategies it attenuates. Four STRIDE analyses of concrete scenarios show that changing one design choice, such as moving from cloud to on-device deplo

What carries the argument

The paper's central organizing device is the canonical LLM scenario string, ST:/UC:/DP:/DI:/SL:/SI:/IO:/AR:/CL:, which encodes the life-cycle stage, the use case, and seven design choices (data provenance, development/deployment infrastructure, software libraries and dependencies, shared infrastructure, prompt input origin, access to resources, and continuous learning). This string carries the argument: every threat is mapped to the scenarios it can affect, every mitigation to the life-cycle phases and design choices it touches, and the four STRIDE threat models are built by instantiating particular strings. Secondary machinery includes the CIA-coded threat taxonomy with attack strategies an

Load-bearing premise

The load-bearing assumption is that the 198 papers kept after the authors' subjective screening (which excluded 106 papers for lacking substantial discussion and 93 for tangential relevance) represent the full LLM security literature; if that filtering dropped a whole threat class, the taxonomy, severity scores, and scenario analyses built from it would be incomplete.

What would settle it

Take two identical systems differing in exactly one design choice — the same chat-bot model and prompts running on-device versus on-cloud — and run the same attack suite against both. If successful attacks and impacts are indistinguishable, the scenario-dependence claim collapses. A cheaper check: rescreen a random sample of the 199 excluded full-text papers for threat classes absent from Tables IV and V; finding one in, say, ten percent of the sample would indicate the corpus is incomplete.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A team deploying an LLM can determine its relevant threat set by writing its scenario string and reading off the threats marked for that use case and design choices, then rank defenses by the severity scores.
  • Design choices are not security-neutral: on-device deployment invites reverse engineering and malware but removes remote insider and shared-cache leakage, while cloud deployment inverts that picture.
  • No mitigation is a silver bullet; the paper's explicit conclusion is that effective protection requires defense-in-depth, with several techniques from different categories tailored to the scenario.
  • Because severity scores differ between methodologies (six threats are 'critical' only under OWASP), relying on a single scoring method can change prioritization decisions.
  • The framework is stated to extend beyond LLMs to generative AI systems generally, so the scenario-mapping method could be reused for other GenAI modalities.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The scenario string could be automated into a decision-support tool that takes a configuration and returns its candidate threat list and suggested mitigations; the paper provides the mapping but does not build such a tool.
  • The severity scores are the authors' interpretations of illustrative attack scenarios rather than measurements; a natural test is to red-team real systems under a known scenario string and check whether the threats scored 'high' or 'critical' are the ones that dominate observed failures.
  • Because the authors excluded 199 full-text papers on subjective grounds and publish no list of included studies, an independent audit of corpus representativeness is impossible from the paper alone; transparency on the inclusion list would let readers test the completeness of Tables IV and V.
  • By explicitly leaving alignment-related harms (such as hate speech or unethical content) out of the CIA framing, the map underweights harm caused by the model's own outputs rather than by an adversary; a practitioner using only this survey would miss that class of risk.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper is a systematic review of security and privacy threats to LLM-based systems. It proposes a life-cycle model spanning development and operation, six LLM use cases (foundation model, fine-tuning, RAG, chat-bot, integrated application, agent), and seven design choices encoded in a canonical scenario string. It contributes a threat taxonomy (Tables IV/V) with CIA classification and attack strategies, severity scores for each threat using CVSS 3.1 and OWASP Risk Rating (Table VI), a mitigation catalogue organized into eight categories and mapped to attack strategies (Tables VII/VIII), and STRIDE-based threat models for four representative deployment scenarios (Section VIII). The central claim is that no prior survey combines scenario-aware use cases, design choices, life-cycle coverage, severity scoring, and defensive mappings for LLM systems.

Significance. If the taxonomy is accepted, it is a genuinely useful reference for practitioners and a reasonable basis for future scenario-specific threat modeling. The paper's strengths include a clearly described PRISMA-style selection flow, machine-readable CVSS vector strings, explicit mappings from threats to mitigations and life-cycle phases, and four concrete STRIDE analyses. I also checked the CVSS v3.1 arithmetic for the vectors flagged in the review: I14 (NHNRCHHH) computes to 8.3, I15 (NHLNCHHH) to 8.5, and I22 (NHNRCLHL) to 7.5, so the printed severity scores are mechanically consistent with the stated vectors. The novelty claim is plausible relative to the surveyed related work, although the completeness of the underlying corpus cannot currently be audited.

major comments (3)
  1. [VII (Mitigation categories 4 and 5)] The text refers to threats I47 and I50 ('via I47 attack' and 'for the I50 attack'), but Tables IV and V only define threats I01 through I28, and no I47/I50 appears elsewhere. Section VIII-B5 similarly refers to 'I09-I29' although the highest integrity ID is I28. This breaks the threat-to-mitigation mapping: a reader cannot tell which threat is being mitigated. Please renumber, add the missing threat definitions, or replace the references with correct IDs.
  2. [IV-D (Shared Infrastructure)] The sentence 'In a shared infrastructure, the storage of context information per-user or per-app should be [334]' is incomplete: it lacks the predicate that states what the storage should be (isolated, partitioned, encrypted, etc.). Since SI is one of the seven design choices used in the canonical scenario strings and in the later scenario analysis, this is a definitional gap rather than a mere typo. The example scenario string in IV-E1 also omits the values after IO and AR ('ST:D/UC:FM/DP:U/DI:P/SL:H/SI:N/IO/AR/CL'), so the formal representation is not consistently applied.
  3. [II (PRISMA methodology)] The review's reproducibility claim is weakened by the absence of a list of the 198 included studies and by the unoperationalized full-text exclusion criteria 'lacked substantial discussion' (106 papers) and 'only tangential relevance' (93 papers). Because the paper's central claim is that no prior work covers the same scenario-aware scope, the completeness of Tables IV and V depends on the representativeness of the selected corpus. Please provide a supplementary file with the included/excluded studies and the reasons for exclusion, as expected under PRISMA.
minor comments (5)
  1. [VII] Typo: 'tensorfow' should be 'TensorFlow'. In the same section, 'IA' should be 'AI'.
  2. [References] Reference [123] is titled 'Writing Secude Code'; should be 'Writing Secure Code'.
  3. [Table VI] The columns 'Exploitability' and 'Technical Impact' are not defined in the table or caption. For OWASP, the numeric 'Overall Score' is the product of Likelihood and Technical Impact averages, not an official OWASP score; please make this explicit in the caption.
  4. [Table II] The filled/half-filled circle legend is difficult to reproduce in text and is not machine-readable. Consider replacing with explicit Y/P/N or a clearer symbol set.
  5. [IV-D] Grammar: 'since LLMs require massive amounts of data to train the models, is generally infeasible' should be 'it is generally infeasible'.

Circularity Check

0 steps flagged

No significant circularity; systematic survey is self-contained against external literature and its framework is a taxonomy, not a derivation.

full rationale

This paper is a systematic literature review and threat/defense taxonomy, not a derivation. The load-bearing contribution is the scenario framework, the threat tables (IV and V), the severity analysis (Table VI), the mitigation mapping (Tables VII–VIII), and the STRIDE-based scenario analyses (Tables X–XIII). These are constructed from 198 externally sourced references plus the authors' stated assumptions; no threat category, severity score, or defense mapping is defined in terms of the paper's own conclusions. The novelty claim ('To the best of our knowledge, no prior work covers LLM threats and defensive strategies considering multiple LLM scenarios...') is a comparative literature claim, not a circular derivation. The CVSS/OWASP severity scores are outputs of standardized formulas applied to author-selected parameter vectors; even if some printed scores are arithmetically inconsistent with their own CVSS strings (a correctness defect), the score is not equivalent to the input by construction. The paper explicitly acknowledges subjectivity as a limitation ('these scores reflect our interpretations... serving as references rather than exact measurements'), which is an honest caveat rather than circularity. Self-citations (e.g., Lindqvist's intrusion classification [186]–[188], Hitaj's federated learning leakage [117]–[120]) appear only as background support for general security concepts and are not load-bearing for the central scenario framework, threat inventory, or severity mapping. There is no imported uniqueness theorem, no ansatz smuggled via self-citation, and no fitted parameter renamed as a prediction. The PRISMA filtering subjectivity and the absence of a data file are reproducibility/auditability concerns, not circularity. Overall score 0.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The paper's central claims rest on subjective severity scores and on the representativeness of a hand-filtered PRISMA selection. The scenario framework is an organizing structure, not an empirically derived model.

free parameters (1)
  • Per-threat severity scores (CVSS 3.1 and OWASP Risk Rating)
    All scores in Tables VI and X-XIII are assigned by the authors based on constructed attack scenarios; they are not measured or validated. They directly drive the prioritization conclusions.
axioms (4)
  • domain assumption The CIA triad is a sufficient model for classifying LLM threats
    Section V uses CIA instead of NIST's misuse category, treating jailbreaks as integrity violations.
  • domain assumption The 198 selected references are representative of the LLM security literature
    Section II PRISMA selection excludes 106 'insubstantial' and 93 'tangential' papers via subjective criteria; completeness of the threat list depends on this.
  • ad hoc to paper The seven design choices (DP, DI, SL, SI, IO, AR, CL) capture the security-relevant variability of LLM scenarios
    Section IV-D postulates these axes without a derivation from first principles.
  • domain assumption CVSS v3.1 and OWASP Risk Rating methodologies are applicable to generative-AI threats
    Section VI applies software vulnerability scoring to threat scenarios, acknowledging but not resolving the mismatch (e.g., insider threat severity is identical across deployments).

pith-pipeline@v1.3.0-alltime-deepseek · 54111 in / 10215 out tokens · 111338 ms · 2026-08-04T17:39:43.839876+00:00 · methodology

0 comments
read the original abstract

The success and wide adoption of generative AI (GenAI), particularly large language models (LLMs), has attracted the attention of cybercriminals seeking to abuse models, steal sensitive data, or disrupt services. Moreover, providing security to LLM-based systems is a great challenge, as both traditional threats to software applications and threats targeting LLMs and their integration must be mitigated. In this survey, we shed light on security and privacy concerns of such LLM-based systems by performing a systematic review and comprehensive categorization of threats and defensive strategies considering the entire software and LLM life cycles. We analyze real-world scenarios with distinct characteristics of LLM usage, spanning from development to operation. In addition, threats are classified according to their severity level and to which scenarios they pertain, facilitating the identification of the most relevant threats. Recommended defense strategies are systematically categorized and mapped to the corresponding life cycle phase and possible attack strategies they attenuate. This work paves the way for consumers and vendors to understand and efficiently mitigate risks during integration of LLMs in their respective solutions or organizations. It also enables the research community to benefit from the discussion of open challenges and edge cases that may hinder the secure and privacy-preserving adoption of LLM-based systems.

Figures

Figures reproduced from arXiv: 2509.10682 by Briland Hitaj, Gabriel Antonio Fontes Rebello, Igor Jochem Sanz, Rodrigo Duarte de Meneses, Ulf Lindqvist, Vitor Hugo Galhardo Moia.

Figure 1
Figure 1. Figure 1: Steps and results from the application of the PRISMA guidelines – [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Adopted life cycle model of an LLM system. The phases can be classified along two axes according to their nature: system-related phases (DevOps) [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Scenario 1: LLM development process (on company premises) for a [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 6
Figure 6. Figure 6: Scenario 4: LLM-based Agent system deployed on user device and [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 5
Figure 5. Figure 5: Scenario 3: An LLM-integrated application deployed on-cloud and [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Possible attack strategies on a general view of an LLM-based system architecture, from development to operation. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Example on how to interpret the severity score string values. On part (a) we provide the scoring produced using CVSS 3.1 ratings, whereas on (b), [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

300 extracted references · 110 linked inside Pith

  1. [1]

    Securing large lan- guage models: Threats, vulnerabilities and responsible practices,

    S. Abdali, R. Anarfi, C. Barberan, and J. He, “Securing large lan- guage models: Threats, vulnerabilities and responsible practices,”arXiv preprint arXiv:2403.12503, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Evaluating privacy leakage and memorization attacks on large language models (LLMs) in generative AI applications,

    H. Aditya, S. Chawla, G. Dhingra, P. Rai, S. Sood, T. Singh, Z. M. Wase, A. Bahga, and V . K. Madisetti, “Evaluating privacy leakage and memorization attacks on large language models (LLMs) in generative AI applications,”Journal of Software Engineering and Applications, vol. 17, no. 5, pp. 421–447, 2024

    2024

  3. [3]

    Prompt leakage effect and mitigation strategies for multi-turn LLM applications,

    D. Agarwal, A. Fabbri, B. Risher, and P. Laban, “Prompt leakage effect and mitigation strategies for multi-turn LLM applications,” aclanthology.org, 2024. [Online]. Available: https://aclanthology.org/2 024.emnlp-industry.94/

    2024

  4. [4]

    Investigating the prompt leakage effect and black- box defenses for multi-turn LLM interactions,

    D. Agarwal, A. R. Fabbri, P. Laban, B. Risher, S. Joty, C. Xiong, and C.-S. Wu, “Investigating the prompt leakage effect and black- box defenses for multi-turn LLM interactions,”arXiv preprint arXiv:2404.16251, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Hallucinat- ing AI hijacking attack: Large language models and malicious code recommenders,

    W. Agnew, H. H. Jiang, C. Sum, M. Sap, and S. Das, “Hallucinat- ing AI hijacking attack: Large language models and malicious code recommenders,”arXiv preprint arXiv:2410.06462, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Jailbreak attacks on large language models and possible defenses: Present status and future possibilities,

    S. S. Ahmed and J. A. A. Jothi, “Jailbreak attacks on large language models and possible defenses: Present status and future possibilities,” in2024 IEEE Int. Symp. on Technology and Society (ISTAS), 9 2024, pp. 1–7

    2024

  7. [7]

    Securing artificial intelligence: Exploring attack scenarios and defense strategies,

    ˙I. Z. Altun and A. E. ¨Ozk¨ok, “Securing artificial intelligence: Exploring attack scenarios and defense strategies,” in2024 12th Int. Symp. on Digital Forensics and Security (ISDFS), 2024, pp. 1–6

    2024

  8. [8]

    Sok: Reducing the vulnera- bility of fine-tuned language models to membership inference attacks,

    G. Amit, A. Goldsteen, and A. Farkash, “Sok: Reducing the vulnera- bility of fine-tuned language models to membership inference attacks,” arXiv preprint arXiv:2403.08481, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Jailbreaking leading safety-aligned LLMs with simple adaptive attacks,

    M. Andriushchenko, F. Croce, and N. Flammarion, “Jailbreaking leading safety-aligned LLMs with simple adaptive attacks,” in13th Int. Conf. on Learning Representations (ICLR) Poster, 2025

    2025

  10. [10]

    How do you use personal data in model training?

    Anthropic, “How do you use personal data in model training?” 2025, accessed: Aug. 01, 2025. [Online]. Available: https://privacy.anthropi c.com/en/articles/10023555-how-do-you-use-personal-data-in-model -training

    arXiv 2025

  11. [11]

    AutoGPT: Build, deploy, and run AI agents,

    AutoGPT, “AutoGPT: Build, deploy, and run AI agents,” 2023, accessed: Aug. 01, 2025. [Online]. Available: https://github.com/Signi ficant-Gravitas/AutoGPT

    2023

  12. [12]

    Obfus- cated activations bypass LLM latent-space defenses,

    L. Bailey, A. Serrano, A. Sheshadri, M. Seleznyov, J. Taylor, E. Jen- ner, J. Hilton, S. Casper, C. Guestrin, and S. Emmons, “Obfus- cated activations bypass LLM latent-space defenses,”arXiv preprint arXiv:2412.09565, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    Understanding jailbreak success: A study of latent space dynamics in large language models,

    S. Ball, F. Kreuter, and N. Panickssery, “Understanding jailbreak success: A study of latent space dynamics in large language models,” arXiv preprint arXiv:2406.09289, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Identifying and mitigating the security risks of generative AI,

    C. Barrett, B. Boyd, E. Bursztein, N. Carlini, B. Chen, J. Choi, A. R. Chowdhury, M. Christodorescu, A. Datta, S. Feizi, K. Fisher, T. Hashimoto, D. Hendrycks, S. Jha, D. Kang, F. Kerschbaum, E. Mitchell, J. Mitchell, Z. Ramzan, K. Shams, D. Song, A. Taly, and D. Yang, “Identifying and mitigating the security risks of generative AI,”Foundations and Trends...

    2023

  15. [15]

    RatGPT: Turning online LLMs into proxies for malware attacks,

    M. Beckerich, L. Plein, and S. Coronado, “RatGPT: Turning online LLMs into proxies for malware attacks,”ACM/JMS Journal of Data Science, vol. 37, 8 2023. [Online]. Available: http: //arxiv.org/abs/2308.09183

    Pith/arXiv arXiv 2023

  16. [16]

    International AI safety report,

    Y . Bengioet al., “International AI safety report,” UK AI Safety Institute, 2025, accessed: Aug. 01, 2025. [Online]. Available: https://www.gov.uk/government/publications/international-ai-safety-r eport-2025

    2025

  17. [17]

    Systematically analyzing prompt injection vulnerabilities in diverse LLM architectures,

    V . Benjamin, E. Braca, I. Carter, H. Kanchwala, N. Khojasteh, C. Landow, Y . Luo, C. Ma, A. Magarelli, R. Mirin, A. Moyer, K. Simpson, A. Skawinski, and T. Heverin, “Systematically analyzing prompt injection vulnerabilities in diverse LLM architectures,”arXiv preprint arXiv:2410.23308, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    The TIP of the iceberg: Revealing a hidden class of task-in-prompt adversarial attacks on LLMs,

    S. Berezin, R. Farahbakhsh, and N. Crespi, “The TIP of the iceberg: Revealing a hidden class of task-in-prompt adversarial attacks on LLMs,”arXiv preprint arXiv:2501.18626, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    Adversarial attacks on automatic speech recognition (asr): A survey,

    A. R. Bhanushali, H. Mun, and J. Yun, “Adversarial attacks on automatic speech recognition (asr): A survey,”IEEE Access, 2024

    2024

  20. [20]

    ALERT: A comprehensive benchmark for assessing large language models’ safety through red teaming,

    M. Bhatt, S. Chennabasappa, Y . Li, C. Nikolaidis, D. Song, S. Wan, F. Ahmad, C. Aschermann, Y . Chen, D. Kapil, D. Molnar, S. Whitman, and J. Saxe, “ALERT: A comprehensive benchmark for assessing large language models’ safety through red teaming,”arXiv preprint arXiv:2404.08676, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Talk too much: Poisoning large language models under token limit,

    ——, “Talk too much: Poisoning large language models under token limit,”arXiv preprint arXiv:2404.14795, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Exfiltration of personal information from chatGPT via prompt injection,

    D. Bhusal, T. Alam, L. Nguyen, and B. A. Blakely, “Exfiltration of personal information from chatGPT via prompt injection,”arXiv preprint arXiv:2406.00199, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Large language models are vulnerable to bait-and-switch attacks for generating harmful content,

    F. Bianchi and J. Zou, “Large language models are vulnerable to bait-and-switch attacks for generating harmful content,”arXiv preprint arXiv:2402.13926, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Model leeching: An extraction attack targeting LLMs,

    L. Birch, W. Hackett, S. Trawicki, N. Suri, and P. Garraghan, “Model leeching: An extraction attack targeting LLMs,” inProc. Conf. on Applied Machine Learning in Information Security (CAMLIS), Oct. 2023, pp. 91–104

    2023

  25. [25]

    GPThreats-3: Is automatic malware generation a threat?

    M. Botacin, “GPThreats-3: Is automatic malware generation a threat?” in2023 IEEE Security and Privacy Workshops (SPW), 5 2023, pp. 238–254

    2023

  26. [26]

    Insights and current gaps in open-source LLM vulnerability scanners: A comparative anal- ysis,

    J. Brokman, O. Hofman, O. Rachmil, I. Singh, V . Pahuja, R. S. A. Priya, A. Giloni, R. Vainshtein, and H. Kojima, “Insights and current gaps in open-source LLM vulnerability scanners: A comparative anal- ysis,”arXiv preprint arXiv:2410.16527, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language mod- els are few-shot learners,”Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901, 2020

    1901

  28. [28]

    Lessons from red teaming 100 generative AI products,

    B. Bullwinkel, A. Minnich, S. Chawla, G. Lopez, M. Pouliot, W. Maxwell, J. de Gruyter, K. Pratt, S. Qi, N. Chikanov, R. Lutz, R. S. R. Dheekonda, B.-E. Jagdagdorj, E. Kim, J. Song, K. Hines, D. Jones, G. Severi, R. Lundeen, S. Vaughan, V . Westerhoff, P. Bryan, R. S. S. Kumar, Y . Zunger, C. Kawaguchi, and M. Russinovich, “Lessons from red teaming 100 gen...

    Pith/arXiv arXiv 2025

  29. [29]

    A new foundation for AI on android,

    D. Burke, “A new foundation for AI on android,” 2023, accessed: Aug. 01, 2025. [Online]. Available: https://android-developers.google blog.com/2023/12/a-new-foundation-for-ai-on-android.html

    2023

  30. [30]

    Rethinking how to evaluate language model jailbreak,

    H. Cai, A. Arunasalam, L. Y . Lin, A. Bianchi, and Z. B. Celik, “Rethinking how to evaluate language model jailbreak,”arXiv preprint arXiv:2404.06407, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Are large language models really bias-free? jailbreak prompts for assessing adversarial robustness to bias elicitation,

    R. Cantini, G. Cosenza, A. Orsino, and D. Talia, “Are large language models really bias-free? jailbreak prompts for assessing adversarial robustness to bias elicitation,” inDiscovery Science. Cham: Springer Nature Switzerland, 2025, pp. 52–68

    2025

  32. [32]

    Red teaming chatGPT via jailbreaking: Bias, robustness, reliability and toxicity,

    B. Cao, H. Lin, X. Han, F. Liu, and L. Sun, “Red teaming chatGPT via jailbreaking: Bias, robustness, reliability and toxicity,”arXiv preprint arXiv:2301.12867, 2023

    Pith/arXiv arXiv 2023

  33. [33]

    Guide for defense (G4D): Dynamic guidance for robust and balanced defense in large language models,

    H. Cao, W. Luo, Y . Wang, Z. Liu, B. Feng, Y . Yao, and Y . Li, “Guide for defense (G4D): Dynamic guidance for robust and balanced defense in large language models,”arXiv preprint arXiv:2410.17922, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    Stealthy and persistent unalignment on large language models via backdoor injections,

    Y . Cao, B. Cao, and J. Chen, “Stealthy and persistent unalignment on large language models via backdoor injections,”Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024, vol. 1, pp. 4920–4935, 2024. [Online]. Available: https://aclanthology.org/2024.naac...

    2024

  35. [35]

    Extracting training data from large language models,

    N. Carlini, F. Tram `er, E. Wallace, M. Jagielski, A. Herbert-V oss, K. Lee, A. Roberts, T. Brown, D. Song, ´U. Erlingsson, A. Oprea, and C. Raffel, “Extracting training data from large language models,” in30th USENIX Security Symp. (USENIX Security 21). USENIX Association, Aug. 2021, pp. 2633–2650

    2021

  36. [36]

    Privacy by design: The seven foundational principles,

    A. Cavoukianet al., “Privacy by design: The seven foundational principles,”IAPP Resource Center, 2021

    2021

  37. [37]

    Speak easy: Eliciting harmful jailbreaks from LLMs with simple interactions,

    Y . S. Chan, N. Ri, Y . Xiao, and M. Ghassemi, “Speak easy: Eliciting harmful jailbreaks from LLMs with simple interactions,”arXiv preprint arXiv:2502.04322, 2025

    Pith/arXiv arXiv 2025

  38. [38]

    Context-aware membership inference attacks against pre-trained large language models,

    H. Chang, A. S. Shamsabadi, K. Katevas, H. Haddadi, and R. Shokri, “Context-aware membership inference attacks against pre-trained large language models,”arXiv preprint arXiv:2409.13745, 2024

    arXiv 2024

  39. [39]

    Jailbreakbench: An open robustness benchmark for jailbreaking large language models,

    P. Chao, E. Debenedetti, A. Robey, M. Andriushchenko, F. Croce, V . Sehwag, E. Dobriban, N. Flammarion, G. J. Pappas, F. Trameret al., “Jailbreakbench: An open robustness benchmark for jailbreaking large language models,”Advances in Neural Information Processing Systems, vol. 37, pp. 55 005–55 029, 2024

    2024

  40. [40]

    Jailbreaking black box large language models in twenty queries,

    P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, “Jailbreaking black box large language models in twenty queries,” in 2023 NeurIPS Workshop on robustness of zero/few-shot learning in foundation models (R0-FoMo), 2023

    2023

  41. [41]

    The dark side of human feedback: Poisoning large language models via user inputs,

    B. Chen, H. Guo, G. Wang, Y . Wang, and Q. Yan, “The dark side of human feedback: Poisoning large language models via user inputs,” arXiv preprint arXiv:2409.00787, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    FlexLLM: Exploring LLM customiza- tion for moving target defense on black-box LLMs against jailbreak attacks,

    B. Chen, H. Guo, and Q. Yan, “FlexLLM: Exploring LLM customiza- tion for moving target defense on black-box LLMs against jailbreak attacks,”arXiv preprint arXiv:2412.07672, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Multi-turn hidden backdoor in large language model-powered chatbot models,

    B. Chen, N. Ivanov, G. Wang, and Q. Yan, “Multi-turn hidden backdoor in large language model-powered chatbot models,” inProc. 19th ACM Asia Conf. on Computer and Communications Security. Association for Computing Machinery, 2024, pp. 1316–1330. [Online]. Available: https://doi.org/10.1145/3634737.3656289

    arXiv 2024

  44. [44]

    A statistical and multi-perspective re- visiting of the membership inference attack in large language models,

    B. Chen, N. Han, and Y . Miyao, “A statistical and multi-perspective re- visiting of the membership inference attack in large language models,” arXiv preprint arXiv:2412.13475, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    LLM for mobile: An initial roadmap,

    D. Chen, Y . Liu, M. Zhou, Y . Zhao, H. Wang, S. Wang, X. Chen, T. F. Bissyand´e, J. Klein, and L. Li, “LLM for mobile: An initial roadmap,” ACM Transactions on Software Engineering and Methodology, 2024

    2024

  46. [46]

    Deceptive delight: Jailbreak LLMs through camouflage and distraction,

    J. Chen and R. Lu, “Deceptive delight: Jailbreak LLMs through camouflage and distraction,” 2024, accessed: Aug. 01, 2025. [Online]. Available: https://unit42.paloaltonetworks.com/jailbreak-llms-through -camouflage-distraction/

    2024

  47. [47]

    Unlearn what you want to forget: Efficient unlearning for LLMs,

    J. Chen and D. Yang, “Unlearn what you want to forget: Efficient unlearning for LLMs,” inProc. 2023 Conf. on Empirical Methods in Natural Language Processing (EMNLP). Association for Computa- tional Linguistics, Dec. 2023, pp. 12 041–12 052

    2023

  48. [48]

    Defending against gcg jailbreak attacks with syntax trees and perplexity in LLMs,

    Q. Chen, S. Yamaguchi, and Y . Yamamoto, “Defending against gcg jailbreak attacks with syntax trees and perplexity in LLMs,” in2024 IEEE 13th Global Conf. on Consumer Electronics (GCCE), 10 2024, pp. 1411–1415

    2024

  49. [49]

    Security of language models for code: A systematic literature review,

    Y . Chen, W. Sun, C. Fang, Z. Chen, Y . Ge, T. Han, Q. Zhang, Y . Liu, Z. Chen, and B. Xu, “Security of language models for code: A systematic literature review,”arXiv preprint arXiv:2410.15631, 2024

    Pith/arXiv arXiv 2024

  50. [50]

    Agentpoison: Red- teaming LLM agents via poisoning memory or knowledge bases,

    Z. Chen, Z. Xiang, C. Xiao, D. Song, and B. Li, “Agentpoison: Red- teaming LLM agents via poisoning memory or knowledge bases,”arXiv preprint arXiv:2407.12784, 2024

    Pith/arXiv arXiv 2024

  51. [51]

    Black- box opinion manipulation attacks to retrieval-augmented generation of large language models,

    Z. Chen, J. Liu, H. Liu, Q. Cheng, F. Zhang, W. Lu, and X. Liu, “Black- box opinion manipulation attacks to retrieval-augmented generation of large language models,”arXiv preprint arXiv:2407.13757, 2024

    Pith/arXiv arXiv 2024

  52. [52]

    TrojanRAG: Retrieval-augmented generation can be backdoor driver in large language models,

    P. Cheng, Y . Ding, T. Ju, Z. Wu, W. Du, P. Yi, Z. Zhang, and G. Liu, “TrojanRAG: Retrieval-augmented generation can be backdoor driver in large language models,”arXiv preprint arXiv:2405.13401, 2024

    Pith/arXiv arXiv 2024

  53. [53]

    Syntactic ghost: An imperceptible general-purpose backdoor attacks on pre- trained language models,

    P. Cheng, W. Du, Z. Wu, F. Zhang, L. Chen, and G. Liu, “Syntactic ghost: An imperceptible general-purpose backdoor attacks on pre- trained language models,”arXiv preprint arXiv:2402.18945, 2024

    Pith/arXiv arXiv 2024

  54. [54]

    Transferring backdoors between large language models by knowledge distillation,

    P. Cheng, Z. Wu, T. Ju, W. Du, Z. Zhang, and G. Liu, “Transferring backdoors between large language models by knowledge distillation,” arXiv preprint arXiv:2408.09878, 2024

    Pith/arXiv arXiv 2024

  55. [55]

    Forensic analysis of indirect prompt injection attacks on LLM agents,

    M. Chernyshev, Z. Baig, and R. Doss, “Forensic analysis of indirect prompt injection attacks on LLM agents,” in2024 IEEE 6th Int. Conf. on Trust, Privacy and Security in Intelligent Systems, and Applications (TPS-ISA), 10 2024, pp. 409–411

    2024

  56. [56]

    Breaking down the defenses: A compar- ative survey of attacks on large language models,

    A. G. Chowdhury, M. M. Islam, V . Kumar, F. H. Shezan, V . Kumar, V . Jain, and A. Chadha, “Breaking down the defenses: A compar- ative survey of attacks on large language models,”arXiv preprint arXiv:2403.04786, 2024

    arXiv 2024

  57. [57]

    Compre- hensive assessment of jailbreak attacks against LLMs,

    J. Chu, Y . Liu, Z. Yang, X. Shen, M. Backes, and Y . Zhang, “Compre- hensive assessment of jailbreak attacks against LLMs,”arXiv preprint arXiv:2402.05668, 2024

    Pith/arXiv arXiv 2024

  58. [58]

    AI safety in generative AI large language models: A survey,

    J. Chua, Y . Li, S. Yang, C. Wang, and L. Yao, “AI safety in generative AI large language models: A survey,”arXiv preprint arXiv:2407.18369, 2024

    Pith/arXiv arXiv 2024

  59. [59]

    PrisonBreak: Jailbreaking large language models with fewer than twenty-five targeted bit-flips,

    Z. Coalson, J. Woo, S. Chen, Y . Sun, L. Yang, P. Nair, B. Fang, and S. Hong, “PrisonBreak: Jailbreaking large language models with fewer than twenty-five targeted bit-flips,”arXiv preprint arXiv:2412.07192, 2024

    arXiv 2024

  60. [60]

    Here comes the AI worm: Unleashing zero-click worms that target genAI-powered applications,

    S. Cohen, R. Bitton, and B. Nassi, “Here comes the AI worm: Unleashing zero-click worms that target genAI-powered applications,” 2024, accessed: Aug. 01, 2025. [Online]. Available: https://sites.google.com/view/compromptmized

    2024

  61. [61]

    Common crawl maintains a free, open repository of web crawl data that can be used by anyone,

    CommonCrawl, “Common crawl maintains a free, open repository of web crawl data that can be used by anyone,” 2025, accessed: Aug. 01, 2025. [Online]. Available: https://commoncrawl.org/

    2025

  62. [62]

    Intel SGX explained,

    V . Costan and S. Devadas, “Intel SGX explained,”IACR Cryptology ePrint Archive, vol. 2016, p. 86, 2016. [Online]. Available: https://eprint.iacr.org/2016/086

    2016

  63. [63]

    Recent advances in attack and defense approaches of large language models,

    J. Cui, Y . Xu, Z. Huang, S. Zhou, J. Jiao, and J. Zhang, “Recent advances in attack and defense approaches of large language models,” arXiv preprint arXiv:2409.03274, 2024

    Pith/arXiv arXiv 2024

  64. [64]

    Risk taxonomy, mitigation, and assess- ment benchmarks of large language model systems,

    T. Cui, Y . Wang, C. Fu, Y . Xiao, S. Li, X. Deng, Y . Liu, Q. Zhang, Z. Qiu, P. Liet al., “Risk taxonomy, mitigation, and assess- ment benchmarks of large language model systems,”arXiv preprint arXiv:2401.05778, 2024

    Pith/arXiv arXiv 2024

  65. [65]

    AI worm that infects computers and reads emails created by researchers,

    A. Cuthbertson, “AI worm that infects computers and reads emails created by researchers,” 2024, accessed: Aug. 01, 2025. [Online]. Available: https://www.independent.co.uk/tech/ai-worm-computer-sec urity-chatgpt-malware-b2506594.html

    2024

  66. [66]

    Security and privacy challenges of large language models: A survey,

    B. C. Das, M. H. Amini, and Y . Wu, “Security and privacy challenges of large language models: A survey,”ACM Comput. Surv., vol. 57, no. 6, Feb. 2025

    2025

  67. [67]

    Human-readable adversarial prompts: An investigation into LLM vulnerabilities using situational context,

    N. Das, E. Raff, U. Booz, A. Hamilton, and M. Gaur, “Human-readable adversarial prompts: An investigation into LLM vulnerabilities using situational context,”arXiv preprint arXiv:2412.16359, 2024

    Pith/arXiv arXiv 2024

  68. [68]

    AttentionBreaker: Adaptive evolutionary optimization for unmasking vulnerabilities in LLMs through bit-flip attacks,

    S. Das, S. Bhattacharya, S. Kundu, S. Kundu, A. Menon, A. Raha, and K. Basu, “AttentionBreaker: Adaptive evolutionary optimization for unmasking vulnerabilities in LLMs through bit-flip attacks,”arXiv preprint arXiv:2411.13757, 2024

    Pith/arXiv arXiv 2024

  69. [69]

    AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents,

    E. Debenedetti, J. Zhang, M. Balunovi ´c, L. Beurer-Kellner, M. Fischer, and F. Tram`er, “AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents,” in38th Annual Conf. on Neural Information Processing Systems (NeurIPS) - Datasets and Benchmarks, 2024

    2024

  70. [70]

    Attack prompt generation for red teaming and defending large language models,

    B. Deng, W. Wang, F. Feng, Y . Deng, Q. Wang, and X. He, “Attack prompt generation for red teaming and defending large language models,” inFindings of the Association for Computational Linguistics (EMNLP 2023). Association for Computational Linguistics, 2023, pp. 2176–2189

    2023

  71. [71]

    Adversaflow: Visual red teaming for large language models with multi-level adversar- ial flow,

    D. Deng, C. Zhang, H. Zheng, Y . Pu, S. Ji, and Y . Wu, “Adversaflow: Visual red teaming for large language models with multi-level adversar- ial flow,”IEEE Transactions on Visualization and Computer Graphics, vol. 31, pp. 492–502, 1 2025

    2025

  72. [72]

    MASTERKEY: Automated jailbreaking of large language model chatbots,

    G. Deng, Y . Liu, Y . Li, K. Wang, Y . Zhang, Z. Li, H. Wang, T. Zhang, and Y . Liu, “MASTERKEY: Automated jailbreaking of large language model chatbots,” inNetwork and Distributed System Security (NDSS) Symp. 2024, Jan. 2024

    2024

  73. [73]

    ThreatFinderAI: Automated threat modeling applied to LLM system integration,

    J. der Assen, A. Huertas, J. Sharif, C. Feng, G. Bovet, and B. Stiller, “ThreatFinderAI: Automated threat modeling applied to LLM system integration,” in2024 20th Int. Conf. on Network and Service Manage- ment (CNSM), Oct. 2024, pp. 1–3

    2024

  74. [74]

    Beyond the safeguards: Exploring the security risks of chatGPT,

    E. Derner and K. Batisti ˇc, “Beyond the safeguards: Exploring the security risks of chatGPT,”arXiv preprint arXiv:2305.08005, 2023

    Pith/arXiv arXiv 2023

  75. [75]

    A wolf in sheep’s clothing: Generalized nested jailbreak prompts can fool large language models easily,

    P. Ding, J. Kuang, D. Ma, X. Cao, Y . Xian, J. Chen, and S. Huang, “A wolf in sheep’s clothing: Generalized nested jailbreak prompts can fool large language models easily,”Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024, vol. 1, pp. 2136–2153, 2024

    2024

  76. [76]

    Attacks, defenses and evaluations for LLM conversation safety: A survey,

    Z. Dong, Z. Zhou, C. Yang, J. Shao, and Y . Qiao, “Attacks, defenses and evaluations for LLM conversation safety: A survey,” inProc. 2024 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Jan. 2024, pp. 6734–6747. 30

    2024

  77. [77]

    When LLM meets drl: Advancing jailbreaking efficiency via drl-guided search,

    A. Draguns, A. Gritsevskiy, S. R. Motwani, C. Rogers-Smith, J. Ladish, and C. S. D. Witt, “When LLM meets drl: Advancing jailbreaking efficiency via drl-guided search,” in38th Annual Conf. on Neural Information Processing Systems (NeurIPS) Poster, 2024

    2024

  78. [78]

    Privacy in fine-tuning large language models: Attacks, defenses, and future directions,

    H. Du, S. Liu, L. Zheng, Y . Cao, A. Nakamura, and L. Chen, “Privacy in fine-tuning large language models: Attacks, defenses, and future directions,”arXiv preprint arXiv:2412.16504, 2024

    Pith/arXiv arXiv 2024

  79. [79]

    UOR: Universal backdoor attacks on pre-trained language models,

    W. Du, P. Li, H. Zhao, T. Ju, G. Ren, and G. Liu, “UOR: Universal backdoor attacks on pre-trained language models,” inFindings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics, Aug. 2024, pp. 7865–7877

    2024

  80. [80]

    Analyzing the inherent response tendency of LLMs: Real-world instructions-driven jailbreak,

    Y . Du, S. Zhao, M. Ma, Y . Chen, and B. Qin, “Analyzing the inherent response tendency of LLMs: Real-world instructions-driven jailbreak,” arXiv preprint arXiv:2312.04127, 2023

    Pith/arXiv arXiv 2023

Showing first 80 references.