Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

A Survey on Model Extraction Attacks and Defenses for Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This survey tries to show that every LLM extraction attack falls into one of three families with matching defenses.

desk verdict Useful high-level map of LLM extraction, but the Carlini et al. misclassification is a load-bearing error and the defense ratings are asserted without evidence. read the letter →

arxiv 2506.22521 v1 pith:O6YQAYC3 submitted 2025-06-26 cs.CR cs.AIcs.LG

Kaixiang Zhao , Lincan Li , Kaize Ding , Neil Zhenqiang Gong , Yue Zhao , Yushun Dong This is my paper
classification cs.CRcs.AIcs.LG
keywords largelanguagemodelsmodelextractionattacksLLMsecuritytaxonomydefensespromptstealingtrainingdataevaluationmetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is a survey that organizes the field of model extraction attacks against large language models into a single coherent map. Its central claim is that every extraction attack falls into one of three families — functionality extraction, training data extraction, or prompt-targeted attacks — and every defense falls into one of three corresponding families. If that map is correct, researchers and engineers can compare attacks and defenses on common terms and see which protections actually block which threats. The survey also proposes evaluation metrics for attack success and defense performance that are tailored to generative models.

What carries the argument

The organizing device is the three-by-three taxonomy: three attack families (functionality extraction, training data extraction, and prompt-targeted attacks) mapped against three defense families (model protection, data privacy protection, and prompt protection). The comparison is carried by a defense-effectiveness matrix that assigns each defense family an ordinal rating — High, Medium, Low, or Minimal — against each attack family, alongside proposed evaluation metrics for measuring both sides.

What would settle it

Build a benchmark that applies the paper's proposed metrics to a sample of the cited attacks and defenses, and check whether the ordinal effectiveness ratings in Table 1 survive. If, for instance, measured defense success against prompt stealing ranked query monitoring above prompt protection, the survey's comparative defense claim would be falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLM model extraction is not an unstructured collection of exploits but a small set of attack families with matching defense families. Functionality extraction tries to clone the model's behavior through black-box queries; training data extraction tries to recover examples the model memorized; prompt-targeted attacks try to steal the prompts that steer the model. Defenses are grouped as model protection, data privacy protection, and prompt protection. The paper further claims that these families can be compared using generative-model-specific metrics such as functional similarity, data recovery rate, security metrics, and utility metrics, and it summarizes the expected effectiveness of each defense against each attack in a single table.

Load-bearing premise

The comparison of defenses rests on the assumption that the qualitative effectiveness ratings in the summary table accurately reflect how well each cited defense works against each attack type, even though no scoring rubric or quantitative aggregation is supplied.

Editorial extensions

If this is right

  • A practitioner can take any new extraction attack and classify it as functionality, data, or prompt-targeted, which immediately identifies which defense families are thought to apply.
  • Defenders can use the effectiveness matrix to choose a first-line defense: architectural protection for functionality extraction, training-data security for data extraction, and prompt protection for prompt stealing.
  • The proposed metrics give attack and defense papers a shared language for reporting success, so results become easier to compare across studies.
  • The taxonomy exposes gaps: no single defense is rated effective against all attack types, so layered defenses are likely to be needed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The Table 1 ratings are qualitative and are not backed by a stated scoring protocol, so the relative effectiveness of defenses should be treated as a hypothesis to test rather than a measured ranking.
  • Because the taxonomy is built from attacks that have already appeared, a genuinely new attack that straddles families, such as a prompt injection that also reveals training data, may require the categories to bend or split.
  • The metrics the paper proposes are defined conceptually rather than operationalized with a concrete benchmark; turning them into an open test suite would be a direct way to validate the survey's comparative claims.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper surveys model extraction attacks against large language models and defenses against them. It proposes a three-part taxonomy of attacks (functionality extraction, training data extraction, prompt-targeted attacks) and a corresponding three-part taxonomy of defenses (model protection, data privacy protection, prompt protection). The paper formalizes the attack and defense families with objective functions (Eqs. 1-6), summarizes defense effectiveness in a qualitative table (Table 1), and proposes evaluation metrics for attack and defense performance (Section 6). It closes with limitations and future directions, positioning the taxonomy and the metric framework as its main contributions.

Significance. If the taxonomy and Table 1 were reliable, the paper would be a useful organizing reference for researchers and practitioners comparing LLM extraction attacks and defenses. The paper's strengths include explicit threat-model definitions, a broad citation base, and a clear visual taxonomy in Fig. 2. However, the central taxonomic claim is weakened by concrete misclassifications of landmark papers, and the defense-comparison table lacks any stated scoring protocol. These issues are local and correctable, but they currently prevent the survey from serving as a dependable map of the field.

major comments (4)
  1. [Fig. 2 / §4.1.1 / §4.1.3] Carlini et al. [5] is listed under 'General Functionality Extraction' in Fig. 2 and §4.1.1, with the text describing it as an extraction attack against production-scale language models. The cited paper, 'Stealing Part of a Production Language Model,' actually recovers the embedding projection (unembedding) matrix via API queries, which is precisely what §4.1.3 defines as Parameter/Architecture Recovery ('reverse-engineer specific model components, such as weights or structural designs'). This is an objective mismatch with the cited work and with the paper's own category definitions, and it undermines the reliability of the taxonomy for readers who use it to organize the literature.
  2. [Fig. 2 / §4.1.2] A related inconsistency appears with Krishna et al. [33]: Fig. 2 places it under 'General Functionality Extraction,' but §4.1.2 cites it as a demonstration of targeted functionality extraction. The paper never reconciles these two placements. Since one of the survey's central claims is that its taxonomy provides a consistent categorization, duplicate placement of the same work in different subcategories without explanation weakens that claim.
  3. [Fig. 2 / Table 1 / §5.2.1] Feng and Tramèr [17] is listed under the defense category 'Training Data Security' in Fig. 2 and in Table 1's footnote [3]. The cited paper is 'Privacy Backdoors: Stealing Data with Corrupted Pretrained Models,' which presents an attack that uses corrupted pretrained models to steal data. Listing an attack paper as a defense mechanism misrepresents the defense literature and makes the defense taxonomy appear to cover a defensive technique that does not exist in the cited work.
  4. [Table 1 / Section 6] The effectiveness ratings in Table 1 (High/Medium/Low/Minimal) are asserted without a scoring rubric, aggregation procedure, or citation-to-cell mapping. Section 6 lists candidate metrics, such as attack prevention rate, query detection accuracy, extraction cost increase, and watermark robustness, but it does not explain how any of these were measured for Table 1 or how conflicting results across the cited defenses were reconciled. Because the defense-comparison table is a central contribution, the ratings need to be derived from a stated protocol or explicitly labeled as an informal qualitative summary.
minor comments (4)
  1. [§5] The sentence in Section 5 reads 'Defending against model extraction attacks requires a approach'; it should be 'requires an approach.'
  2. [§5.2] The phrase 'Similarly as we mentioned in the Section 4.2' is awkward; consider 'As mentioned in Section 4.2.'
  3. [Eqs. (2) and (3)] Equations (2) and (3) use sim(·,·) without specifying a similarity metric or its domain; since the paper proposes these as formal threat-model definitions, a concrete choice, such as embedding cosine similarity or exact-match overlap, would make them more operational.
  4. [§6.1] The citation support for several metric definitions is unclear; for example, §6.1 attributes the agreement-rate metric to Carlini et al. [5], but that paper's evaluation is centered on parameter recovery rather than functional agreement between extracted and target models.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the survey's taxonomy and evaluation metrics are external-literature restatements, not derived from a fitted input or self-citation chain.

full rationale

This is a literature-organizing survey, not a derivation or prediction pipeline. The taxonomy is built by assigning cited attacks and defenses to categories, and the categories are not defined in terms of one another in a way that forces the assignment. The formal equations (1)-(6) restate standard threat models and do not produce fitted values that are later called predictions. The proposed evaluation metrics are enumerations of known measurement practices (agreement rate, perplexity similarity, data recovery rate, watermark robustness), not quantities derived from the survey's own parameters. The only self-references are to the authors' earlier survey [92] and group papers ([8], [9], [72]), used as examples or as a pointer for distributed-computing extraction; none is load-bearing for the central taxonomy or metrics. The skeptic's concern that Carlini et al. [5] is placed under general functionality extraction instead of parameter/architecture recovery is a correctness and classification objection; even if valid, it does not make the paper's claims equivalent to their inputs. No circular step can be exhibited under the required standard.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The survey is descriptive and introduces no free parameters or invented entities. Its central claims rest on two domain assumptions: that the proposed category partitions are exhaustive, and that the qualitative effectiveness ratings in Table 1 are valid summaries of the cited defenses. Neither is formally justified in the paper.

assumptions (2)
  • domain assumption The three-category partition of attacks (functionality, training data, prompt-targeted) and the three-category partition of defenses (model protection, data privacy protection, prompt protection) are exhaustive and mutually exclusive.
    Underlies the taxonomy in Figure 2 and all section headings; no formal argument, meta-analysis, or coverage audit is provided.
  • domain assumption The qualitative effectiveness ratings in Table 1 correctly summarize the cited defenses against each attack type.
    The High/Medium/Low/Minimal ratings are asserted without a scoring rubric, quantitative results, or stated method for aggregation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Survey on Model Extraction Attacks and Defenses for Large Language Models." pith.science (2026). https://pith.science/paper/O6YQAYC3

@misc{pith2026250622521,
  author       = {Pith},
  title        = {Pith review of: A Survey on Model Extraction Attacks and Defenses for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O6YQAYC3}},
  note         = {Machine review of arXiv:2506.22521}
}
read the original abstract

Model extraction attacks pose significant security threats to deployed language models, potentially compromising intellectual property and user privacy. This survey provides a comprehensive taxonomy of LLM-specific extraction attacks and defenses, categorizing attacks into functionality extraction, training data extraction, and prompt-targeted attacks. We analyze various attack methodologies including API-based knowledge distillation, direct querying, parameter recovery, and prompt stealing techniques that exploit transformer architectures. We then examine defense mechanisms organized into model protection, data privacy protection, and prompt-targeted strategies, evaluating their effectiveness across different deployment scenarios. We propose specialized metrics for evaluating both attack effectiveness and defense performance, addressing the specific challenges of generative language models. Through our analysis, we identify critical limitations in current approaches and propose promising research directions, including integrated attack methodologies and adaptive defense mechanisms that balance security with model utility. This work serves NLP researchers, ML engineers, and security professionals seeking to protect language models in production environments.

Figures

Figures reproduced from arXiv: 2506.22521 by the authors.

Figure 1
Figure 1. Illustration of the Model Extraction Attack pipeline [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. A Taxonomy of Model Extraction Attacks & Defenses on Large Language Models. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Intellectual Property in Graph-Based Machine Learning as a Service: Attacks and Defenses

    cs.CR 2025-08 conditional novelty 4.0 of 10

    A systematic review that organizes graph-ML IP protection into model-level and data-level attacks and defenses, and ships a benchmark library, PyGIP.

  2. DESIGN: Encrypted GNN Inference via Server-Side Input Graph Pruning

    cs.CR 2025-07 reject novelty 4.0 of 10

    DESIGN uses encrypted node degrees to prune graphs and adaptively choose polynomial activations, reporting 1.7x-2.4x speedups over a basic FHE GNN baseline.

Reference graph

Works this paper leans on

101 extracted references · 28 canonical work pages · cited by 2 Pith papers

  1. [5]

    Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Itay Yona, Eric Wallace, David Rolnick, and Florian Tram‘er

    Nicholas Carlini, Daniel Paleka, Krishnamurthy Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Itay Yona, Eric Wallace, David Rolnick, and Florian Tram‘er. 2024. Stealing Part of a Production Language Model. arXiv preprint arXiv:2403.06634 (2024). https://arxiv.org/abs/2403.06634

    arXiv 2024

  2. [6]

    Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting Training Data from Large Lan- guage Models. In 30th USENIX Security Symposium (USENIX Security 21) . USENIX Association, 2633–2650

    2021

  3. [60]

    Zeyang Sha and Yang Zhang. 2024. Prompt Stealing Attacks Against Large Language Models. arXiv preprint arXiv:2402.12959 (2024)

    arXiv 2024

  4. [33]

    Kalpesh Krishna, Gaurav Singh Tomar, Ankur P Parikh, Nicolas Papernot, and Mohit Iyyer. 2020. Thieves on Sesame Street! Model Extraction of BERT-based APIs. In International Conference on Learning Representations. https://openreview. net/forum?id=Byl5NREFDr

    2020

  5. [17]

    Shanglun Feng and Florian Tramèr. 2024. Privacy backdoors: stealing data with corrupted pretrained models. arXiv preprint arXiv:2404.00473 (2024)

    arXiv 2024

  6. [3]

    Anahita Baninajjar, Kamran Hosseini, Ahmed Rezine, and Amir Aminifar. 2024. Verified relative safety margins for neural network twins. arXiv preprint arXiv:2409.16726 (2024)

    work page Pith review arXiv 2024

  7. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  8. [2]

    Francisco Aguilera-Martínez and Fernando Berzal. 2025. LLM Security: Vulnera- bilities, Attacks, Defenses, and Countermeasures. arXiv preprint arXiv:2505.01177 (2025)

    arXiv 2025

Show all 101 references
  1. [4]

    Lewis Birch, William Hackett, Stefan Trawicki, Neeraj Suri, and Peter Garraghan

  2. [7]

    Chen Chen, Xuanli He, Lingjuan Lyu, and Fangzhao Wu. 2021. Killing one bird with two stones: model extraction and attribute inference attacks against bert-based apis. arXiv preprint arXiv:2105.10909 (2021)

    2021 arXiv

  3. [8]

    Xueqi Cheng, Minxing Zheng, Shixiang Zhu, and Yushun Dong. 2025. MIS- LEADER: Defending against Model Extraction with Ensembles of Distilled Mod- els. arXiv preprint arXiv:2506.02362 (2025)

    2025 arXiv

  4. [9]

    Zhan Cheng, Bolin Shen, Tianming Sha, Yuan Gao, Shibo Li, and Yushun Dong

  5. [10]

    Jing Cui, Yishi Xu, Zhewei Huang, Shuchang Zhou, Jianbin Jiao, and Junge Zhang

  6. [11]

    Chenxi Dai, Lin Lu, and Pan Zhou. 2025. Stealing Training Data from Large Language Models in Decentralized Training through Activation Inversion Attack. arXiv preprint arXiv:2502.16086 (2025)

    2025 arXiv

  7. [12]

    Badhan Chandra Das, M Hadi Amini, and Yanzhao Wu. 2025. Security and privacy challenges of large language models: A survey. Comput. Surveys 57, 6 (2025), 1–39

    2025

  8. [13]

    Gunika Dhingra, Saumil Sood, Zeba Mohsin Wase, Arshdeep Bahga, and Vijay K Madisetti. 2024. Protecting LLMs against Privacy Attacks While Preserving Utility. Journal of Information Security 15, 4 (2024), 448–473

    2024

  9. [14]

    Yi Dong, Ronghui Mu, Yanghao Zhang, Siqi Sun, Tianle Zhang, Changshun Wu, Gaojie Jin, Yi Qi, Jinwei Hu, Jie Meng, et al. 2024. Safeguarding large language models: A survey. arXiv preprint arXiv:2406.02622 (2024)

    2024 arXiv

  10. [15]

    Aysan Esmradi, Daniel Wankit Yip, and Chun Fai Chan. 2023. A comprehensive survey of attack techniques, implementation, and mitigation strategies in large language models. In International Conference on Ubiquitous Security . Springer, 76–95

    2023

  11. [16]

    Luyang Fang, Xiaowei Yu, Jiazhang Cai, Yongkai Chen, Shushan Wu, Zhengliang Liu, Zhenyuan Yang, Haoran Lu, Xilin Gong, Yufang Liu, et al. 2025. Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions. arXiv p...

    2025

  12. [18]

    Denis V Gerasimenko and Dmitry Namiot. 2024. Extracting Training Data: Risks and solutions in the context of LLM security. International Journal of Open Information Technologies 12, 11 (2024), 9–19

    2024

  13. [19]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

    2025 arXiv

  14. [20]

    Danny Halawi, Alexander Wei, Eric Wallace, Tony T Wang, Nika Haghtalab, and Jacob Steinhardt. 2024. Covert malicious finetuning: Challenges in safeguarding llm adaptation. arXiv preprint arXiv:2406.20053 (2024)

    2024 arXiv

  15. [21]

    Xuanli He, Lingjuan Lyu, Qiongkai Xu, and Lichao Sun. 2021. Model extrac- tion and adversarial transferability, your BERT is vulnerable! arXiv preprint arXiv:2103.10013 (2021)

    2021 arXiv

  16. [22]

    Xuanli He, Qiongkai Xu, Yi Zeng, Lingjuan Lyu, Fangzhao Wu, Jiwei Li, and Ruoxi Jia. 2022. Cater: Intellectual property protection on text generation apis via conditional watermarks. Advances in Neural Information Processing Systems 35 (2022), 5431–5445

    2022

  17. [23]

    Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint a...

    2023 arXiv

  18. [24]

    Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. 2022. Are large pre- trained language models leaking your personal information? arXiv preprint arXiv:2205.12628 (2022)

    2022 arXiv

  19. [25]

    Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. 2024. O1 Replication Journey–Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson? arXiv preprint arXiv:2411.164...

    2024 arXiv

  20. [26]

    Bo Hui, Haolin Yuan, Neil Gong, Philippe Burlina, and Yinzhi Cao. 2024. Pleak: Prompt leaking attacks against large language model applications. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security. 3600–3614

    2024

  21. [27]

    Tyler Hunt, Congzheng Song, Reza Shokri, Vitaly Shmatikov, and Emmett Witchel. 2018. Chiron: Privacy-preserving machine learning as a service. arXiv preprint arXiv:1803.05961 (2018)

    2018 arXiv

  22. [28]

    Yoichi Ishibashi and Hidetoshi Shimodaira. 2023. Knowledge sanitization of large language models. arXiv preprint arXiv:2309.11852 (2023)

    2023 arXiv

  23. [29]

    Ziyou Jiang, Mingyang Li, Guowei Yang, Junjie Wang, Yuekai Huang, Zhiyuan Chang, and Qing Wang. 2025. Mimicking the Familiar: Dynamic Command Generation for Information Theft Attacks in LLM Tool-Learning System. arXiv preprint arXiv:2502.11358 (2025)

    2025 arXiv

  24. [30]

    Manish Kesarwani, Bhaskar Mukhoty, Vijay Arya, and Sameep Mehta. 2018. Model extraction warning in MLaaS paradigm. In Proceedings of the 34th annual computer security applications conference . 371–380

    2018

  25. [31]

    Hanjoo Kim, Minkyu Kim, Dongjoo Seo, Jinwoong Kim, Heungseok Park, Soeun Park, Hyunwoo Jo, KyungHyun Kim, Youngil Yang, Youngkwan Kim, et al. 2018. Nsml: Meet the mlaas platform with a real-world case study. arXiv preprint arXiv:1810.09957 (2018)

    2018 arXiv

  26. [32]

    Minjae Kim, Taehyeong Kwon, Kibeom Shim, and Beonghoon Kim. 2024. Pro- tection of LLM Environment Using Prompt Security. In 2024 15th International Conference on Information and Communication Technology Convergence (ICTC) . IEEE, 1715–1719

    2024

  27. [34]

    Sunbowen Lee, Junting Zhou, Chang Ao, Kaige Li, Xinrun Du, Sirui He, Haihong Wu, Tianci Liu, Jiaheng Liu, Hamid Alinejad-Rokny, Min Yang, Yitao Liang, Zhoufutu Wen, and Shiwen Ni. 2025. Quantification of Large Language Model Distillation. arXiv:2501.12619

    2025 arXiv

  28. [35]

    Chenyang Li, Zhao Song, Weixin Wang, and Chiwun Yang. 2023. A theoretical insight into attack and defense of gradient leakage in transformer. arXiv preprint arXiv:2311.13624 (2023)

    2023 arXiv

  29. [36]

    Qinbin Li, Junyuan Hong, Chulin Xie, Jeffrey Tan, Rachel Xin, Junyi Hou, Xavier Yin, Zhun Wang, Dan Hendrycks, Zhangyang Wang, et al. 2024. Llm-pbe: As- sessing data privacy in large language models. arXiv preprint arXiv:2408.12787 (2024)

    2024 arXiv

  30. [37]

    Qinfeng Li, Zhiqiang Shen, Zhenghan Qin, Yangfan Xie, Xuhong Zhang, Tianyu Du, Sheng Cheng, Xun Wang, and Jianwei Yin. 2024. TransLinkGuard: Safe- guarding Transformer Models Against Model Stealing in Edge Deployment. In Proceedings of the 32nd ACM International Conference on ...

    2024

  31. [38]

    Qinfeng Li, Yangfan Xie, Tianyu Du, Zhiqiang Shen, Zhenghan Qin, Hao Peng, Xinkui Zhao, Xianwei Zhu, Jianwei Yin, and Xuhong Zhang. 2024. CoreGuard: Safeguarding Foundational Capabilities of LLMs Against Model Stealing in Edge Deployment. arXiv preprint arXiv:2410.13903 (2024)

    2024 arXiv

  32. [39]

    Zi Liang, Haibo Hu, Qingqing Ye, Yaxin Xiao, and Haoyang Li. 2024. Why Are My Prompts Leaked? Unraveling Prompt Extraction Threats in Customized Large Language Models. arXiv preprint arXiv:2408.02416 (2024)

    2024 arXiv

  33. [40]

    Aiwei Liu, Leyi Pan, Yijian Lu, Jingjing Li, Xuming Hu, Xi Zhang, Lijie Wen, Irwin King, Hui Xiong, and Philip Yu. 2024. A survey of text watermarking in the era of large language models. Comput. Surveys 57, 2 (2024), 1–36

    2024

  34. [41]

    Feng Liu, Jiaqi Jiang, Yating Lu, Zhanyi Huang, and Jiuming Jiang. 2025. The ethical security of large language models: A systematic review. Frontiers of Engineering Management (2025), 1–13

    2025

  35. [42]

    Jiaheng Liu, Chenchen Zhang, Jinyang Guo, Yuanxing Zhang, Haoran Que, Ken Deng, Jie Liu, Ge Zhang, Yanan Wu, Congnan Liu, et al. 2024. Ddk: Distilling domain knowledge for efficient large language models. Advances in Neural Information Processing Systems 37 (2024), 98297–98319

    2024

  36. [43]

    Yupei Liu, Jinyuan Jia, Hongbin Liu, and Neil Zhenqiang Gong. 2022. Stolenen- coder: stealing pre-trained encoders in self-supervised learning. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security . 2115–2128

    2022

  37. [44]

    Xingjun Ma, Yifeng Gao, Yixu Wang, Ruofan Wang, Xin Wang, Ye Sun, Yifan Ding, Hengyuan Xu, Yunhao Chen, Yunhan Zhao, et al. 2025. Safety at Scale: A Comprehensive Survey of Large Model Safety. arXiv preprint arXiv:2502.05206 (2025). KDD ’25, August 3–7, 2025, Toronto, ON, Cana...

    2025 arXiv

  38. [45]

    Loïc Maréchal. 2024. The Flow of Investments in the LLM Space. In Large Language Models in Cybersecurity: Threats, Exposure and Mitigation . Springer Nature Switzerland Cham, 129–135

    2024

  39. [46]

    Eleena Mathew. 2024. Enhancing Security in Large Language Models: A Compre- hensive Review of Prompt Injection Attacks and Defenses. Authorea Preprints (2024)

    2024

  40. [47]

    Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. 2024. Tree of attacks: Jailbreaking black-box llms automatically. Advances in Neural Information Processing Systems 37 (2024), 61065–61105

    2024

  41. [48]

    Najmeh Nazari, Furi Xiang, Chongzhou Fang, Hosein Mohammadi Makrani, Aditya Puri, Kartik Patwari, Hossein Sayadi, Setareh Rafatirad, Chen-Nee Chuah, and Houman Homayoun. 2024. LLM-FIN: Large Language Models Fingerprint- ing Attack on Edge Devices. In 2024 25th International Sy...

    2024

  42. [49]

    Kaiyi Pang, Tao Qi, Chuhan Wu, Minhao Bai, Minghu Jiang, and Yongfeng Huang

  43. [50]

    Rahil Parikh, Christophe Dupuy, and Rahul Gupta. 2022. Canary extraction in natural language understanding models. arXiv preprint arXiv:2203.13920 (2022)

    2022 arXiv

  44. [51]

    Vaidehi Patil, Peter Hase, and Mohit Bansal. 2023. Can sensitive information be deleted from llms? objectives for defending against extraction attacks. arXiv preprint arXiv:2309.17410 (2023)

    2023 arXiv

  45. [52]

    Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527 (2022)

    2022 arXiv

  46. [53]

    IEEE Transactions on Information Forensics and Security (2025)

    ModelShield: Adaptive and Robust Watermark against Model Extraction Attack. IEEE Transactions on Information Forensics and Security (2025)

    2025

  47. [54]

    Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, et al. 2024. O1 Replication Journey: A Strategic Progress Report–Part 1. arXiv preprint arXiv:2410.18982 (2024)

    2024 arXiv

  48. [55]

    Adnan Siraj Rakin, Md Hafizul Islam Chowdhuryy, Fan Yao, and Deliang Fan. 2022. Deepsteal: Advanced model extractions leveraging efficient weight stealing in memories. In 2022 IEEE symposium on security and privacy (SP) . IEEE, 1157–1174

    2022

  49. [56]

    Vishal Rathod, Seyedsina Nabavirazavi, Samira Zad, and Sundararaja Sitharama Iyengar. 2025. Privacy and security challenges in large language models. In 2025 IEEE 15th Annual Computing and Communication Workshop and Conference (CCWC). IEEE, 00746–00752

    2025

  50. [57]

    Robert Philipp, Andreas Mladenow, Christine Strauss, and Alexander Völz. 2020. Machine learning as a service: Challenges in research and applications. InProceed- ings of the 22nd International Conference on Information Integration and Web-based Applications & Services. 396–406

    2020

  51. [58]

    Mauro Ribeiro, Katarina Grolinger, and Miriam AM Capretz. 2015. Mlaas: Machine learning as a service. In 2015 IEEE 14th international conference on machine learning and applications (ICMLA) . IEEE, 896–902

    2015

  52. [59]

    Goldman Sachs. 2023. AI investment forecast to approach $200 billion globally by 2025. Artificial intelligence outlooks–01 AUG (2023)

    2023

  53. [61]

    Robert Nikolai Reith, Thomas Schneider, and Oleksandr Tkachenko. 2019. Effi- ciently stealing your machine learning models. In Proceedings of the 18th ACM Workshop on Privacy in the Electronic Society . 198–210

    2019

  54. [62]

    Anup Shirgaonkar, Nikhil Pandey, Nazmiye Ceren Abay, Tolga Aktas, and Vijay Aski. 2024. Knowledge Distillation Using Frontier Open-source LLMs: Gen- eralizability and the Role of Synthetic Data. arXiv preprint arXiv:2410.18588 (2024)

    2024 arXiv

  55. [63]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

    2023 arXiv

  56. [64]

    Stephen Burabari Tete. 2024. Threat modelling and risk analysis for large language model (llm)-powered applications. arXiv preprint arXiv:2406.11007 (2024)

    2024 arXiv

  57. [65]

    Xinyue Shen, Yiting Qu, Michael Backes, and Yang Zhang. 2024. Prompt Stealing Attacks Against{Text-to-Image} Generation Models. In 33rd USENIX Security Symposium (USENIX Security 24) . 5823–5840

    2024

  58. [66]

    Anvesh Rao Vijjini, Somnath Basu Roy Chowdhury, and Snigdha Chaturvedi

  59. [67]

    Jeffrey G Wang, Jason Wang, Marvin Li, and Seth Neel. 2024. Pandora’s White- Box: Precise Training Data Detection and Extraction in Large Language Models. arXiv preprint arXiv:2402.17012 (2024)

    2024 arXiv

  60. [68]

    Liaoyaqi Wang and Minhao Cheng. 2024. GuardEmb: Dynamic Watermark for Safeguarding Large Language Model Embedding Service Against Model Stealing Attack. In Findings of the Association for Computational Linguistics: EMNLP 2024 . 7518–7534

    2024

  61. [69]

    Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart

  62. [70]

    Xinyi Wang, Antonis Antoniades, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, and William Yang Wang. 2025. Generalization v.s. Memorization: Tracing Language Models’ Capabilities Back to Pretraining Data. InThe Thirteenth International Conference on Learning Repr...

    2025

  63. [71]

    Yu Wang, Cailing Cai, Zhihua Xiao, and Peifung E Lam. 2025. LLM Access Shield: Domain-Specific LLM Framework for Privacy Policy Compliance. arXiv preprint arXiv:2505.17145 (2025)

    2025 arXiv

  64. [72]

    arXiv preprint arXiv:2406.11107 (2024)

    Exploring safety-utility trade-offs in personalized language models. arXiv preprint arXiv:2406.11107 (2024)

    2024 arXiv

  65. [73]

    Zezhong Wang, Fangkai Yang, Lu Wang, Pu Zhao, Hongru Wang, Liang Chen, Qingwei Lin, and Kam-Fai Wong. 2023. Self-guard: Empower the llm to safeguard itself. arXiv preprint arXiv:2310.15851 (2023)

    2023 arXiv

  66. [74]

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems 36 (2023), 80079–80110

    2023

  67. [75]

    Shang Wang, Tianqing Zhu, Bo Liu, Ming Ding, Xu Guo, Dayong Ye, Wanlei Zhou, and Philip S Yu. 2024. Unique security and privacy threats of large language model: A comprehensive survey. arXiv preprint arXiv:2406.07973 (2024)

    2024

  68. [76]

    Jiashu Xu, Fei Wang, Mingyu Derek Ma, Pang Wei Koh, Chaowei Xiao, and Muhao Chen. 2024. Instructional fingerprinting of large language models. arXiv preprint arXiv:2401.12255 (2024)

    2024 arXiv

  69. [77]

    Qiongkai Xu, Xuanli He, Lingjuan Lyu, Lizhen Qu, and Gholamreza Haffari

  70. [78]

    Zebin Wang, Menghan Lin, Bolin Shen, Ken Anderson, Molei Liu, Tianxi Cai, and Yushun Dong. 2025. CEGA: A Cost-Effective Approach for Graph-Based Model Extraction and Acquisition. arXiv:2506.17709 [cs.LG] https://arxiv.org/ abs/2506.17709

    2025 arXiv

  71. [79]

    Biwei Yan, Kun Li, Minghui Xu, Yueyan Dong, Yue Zhang, Zhaochun Ren, and Xiuzhen Cheng. 2024. On protecting the data privacy of large language models (llms): A survey. arXiv preprint arXiv:2403.05156 (2024)

    2024 arXiv

  72. [80]

    Mingke Yang, Yuqi Chen, Yi Liu, and Ling Shi. 2024. DistillSeq: A Framework for Safety Alignment Testing in Large Language Models using Knowledge Dis- tillation. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 578–589

    2024

  73. [81]

    Fangzhou Wu, Ning Zhang, Somesh Jha, Patrick McDaniel, and Chaowei Xiao

  74. [82]

    arXiv preprint arXiv:2402.18649 (2024)

    A new era in llm security: Exploring security concerns in real-world llm-based systems. arXiv preprint arXiv:2402.18649 (2024)

    2024 arXiv

  75. [83]

    Zhou Yang, Zhipeng Zhao, Chenyu Wang, Jieke Shi, Dongsun Kim, Donggyun Han, and David Lo. 2024. Unveiling memorization in code models. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering . 1–13

    2024

  76. [84]

    Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. 2024. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing (2024), 100211

    2024

  77. [85]

    Yuanshun Yao, Zhujun Xiao, Bolun Wang, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. 2017. Complexity vs. performance: empirical analysis of machine learning as a service. In Proceedings of the 2017 Internet Measurement Conference . 384–397

    2017

  78. [86]

    Wenrui Xu and Keshab K Parhi. 2025. A Survey of Attacks on Large Language Models. arXiv preprint arXiv:2505.12567 (2025)

    2025 arXiv

  79. [87]

    Collin Zhang, John X Morris, and Vitaly Shmatikov. 2024. Extracting prompts by inverting llm outputs. arXiv preprint arXiv:2405.15012 (2024)

    2024 arXiv

  80. [88]

    Ruisi Zhang, Seira Hidano, and Farinaz Koushanfar. 2022. Text revealer: Private text reconstruction via model inversion attacks against transformers. arXiv preprint arXiv:2209.10505 (2022)

    2022 arXiv

  81. [89]

    Wenkai Yang, Yankai Lin, Jie Zhou, and Ji-Rong Wen. 2025. Distilling Rule-based Knowledge into Large Language Models. In Proceedings of the 31st International Conference on Computational Linguistics . 913–932

    2025

  82. [90]

    Yong Yang, Changjiang Li, Yi Jiang, Xi Chen, Haoyu Wang, Xuhong Zhang, Zonghui Wang, and Shouling Ji. 2024. PRSA: PRompt Stealing Attacks against large language models. arXiv preprint arXiv:2402.19200 (2024)

    2024 arXiv

  83. [91]

    Zhexin Zhang, Jiaxin Wen, and Minlie Huang. 2023. Ethicist: Targeted training data extraction through loss smoothed soft prompting and calibrated confidence estimation. arXiv preprint arXiv:2307.04401 (2023)

    2023 arXiv

  84. [92]

    Kaixiang Zhao, Lincan Li, Kaize Ding, Neil Zhenqiang Gong, Yue Zhao, and Yushun Dong. 2025. A Survey of Model Extraction Attacks and Defenses in Distributed Computing Environments. arXiv preprint arXiv:2502.16065 (2025)

    2025 arXiv

  85. [93]

    Zhengyue Zhao, Xiaogeng Liu, Somesh Jha, Patrick McDaniel, Bo Li, and Chaowei Xiao. [n. d.]. Can Watermarks be Used to Detect LLM IP Infringement For Free?. In The Thirteenth International Conference on Learning Representations

  86. [94]

    Yizhen Yuan, Rui Kong, Yuanchun Li, and Yunxin Liu. 2024. Wip: An on-device llm-based approach to query privacy protection. In Proceedings of the Workshop on Edge and Mobile Foundation Models . 7–9

    2024

  87. [97]

    2024.{REMARK-LLM}: A robust and efficient watermarking framework for generative large language models

    Ruisi Zhang, Shehzeen Samarah Hussain, Paarth Neekhara, and Farinaz Koushan- far. 2024.{REMARK-LLM}: A robust and efficient watermarking framework for generative large language models. In 33rd USENIX Security Symposium (USENIX Security 24). 1813–1830

    2024

  88. [98]

    Yuehan Zhang, Peizhuo Lv, Yinpeng Liu, Yongqiang Ma, Wei Lu, Xiaofeng Wang, Xiaozhong Liu, and Jiawei Liu. 2024. PersonaMark: Personalized LLM watermark- ing for model protection and user attribution. arXiv preprint arXiv:2409.09739 (2024)

    2024 arXiv

  89. [2016]

    In 25th USENIX security symposium (USENIX Security 16)

    Stealing machine learning models via prediction{APIs}. In 25th USENIX security symposium (USENIX Security 16) . 601–618

  90. [2021]

    arXiv preprint arXiv:2108.13873 (2021)

    Student surpasses teacher: Imitation attack for black-box NLP APIs. arXiv preprint arXiv:2108.13873 (2021)

    2021 arXiv

  91. [2023]

    arXiv preprint arXiv:2309.10544 (2023)

    Model leeching: An extraction attack targeting llms. arXiv preprint arXiv:2309.10544 (2023)

    2023 arXiv

  92. [2024]

    arXiv preprint arXiv:2409.03274 (2024)

    Recent advances in attack and defense approaches of large language models. arXiv preprint arXiv:2409.03274 (2024)

    2024 arXiv

  93. [2025]

    arXiv preprint arXiv:2503.16693 (2025)

    ATOM: A Framework of Detecting Query-Based Model Extraction Attacks for Graph Neural Networks. arXiv preprint arXiv:2503.16693 (2025)

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.