Pith. sign in

REVIEW 4 major objections 5 minor 71 references

A black-box attacker can extract a large fraction of a private RAG knowledge base, even when the backend LLM is safety-aligned and told not to disclose.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 12:38 UTC pith:KEXEUKJ7

load-bearing objection A genuinely useful framework and a clever attack, but the 'beats all baselines' claim is not established as stated because the baselines were modified. the 4 major comments →

arxiv 2510.02964 v2 pith:KEXEUKJ7 submitted 2025-10-03 cs.CR

External Data Extraction Attacks against Retrieval-Augmented Large Language Models

Yu He , Yifei Chen , Yiming Li , Shuo Shao , Leyi Qi , Boheng Li , Dacheng Tao , Zhan Qin This is my paper
classification cs.CR
keywords retrieval-augmented generationexternal data extraction attackjailbreak prompt optimizationcluster-focused triggeringblack-box privacy attackprompt injectionLLM data leakageRAG security
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that retrieval-augmented generation (RAG) systems, which let an LLM answer from a private knowledge base, are vulnerable to a practical black-box data-extraction attack. The authors formalize these attacks as having three components—an extraction instruction, a jailbreak operator, and a retrieval trigger—and show that prior attacks, which strengthen only one or two of these, fail against safety-aligned commercial models. They then build SECRET, which uses an LLM-driven optimizer to generate a natural jailbreak prompt and alternates broad exploration with local semantic exploitation to find new documents efficiently. Across 16 RAG instances built on four models, SECRET extracts roughly a third to a half of a 1,000-document private database, including 35% from a RAG powered by Claude 3.7 Sonnet where all prior attacks extracted 0%. If correct, the default RAG deployment—a dense retriever, top-10 context, no output filtering—offers little privacy protection, and defenders need output-side guards, access control, or stricter retrieval thresholds.

Core claim

The paper claims that external data extraction from retrieval-augmented LLMs is a realistic black-box threat: an attacker who only submits prompts and reads responses can extract a large fraction of a private knowledge base verbatim, even when the backend model is safety-aligned and the system prompt forbids disclosure. Prior attacks fail, the paper argues, because they strengthen only one piece of the puzzle; a successful attack must simultaneously force the retriever to surface unextracted documents and force the generator to repeat them verbatim. SECRET does this with an LLM-driven jailbreak-prompt optimizer and cluster-focused triggering. Across 16 RAG instances on four models it reaches

What carries the argument

The load-bearing object is SECRET's two-phase pipeline. Phase one is an adaptive jailbreak-prompt optimizer: an optimizer LLM proposes candidate prompts, an evaluator LLM scores them by whether the target model repeats retrieved documents verbatim, and only the best candidate enters a score-sorted history, with curriculum seeding from weaker models for heavily aligned targets. Phase two is cluster-focused triggering (CFT), which alternates global exploration (random natural-text chunks as triggers) with local exploitation (LLM-fused semantic shifts of discovered documents) to navigate embedding space and surface new documents. The framework's conceptual move is decomposing any EDEA query int

Load-bearing premise

The headline numbers assume the target RAG deployment matches the lab setup: a dense retriever returning a few documents into the context, an API that returns the LLM's full unmoderated response, and no output-side filtering, per-document access control, or authentication—if production systems add any of those, the extraction rates do not automatically transfer.

What would settle it

Run SECRET against the same RAG setup but with an output-side guard that blocks any response whose ROUGE-L or semantic similarity to the retrieved documents exceeds a threshold, both with and without light obfuscation; if extraction still approaches 35% of a 1,000-document database, the threat is real, but if it drops to the level of prior attacks, the paper's headline result depends on the absence of output filtering.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If SECRET works as reported, any RAG service that returns unmoderated LLM output can be drained of a large fraction of its knowledge base by a black-box user, with no knowledge of the retriever, system prompt, or database contents.
  • Simple system-prompt 'do not repeat' defenses are insufficient; the attack adapts its jailbreak prompt to bypass them, and generic jailbreak or prompt-stealing methods fail where SECRET succeeds.
  • Optimized jailbreak prompts transfer across models to a useful degree, so an attacker can optimize on a weaker model and still attack stronger commercial systems effectively.
  • Retrieval similarity thresholds reduce extraction but do not stop it; the paper measures 24.6% extraction even when about half of relevant documents are filtered out, implying a privacy–utility trade-off.
  • Because extraction is verbatim, the attack is a direct exfiltration path for medical records, financial data, and internal documents in customized RAG deployments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural next step the paper leaves implicit is that defenders should focus on output-side filtering, per-document access control, rate limits, and response monitoring rather than prompt wording alone.
  • Cluster-focused triggering could generalize to other high-dimensional search problems, such as membership inference or dataset auditing, where global distance metrics are uninformative but local cluster structure persists.
  • The 35% figure should be read as a lower bound under adversarial conditions; real deployments with logging, anomaly detection, and query budgets could blunt or detect the attack, and testing under those operational constraints is a concrete extension.
  • The formal advantage proof assumes idealized cluster geometry; the Enron experiments suggest the advantage shrinks when documents are short and unstructured, so real database clustering quality is the key contingency.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper formalizes external data extraction attacks (EDEAs) against retrieval-augmented LLMs, decomposing attacks into extraction instruction, jailbreak operator, and retrieval trigger; proposes SECRET, combining LLM-driven jailbreak-prompt optimization with cluster-focused triggering (CFT); and evaluates SECRET across 4 LLMs (Claude 3.7 Sonnet, Gemini 2.0 Flash, GPT-4o mini, DeepSeek-V3), 2 datasets, and 2 defense settings. The main claimed findings are that SECRET outperforms five prior/adapted EDEA baselines, extracts up to 34-36% of documents from a 1000-document RAG database even against safety-aligned commercial models, and that CFT enjoys a formal query-efficiency advantage over non-adaptive global exploration. The paper also studies several defenses, showing that simple system-prompt defenses and similarity-threshold defenses are insufficient to stop SECRET.

Significance. If the empirical claims hold, the paper would be a substantial contribution: it provides the first systematic decomposition of EDEAs, a scalable attack that defeats state-of-the-art aligned LLMs in a black-box RAG setting, and broad evidence that existing defenses are inadequate. The strength of the paper is its breadth: 16 RAG instances, two sensitive datasets, an additive ablation isolating each component, robustness checks across embedding mismatches and RAG templates, and evaluation of several defenses. The design of SECRET is also plausible and well-motivated, especially the curriculum-based jailbreak optimization and the alternating GE/LE trigger strategy. However, the paper's comparative claim depends on modified or reimplemented baselines, and the formal analysis of CFT contains gaps. These issues affect the headline 'significantly outperforms previous attacks' and the claimed proof of CFT's advantage, so the significance is currently conditional on revisions.

major comments (4)
  1. [Appendix II-A and Tables II/III/VIII/IX] The head-to-head comparison is not against the attacks as published. Appendix II-A states that all baselines were 'combined with our base extraction instruction pe' because their original prompts were ineffective; Jiang et al.'s RAG-Thief was reimplemented from a paper description; Cohen et al.'s original prompt was replaced; and GCG-Leak is an adapted baseline using a LLaMA2-7B surrogate. The abstract's 'significantly outperforms previous attacks' and the specific '0% vs 35%' against Claude 3.7 are therefore comparisons to modified variants, and the 0% could be an artifact of the modifications. Please run the original released attacks if code is available, or otherwise clearly report both original-version and modified-version results, and temper the superiority claim to what the experiments actually support.
  2. [Appendix A, Theorem I.1] The theorem asserts a unique threshold Q* at which CFT strictly outperforms GE, but the proof requires an unstated condition. Monotonicity of G' across breakpoints is established only when Eq. (13) holds, and the appendix itself notes that one must choose b so that Δ_dec ≳ c(b). Without that condition, uniqueness of the crossing is not proved. In addition, L_CFT(Q) is never explicitly defined; the statement that it is piecewise linear with non-decreasing slope is not reconciled with the cycle structure, and the proof compares bounds (U_GE, L_CFT) without formalizing the expectations D_GE, D_CFT. As written, the formal analysis does not establish the claimed advantage. Please repair the proof, state the theorem with all needed conditions, or downgrade the claim to a heuristic supported by the empirical comparisons.
  3. [Section V-A, MER metric] MER is defined at k×TMQ with k=3, chosen 'empirically' because extraction has 'largely converged.' This penalizes slower-converging baselines and is circular when the convergence behavior is itself the property being compared. Report extraction-rate curves at multiple budgets (e.g., k=3,5,10) or use a budget-independent measure such as area under the extraction curve. The paper also reports no confidence intervals or repeated-seed variation for any table, despite the stochasticity of LLM optimization and retrieval; please add multiple runs or bootstrap CIs for the headline numbers.
  4. [Section V-B and Appendix IV-D] Takeaway 3 and the abstract attribute the 35% Claude result to SECRET being 'highly effective against all 16 tested RAG instances.' This is supported by Tables II/III, but the case study in Appendix IV-D shows that the adapted generic jailbreaks (PAP, LLM-Fuzzer, PLeak) fail against Claude; those results are useful, but they are not published EDEA attacks and should not be used as evidence that SECRET outperforms all prior task-specific attacks. The central limitation remains the baseline fairness issue raised above.
minor comments (5)
  1. [Appendix references] The body text uses inconsistent appendix labels: Section II-A refers to 'Appendix V' (there is no Appendix V; the confidentiality discussion is Appendix E), and Section IV-C refers to 'Appendix IV-D' while the actual appendix is labeled D. Please harmonize the appendix numbering throughout.
  2. [Notation conflict] The symbol k is used both for the number of retrieved documents per query and for the MER query-budget multiplier k×TMQ. This can confuse the reader, especially in Section V-A and Figure 3. Consider renaming one of them (e.g., K_r for retrieved count and m for the multiplier).
  3. [Table X] In the transferability matrix, the GPT-4o mini row appears to contain five symbols (✗ ✔ ✗ ✔ ✔) while the header expects four columns; the row should be corrected. Table X is also not referenced in the main text; please add a pointer in Section V or Appendix IV-C.
  4. [Appendix A, Lemma I.1] The proof of the query-complexity bound assumes a BFS-like branching factor k on the δmax-adjacency graph, but no argument is given that queries generated by the LE operator correspond to edges of this graph or that the graph diameter bound is achievable with the specific LLM-based fshift. Please state these as assumptions explicitly.
  5. [Section VI-B, threshold naming] The retriever similarity threshold τ in Figure 5 and Appendix II-D is notationally identical to the normalized Levenshtein threshold τ in Definition III.1, but they are different quantities. Please use distinct symbols (e.g., ρ for the retriever threshold).

Circularity Check

0 steps flagged

No significant circularity: SECRET's extraction rates are independently measured, and the CFT formalization is a conditional theorem rather than a by-construction reduction.

full rationale

The paper's central claim—that SECRET extracts a substantial fraction of RAG documents—is not a fitted parameter masquerading as a prediction. Extraction success is measured with normalized Levenshtein distance against the actual documents (Definition III.1, Eq. 2), and the jailbreak optimization loop (Algorithm 1) uses only the target model's responses and an LLM evaluator, not the ground-truth database. The extraction rates in Tables II/III are therefore independent measurements, not quantities that equal their inputs by construction. The CFT advantage theorem in Appendix A is conditional on explicit cluster-structure, separation, and LE-connectivity assumptions; it is an idealized model and should be discounted if those premises are not met, but it does not define its conclusion into its assumptions. The admitted baseline modifications/reimplementations in Appendix B-A raise a fair-comparison and reproducibility concern, not a circularity. Self-citations (e.g., [15], [16]) appear only as contextual support for the value of protected data and are not load-bearing for the attack's derivation. I therefore find no by-construction reduction between the paper's inputs and its claimed results.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central empirical result rests on a small number of hand-chosen tolerances and deployment assumptions, most notably tau=0.1, the MER multiplier k=3, and the absence of output-side filtering. The formal CFT proof adds many idealized geometric parameters that are not tied to real embeddings, so the proof should be read as a stylized model, not as evidence for the empirical extraction rates.

free parameters (5)
  • Normalized Levenshtein threshold tau = 0.1
    Hand-set in Definition III.1 to tolerate whitespace and spelling noise; every RR, ER-TMQ, and MER number depends on this success threshold.
  • MER convergence multiplier k = 3
    Chosen in Section V-A because extraction rate has 'largely converged'; the headline 35% Claude number is MER at this operating point.
  • Top-k retrieved documents per query = 10
    Set in Appendix II-B; this batch size defines TMQ and therefore all extraction-efficiency metrics.
  • Jailbreak optimization hyperparameters = Niter=20, Ncand=3, alpha=1.0, LE budget=30, temperature=0.0
    Set in Section V-A; only Ncand and LE budget receive partial sensitivity analysis.
  • CFT formal-analysis parameters = M, V, gamma_c, gamma_s, delta_max, b (assumed, not estimated)
    Appendix I assumes homogeneous, well-separated, connected clusters and bounded LE perturbations; the Q-star formula depends on these idealized quantities.
axioms (5)
  • domain assumption Document embeddings form locally clustered and connected structures that LE can navigate with delta_max-bounded perturbations, with separation gamma_s > gamma_c + 2 delta_max.
    Section IV-D and Appendix I-A; the formal advantage theorem depends on this cluster geometry and it is not verified on the real embedding spaces used in the experiments.
  • domain assumption The LLM evaluator's 0-1 rubric is a reliable reward signal for verbatim extraction without access to the database D.
    Section IV-C; fe scores responses in a zero-shot manner. If the evaluator is biased or gameable, the optimizer may chase prompt quirks rather than true leakage.
  • domain assumption The attacker can observe full unmoderated responses from the RA-LLM, with no output-side filtering, data-loss prevention, or rate limiting.
    Threat model in Section III-A; central to whether the lab results transfer to production RAG deployments.
  • domain assumption Extracted strings come from the retrieved context, not from the LLM's pretraining memory.
    Supported by the paper's 0% extraction on bare LLMs without RAG in Section V-A, but it remains an assumption for each extracted document in the main runs.
  • standard math Standard probability and metric-space facts used in Appendix I: triangle inequality, binomial/coupon-collector bounds, BFS covering, concavity of the GE upper bound.
    Used in the proof of Theorem I.1; these background facts are not new and are not the paper's contribution.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of External Data Extraction Attacks against Retrieval-Augmented Large Language Models." pith.science (2026). https://pith.science/paper/KEXEUKJ7

@misc{pith2026251002964,
  author       = {Pith},
  title        = {Pith review of: External Data Extraction Attacks against Retrieval-Augmented Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KEXEUKJ7}},
  note         = {Machine review of arXiv:2510.02964}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

In recent years, RAG has emerged as a key paradigm for enhancing large language models (LLMs). By integrating externally retrieved information, RAG alleviates issues like outdated knowledge and, crucially, insufficient domain expertise. While effective, RAG introduces new risks of external data extraction attacks (EDEAs), where sensitive or copyrighted data in its knowledge base may be extracted verbatim. These risks are particularly acute when RAG is used to customize specialized LLM applications with private knowledge bases. Despite initial studies exploring these risks, they often lack a formalized framework, robust attack performance, and comprehensive evaluation, leaving critical questions about real-world EDEA feasibility unanswered. In this paper, we present the first comprehensive study to formalize EDEAs against retrieval-augmented LLMs. We first formally define EDEAs and propose a unified framework decomposing their design into three components: extraction instruction, jailbreak operator, and retrieval trigger, under which prior attacks can be considered instances within our framework. Guided by this framework, we develop SECRET: a Scalable and EffeCtive exteRnal data Extraction aTtack. Specifically, SECRET incorporates (1) an adaptive optimization process using LLMs as optimizers to generate specialized jailbreak prompts for EDEAs, and (2) cluster-focused triggering, an adaptive strategy that alternates between global exploration and local exploitation to efficiently generate effective retrieval triggers. Extensive evaluations across 4 models reveal that SECRET significantly outperforms previous attacks, and is highly effective against all 16 tested RAG instances. Notably, SECRET successfully extracts 35% of the data from RAG powered by Claude 3.7 Sonnet for the first time, whereas other attacks yield 0% extraction. Our findings call for attention to this emerging threat.

Figures

Figures reproduced from arXiv: 2510.02964 by Boheng Li, Dacheng Tao, Leyi Qi, Shuo Shao, Yifei Chen, Yiming Li, Yu He, Zhan Qin.

Figure 1
Figure 1. Figure 1: The pipeline of Retrieval-Augmented Generation. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overall attack pipeline of our SECRET. This analysis exposes a key limitation of prior work, i.e., their focus on isolated improvements. For instance, an attack with a sophisticated trigger but no jailbreak operator still fails against a well-aligned LLM. Our framework not only unifies these fragmented efforts but also motivates a more holistic design. SECRET is the first to adopt this approach, simultaneo… view at source ↗
Figure 3
Figure 3. Figure 3: Evolution of extraction rates of SECRET under different k. The ER-TMQ results are annotated in the figure. D. Robustness Assessment In this section, we assess the robustness of SECRET under various external parameter settings. We use Gemini 2.0 Flash as the backend LLM unless otherwise specified. Retrieved Document Count (k). The number of retrieved documents k is set by the application owner. We test SECR… view at source ↗
Figure 4
Figure 4. Figure 4: The effectiveness of SECRET after varying the retrieval settings. The dataset is HealthcareMagic-101 [60]. TABLE V: The effectiveness of SECRET under different RAG templates. The backend LLM is GPT-4o mini. RAG Templates RR ↓ ER-TMQ ↑ You’ve been equipped with Retrieval-A. . . 6.0% 29.0% You have to answer the next question:{Q. . . 8.0% 27.8% Given the following question and context. . . 0.0% 33.7% Answer … view at source ↗
Figure 5
Figure 5. Figure 5: The effectiveness of SECRET under different τ . The dataset is HealthcareMagic-101. restrict the retriever to return only documents whose distance from the query is below a threshold τ . To evaluate the effectiveness of this defense under practical settings, we test a range of τ values and measure the changes in ER-TMQ. The threshold design methodology is detailed in Appendix II-D. As shown in [PITH_FULL_… view at source ↗
Figure 6
Figure 6. Figure 6: Impact of LE query budget (ϵlocal) on MER. 1 2 3 4 5 7 10 Ncand 0 4 8 12 16 20 Total Queries [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Average total queries to target f during optimization vs. Ncand. a reasonable range [10, 100] and measure the resulting MER on the HealthcareMagic-101 dataset using the GPT-4o mini as backend LLM. The results are plotted in [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Evolution of extraction rates of different attacks against naive defenses. The dataset is HealthcareMagic-101. [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Evolution of extraction rates of different attacks. The dataset is HealthcareMagic-101. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Evolution of extraction rates of different attacks against naive defenses. The dataset is Enron Mail. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Evolution of extraction rates of different attacks. The dataset is Enron Mail. [PITH_FULL_IMAGE:figures/full_fig_p021_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

71 extracted references · 21 linked inside Pith

  1. [1]

    Finetuned language models are zero-shot learners,

    J. Wei, M. Bosma, V . Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V . Le, “Finetuned language models are zero-shot learners,” inICLR, 2022

    2022

  2. [2]

    Codegen: An open large language model for code with multi-turn program synthesis,

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” inICLR, 2023

    2023

  3. [3]

    Prompting large language model for machine translation: A case study,

    B. Zhang, B. Haddow, and A. Birch, “Prompting large language model for machine translation: A case study,” inICML, 2023

    2023

  4. [4]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,

    L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qinet al., “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” arXiv preprint arXiv:2311.05232, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    A survey of large language models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Donget al., “A survey of large language models,”arXiv preprint arXiv:2303.18223, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Retrieval- augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschelet al., “Retrieval- augmented generation for knowledge-intensive nlp tasks,”NeurIPS, 2020

    2020

  7. [7]

    Retrieval-augmented generation for large language models: A survey,

    Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Sun, and H. Wang, “Retrieval-augmented generation for large language models: A survey,”arXiv preprint arXiv:2312.10997, 2023

    Pith/arXiv arXiv 2023

  8. [8]

    A survey on rag meeting llms: Towards retrieval-augmented large language models,

    W. Fan, Y . Ding, L. Ning, S. Wang, H. Li, D. Yin, T.-S. Chua, and Q. Li, “A survey on rag meeting llms: Towards retrieval-augmented large language models,” inKDD, 2024

    2024

  9. [9]

    Develop secure, reliable medical apps with rag and nvidia nemo guardrails,

    S. Ganju, “Develop secure, reliable medical apps with rag and nvidia nemo guardrails,” https://developer .nvidia.com/blog/develop-secure- reliable-medical-apps-with-rag-and-nvidia-nemo-guardrails/, 2024

    2024

  10. [10]

    Harnessing rag in healthcare: Use-cases, impact, & solutions,

    M. Malec, “Harnessing rag in healthcare: Use-cases, impact, & solutions,” https://hatchworks.com/blog/gen-ai/rag-for-healthcare/, 2025

    2025

  11. [11]

    Ai in finance: The promise and risks of rag,

    Lumenova, “Ai in finance: The promise and risks of rag,” https:// www.lumenova.ai/blog/ai-finance-retrieval-augmented-generation/, 2024

    2024

  12. [12]

    Leveraging retrieval-augmented generation (rag) in banking: A new era of finance transformation,

    Revvence, “Leveraging retrieval-augmented generation (rag) in banking: A new era of finance transformation,” https://revvence .com/blog/rag-in- banking, 2023

    2023

  13. [13]

    watsonx assistant,

    IBM, “watsonx assistant,” https://www .ibm.com/products/watsonx- assistant, 2024

    2024

  14. [14]

    Cpr: Retrieval augmented generation for copyright protection,

    A. Golatkar, A. Achille, L. Zancato, Y .-X. Wang, A. Swaminathan, and S. Soatto, “Cpr: Retrieval augmented generation for copyright protection,” inCVPR, 2024. PREPRINT 13

    2024

  15. [15]

    Rethinking data protection in the (generative) artificial intelligence era,

    Y . Li, S. Shao, Y . He, J. Guo, T. Zhang, Z. Qin, P.-Y . Chen, M. Backes, P. Torr, D. Taoet al., “Rethinking data protection in the (generative) artificial intelligence era,”arXiv preprint arXiv:2507.03034, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Databench: Evaluating dataset auditing in deep learning from an adversarial perspective,

    S. Shao, Y . Li, M. Zheng, Z. Hu, Y . Chen, B. Li, Y . He, J. Guo, T. Zhang, D. Taoet al., “Databench: Evaluating dataset auditing in deep learning from an adversarial perspective,”arXiv preprint arXiv:2507.05622, 2025

    arXiv 2025

  17. [17]

    The good and the bad: Exploring privacy issues in retrieval-augmented generation (RAG),

    S. Zeng, J. Zhang, P. He, Y . Liu, Y . Xing, H. Xu, J. Ren, Y . Chang, S. Wang, D. Yin, and J. Tang, “The good and the bad: Exploring privacy issues in retrieval-augmented generation (RAG),” inFindings of ACL, 2024

    2024

  18. [18]

    Follow my instruction and spill the beans: Scalable data extraction from retrieval- augmented generation systems,

    Z. Qi, H. Zhang, E. P. Xing, S. M. Kakade, and H. Lakkaraju, “Follow my instruction and spill the beans: Scalable data extraction from retrieval- augmented generation systems,” inICLR, 2025

    2025

  19. [19]

    Unleashing worms and extracting data: Escalating the outcome of attacks against rag-based inference in scale and severity using jailbreaking,

    S. Cohen, R. Bitton, and B. Nassi, “Unleashing worms and extracting data: Escalating the outcome of attacks against rag-based inference in scale and severity using jailbreaking,”arXiv preprint arXiv:2409.08045, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Universal and transferable adversarial attacks on aligned language models,

    A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,”arXiv preprint arXiv:2307.15043, 2023

    Pith/arXiv arXiv 2023

  21. [21]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models,

    X. Liu, N. Xu, M. Chen, and C. Xiao, “Autodan: Generating stealthy jailbreak prompts on aligned large language models,” inICLR, 2024

    2024

  22. [22]

    {LLM-Fuzzer}: Scaling assessment of large language model jailbreaks,

    J. Yu, X. Lin, Z. Yu, and X. Xing, “ {LLM-Fuzzer}: Scaling assessment of large language model jailbreaks,” inUSENIX Security, 2024

    2024

  23. [23]

    Jailbreaking leading safety-aligned llms with simple adaptive attacks,

    M. Andriushchenko, F. Croce, and N. Flammarion, “Jailbreaking leading safety-aligned llms with simple adaptive attacks,”arXiv preprint arXiv:2404.02151, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Large language models as optimizers,

    C. Yang, X. Wang, Y . Lu, H. Liu, Q. V . Le, D. Zhou, and X. Chen, “Large language models as optimizers,” inICLR, 2024

    2024

  25. [25]

    When is

    K. S. Beyer, J. Goldstein, R. Ramakrishnan, and U. Shaft, “When is ”nearest neighbor” meaningful?” inProceedings of the 7th International Conference on Database Theory, 1999, p. 217–235

    1999

  26. [26]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,”arXiv preprint arXiv:1908.10084, 2019

    Pith/arXiv arXiv 1908

  27. [27]

    Simcse: Simple contrastive learning of sentence embeddings,

    T. Gao, X. Yao, and D. Chen, “Simcse: Simple contrastive learning of sentence embeddings,”arXiv preprint arXiv:2104.08821, 2021

    Pith/arXiv arXiv 2021

  28. [28]

    Rag-thief: Scalable ex- traction of private data from retrieval-augmented generation applications with agent-based attacks,

    C. Jiang, X. Pan, G. Hong, C. Bao, and M. Yang, “Rag-thief: Scalable ex- traction of private data from retrieval-augmented generation applications with agent-based attacks,”arXiv preprint arXiv:2411.14110, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    A neural probabilistic language model,

    Y . Bengio, R. Ducharme, and P. Vincent, “A neural probabilistic language model,”NeurIPS, 2000

    2000

  30. [30]

    Recurrent neural network based language model

    T. Mikolov, M. Karafiát, L. Burget, J. Cernock `y, and S. Khudanpur, “Recurrent neural network based language model.” inInterspeech, 2010

    2010

  31. [31]

    Improving language understanding by generative pre-training. openai,

    A. Radford, K. Narasimhan, T. Salimans, I. Sutskeveret al., “Improving language understanding by generative pre-training. openai,”Preprint, pp. 1–12, 2018

    2018

  32. [32]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  33. [33]

    Claude 3.7 sonnet and claude code,

    Anthropic, “Claude 3.7 sonnet and claude code,” https: //www.anthropic.com/news/claude-3-7-sonnet, 2025

    2025

  34. [34]

    Pleak: Prompt leaking attacks against large language model applications,

    B. Hui, H. Yuan, N. Gong, P. Burlina, and Y . Cao, “Pleak: Prompt leaking attacks against large language model applications,” inCCS, 2024

    2024

  35. [35]

    Covid-qa: A question answering dataset for covid-19,

    T. Möller, A. Reina, R. Jayakumar, and M. Pietsch, “Covid-qa: A question answering dataset for covid-19,” inWorkshop on NLP for COVID-19 at ACL 2020, 2020

    2020

  36. [36]

    Llmmaps–a visual metaphor for stratified evaluation of large language models,

    P. Puchert, P. Poonam, C. van Onzenoodt, and T. Ropinski, “Llmmaps–a visual metaphor for stratified evaluation of large language models,”arXiv preprint arXiv:2304.00457, 2023

    Pith/arXiv arXiv 2023

  37. [37]

    A comprehensive survey of hallucination mitigation techniques in large language models,

    S. Tonmoy, S. Zaman, V . Jain, A. Rani, V . Rawte, A. Chadha, and A. Das, “A comprehensive survey of hallucination mitigation techniques in large language models,”arXiv preprint arXiv:2401.01313, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Survey of hallucination in natural language generation,

    Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y . Xu, E. Ishii, Y . J. Bang, A. Madotto, and P. Fung, “Survey of hallucination in natural language generation,”ACM Computing Surveys, 2023

    2023

  39. [39]

    Agentpoison: Red- teaming llm agents via poisoning memory or knowledge bases,

    Z. Chen, Z. Xiang, C. Xiao, D. Song, and B. Li, “Agentpoison: Red- teaming llm agents via poisoning memory or knowledge bases,”Advances in Neural Information Processing Systems, 2024

    2024

  40. [40]

    Is my data in your retrieval database? membership inference attacks against retrieval augmented generation,

    M. Anderson, G. Amit, and A. Goldsteen, “Is my data in your retrieval database? membership inference attacks against retrieval augmented generation,”arXiv preprint arXiv:2405.20446, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Seeing is believing: Black-box membership inference attacks against retrieval augmented generation,

    Y . Li, G. Liu, Y . Yang, and C. Wang, “Seeing is believing: Black-box membership inference attacks against retrieval augmented generation,” arXiv preprint arXiv:2406.19234, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    Extracting training data from large language models,

    N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-V oss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingssonet al., “Extracting training data from large language models,” inUSENIX Security, 2021

    2021

  43. [43]

    Truvrf: Towards triple-granularity verification on machine unlearning,

    C. Zhou, Y . Gao, A. Fu, K. Chen, Z. Zhang, M. Xue, Z. Dai, S. Ji, and Y . Zhang, “Truvrf: Towards triple-granularity verification on machine unlearning,”IEEE TIFS, 2025

    2025

  44. [44]

    Hardening llm fine-tuning: From differentially private data selection to trustworthy model quantization,

    Z. Deng, R. Sun, M. Xue, W. Ma, S. Wen, S. Nepal, and Y . Xiang, “Hardening llm fine-tuning: From differentially private data selection to trustworthy model quantization,”IEEE TIFS, 2025

    2025

  45. [45]

    Membership inference attacks against in-context learning,

    R. Wen, Z. Li, M. Backes, and Y . Zhang, “Membership inference attacks against in-context learning,” inCCS, 2024

    2024

  46. [46]

    SILO language models: Isolating legal risk in a nonparametric datastore,

    S. Min, S. Gururangan, E. Wallace, W. Shi, H. Hajishirzi, N. A. Smith, and L. Zettlemoyer, “SILO language models: Isolating legal risk in a nonparametric datastore,” inICLR, 2024

    2024

  47. [47]

    Membership inference attacks against machine learning models,

    R. Shokri, M. Stronati, C. Song, and V . Shmatikov, “Membership inference attacks against machine learning models,” inS&P, 2017

    2017

  48. [48]

    Rethinking membership inference attacks against transfer learning,

    C. Wu, J. Chen, Q. Fang, K. He, Z. Zhao, H. Ren, G. Xu, Y . Liu, and Y . Xiang, “Rethinking membership inference attacks against transfer learning,”IEEE TIFS, vol. 19, pp. 6441–6454, 2024

    2024

  49. [49]

    Privacy implications of retrieval-based language models,

    Y . Huang, S. Gupta, Z. Zhong, K. Li, and D. Chen, “Privacy implications of retrieval-based language models,” inEMNLP, 2023

    2023

  50. [50]

    Generalization through memorization: Nearest neighbor language mod- els,

    U. Khandelwal, O. Levy, D. Jurafsky, L. Zettlemoyer, and M. Lewis, “Generalization through memorization: Nearest neighbor language mod- els,” inICLR, 2020

    2020

  51. [51]

    A normalized levenshtein distance metric,

    L. Yujian and L. Bo, “A normalized levenshtein distance metric,”TPAMI, 2007

    2007

  52. [52]

    UPAM: Unified prompt attack in text-to- image generation models against both textual filters and visual checkers,

    D. Peng, Q. Ke, and J. Liu, “UPAM: Unified prompt attack in text-to- image generation models against both textual filters and visual checkers,” inICML, 2024

    2024

  53. [53]

    Masterkey: Automated jailbreaking of large language model chatbots,

    G. Deng, Y . Liu, Y . Li, K. Wang, Y . Zhang, Z. Li, H. Wang, T. Zhang, and Y . Liu, “Masterkey: Automated jailbreaking of large language model chatbots,”NDSS, 2024

    2024

  54. [54]

    Baseline defenses for adversarial attacks against aligned language models,

    N. Jain, A. Schwarzschild, Y . Wen, G. Somepalli, J. Kirchenbauer, P. yeh Chiang, M. Goldblum, A. Saha, J. Geiping, and T. Goldstein, “Baseline defenses for adversarial attacks against aligned language models,”arXiv preprint arXiv:2309.00614, 2023

    Pith/arXiv arXiv 2023

  55. [55]

    Papillon: Efficient and stealthy fuzz testing-powered jailbreaks for llms,

    X. Gong, M. Li, Y . Zhang, F. Ran, C. Chen, Y . Chen, Q. Wang, and K.-Y . Lam, “Papillon: Efficient and stealthy fuzz testing-powered jailbreaks for llms,”arXiv preprint arXiv:2409.14866, 2025

    Pith/arXiv arXiv 2025

  56. [56]

    Curriculum learning,

    Y . Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” inICML, 2009

    2009

  57. [57]

    Gemini: a family of highly capable multimodal models,

    G. Team, R. Anil, S. Borgeaud, Y . Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauthet al., “Gemini: a family of highly capable multimodal models,”arXiv preprint arXiv:2312.11805, 2023

    Pith/arXiv arXiv 2023

  58. [58]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruanet al., “Deepseek-v3 technical report,”arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  59. [59]

    The enron corpus: A new dataset for email classification research,

    B. Klimt and Y . Yang, “The enron corpus: A new dataset for email classification research,” inECML, 2004

    2004

  60. [60]

    Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge,

    Y . Li, Z. Li, K. Zhang, R. Dan, S. Jiang, and Y . Zhang, “Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge,”Cureus, 2023

    2023

  61. [61]

    Scalable extraction of training data from (production) language models,

    M. Nasr, N. Carlini, J. Hayase, M. Jagielski, A. F. Cooper, D. Ippolito, C. A. Choquette-Choo, E. Wallace, F. Tramèr, and K. Lee, “Scalable extraction of training data from (production) language models,”arXiv preprint arXiv:2311.17035, 2023

    Pith/arXiv arXiv 2023

  62. [62]

    C-pack: Packaged resources to advance general chinese embedding,

    S. Xiao, Z. Liu, P. Zhang, and N. Muennighoff, “C-pack: Packaged resources to advance general chinese embedding,” 2023

    2023

  63. [63]

    Open source strikes bread - new fluffy embeddings model,

    S. Lee, A. Shakir, D. Koenig, and J. Lipp, “Open source strikes bread - new fluffy embeddings model,” 2024. [Online]. Available: https://www.mixedbread.ai/blog/mxbai-embed-large-v1

    2024

  64. [64]

    How johnny can persuade LLMs to jailbreak them: Rethinking persuasion to challenge AI safety by humanizing LLMs,

    Y . Zeng, H. Lin, J. Zhang, D. Yang, R. Jia, and W. Shi, “How johnny can persuade LLMs to jailbreak them: Rethinking persuasion to challenge AI safety by humanizing LLMs,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L.-W. Ku, A. Martins, and V . Srikumar, Eds., 2024, pp. 14 322–14 350

    2024

  65. [65]

    Jailbroken: How does llm safety training fail?

    A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does llm safety training fail?”NeurIPS, 2024

    2024

  66. [66]

    Angle-optimized text embeddings,

    X. Li and J. Li, “Angle-optimized text embeddings,”arXiv preprint arXiv:2309.12871, 2023

    Pith/arXiv arXiv 2023

  67. [67]

    Optimizing llms for accuracy,

    OpenAI, “Optimizing llms for accuracy,” https://platform .openai.com/ docs/guides/optimizing-llm-accuracy

  68. [68]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” inText summarization branches out, 2004

    2004

  69. [69]

    Datasentinel: A game- theoretic detection of prompt injection attacks,

    Y . Liu, Y . Jia, J. Jia, D. Song, and N. Z. Gong, “Datasentinel: A game- theoretic detection of prompt injection attacks,” inIEEE S&P, 2025

    2025

  70. [70]

    Qwen2.5: A party of foundation models,

    Q. Team, “Qwen2.5: A party of foundation models,” September 2024. [Online]. Available: https://qwenlm.github.io/blog/qwen2.5/ PREPRINT 14

    2024

  71. [71]

    Does safety training of llms generalize to semantically related natural prompts?

    S. Addepalli, Y . Varun, A. Suggala, K. Shanmugam, and P. Jain, “Does safety training of llms generalize to semantically related natural prompts?” inICLR, 2025. APPENDIXA OMITTEDPROOFS In this section, we formally analyze when Cluster-Focused Triggering (CFT) is provably more query-efficient than Global Exploration (GE) alone. A. Model and Assumptions We ...

    arXiv 2025

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.