Pith. sign in

REVIEW 3 major objections 5 minor 18 references

RAG with Differential Privacy

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Differentially private token generation makes retrieval-augmented generation formally private end to end, and the approach is usable whenever many documents corroborate the same fact.

desk verdict A promising DP-RAG construction with a real sensitivity gap in its top-p retrieval utility; the top-k version and ICL aggregation are solid, but the whole-pipeline guarantee is not yet proven as stated. read the letter →

arxiv 2412.19291 v2 pith:W6IKTQRY submitted 2024-12-26 cs.LG cs.AIcs.CR

Nicolas Grislain This is my paper · ORCID
classification cs.LGcs.AIcs.CR
keywords differentialprivacyretrieval-augmentedgenerationin-contextlearningexponentialmechanismtokenunitlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Retrieval-augmented generation gives large language models fresh context by injecting retrieved documents into the prompt, but those documents can leak into the answer. The paper claims the leak can be closed by generating each response token differentially privately: no single document can measurably change the published answer. The pipeline selects documents with a privately sampled similarity threshold, prompts the model once per document, and builds each output token from a privately aggregated mixture of the resulting token distributions. On a synthetic medical dataset with roughly 5000 documents, $\varepsilon \approx 5$, and $\delta = 10^{-3}$, answers become accurate once about 100 documents carry the same fact, while facts held by a single individual are censored by design. If this holds, private question-answering over personal records becomes a matter of corroboration: enough independent documents, and the answer is safe to release.

What carries the argument

The load-bearing object is the exponential mechanism, a differentially private selection rule that samples an output with probability growing exponentially in a utility score, applied twice in a chain. On the retrieval side the utility is a function of a candidate similarity threshold — $U_{\mathrm{top-k}}(\tau)=-\bigl|\sum_i \mathbf{1}_{[0,s_i]}(\tau)-k\bigr|$, or the weighted $U_{\mathrm{top-p}}$ variant with weights $w(s)=\exp(\alpha(s-s_{\max})/(s_{\max}-s_{\min}))$ — which has sensitivity 1, and the sampled threshold $\tau_{\mathrm{DP}}$ selects the private context set. On the generation side, each selected document is prompted alone; the per-document next-token distributions are mapped through $l^{\mathrm{norm}}$, an $\alpha$-interpolation between probabilities, log-probabilities, and top-token indicators, then centered and clipped to bound $C$; and a token is sampled proportionally to $\exp(\varepsilon U_{\mathrm{ICL}}(r)/(2C))$, where $U_{\mathrm{ICL}}(r)=\theta\,\ln L_{\mathrm{pub}}(r)+\sum_j l^{\mathrm{clipped}}_{j,i_j}(r)$ blends a public prior. The chain works because the DP threshold makes the selected document set a private object before it reaches the LLM, so the downstream aggregation can be analyzed independently.

What would settle it

Run the released implementation on the released synthetic medical dataset at $\varepsilon \approx 5$, $\delta = 10^{-3}$ and check two predictions: accuracy becomes reasonable only when roughly 100 documents corroborate the same fact, and a fact present in a single document is never emitted. A second experiment targets the load-bearing assumption directly: write one document that describes two individuals, remove just one of them from the dataset, and compare the output distributions — a shift in the response would show the per-individual guarantee failing, the boundary the paper's own privacy-unit assumption concedes.

Watch

Extended reading notes

Core claim

DP-RAG claims that the complete retrieval-and-generation chain can satisfy $(\varepsilon,\delta)$-differential privacy while remaining useful for knowledge-extraction questions. It replaces the standard prompt containing the top-$k$ documents with two private stages: document selection samples a similarity threshold $\tau_{\mathrm{DP}}$ from an exponential mechanism, so the chosen document set is itself a private output; generation then prompts the model once per selected document, transforms and clips the per-document next-token distributions to bound their sensitivity, and samples a token from an exponential mechanism over their aggregated utility blended with a public-prior term. The privacy losses of both stages compose in a standard accountant, yielding a single formal guarantee for the whole system. The empirical finding is that accuracy becomes reasonable once at least 100 documents hold a similar piece of information, so the method is viable precisely when a fact is corroborated across many privacy units — and by design it censors knowledge that is too specific to one individual.

Load-bearing premise

The entire guarantee rests on the paper's explicit assumption that each document belongs to exactly one individual, the privacy unit: if one document contains information about several people, deleting that document removes them all together, and the differential privacy promise protects documents, not the people.

Editorial extensions

If this is right

  • A single formal $(\varepsilon,\delta)$-DP guarantee covers both retrieval and generation, so a RAG service over personal data can be audited without relying on the model's memory behaviour.
  • Utility is tied to corroboration: facts attested by roughly 100 or more privacy units are released accurately, while knowledge unique to one individual is censored — a predictable selectivity property of the mechanism.
  • Off-the-shelf LLMs can serve private knowledge bases without private fine-tuning, since the model is queried once per document and only the aggregated token distributions are released.
  • The accuracy/privacy tradeoff can be steered by the controls the paper identifies: shorter responses, more relevant documents per query, and a stronger public prior $\theta$ for parts of the answer that are already public.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The corroboration threshold of about 100 documents at $\varepsilon \approx 5$ is the quantity practitioners would need as a curve: measuring it as a function of $\varepsilon$, of model size, and of question type would map where the approach is usable, a measurement the paper does not report.
  • The DP threshold-on-similarity trick is a generic way to release a data-dependent cutoff; the same utility-on-cumulative-curve construction could transfer to private clustering, private top-$k$ feature selection, or other settings whose output is a cut point.
  • Under the privacy-unit assumption, the practical burden shifts to preprocessing: documents that mention several people must be split so that each document names exactly one individual, and this segmentation is a decision the framework does not automate or protect.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces DP-RAG, a retrieval-augmented generation pipeline with differential privacy guarantees. The system has two components: (1) a privacy-preserving document retrieval mechanism that selects relevant documents by sampling a threshold with the exponential mechanism, using either a top-k utility or a top-p utility with score weighting; and (2) a differentially private in-context learning component that, for each generated token, aggregates per-document next-token distributions via an exponential mechanism with clipped, centered log-probability utilities. The paper claims formal (ε,δ)-DP for the whole pipeline and presents experiments on a synthetic medical dataset showing that accuracy improves with the number of documents containing the same information, becoming 'reasonable' at around 100 redundant documents.

Significance. The work addresses a timely problem—privacy in RAG—and proposes a practical method that does not require fine-tuning or synthetic data generation. The use of the exponential mechanism for top-k threshold selection is a clean idea, and the per-token aggregation of in-context learning distributions is a meaningful extension of recent DP-ICL work. The code and synthetic dataset are publicly available, which supports reproducibility. If the privacy analysis is completed, this could be a useful baseline for private RAG. However, the formal guarantees are currently incomplete, and the empirical evidence is preliminary, so the significance is contingent on resolving the technical gaps.

major comments (3)
  1. [Privacy Unit Preserving Document Retrieval (top-p utility)] The exponential mechanism for top-p retrieval is sampled as τ ∝ exp(ε U_top-p(τ)/2), but no sensitivity bound is provided for U_top-p. The weights w(s) = exp(α(s - smax)/(smax - smin)) depend on smax and smin, which appear to be the empirical maximum and minimum similarities over the private document set. Adding or removing one document can shift these extrema, rescaling every weight and changing the utility by an amount that may grow with the number of documents N. The exponential mechanism is only ε-DP if the sensitivity is bounded (e.g., by 1, as claimed for the top-k utility). As written, the top-p variant has no proven privacy guarantee, and since it is presented as a component of DP-RAG, this undermines the central claim for that configuration.
  2. [Differentially Private In-Context Learning and privacy accounting] The ICL mechanism is applied at every autoregressive decoding step; each generated token consumes a fresh privacy loss. The paper states that the privacy losses of the retrieval phase and the ICL phase are aggregated in a privacy accountant, but it does not state that the ICL loss multiplies by the number of generated tokens, nor does it explain how the total budget is split between retrieval and the per-token mechanisms. The evaluation reports ε≈5 without clarifying whether this is a per-token budget or the total for the whole response. Without a precise accounting that includes the number of tokens, the claim of formal DP for the full generated response is ambiguous and unverifiable.
  3. [Evaluation] The empirical validation is thin relative to the claim that DP-RAG is a 'viable approach to private RAG.' The experiments use a single synthetic dataset, report no confidence intervals or error bars, do not compare against a non-private RAG baseline, and do not ablate the effects of the key privacy/utility parameters (ε, C, α, θ) on accuracy. Figure 6 shows one curve with no indication of variability. More systematic experiments are needed to support the conclusions and to understand the privacy-utility tradeoff in realistic settings.
minor comments (5)
  1. [Privacy Unit Preserving Document Retrieval] The definition of w(s) has a division by zero when smax = smin (e.g., when all similarity scores are equal). The paper should handle this edge case explicitly.
  2. [Differentially Private In-Context Learning] The clipping factor min(1, C / max_s |l_centered(s)|) divides by zero if max_s |l_centered(s)| = 0. This can occur when all log-probabilities are equal; the paper should specify a default behavior.
  3. [Privacy Unit Preserving Document Retrieval] The exponential mechanism for the continuous threshold τ requires a base measure on [0,1]; the paper should state that τ is sampled uniformly on [0,1] (or another explicit distribution) so that the mechanism is well-defined.
  4. [Differential Privacy and its application to RAG] The paper's abstract and conclusion state that DP-RAG protects individuals, but the formal guarantee protects documents under the assumption that each document belongs to one privacy unit. The text should consistently qualify that the guarantee is for documents, and that violations of the assumption (e.g., multiple individuals in one document) are outside the scope.
  5. [Throughout] There are several typographical errors (e.g., 'garantees', 'approache', 'parametrized') and inconsistent use of math notation (e.g., the subscript in l_{j+1,i_j}^{clipped}). A careful proofreading pass is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DP-RAG's mechanisms are constructed from standard DP building blocks and evaluated on synthetic data; no claim reduces to its own inputs by construction.

full rationale

The derivation chain is not circular. The retrieval phase uses an exponential mechanism over a threshold utility; for the top-k utility the sensitivity-1 bound is stated and the mechanism follows the standard textbook construction. The ICL phase builds a per-token utility by clipping a centered and normalized transformation of the LLM's log-probabilities and then applies the exponential mechanism with a sensitivity divisor; this is a first-principles mechanism rather than a fitted function. The empirical 'viable approach' claim is tested on synthetic documents that are independent of the mechanism derivation, so the result is not a fitted input renamed as a prediction. No self-citations are load-bearing: the cited prior works (Tang et al. 2024, Wu et al. 2023) are external, and the paper explicitly modifies their mechanisms rather than relying on an unverified uniqueness theorem. Flagged for completeness, but not as circularity: in the section 'Privacy Unit Preserving Document Retrieval', the top-p utility U_top-p uses data-dependent smax/smin in w(s) = exp(alpha (s - smax)/(smax - smin)), and the statement 'Once the tau_top-p threshold is sampled with DP, incurring a small privacy loss' does not supply a sensitivity bound for U_top-p. This is an omitted proof or correctness gap, not a reduction of a conclusion to an input.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method relies on standard DP mechanisms and a set of hyperparameters that are not reported. The key domain assumption is the one-document-per-privacy-unit structure.

free parameters (6)
  • k (target document count in U_top-k) = not reported
    Controls the number of documents the threshold aims to select; set by the user, not specified in the paper.
  • p (target weight share in U_top-p) = not reported
    Used in the alternative utility to select a fraction of total document weight; not specified.
  • α (contrast parameter) = not reported
    Controls how much the utility emphasizes high-probability tokens (in l_norm) and score differences (in w); likely tuned per experiment.
  • C (clipping bound) = not reported
    Bounds the ∞-norm of each token utility contribution, thereby setting the sensitivity of the ICL mechanism; not specified.
  • θ (public prior strength) = not reported
    Weights the public log-probability term in the ICL utility; not specified.
  • ε_retrieval and ε_ICL privacy split = total ε ≈ 5 in experiments
    The paper says losses are aggregated with a privacy accountant, but does not report how ε is split between retrieval and ICL.
assumptions (4)
  • standard math Standard differential privacy definition and exponential mechanism guarantee
    Used throughout to claim DP for the threshold and token sampling; these are textbook results (Dwork, Roth).
  • domain assumption Each document is associated with exactly one privacy unit
    Stated in the paper; needed for the neighboring-dataset relation to correspond to adding/removing one document.
  • domain assumption The query and public context are not private
    The query q and public document d_pub are treated as public inputs; if the query itself is sensitive, the privacy guarantee would need to cover it.
  • domain assumption Embedding model E and LLM L are public, fixed functions
    Used to compute similarities and token distributions; their public status means their internal parameters are not part of the private dataset.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RAG with Differential Privacy." pith.science (2026). https://pith.science/paper/W6IKTQRY

@misc{pith2026241219291,
  author       = {Pith},
  title        = {Pith review of: RAG with Differential Privacy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W6IKTQRY}},
  note         = {Machine review of arXiv:2412.19291}
}
read the original abstract

Retrieval-Augmented Generation (RAG) has emerged as the dominant technique to provide \emph{Large Language Models} (LLM) with fresh and relevant context, mitigating the risk of hallucinations and improving the overall quality of responses in environments with large and fast moving knowledge bases. However, the integration of external documents into the generation process raises significant privacy concerns. Indeed, when added to a prompt, it is not possible to guarantee a response will not inadvertently expose confidential data, leading to potential breaches of privacy and ethical dilemmas. This paper explores a practical solution to this problem suitable to general knowledge extraction from personal data. It shows \emph{differentially private token generation} is a viable approach to private RAG.

Figures

Figures reproduced from arXiv: 2412.19291 by the authors.

Figure 1
Figure 1. A broad picture of how RAG works In the private variant of the problem (DP-RAG), we also assume the documents are privacy sensitive, and make the additional assumption that each document relates to only one individual that we call privacy unit (PU)4 . Differential Privacy and its application to RAG A (randomized) algorithm: A provides (ϵ, δ)-Differential Privacy if and only if for all event S and neighboring dataset… view at source ↗
Figure 2
Figure 2. A broad picture of the main problems to overcome when considering DP [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. In DP-RAG, k smaller queries are sent to the LLM, rather than a single query (approximately) k times larger. Privacy Unit Preserving Document Retrieval As mentioned above, DP deals with the concept of neighboring datasets. For this reason, it is convenient to assign each document to one and only one individual, or privacy unity (PU). Adding or removing one PU, comes down to adding or removing 6 [PITH_FULL_IMAGE:fig… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The exponential mechanism for the top-k DP-threshold. For the sake of [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The exponential mechanism for the top-p DP-threshold. For the sake of [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: DP-RAG accuracy as a function of knowledge specificity [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 4 canonical work pages

  1. [1]

    Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang

    Abadi, Martin, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. ``Deep Learning with Differential Privacy.'' In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. CCS'16. ACM. https://doi.org/10.1145/2976749.2978318

    arXiv 2016

  2. [2]

    Bu, Zhiqi, Yu-Xiang Wang, Sheng Zha, and George Karypis. 2023. ``Differentially Private Optimization on Large Model at Small Cost.'' In International Conference on Machine Learning, 3192--3218. PMLR

    work page 2023

  3. [3]

    Carlini, Nicholas, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, et al. 2021. ``Extracting Training Data from Large Language Models.'' In 30th USENIX Security Symposium (USENIX Security 21), 2633--50. USENIX Association. https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting

    work page 2021

  4. [4]

    Dwork, Cynthia, Aaron Roth, et al. 2014. ``The Algorithmic Foundations of Differential Privacy.'' Foundations and Trends in Theoretical Computer Science 9 (3--4): 211--407

    work page 2014

  5. [5]

    Wang, Chenhui Zhang, Zhangheng Li, Bo Li, and Zhangyang Wang

    Hong, Junyuan, Jiachen T. Wang, Chenhui Zhang, Zhangheng Li, Bo Li, and Zhangyang Wang. 2024. ``DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer.'' https://arxiv.org/abs/2312.03724

    arXiv 2024

  6. [6]

    Lebensold, Jonathan, Maziar Sanjabi, Pietro Astolfi, Adriana Romero-Soriano, Kamalika Chaudhuri, Mike Rabbat, and Chuan Guo. 2024. ``DP-RDM: Adapting Diffusion Models to Private Domains Without Fine-Tuning.'' https://arxiv.org/abs/2403.14421

    work page Pith review arXiv 2024

  7. [7]

    Lewis, Patrick, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, et al. 2021. ``Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.'' https://arxiv.org/abs/2005.11401

    arXiv 2021

  8. [8]

    Lin, Zinan, Sivakanth Gopi, Janardhan Kulkarni, Harsha Nori, and Sergey Yekhanin. 2024. ``Differentially Private Synthetic Data via Foundation Model APIs 1: Images.'' https://arxiv.org/abs/2305.15560

    arXiv 2024

Show all 18 references
  1. [9]

    Brendan McMahan, Sergei Vassilvitskii, Steve Chien, and Abhradeep Guha Thakurta

    Ponomareva, Natalia, Hussein Hazimeh, Alex Kurakin, Zheng Xu, Carson Denison, H. Brendan McMahan, Sergei Vassilvitskii, Steve Chien, and Abhradeep Guha Thakurta. 2023. ``How to DP-Fy ML: A Practical Guide to Machine Learning with Differential Privacy.'' Journal of Artificial I...

    2023 doi

  2. [10]

    Qi, Zhenting, Hanlin Zhang, Eric Xing, Sham Kakade, and Himabindu Lakkaraju. 2024. ``Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems.'' https://arxiv.org/abs/2402.17840

    2024 arXiv

  3. [11]

    Shokri, Reza, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. ``Membership Inference Attacks Against Machine Learning Models.'' In 2017 IEEE Symposium on Security and Privacy (SP), 3--18. https://doi.org/10.1109/SP.2017.41

    2017 doi

  4. [12]

    Sordoni, Alessandro, Xingdi Yuan, Marc-Alexandre Côté, Matheus Pereira, Adam Trischler, Ziang Xiao, Arian Hosseini, Friederike Niedtner, and Nicolas Le Roux. 2023. ``Joint Prompt Optimization of Stacked LLMs Using Variational Inference.'' https://arxiv.org/abs/2306.12509

    2023 arXiv

  5. [13]

    Inan, Andre Manoel, Fatemehsadat Mireshghallah, Zinan Lin, Sivakanth Gopi, Janardhan Kulkarni, and Robert Sim

    Tang, Xinyu, Richard Shin, Huseyin A. Inan, Andre Manoel, Fatemehsadat Mireshghallah, Zinan Lin, Sivakanth Gopi, Janardhan Kulkarni, and Robert Sim. 2024. ``Privacy-Preserving in-Context Learning with Differentially Private Few-Shot Generation.'' https://arxiv.org/abs/2309.11765

    2024 arXiv

  6. [14]

    Wang, and Prateek Mittal

    Wu, Tong, Ashwinee Panda, Jiachen T. Wang, and Prateek Mittal. 2023. ``Privacy-Preserving in-Context Learning for Large Language Models.'' https://arxiv.org/abs/2305.01639

    2023 arXiv

  7. [15]

    Xie, Chulin, Zinan Lin, Arturs Backurs, Sivakanth Gopi, Da Yu, Huseyin A Inan, Harsha Nori, et al. 2024. ``Differentially Private Synthetic Data via Foundation Model APIs 2: Text.'' https://arxiv.org/abs/2403.01749

    2024 arXiv

  8. [16]

    Inan, Xuechen Li, Girish Kumar, Julia McAnallen, Hoda Shajari, Huan Sun, David Levitan, and Robert Sim

    Yue, Xiang, Huseyin A. Inan, Xuechen Li, Girish Kumar, Julia McAnallen, Hoda Shajari, Huan Sun, David Levitan, and Robert Sim. 2023. ``Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe.'' https://arxiv.org/abs/2210.14348

    2023 arXiv

  9. [17]

    Zeng, Shenglai, Jiankun Zhang, Pengfei He, Jie Ren, Tianqi Zheng, Hanqing Lu, Han Xu, Hui Liu, Yue Xing, and Jiliang Tang. 2024. ``Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Data.'' https://arxiv.org/abs/2406.14773

    2024 arXiv

  10. [18]

    Zhou, Yongchao, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. ``Large Language Models Are Human-Level Prompt Engineers.'' https://arxiv.org/abs/2211.01910. CSLReferences document

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.