{"id":"ac4d621d-fbe0-42ac-a20d-06ea1e59b39f","arxiv_id":"2608.09001","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An active-path audit that maps each privacy defense to the pipeline hook it modifies shows that retrieval-only DP-style stubs do not reduce named-entity leakage in generated text, while an end-to-end substitution defense does.","lead":"This paper proposes auditing RAG privacy defenses by inspecting which pipeline component they actually modify, rather than only looking at black-box leakage scores. In the authors' testbed, three differential-privacy-style defenses modified only retrieval scores, while one entity-substitution defense passed an independent email-canary test.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The DP-vs-baseline null on generated text depends on NEL_strict, whose recall is unaudited; a scorer floor effect could conceal real leakage differences, so the channel-split confirmation is weaker than claimed.","rationale":"I agree with the reader's weakest-assumption identification. The empirical weight of the silent-stub case study is the observed dissociation between MI AUC movement and NEL_strict flatness. That dissociation is only meaningful if NEL_strict can actually detect changes in generated-text entity leakage; with recall unaudited, the null could be a measurement artifact. This is a genuine load-bearing concern for the strength of the case study's confirmation, though not for the core source-level finding, which is supported by direct code quotes and Table I. The canary-validated LPRAG email result is independent and strong, and the methodology remains sound. Therefore the verdict should remain CONDITIONAL, pending the promised artifact release and the stated follow-on NEL_strict recall audit and person-name canary. I do not see a basis to reject or accept unconditionally.","tokens_in":9673,"tokens_out":16060,"duration_ms":151567,"concrete_test":"Run a stratified recall audit of NEL_strict on a random sample of generated responses from the DP and No-Defense cells (e.g., 50 responses per condition), with two independent human annotators labeling all named entities, and recompute the DP-vs-No-Defense comparison using the recall-complete gold labels. In parallel, inject out-of-vocabulary person-name canaries (e.g., a unique string outside any defense vocabulary) into the synthetic KB and run the same PS attack under DP-R, CA-DP, PRIVATE-RAG, and No-Defense; if exact-match recovery rates are statistically indistinguishable, the NEL_strict null is corroborated; if DP reduces recovery, the reported 'tracks No-Defense' claim is a scorer artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central case-study claim that the three DP-style implementations 'track No-Defense' on generated-text leakage (Section VI-B, Table II) depends entirely on NEL_strict, a regex+lexicon scorer whose recall is explicitly unaudited (Section V: only a 20/20 precision spot-check on a 10-per-class sample; 'not a recall audit'). A precision-only lower bound cannot rule out a floor effect: if NEL_strict misses most person-name or other entities in generated text, genuine leakage differences between DP and No-Defense could be compressed or hidden. The paper's own limitation statement (Section VII) concedes NEL_strict is 'precision-only' and says a larger stratified audit and a person-name OOV canary are follow-on work. Since the empirical pattern 'MI down, NEL_strict flat' is the public face of the silent-stub demonstration, the absence of recall validation leaves open that the flatness is a scorer artifact rather than a true null on the generated-text channel. The source-level stub finding itself is direct and well-quoted, so the core conclusion about hook status is not in doubt; what is unverified is the empirical claim that the stub yields no measurable generated-text effect.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces an \"active-path audit\" methodology for RAG privacy defenses, consisting of three steps: (1) a source-level hook inventory, (2) a metric-to-channel map, and (3) exact-match canary validation of generated-text effects. The methodology is applied to a fixed open-source stack (Phi-3-mini, FAISS, MiniLM embeddings) with six defense implementations and six attack families arranged in a 432-cell grid. The principal empirical findings are that three DP-style implementations in the benchmark modify only retrieval scores while their generation hooks are TODO-flagged pass-throughs; these implementations reduce membership-inference AUC on Synthetic-Corp while tracking No-Defense on the NEL_strict generated-text metric; PAD removes email patterns but not person-name leakage; and LPRAG reduces email leakage, independently validated by an out-of-vocabulary exact-match canary (53/150 vs 0/150). The paper carefully scopes all findings to its own reimplementations and stack, disclaiming any generalization to defense families or upstream releases.","tokens_in":9909,"tokens_out":4685,"duration_ms":42648,"significance":"The active-path audit is a timely and genuinely useful methodological contribution. If the results hold, they demonstrate a concrete silent-stub failure mode: a black-box benchmark can rank implementations as privacy-reducing based on a metric whose channel is not affected by the implementation's active hooks. Strengths include the explicit self-limitation throughout, direct source-level evidence (including quoted TODO stubs), a canary design that is independent of the heuristic scorer, a consistent per-seed pattern in the canary experiment, and the stated plan to release artifacts. The main weakness is that the empirical channel-split claim rests on NEL_strict, a precision-only heuristic with unaudited recall, so the null result for generated-text leakage is not as strongly supported as the source-level hook finding.","major_comments":[{"comment":"The claim that the three DP-style implementations \"track No-Defense\" on NEL_strict (within +/-7%) is used as the empirical face of the silent-stub demonstration. However, NEL_strict is validated only by a 20/20 precision spot-check on a 10-per-class positive sample, with no recall audit (Section V: \"not a recall audit\"). If the scorer systematically misses person-name or other named-entity leakage in generated text, the flatness of the DP-vs-baseline comparison could be a floor effect rather than a true null on the generated-text channel. I ask the authors to either (a) add a recall-oriented audit or a person-name OOV canary for at least one DP implementation, or (b) explicitly downgrade the claim in the abstract and Section VI-B from \"track No-Defense\" to \"no effect measurable by a precision-oriented lower-bound heuristic with unknown recall.\"","section":"Section V, Table II, Section VI-B"},{"comment":"Every cell in the 432-cell grid is a single run, and every bootstrap interval is within-instantiation only. The paper is admirably explicit about this, but the headline comparative statement that NEL_strict \"does not move appreciably\" rests on overlapping within-instantiation intervals. Without independent seeds or corpus realizations for at least the DP-style versus No-Defense comparison, the interval overlap is only descriptive and cannot support any statement about whether the small observed differences (e.g., -1.3% and -6.1%) are meaningful. I would like to see either a few independent seeds for the central channel-split cells or an explicit statement that even the sign of the DP-vs-baseline difference on NEL_strict is not estimated with any between-instantiation uncertainty.","section":"Section IV-A, Section V, Section VI-B"},{"comment":"The PAD result (EMAIL_REAL driven to 0.00 while PERSON_NAME rises from 4.00 to 6.15) is reported descriptively and flagged as not mechanistic. This is appropriately cautious, but the same precision-only scorer issue applies to the conclusion that \"PAD is not a general person-entity defense.\" The source-level fact that PAD is a regex masker supports the qualitative conclusion, but the empirical magnitude and even the sign of the person-name delta are not supported by the current validation of NEL_strict. The paper should either soften the language to indicate that the person-name comparison is an unvalidated descriptive observation or provide additional validation for that sub-class.","section":"Section VI-C, Table II"}],"minor_comments":[{"comment":"There is a typo in the abstract: \"NO-DEFENSEon\" should be \"NO-DEFENSE on\", and the formatting of NEL_strict is inconsistent (NELstrict vs NEL_strict).","section":"Abstract"},{"comment":"The function name \"apply_defense_generation\" should be set in monospace or code font for clarity.","section":"Section III"},{"comment":"The table footnotes use abbreviations \"Ex\", \"SE\", and \"canary-val.\" that are not defined in the caption or the immediately surrounding text; please define them explicitly.","section":"Table II"},{"comment":"The paper states that the scorer buckets extracted items into eleven types but names only a few; please provide the full list of types in an appendix or a supplementary table for reproducibility.","section":"Section V"},{"comment":"The canary protocol says that the first 10 queries \"match the shared extraction pool\" but does not specify whether these are the exact same query strings or a semantically equivalent subset; please clarify.","section":"Section VI-A"},{"comment":"Reference [4] includes an editorial note about v1 title and later revisions; such provenance information is better placed in a footnote or removed from the reference entry.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is well within scope for a security or privacy venue, and the active-path audit methodology is a useful contribution. The main risk is overinterpretation of the NEL_strict null given the unaudited recall; the authors' own caveats partially mitigate this, but the abstract and headline findings should be aligned with the supporting evidence. The paper also cites several closely related arXiv preprints by the same authors; the editor may wish to verify that those citations are substantive and not padding."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time. The active-path audit is a small but real methodological contribution: inventory where a defense actually hooks in the code, map each metric to the channel it observes, and use exact-match canaries to validate generated-text effects. The paper catches a failure mode that black-box benchmarking would miss: three DP-style implementations in their benchmark have TODO-flagged pass-through generation hooks, so they only perturb retrieval scores. That explains why membership-inference AUC drops while NEL_strict, scored on generated text, does not move. The source-level evidence is direct—they quote the TODO code—and the email canary for LPRAG is clean: 53/150 recovery under no-defense, 0/150 under LPRAG, exact-match, OOV outside the substitution dictionary. The authors are also honest about scope: findings are about their reimplementations on one fixed stack, not about DP-RAG in general or upstream releases.\n\nThe soft spot is exactly where the stress-test points. The empirical claim that the stubs have no measurable generated-text effect depends entirely on NEL_strict, a regex+lexicon scorer whose recall is explicitly unaudited. They spot-checked precision 20/20 on a small sample and call it precision-only. If that scorer misses most person-name leakage, a real difference between DP and no-defense could be compressed to \"no move.\" So the channel-split confirmation is weaker than the source-code finding. That is a real limitation, but it does not break the methodology: the hook inventory and metric-to-channel map still stand, and the canary-validated email channel is independent of the heuristic scorer.\n\nOther issues are minor but worth naming: each grid cell is a single run, so all bootstrap intervals are within-instantiation only; the person-name reduction for LPRAG is flagged as scorer-entangled and not used as evidence; code and data are promised but not yet public. The self-citation cluster at the end is noticeable but the cited audits are genuinely on-topic.\n\nWho is this for: anyone measuring or benchmarking RAG privacy defenses, and to a lesser extent anyone designing audit methods for ML systems. It deserves a serious referee. I would accept it for review and ask for the artifacts plus a stratified recall audit of NEL_strict, or a narrowed claim.","headline":"A modest but real methodology paper: the active-path audit catches a silent-stub failure mode that black-box RAG privacy benchmarks miss, though the generated-text null rests on an under-validated scorer.","tokens_in":10444,"tokens_out":2305,"would_cite":true,"duration_ms":22321,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Black-box privacy scores for retrieval-augmented generation defenses are uninterpretable unless the defense's active pipeline hook is known; source-level auditing reveals silent-stub failure modes.","keywords":["retrieval-augmented generation","privacy audit","source-level auditing","differential privacy","membership inference","named-entity leakage","canary validation","silent-stub failure mode"],"falsifier":"Inspect the released source of the three DP-style modules: if any generation hook performs an actual transformation instead of returning the response unchanged, the paper's explanation of the MI-versus-NEL_strict split is false.","tokens_in":9430,"feed_emoji":"🔍","tokens_out":8993,"duration_ms":74494,"temperature":0.7,"pith_summary":"The paper argues that a black-box privacy number for a retrieval-augmented generation (RAG) system is uninterpretable unless you know which part of the pipeline the audited defense actually modifies. It proposes an active-path audit: inventory the source-level hooks a defense touches, map each privacy metric to the channel it observes, and validate generated-text effects with exact-match canaries that bypass the heuristic scorer. Applied to reimplementations on one fixed stack, the audit finds that three differential-privacy-style defenses modify only retrieval scores and their generation hooks are TODO-flagged pass-throughs, which explains why they lower membership-inference AUC but leave generated-text named-entity leakage at the no-defense level. The point is that benchmark wrappers can silently evaluate an inactive path, so defense rankings need this check.","feed_headline":"Black-box RAG privacy scores can conceal inactive defenses","feed_subtitle":"Without a source-level hook check, a defense can look privacy-preserving while leaving generated text untouched.","key_machinery":"The load-bearing mechanism is the active-path audit protocol. Step one inventories each defense module's hooks and records any TODO-flagged or pass-through generation hook as inactive. Step two maps each metric to the channel it observes: NEL_strict reads generated text, while the black-box membership-inference AUC is treated as retrieval-sensitive channel-location evidence rather than a calibrated privacy estimate. Step three validates generated-text effects with exact-match out-of-vocabulary canaries, which are scored by string matching and therefore cannot be inflated by the heuristic classifier's placeholder vocabulary. This machinery is what lets the paper reconcile the MI-versus-NEL_strict split as a property of the audited stubs rather than of differential privacy on RAG retrieval in general.","core_discovery":"The central discovery is a silent-stub failure mode in RAG privacy benchmarking. In the paper's benchmark reimplementations, the three DP-style modules (DP-R, CA-DP, PRIVATE-RAG) ship with apply_defense_generation hooks that are TODO-flagged pass-throughs, so they act only on retrieval scores. This makes the observed pattern interpretable: membership-inference AUC on Synthetic-Corp drops from 70.5 to 41.5-59.3 because perturbed retrieval changes which documents surface, while the named-entity leakage metric NEL_strict, scored on generated text, stays within ±7% of the no-defense baseline because the stubs do not transform output. By contrast, the end-to-end entity-substitution defense LPRAG is validated on the email channel by an exact-match out-of-vocabulary canary: 53/150 emails recovered under no defense and 0/150 under LPRAG. These findings are explicitly about these reimplementations on one stack, not about the cited defense families.","pith_inferences":["Editorial extension: applying the same hook audit to released defense codebases, rather than reimplementations, would likely surface similar inactive-path drift; the paper explicitly stops short of claiming upstream stubs.","Editorial extension: a natural next experiment is an out-of-vocabulary person-name canary analogous to the email canary; if it also showed zero recovery under LPRAG, the person-name channel would move from scorer-entangled to independently validated.","Editorial extension: benchmark suites could adopt a machine-readable hook manifest declaring each defense's active pipeline stage, making channel-overlap checks a standard part of privacy evaluations.","Editorial extension: because the black-box MI score can move with output length or masking, the audit implies that any defense that merely shortens output could be misread as membership-private unless the active-path check rules out surface-corruption nuisance."],"forward_implications":["Black-box RAG privacy evaluations should state which pipeline hook each defense activates, or at least flag when no active hook overlaps the metric's channel.","Lower membership-inference AUC from retrieval-only perturbation is channel-location evidence, not a calibrated privacy guarantee, because it can also respond to output-surface nuisance.","Generated-text leakage claims need exact-match canary validation: heuristic scorers can systematically exclude a defense's own placeholder vocabulary and inflate apparent protection.","Aggregate leakage numbers can hide channel-specific behavior: a regex masker can eliminate email-format strings while leaving person-name leakage unchanged or higher.","The methodology generalizes across RAG stacks even though the numeric findings are confined to the audited single-stack benchmark."],"supporting_citations":[{"why":"Supplies the canary methodology used to validate generated-text effects by exact-match recovery.","marker":"[10]"},{"why":"Supplies the black-box membership-inference framing used for the MI attack.","marker":"[5]"},{"why":"Supplies the locally-private entity-perturbation defense that the audited LPRAG implementation simplifies.","marker":"[6]"},{"why":"Supplies the differentially-private RAG mechanism that the PRIVATE-RAG implementation is inspired by.","marker":"[11]"},{"why":"Supplies the agent-based extraction attack reproduced as the RAG-Thief baseline.","marker":"[4]"},{"why":"Supplies the language model used as the fixed generator in the stack.","marker":"[1]"},{"why":"Supplies the similarity-search index used for retrieval.","marker":"[2]"},{"why":"Supplies the sentence-embedding model used to index the corpora.","marker":"[3]"},{"why":"Supplies the PubMedQA corpus used as the realistic QA workload.","marker":"[17]"},{"why":"Contextualizes the auditing perspective that keeps MI from being over-read as a calibrated membership estimate.","marker":"[9]"}],"fun_headline_variants":["RAG privacy scores hide stubbed defense hooks","Silent stub defenses fool RAG privacy audits","DP-style RAG defenses leave generation untouched","Canary test exposes inactive RAG defenses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that NEL_strict, a heuristic named-entity scorer validated only by an informal 20/20 spot-check with no recall audit, correctly measures leakage on generated text; if it systematically misses or mislabels leaked entities, the paper's null result for the DP-style defenses on that channel collapses.","fun_headline_variants_meta":{"raw":{"variants":["RAG privacy scores hide stubbed defense hooks","Silent stub defenses fool RAG privacy audits","DP-style RAG defenses leave generation untouched","Canary test exposes inactive RAG defenses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00049,"raw_usage":{"total_tokens":2405,"prompt_tokens":936,"completion_tokens":1469,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":1412}},"tokens_in":552,"tokens_out":1469,"duration_ms":12136,"temperature":1.0,"reasoning_tokens":1412,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:18:06.886156+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the released source of the three DP-style modules: if any generation hook performs an actual transformation instead of returning the response unchanged, the paper's explanation of the MI-versus-NEL_strict split is false.","supporting_citations":[{"cited_title":"The secret sharer: Evaluating and testing unintended memorization in neural networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the canary methodology used to validate generated-text effects by exact-match recovery."},{"cited_title":"Membership inference attacks against machine learning models,","cited_arxiv_id":null,"evidence_quote":"Supplies the black-box membership-inference framing used for the MI attack."},{"cited_title":"Mitigating privacy risks in retrieval-augmented generation via locally private entity perturbation,","cited_arxiv_id":null,"evidence_quote":"Supplies the locally-private entity-perturbation defense that the audited LPRAG implementation simplifies."},{"cited_title":"Sentence-BERT: Sentence embeddings using Siamese BERT-networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the sentence-embedding model used to index the corpora."},{"cited_title":"PubMedQA: A dataset for biomedical research question answering,","cited_arxiv_id":null,"evidence_quote":"Supplies the PubMedQA corpus used as the realistic QA workload."},{"cited_title":"Membership inference attacks from first principles,","cited_arxiv_id":null,"evidence_quote":"Contextualizes the auditing perspective that keeps MI from being over-read as a calibrated membership estimate."}],"review_version":1}