{"id":"0c5ce0c3-a0e7-4f99-9255-23d804e67bd2","arxiv_id":"2504.21036","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Differentially private fine-tuning sharply reduces both data-extraction and membership-inference risk on GPT-2 models, while full fine-tuning and LoRA preserve utility best.","lead":"This paper tests whether differentially private training actually protects language models from privacy attacks under different fine-tuning methods. It finds that privacy protection improves strongly, but some methods such as prefix-tuning lose too much accuracy to be practically useful.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Extraction-risk claim hinges on a model-generated candidate space that conflates DP-induced utility loss with privacy protection; a fixed-adversary test is needed.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the extraction-risk conclusion depends on whether the 1,000 model-generated candidates represent the space an actual adversary would guess from. My stress-test sharpens this concern: the candidate generation procedure is not merely unrepresentative; it is definitionally tied to the model's own post-DP sampling distribution. Since DP changes that distribution both by reducing memorization and by degrading utility, the exposure metric cannot separate 'the model is harder to extract from' from 'the model is too broken to output the canary.' This makes the extraction-risk portion of the headline claim conditional on an external, fixed adversary model. The MIA results using SPV-MIA are a genuine attack and provide some independent support for DP's effectiveness, which is why I do not recommend rejection. The paper also explicitly acknowledges the non-attack nature of its extraction measure in Section 3.1 and lists scope limitations in Section 5. Given that the reader already assigned CONDITIONAL based on this weakness, my stress-test does not change the verdict; it reinforces the condition. A concrete fixed-candidate-space rerun would settle whether the extraction-risk finding is real or an artifact.","tokens_in":11265,"tokens_out":4289,"duration_ms":46778,"concrete_test":"Recompute exposure on GPT-2 full fine-tuning and LoRA at epsilon in {1, 10, 50, infinity} using a fixed candidate space of the true canary plus 999 randomly generated 8-character alphanumeric secrets sharing the same prompt prefix, instead of model-generated candidates; also measure exact-match extraction success over 10,000 sampled continuations. If the DP-induced exposure drop largely disappears, or if exact-match success remains high under DP, the extraction-risk claim is an artifact of the model-generated candidate space and Section 4.5 should be re-scoped to MIA only. If the drop persists under the fixed candidate space and exact-match success, the claim survives this test.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that DP substantially reduces extraction risk (Finding 3 and the abstract) rests on the exposure metric defined in Section 3.1, where the 1,000 candidates are sampled from the fine-tuned model itself. The authors concede that this 'is not a true attack but rather an evaluation technique.' Because the candidate set is drawn from the same distribution that DP is supposed to alter, the rank of the canary in Definition 1 measures the model's propensity to emit the canary under its own sampling procedure, not an adversary's ability to guess it from a plausible candidate space. When DP degrades utility, the model may fail to produce the canary within 1,000 samples simply because it is less fluent, so exposure falls even if a real adversary with a fixed candidate list would still identify the canary. This conflation directly affects the extraction-risk component of the privacy-utility trade-off conclusion in Section 4.5 and Finding 3. The MIA component is a recognized attack and is not undermined by this issue, but the headline claim bundles both attacks, so the extraction-risk portion is not yet established as a measure of real attack resistance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper empirically studies whether DP-SGD-style training protects GPT-2 and GPT-2 XL during fine-tuning, comparing full fine-tuning with LoRA, prefix-tuning, and P-tuning. Privacy risk is measured with a canary-based generation/exposure probe and with SPV-MIA membership inference, while utility is measured by perplexity. The central claims are that DP degrades utility unevenly across methods (prefix-tuning most), that without DP full fine-tuning and LoRA memorize much more than prefix-tuning and P-tuning, that DP substantially reduces exposure and MIA AUC even at relatively high privacy budgets, and that full fine-tuning offers the best privacy-utility trade-off under DP, with LoRA and P-tuning as practical alternatives and prefix-tuning unsuitable.","tokens_in":11507,"tokens_out":8707,"duration_ms":95220,"significance":"If the empirical claims hold, this is a useful comparative map for practitioners choosing fine-tuning strategies under DP, and it is one of the few studies to vary both fine-tuning method and privacy budget while using two attack families. The paper has clear strengths: it uses established attack implementations (SPV-MIA and Carlini et al.'s exposure), covers two model sizes and four fine-tuning methods, includes a GPT-J control for architecture effects, and reports both utility and attack metrics. The MIA component is credible and internally consistent. However, the extraction-risk component is not yet a valid measure of real attack resistance, and the lack of error bars weakens the quantitative rankings. The paper's practical guidance therefore needs additional support before the headline privacy-utility conclusions can be accepted.","major_comments":[{"comment":"The exposure score used for the extraction-risk claims is computed against a candidate set generated by sampling from the fine-tuned model itself (1,000 unique outputs). As the authors acknowledge in Section 3.1, this 'is not a true attack but rather an evaluation technique.' With this construction, the rank in Definition 1 measures how readily the model happens to emit the canary under its own stochastic decoding distribution, not whether an adversary who guesses from a plausible fixed candidate list would identify the canary. Under DP, the model can produce fewer fluent or diverse candidates, so the canary may drop out of the 1,000-sample set even though the canary has the lowest loss among the strings an actual attacker would try. This conflation directly supports Finding 3 and the Section 4.5 privacy-utility trade-off, so the extraction component of those claims is not yet established as real attack resistance. Please add a control with an adversary-independent candidate space (e.g., a fixed list of plausible secret codes or samples from a held-out reference model), or explicitly rescope the claims to 'generation-based memorization' rather than 'data extraction attack resistance.'","section":"3.1 (Definition 1 and Candidates Generation)"},{"comment":"All figures and the narrative report a single run per configuration; there is no mention of random seeds, repeated trials, or error bars. Several of the paper's practical conclusions are comparative rankings that could easily be reversed by run-to-run variance, for example the ordering of LoRA versus P-tuning in Fig. 5 and the relative drops at epsilon=30 versus epsilon=50 in Fig. 2. Please report multiple seeds with standard deviations or confidence intervals for at least the headline comparisons, or soften the ordering claims accordingly.","section":"4.4 (Figs. 2-5)"},{"comment":"The sentence claiming that with epsilon=1 differential privacy 'guarantees that the success rate of MIA remains below 73%' is not supported by any derivation in the paper and appears to overstate what a DP guarantee provides. DP bounds the advantage of any adversary, but it does not by itself imply a 73% AUC ceiling for SPV-MIA, and the observed values in Fig. 3 are not a theoretical guarantee. This statement should be removed or replaced with a proper derivation, since it is currently used to argue that current MIAs are insufficient against DP-trained models.","section":"4.4 (MIA discussion)"}],"minor_comments":[{"comment":"The figure caption labels the weak attack as 'Short Prefix' and the strong attack as 'Long Prefix,' but Section 3.1 defines the strong attack as asking for only the final character of the secret code; the caption terminology should be aligned with the threat-model description.","section":"4.4 / Fig. 2 caption"},{"comment":"The Future Work section states that the experiments focused solely on the GPT-2 family, but Section 4.4 reports a GPT-J experiment to rule out architecture effects; this contradiction should be fixed.","section":"5 (Future Work)"},{"comment":"The text says the PEFT methods are implemented using 'the peft module available in the Transformers library,' but peft is a separate Hugging Face library; please correct the attribution.","section":"4.2 (Experimental Setup)"},{"comment":"References [5] and [6] appear to be the same paper and should be merged or clearly distinguished.","section":"References"},{"comment":"The notation in Definition 1 says 'candidate space C represents the number of candidates generated' while the formula uses |C|; clarify that C is the set of candidates and |C| is its cardinality.","section":"3.1 (Definition 1)"},{"comment":"The caption of Fig. 1 does not identify the subplots or the two experimental settings; please add panel labels or a description of which panels correspond to Wikitext-2 and AG News.","section":"4.3 / Fig. 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the comparative empirical question is timely. The main risk is not novelty or internal inconsistency but measurement validity: the extraction-risk claim rests on a self-generated candidate space, and the absence of repeated-seed error bars makes the fine-grained rankings fragile. I would ask the authors to add a fixed-candidate-list control or rescope the extraction claims, and to address the unsupported DP guarantee statement, before publication. The comparison with the directly related Marchyok et al. result cited as [22] also deserves a more detailed discussion, since Remark 2 claims a contradiction with prior findings."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a useful empirical grid, but the extraction half of the paper has a load-bearing caveat. The headline claim that DP reduces extraction risk rests on an exposure metric whose candidate space is sampled from the model itself, not from a fixed adversarial list. The authors admit in Section 3.1 that their method 'is not a true attack but rather an evaluation technique.' When DP degrades fluency, the canary is less likely to appear among 1,000 samples, so exposure falls even if a real attacker with a fixed candidate set would still identify the secret. That conflates utility loss with privacy protection, and it directly affects Finding 3 and the abstract. The MIA results, using SPV-MIA, are a recognized attack and are not subject to this flaw.\n\nWhat is genuinely new: a systematic comparison of four fine-tuning methods (FFT, LoRA, prefix-tuning, P-tuning), two model sizes, two attack families, and multiple epsilon values. The reported contradiction with Marchyok et al. on LoRA exposure is interesting, though the paper does not investigate why the results diverge. The experimental design is otherwise careful—fastDP for DP training, integration of SPV-MIA with PEFT, and honest limitations in the Future Work section.\n\nSoft spots in order of severity. First, the candidate-space issue above; a fixed-adversary test (e.g., a scripted list of plausible secrets) would make the extraction claim transferable. Second, no repeated seeds or error bars; single runs are risky for claims like 'all methods converge to 58% AUC.' Third, no code or data release, which hurts reproducibility. Fourth, the weak/strong attack description in Section 3.1 is confusing—the text and figure captions seem to disagree on whether the strong attack uses a longer or shorter prefix. These are fixable.\n\nThe paper's practical guidance—full fine-tuning offers the best privacy-utility trade-off under DP, prefix-tuning is unsuitable, LoRA and P-tuning are viable alternates—is plausible and supported by both attacks. The MIA findings are the stronger evidence.\n\nWho is this for? Anyone choosing a fine-tuning strategy under DP constraints. It deserves serious peer review, but I would ask for a fixed-adversary extraction test, error bars, and code release before acceptance. As is, the extraction-risk language should be softened.\n\nRecommendation: send to peer review, conditional on revisions addressing the candidate-space issue.","headline":"Useful empirical grid of DP fine-tuning methods, but the extraction-risk claim rests on a self-generated candidate space and needs a fixed-adversary check before it lands.","tokens_in":12001,"tokens_out":4598,"would_cite":true,"duration_ms":43020,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Differentially private fine-tuning substantially reduces both data-extraction and membership-inference risk in language models, even at high privacy budgets, and full fine-tuning gives the best privacy-utility trade-off under DP.","keywords":["differential privacy","LLM fine-tuning","membership inference attack","data extraction attack","parameter-efficient fine-tuning","privacy-utility trade-off","exposure metric"],"falsifier":"Fine-tune a GPT-2 model with full fine-tuning under DP at ε=50 using the paper's canary setup, then compute exposure not against 1,000 model-generated candidates but against a candidate set built from the true secret-code alphabet or an independent external guess list. If the true canary ranks near the top under that alternative candidate space, or if a stronger membership inference attack lifts AUC well above 58%, the claim that even high privacy budgets substantially reduce empirical privacy risk would fail in that setting.","tokens_in":11081,"feed_emoji":"🔒","tokens_out":7913,"duration_ms":72926,"temperature":0.7,"pith_summary":"The paper asks whether differentially private (DP) fine-tuning protects large language models against concrete privacy attacks, not just in theory. It fine-tunes GPT-2 and GPT-2 XL with four methods—full fine-tuning, LoRA, prefix-tuning, and P-tuning—under several privacy budgets, and measures risk with a canary-based data extraction attack and a membership inference attack. The central empirical claim is that DP reduces both risks substantially even at a relatively high budget, while the size of the utility loss depends strongly on the fine-tuning method. Under DP, full fine-tuning achieves the best privacy-utility trade-off, LoRA is the recommended efficient choice, P-tuning favors privacy, and prefix-tuning suffers severe utility degradation and should be avoided. The results matter because they give deployment guidance for privacy-sensitive applications: DP can be practically useful, but only with the right fine-tuning method.","feed_headline":"Differential privacy cuts LLM attack risk, even at high budgets","feed_subtitle":"Full fine-tuning wins the privacy-utility trade-off, while prefix-tuning fails under differential privacy.","key_machinery":"The argument rests on three measurement instruments. The first is DP-Adam, implemented with book-keeping per-sample gradient clipping, which injects calibrated noise so that no single training sample can strongly influence the model. The second is canary-based extraction: a secret-code sentence is inserted into 0.25% of the training set, the model generates 1,000 candidate continuations from a shared prefix, and the rank of the true canary among those candidates is converted into an exposure score, with higher exposure meaning stronger memorization. The third is SPV-MIA, a membership inference attack that fine-tunes a reference model on self-prompted text from the target model and uses a second-order directional-derivative signal to decide membership. Together these instruments define both sides of the reported privacy-utility trade-off: perplexity measures utility, while exposure and attack AUC measure empirical privacy risk.","core_discovery":"On its own terms, the paper establishes an empirical, rather than purely theoretical, privacy benefit of DP in fine-tuned LLMs. Without DP, full fine-tuning and LoRA memorize training data strongly: the inserted canary is output at maximum exposure, and SPV-MIA reaches AUC above 80% for both, up to 97.8% for full fine-tuning. Prefix-tuning and P-tuning show markedly lower memorization and MIA risk even without DP. Adding DP with a privacy budget as high as ε=50 cuts exposure strongly for full fine-tuning and LoRA and drives all methods' MIA AUC down to roughly 58%; lowering the budget further gives only marginal additional MIA reduction. In the privacy-utility plane, full fine-tuning sits closest to the lower-left corner and is therefore the best overall option under DP, while prefix-tuning becomes unusable because its perplexity explodes.","pith_inferences":["Inference: the apparent 'natural privacy' of prefix-tuning and P-tuning may reflect limited capacity in the updated parameter set; if so, deliberately constraining where PEFT updates land could reduce both memorization and DP noise sensitivity.","Inference: because MIA AUC saturates near 58%, the residual signal at high ε may be an artifact of current attacks; a stronger future MIA could reveal membership leakage that this measurement misses.","Inference: the model-generated candidate space may underestimate real extraction risk, since a real adversary could guess from a better-informed candidate distribution; the DP exposure results should be re-checked against such distributions before deployment.","Inference: the observation that larger models tolerate DP better suggests a testable design rule—when fine-tuning under a fixed budget, prefer the largest model that the compute budget allows."],"forward_implications":["DP at a high budget like ε=50 still gives large empirical privacy gains, so practitioners do not necessarily need very low budgets to blunt extraction and membership attacks.","For privacy-sensitive deployment, full fine-tuning under DP is the best choice when compute allows; otherwise LoRA is preferable when utility matters and P-tuning when privacy matters.","Prefix-tuning should not be combined with DP in its current form, since noise destroys its utility.","Deploying full fine-tuning or LoRA without DP on sensitive data is risky, since both exhibit near-maximal canary exposure and high MIA success.","Beyond the initial DP application, further lowering the budget yields diminishing returns for MIA resistance, so the added utility cost of tiny budgets may not be justified by these attacks."],"supporting_citations":[{"why":"Defines the exposure metric used to score canary extraction risk in all data-extraction experiments.","marker":"[7]"},{"why":"Provides the SPV-MIA membership inference attack whose AUC and ROC results drive the MIA findings.","marker":"[11]"},{"why":"Supplies the earlier DP-and-PII study that this paper extends from full fine-tuning to PEFT methods.","marker":"[20]"},{"why":"Gives earlier PEFT privacy results that this paper directly contrasts, especially for LoRA exposure.","marker":"[22]"},{"why":"Implements the book-keeping per-sample gradient clipping that makes DP-Adam efficient enough to run on the tested models.","marker":"[5]"},{"why":"Origin of DP-SGD, the clipping-and-noising mechanism underlying the DP training analyzed here.","marker":"[1]"},{"why":"Defines LoRA, one of the four fine-tuning methods whose privacy-utility trade-off is compared.","marker":"[13]"},{"why":"Defines prefix-tuning, the method found to suffer severe utility loss under DP.","marker":"[16]"},{"why":"Defines P-tuning, the method found to offer stronger privacy at some utility cost.","marker":"[19]"}],"fun_headline_variants":["DP cuts LLM privacy risk even at high budgets","Empirical privacy win: DP tames LLM attacks at ε=50","Full fine-tuning beats prefix-tuning under differential privacy","High-budget DP still slashes LLM memorization and MIA","DP on LLMs: utility trade-off favors full fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The extraction-risk conclusion assumes that the 1,000 model-generated candidates are a meaningful stand-in for the guesses a real adversary would try; if a real adversary's candidate distribution differs, the measured exposure levels may not reflect actual extraction risk.","fun_headline_variants_meta":{"raw":{"variants":["DP cuts LLM privacy risk even at high budgets","Empirical privacy win: DP tames LLM attacks at ε=50","Full fine-tuning beats prefix-tuning under differential privacy","High-budget DP still slashes LLM memorization and MIA","DP on LLMs: utility trade-off favors full fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000168,"raw_usage":{"total_tokens":1253,"prompt_tokens":928,"completion_tokens":325,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":239}},"tokens_in":544,"tokens_out":325,"duration_ms":3427,"temperature":1.0,"reasoning_tokens":239,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:51:23.889665+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune a GPT-2 model with full fine-tuning under DP at ε=50 using the paper's canary setup, then compute exposure not against 1,000 model-generated candidates but against a candidate set built from the true secret-code alphabet or an independent external guess list. If the true canary ranks near the top under that alternative candidate space, or if a stronger membership inference attack lifts AUC well above 58%, the claim that even high privacy budgets substantially reduce empirical privacy risk would fail in that setting.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the exposure metric used to score canary extraction risk in all data-extraction experiments."},{"cited_title":"In: 2023 IEEE Symposium on Security and Privacy","cited_arxiv_id":null,"evidence_quote":"Supplies the earlier DP-and-PII study that this paper extends from full fine-tuning to PEFT methods."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives earlier PEFT privacy results that this paper directly contrasts, especially for LoRA exposure."},{"cited_title":"In: Proceedings of the 40th International Conference on Machine Learning (2023)","cited_arxiv_id":null,"evidence_quote":"Implements the book-keeping per-sample gradient clipping that makes DP-Adam efficient enough to run on the tested models."},{"cited_title":"In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pro- cessing","cited_arxiv_id":null,"evidence_quote":"Defines prefix-tuning, the method found to suffer severe utility loss under DP."},{"cited_title":"AI Open5, 208–215 (2024)","cited_arxiv_id":null,"evidence_quote":"Defines P-tuning, the method found to offer stronger privacy at some utility cost."}],"review_version":1}