{"id":"1e91707d-3538-4eea-9b20-5d4d0ee8268b","arxiv_id":"2506.10424","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"SOFT paraphrases low-loss fine-tuning samples before training, reducing MIA AUC from about 0.82 to about 0.54 across six datasets at roughly 7% perplexity cost.","lead":"This paper proposes a defense that rewrites the most memorized fine-tuning examples so membership inference attacks can no longer tell which texts a model trained on. The approach cuts average attack success roughly in half while keeping model quality close to full fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOFT's selection rule leaves above-threshold members exposed; the paper's own Selection Only adaptive attack (AUC 0.651, TPR@1%FPR 0.086) contradicts the near-random guessing claim.","rationale":"The reader's weakest_assumption identifies the data selection rule's safety assumption as the critical point: members with loss above the validation average may still be vulnerable, and the validation threshold may be a poor proxy for attack risk. The paper's own 'Selection Only' adaptive attack provides direct empirical evidence for this concern, with AUC 0.651 and TPR@1%FPR 0.086 on the unselected subset. This is the same load-bearing concern, now supported by the paper's internal results. The reader's CONDITIONAL verdict already accounts for the uncertainty; my stress-test does not move the verdict because the existing conditional status is appropriate. However, the conclusion's overbroad phrasing 'near-random guessing' should be revised to acknowledge the residual leakage on unselected samples. The concrete test would quantify how general this residual leakage is and whether the central claim needs further qualification.","tokens_in":28962,"tokens_out":7681,"duration_ms":102958,"concrete_test":"Run the Selection Only adaptive attack (Table 7) on all six datasets using the same protocol, reporting pooled AUC-ROC and TPR@1%FPR restricted to the unselected (above-threshold) members, together with the per-dataset fraction of samples paraphrased by SOFT. If the pooled AUC exceeds 0.6 or TPR@1%FPR exceeds 0.05, the near-random guessing claim is falsified for the unselected subset; if the pooled AUC stays at or below 0.55 and TPR near 0.01, the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SOFT reduces attack success to near-random guessing is load-bearing on Algorithm 1's data selection rule (DATA_SELECTION, Lines 20-30): samples with loss below the average validation loss are paraphrased, while samples with loss above the threshold are left untouched and assumed safe. The paper's own adaptive attack in Section 5.5, 'Selection Only', directly tests this assumption by attacking only the remaining un-paraphrased sentences. On ArXiv it achieves AUC-ROC 0.651 and TPR@1%FPR 0.086, far above random (0.5 and 0.01 respectively). This shows that members whose loss is above the validation average, i.e., exactly the samples SOFT chooses not to obfuscate, remain distinguishable by a straightforward loss-based attack. The conclusion's unqualified statement that SOFT 'reduces the attack success rate to near-random guessing' is therefore not supported for the subset the defense intentionally leaves unmodified. The paper does not report the fraction of samples selected, error bars for Table 7, or analogous results on the other five datasets, so the extent of this residual leakage is unknown. If the Selection Only attack performs similarly across datasets, SOFT's privacy-utility trade-off is materially weaker than claimed: it protects only the low-loss fraction of the data, and an adversary aware of the selection strategy can concentrate attacks on the high-loss remainder.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Zhang et al. study membership inference attacks (MIAs) against fine-tuned large language models and propose SOFT, a defense that paraphrases the subset of the fine-tuning dataset whose per-sample loss falls below the average validation loss. The paper presents an empirical analysis on six Pile subsets with Llama-3.2 and Pythia models, evaluates ten MIAs (including a proposed Ensemble attack), reports average AUC-ROC dropping from 0.819 under full fine-tuning to 0.540 under SOFT, and compares SOFT against DP-LoRA. The central claim is that SOFT reduces attack success to near-random guessing while incurring only a small utility cost.","tokens_in":29221,"tokens_out":5366,"duration_ms":59314,"significance":"If the near-random-guessing claim is supported, SOFT would be a practical and light-weight privacy defense for fine-tuning, avoiding the utility and computational overhead of DP-based methods. The paper also contributes a systematic empirical mapping of MIA effectiveness across model sizes, LoRA ranks, epoch counts, and dataset properties, and it is commendable that the authors attempt adaptive attacks and release their code on GitHub. The residual leakage identified by the paper's own Selection Only adaptive attack, together with missing variance estimates in the main evaluation tables and an underspecified Ensemble attack protocol, currently prevent me from taking the headline privacy claim at face value.","major_comments":[{"comment":"The 'Selection Only' adaptive attack, which attacks exactly the un-paraphrased members that Algorithm 1 leaves untouched, achieves AUC-ROC 0.651 and TPR@1%FPR 0.086 on ArXiv. These numbers are far above the random baselines of 0.5 and 0.01, respectively. The conclusion in Section 7 that SOFT 'reduces the attack success rate to near-random guessing' is therefore not supported for the subset of members that the defense deliberately does not obfuscate. Please report Selection Only results on all six datasets, state the fraction of samples selected by the threshold in each dataset, and qualify the conclusion, or demonstrate that the residual leakage is acceptable under a concrete risk criterion.","section":"Section 5.5, Table 7"},{"comment":"The metrics section states that AUC is computed on 1,000 bootstrapped subsets and that both mean and standard deviation are reported, but Tables 1-4 list only point estimates without error bars or confidence intervals. Without variance measures, the reader cannot judge whether SOFT's average AUC-ROC of 0.540 is statistically distinguishable from the pre-trained model's near-0.5 values, nor can differences such as 0.525 vs. 0.508 be assessed. Please report the standard deviations (or confidence intervals) for all tables, and state the number of independent fine-tuning runs; bootstrap over the attack evaluation does not capture variance due to the fine-tuning process itself.","section":"Section 5.1 and Tables 1-4"},{"comment":"The Ensemble attack's training and evaluation protocol is not described. The text says it aggregates twelve features but does not state the classifier family, the training set construction from member/non-member samples, the train/test split, whether the attack is trained on labels from the same distribution used during evaluation, or how the reference model is incorporated. Because Ensemble is one of the two reference-based attacks and is the strongest attack in several settings (e.g., Table 1, GitHub, AUC 0.669), this omission prevents replication and weakens the claim of a systematic vulnerability study. Please provide the full protocol in the paper or an appendix.","section":"Section 3.1 and Figure 2"},{"comment":"The DP-LoRA comparison is run on Llama-3.2-1B, whereas the main SOFT evaluation in Tables 1-4 uses Llama-3.2-3B. If the SOFT row in Table 5 corresponds to the 1B model, this should be stated explicitly; if it corresponds to the 3B model, the comparison is confounded by model size. The SOFT AUC-ROC values in Table 5 (e.g., Ensemble 0.573) also differ from the 3B values in Table 1 (ArXiv Ensemble 0.568), so the reader cannot infer which model is used. Please make the model size explicit and, ideally, compare SOFT and DP-LoRA at the same model scale.","section":"Section 5.4, Tables 5-6"},{"comment":"The selection rule in DATA_SELECTION paraphrases samples with loss below the average validation loss and leaves all others unchanged. Since the reference-free MIAs under evaluation are loss-based, the reduction achieved by SOFT is partly by construction: it removes precisely the low-loss members that those attacks flag. The paper's adaptive attacks are a genuine check, but the paper's own Selection Only result shows that the remaining high-loss members are still distinguishable. The manuscript should therefore present the Selection Only result as an explicit bound on worst-case residual leakage, analyze how the threshold τ controls the trade-off between selection coverage and privacy leakage, and reconcile this with the unqualified near-random-guessing statement in the conclusion.","section":"Algorithm 1 and Section 5.5"}],"minor_comments":[{"comment":"The text describes α as 'a paraphrasing strength factor' where α=0.5 preserves the first half and paraphrases the remaining half, then says 'a higher paraphrasing ratio may reduce the utility of the data.' Since α is the fraction of text retained, a higher paraphrasing ratio corresponds to a lower α; the phrasing is ambiguous and should be clarified.","section":"Section 4.2"},{"comment":"The text states that 'even when subjected to the adaptive attack, SOFT's performance remains unaffected,' but Table 7 shows Paraphrase & Selection AUC-ROC 0.595 versus 0.568 for 'No Adaptive (w/ SOFT),' and Paraphrase Only 0.575. The change is modest but not zero; 'unaffected' overstates the result.","section":"Section 5.5"},{"comment":"The sentence 'Table 1 and 3 present the AUC-ROC scores, showing that LoRA fine-tuning remains vulnerable' is confusing because Table 1 reports SOFT and full fine-tuning, while Table 3 is the LoRA table. Please rephrase to point the reader to the correct table for each comparison.","section":"Section 5.2"},{"comment":"The adaptive attack and the data-selection ablation report small numerical differences, but no error bars are provided for Table 7 or for Figures 9-10's box plots beyond the mean markers. Please add the number of repetitions and statistical significance or confidence intervals so readers can assess the reliability of these differences.","section":"Section 5.5 and Section 5.6"},{"comment":"The α ablation reports 'slight' differences in TPR@1%FPR, but the y-axis ranges in the figure are narrow; if the differences are not statistically significant, please say so explicitly rather than relying on visual inspection.","section":"Section 5.6, Figure 10"},{"comment":"The text says 'we evaluate the Pythia family ... covering six model sizes: 70M, 160M, 1B, 1.4B, 2.8B, and 6.9B' but does not mention the 410M model that appears in the model list in Section 3.1; please reconcile the counts.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a security/ML venue and the empirical direction is useful, but the headline claim needs to be scaled back to match the Selection Only result. The missing error bars and the underspecified Ensemble attack are fixable in revision. I recommend a major revision rather than rejection because the core idea is sound and the adaptive evaluation, while currently partial, is the right kind of test."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a competent, well-scoped empirical paper that introduces a simple defense (SOFT) and evaluates it more broadly than prior fine-tuning MIA work. The main problem is that the conclusion oversells the defense. SOFT paraphrases samples below the validation-loss threshold and leaves everything else untouched; the paper's own Selection Only adaptive attack shows those untouched members are still detectable (AUC 0.651, TPR@1%FPR 0.086 on ArXiv, versus 0.5/0.01 random). That is not near-random guessing for the subset the defense deliberately does not obfuscate.\n\nWhat's actually new: the first systematic study of MIAs against fine-tuned LLMs across two model families, six datasets, and a ten-attack battery, including a new ensemble attack. SOFT itself—loss-threshold selection plus paraphrase obfuscation—is new relative to the cited defenses. The code and data are released, which is real evidence and makes the results checkable. The ablations (paraphraser quality, alpha, LoRA integration) are genuinely useful.\n\nSoft spots, in proportion: (1) Selection Only is the load-bearing one, as above. The paper acknowledges that row but does not reconcile it with the conclusion's 'near-random' claim, nor report the fraction of samples selected or analogous numbers for the other datasets. (2) Tables 1–4 omit the standard deviations that the metrics section says are computed; the ensemble attack's train/test protocol is not described, so its reported AUCs are hard to trust. (3) The DP-LoRA comparison in Table 5 compares against a different model size (1B vs 3B), which weakens the utility trade-off claim. (4) Utility is perplexity plus a judge-based QA test; it does not directly measure factual retention.\n\nOn the circularity concern: yes, paraphrasing low-loss samples targets exactly the signal loss-based MIAs use, so the average AUC drop is partly by construction. But the adaptive attacks are a genuine effort to go beyond that, and the defense does something real—it just does not achieve what the conclusion says.\n\nWho this is for: researchers working on LLM privacy defenses or MIA evaluation. They will find the empirical map useful and the defense worth trying. It deserves a serious referee, but the revision needs to tone down the near-random claim, report selection fractions and error bars for the adaptive attacks, and fix the DP comparison. Send it to review, with major revisions expected.","headline":"A solid, broad empirical study of MIAs on fine-tuned LLMs with a practical defense, but the conclusion oversells SOFT: its own Selection Only adaptive attack shows the un-paraphrased subset still leaks.","tokens_in":29805,"tokens_out":1959,"would_cite":true,"duration_ms":22924,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned LLMs leak membership through lower loss, and selectively paraphrasing the riskiest samples drives attack accuracy back to near random guessing.","keywords":["membership inference attack","LLM fine-tuning","data obfuscation","paraphrasing defense","influential data selection","loss-based attack","privacy-utility trade-off","differential privacy comparison"],"falsifier":"Construct a fine-tuning set in which some members are rare-format or high-entropy texts (so their loss stays above the validation average), run SOFT, and then evaluate an Ensemble attack restricted to those unparaphrased high-loss members; if their membership AUC exceeds about 0.6, the loss-threshold selection rule has missed the vulnerable population and the defense's central premise fails.","tokens_in":28777,"feed_emoji":"🔒","tokens_out":7503,"duration_ms":83369,"temperature":0.7,"pith_summary":"Fine-tuned large language models leak which texts were in their training sets: three epochs of full fine-tuning let ten membership inference attacks reach an average AUC-ROC of 0.819. The paper attributes this to loss reduction — members end up with lower loss, and every strong attack exploits that signal. It proposes SOFT, which each epoch computes per-sample training loss, flags samples whose loss falls below the average validation loss as \"influential\", and replaces them with paraphrased versions before fine-tuning continues. Across six Pile domains and Llama-3.2 models, this brings average attack AUC down to 0.540 and TPR at 1 percent FPR down to 0.029, while ArXiv perplexity rises only from 9.78 to 10.49. The claim is that the loss-threshold selection rule identifies the samples that actually drive membership leakage, so a defender can obfuscate only those and keep the rest of the data intact.","feed_headline":"Paraphrasing risky samples cuts LLM membership attack AUC to 0.54","feed_subtitle":"Only the lowest-loss samples get obfuscated, so the model keeps most of its fine-tuning utility while members stop being identifiable.","key_machinery":"The load-bearing object is the loss-threshold selection rule in Algorithm 1 (DATA_SELECTION). It computes the loss of every fine-tuning sample under the current model, sets the threshold as the mean loss on a small validation set, and declares every sample below that threshold \"influential\" — the ones a membership attacker can exploit because the model has fitted them. Those samples are then replaced by PARAPHRASE(x, α) with α = 0.5 (keep the first half, paraphrase the rest), and only this obfuscated set participates in the next gradient step. The same idea drives the paper's Ensemble attack: it aggregates loss, perplexity, lowercase, zlib, Min-K%++ features at several k, and ratio, which is what exposes the loss signal that SOFT then suppresses.","core_discovery":"The paper's central claim is that membership inference against fine-tuned LLMs is primarily a loss phenomenon and therefore a targeted data-obfuscation defense can neutralize it. In full fine-tuning on Llama-3.2-3B, ten MIAs (Loss, Zlib, Lowercase, Min-K% Prob, Min-K%++, Ratio, Bag of Words, ReCall, CON-ReCall, and an Ensemble attack that combines twelve features) average AUC-ROC 0.819 across six Pile datasets; SOFT reduces this to 0.540, close to the 0.509 average of the pre-trained model, and reduces TPR@1%FPR from 0.360 to 0.029. The mechanism is iterative: fine-tune once to get a warm-up model, then at each epoch paraphrase exactly those training samples whose current loss is below the average validation loss, leaving all other samples untouched. The paper reports that the resulting model answers domain questions almost as well as a fully fine-tuned one (LLM-as-a-Judge 0.66 vs 0.68) and beats DP-LoRA on the privacy-utility trade-off at comparable computational overhead.","pith_inferences":["The authors do not explore it, but the loss-threshold rule could be reused for other memorization-related threats such as data extraction or verbatim regurgitation, because those attacks also rely on low-loss overfitted samples.","The defense shifts the burden to the paraphrase model: with a weaker local paraphraser (BART, seq2seq) privacy still drops but utility degrades more, so deployment without access to a strong commercial API may face a worse trade-off than the headline numbers.","The \"Selection Only\" adaptive result (AUC 0.651, TPR@1%FPR 0.086) suggests the unparaphrased remainder still carries a residual membership signal; an attacker that can predict which samples were chosen might concentrate on them, so the privacy guarantee is only as good as the selection rule's unpredictability.","A natural testable extension is to apply the loss-threshold rule to other fine-tuning paradigms (adapters, prompt tuning) and to measure whether the defense holds against attacks that use the paraphrase API itself as a shadow model."],"forward_implications":["A fine-tuned model protected by SOFT makes ten standard MIAs perform only slightly better than random guessing (average AUC 0.540), so membership attacks no longer reveal training membership at scale.","At 1 percent false positive rate, SOFT reduces the true positive rate from 0.360 (full fine-tuning) to 0.029, meaning almost no member can be singled out with high confidence.","The utility cost is small: ArXiv perplexity rises from 9.78 to 10.49, and LLM-as-a-Judge quality scores fall only from 0.68 to 0.66.","SOFT preserves most of full fine-tuning's utility while providing stronger privacy than DP-LoRA at the same or lower computational overhead, and it can be combined with LoRA for even lower attack AUC.","Adaptive adversaries who know the paraphrase model and selection rule still stay near random (AUC 0.575–0.595) unless they attack only unparaphrased samples, which recovers only a small fraction of members."],"supporting_citations":[{"why":"Supplies the Loss, Zlib, Lowercase, and Ratio attack baselines and the extraction-threat framing that motivates the study.","marker":"[16]"},{"why":"Provides the MIMIR 13-gram data partition methodology used to build member and non-member sets without temporal shift.","marker":"[23]"},{"why":"Supplies the Pile dataset from which the six fine-tuning domains are drawn.","marker":"[34]"},{"why":"Motivates the ensemble feature-aggregation attack used as the strongest MIA baseline.","marker":"[60]"},{"why":"Supplies the Bag-of-Words sanity check and the argument that MIA on pre-trained LLMs is mostly dataset shift rather than leakage.","marker":"[62]"},{"why":"Supplies the Min-K% Prob attack and the Wiki-MIA dataset whose temporal-shift flaw motivates careful data splits.","marker":"[73]"},{"why":"Supplies the CON-ReCall contrastive-decoding attack evaluated as a reference-free baseline.","marker":"[82]"},{"why":"Supplies the ReCall background-prefix attack evaluated as a reference-free baseline.","marker":"[87]"},{"why":"Supplies the DP-LoRA baseline that SOFT is compared against on the privacy-utility trade-off.","marker":"[94]"},{"why":"Supplies the Min-K%++ attack whose token-level features are used inside the Ensemble attack.","marker":"[98]"}],"fun_headline_variants":["SOFT: obfuscate only low-loss samples to cut MIA AUC to 0.54","Loss-based obfuscation kills LLM membership leaks","Fine-tune LLMs privately by masking only easy samples","Selective obfuscation drops MIA success to near-pre-trained","Loss-targeted obfuscation keeps fine-tuning utility, blocks MIAs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The defense assumes that the samples a membership attacker can identify are exactly those whose loss sits below the average validation loss, so it paraphrases only those and leaves every higher-loss sample untouched; if any member with loss above that threshold can still be recognized, the near-random AUC claim does not generalize.","fun_headline_variants_meta":{"raw":{"variants":["SOFT: obfuscate only low-loss samples to cut MIA AUC to 0.54","Loss-based obfuscation kills LLM membership leaks","Fine-tune LLMs privately by masking only easy samples","Selective obfuscation drops MIA success to near-pre-trained","Loss-targeted obfuscation keeps fine-tuning utility, blocks MIAs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000765,"raw_usage":{"total_tokens":3408,"prompt_tokens":972,"completion_tokens":2436,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":2348}},"tokens_in":588,"tokens_out":2436,"duration_ms":21142,"temperature":1.0,"reasoning_tokens":2348,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:27:00.426093+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a fine-tuning set in which some members are rare-format or high-entropy texts (so their loss stays above the validation average), run SOFT, and then evaluate an Ensemble attack restricted to those unparaphrased high-loss members; if their membership AUC exceeds about 0.6, the loss-threshold selection rule has missed the vulnerable population and the defense's central premise fails.","supporting_citations":[{"cited_title":"LLM dataset inference: Did you train on my dataset? In The Thirty-eighth Annual Confer- ence on Neural Information Processing Systems, 2024","cited_arxiv_id":null,"evidence_quote":"Motivates the ensemble feature-aggregation attack used as the strongest MIA baseline."},{"cited_title":"ReCaLL: Membership inference via relative conditional log-likelihoods","cited_arxiv_id":null,"evidence_quote":"Supplies the ReCall background-prefix attack evaluated as a reference-free baseline."},{"cited_title":"Differentially private fine-tuning of language models","cited_arxiv_id":null,"evidence_quote":"Supplies the DP-LoRA baseline that SOFT is compared against on the privacy-utility trade-off."}],"review_version":1}