{"id":"b2a3c5d6-1335-4c99-add4-e93345778b06","arxiv_id":"2501.11525","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Per-sample gradient noise and exponential down-weighting of LIRA-identified vulnerable points reduce membership inference success on CIFAR-10 while keeping test accuracy near baseline.","lead":"This paper proposes a way to train image classifiers so that specific, vulnerable training images are harder to detect as having been used in training, by adding noise and giving those images lower weight during gradient updates. The authors test it on CIFAR-10 with a ResNet-18 model and report that membership inference attacks become much weaker while accuracy stays roughly the same.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline claim rests on a metric-alignment risk: the defense suppresses the same LIRA Gaussian IN/OUT gap that the evaluation measures, so a LIRA-only order-of-magnitude reduction does not establish that membership leakage is prevented.","rationale":"The paper is transparent and intellectually honest: it is framed as a technical report, it presents full privacy-utility trade-off curves (Figure 3), it reports confidence bands for the σ=0.03 cases (Figure 7), and Sections 2.8 and 5.2 explicitly list the missing validations (other datasets, stronger attacks, DP baselines, distributional assumptions). The method's internal logic is coherent: shadow-model-derived t-scores (Eq. 7) rank vulnerability, exponential weights (Eq. 12) suppress high-vulnerability samples, and the reported LIRA metrics do drop substantially (Table 3: τ@0.001 FPR from 5.2 to 1.7 at weighted σ=0.01) with accuracy approximately preserved. I found no internal inconsistency in the derivations; Eq. 8 is a valid per-sample weighted-gradient update, and implementing the weighting as a weighted loss is correct for constant weights. The identified soft spot is not the mathematics but the strength of the evidence: the defense's target quantity and the evaluation's measured quantity are the same statistic—the IN/OUT separation of Gaussian-fitted, logit-scaled losses (Eqs. 2–4 versus Eqs. 7 and 12). Suppressing the top of the t-score distribution will, by construction, collapse the LIRA likelihood-ratio signal, so the measured order-of-magnitude reduction cannot distinguish 'the model no longer memorizes' from 'the meter used to check was the same meter the method was designed to move.' The paper itself cites Aerni et al. [40] for the proposition that this class of evaluation artifact has previously led to order-of-magnitude overstatements of privacy. The decisive experiment is evaluation under a structurally independent attack; RMIA is the natural candidate because it uses pairwise likelihood ratios rather than per-point Gaussian fits, and it is already named in the paper's future work. If RMIA's low-FPR TPR stays near random on the protected model, the concern is settled in the authors' favor and the CONDITIONAL verdict can be upgraded. If RMIA re-identifies members, the headline claim should be re-scoped to LIRA-measured risk and the method re-evaluated against a DP-SGD baseline at matched accuracy. Because this is exactly the condition the reader attached, the verdict remains CONDITIONAL (UNCHANGED), with the condition sharpened from 'test stronger attacks' to 'test whether claimed forgetting survives an attack that does not share the defense's sufficient statistic.'","tokens_in":22111,"tokens_out":13289,"duration_ms":123993,"concrete_test":"Retrain the weighted scenario (α=2, β=2, σ=0.01) on the same audit split, then attack the resulting model with RMIA (Zarifzadeh et al., 2023) using at least 300 reference/shadow models, and report TPR@0.1% FPR, τ, and AUC alongside the LIRA values in Table 3. Decision rule: if RMIA's TPR@0.1% FPR is near the random baseline (≈0.1%) and its AUC is within noise of 0.5, the concern is refuted and the order-of-magnitude claim survives. If RMIA recovers membership at materially above-random TPR (e.g., >5% at 0.1% FPR) or AUC ≳ 0.6, the reported LIRA-only reduction reflects suppression of the defense's own target statistic, and the headline claim must be re-scoped to LIRA-measured leakage pending a DP-SGD comparison at matched utility.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that instance-specific gradient obfuscation (Eq. 8) reduces privacy risk by at least an order of magnitude and prevents sensitive data from being embedded—is measured with the same statistical machinery that defines the defense's target. The vulnerability score t_j (Eq. 7) is the normalized IN–OUT separation of the Gaussian fits over shadow models; the weight w(t)=exp(−max(0,αt−β)) (Eq. 12) suppresses exactly the samples with the largest IN–OUT gap; and the headline metrics (AUC, τ=log(TPR/FPR), Table 3) come from LIRA, which thresholds precisely that same Gaussian IN–OUT likelihood ratio (Eqs. 2–4). The defense minimizes, and the audit measures, the same one-dimensional statistic. The drop in τ from 5.2 to 1.7 at σ=0.01 (weighted) is therefore expected by construction for any down-weighting of large-t points; it does not establish that no membership signal remains in the model. The paper's own limitations (§5.2) concede that stronger attacks are untested, and §5.1 cites Aerni et al. [40]—the result that empirical defenses evaluated against the attacks they were designed against can overstate privacy by orders of magnitude—without supplying the requested RMIA or DP-SGD comparison. The load-bearing assumption is not merely that LIRA is representative; it is that the evaluation and the defense do not share a sufficient statistic. Because they do share one, the measured quantity (LIRA leakage) is not a valid estimate of the claimed quantity (membership leakage against arbitrary adversaries). The claim of 'preventing embedding in the first place' is an existential statement about the model; an audit that reuses the defense's own score as its meter cannot support it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This technical report proposes 'Forgotten by Design,' a training-time defense that combines per-example Gaussian gradient noise with a LIRA-derived weighting of vulnerable samples. The authors define a privacy vulnerability t-score from shadow-model Gaussian fits (Eq. 7), assign each sample a weight w(t)=exp(-max(0,alpha t - beta)) (Eq. 12), and add noise to the gradient sum (Eq. 8). On CIFAR-10 with ResNet-18, they report that the weighted scheme at sigma=0.01 reduces LIRA tau@0.001FPR from 5.217 to 1.712 and AUC from 0.885 to 0.671 while accuracy remains around 70-70.5%, and that stronger noise pushes AUC toward 0.5. They interpret these results as preventing sensitive data from being embedded and as an order-of-magnitude privacy improvement, and they frame the approach in terms of motivated forgetting and the right to be forgotten.","tokens_in":22397,"tokens_out":7842,"duration_ms":78142,"significance":"The idea of targeting gradient obfuscation to samples that are individually vulnerable to a specified membership-inference attack is a reasonable engineering step, and the paper has several strengths: it uses at least 300 shadow models per configuration, reports ROC curves with confidence bands, visualizes the privacy-utility frontier, and explicitly lists its limitations. If the claims were restricted to LIRA-measured leakage, the results would be a useful empirical data point for privacy audit practice. However, the main claim that the method 'prevents sensitive data from being embedded' is not established: the defense and the evaluation share the same LIRA statistic, no comparison to stronger attacks (e.g., RMIA) or DP-SGD is provided, and no confidence intervals are given for the headline privacy metrics. As it stands, the paper reads as a feasibility study rather than a validated privacy guarantee.","major_comments":[{"comment":"The reported privacy reduction is partly engineered because the defense uses the same LIRA statistic that defines the evaluation. The vulnerability weight w(t)=exp(-max(0,alpha t - beta)) suppresses samples with the largest normalized IN-OUT gap t_j (Eq. 7), while the attack metrics tau and AUC are computed by thresholding the LIRA log-likelihood ratio of precisely that gap (Eq. 4). Down-weighting high-t points therefore lowers the measured leakage by construction, independently of whether a different adversary would still succeed. The paper's own Section 5.1 cites Aerni et al. [40] as showing that empirical defenses evaluated against the attacks they are designed against can overstate privacy, and Section 5.2 concedes that RMIA is untested. I request an evaluation with RMIA (or a simple non-LIRA threshold attack) and a DP-SGD baseline; absent that, the abstract's 'prevents sensitive data from being embedded' should be replaced by 'reduces LIRA-measured membership leakage.'","section":"Sections 3.2 and 3.4, Eqs. (4), (7), (12)"},{"comment":"The abstract's claim of 'at 95% significance' is not supported by any confidence interval, p-value, or hypothesis test for AUC or tau in the main text. Section 3.2 explains how shadow models can provide confidence bands, and Figure 7 displays such bands, but Table 3 reports only point estimates for AUC and tau, with standard deviations only for accuracy. With at least 300 shadow models per configuration, the authors should report bootstrap or shadow-model-based confidence intervals for AUC and tau, and a test for the key comparison (e.g., tau@0.001FPR = 5.217 at baseline versus 1.712 for weighted sigma=0.01). Without this, the headline statistical claim is unverifiable.","section":"Abstract and Table 3"},{"comment":"The headline operating point (alpha=2, beta=2, sigma=0.01) was selected after inspecting the privacy-utility trade-off curves, so the reported tau of 1.712 and 70.5% accuracy are in-sample point estimates for a chosen configuration. The order-of-magnitude claim is sensitive to this choice: Table 3 shows weighted-row tau values ranging from 4.525 to 0.159 depending on sigma. Please report a sensitivity analysis over alpha, beta, and sigma, or a pre-specified selection rule with a validation split, so that the reader can assess whether the improvement is robust rather than an artifact of post-hoc parameter choice.","section":"Section 4.1 and Table 2"},{"comment":"The conclusion that the method 'prevents sensitive data from being embedded' is not supported by the evidence even for the LIRA adversary. At the headline setting (weighted sigma=0.01), AUC is 0.671 and tau@0.001FPR is 1.712, so the attacker's TPR/FPR ratio is still around exp(1.712) approximately 5.5; only at sigma=0.03-0.05 does AUC approach 0.5, at the cost of a 5-13 percentage-point accuracy drop. Moreover, membership inference is only one type of privacy harm, so the absolute wording claims more than the experiments can show. I recommend reporting residual leakage at each operating point and rewording the abstract and conclusions to describe a reduction in LIRA-measured leakage rather than the prevention of sensitive-data embedding.","section":"Section 5.2, Table 3, and Conclusions"}],"minor_comments":[{"comment":"References [34] and [40] are the same paper (Aerni, Zhang, and Tramèr, CCS 2024); please consolidate the duplicate entry.","section":"Section 2.7 and References"},{"comment":"The caption says 'uniform noise addition (0.01 and 0.02)' but the legend and the text refer to 0.01 and 0.03; the caption should match the plotted curves.","section":"Figure 2 caption"},{"comment":"The 'weighted: 0.007' row is duplicated; one copy should be removed.","section":"Table 4"},{"comment":"The phrase 'the cross-entropy loss nor the logits' should be 'the cross-entropy loss or the logits'; the definition of p_j as exp(-ell) also appears only later, so an early forward reference would improve readability.","section":"Section 3.1"},{"comment":"Accuracies are reported as fractions in Table 3 (e.g., 0.7) and as percentages in Table 4 (e.g., 70.043); please use one convention consistently.","section":"Section 4.1 and Table 3"},{"comment":"The paper does not provide the full training setup (epochs, batch size, learning-rate schedule, optimizer, and number of gradient steps) needed to reproduce the experiments; adding a reproducibility appendix would strengthen the report.","section":"General / Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"To the editor: this manuscript reads as a project technical report rather than a completed research article. The empirical core is plausible but narrow, and the central claim is currently overstated. The authors are transparent about the LIRA-only evaluation, which argues for requiring an additional attack rather than rejecting the paper. I recommend major revision: the authors should add an RMIA or otherwise non-LIRA evaluation, a DP-SGD comparison, and confidence intervals for the headline privacy metrics before the paper can be accepted. If the journal is oriented toward archival research contributions, the missing validation is essential for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my take on the Forgotten-by-Design report. The genuinely new thing is the specific combination: instance-specific gradient noise plus exponential down-weighting of samples by LIRA t-scores, and the tau metric for privacy-utility plots. The pieces exist, but this particular recipe hasn't been tested before. The authors also do some things right: they train hundreds of shadow models, report tail-risk metrics (TPR at low FPR) rather than just AUC, include full parameter sweeps, and spend real space in Sections 2.8 and 5.2 acknowledging what they didn't test. That level of candor is not usual and should be credited.\n\nThe problem is the central claim. The defense weights down the very samples that LIRA's t-score flags, and then evaluates success with LIRA's own AUC and tau. So the measured reduction is partly built into the construction. The abstract says 'reduces privacy risks by at least an order of magnitude,' but that is only against LIRA, not against membership inference in general. The authors themselves cite Aerni et al. showing that empirical defenses evaluated only on the attack they were designed against can overstate privacy by orders of magnitude, and they concede in Section 5.2 that RMIA and DP-SGD comparisons are missing. That concession is honest, but it means the headline is not backed by the evidence they present.\n\nOther soft spots: the operating point (alpha=2, beta=2, sigma=0.01) is chosen after looking at the trade-off curves, so the 'order of magnitude' number is a selected result. The tau/AUC values in Table 3 have no confidence intervals, and the abstract's 'at 95% significance' is not substantiated with an actual test for the privacy metrics. The t-score variant (denominator sqrt of sum of variances instead of sum of stds) is a minor change, fine, but not a contribution on its own.\n\nWho gets value from this? People working on practical RTBF audits and empirical MIA defenses. It's a useful starting point—the weighting idea and the tau visualization are worth discussing. But the paper should be treated as a technical report, not a settled result.\n\nIf I were a referee, I would ask for: a DP-SGD baseline, an RMIA (or at least a non-LIRA) evaluation, confidence intervals on tau and AUC, and a pre-registered or at least clearly indicated parameter selection procedure. The method may well survive that stronger evaluation; the current paper doesn't show it.\n\nRecommendation: send it to peer review, but require major revision. The idea is worth referee time; the evidence is not there yet.","headline":"The paper's core idea is plausible and the authors are honest about limits, but the headline 'order of magnitude' claim is not yet supported because the defense and the evaluation both operate on the same LIRA statistic.","tokens_in":23026,"tokens_out":2867,"would_cite":false,"duration_ms":29971,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Instance-specific gradient obfuscation during training prevents sensitive data from being embedded, reducing membership-inference risk by an order of magnitude while preserving accuracy.","keywords":["right to be forgotten","membership inference attack","LIRA","gradient obfuscation","instance-specific noise","privacy-utility trade-off","differential privacy","machine learning privacy"],"falsifier":"Run the stronger RMIA attack on the protected weighted sigma=0.01 checkpoints from Table 3; if its true positive rate at 0.001 false positive rate is materially higher than the LIRA-derived tau near 1.7 implies, the order-of-magnitude claim holds only against LIRA, not against membership inference in general.","tokens_in":21869,"feed_emoji":"🔒","tokens_out":6569,"duration_ms":64366,"temperature":0.7,"pith_summary":"This paper argues that the right to be forgotten in AI should be achieved not by erasing data after training but by preventing sensitive data from being learnable in the first place. It proposes Forgotten by Design: during training, each data point receives a weight and a noise level based on how vulnerable it is to the LIRA membership-inference attack, so the model shifts its parameters away from points that would otherwise be identifiable. Experiments on a ResNet-18 model trained on CIFAR-10 show that the combined scheme cuts the LIRA attack's advantage by at least an order of magnitude while holding test accuracy at the baseline level, and at higher noise levels drives the attack toward random guessing. If correct, this offers a practical route to privacy compliance without retraining models and without committing to formal differential privacy.","feed_headline":"Targeted gradient noise cuts membership leakage tenfold","feed_subtitle":"A ResNet-18 on CIFAR-10 keeps its accuracy while LIRA attack scores fall toward random guessing.","key_machinery":"The load-bearing object is the instance-specific gradient update $$\\theta_{t+1} = \\theta_t - \\eta \\frac{1}{|B|}\\sum_{j\\in B}(w_j g_j + u_j),$$ in which each data point contributes a weight $w_j$ and a Gaussian noise $u_j \\sim \\mathcal{N}(0,\\sigma_j^2 I)$. The weight is a clipped exponential of the privacy vulnerability t-score $$t_j = \\frac{$m^{{\\mathrm{IN}}$}_j - $m^{{\\mathrm{OUT}}$}_j}{\\sqrt{$V^{{\\mathrm{IN}}$}_j + $V^{{\\mathrm{OUT}}$}_j}},$$ computed from LIRA shadow-model fits of scaled logits, so a point with a higher vulnerability score receives a lower weight $w_{\\mathrm{exp}}(t) = e^{-\\max(0,\\alpha t - \\beta)}$. This redirects the gradient update away from points LIRA can identify, while uniform noise blurs whatever signal remains, and the paper shows the two effects compound in the privacy-utility trade-off.","core_discovery":"The paper claims that a model can be made to forget sensitive data during training rather than after it, by obfuscating the gradients of the data points a membership-inference adversary would most reliably identify. On CIFAR-10 with ResNet-18, combining additive Gaussian noise with instance-specific down-weighting based on LIRA vulnerability scores reduces LIRA's tau at 0.001 FPR from 5.217 to 1.712 at sigma = 0.01, with test accuracy statistically unchanged, and at sigma = 0.03 drives AUC to about 0.54-0.56 and tau toward zero, close to the random-guessing baseline. The authors conclude that non-inference under a strong membership-inference test is a workable operational definition of forgetting, satisfying a practical reading of the right to be forgotten.","pith_inferences":["The method's practicality depends on the t-score ranking produced by a first round of shadow models; a natural test is whether the same ranking can be reused across architectures, datasets, or data splits without retraining shadow models.","One unstated direction is combining the instance-specific weights with DP-SGD's gradient clipping; the paper does not claim formal guarantees, but applying weights before clipping might preserve some of the targeted forgetting while recovering a bound.","Equating forgetting with non-inference implies that a model passing an accepted membership-inference audit could be treated as having forgotten a data point; that regulatory equivalence is the authors' framing, not a legal result, and would need regulator acceptance.","The strongest untested risk is attack transferability: if a stronger attack exploits different signals than LIRA, the reported reduction could shrink, so re-running the evaluation with RMIA would delimit the claim."],"forward_implications":["At sigma=0.01 with the exponential weighting, LIRA's tau at 0.001 FPR drops from 5.217 to 1.712 while test accuracy stays at 0.705, so the attacker's success per false positive falls by more than an order of magnitude.","At sigma=0.03, both schemes push AUC to about 0.54-0.56 and tau near 0.2-0.3, close to the random-guessing baseline where the attacker has little advantage over chance.","The weighted scheme sits on a more favorable privacy-utility frontier than noise-only obfuscation: at intermediate risk levels it achieves the same accuracy with lower tau at every FPR level tested.","Even at the strongest obfuscation tested, accuracy remains in the sixties, far above the roughly 10 percent random-label floor for CIFAR-10.","Because the obfuscation is applied during training, no post-training unlearning step is needed for the protected data points."],"supporting_citations":[{"why":"Defines the LIRA attack whose shadow-model Gaussian fits, scaled logits, and likelihood-ratio test are both the vulnerability measure and the evaluation metric.","marker":"[2]"},{"why":"Supplies the CIFAR-10 benchmark dataset used in all training and audit experiments.","marker":"[3]"},{"why":"Supplies the ResNet-18 architecture used as the target model and as the basis for shadow models.","marker":"[4]"},{"why":"Provides the DP-SGD update that the instance-specific obfuscation modifies, and the baseline privacy framework the paper contrasts with its approach.","marker":"[30]"},{"why":"Introduces the privacy-onion effect that motivates focusing defenses on the most vulnerable data points.","marker":"[33]"},{"why":"Sets the evaluation standard showing empirical defenses must be measured against strong attacks and practical DP baselines, which the paper partly adopts and partly leaves for future work.","marker":"[34]"},{"why":"Provides the audit implementation used to train shadow models, extract log-likelihood ratios, and compute the vulnerability t-scores.","marker":"[35]"},{"why":"Supplies the concept of instance-specific noise that the per-point noise component of the method extends to gradient descent.","marker":"[37]"},{"why":"Introduces the stronger RMIA attack that the paper identifies as the main untested threat to its LIRA-based conclusions.","marker":"[39]"}],"fun_headline_variants":["Train-time noise cuts membership inference tenfold","Forget data at training time, not after - obfuscate gradients","LIRA attack falls toward random with gradient obfuscation","Proactive obfuscation preserves accuracy while ensuring privacy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire guarantee is measured and targeted through LIRA's Gaussian approximation of shadow-model losses, so if LIRA is not a representative proxy for membership inference, the weighting protects the wrong points and the reported reduction may not transfer to stronger attacks.","fun_headline_variants_meta":{"raw":{"variants":["Train-time noise cuts membership inference tenfold","Forget data at training time, not after - obfuscate gradients","LIRA attack falls toward random with gradient obfuscation","Proactive obfuscation preserves accuracy while ensuring privacy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000629,"raw_usage":{"total_tokens":2893,"prompt_tokens":919,"completion_tokens":1974,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":1905}},"tokens_in":535,"tokens_out":1974,"duration_ms":13574,"temperature":1.0,"reasoning_tokens":1905,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:09:27.729538+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the stronger RMIA attack on the protected weighted sigma=0.01 checkpoints from Table 3; if its true positive rate at 0.001 false positive rate is materially higher than the LIRA-derived tau near 1.7 implies, the order-of-magnitude claim holds only against LIRA, not against membership inference in general.","supporting_citations":[{"cited_title":"Mem- bership inference attacks from first principles","cited_arxiv_id":null,"evidence_quote":"Defines the LIRA attack whose shadow-model Gaussian fits, scaled logits, and likelihood-ratio test are both the vulnerability measure and the evaluation metric."},{"cited_title":"Learning multiple layers of features from tiny images","cited_arxiv_id":null,"evidence_quote":"Supplies the CIFAR-10 benchmark dataset used in all training and audit experiments."},{"cited_title":"Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang","cited_arxiv_id":null,"evidence_quote":"Provides the DP-SGD update that the instance-specific obfuscation modifies, and the baseline privacy framework the paper contrasts with its approach."},{"cited_title":"The privacy onion effect: Memorization is relative","cited_arxiv_id":null,"evidence_quote":"Introduces the privacy-onion effect that motivates focusing defenses on the most vulnerable data points."},{"cited_title":"Evaluations of machine learning privacy defenses are misleading","cited_arxiv_id":null,"evidence_quote":"Sets the evaluation standard showing empirical defenses must be measured against strong attacks and practical DP baselines, which the paper partly adopts and partly leaves for future work."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the audit implementation used to train shadow models, extract log-likelihood ratios, and compute the vulnerability t-scores."},{"cited_title":"Smooth sensitivity and sampling in private data analysis","cited_arxiv_id":null,"evidence_quote":"Supplies the concept of instance-specific noise that the per-point noise component of the method extends to gradient descent."}],"review_version":1}