{"id":"bb9aba70-4d25-40ac-a485-89e355f8a839","arxiv_id":"2601.05134","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Sequential subspace noise injection lets noisy fine-tuning for certified unlearning keep its privacy budget while avoiding the severe accuracy drop from one-shot noise.","lead":"This paper proposes a block-wise noise schedule for certified machine unlearning: instead of adding Gaussian noise to all model weights at once, it adds noise to one orthogonal subspace at a time. The authors claim this preserves the same differential-privacy-style guarantee while reducing the accuracy collapse seen with standard noisy fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Certificate is conditional on an unmeasured, likely too-small initial discrepancy Δ(ρ); Theorem 2's stated (ε,δ) budget also omits the failure probability ρ, so the central guarantee may be vacuous.","rationale":"The reader's weakest assumption correctly identifies the unmeasured initial discrepancy Δ(ρ) as the central vulnerability. I reach the same conclusion: without measuring or bounding the actual coupled distance between the fully trained and retained-data models, the claimed certificate is conditional to the point of being untestable. My read adds a second, related formal gap: the high-probability statement in Definition 3 introduces a failure probability ρ that never appears in Theorem 2's (ε,δ) formula. A rigorous conversion from the conditional proximity event to the unconditional distributional guarantee of Definition 1 would require adding ρ to δ or increasing ε, so the theorem as stated is incomplete even if Δ were measured. This does not change the overall verdict: the method may be salvageable by measuring Δ, choosing a conservative ρ, and restating the guarantee, but the paper as written overclaims that the original worst-case certification is preserved. The code release is a genuine asset because it makes the proposed measurement check immediately executable. I therefore keep the reader's CONDITIONAL verdict unchanged, while noting that the concern is slightly broader than the reader's wording.","tokens_in":22832,"tokens_out":12291,"duration_ms":138172,"concrete_test":"In the released repository, add logging to the retrain-from-scratch reference model used as x̂', preserving the same coupling (matched seed, minibatch ordering, noise schedule) as the fully trained model. For each of the five runs in each scenario (MNIST random 10%, CIFAR-10 random 10%, CIFAR-10 class-5 deletion), record the L2 parameter distance ∥x̂−x̂'∥ after training. Report the mean, max, and 95th percentile, and compare with the Δ values 0.01/0.05 used in Tables 5 and 7. If the 95th percentile exceeds Δ, Theorem 2's precondition is not met at the implied ρ, and the stated (ε,δ) guarantee does not hold for the evaluated models.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the replacement of the worst-case clipping radius C0 in Theorem 1 by Δ(ρ)/2 (Section 3.2, Definition 3) and the claim in Theorem 2 that the same (ε,δ) budget is preserved. This holds only if the coupled fully-trained and retained-trained models satisfy ∥x̂−x̂'∥ ≤ Δ(ρ) with probability ≥ 1−ρ. In Section 5 the authors set Δ=0.01 (random 10% deletion) and Δ=0.05 (classwise deletion), calling Δ a 'tunable calibration hyperparameter' (Appendix D.2). They never measure or upper-bound the actual distance for ResNet-18/MNIST. For a network with millions of parameters, an L2 distance of 0.01 between two converged coupled SGD solutions is far from obviously attainable; removing 10% of the data or an entire class can plausibly move the weights much more. If the true distance exceeds Δ, the noise schedule computed from the smaller Δ does not satisfy the Rényi bound, and the (ε,δ) certificate is vacuous. Moreover, Definition 3 is a high-probability statement, yet Theorem 2's formula contains no ρ term. Converting a conditional guarantee into the unconditional Definition 1 requires absorbing ρ into δ (or increasing ε), which the paper never does. The strong MIA/UA numbers are therefore empirical evidence, not evidence that the stated certificate applies to the actual models.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes block-wise noisy fine-tuning (Sequential Subspace Noise Injection) for certified unlearning. The parameter space is partitioned into k orthogonal subspaces, and noisy fine-tuning is applied sequentially to one block at a time, with the noise variance computed from the NFT analysis of Koloskova et al. after replacing the initial clipping radius C0 by a high-probability initial-discrepancy Δ(ρ)/2. Theorem 2 claims that the same (ε,δ) budget is retained via Rényi DP composition, and Corollary 1 states that equal-size blocks preserve the total budget with k times the step budget. Experiments on MNIST and CIFAR-10 with MLPs, ResNet-18, and ViT-Tiny report substantially improved post-unlearning accuracy and strong MIA robustness compared to NFT and several empirical baselines. The central theoretical claim is conditional on Definition 3, and the proof of Theorem 2 in Appendix F.2 is only a sketch.","tokens_in":1533,"tokens_out":1937,"duration_ms":105619,"significance":"If the certificate were established unconditionally, the paper would make an important practical contribution: it would mitigate the accuracy collapse that has limited DP-based certified unlearning in deep networks while retaining formal guarantees. The empirical study is broad, covering multiple architectures, deletion scenarios, block constructions, and privacy budgets, and the authors release code, which strengthens reproducibility. However, the certificate is conditional on an unmeasured quantity Δ(ρ), and the proof of the main theorem is not fully spelled out. As it stands, the utility results are credible empirical evidence, but the central claim of preserving the same (ε,δ) guarantee is not yet established.","major_comments":[{"comment":"The certificate is conditional on the event ∥x̂−x̂′∥≤Δ(ρ), but Theorem 2's formula (6) contains no ρ term and no conversion of the failure event into δ or ε is given. Section 5 sets Δ=0.01 (random 10%) and 0.05 (classwise) by hand, and Appendix D.2 describes Δ as a 'tunable calibration hyperparameter'; the actual coupled distance between the fully trained and retained models is never measured or bounded. Thus the reported (ε,δ) may not apply to the models actually used. The authors should measure or upper-bound the empirical quantile of ∥x̂−x̂′∥ for the exact coupling used in experiments, and then either absorb the failure probability ρ into the stated (ε,δ) guarantee or clearly present the result as a conditional guarantee that is not Definition 1.","section":"§3.2, Definition 3, Theorem 2, Algorithm 1 (Step 5)"},{"comment":"The proof of Theorem 2 is only a sketch. The text says that by 'zeroing out the coordinates sequentially' the Rényi increments can be summed, but the induction, the per-block conditioning events, and the interaction with the decomposed Shift Reduction Lemma are not formally spelled out. Corollary 1 additionally relies on the claim that group norms are 'approximately equal' to justify treating C0/√k and C1/√k as per-block clipping bounds, which is a heuristic rather than a proof. Since the same-(ε,δ)-budget claim is the core theoretical contribution, a complete proof with explicit assumptions on block sizes, step counts, and noise schedule is needed.","section":"Appendix F.2, Theorem 2 and Corollary 1"},{"comment":"There is a circularity in the empirical validation: MIA is used both as the auditing metric for successful unlearning and as the heuristic for calibrating Δ(ρ), with Appendix D.2 stating that 'overly small values of Δ(ρ) lead to detectable leakage, while conservative values suppress the signal.' Table 1's MIA=100 is therefore not an independent confirmation of the certificate. The authors should report MIA results for a pre-registered or conservatively chosen Δ, or use a separate auditing procedure that is not part of the calibration loop.","section":"§5.2 and Appendix D.2"}],"minor_comments":[{"comment":"The notation ε_i in Eq. (6) is used before it is defined; the definition 'ε_i := ε_i^{rényi} + log(1/δ)/(q−1)' appears only after the equation. Please define it before the theorem or state it in the theorem.","section":"§4.2, Theorem 2"},{"comment":"Several figure captions show missing ε and δ labels, e.g., 'CIFAR10: = 5.0, = 1e 05'. Please insert the symbols.","section":"Figures 1 and 3"},{"comment":"The text says 'the per-step perturbation is smaller' and the abstract/reader summary describes it as 'k-fold smaller.' Since the per-step noise vector has ℓ2-norm σ√(d/k), the reduction is by a factor of √k, not k. Please correct the wording.","section":"Remark 4"},{"comment":"Proposition 1 establishes an inequality about NFT started from a random initialization, but the jump to 'retraining from scratch can reach accuracy close to α in T steps' is not a formal implication. The limitations paragraph in Appendix C is helpful; please mark the inference as heuristic in the main text as well.","section":"§3.2, Proposition 1 and Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The empirical results are promising and the general idea of sequential subspace noise injection is worth pursuing, but the paper's main claim—preserving the same (ε,δ) guarantee—is currently conditional in a load-bearing way. I would not accept without a complete proof of Theorem 2 and a valid, non-circular estimate of Δ(ρ) for the actual coupled model pairs. If the true distance is much larger than 0.01/0.05, the utility advantage may largely disappear; the authors should report this quantity directly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The core idea is simple and likely useful: partition parameters into orthogonal blocks, run noisy fine-tuning block by block, and you get a large empirical accuracy improvement over standard NFT while keeping the noise budget per step. The decomposed Wasserstein / shifted Rényi machinery is a sensible adaptation of Koloskova et al., and the thought experiment about initialization-agnostic guarantees, while not a formal impossibility, makes a real conceptual point about why worst-case clipping is overkill. Code is released, and the experiments are reasonably thorough.\n\nThe soft spots are concentrated where the stress-test puts them. The certificate is only valid if ||x̂ − x̂'|| ≤ Δ(ρ) with probability 1−ρ, but the authors never measure this for the actual ResNet-18/MNIST models. They set Δ=0.01 or 0.05 by hand and justify it through MIA auditing, which creates a fitting loop: the same MIA score that calibrates Δ is later reported as evidence of successful unlearning. More importantly, Theorem 2 as stated omits ρ entirely. The stated (ε,δ) budget is not the unconditional guarantee from Definition 1 unless the failure probability is absorbed into δ or ε; that is a real gap. The proof of Theorem 2 is also a sketch, not a full block-by-block composition, so a referee should push for a complete proof.\n\nI would not call this fatal. The paper is transparent about conditioning on Δ, and the empirical method stands on its own as a practical improvement over NFT. But the abstract overstates the guarantee, and the theory needs to be cleaned up before I would trust the certificate as stated. The empirical MIA/UA numbers are good evidence of utility, not evidence that the stated (ε,δ) applies to the actual models.\n\nSend it to peer review, but with a clear request to fix the Δ issue, absorb ρ into the guarantee, and complete the proof of Theorem 2. The right audience is the unlearning/DP subfield; this is not a general-ML paper. A serious referee will find it worth the time, but the revision needs to make the conditioning explicit and honest throughout.","headline":"Useful empirical trick and a real proof adaptation, but the certificate is conditional on an unmeasured Δ(ρ) and Theorem 2 omits the failure probability — the practical guarantee is weaker than the abstract claims.","tokens_in":23658,"tokens_out":2066,"would_cite":true,"duration_ms":24031,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Certified unlearning can keep its formal guarantee and its accuracy by injecting noise into one orthogonal weight block at a time.","keywords":["certified unlearning","machine unlearning","differential privacy","noisy fine-tuning","orthogonal subspaces","Rényi differential privacy","membership inference attack","accuracy preservation"],"falsifier":"Run many random 10% deletions with the same coupled randomness used in the paper, record ∥x̂−x̂′∥ after training, and compare the empirical 95th percentile with Δ(ρ)=0.01; if the measured percentile is much larger than 0.01, then at the claimed noise levels the mechanism is not certified for those deletions, and the accuracy and MIA numbers would be empirical rather than guaranteed.","tokens_in":22765,"feed_emoji":"🧩","tokens_out":8687,"duration_ms":92895,"temperature":0.7,"pith_summary":"Certified machine unlearning normally pays for its guarantee with destroyed accuracy: the calibrated Gaussian noise added while fine-tuning on retained data drowns the gradient in large networks. This paper claims that the same (ε,δ) guarantee survives if the weight space is split into k orthogonal subspaces and the noise budget is spent sequentially on one block at a time. Each step then perturbs only a 1/k fraction of coordinates, so the model does not take a single full-dimensional noise hit; the block-wise privacy-loss terms add, leaving the total budget unchanged. The paper also drops worst-case initial clipping in favour of a high-probability bound Δ(ρ) on the distance between the fully trained and the retrained models, which tightens the constants. If the claim is right, regulated forgetting requests can be served with formal guarantees without collapsing accuracy.","feed_headline":"Block-by-block noise keeps certified unlearning accurate","feed_subtitle":"Spreading the noise budget across orthogonal weight blocks preserves the (ε,δ) guarantee and avoids the accuracy cliff.","key_machinery":"The load-bearing object is the orthogonal decomposition W = Σ A_i B_i with A = [A1 ... Ak] orthonormal; the unlearning schedule freezes all blocks except the active one and adds Gaussian noise only there. The analytical engine is a block-adapted version of the shifted Rényi divergence and a decomposed Wasserstein distance: each step is compared against a reference run with the same noise coupling, and the divergence increment is charged to the active block. A decomposed Shift Reduction Lemma bounds how much one Gaussian step can increase the divergence when only one block is shifted, which lets the paper sum per-block privacy losses and conclude the standard (ε,δ) guarantee. The same equival","core_discovery":"The central claim is that noisy fine-tuning can be executed block-wise with no loss of certificate. Choose k orthogonal subspaces spanning the weight space; the algorithm runs clipped noisy gradient steps on one block at a time, cycling through all k blocks, then fine-tunes the full model. The proof shows the (ε,δ) unlearning budget is preserved: if the i-th block consumes a Rényi privacy budget ε_i, the total satisfies ε = Σ_i ε_i − (k−1) log(1/δ)/(q−1), so splitting into equal blocks keeps the same overall ε. Because the blocks are orthogonal, independent Gaussian noise per block is exactly equal in distribution to one isotropic Gaussian on the full vector, while the per-step perturbation","pith_inferences":["A natural extension is to estimate the full distribution of ∥x̂−x̂′∥ and set Δ(ρ) to its empirical quantile; this would make the certificate self-contained rather than conditional on an unverified choice.","Adaptive block selection is the obvious next knob: allocating larger privacy sub-budgets to layers that tolerate noise, or smaller ones to the classification head, could further close the gap to retraining.","The sequential-subspace argument is not specific to unlearning: private fine-tuning or differentially private training of deep networks suffers the same noise-dominated updates, so the same budget-splitting could stabilize training there too.","The thought experiment implies a broader design principle for unlearning definitions: certificates that are uniform over all initializations cannot simultaneously promise speed and utility; future definitions may need to condition on the actual training distribution."],"forward_implications":["Over-parameterized models no longer hit the accuracy cliff during unlearning: with k blocks, each noise step disturbs only a 1/k fraction of coordinates, so the model spends less time in the low-accuracy regime.","The number of blocks k becomes a utility/privacy knob: the total (ε,δ) budget is unchanged as k grows, while per-step distortion shrinks, at the cost of up to k times as many unlearning steps.","Formal certificates can be issued without worst-case assumptions on initialization; conditioning on the actual coupled distance between full and retained training runs yields tighter constants and removes the initial clipping step.","On standard image benchmarks, the unlearned model matches retraining on forgetting metrics (UA=100, MIA=100 for random and class-wise deletion) while keeping retain and test accuracy close to retraining.","Because the block decomposition is computed once and reused, the approach carries over to different architectures, including transformer backbones, with the same stability gains."],"fun_headline_variants":["Subspace noise scheduling stops unlearning accuracy collapse","Certified unlearning keeps accuracy with orthogonal noise","Block-wise noise preserves ε,δ while boosting unlearning utility","Sequential noise injection: unlearning without the accuracy hit"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper's certificate is conditional on the high-probability bound Δ(ρ) on the distance between the fully trained and retrained models; the paper treats Δ(ρ) as a tunable calibration hyperparameter (Appendix D.2) rather than measuring it, so if the true distance exceeds the chosen value, the stated (ε,δ) guarantee does not actually hold.","fun_headline_variants_meta":{"raw":{"variants":["Subspace noise scheduling stops unlearning accuracy collapse","Certified unlearning keeps accuracy with orthogonal noise","Block-wise noise preserves ε,δ while boosting unlearning utility","Sequential noise injection: unlearning without the accuracy hit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1058,"prompt_tokens":658,"completion_tokens":400,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":402,"completion_tokens_details":{"reasoning_tokens":336}},"tokens_in":402,"tokens_out":400,"duration_ms":4499,"temperature":1.0,"reasoning_tokens":336,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T11:46:10.400619+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run many random 10% deletions with the same coupled randomness used in the paper, record ∥x̂−x̂′∥ after training, and compare the empirical 95th percentile with Δ(ρ)=0.01; if the measured percentile is much larger than 0.01, then at the claimed noise levels the mechanism is not certified for those deletions, and the accuracy and MIA numbers would be empirical rather than guaranteed.","supporting_citations":[],"review_version":1}