{"id":"892ab440-a57a-40f1-a785-9daa8651a991","arxiv_id":"2507.10886","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A 'healing' procedure that uses similar real examples as surrogates for unlearned data during fine-tuning mitigates accuracy loss from machine unlearning in several classification benchmarks.","lead":"This paper introduces 'healing', a method that replaces data points requested for unlearning with similar spare examples and fine-tunes the model on them, to counter accuracy loss from adversarial or routine unlearning requests. It reports experiments on MNIST, CIFAR-10 and AFHQ showing that healing often recovers accuracy close to a full retrain, with twin-based replacement sometimes beating random or no replacement.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Twin-based healing underperforms plain fine-tuning/random replacement in the paper's strongest adversarial setting (MNIST worst-25-logits, Influence path), so the claim of protecting against adversary actions lacks support.","rationale":"The reader's weakest assumption concerns the availability and quality of twins; our concern goes further: even in the paper's own experiments, twin-based healing often fails to outperform plain fine-tuning or random replacement. This is the single most load-bearing issue because the paper's headline novelty is the twin mechanism, not generic fine-tuning. The strongest adversarial scenario (worst-25-logits, Influence) shows a 5-point deficit for twins over random, directly undermining the abstract's claim of protecting against adversary actions. We still agree with the CONDITIONAL verdict because the idea is novel and the reported deficiencies are addressable with variance reporting, a better adversarial protocol, and honest caveats in the claims; the paper should not be rejected outright, but it must be revised to avoid overclaiming. Our concern is distinct from a mere 'outside consensus' critique; it is an internal inconsistency between the stated contribution and the reported results.","tokens_in":14516,"tokens_out":3385,"duration_ms":36725,"concrete_test":"Re-run the MNIST worst-25-logits experiment with Influence unlearning using 10 random seeds; for each seed, compare after 5 epochs: (a) fine-tuning on remaining data plus RawMahalanobis twins, (b) remaining data plus 25 random samples, (c) remaining data only. Report mean ± std of test accuracy and paired differences. If (a) is not significantly better than (b) or (c) by a paired test, the claimed benefit of twin selection for adversarial unlearning is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Table 5, for the only truly adversarial deletion rule (MNIST worst-25-logits, Influence-initiated unlearning), the twin-based healing variant reported as best (T-RawMahal) reaches 87.73% after 5 epochs, while the random-replacement baseline reaches 92.65% and the gold standard is 98.72%. This is a direct counterexample to the abstract's claim that healing with similar examples protects model performance from adversary actions. The same table shows FS heals well, but the IF path is the one with severe initial degradation (11.71%) and is precisely where a protective method is most needed. Moreover, in Table 4 the remain-only condition (R) is best or tied for best in several dataset/start-point combinations (MNIST FS 1-epoch, CIFAR-10 IF N/2, AFHQ FS 1-epoch and IF N/2), so the specific twin mechanism adds little beyond simple fine-tuning. Since the reported numbers come from a single representative run and 'best' is selected over five configurations, the observed 87.73 vs 92.65 gap could easily vanish under repeated sampling. This internal evidence directly contradicts the paper's central contribution: the novelty of healing rests on twin selection rather than generic fine-tuning, yet the strongest adversarial test shows twin selection is worse than random replacement, and several settings show no advantage over remain-only fine-tuning.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies adversarial machine unlearning, in which a malicious party submits deletion requests intended to degrade a trained model's accuracy. It proposes \"healing\": after an unlearning operation, the model owner fine-tunes on the remaining training data plus a small set of real examples selected to be similar to the deleted examples, drawn from a withheld spare set. The authors introduce two selection strategies (General Spare Set and Twins Strategy) and several similarity metrics (pixel L2, feature cosine, Mahalanobis). Experiments on MNIST, CIFAR-10, and AFHQ with Fisher and Influence unlearning show that fine-tuning generally restores accuracy, often close to the gold-standard retrained model, but with notable exceptions in the strongest adversarial setting.","tokens_in":14706,"tokens_out":5631,"duration_ms":65524,"significance":"The question of how to mitigate performance loss from adversarial unlearning is timely and practically relevant. The paper contributes a broad experimental sweep across three datasets, several unlearning methods, multiple similarity metrics, and an explicit adversary taxonomy. It also honestly acknowledges that healing is not true unlearning. However, the central claim that healing with similar examples protects against adversary actions is not consistently supported by the reported experiments, particularly in the worst-25-logits Influence scenario, where random replacement outperforms twin-based healing. If the authors can provide multi-run statistics, add the missing baselines, and reframe the contribution as an empirical study of post-unlearning fine-tuning, the paper could become a useful reference; in its current form the overstatement is load-bearing.","major_comments":[{"comment":"The strongest adversarial scenario contradicts the abstract's claim of protecting against adversary actions. In the MNIST worst-25-logits experiment with Influence-initiated unlearning, the initial accuracy is 11.71%, and after 5 healing epochs the best twin variant (T-RawMahal) reaches 87.73%, while the random-replacement baseline reaches 92.65% and the gold standard is 98.72%. Thus twin-based healing is worse than randomly chosen replacement samples precisely where the model is most degraded and protection is most needed. Because Table 5 is based on a single representative run, the reported 5-percentage-point gap may not be robust; please report means and standard deviations over repeated healing runs.","section":"Section 5.3.2, Table 5"},{"comment":"The benefit of twin-based healing over plain fine-tuning is not consistently demonstrated. RemainOnly (R) achieves the best accuracy in 4 of the 12 settings (including 2 of 6 N/2-epoch settings), and in Table 5 random replacement beats the best twin variant in the IF 5-epoch row. Because \"Best Acc\" is the maximum over five data configurations, the reported twin advantage could be an artifact of selection. A direct statistical comparison (e.g., paired tests across repeated runs) among RemainOnly, random replacement, and each twin metric is needed to establish that similarity-based selection, rather than additional fine-tuning data, drives recovery.","section":"Section 5.3.2, Table 4"},{"comment":"The method is explicitly not an unlearning method. The text states healing \"is not unlearning or even a type of performance recovery\" and \"leads to a certain paradox\" by re-introducing similar real examples, and Section 6 concedes it \"is not essentially unlearned in the sense of metrics on the forgetting quality.\" The abstract's phrase \"protecting model performance from these side effects\" should be qualified to indicate that the method provides apparent unlearning without forgetting guarantees. Otherwise the contribution risks being circular: fine-tuning on surrogate examples similar to the removed ones should, by construction, restore accuracy on data resembling the removed instances.","section":"Sections 4 and 6"}],"minor_comments":[{"comment":"Lines 2-3 contain a typographical error: the argmin is written as d(z', z*) but should be d(z, z'), and the condition line reads \"if then d(z, z*) < delta.\"","section":"Algorithm 1"},{"comment":"There are typos such as \"Unleraning procedures\" and \"M' F isher\"; the manuscript would benefit from a careful proofread.","section":"Section 5.1"},{"comment":"In the adversary taxonomy, the phrases \"chosen trading set\" and \"known training set\" are used with overlapping and confusing wording; please clarify the distinction between knowing the training set and being able to add elements to it.","section":"Section 3"},{"comment":"The five data setups are described as \"(i) using the remaining primary training data combined with the method-based twins; (ii) using only the remaining primary training data,\" which lists only two of the five; please specify the four twin variants explicitly.","section":"Section 5.3.1"},{"comment":"The text around Table 5 states that \"in most scenarios presented, employing twins ... provides better healing outcomes than randomly chosen samples,\" but Table 5 shows that random replacement is better in the IF 5-epoch row; please reconcile the statement with the table.","section":"Section 5.3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant problem and contains a useful empirical study, but the central novelty claim is currently overstated. The evidence in Table 5 directly undercuts the abstract's promise of protection against adversary actions, and the inconsistent advantage over RemainOnly in Table 4 suggests the twin mechanism is not the main driver. A major revision with multi-run statistics, explicit random-replacement baselines in all tables, and a reframed contribution would be needed before the paper can be accepted. If the editors value the empirical taxonomy and unlearning-susceptibility findings, the revised version could still fit the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi,\n\nThe one thing you should know: the healing idea—keep a reserve of similar examples and fine-tune the unlearned model on them—is genuinely new in the unlearning literature, and the paper is honest that this is not real unlearning. But the evidence for the headline claim, that healing protects against adversarial unlearning, is thin, and the paper's own strongest adversarial experiment undercuts it.\n\nWhat is actually new is the spare-set/twin mechanism: instead of just fine-tuning on remaining data, you match removed examples to similar real examples held out before training and use those for fine-tuning. That specific procedure is not in the cited attacks or defenses. The paper does several things well. It documents that Fisher unlearning collapses even with small random deletions—useful negative evidence. It compares four similarity metrics and two approximate unlearning methods across three datasets, and it reports mean/std dev for the starting models in Table 6. The authors explicitly concede that healing reintroduces properties of the removed data, which is the right kind of honesty.\n\nThe soft spots are real but fixable. The healing tables report a single representative run and select the best of five data configurations, so the central comparison has no variance. In the one genuinely adversarial deletion rule—MNIST worst-25-logits with Influence unlearning—the best twin healing reaches 87.73% after 5 epochs, while random replacement reaches 92.65% and the gold standard is 98.72%. That is a direct counterexample to the abstract's claim. In several other settings, remain-only fine-tuning is competitive or best, so the twin mechanism often adds little. The method is also partially circular by design: fine-tuning on similar real examples should restore accuracy on data resembling the removed points. All of this is addressable with variance reporting, a harder adversarial protocol, and stronger baselines. The citation pattern looks fine; the prior attacks are cited and the distinction from [13] is addressed.\n\nBottom line: this paper is for people working on unlearning robustness, not for practitioners needing a ready defense. The idea is worth replicating and deserves a serious referee, but the abstract overclaims and the experiments need major revision before the central claim is supported. I'd send it to review with that expectation.","headline":"The healing idea is genuinely new, but the paper's strongest adversarial experiment shows twin selection losing to random replacement, so the headline claim outruns the data.","tokens_in":15331,"tokens_out":3041,"would_cite":true,"duration_ms":32080,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning on similar 'twin' examples restores accuracy lost in machine unlearning.","keywords":["machine unlearning","adversarial unlearning","model healing","data removal","right to be forgotten","fine-tuning","influence functions","SISA"],"falsifier":"A single experiment would settle the role of twin matching: request unlearning of the 25 training points that are farthest from every held-out spare example under all three metrics, heal with the closest available spares, and compare accuracy against the gold-standard retrained model. If the gap is no larger than when random training points are removed, the twin mechanism is not load-bearing and healing is just fine-tuning on remaining data.","tokens_in":14224,"feed_emoji":"🩹","tokens_out":6780,"duration_ms":74069,"temperature":0.7,"pith_summary":"AI systems can be forced to forget training data, whether to comply with regulations such as the GDPR or because a malicious party files deletion requests, and the removal itself can badly damage accuracy. This paper claims that much of the damage can be repaired by 'healing': keep a spare set of real examples out of training, and when a deletion request arrives, fine-tune the model on the remaining data plus a spare example judged similar to the one removed. The authors test healing after Fisher and influence-function unlearning on MNIST, CIFAR-10, and AFHQ, and report accuracy restored to within a few tenths of a percentage point of retraining from scratch in most settings, sometimes slightly above it. They also report that raw approximate unlearning is fragile, with Fisher updates collapsing models to near single-class accuracy even under only 5% random deletion.","feed_headline":"Look-alike examples can heal models after forced forgetting","feed_subtitle":"Keeping similar real samples out of training, then fine-tuning with them after a delete request, restores accuracy to near retraining…","key_machinery":"The central object is the spare set and the twin-replacement rule. Healing is defined in Section 4 through Algorithm 1 (General Spare Set) and the Twins strategy: before training, a set DS of spare elements is withheld; when an element z is requested for unlearning, the model picks the most similar element z* in DS under a metric d, uses it for a fine-tuning step only if d(z,z*) < δ, and then removes z* from the spare set. The Twins strategy precomputes a similar surrogate for each protected training element, possibly extending to triplets and quadruplets. The similarity metrics tested are raw-pixel L2 distance, feature-space cosine similarity, and Mahalanobis distance computed on model embeddings. The mechanism does the work of replacing removed knowledge with transferable properties of a near-duplicate real example rather than synthetic data.","core_discovery":"The central claim is that replacing deleted training examples with similar real examples is an effective protection against both spontaneous and adversarial unlearning. In Section 4 healing is defined as retraining the unlearned model on the retained data together with a surrogate z* drawn from a withheld spare set, where z* is the closest example under a chosen metric and is used only if the distance is below a threshold δ. Experiments with 25 forgotten images show healing recovers accuracy close to the gold-standard retrained model MGold: on MNIST the best healed model reached 99.03% against 98.87% gold, on CIFAR-10 91.41% against 90.97%, and on AFHQ 99.73% against 99.40%, after roughly half the original training epochs of fine-tuning. The paper also claims that model susceptibility to unlearning differs by backbone and adversary knowledge, and that exact unlearning via SISA degrades steadily as deletion fraction grows while naive retraining is robust.","pith_inferences":["The paper leaves implicit that healing deliberately keeps some functional trace of the removed data through its surrogates, so it is likely to satisfy a formal deletion request while falling short of strict definitions of forgetting; the paper itself notes healing is not unlearning in a forgetting-quality sense.","A stress test the paper does not run: an adversary who can choose the forget set will pick points with no twin within the threshold, reducing healing to plain fine-tuning; whether plain fine-tuning already matches the gold model determines how much the twin mechanism contributes.","The spare set could be shrunk by exploiting the memorization-versus-generalization distinction the paper cites as future work: models need fewer surrogates for items they have generalised from, so the minimal protective set may be much smaller than the training set.","Group healing, replacing a deleted subset by a similar subset of surrogates, could reduce the required spare-set size for large deletion requests, a direction the paper lists but does not evaluate."],"forward_implications":["A model owner who withholds a spare set before training can answer any unlearning request by fine-tuning on the retained data plus a matched twin, typically closing the accuracy gap to retraining from scratch without paying the full retraining cost.","The protection is not limited to sophisticated attacks: even random deletion under the exact SISA method steadily lowers accuracy as the deleted fraction grows, so healing is relevant for routine deletion requests too.","Raw Fisher unlearning without healing collapses models to near single-class accuracy on MNIST, CIFAR-10, and AFHQ, so approximate unlearning pipelines that use it need a healing step to be usable.","Longer healing fine-tuning of about half the original training epochs is consistently better than a single epoch, and adding matched twins often gives the best final accuracy, while twins give a faster initial correction after just one epoch.","Healing works from both Fisher and Influence starting points, bringing best-case healed accuracy to within a few tenths of a point of the gold-standard model in the tested settings."],"supporting_citations":[{"why":"supplies the SISA sharded unlearning method used as an exact-unlearning baseline whose accuracy degrades with deletion fraction.","marker":"[2]"},{"why":"defines the Fisher-information unlearning update that the paper shows collapses without healing.","marker":"[16]"},{"why":"defines influence-function unlearning via inverse Hessian-gradient products, the other approximate method healed in experiments.","marker":"[24]"},{"why":"provides the sequential mini-batch unlearning update protocol used in the experiments.","marker":"[29]"},{"why":"supplies LiSSA, the algorithm used to approximate inverse-Hessian products for influence unlearning.","marker":"[1]"},{"why":"MNIST is the first benchmark dataset for the healing experiments.","marker":"[27]"},{"why":"CIFAR-10 is the second benchmark dataset, where healing is tested after Fisher and influence unlearning.","marker":"[25]"},{"why":"AFHQ supplies the high-resolution animal-face benchmark for the third experiment.","marker":"[7]"}],"fun_headline_variants":["Healing unlearned models with look-alike data","Look-alike examples restore models after forced forgetting","Surrogate examples undo damage from malicious unlearn requests","Retraining on look-alikes fixes models after unlearning attacks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a stock of real examples similar enough to the data that may later be deleted exists and has been withheld from training, and that the chosen similarity metric identifies examples that can stand in for the removed ones during fine-tuning.","fun_headline_variants_meta":{"raw":{"variants":["Healing unlearned models with look-alike data","Look-alike examples restore models after forced forgetting","Surrogate examples undo damage from malicious unlearn requests","Retraining on look-alikes fixes models after unlearning attacks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000158,"raw_usage":{"total_tokens":1200,"prompt_tokens":893,"completion_tokens":307,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":242}},"tokens_in":509,"tokens_out":307,"duration_ms":3814,"temperature":1.0,"reasoning_tokens":242,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:22:17.547215+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A single experiment would settle the role of twin matching: request unlearning of the 25 training points that are farthest from every held-out spare example under all three metrics, heal with the closest available spares, and compare accuracy against the gold-standard retrained model. If the gap is no larger than when random training points are removed, the twin mechanism is not load-bearing and healing is just fine-tuning on remaining data.","supporting_citations":[{"cited_title":"Bourtoule, V","cited_arxiv_id":null,"evidence_quote":"supplies the SISA sharded unlearning method used as an exact-unlearning baseline whose accuracy degrades with deletion fraction."},{"cited_title":"Golatkar, A","cited_arxiv_id":null,"evidence_quote":"defines the Fisher-information unlearning update that the paper shows collapses without healing."},{"cited_title":"Agarwal, B","cited_arxiv_id":null,"evidence_quote":"supplies LiSSA, the algorithm used to approximate inverse-Hessian products for influence unlearning."},{"cited_title":"Jeong, S","cited_arxiv_id":null,"evidence_quote":"CIFAR-10 is the second benchmark dataset, where healing is tested after Fisher and influence unlearning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AFHQ supplies the high-resolution animal-face benchmark for the third experiment."}],"review_version":1}