REVIEW 10 cited by
Are we making progress in unlearning? Findings from the first NeurIPS unlearning competition
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We present the findings of the first NeurIPS competition on unlearning, which sought to stimulate the development of novel algorithms and initiate discussions on formal and robust evaluation methodologies. The competition was highly successful: nearly 1,200 teams from across the world participated, and a wealth of novel, imaginative solutions with different characteristics were contributed. In this paper, we analyze top solutions and delve into discussions on benchmarking unlearning, which itself is a research problem. The evaluation methodology we developed for the competition measures forgetting quality according to a formal notion of unlearning, while incorporating model utility for a holistic evaluation. We analyze the effectiveness of different instantiations of this evaluation framework vis-a-vis the associated compute cost, and discuss implications for standardizing evaluation. We find that the ranking of leading methods remains stable under several variations of this framework, pointing to avenues for reducing the cost of evaluation. Overall, our findings indicate progress in unlearning, with top-performing competition entries surpassing existing algorithms under our evaluation framework. We analyze trade-offs made by different algorithms and strengths or weaknesses in terms of generalizability to new datasets, paving the way for advancing both benchmarking and algorithm development in this important area.
Forward citations
Cited by 10 Pith papers
-
Distribution Preference Optimization: A Fine-grained Perspective for LLM Unlearning
DiPO is a distribution-level unlearning method that constructs preference distributions from the model's own high-confidence logits and achieves state-of-the-art forget quality on TOFU while preserving utility.
-
Invisible Watermarks, Visible Gains: Steering Machine Unlearning with Bi-Level Watermarking Design
Water4MU tunes an invisible watermark on data so that machine unlearning algorithms can remove requested images more effectively, beating prior methods on 'challenging forgets'.
-
Rectifying Privacy and Efficacy Measurements in Machine Unlearning: A New Inference Attack Perspective
RULI is a per-sample, dual-objective inference attack that measures privacy leakage and unlearning efficacy, showing average-case evaluations understate privacy risk.
-
Certified Unlearning for Neural Networks
Noisy fine-tuning with gradient or model clipping on retained data provably removes the influence of forget data, with guarantees that need no smoothness or convexity assumptions.
-
Improved Localized Machine Unlearning Through the Lens of Memorization
DEL, which pairs channel-level weighted-gradient localization with reset-and-finetune, reports state-of-the-art unlearning metrics on CIFAR-10, SVHN, and ImageNet-100.
-
Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks
A system-first taxonomy and literature synthesis of multimodal unlearning across vision, language, video, and audio, with datasets, benchmarks, metrics, applications, and open challenges.
-
When unlearning is free: leveraging low influence points to reduce computational costs
Low-influence training points can be dropped from forget/retain sets before unlearning, cutting runtime up to ~50% with little measured loss in accuracy or MIA-based privacy.
-
Verifying Machine Unlearning with Explainable AI
The authors use SIDU heatmaps and two new metrics to test whether machine unlearning removes reliance on human patterns in a thermal object-counting model.
-
On the importance of multiple training seeds for evaluating machine unlearning
Machine-unlearning evaluation with a single training seed can misrepresent method performance, particularly for deterministic unlearning methods, and extra unlearning seeds do not fix it.
-
Lifting Data-Tracing Machine Unlearning to Knowledge-Tracing for Foundation Models
A position paper urging a shift from data-tracing to knowledge-tracing machine unlearning for foundation models, supported by a CLIP case study that shows current methods struggle to generalize.
Discussion (0). Continue with ORCID to comment.