Pith. sign in

REVIEW 39 cited by

MUSE: Machine Unlearning Six-Way Evaluation for Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.06460 v2 pith:NFZUJRGF submitted 2024-07-08 cs.CL cs.AI

Weijia Shi , Jaechan Lee , Yangsibo Huang , Sadhika Malladi , Jieyu Zhao , Ari Holtzman , Daogao Liu , Luke Zettlemoyer
show 2 more authors
Noah A. Smith Chiyuan Zhang
This is my paper · ORCID
classification cs.CLcs.AI
keywords unlearningalgorithmsdatamodelsmemorizationremovalbenchmarkevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Language models (LMs) are trained on vast amounts of text data, which may include private and copyrighted content. Data owners may request the removal of their data from a trained model due to privacy or copyright concerns. However, exactly unlearning only these datapoints (i.e., retraining with the data removed) is intractable in modern-day models. This has led to the development of many approximate unlearning algorithms. The evaluation of the efficacy of these algorithms has traditionally been narrow in scope, failing to precisely quantify the success and practicality of the algorithm from the perspectives of both the model deployers and the data owners. We address this issue by proposing MUSE, a comprehensive machine unlearning evaluation benchmark that enumerates six diverse desirable properties for unlearned models: (1) no verbatim memorization, (2) no knowledge memorization, (3) no privacy leakage, (4) utility preservation on data not intended for removal, (5) scalability with respect to the size of removal requests, and (6) sustainability over sequential unlearning requests. Using these criteria, we benchmark how effectively eight popular unlearning algorithms on 7B-parameter LMs can unlearn Harry Potter books and news articles. Our results demonstrate that most algorithms can prevent verbatim memorization and knowledge memorization to varying degrees, but only one algorithm does not lead to severe privacy leakage. Furthermore, existing algorithms fail to meet deployer's expectations because they often degrade general model utility and also cannot sustainably accommodate successive unlearning requests or large-scale content removal. Our findings identify key issues with the practicality of existing unlearning algorithms on language models, and we release our benchmark to facilitate further evaluations: muse-bench.github.io

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 39 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Suppression Sticks, Locality Is Fragile: A Closed-Loop Target-and-Control Audit of Task-Vector Negation in VLA Policies

    cs.RO 2026-08 conditional novelty 7.0 of 10

    Subtracting a task vector from a vision-language-action robot policy suppresses the target skill but not its collateral damage: only five of ten LIBERO-Goal skills separate cleanly, and held-out control retention aver...

  2. Stochastic Dynamic Barrier Perturbed Gradient Methods for Nonconvex Simple Bilevel Optimization

    math.OC 2026-07 accept novelty 7.0 of 10

    SDBPG and its penalty/variance-reduced variants achieve the first explicit (ε_f, ε_g)-stationarity rates for stochastic nonconvex simple bilevel problems by regularizing unbounded dual multipliers.

  3. De-attribute to Forget for LLM Unlearning

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    DareU unlearns LLMs by PPO-optimizing attribution rewards so outputs are no longer attributable to forget owners, outperforming loss-based methods on forget-utility trade-offs.

  4. ICU-Bench:Benchmarking Continual Unlearning in Multimodal Large Language Models

    cs.AI 2026-05 unverdicted novelty 7.0 of 10

    ICU-Bench is a new continual unlearning benchmark for MLLMs using 1000 privacy profiles, 9500 images, and 100 forget tasks, showing existing methods fail to balance forgetting, utility, and scalability.

  5. FlexOlmo: Open Language Models for Flexible Data Use

    cs.CL 2025-07 conditional novelty 7.0 of 10

    FlexOlmo merges independently trained language-model experts, trained on private data, into a single mixture-of-experts model without joint training.

  6. Provable unlearning in topic modeling and downstream tasks

    cs.LG 2024-11 conditional novelty 7.0 of 10

    Provable (epsilon, delta)-unlearning algorithms for topic models achieve deletion capacity O~(m/(r^2 sqrt(nr))) before fine-tuning and O~(m q/(r sqrt(nr))) after fine-tuning, with the base model untouched in the downs...

  7. Hierarchical Semantic Correlation-Aware Masked Autoencoder for Unsupervised Audio-Visual Representation Learning

    cs.MM 2026-04 conditional novelty 6.5 of 10

    Localized Muon-style spectral orthogonalization of primary and constraint momentum subspaces (SIFT) reduces objective conflicts and improves constrained LLM steering across four tasks.

  8. Leak-Resistant Unlearning: A New Benchmark for Evaluating Multi-Hop Reasoning Consistency and Recovery Robustness

    cs.AI 2026-08 conditional novelty 6.0 of 10

    A new benchmark with six multi-hop reasoning structures and three recovery attacks shows that existing LLM unlearning methods leak knowledge through indirect queries and post-unlearning attacks.

  9. Stochastic Meta-Unlearning: Bridging Language Backbone and Multimodal Unlearning

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A bilevel meta-training method aligns text-only backbone unlearning with full vision-language model forgetting, improving forget-retain trade-offs across two VLMs and two meme datasets.

  10. Leak@$k$: Unlearning Does Not Make LLMs Forget Under Probabilistic Decoding

    cs.LG 2025-11 reject novelty 6.0 of 10

    LLM unlearning methods that pass greedy-decoding benchmarks leak forgotten facts when the model is sampled repeatedly, and the new leak@k metric quantifies this.

  11. Towards Evaluation for Real-World LLM Unlearning

    cs.AI 2025-08 conditional novelty 6.0 of 10

    DCUE evaluates LLM unlearning by comparing core-token confidence score distributions of the unlearned model and the original model, corrected by a validation set, using the Kolmogorov-Smirnov test.

  12. Learning-Time Encoding Shapes Unlearning in LLMs

    cs.CL 2025-06 conditional novelty 6.0 of 10

    How knowledge is encoded during LLM fine-tuning strongly affects later unlearning: paraphrased training data helps unlearning, while entangled chunks hinder selective forgetting.

  13. Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputs

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Unlearning in LLMs leaves detectable 'fingerprints' that let a simple classifier distinguish an unlearned model from its original, even on unrelated prompts.

  14. LLM Unlearning Should Be Form-Independent

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Existing LLM unlearning is form-dependent; the new ORT benchmark measures this, and the training-free ROCR edit reduces it by redirecting concept representations.

  15. Existing Large Language Model Unlearning Evaluations Are Inconclusive

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Existing LLM unlearning evaluations are inconclusive: they can inject new information, depend heavily on task format, and rely on spurious correlations.

  16. Soft Weighted Machine Unlearning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Soft-weighted unlearning replaces binary data removal with per-sample weights from a convex quadratic program, improving fairness and robustness gains while preserving utility.

  17. DUSK: Do Not Unlearn Shared Knowledge

    cs.CL 2025-05 conditional novelty 6.0 of 10

    DUSK benchmarks machine unlearning under overlapping forget and retain documents, showing existing methods remove surface text but fail to preserve shared knowledge while erasing unique content.

  18. GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detection

    cs.CL 2025-05 conditional novelty 6.0 of 10

    GUARD performs inference-time unlearning by classifying prompts, retrieving original answers, and penalizing token matches during beam search, preserving utility but with forget quality that collapses on larger TOFU f...

  19. Unilogit: Robust Machine Unlearning for LLMs Using Uniform-Target Self-Distillation

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Unilogit unlearns text from LLMs by setting the target token probability to 1/|V| and distilling from the current model, reporting stronger forgetting at comparable retained utility than NPO and UnDIAL.

  20. A mean teacher algorithm for unlearning of language models

    cs.LG 2025-04 conditional novelty 6.0 of 10

    A mean teacher optimizer that approximates slow natural gradient descent, paired with a new negative log-unlikelihood loss, reduces memorization and privacy leakage on MUSE-News and MUSE-Books, with the strongest vari...

  21. Position: Adversarial ML for LLMs Is Not Making Any Progress

    cs.LG 2025-02 conditional novelty 6.0 of 10

    The authors argue that LLM-era adversarial machine learning is less well-defined, harder to solve, and harder to evaluate, so meaningful progress may not be achievable or trackable in the current paradigm.

  22. Investigating the Feasibility of Mitigating Potential Copyright Infringement via Large Language Model Unlearning

    cs.CL 2024-12 conditional novelty 6.0 of 10

    SSU combines task-vector negation, random-label loss, and weight saliency to forget copyrighted books sequentially while retaining more general language ability than existing baselines.

  23. SEUF: Is Unlearning One Expert Enough for Mixture-of-Experts LLMs?

    cs.LG 2024-11 conditional novelty 6.0 of 10

    SEUF restricts MoE LLM unlearning to the top-1 most relevant expert and adds a router anchor loss, improving utility by up to 35% while keeping forget efficacy roughly comparable.

  24. GROM: Gradient-Free Rapid One-Shot Machine Unlearning

    cs.LG 2026-08 conditional novelty 5.0 of 10

    A single closed-form ridge update to selected MLP layers removes targeted knowledge from LLMs in seconds, with state-of-the-art forgetting-utility trade-offs and quantization robustness.

  25. Representation-Aware Unlearning via Activation Signatures: From Suppression to Entity-Signature Erasure

    cs.CL 2026-01 reject novelty 5.0 of 10

    KIF reports near-oracle entity unlearning on TOFU (FQ=0.99, MU=0.62) by suppressing subject-specific activation directions and distilling the behavior into LoRA, but its 'internal attenuation' evidence rests on an out...

  26. Reliable Unlearning Harmful Information in LLMs with Metamorphosis Representation Projection

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    An LLM unlearning method that projects hidden states so harmful information is irreversibly removed while useful knowledge is preserved.

  27. Reviving Your MNEME: Predicting The Side Effects of LLM Unlearning and Fine-Tuning via Sparse Model Diffing

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Sparse model diffing on task-agnostic text can flag which academic, safety, or behavioral categories a fine-tuned or unlearned LLM has unintentionally changed.

  28. SoK: Machine Unlearning for Large Language Models

    cs.LG 2025-06 conditional novelty 5.0 of 10

    A new taxonomy for LLM unlearning distinguishes removal-intended from suppression-intended methods, and argues that gradient ascent methods functionally behave like suppression.

  29. Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearning

    cs.CL 2025-06 reject novelty 5.0 of 10

    SU uses two assistant models trained on different data splits to score tokens, then unlearns only tokens whose scores diverge, claiming better retain-set utility with comparable forget quality.

  30. BLUR: A Benchmark for LLM Unlearning Robust to Forget-Retain Overlap

    cs.LG 2025-05 conditional novelty 5.0 of 10

    BLUR adds forget-retain overlapping combined queries and relearning data to four unlearning benchmarks, and shows existing unlearning methods degrade on them.

  31. Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning

    cs.LG 2025-05 conditional novelty 5.0 of 10

    The authors propose SatImp, a product of a saturation weight and an importance weight, and show it improves the unlearn-retain trade-off on TOFU, WMDP, and MUSE.

  32. Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond

    cs.LG 2025-02 conditional novelty 5.0 of 10

    Adding sharpness-aware minimization to NPO unlearning slows relearning attacks on WMDP and MUSE benchmarks.

  33. Forget Vectors at Play: Universal Input Perturbations Driving Machine Unlearning in Image Classification

    cs.LG 2024-12 reject novelty 5.0 of 10

    A single optimized input perturbation can make a fixed image classifier misclassify targeted classes, mimicking unlearning without any weight update.

  34. Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning

    cs.LG 2024-12 conditional novelty 5.0 of 10

    Standard random-data LLM unlearning evaluations understate privacy leakage for minority data, as shown by canary and real rare-PII experiments across three datasets and two models.

  35. LLM Unlearning for Cyber Defense: A Survey on Methods, Challenges, and Emerging Threats

    cs.LG 2026-06 conditional novelty 4.0 of 10

    Most gradient-based LLM unlearning methods achieve behavioral suppression, not true forgetting, and current benchmarks cannot certify that knowledge has been removed.

  36. Generative Model Unlearning: A Survey through Target Events, Unlearning Operators, and Evaluation Protocols

    cs.LG 2025-07 conditional novelty 4.0 of 10

    A survey and framework that categorizes generative model unlearning by point-wise versus concept-wise objectives, parameter-based versus non-parametric methods, and completeness/utility/efficiency evaluation.

  37. Lifting Data-Tracing Machine Unlearning to Knowledge-Tracing for Foundation Models

    cs.CV 2025-06 conditional novelty 4.0 of 10

    A position paper urging a shift from data-tracing to knowledge-tracing machine unlearning for foundation models, supported by a CLIP case study that shows current methods struggle to generalize.

  38. UCD: Unlearning in LLMs via Contrastive Decoding

    cs.CL 2025-06 conditional novelty 4.0 of 10

    UCD steers an LLM away from forget-set content at inference time by mixing in the difference between forget-tuned and retain-tuned small models.

  39. Open Problems in Machine Unlearning for AI Safety

    cs.LG 2025-01 conditional novelty 4.0 of 10

    Machine unlearning is not a comprehensive solution for AI safety; it is best suited to data removal, while capability control faces fundamental and unresolved challenges.

Pith tools