REVIEW 11 cited by
Machine Unlearning of Pre-trained Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Machine Unlearning of Pre-trained Large Language Models
read the original abstract
This study investigates the concept of the `right to be forgotten' within the context of large language models (LLMs). We explore machine unlearning as a pivotal solution, with a focus on pre-trained models--a notably under-researched area. Our research delineates a comprehensive framework for machine unlearning in pre-trained LLMs, encompassing a critical analysis of seven diverse unlearning methods. Through rigorous evaluation using curated datasets from arXiv, books, and GitHub, we establish a robust benchmark for unlearning performance, demonstrating that these methods are over $10^5$ times more computationally efficient than retraining. Our results show that integrating gradient ascent with gradient descent on in-distribution data improves hyperparameter robustness. We also provide detailed guidelines for efficient hyperparameter tuning in the unlearning process. Our findings advance the discourse on ethical AI practices, offering substantive insights into the mechanics of machine unlearning for pre-trained LLMs and underscoring the potential for responsible AI development.
Forward citations
Cited by 11 Pith papers
-
Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
NPO enables stable unlearning of 50%+ training data in LLMs on TOFU by making collapse exponentially slower than gradient ascent, preserving sensible outputs where prior methods fail.
-
REMEDI: A Benchmark for Retention and Unlearning Evaluation in Multi-label Clinical Disease Inference
REMEDI is a new benchmark for evaluating machine unlearning in multi-label clinical disease inference on MIMIC-III data that reveals trade-offs in existing methods.
-
POPS: Recovering Unlearned Multi-Modality Knowledge in MLLMs with Prompt-Optimized Parameter Shaking
Prompt-optimized suffixes plus synthetic fine-tuning recover ~82% of knowledge that multimodal unlearning methods claim to erase from MLLMs.
-
MLUBench: A Benchmark for Lifelong Unlearning Evaluation in MLLMs
MLUBench benchmarks lifelong unlearning in MLLMs, revealing severe cumulative degradation in baselines due to multimodal alignment constraints and introducing LUMoE to mitigate the issue.
-
ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models
ZeroUnlearn reformulates machine unlearning as knowledge re-mapping via model editing, using multiplicative updates with closed-form solutions for efficient few-shot removal of sensitive representations while preservi...
-
Representation-Guided Parameter-Efficient LLM Unlearning
REGLU guides LoRA-based unlearning via representation subspaces and orthogonal regularization to outperform prior methods on forget-retain trade-off in LLM benchmarks.
-
Mechanism-Guided Selective Unlearning for RLVR-Induced Reasoning
MAST ranks attention-projection tensors by off-principal energy, update magnitude, and forget-gradient coupling to selectively unlearn RLVR-induced reasoning, achieving significant forgetting on MATH while preserving ...
-
ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models
ZeroUnlearn is a few-shot unlearning method that overwrites sensitive inputs with neutral targets via closed-form multiplicative parameter updates enforcing representational orthogonality in LLMs.
-
ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models
ZeroUnlearn is a few-shot unlearning method that maps sensitive inputs to neutral states and enforces representational orthogonality through a closed-form multiplicative update, outperforming baselines while preservin...
-
Forgetting to Witness: Efficient Federated Unlearning and Its Visible Evaluation
A complete pipeline for federated unlearning via knowledge distillation for efficient removal and a GAN-integrated classifier for visual evaluation of forgetting capacity.
-
Machine Unlearning: A Comprehensive Survey
A survey classifying machine unlearning into centralized (exact and approximate), distributed/irregular data, verification, and privacy/security categories with technique overviews.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.