REVIEW 10 cited by
Knowledge Unlearning for LLMs: Tasks, Methods, and Challenges
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Knowledge Unlearning for LLMs: Tasks, Methods, and Challenges
read the original abstract
In recent years, large language models (LLMs) have spurred a new research paradigm in natural language processing. Despite their excellent capability in knowledge-based question answering and reasoning, their potential to retain faulty or even harmful knowledge poses risks of malicious application. The challenge of mitigating this issue and transforming these models into purer assistants is crucial for their widespread applicability. Unfortunately, Retraining LLMs repeatedly to eliminate undesirable knowledge is impractical due to their immense parameters. Knowledge unlearning, derived from analogous studies on machine unlearning, presents a promising avenue to address this concern and is notably advantageous in the context of LLMs. It allows for the removal of harmful knowledge in an efficient manner, without affecting unrelated knowledge in the model. To this end, we provide a survey of knowledge unlearning in the era of LLMs. Firstly, we formally define the knowledge unlearning problem and distinguish it from related works. Subsequently, we categorize existing knowledge unlearning methods into three classes: those based on parameter optimization, parameter merging, and in-context learning, and introduce details of these unlearning methods. We further present evaluation datasets used in existing methods, and finally conclude this survey by presenting the ongoing challenges and future directions.
Forward citations
Cited by 10 Pith papers
-
Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
NPO enables stable unlearning of 50%+ training data in LLMs on TOFU by making collapse exponentially slower than gradient ascent, preserving sensible outputs where prior methods fail.
-
ICU-Bench:Benchmarking Continual Unlearning in Multimodal Large Language Models
ICU-Bench is a new continual unlearning benchmark for MLLMs using 1000 privacy profiles, 9500 images, and 100 forget tasks, showing existing methods fail to balance forgetting, utility, and scalability.
-
Representation-Guided Parameter-Efficient LLM Unlearning
REGLU guides LoRA-based unlearning via representation subspaces and orthogonal regularization to outperform prior methods on forget-retain trade-off in LLM benchmarks.
-
Exclusive Unlearning
Exclusive Unlearning makes LLMs safe by forgetting all but retained domain knowledge, protecting against jailbreaks while preserving useful responses in areas like medicine and math.
-
Anatomy of Unlearning: The Dual Impact of Fact Salience and Model Fine-Tuning
Fact popularity and training stage jointly change how unlearning behaves: popular facts are harder to erase, and a model fine-tuned on the evaluation data forgets more smoothly and retains more than a pretrained model.
-
Anatomy of Unlearning: The Dual Impact of Fact Salience and Model Fine-Tuning
SFT models forget facts more stably than pretrained models, with 10-50% higher retention of unrelated knowledge when using the DUET benchmark of 28.6k Wikidata triplets.
-
Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearning
PALU shows that unlearning only needs local intervention—the first few tokens of the sensitive span and the top-k logits—not full-sequence, full-vocabulary suppression.
-
Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearning
PALU improves LLM unlearning by restricting entropy maximization to sensitive prefixes and top-k logits, achieving better forgetting with less utility loss.
-
Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning
An MLLM unlearning method and benchmark that aim to erase targeted private facts while preserving image understanding.
-
Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks
A system-first taxonomy and literature synthesis of multimodal unlearning across vision, language, video, and audio, with datasets, benchmarks, metrics, applications, and open challenges.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.