REVIEW 7 cited by
Large Scale Knowledge Washing
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large language models show impressive abilities in memorizing world knowledge, which leads to concerns regarding memorization of private information, toxic or sensitive knowledge, and copyrighted content. We introduce the problem of Large Scale Knowledge Washing, focusing on unlearning an extensive amount of factual knowledge. Previous unlearning methods usually define the reverse loss and update the model via backpropagation, which may affect the model's fluency and reasoning ability or even destroy the model due to extensive training with the reverse loss. Existing works introduce additional data from downstream tasks to prevent the model from losing capabilities, which requires downstream task awareness. Controlling the tradeoff of unlearning and maintaining existing capabilities is also challenging. To this end, we propose LAW (Large Scale Washing) to update the MLP layers in decoder-only large language models to perform knowledge washing, as inspired by model editing methods and based on the hypothesis that knowledge and reasoning are disentanglable. We derive a new objective with the knowledge to be unlearned to update the weights of certain MLP layers. Experimental results demonstrate the effectiveness of LAW in forgetting target knowledge while maintaining reasoning ability. The code will be open-sourced at https://github.com/wangyu-ustc/LargeScaleWashing.
Forward citations
Cited by 7 Pith papers
-
Learning-Time Encoding Shapes Unlearning in LLMs
How knowledge is encoded during LLM fine-tuning strongly affects later unlearning: paraphrased training data helps unlearning, while entangled chunks hinder selective forgetting.
-
SHA256 at SemEval-2025 Task 4: Selective Amnesia -- Constrained Unlearning for Large Language Models via Knowledge Isolation
A causal-tracing-guided method that freezes upper layers and retrains early MLP layers with a joint forget/retain loss achieved 2nd place in the SemEval-2025 Task 4 1B unlearning track.
-
SEUF: Is Unlearning One Expert Enough for Mixture-of-Experts LLMs?
SEUF restricts MoE LLM unlearning to the top-1 most relevant expert and adds a router anchor loss, improving utility by up to 35% while keeping forget efficacy roughly comparable.
-
Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks
A system-first taxonomy and literature synthesis of multimodal unlearning across vision, language, video, and audio, with datasets, benchmarks, metrics, applications, and open challenges.
-
SoK: Machine Unlearning for Large Language Models
A new taxonomy for LLM unlearning distinguishes removal-intended from suppression-intended methods, and argues that gradient ascent methods functionally behave like suppression.
-
UCD: Unlearning in LLMs via Contrastive Decoding
UCD steers an LLM away from forget-set content at inference time by mixing in the difference between forget-tuned and retain-tuned small models.
-
Open Problems in Machine Unlearning for AI Safety
Machine unlearning is not a comprehensive solution for AI safety; it is best suited to data removal, while capability control faces fundamental and unresolved challenges.
Discussion (0). Continue with ORCID to comment.