REVIEW 15 cited by
Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The LLM unlearning technique has recently been introduced to comply with data regulations and address the safety and ethical concerns of LLMs by removing the undesired data-model influence. However, state-of-the-art unlearning methods face a critical vulnerability: they are susceptible to ``relearning'' the removed information from a small number of forget data points, known as relearning attacks. In this paper, we systematically investigate how to make unlearned models robust against such attacks. For the first time, we establish a connection between robust unlearning and sharpness-aware minimization (SAM) through a unified robust optimization framework, in an analogy to adversarial training designed to defend against adversarial attacks. Our analysis for SAM reveals that smoothness optimization plays a pivotal role in mitigating relearning attacks. Thus, we further explore diverse smoothing strategies to enhance unlearning robustness. Extensive experiments on benchmark datasets, including WMDP and MUSE, demonstrate that SAM and other smoothness optimization approaches consistently improve the resistance of LLM unlearning to relearning attacks. Notably, smoothness-enhanced unlearning also helps defend against (input-level) jailbreaking attacks, broadening our proposal's impact in robustifying LLM unlearning. Codes are available at https://github.com/OPTML-Group/Unlearn-Smooth.
Forward citations
Cited by 15 Pith papers
-
Crossing the Margin Cliff: Toward Relearn-Robust LLM Unlearning via Margin Calibration
Margin Calibration, a non-saturating margin-anchored LoRA polish, crosses the margin cliff and cuts post-attack relearn recovery on all 97 populated cells in the paper's stress matrix.
-
TRACER: Token ReAssignment for Concept ERasure in Generative Recommendation
TRACER uses token reassignment for concept-related items plus a coherence regularizer to unlearn specific concepts in generative recommendation while preserving utility better than baselines.
-
Measuring the Depth of LLM Unlearning via Activation Patching
Introduces Unlearning Depth Score (UDS) via activation patching to quantify LLM unlearning depth and claims it outperforms 20 other metrics in faithfulness and robustness on 150 models.
-
One Step to the Side: Why Defenses Against Malicious Finetuning Fail Under Adaptive Adversaries
A unified adaptive attack exploits the common weakness across 15 defenses against malicious fine-tuning, showing they only obscure rather than remove harmful model capabilities.
-
Efficient Unlearning through Maximizing Relearning Convergence Delay
The Influence Eliminating Unlearning framework maximizes relearning convergence delay via weight decay and noise injection to remove the influence of a forgetting set while preserving accuracy on retained data.
-
CURE:Circuit-Aware Unlearning for LLM-based Recommendation
CURE disentangles LLM recommendation circuits into forget-specific, retain-specific, and task-shared modules with tailored update rules to achieve more effective unlearning than weighted baselines.
-
Understanding Machine Unlearning Through the Lens of Mode Connectivity
Unlearned models usually connect to their originals by smooth low-loss paths, and the smoothness of that path can predict how hard the unlearning task was.
-
Understanding Machine Unlearning Through the Lens of Mode Connectivity
Unlearned models often lie in connected low-loss basins, and that geometry can predict unlearning difficulty and improve robustness to relearning attacks.
-
A Mechanistic Perspective and Circuit-Guided Difficulty Metric for Unlearning
A circuit-similarity score predicts which samples an LLM unlearning method will fail to erase, with hard samples relying on deeper, output-facing pathways.
-
Leak@$k$: Unlearning Does Not Make LLMs Forget Under Probabilistic Decoding
LLM unlearning methods that pass greedy-decoding benchmarks leak forgotten facts when the model is sampled repeatedly, and the new leak@k metric quantifies this.
-
Downgrade to Upgrade: Optimizer Simplification Enhances Robustness in LLM Unlearning
Downgrading optimizers to lower-information variants during LLM unlearning yields more robust forgetting on MUSE and WMDP benchmarks by converging to harder-to-perturb loss basins.
-
BARRIER: Bounded Activation Regions for Robust Information Erasure
BARRIER applies interval arithmetic to SVD-based activation projections to create bounded forget regions that enable aggressive unlearning while providing formal protection for retain distributions via tail bounds on ...
-
LLM Unlearning for Cyber Defense: A Survey on Methods, Challenges, and Emerging Threats
Most gradient-based LLM unlearning methods achieve behavioral suppression, not true forgetting, and current benchmarks cannot certify that knowledge has been removed.
-
A Numerical PDEs Approach to Evolution Equations in Shape Analysis Based on Regularized Morphoelasticity
Regularized morphoelasticity yields a high-order elliptic system for continuous shape evolution that is solved by mixed finite elements in FEniCSx within an LDDMM-style optimal-control growth model.
-
A Survey on Generative Model Unlearning: Fundamentals, Taxonomy, Evaluation, and Future Direction
A survey and framework that categorizes generative model unlearning by point-wise versus concept-wise objectives, parameter-based versus non-parametric methods, and completeness/utility/efficiency evaluation.
Discussion (0). Sign in to comment.