Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Masks over attention heads and feed-forward filters, chosen by a Taylor-derived importance score, let Transformer unlearning freeze most parameters while improving both forgetting and fidelity.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

MAPE-Unlearn uses Fisher-information-based scores and greedy search to select important heads and filters, then applies sparse unlearning updates, claiming improved efficacy-fidelity trade-offs on Transformers.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Practical module-level unlearning heuristic, but the theoretical derivation doesn't survive scrutiny - still worth refereeing for its experimental breadth. the 4 major comments →

arxiv 2508.17233 v1 pith:HYQQIFOC submitted 2025-08-24 cs.LG cs.AI

Module-Aware Parameter-Efficient Machine Unlearning on Transformers

Wenjie Bao , Jian Lou , Yuke Hu , Xiaochen Li , Zhihao Liu , Jiaqi Liu , Zhan Qin , Kui Ren This is my paper
classification cs.LG cs.AI
keywords machine unlearningparameter-efficient unlearningTransformerattention headsfeed-forward filtersbinary masksFisher informationrelearning attacks
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Machine unlearning usually means fine-tuning a trained model so it no longer behaves as if it saw specific data. For Transformers, most parameter-efficient methods pick important weights heuristically, and that misses the way behavior is organized into attention heads and feed-forward filters. MAPE-Unlearn instead learns two binary masks, one over heads and one over filters, using a Taylor expansion of the unlearning objective; the resulting importance score combines the forget-set gradient with a diagonal Fisher information term, refined by a layer-wise greedy swap. Applied to second-order, gradient ascent, gradient difference, and negative preference optimization unlearning, the masked updates reach comparable or better forgetting with about 90% of modules frozen, and they delay degradation in successive unlearning and resist relearning attacks. A reader should care because this is a concrete path to cheaper, less destructive data removal in large language models.

Core claim

The paper's central claim is that the right unit for locating what must change during unlearning is not the individual weight but the module: an attention head or a feed-forward filter. It models the search for these units as a constrained mask-optimization problem, freeze most modules and update the rest, and derives the mask objective by expanding the unlearning loss around the all-ones mask. With the original model assumed to be at a local minimum, the first-order term becomes the negative forget-set gradient, and the Hessian is approximated by a block-diagonal Fisher information matrix; the result is an importance score per module, refined by a greedy swap that accounts for intra-layer i

What carries the argument

The load-bearing object is a pair of learnable binary masks over attention heads and feed-forward filters, with mask value 1 meaning update this module and 0 meaning freeze it. The derivation assigns each module an importance score: the forget-set gradient with respect to the mask plus half the diagonal Fisher information on the retain set, obtained from a second-order Taylor expansion of the unlearning objective under a sparsity constraint. A warm-start greedy search then swaps selected and unselected modules within each layer to capture intra-layer interactions. This same mechanism carries the argument for all four unlearning variants.

Load-bearing premise

The derivation assumes the trained model sits at a local minimum of the loss with respect to the head and filter scaling variables themselves; if that gradient is not near zero, the Taylor-based importance score is not a valid approximation and the method becomes a heuristic.

What would settle it

Compute the actual gradient of the retain-loss with respect to the mask variables at a converged Transformer and check whether it is near zero; if it is not, the claimed Taylor derivation of the importance score collapses. A second test: re-run mask selection using the exact layer-wise Hessian instead of the diagonal Fisher approximation and see whether the selected modules survive unchanged.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • At 90% sparsity, existing unlearning methods can achieve comparable or better forgetting while updating far fewer parameters, lowering the computational cost of Hessian-based unlearning.
  • Confining second-order approximation errors to a small set of modules lets the model handle more successive removal requests before retraining becomes necessary.
  • Freezing most modules disrupts the pathways that relearning attacks exploit, so masked unlearning resists longer attack durations than full-parameter updates for GA, GD, and NPO.
  • The masked methods preserve retain and test performance and reduce membership-inference attack success relative to their full-update counterparts.
  • The same mask-selection mechanism transfers across classification, question answering, fictitious-knowledge unlearning, and hazardous-knowledge removal tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the robustness to relearning attacks may be a side effect of freezing most weights rather than of the masks' quality; comparing against random masks at the same 90% sparsity would separate those two explanations.
  • Editorial inference: the mask-selection recipe depends only on a per-module saliency signal and an unlearning objective, so it plausibly transfers to other modular architectures such as vision transformers or mixture-of-experts models.
  • Editorial inference: because the method leaves most parameters untouched, it should compose naturally with certified-removal or privacy-accounting methods that require bounded parameter changes, though the paper does not test this.
  • Editorial inference: on small forget sets the paper's own TOFU results show the masked method underperforms full updates, suggesting the failure boundary is when the forget set is too small to give a reliable gradient signal for mask selection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MAPE-Unlearn, a module-level parameter-efficient unlearning method for Transformers. It introduces binary masks over attention heads and feed-forward filters, with importance scores derived from a second-order Taylor expansion of the unlearning objective (minimizing retain loss or maximizing forget loss). The masks are then applied to existing unlearning methods (SO, GA, GD, NPO, DPO) to perform sparse updates. The method is evaluated on GLUE/SQuAD, TOFU, and WMDP benchmarks using BERT-base, RoBERTa-large, Llama2-7b-chat, and Zephyr-7B-beta, with claims of improved unlearning efficacy, preserved fidelity, robustness to successive unlearning, and resistance to relearning attacks.

Significance. If the empirical claims hold under rigorous comparison, the module-aware sparsification idea is practically valuable: it targets a small number of Transformer modules rather than individual parameters, integrates with several unlearning objectives, and is evaluated on a broad range of models and tasks. The successive-unlearning and relearning-attack experiments are useful and go beyond standard single-request evaluation. However, the theoretical derivation of the importance score is invalid as stated, and the experimental protocol does not currently support the strength of the claims. The core approach may still work as a heuristic, but the paper's novelty relies on the claimed derivation, so this needs substantial revision.

major comments (4)
  1. [Sec. 4.1, Eq. (5)-(8)] The step from Eq. (5) to Eq. (6) uses the assumption ∇_m L(1; θ*, D)=0. This is not justified: m is a scaling on module outputs (heads/filters), not a parameter over which θ* was optimized. For a mask m_h scaling head output o_h(x), ∂L/∂m_h = Σ_x ∇_{o_h}ℓ(θ*,x)·o_h(x), which is generically nonzero at a parameter minimum. The same invalid step is reused in Appendix B.1, Eq. (16)-(17). Thus Eq. (6)/(17) are not valid quadratic approximations of the mask objective, and the importance scores are heuristics. The paper should either state and justify an additional assumption, or reframe the contribution as heuristic and remove the 'theoretically derives' claim.
  2. [App. D.2 / Sec. 5] Hyperparameters are chosen separately for the full-update and MAPE variants, often with different, wider ranges (e.g., TOFU GA: full [2e-6,2e-5] vs MAPE [1e-5,1e-4]; WMDP GA: 1.5e-7 vs 6e-7). All reported tables are single-run numbers with no error bars or seed-averaged statistics. With 128-sample forget sets and discrete mask selection, variance is likely non-negligible. The reported improvements could therefore be due to more favorable hyperparameter search rather than to the module-aware masks. Please report mean±std over at least 3 seeds, fix a common validation protocol, or ablate hyperparameter strength.
  3. [Sec. 5.3 / Tables 8-9] The abstract and Sec. 5.2 claim that MAPE-Unlearn 'consistently outperforms counterparts in unlearning efficacy,' but the data show mixed results. For example, TOFU Forget10 (Table 9) reports FQ = 7.28e-9 for GA vs 8.78e-12 for MAPE-GA, and FQ = 0.02 for DPO vs 3.11e-3 for MAPE-DPO; in both cases the MAPE variant has worse forget quality. The paper itself notes in Sec. 5.3 that gains are absent for GA and that small forget sets underperform full updates. Please soften the claim and provide a rigorous comparison of wins/ties/losses across all table entries.
  4. [Sec. 4.1 / App. B.2] The core method is under-specified. The paper does not define how the mask variables multiply module outputs (hard binary vs. soft scaling), how ∇_m ℓ is computed, how the diagonal/block-diagonal FIM is formed for mask variables, or how many swap iterations the greedy search performs and at what cost. No pseudo-code or algorithm box is provided. Without these details, the method is not reproducible from the manuscript. Please add a precise algorithmic description, including the exact objective optimized in Eq. (9) and the termination criterion.
minor comments (5)
  1. [Sec. 5.2] The text states that MAPE-Unlearn is 'evidenced by higher F1 scores,' but in Tables 2, 4-7, a lower forget-set F1/accuracy indicates stronger unlearning. The metric direction is inverted; please correct the wording.
  2. [App. B.2] Typo: 'MAPEcific neurons' should be 'MAPE-specific neurons.'
  3. [App. A.2, Eq. (12)] Equation (12) is malformed: it displays only 'mt ◦ { ... }' without the left-hand side 'θ_t ≈ θ* + ...'. Please fix the equation.
  4. [Fig. 2 / Sec. 5.2] The discussion says 'when sparsity surpasses 90%, SO experiences a sharp decline,' but the x-axis in Figure 2 appears to end at 100 and the curve behavior at exact 90/100 is not clear. Consider marking the 90% point and clarifying whether the decline is before or after 90%.
  5. [Sec. 4.1, citation [26]] The zero-gradient assumption is attributed to Optimal Brain Damage, but that result applies to trained weight parameters, not to output-scaling masks. This citation should be removed or replaced with an explicit justification.

Circularity Check

0 steps flagged

No significant circularity; the derivation's zero-gradient step is a correctness concern, not a circular one.

full rationale

MAPE-Unlearn's mask importance scores are computed from the same forget/retain losses and FIM that define the unlearning objectives, so the mask selection is aligned with the update objective rather than an independent prediction. However, this alignment is the method's design, not a circular reduction: the paper does not fit a parameter to a target metric and then report that metric as a predicted result; efficacy and fidelity are measured on held-out benchmarks and compared against baselines (SURE, SA, GA, GD, NPO, DPO, SO, retraining). No load-bearing step is justified only by a self-citation: [19], [30], and [31] are contextual (scenario definitions and prior second-order work) and are supported by multiple external citations. The main mathematical weakness is the assumption in §4.1 that ∇_m L(1;θ*,D)=0 for output-scaling mask variables; this is generally false at a parameter minimum and undermines the claim that Eq. (6)/(16) are valid quadratic approximations. That is a correctness/rigor issue, not circularity, and it does not make the empirical comparisons circular.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

The central method depends on a sparsity hyperparameter (S=90%) and on two modeling assumptions about the validity of the Taylor expansion and the block-diagonal FIM. No new physical or conceptual entities are introduced.

free parameters (1)
  • S (sparsity) = 90%
    The sparsity level that freezes the fraction of modules is set to 90% in all main comparisons; Figure 2 shows optional sparsity differs by method (e.g., SO is optimal near 70%), so this choice is a tuned hyperparameter.
axioms (3)
  • standard math Taylor expansion of the mask-loss and diagonal/block FIM approximation to the Hessian
    Used in Eq. (5)-(9) to convert the L0-constrained mask optimization into a score-based ranking.
  • domain assumption ∇_m L(1; θ*, D) = 0 because the original model is a local minimum
    Invoked in Section 4.1 to drop the first-order term and replace it with the forget-set gradient; not generally true for mask variables.
  • domain assumption Block diagonal FIM per layer captures intra-layer interactions enough for greedy swap to improve the mask
    Invoked in Section 4.1 to justify the layer-wise decomposition and the warm-start greedy search.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Module-Aware Parameter-Efficient Machine Unlearning on Transformers." pith.science (2026). https://pith.science/paper/HYQQIFOC

@misc{pith2026250817233,
  author       = {Pith},
  title        = {Pith review of: Module-Aware Parameter-Efficient Machine Unlearning on Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HYQQIFOC}},
  note         = {Machine review of arXiv:2508.17233}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Transformer has become fundamental to a vast series of pre-trained large models that have achieved remarkable success across diverse applications. Machine unlearning, which focuses on efficiently removing specific data influences to comply with privacy regulations, shows promise in restricting updates to influence-critical parameters. However, existing parameter-efficient unlearning methods are largely devised in a module-oblivious manner, which tends to inaccurately identify these parameters and leads to inferior unlearning performance for Transformers. In this paper, we propose {\tt MAPE-Unlearn}, a module-aware parameter-efficient machine unlearning approach that uses a learnable pair of masks to pinpoint influence-critical parameters in the heads and filters of Transformers. The learning objective of these masks is derived by desiderata of unlearning and optimized through an efficient algorithm featured by a greedy search with a warm start. Extensive experiments on various Transformer models and datasets demonstrate the effectiveness and robustness of {\tt MAPE-Unlearn} for unlearning.

Figures

Figures reproduced from arXiv: 2508.17233 by Jian Lou, Jiaqi Liu, Kui Ren, Wenjie Bao, Xiaochen LI, Yuke Hu, Zhan Qin, Zhihao Liu.

Figure 1
Figure 1. Figure 1: Illustration of our method applied to obtain important heads. Starting with the original [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: F1 scores of unlearning methods on BERT-base at different sparsity levels. Module-Aware sparse unlearning is effective [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Results on BERT-base under different unlearning scenarios with varying removal requests. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Forget quality for different unlearning methods with varying relearning epochs on [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance comparison of unlearning methods on WMDP under Zephyr-7B-beta. Method Efficacy Fidelity AccBio. ↓ AccCyber. ↓ Avg. ↓ MMLU ↑ Original 0.6465 0.4449 0.5457 0.5845 GA 0.2679 0.3301 0.2990 0.4083 SURE-GA 0.2569 0.3296 0.2933 0.4030 MAPE-GA 0.2726 0.3191 0.2959 0.4145 GD 0.3370 0.3709 0.3540 0.4529 SURE-GD 0.3346 0.3749 0.3548 0.4603 MAPE-GD 0.3236 0.3629 0.3433 0.4557 NPO 0.4540 0.4051 0.4396 0.490… view at source ↗
Figure 6
Figure 6. Figure 6: Overall results of unlearning performance are [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: F1 scores for various sparsity applied to BERT-base after three kinds of unlearning strategies [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. TOUR: A Trajectory-Level Unlearning Benchmark for Offline Reinforcement Learning

    cs.LG 2026-07 conditional novelty 6.0

    In offline RL, evaluating trajectory deletion with a single membership score is unreliable: the same deletion method can look private under one attack and leak under another.

Reference graph

Works this paper leans on

60 extracted references · 20 canonical work pages · cited by 1 Pith paper · 4 internal anchors

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Fisher information and natural gradient learning in random deep networks

    Shun-ichi Amari, Ryo Karakida, and Masafumi Oizumi. Fisher information and natural gradient learning in random deep networks. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 694–702. PMLR, 2019

    work page 2019

  3. [3]

    Machine unlearning

    Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141–159. IEEE, 2021

    2021

  4. [4]

    Towards making systems forget with machine unlearning

    Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pages 463–480. IEEE, 2015

    2015

  5. [5]

    Machine unlearning in large language models

    Kongyang Chen, Zixin Wang, Bing Mi, Waixi Liu, Shaowei Wang, Xiaojun Ren, and Jiaxing Shen. Machine unlearning in large language models. arXiv preprint arXiv:2404.16841, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    Pith/arXiv arXiv 2018

  7. [7]

    Does unlearning truly unlearn? a black box evaluation of llm unlearning methods

    Jai Doshi and Asa Cooper Stickland. Does unlearning truly unlearn? a black box evaluation of llm unlearning methods. arXiv preprint arXiv:2411.12103, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Who’s harry potter? approximate unlearning in llms

    Ronen Eldan and Mark Russinovich. Who’s harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Fast machine unlearning without retraining through selective synaptic dampening

    Jack Foster, Stefan Schoepf, and Alexandra Brintrup. Fast machine unlearning without retraining through selective synaptic dampening. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 12043–12051, 2024

    2024

  10. [10]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018

    Pith/arXiv arXiv 2018

  11. [11]

    Eternal sunshine of the spotless net: Selective forgetting in deep networks

    Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9304–9312, 2020

    2020

  12. [12]

    Second-order infor- mation matters: Revisiting machine unlearning for large language models

    Kang Gu, Md Rafi Ur Rashid, Najrin Sultana, and Shagufta Mehnaz. Second-order infor- mation matters: Revisiting machine unlearning for large language models. arXiv preprint arXiv:2403.10557, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Certified data removal from machine learning models

    Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. In International Conference on Machine Learning, pages 3832–3842. PMLR, 2020

    work page 2020

  14. [14]

    Visualizing and Understanding the Effectiveness of BERT

    Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Visualizing and understanding the effectiveness of bert. arXiv preprint arXiv:1908.05620, 2019. 10

    work page internal anchor Pith review Pith/arXiv arXiv 1908

  15. [15]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    Pith/arXiv arXiv 2009

  16. [16]

    The euro- pean union general data protection regulation: what it is and what it means

    Chris Jay Hoofnagle, Bart Van Der Sloot, and Frederik Zuiderveen Borgesius. The euro- pean union general data protection regulation: what it is and what it means. Information & Communications Technology Law, 28(1):65–98, 2019

    work page 2019

  17. [17]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    Pith/arXiv arXiv 2021

  18. [18]

    Jogging the memory of unlearned llms through targeted relearning attacks

    Shengyuan Hu, Yiwei Fu, Steven Wu, and Virginia Smith. Jogging the memory of unlearned llms through targeted relearning attacks. In Neurips Safe Generative AI Workshop 2024, 2024

    work page 2024

  19. [19]

    Eraser: Machine unlearning in mlaas via an inference serving-aware approach

    Yuke Hu, Jian Lou, Jiaqi Liu, Feng Lin, Zhan Qin, and Kui Ren. Eraser: Machine unlearning in mlaas via an inference serving-aware approach. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS ’24, page 3883–3897, 2024

    work page 2024

  20. [20]

    Fadam: Adam is a natural gradient optimizer using diagonal empirical fisher information

    Dongseong Hwang. Fadam: Adam is a natural gradient optimizer using diagonal empirical fisher information. arXiv preprint arXiv:2405.12807, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Approximate data deletion from machine learning models

    Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pages 2008–2016. PMLR, 2021

    work page 2008

  22. [22]

    Knowledge unlearning for mitigating privacy risks in language models

    Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint arXiv:2210.01504, 2022

    Pith/arXiv arXiv 2022

  23. [23]

    Soul: Unlocking the power of second-order optimization for llm unlearning

    Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. Soul: Unlocking the power of second-order optimization for llm unlearning. arXiv preprint arXiv:2404.18239, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Understanding black-box predictions via influence functions

    Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885–1894. PMLR, 2017

    2017

  25. [25]

    A fast post-training pruning framework for transformers

    Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. A fast post-training pruning framework for transformers. Advances in Neural Information Processing Systems, 35:24101–24116, 2022

    work page 2022

  26. [26]

    Optimal brain damage

    Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989

    1989

  27. [27]

    The wmdp benchmark: Measuring and reducing malicious use with unlearning

    Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Continual learning and private unlearning

    Bo Liu, Qiang Liu, and Peter Stone. Continual learning and private unlearning. In Conference on Lifelong Learning Agents, pages 243–254. PMLR, 2022

    work page 2022

  29. [29]

    Model sparsity can simplify machine unlearning

    Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, PRANAY SHARMA, Sijia Liu, et al. Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  30. [30]

    Certified minimax unlearning with generalization rates and deletion capacity

    Jiaqi Liu, Jian Lou, Zhan Qin, and Kui Ren. Certified minimax unlearning with generalization rates and deletion capacity. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  31. [31]

    Muter: Machine unlearning on adversarially trained models

    Junxu Liu, Mingsheng Xue, Jian Lou, Xiaoyu Zhang, Li Xiong, and Zhan Qin. Muter: Machine unlearning on adversarially trained models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4892–4902, 2023

    work page 2023

  32. [32]

    Rethinking machine unlearning for large language models

    Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. Rethinking machine unlearning for large language models. arXiv preprint arXiv:2402.08787, 2024. 11

    Pith/arXiv arXiv 2024

  33. [33]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    Pith/arXiv arXiv 1907

  34. [34]

    Unlearning with fisher masking

    Yufang Liu, Changzhi Sun, Yuanbin Wu, and Aimin Zhou. Unlearning with fisher masking. arXiv preprint arXiv:2310.05331, 2023

    Pith/arXiv arXiv 2023

  35. [35]

    An adversarial perspective on machine unlearning for ai safety, 2024

    Jakub Łucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tramer, and Javier Rando. An adversarial perspective on machine unlearning for ai safety, 2024. URL https://arxiv. org/abs/2409.18025, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Eight methods to evaluate robust unlearning in llms

    Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402.16835, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Learn to forget: Machine unlearning via neuron masking

    Zhuo Ma, Yang Liu, Ximeng Liu, Jian Liu, Jianfeng Ma, and Kui Ren. Learn to forget: Machine unlearning via neuron masking. IEEE Transactions on Dependable and Secure Computing, 20 (4):3194–3207, 2022

    work page 2022

  38. [38]

    Tofu: A task of fictitious unlearning for llms

    Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. Tofu: A task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06121, 2024

    Pith/arXiv arXiv 2024

  39. [39]

    Ssse: Efficiently erasing samples from trained machine learning models

    Alexandra Peste, Dan Alistarh, and Christoph H Lampert. Ssse: Efficiently erasing samples from trained machine learning models. arXiv preprint arXiv:2107.03860, 2021

    Pith/arXiv arXiv 2021

  40. [40]

    Dissecting language models: Machine unlearning via selective pruning

    Nicholas Pochinkov and Nandi Schoots. Dissecting language models: Machine unlearning via selective pruning. arXiv preprint arXiv:2403.01267, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  42. [42]

    Squad: 100,000+ questions for machine comprehension of text

    P Rajpurkar. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016

    Pith/arXiv arXiv 2016

  43. [43]

    Parameter-tuning-free data entry error unlearning with adaptive selective synaptic dampening

    Stefan Schoepf, Jack Foster, and Alexandra Brintrup. Parameter-tuning-free data entry error unlearning with adaptive selective synaptic dampening. arXiv preprint arXiv:2402.10098, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  44. [44]

    DeepClean: Machine Unlearning on the Cheap by Resetting Privacy Sensitive Weights using the Fisher Diagonal

    Jiaeli Shi, Najah Ghalyan, Kostis Gourgoulias, John Buford, and Sean Moran. Deepclean: Machine unlearning on the cheap by resetting privacy sensitive weights using the fisher diagonal. arXiv preprint arXiv:2311.10448, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  45. [45]

    Privacy risks of securing machine learning models against adversarial examples

    Liwei Song, Reza Shokri, and Prateek Mittal. Privacy risks of securing machine learning models against adversarial examples. In Proceedings of the 2019 ACM SIGSAC conference on computer and communications security, pages 241–257, 2019

    work page 2019

  46. [46]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023

    Pith/arXiv arXiv 2023

  47. [47]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  48. [48]

    Zephyr: Direct distillation of lm alignment

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944, 2023

    Pith/arXiv arXiv 2023

  49. [49]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  50. [50]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018. 12

    Pith/arXiv arXiv 2018

  51. [51]

    Machine un- learning of features and labels

    Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine un- learning of features and labels. arXiv preprint arXiv:2108.11577, 2021

    Pith/arXiv arXiv 2021

  52. [52]

    Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning

    Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning. Advances in Neural Information Processing Systems, 34:16158–16170, 2021

    work page 2021

  53. [53]

    Scissorhands: Scrub data influence via connection sensitivity in networks

    Jing Wu and Mehrtash Harandi. Scissorhands: Scrub data influence via connection sensitivity in networks. arXiv preprint arXiv:2401.06187, 2024

    Pith/arXiv arXiv 2024

  54. [54]

    Machine unlearning of pre-trained large language models

    Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. Machine unlearning of pre-trained large language models. arXiv preprint arXiv:2402.15159, 2024

    Pith/arXiv arXiv 2024

  55. [55]

    Large language model unlearning.arXiv preprint arXiv:2310.10683, 2023

    Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning.arXiv preprint arXiv:2310.10683, 2023

    Pith/arXiv arXiv 2023

  56. [56]

    Negative preference optimization: From catastrophic collapse to effective unlearning

    Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868, 2024

    Pith/arXiv arXiv 2024

  57. [57]

    Unveiling Linguistic Regions in Large Language Models

    Zhihao Zhang, Jun Zhao, Qi Zhang, Tao Gui, and Xuanjing Huang. Unveiling linguistic regions in large language models. arXiv preprint arXiv:2402.14700, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  58. [58]

    Catastrophic failure of llm unlearning via quantization

    Zhiwei Zhang, Fali Wang, Xiaomin Li, Zongyu Wu, Xianfeng Tang, Hui Liu, Qi He, Wenpeng Yin, and Suhang Wang. Catastrophic failure of llm unlearning via quantization. arXiv preprint arXiv:2410.16454, 2024. 13 A Additional Details of Second-Order Unlearning Update A.1 Simplified Second-Order Unlearning Update Second-order unlearning involves the inverse Hes...

    Pith/arXiv arXiv 2024

  59. [59]

    Unlike previous methods, this approach directly employs a Taylor expansion around the original model parameters

    introduced a successive unlearning scenario leveraging second-order updates, requiring the retention of data information (e.g., gradients and FIM) for efficient unlearning on the original model. Unlike previous methods, this approach directly employs a Taylor expansion around the original model parameters. Specifically, at the t-th unlearning request, MAP...

    work page

  60. [60]

    To further investigate this claim, we conducted a comparative analysis of three selective parameter update strategies: heads-only, filters- only, and heads&filters in Figure 7

    argued that pruning filters are more effective than pruning heads. To further investigate this claim, we conducted a comparative analysis of three selective parameter update strategies: heads-only, filters- only, and heads&filters in Figure 7. All experiments were designed to provide comparable unlearning guarantees varying sparsity. The heads-only strate...

    work page

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.