Pith. sign in

REVIEW 4 major objections 4 minor 122 references

Current LLM unlearning methods suppress knowledge rather than truly remove it, leaving attackers able to recover it.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 10:20 UTC pith:LUKUMOT7

load-bearing objection A security-focused unlearning survey with a genuinely useful three-level framework; the bottom-line 'no true forgetting' claim is probably right but over-scoped relative to the evidence. the 4 major comments →

arxiv 2607.16227 v1 pith:LUKUMOT7 submitted 2026-06-23 cs.LG cs.CR

LLM Unlearning for Cyber Defense: A Survey on Methods, Challenges, and Emerging Threats

Ruppikha Sree Shankar , Abhishek Bhardwaj , Arnav Doshi , Anusri Nagarajan , Troy Paulus Asia , Saptarshi Sengupta This is my paper
classification cs.LG cs.CR
keywords LLM unlearningmachine unlearninggradient ascentbehavioral suppressiontrue forgettingadversarial robustnessmembership inferencecyber defense
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This survey argues that current LLM unlearning methods - especially gradient-based ones - do not actually make a model forget in the strong sense of matching a model retrained without the data. They typically achieve behavioral suppression: the model stops producing unwanted outputs under normal prompts, but the underlying representations remain. The survey introduces a three-level framework separating behavioral suppression, representation-level attenuation, and true forgetting, and classifies existing methods mostly at the first level. It shows through adversarial audits that supposedly forgotten knowledge can be recovered via fine-tuning, jailbreak prompts, or white-box probes, and that current benchmarks certify only output-level suppression, not removal. This matters because without true forgetting, unlearning cannot serve as a verifiable cyber defense or meet legal erasure obligations.

Core claim

On the paper's own terms, its central claim is that no family of LLM unlearning methods reviewed - gradient ascent/descent, influence-based, or constrained/localized editing - achieves true forgetting. Gradient ascent fails because it starts a new optimization trajectory instead of inverting the original training path, leaving latent representations largely intact. Influence methods model per-sample contributions but rely on local linearity assumptions that fail in non-convex networks. Constrained and localized editing methods assume knowledge is localizable, which holds only approximately. The consequence is that most methods sit at the behavioral-suppression tier, and even the strongest av

What carries the argument

The survey's central analytical object is a three-level framework distinguishing behavioral suppression (no unwanted output under standard prompts), representation-level attenuation (weakened but not removed internal encoding), and true forgetting (equivalence to retraining without the forget set). This framework doubles as a robustness ladder keyed to adversary access: black-box, gray-box, and white-box. The mechanism that explains the field-wide failure is that gradient ascent on the forget set is not the inverse of training: it introduces a new optimization path that changes output behavior without disturbing the distributed, entangled parameter-level representations that encode the knowl

Load-bearing premise

The survey's blanket conclusion that no method achieves true forgetting rests on generalizing from a small set of white-box audits and negative theoretical results to all gradient-difference, saliency, influence, orthogonality, low-rank, and constrained unlearning methods; if those audits characterize only certain baselines, or if the practical target is output-level behavior rather than equivalence to retraining without the data, the conclusion is too strong.

What would settle it

A concrete audit would settle the central claim: take a model unlearned by any leading method, give a white-box adversary the full weights and the unlearning procedure, and allow them to run orthogonalization, pruning, or embedding-space probes; then compare recovered forget-set accuracy and internal activations against a model retrained from scratch without the forget set. If any probe fails to raise forget-set accuracy above the retrained-reference level, and activation-level signals become indistinguishable from the retrained reference, the claim that every current method only suppresses wo

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If this is right, benchmark scores on TOFU, RWKU, WMDP, and MUSE should not be read as evidence of deletion; they certify suppression under fixed, non-adaptive prompts.
  • White-box robustness becomes the minimum security bar: any unlearning claim must hold against an adversary who knows the procedure and has access to weights, gradients, or activations.
  • Relearning attacks become a standard test, because fine-tuning an unlearned model on a handful of unrelated examples can restore hazardous capabilities; robust methods must flatten the loss landscape around the unlearned solution.
  • Regulatory compliance via unlearning is not yet demonstrable: current methods cannot certify parametric removal, and differential-privacy-based guarantees exist only for models trained with DP from the outset.
  • Evaluation must be redesigned to probe internal representations, not just output distributions, so that suppression and forgetting can be distinguished.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A practical extension of the paper's threat-model logic: for threats like casual copyrighted-text extraction, behavioral suppression may be a sufficient defense, so 'unlearning success' should be defined relative to the deployment's adversary class rather than the idealized retraining-equivalence standard.
  • If the white-box recovery findings generalize, open-weight models advertised as 'unlearned' are effectively not unlearned against a determined adversary; closed-API deployments with strict output filtering may have a lower practical exposure despite the same parametric retention.
  • A testable next step follows directly from the framework: build an evaluation that measures representation-level removal using activation-distance or linear-probe comparisons against a retrained reference model; this would either localize which methods achieve true attenuation or confirm that none do.
  • The paper hints at, but does not develop, a hybrid direction: combine causal tracing of where a fact is encoded with gradient-based removal targeted at those representational pathways, rather than output-level loss signals.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This survey reviews LLM unlearning through a cybersecurity lens, focusing predominantly on gradient-based methods. It introduces a three-level framework distinguishing behavioral suppression, representation-level attenuation, and true forgetting, and uses it to classify three families: gradient ascent/descent variants, influence-based methods, and constrained/parameter-localized approaches. The paper's central claim is that none of the reviewed families reliably achieves true forgetting in the sense of distributional equivalence to retraining without the forget set; current methods mostly achieve behavioral suppression that dissolves under gray- or white-box adversarial access. It supports this with cited adversarial audits (notably Lucki et al. on WMDP/RMU), theoretical results on the failure of gradient ascent, and a softmax 'squeezing effect,' concluding that evaluation protocols such as TOFU, RWKU, WMDP, and MUSE certify output-level suppression rather than representational removal. The survey also positions unlearning as a corrective security control, documents motivating legal cases, and identifies open challenges including adaptive evaluation and verifiable removal.

Significance. If the negative synthesis is accepted, the paper provides a useful corrective to benchmark-driven claims in LLM unlearning. Its three-level framework is a practical diagnostic, and the adversary taxonomy plus the mapping of threat categories to required defense levels (Table 2) are valuable for practitioners. The paper is honest about limitations of existing evaluation, and it collects recent theoretical and empirical negative results, including concrete numbers from [21] (e.g., residual-stream orthogonalization recovering WMDP-Biology accuracy to 64.7% versus a 64.4% baseline; pruning 0.9--2.4% of weights restoring hazardous knowledge). The literature collection is up to date through mid-2026. The main value is as a synthesis and warning rather than a new method; the central conclusion would be stronger if its scope were aligned with the direct evidence. The absence of machine-checked proofs is not a flaw for a survey, but the systematic-review protocol is incompletely reported.

major comments (4)
  1. [Abstract; §7.2; §6.4] The universal negative 'None of the families reviewed achieve true forgetting' is broader than the cited evidence. Direct support comes from Lucki et al. [21] on RMU/WMDP (§2.2, §6.4), the gradient-ascent non-reversibility theorem of Mavrothalassitis et al. [71] (§4.2.2), and the softmax squeezing argument (§4.2.2). Influence-based methods (§4.3, Eqs. 17--18) are discussed only through the fragility of influence functions, and constrained/saliency/low-rank methods (§4.4) only through localizability assumptions; no adversarial recovery audit is reported for representatives such as NPO, SGA, SalUn, GRAIL, CATNIP, or GD+Sine. Either provide direct audits for each family or narrow the claim to 'no method in the sampled families has yet been demonstrated to achieve true forgetting.' As written, the abstract converts absence of evidence into a field-wide impossibility.
  2. [§3.3, Eq. (2); §6.1; §7.2] The definition of true forgetting as exact distributional equivalence to a retrained model, Eq. (2), is acknowledged in §6.1 to be infeasible to evaluate at LLM scale because retraining is impractical. Consequently, the paper's central negative statement is not crisply distinguishable from 'no method has been shown to satisfy an untestable criterion.' The distinction matters because Table 2 prescribes true forgetting as the minimum defense for hazardous knowledge and backdoors, while §7.2 concludes no existing method achieves it. The paper should separate an absence-of-evidence claim from an impossibility claim, and should discuss feasible approximate-unlearning criteria (Eq. 3) or adversarial-probe-based certificates that could be checked in practice.
  3. [§4.2.2; §4.2.3] The squeezing effect is an output-level phenomenon: softmax normalization displaces probability mass to semantically related paraphrases. The text infers from this that 'the underlying mechanism may be redistribution rather than removal' and then uses this to classify methods at the behavioral-suppression level in §4.2.3. However, output-level probability displacement is compatible with either retention or genuine removal of internal representations. The argument also does not cover influence-based methods (§4.3) or other families that do not optimize a next-token softmax. This paragraph should be reworded as evidence against output-level metrics, not as evidence about representational retention.
  4. [§1.2; Table 3; References [68], [70], [74], [75]] The methodology section describes the review as systematic, but no screening counts, number of included papers, exclusion decisions, or quality-assessment details are reported beyond a manual 'quality, empirical evidence, and relevance' screen. Since the negative conclusion depends on complete coverage of three method families, the protocol should be reported in a reproducible way. In addition, several key methods used in the classification (GRAIL [74], Forgetting-MarI [68], CATNIP [75], GD+Sine [70]) lack complete bibliographic metadata, and CATNIP appears only in Table 3 with no prose description. This makes it difficult for readers to verify the survey's coverage and the evidentiary basis for the assigned forgetting levels.
minor comments (4)
  1. [§1.1; References [11], [14], [17]] Citation inconsistencies in the prior-work discussion: [11] and [17] point to the same title by Le-Khac and Vinh, 'Si et al. [14]' does not match reference [14] (Zhang et al.), and 'Le-Khac and Truong [17]' should presumably be Le-Khac and Vinh. Please correct the references and the attributions.
  2. [Eqs. (13)--(14)] The symbol r is used for the retain-set parameter in Eq. (13) and r_s for the smoothing rate in Eq. (14). The paper notes the distinction, but the proximity is confusing; a different symbol for the retain-set parameter would improve readability.
  3. [Table 3; §4.2.1] FG-OrIU [64] and CATNIP [75] are listed in Table 3 but are not named or described in the main text. Either add brief descriptions or remove the table-only entries.
  4. [References] Several references are incomplete, with 'et al.' placeholder author lists or no venue/arXiv identifiers (e.g., [20], [56], [102], [103], [105]). For a survey, full bibliographic entries are expected.

Circularity Check

0 steps flagged

No significant circularity; the survey's negative conclusion rests on external audits and cited theory, not on self-citation or fitted inputs.

full rationale

This is a literature survey, not a parameterized derivation. Its three-level framework (Section 3.4) is an explicitly stipulated taxonomy for organizing existing methods, not a result derived from assumptions, and it is not used to fit any quantity. The central claim that no reviewed family achieves true forgetting in the sense of retrained equivalence (Abstract; Section 7.2) is supported by external, independently reported evidence: the white-box WMDP audits of Lucki et al. (Section 2.2 and Section 6.4), the gradient-ascent failure theorems of Mavrothalassitis et al. (Section 4.2.2), and fragility results for influence functions (Section 4.3). These citations are not self-citations of the present authors, and their assumptions do not include the survey's conclusion. The equations in the paper (e.g., Eqs. 5, 7, 9, 14, 16, 17-18, 20) are reproduced or reformulated from the cited literature and are not used to derive the survey's verdict; no parameter is fitted and then renamed a prediction. The main concern is scope: the universal negative generalizes from audits of RMU/gradient-ascent baselines to all families, and 'true forgetting' is defined so that the infeasibility of retraining makes certification difficult. These are evidentiary and calibration issues, not circularity. Therefore, the circularity score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The survey introduces no fitted parameters and no physical or parametric entities. Its conclusions depend on domain assumptions about distributed knowledge, infeasible retraining, the use of retrained equivalence as the gold standard, the transferability of adversarial audits, and the softmax mathematics of the squeezing effect. The three-level framework is a taxonomy, not an invented entity with an independent empirical handle.

axioms (5)
  • domain assumption Knowledge in LLMs is distributed and entangled across parameters rather than localized to identifiable units.
    Adopted in Sections 1, 3.2, and 4.2.2; it motivates why gradient-ascent cannot cleanly remove information, but is not established by this paper.
  • domain assumption Retraining an LLM without the forget set is computationally infeasible, so the gold standard of retrained equivalence cannot be directly verified.
    Stated in Sections 1 and 3.3; enables the claim that no method achieves 'true forgetting' because the ground-truth model is unavailable.
  • domain assumption Gradient ascent introduces a new optimization trajectory rather than inverting the original training trajectory, leaving underlying representations intact.
    Used in Sections 4.2.2 and 7.1, attributed to Mavrothalassitis et al. [71]; if false, the central negative synthesis loses much of its support.
  • standard math Softmax normalization makes exact removal of probability mass impossible in token-level objectives, so suppression redistributes mass to semantically related outputs.
    The squeezing effect in Section 4.2.2 is mathematically true for normalized distributions, but the claim that the redistributed mass is semantically equivalent is empirical, not formal.
  • domain assumption The black-box, gray-box, white-box adversary classification from adversarial machine learning transfers cleanly to LLM unlearning.
    Used in Section 2.2 to conclude that behavioral suppression only defends against black-box, non-adaptive adversaries; the transfer is asserted rather than justified in detail.

pith-pipeline@v1.3.0-alltime-deepseek · 28025 in / 9329 out tokens · 97213 ms · 2026-08-02T10:20:43.365324+00:00 · methodology

0 comments
read the original abstract

LLMs are increasingly deployed in security-critical systems across healthcare, finance, education, and decision support, yet their inability to forget creates serious cybersecurity, privacy, and safety risks. Sensitive personal information, copyrighted material, hazardous domain knowledge, and memorized training data remain encoded across billions of parameters long after deployment, leaving models vulnerable to extraction, jailbreak attacks, membership inference, and regulatory non-compliance. Real-world incidents, from chatbots regenerating private information to fabricated legal citations producing direct legal and financial cost, place the problem at the center of the emerging-threats landscape rather than the realm of speculation. Because retraining billion-parameter models on revised corpora is computationally infeasible, and because knowledge within an LLM is distributed and entangled across parameters rather than localized to identifiable units, LLM unlearning has emerged as the principal cyber defense response, aiming to remove or suppress targeted knowledge from a trained model without retraining and without eroding what the model should still know. A central question, however, remains unresolved. Do current methods genuinely remove knowledge, or do they only stop the model from expressing it under ordinary prompting conditions? This survey examines LLM unlearning through the lens of security, robustness, and verifiable forgetting, with primary focus on gradient-based methods, which have come to dominate the field due to their compatibility with existing training pipelines and their scalability to billion-parameter models.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

122 extracted references · 12 canonical work pages

  1. [1]

    Maity and Manob Jyoti Saikia

    S. Maity and Manob Jyoti Saikia. Large language models in healthcare and medical applications: A review.Bioengineering, 12(6):631, 2025. doi: 10.3390/ bioengineering12060631

    2025

  2. [2]

    Yuan et al

    M. Yuan et al. Large language models illuminate a progressive pathway to artificial intelligent healthcare assistant.Medicine Plus, 1(2):100030, 2024. doi: 10.1016/j.medp.2024.100030

    arXiv 2024

  3. [3]

    Vrdoljak, Z

    J. Vrdoljak, Z. Boban, M. Vilovi ´c, M. Kumri´c, and J. Boˇ zi´c. A review of large language models in medical education, clinical decision support, and healthcare administration.Healthcare, 13(6):603, 2025. doi: 10.3390/healthcare13060603

    doi:10.3390/healthcare13060603 2025

  4. [4]

    A systematic review of transformer-based pre-trained language models through self-supervised learn- ing.Information, 14(3):187, 2023

    Evans Kotei and Ramkumar Thirunavukarasu. A systematic review of transformer-based pre-trained language models through self-supervised learn- ing.Information, 14(3):187, 2023. doi: 10.3390/info14030187

    work page doi:10.3390/info14030187 2023

  5. [5]

    W. Yang, L. Some, M. Bain, and B. Kang. A comprehensive survey on integrat- ing large language models with knowledge-based methods.Knowledge-Based Systems, 318:113503, 2025. doi: 10.1016/j.knosys.2025.113503

    arXiv 2025

  6. [6]

    Quantifying memorization across neural language models

    Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. InICLR, 2022. URL https://openreview.net/forum?id=TatRHT 1cK. LLM Unlearning for Cyber Defense 35

    2022

  7. [7]

    Karamolegkou, J

    A. Karamolegkou, J. Li, L. Zhou, and Anders Søgaard. Copyright violations and large language models. InOpenReview, 2023. URL https://openreview.net/for um?id=YokfK5VOoz

    2023

  8. [8]

    G. Kim, Y. Li, E. Spiliopoulou, J. Ma, and W. Wang. Detecting training data of large language models via expectation maximization. InEACL, pages 1115– 1129, 2026. URL https://aclanthology.org/2026.eacl-long.49.pdf

    2026

  9. [9]

    Jailbroken: How does llm safety training fail? InNeurIPS, 2023

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? InNeurIPS, 2023. URL https://openreview.net/forum?i d=jA235JGM09

    2023

  10. [10]

    Bernhard Past ¨otter and C. C. Haciahmet. Can people intentionally and selectively forget prose material?Frontiers in Psychology, 13, 2022. doi: 10.3389/fpsyg.20 22.928533

    doi:10.3389/fpsyg.20 2022

  11. [11]

    U. N. Le-Khac and Vinh. A survey on large language models unlearning: Tax- onomy, evaluations, and future directions.Artificial Intelligence Review, 58(12),

  12. [12]

    Lizzo and L

    T. Lizzo and L. Heck. Unlearn: Efficient removal of knowledge in large language models. InFindings of the Association for Computational Linguistics: NAACL 2025, pages 7257–7268, 2025. doi: 10.18653/v1/2025.findings-naacl.405

    work page doi:10.18653/v1/2025.findings-naacl.405 2025

  13. [13]

    Zhang et al

    C. Zhang et al. Rule: Reinforcement unlearning achieves forget-retain pareto optimality, 2025. URL https://neurips.cc/virtual/2025/loc/san-diego/poster/11

    2025

  14. [14]

    A review on machine unlearning.SN Computer Science, 4(4), 2023

    Hongwei Zhang, Takamasa Nakamura, Takamasa Isohara, and Kouichi Sakurai. A review on machine unlearning.SN Computer Science, 4(4), 2023. doi: 10.1007/s42979-023-01767-4

    work page doi:10.1007/s42979-023-01767-4 2023

  15. [15]

    Huan Xu, Tianqing Zhu, Lei Zhang, Wanlei Zhou, and Philip S. Yu. Machine unlearning: A survey.ACM Computing Surveys, 56(1):1–36, 2023. doi: 10.114 5/3603620

    2023

  16. [16]

    A. et al. Blanco-Justicia. Digital forgetting in large language models: a survey of unlearning methods.Artificial Intelligence Review, 58(3), 2025. doi: 10.100 7/s10462-024-11078-6

    2025

  17. [17]

    A survey on large language models unlearning: Taxonomy, evaluations, and future directions.Artificial Intelligence Review, 58 (12), 2025

    Uyen Le-Khac and Tran Vinh. A survey on large language models unlearning: Taxonomy, evaluations, and future directions.Artificial Intelligence Review, 58 (12), 2025. doi: 10.1007/s10462-025-11376-7

    work page doi:10.1007/s10462-025-11376-7 2025

  18. [18]

    Cevallos, Marco E

    Ivan D. Cevallos, Marco E. Benalc´azar, Luis Valdivieso, Juan A. Zea, and Luis I. Barona-L´opez. A systematic literature review of machine unlearning techniques in neural networks.Computers, 14(4):150, 2025. doi: 10.3390/computers14040 150

    work page doi:10.3390/computers14040 2025

  19. [19]

    Towards making systems forget with machine unlearning

    Yang Cao and Junfeng Yang. Towards making systems forget with machine unlearning. InProceedings of the IEEE Symposium on Security and Privacy Workshops, pages 463–480, 2015. doi: 10.1109/SPW.2015.15

    work page doi:10.1109/spw.2015.15 2015

  20. [20]

    N. et al. Li. The wmdp benchmark: Measuring and reducing malicious use with unlearning, 2024

    2024

  21. [21]

    An adversarial perspective on machine unlearning for AI safety

    Jakub Lucki, Boyi Wen, Maksym Andriushchenko, Nicolas Flammarion, and Florian Tram`er. An adversarial perspective on machine unlearning for AI safety. Transactions on Machine Learning Research, 2025. URL https://arxiv.org/abs/ 2409.18025. arXiv:2409.18025. 36 R. S. Shankar & A. Bhardwaj et al

    Pith/arXiv arXiv 2025

  22. [22]

    Magesh, F

    V. Magesh, F. Surani, M. Dahl, M. Suzgun, Christopher Manning, and Daniel Ho. Hallucination-free? assessing the reliability of leading ai legal research tools,

  23. [23]

    Adversar- ial machine learning: A taxonomy and terminology of attacks and mitigations

    Apostol Vassilev, Alina Oprea, Alie Fordyce, and Harold Anderson. Adversar- ial machine learning: A taxonomy and terminology of attacks and mitigations. Technical Report NIST AI 100-2, National Institute of Standards and Technology,

  24. [24]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023. URL https://arxiv.org/abs/2307.15043

    Pith/arXiv arXiv 2023

  25. [25]

    Schwinn, D

    L. Schwinn, D. Dobre, and S. et al. Xhonneux. Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the embedding space, 2024

    2024

  26. [26]

    URL https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2025.pdf

  27. [27]

    Provvedimento del 30 marzo 2023 [9870832]

    Garante per la Protezione dei Dati Personali. Provvedimento del 30 marzo 2023 [9870832]. https://www.garanteprivacy.it, 2023. Italian Data Protection Authority order against OpenAI. Follow-up press release April 28, 2023

    2023

  28. [28]

    The EU proposal for a general data protection regulation and the roots of the right to be forgotten.Computer Law and Security Review, 29(3):229–235, 2013

    Alessandro Mantelero. The EU proposal for a general data protection regulation and the roots of the right to be forgotten.Computer Law and Security Review, 29(3):229–235, 2013. doi: 10.1016/j.clsr.2013.03.010

    doi:10.1016/j.clsr.2013.03.010 2013

  29. [29]

    United States District Court, S.D.N.Y. Mata v. avianca, inc., no. 22-cv-1461, sanctions order, 2023. Judge P. Kevin Castel. June 22, 2023. Available via PACER

    2023

  30. [30]

    Do large language models understand us?Daedalus, 151(2):183–197, 2022

    Blaise Ag¨ uera y Arcas. Do large language models understand us?Daedalus, 151(2):183–197, 2022. doi: 10.1162/daed a 01909

    work page doi:10.1162/daed 2022

  31. [31]

    Naveed et al

    H. Naveed et al. A comprehensive overview of large language models.ACM Transactions on Intelligent Systems and Technology, 2025. doi: 10.1145/374474 6

    work page doi:10.1145/374474 2025

  32. [32]

    Moffatt v

    Civil Resolution Tribunal of British Columbia. Moffatt v. air canada, 2024 BCCRT 149. https://decisions.civilresolutionbc.ca, 2024. Tribunal Member Christopher Rivers

    2024

  33. [33]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186, 2019

    2019

  34. [34]

    Liu et al

    Q. Liu et al. When moe meets llms: Parameter efficient fine-tuning for multi- task medical applications. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1104– 1114, 2024. doi: 10.1145/3626772.3657722

    arXiv 2024

  35. [35]

    Improving language understanding by generative pre-training, 2018

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018. URL https://cdn.open ai.com/research-covers/language-unsupervised/language understanding paper .pdf

    2018

  36. [36]

    Language models are few-shot learners

    Tom Brown et al. Language models are few-shot learners. InAdvances in Neural Information Processing Systems (NeurIPS), 2020. URL https://proceedings.ne urips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf

    2020

  37. [37]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113, 2023. URL https: //jmlr.org/papers/v24/22-1144.html

    2023

  38. [38]

    M. Li, L. Chen, J. Chen, S. He, J. Gu, and T. Zhou. Selective reflection-tuning: Student-selected data recycling for llm instruction-tuning. InFindings of the Association for Computational Linguistics ACL, pages 16189–16211, 2024. doi: 10.18653/v1/2024.findings-acl.958. LLM Unlearning for Cyber Defense 37

    work page doi:10.18653/v1/2024.findings-acl.958 2024

  39. [39]

    Fields, K

    J. Fields, K. Chovanec, and Praveen Madiraju. A survey of text classification with transformers: How wide? how large? how long? how accurate? how expensive? how safe?IEEE Access, 2024. doi: 10.1109/access.2024.3349952

    arXiv 2024

  40. [40]

    Peykani, F

    P. Peykani, F. Ramezanlou, C. Tanasescu, and S. Ghanidel. Large language models: A structured taxonomy and review of challenges, limitations, solutions, and future directions.Applied Sciences, 15(14):8103, 2025. doi: 10.3390/app1 5148103

    doi:10.3390/app1 2025

  41. [41]

    Acharya, B

    A. Acharya, B. Singh, and Naoyuki Onoe. Llm based generation of item- description for recommendation system. InProceedings of the 17th ACM Con- ference on Recommender Systems (RecSys ’23), pages 1204–1207, 2023. doi: 10.1145/3604915.3610647

    arXiv 2023

  42. [42]

    Transformer feed- forward layers are key-value memories

    Mor Geva, Tal Schuster, Jonathan Berant, and Omer Levy. Transformer feed- forward layers are key-value memories. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. doi: 10.18653/v1/2021.emnlp-main.446

    doi:10.18653/v1/2021.emnlp-main.446 2021

  43. [43]

    Locating and editing factual associations in gpt

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. InAdvances in Neural Information Processing Systems (NeurIPS), 2022. URL https://proceedings.neurips.cc/paper files/paper /2022/file/6f1d43d5a82a37e89b0665b33bf3a182-Paper-Conference.pdf

    2022

  44. [44]

    Trust and R

    P. Trust and R. Minghim. A study on text classification in the age of large language models.Machine Learning and Knowledge Extraction, 6(4):2688–2721, 2024. doi: 10.3390/make6040129

    work page doi:10.3390/make6040129 2024

  45. [45]

    Machine unlearning: Solu- tions and challenges.IEEE Transactions on Emerging Topics in Computational Intelligence, 8(3):2150–2168, 2024

    Jie Xu, Zongyu Wu, Chen Wang, and Xiaohua Jia. Machine unlearning: Solu- tions and challenges.IEEE Transactions on Emerging Topics in Computational Intelligence, 8(3):2150–2168, 2024. doi: 10.1109/TETCI.2024.3379240

    arXiv 2024

  46. [46]

    Arcane: An efficient architecture for exact machine unlearning

    Hong Yan, Xiaoqiang Li, Zhen Guo, Hui Li, Feifei Li, and Xiapu Lin. Arcane: An efficient architecture for exact machine unlearning. InProceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pages 4009– 4015, 2022. URL https://www.ijcai.org/proceedings/2022/556

    2022

  47. [47]

    Roy, and Gintare Karolina Dziugaite

    Nazanin Mohammadi Sepahvand, Anvith Thudi, Berivan Isik, Ashmita Bhattacharyya, Nicolas Papernot, Eleni Triantafillou, Daniel M. Roy, and Gintare Karolina Dziugaite. Leveraging per-instance privacy for machine un- learning. InProceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 ofProceedings of Machine Learning Research, 2025

    2025

  48. [48]

    Unlearning vs

    Guangyu Sun, Potsawee Manakul, Xuan Zhan, and Mark Gales. Unlearning vs. obfuscation: Are we truly removing knowledge? InProceedings of the 2025 38 R. S. Shankar & A. Bhardwaj et al. Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 11468–11478, 2025. doi: 10.18653/v1/2025.emnlp-main.577

    doi:10.18653/v1/2025.emnlp-main.577 2025

  49. [49]

    Re- liable unlearning harmful information in llms with metamorphosis represen- tation projection.arXiv preprint arXiv:2508.15449, 2025

    Chenkai Wu, Zhe Wei, Hong Chen, Yuxiao Dong, and Maosong Sun. Re- liable unlearning harmful information in llms with metamorphosis represen- tation projection.arXiv preprint arXiv:2508.15449, 2025. URL https: //arxiv.org/abs/2508.15449

    Pith/arXiv arXiv 2025

  50. [50]

    Certified data removal from machine learning models.arXiv preprint arXiv:1911.03030, 2019

    Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens van der Maaten. Certified data removal from machine learning models.arXiv preprint arXiv:1911.03030, 2019. doi: 10.48550/arXiv.1911.03030

    Pith/arXiv arXiv doi:10.48550/arxiv.1911.03030 1911

  51. [51]

    Eternal sunshine of the spotless net: Selective forgetting in deep networks.IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020

    Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks.IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020

    2020

  52. [52]

    Gradient-based and task-agnostic machine unlearning

    Daniel Trippa, Cesare Campagnano, Maria Sofia Bucarelli, Gabriele Tolomei, and Fabrizio Silvestri. Gradient-based and task-agnostic machine unlearning. arXiv preprint arXiv:2403.14339, 2024

    Pith/arXiv arXiv 2024

  53. [53]

    Ali Hatami and Ivan E. Monosov. Beyond masking and avoidance: Toward true unlearning. InInternational Conference on Learning Representations (ICLR) (under review), 2026. URL https://openreview.net/forum?id=XXSPoq3uik

    2026

  54. [54]

    et al. Wang. Leak@k: Unlearning does not make llms forget under probabilistic decoding.arXiv preprint, 2024

    2024

  55. [55]

    Towards fair large language model-based recommender systems without costly retraining.arXiv preprint arXiv:2601.17492, 2026

    Jian Li et al. Towards fair large language model-based recommender systems without costly retraining.arXiv preprint arXiv:2601.17492, 2026. URL https: //arxiv.org/abs/2601.17492

    arXiv 2026

  56. [56]

    Llm unlearning using gradient ratio- based influence estimation and noise injection.arXiv preprint arXiv:2508.06467, 2025

    Ameya Anjarlekar and Sandeep Pombra. Llm unlearning using gradient ratio- based influence estimation and noise injection.arXiv preprint arXiv:2508.06467, 2025

    Pith/arXiv arXiv 2025

  57. [57]

    Forget the token and pixel: Rethinking gradient ascent for concept unlearning in multimodal generative models

    Jian Li et al. Forget the token and pixel: Rethinking gradient ascent for concept unlearning in multimodal generative models. InFindings of the Association for Computational Linguistics: ACL 2025, pages 12179–12200, 2025. doi: 10.186 53/v1/2025.findings-acl.630

    2025

  58. [58]

    Is gradient ascent really necessary? memorize to forget for machine unlearning

    Zhen Huang, Qian Wang, Zhiqing Hong, Shuo Ye, Bo Han, and Tongliang Liu. Is gradient ascent really necessary? memorize to forget for machine unlearning. arXiv preprint arXiv:2602.06441, 2026. URL https://arxiv.org/abs/2602.06441

    arXiv 2026

  59. [59]

    Unified gradient-based machine unlearning with remain geometry enhancement

    Zhen Huang et al. Unified gradient-based machine unlearning with remain geometry enhancement. InAdvances in Neural Information Processing Systems (NeurIPS), 2024. URL https://proceedings.neurips.cc/paper files/paper/2024/ file/2e622ac74f66df03b686a12e2e0e4424-Paper-Conference.pdf

    2024

  60. [60]

    Mass-editing memory in a transformer

    Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. InInternational Conference on Learning Representations (ICLR), 2023. URL https://arxiv.org/abs/2210.07229. arXiv:2210.07229

    Pith/arXiv arXiv 2023

  61. [61]

    Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate.arXiv preprint arXiv:2410.22086, 2025

    Zhiqi Bu, Xiaomeng Jin, Bhanukiran Vinzamuri, Anil Ramakrishna, Kai-Wei Chang, Volkan Cevher, and Mingyi Hong. Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate.arXiv preprint arXiv:2410.22086, 2025. URL https://arxiv.org/abs/2410.22086. LLM Unlearning for Cyber Defense 39

    Pith/arXiv arXiv 2025

  62. [62]

    Fine-grained pluggable gradient ascent for knowledge unlearning in language models

    Xiaojun Feng, Chao Chen, Yu Li, and Zhiqiu Lin. Fine-grained pluggable gradient ascent for knowledge unlearning in language models. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024. URL https://aclanthology.org/2024.emnlp-main.566.pdf

    2024

  63. [63]

    Forget for get: A lightweight two-phase gradient method for knowledge editing in large language models

    Yifan Li, Mingyuan Yang, Xiaoyu Hu, and Chao Li. Forget for get: A lightweight two-phase gradient method for knowledge editing in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 7604–7623, 2025. doi: 10.18653/v1/2025.findings-emnlp.402

    work page doi:10.18653/v1/2025.findings-emnlp.402 2025

  64. [64]

    Fg- oriu: Towards better forgetting via feature-gradient orthogonality for incremental unlearning

    Qian Feng, Jun Tu, Meng Kang, Hao Zhao, Cheng Zhang, and Hong Qian. Fg- oriu: Towards better forgetting via feature-gradient orthogonality for incremental unlearning. InProceedings of the IEEE/CVF International Conference on Com- puter Vision (ICCV), 2025. URL https://openaccess.thecvf.com/content/ICCV 2025/papers/Feng FG-OrIU Towards Better Forgetting vi...

    2025

  65. [65]

    Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation.arXiv preprint arXiv:2310.12508, 2023

    Chongyu Fan, Jun Liu, Yifei Zhang, Eric Wong, Dennis Wei, and Sijia Liu. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation.arXiv preprint arXiv:2310.12508, 2023. URL https://arxiv.org/abs/2310.12508

    Pith/arXiv arXiv 2023

  66. [66]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InAdvances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. URL https://arxiv.org/abs/2305.18290. arXiv:2305.18290

    Pith/arXiv arXiv 2023

  67. [67]

    LLM unlearning with LLM beliefs

    Kemou Li, Qizhou Wang, Yue Wang, Fengpeng Li, Jun Liu, Bo Han, and Jiantao Zhou. LLM unlearning with LLM beliefs. InInternational Conference on Learning Representations (ICLR), 2026. URL https://arxiv.org/abs/2510.19422. arXiv:2510.19422

    arXiv 2026

  68. [68]

    Negative preference optimiza- tion: From catastrophic collapse to effective unlearning

    Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimiza- tion: From catastrophic collapse to effective unlearning. InProceedings of the 38th Annual Conference on Neural Information Processing Systems (NeurIPS),

  69. [69]

    arXiv:2404.05868

    URL https://arxiv.org/abs/2404.05868. arXiv:2404.05868

    Pith/arXiv arXiv

  70. [70]

    Stable for- getting: Bounded parameter-efficient unlearning in foundation models, 2026

    Arpit Garg, Hemanth Saratchandran, Ravi Garg, and Simon Lucey. Stable for- getting: Bounded parameter-efficient unlearning in foundation models, 2026

    2026

  71. [71]

    Levi, and Volkan Cevher

    Ioannis Mavrothalassitis, Pol Puigdemont, N. Levi, and Volkan Cevher. Ascent fails to forget. InAdvances in Neural Information Processing Systems (NeurIPS),

  72. [72]

    Forgetting-MarI: LLM unlearning via marginal information regularization, 2026

    Yuchen Xu, Jiayi Guo, Bingbing Wen, and Guosheng Hu. Forgetting-MarI: LLM unlearning via marginal information regularization, 2026

    2026

  73. [73]

    Label smoothing improves gradient ascent in LLM unlearning, 2025

    Zirui Pang, Hao Zheng, Zhijie Deng, Ling Li, Zixin Zhong, and Jiaheng Wei. Label smoothing improves gradient ascent in LLM unlearning, 2025

    2025

  74. [74]

    GRAIL: Gradient-based adaptive unlearning for privacy and copyright in LLMs, 2025

    Taehyun Kim et al. GRAIL: Gradient-based adaptive unlearning for privacy and copyright in LLMs, 2025

    2025

  75. [75]

    CATNIP: LLM unlearning via calibrated and tokenized nega- tive preference alignment, 2026

    Zhiyu Yang et al. CATNIP: LLM unlearning via calibrated and tokenized nega- tive preference alignment, 2026. 40 R. S. Shankar & A. Bhardwaj et al

    2026

  76. [76]

    URL https://neurips.cc/virtual/2025/poster/118666. Poster

    2025

  77. [77]

    Transformer feed- forward layers are key-value memories

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed- forward layers are key-value memories. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. URL https://aclanthology.org/2021.emnlp-main.446.pdf

    2021

  78. [78]

    Machine unlearning in contrastive learning.arXiv preprint arXiv:2405.07317, 2024

    Zhen Wang and Ke Chen. Machine unlearning in contrastive learning.arXiv preprint arXiv:2405.07317, 2024. URL https://arxiv.org/abs/2405.07317

    Pith/arXiv arXiv 2024

  79. [79]

    Gauss-newton unlearning for the llm era.arXiv preprint arXiv:2602.10568, 2026

    Liam McKinney et al. Gauss-newton unlearning for the llm era.arXiv preprint arXiv:2602.10568, 2026. URL https://arxiv.org/abs/2602.10568

    arXiv 2026

  80. [80]

    Lacuna inc

    Aleksey Kudelya and Alexander Shirnin. Lacuna inc. at semeval-2025 task 4: Lora-enhanced influence-based unlearning for llms. InProceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), pages 1528– 1533, 2025. URL https://aclanthology.org/2025.semeval-1.201/

    2025

Showing first 80 references.