Pith. sign in

REVIEW 3 major objections 5 minor 47 references

A mean teacher algorithm for unlearning of language models

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Mean teacher turns LLM unlearning into slow natural gradient descent.

desk verdict Honest empirical LLM unlearning with a novel loss, but the central theory proof has a real gap in Lemma D.1. read the letter →

arxiv 2504.13388 v1 pith:4XBCJFZT submitted 2025-04-18 cs.LG cs.CL

Yegor Klochkov This is my paper · ORCID
classification cs.LGcs.CL
keywords machineunlearningmeanteachernaturalgradientdescentnegativelog-unlikelihoodlanguagemodelmemorizationGauss-NewtonHessianMUSEbenchmarkproximaloptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the mean teacher update—keeping a slowly moving reference model and pulling the optimized model toward it—is not just a heuristic regularizer. Under small step sizes, the trajectory of the mean teacher approximately equals a slow natural gradient descent, an update that moves preferentially along low-curvature directions and so is less likely to wreck the model's general abilities. The paper introduces a negative log-unlikelihood loss to avoid the vanishing-gradient stall that afflicts mean teacher with common unlearning objectives. The combination is tested on the MUSE benchmarks, where it lowers verbatim and sometimes knowledge memorization and reduces privacy leakage, while exposing that apparent knowledge removal can be an artifact of reduced general question-answering ability.

What carries the argument

The load-bearing objects are the sliding reference model $\theta'_t$ and the difference $u_t = \theta_t - \theta'_t$. The reference is updated as an exponential moving average, and the regularized gradient step on $\alpha L + D(\theta,\theta')$ makes $u_t$ behave like an inverse-Hessian vector product: the proof compares the recursion for $u_t$ to IHVP iterations and obtains $u_t \approx -\alpha H_\lambda(\theta_t)^{-1}\nabla L(\theta_t)$. Because $H$ is the Gauss-Newton Hessian of the KL divergence on the retain or pretraining set, the resulting update is slow natural gradient descent conditioned by $H_\lambda^{-1}$. The NLUL loss is the second piece: its gradient equals $(p/(1-p))$ times the log-likelihood gradient, so it escapes the $p \approx 1$ flat start of a memorizing model without the explosive gradients of pure gradient ascent.

What would settle it

Run Algorithm 1 and Algorithm 2 on a small transformer with the same hyperparameters as the MUSE runs and compute the exact Gauss-Newton Hessian on the pretraining batch. If the batched, clipped, momentum trajectory diverges from the slow-NGD trajectory by more than the theorem's $O(\alpha \log(1/\alpha))$ bound, or if the utility preservation vanishes when the reference model is frozen, then the theorem is not the mechanism behind the experimental gains.

Watch

Extended reading notes

Core claim

The central claim is Theorem 3.1: for a generic loss $L$ and divergence $D$ with positive-definite local Hessian $H$, the mean teacher loop with small learning rate $\eta$, small loss weight $\alpha$, and bounded total horizon $T\gamma$ follows the slow natural-gradient trajectory $\theta_{t+1} = \theta_t - \gamma H_\lambda^{-1}\nabla L$, up to error $O(\alpha \log(1/\alpha))$ in parameter space. In the KL/QKL case, $H$ is the Gauss-Newton Hessian of the retain set, so mean teacher is a computationally cheap proximal stand-in for second-order unlearning. The paper also claims that with log-likelihood or NPO, mean teacher can stall near a converged model, and that the new negative-log-unlikelihood loss $\ell_{\mathrm{NLUL}} = -\log(1 - p(y))$ rescues the early gradient while remaining bounded later. On MUSE-News and MUSE-Books, MT+NLUL reaches the retrained-model memorization thresholds, preserves knowledge of retain articles better than the provided baselines in several settings, and yields privacy-leakage values closer to zero. A strong QKL variant does lower knowledge memorization on the forget set, but the paper reports that this comes with a substantial drop in MMLU, and later finetuning on unrelated data restores both, indicating the knowledge remained in the weights.

Load-bearing premise

The theoretical guarantee holds for full-batch updates with tiny step sizes and regularity assumptions, while the experiments use minibatches, gradient clipping, momentum, and many epochs; if the batched clipped mean teacher stops following the low-curvature natural-gradient trajectory, the explanation for why utility is preserved does not cover the reported results.

Editorial extensions

If this is right

  • Mean teacher gives a proximal, low-curvature update for unlearning that can preserve utility while reducing verbatim memorization on MUSE benchmarks.
  • Replacing the curated retain split with a large pretraining subset improves retention and makes the method reusable across unlearning tasks.
  • The NLUL loss lets mean teacher progress from a converged forget set without a separate base model or extra hyperparameter such as NPO's $\beta$.
  • Stronger forget-side unlearning reduces knowledge memorization below the retrained threshold but costs MMLU, so reports of knowledge removal should include general-ability checks.
  • Sequential unlearning requests on MUSE-News show that utility is preserved when pretraining data is used for regularization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The theorem's $O(\alpha \log(1/\alpha))$ guarantee suggests mean teacher could serve as a cheap proxy for influence-function-based unlearning when the Hessian is too large to form explicitly; tuning $\alpha$ might control the forgetting-retention trade-off.
  • Because the natural-gradient interpretation depends on the divergence's curvature, testing other divergences would separate the mean-teacher mechanism from the Gauss-Newton choice.
  • The finding that unrelated SFT data restores forgotten knowledge implies that current MUSE metrics measure surface behavior, not weight-level deletion; a stricter test would probe whether any fine-tuning path can resurrect the memorized text.
  • A testable extension is to run Algorithm 1 with exact full-batch Gauss-Newton inverse on a small language model and compare its trajectory to the batched Algorithm 2; if the batched version deviates beyond the theorem's bound, the experimental results are not explained by Theorem 3.1.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a mean teacher algorithm for LLM unlearning and gives a theoretical result (Theorem 3.1) stating that, under small step sizes and regularity conditions, the mean teacher trajectory approximately matches slow natural gradient descent with error O(alpha log(1/alpha)). A new loss, negative log-unlikelihood (NLUL), is introduced to avoid vanishing gradients at the start of unlearning. Experiments on MUSE-News and MUSE-Books compare mean teacher variants with AdamW baselines on verbatim/knowledge memorization, retain-set utility, privacy leakage, and MMLU. The paper honestly reports that the strongest knowledge-removal variant (MT+NLUL+QKL) reduces MMLU, that a shorter run shows utility loss relative to the retrained model, and that fine-tuning on unrelated alignment data can restore both knowledge memorization and MMLU.

Significance. If Theorem 3.1 were established, the paper would provide a clean proximal-optimization interpretation of mean teacher as an implicit natural-gradient method that prefers low-curvature updates, which is a valuable conceptual contribution to LLM unlearning. The paper also ships reproducible code, reports negative results openly, includes a sequential-unlearning experiment, and is careful to flag the MMLU cost of aggressive knowledge unlearning. These are real strengths. The central theoretical claim is currently not proven as written because the main contraction lemma uses a spectral-norm bound that is false for nonnormal matrices. The experimental claims are suggestive but are not formally connected to the theorem, since the evaluated algorithm differs from the analyzed one in batching, clipping, momentum, and number of epochs. The empirical significance is therefore moderate pending a corrected proof or an explicit heuristic framing.

major comments (3)
  1. [Appendix D.1, Lemma D.1, Eq. (11)] The proof of Lemma D.1 bounds the spectral norm of the 2x2 block B = [[1-eta*h, -mu], [eta*h, mu]] by its spectral radius R = max(sqrt(mu), 1 - eta*lambda/(1-mu)) and then uses ||B^t|| <= R^t. This is not valid because B is nonnormal. For example, with mu = 0.9 and eta*h = 0.5, the spectral radius of B is sqrt(mu) approx 0.949 but ||B|| approx 1.27. Consequently, the error terms in Eq. (11) are not bounded by the claimed geometric contraction, and the IHVP approximation u_t approx -alpha * H_lambda^{-1} nabla L is not established. Since this IHVP approximation is the engine connecting Algorithm 1 to natural gradient descent, the O(alpha log(1/alpha)) bound in Theorem 3.1 does not follow from the written proof. The proof needs either a real norm bound incorporating nonnormal transients (for example via a similarity transform with explicit constants, if such a bound can be proved) or a replacement lemma.
  2. [Section 3, Implementation details; Algorithm 2] Theorem 3.1 is stated and proved for the full-batch Algorithm 1 under small alpha, small eta, and a bounded total step T*gamma. The experiments, however, use Algorithm 2, which draws minibatches, applies per-step gradient clipping with an adaptive contraction parameter, includes momentum, and runs for many epochs with batch size 40. The paper acknowledges this gap in the paragraph beginning 'Finally, notice that Theorem 3.1 assumes full-batch gradient updates', but the paragraph does not supply any argument that the batched, clipped, momentum-based trajectory inherits the low-curvature natural-gradient behavior. The central utility-preservation story is therefore an assertion rather than a consequence of the theorem. I would like the revision to either add a formal robustness statement, give empirical evidence (for example, trajectory or IHVP-error measurements on a small model), or explicitly label the theoretical result as motivational and decouple it from the experimental claims.
  3. [Section D.2, Connecting to NG descent] There is an inconsistency between the theorem statement and the proof about which gradient is used in the comparison trajectory. Theorem 3.1 defines the comparison update as theta_{t+1} = theta_t - gamma * H_lambda^{-1}(theta_t) * nabla L(theta_{t-1}), while the proof's 'Connecting to NG descent' paragraph compares against theta_{t+1} = theta_t - gamma * H_lambda^{-1}(theta_t) * nabla L(theta_t). If the lag is intentional, the proof must track it; if it is a typo, the theorem statement should be corrected. This matters because the claimed bound is about a specific reference trajectory.
minor comments (5)
  1. [Appendix D.1, Eq. (11)] The displayed definition of the matrix A after Eq. (11) is garbled ('A = [[1-eta H_lambda eta(H+lambda) mu]]'); it should be written as the 2x2 block matrix used in the recursion.
  2. [Table 1, MUSE-Books row] The value '37.17 (5.3)' appears with an inconsistent number of decimal places compared with the rest of the table; presumably '37.2 (5.3)' is intended.
  3. [Figure 1 caption] The caption says that for NPO/LL the authors 'additionally perform 2 epochs with AdamW to escape the starting point', but the surrounding text presents these as mean teacher runs. Please clarify whether the plotted NPO/LL curves are warm-started mean teacher runs or baselines, and state this clearly in the caption.
  4. [Section 5.3, Figure 2] The gradient-norm comparison in Figure 2 is qualitative and would be more useful with an axis scale and a statement of which loss each curve corresponds to; currently the legend colors are not described in the caption.
  5. [Section 5.1, PrivLeak] The abstract and Section 5.4 highlight lower privacy leakage, but Table 1 reports PrivLeak from a single run. The single-run caveat should appear in the main text near the claim, not only in the table footnote.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the natural-gradient approximation is a self-contained mathematical claim and the benchmark results are not inputs to the derivation.

full rationale

The only formal derivation is Theorem 3.1, which states that Algorithm 1 approximately follows the slow natural-gradient update (8) under explicit conditions: small step sizes, bounded T*gamma, positive-definite H satisfying the quadratic approximation (5), and the regularity Condition D.2. These assumptions do not contain the conclusion or any benchmark outcome; the proof derives the comparison from the mean-teacher recursion itself, rewriting it as an IHVP-type iteration (Eq. (14)) and bounding the error with Lemma D.1. The lemma is proved in the appendix, so the adjacent citation to Klochkov & Liu (2024) is attribution and not load-bearing. The empirical section is also not circular: NLUL is selected after inspecting training curves on MUSE-News, but the paper states this selection explicitly and does not present it as a prediction; the QKL variant is run until a disclosed knowmem-f threshold, and the paper openly reports the resulting MMLU drop and the recovery of memorization after SFT, which are honest, falsifiable observations. A genuine concern is that Lemma D.1 bounds the spectral norm of a nonnormal 2x2 block by its spectral radius, which appears mathematically invalid and would undermine the stated O(alpha log(1/alpha)) bound; however, that is a correctness gap in the proof, not a circularity, because the theorem and its bound are not assumed as inputs. No load-bearing step reduces to its own inputs, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The theoretical contribution is a derivation, not a fitted model, so the free-parameter count is small and mostly consists of standard optimizer hyperparameters. The main unverified inputs are the regularity assumptions of the theorem and the transfer of the full-batch proof to the batched algorithm.

free parameters (5)
  • loss weight α = 0.05 (MUSE-News), 0.1 (MUSE-Books)
    Scales the unlearning loss relative to the divergence term; the theorem's approximation error is O(α log(1/α)), and values were chosen per benchmark.
  • learning rate η = 0.0005
    Step size in Algorithm 2, chosen by hand for stable training.
  • contraction κ = 10.0
    Controls how quickly the reference model slides toward the optimized model; it sets the effective natural-gradient step size and damping.
  • damping λ = 0.5
    Added to the Hessian for conditioning; affects the preconditioned update and the theorem's convergence rate.
  • momentum µ = not explicitly reported; paper discusses µ=0.9 as typical
    Momentum is used in Algorithm 2 and appears in the theorem's damping correction, but the exact experimental value is not listed in the hyperparameters section.
assumptions (4)
  • domain assumption The KL or QKL divergence is locally quadratic with a symmetric positive-definite Gauss-Newton Hessian H(θ), and the quadratic approximation error is controlled uniformly along the trajectory.
    This is Eq. (5) and Condition D.2 in the proof; it is not verified numerically for Llama-2 on OpenWebText or the MUSE splits.
  • ad hoc to paper The batched, clipped, momentum-based Algorithm 2 inherits the full-batch behavior described by Theorem 3.1.
    Section 3 lists implementation changes as standard practice, but the theorem's assumptions do not cover batching, clipping, or the actual training horizon.
  • domain assumption OpenWebText is representative enough that its Gauss-Newton Hessian aligns with the pretraining directions that preserve general abilities.
    The paper replaces the benchmark retain split with OpenWebText and cites Ghorbani et al. for related eigenvector alignment in other settings.
  • domain assumption The target model is near convergence on the forget set, so the true next-token probability is often close to 1 at the start of unlearning.
    This motivates the NLUL gradient profile and is supported qualitatively by Figure 1, but it is not quantified across the full forget set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A mean teacher algorithm for unlearning of language models." pith.science (2026). https://pith.science/paper/4XBCJFZT

@misc{pith2026250413388,
  author       = {Pith},
  title        = {Pith review of: A mean teacher algorithm for unlearning of language models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4XBCJFZT}},
  note         = {Machine review of arXiv:2504.13388}
}
read the original abstract

One of the goals of language model unlearning is to reduce memorization of selected text instances while retaining the model's general abilities. Despite various proposed methods, reducing memorization of large datasets without noticeable degradation in model utility remains challenging. In this paper, we investigate the mean teacher algorithm (Tarvainen & Valpola, 2017), a simple proximal optimization method from continual learning literature that gradually modifies the teacher model. We show that the mean teacher can approximate a trajectory of a slow natural gradient descent (NGD), which inherently seeks low-curvature updates that are less likely to degrade the model utility. While slow NGD can suffer from vanishing gradients, we introduce a new unlearning loss called "negative log-unlikelihood" (NLUL) that avoids this problem. We show that the combination of mean teacher and NLUL improves some metrics on the MUSE benchmarks (Shi et al., 2024).

Figures

Figures reproduced from arXiv: 2504.13388 by the authors.

Figure 1
Figure 1. NLL loss on the forget set for MT using different unlearning losses (green). For IT we only show the KL divergence between the bad teacher and the target model. For NPO/LL we additionally per￾form 2 epochs with AdamW to “escape” the starting point [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 3
Figure 3. Comparison of mean teacher (blue), baselines with pretraining data (red), original baselines from (Shi et al., 2024) that use retain split for regularization (green), and recent new methods as reported in (Bu et al., 2024; Wang et al., 2024; Fan et al., 2024; Wang et al., 2025) (orange) [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Sustainability of unlearning: how utility preserves with sequential unlearning requests. We perform [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 9 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Second-order stochastic optimization for machine learning in linear time

    Agarwal, N., Bullins, B., and Hazan, E. Second-order stochastic optimization for machine learning in linear time. Journal of Machine Learning Research, 18 0 (116): 0 1--40, 2017

    work page 2017

  3. [3]

    Natural gradient works efficiently in learning

    Amari, S.-I. Natural gradient works efficiently in learning. Neural computation, 10 0 (2): 0 251--276, 1998

    1998

  4. [4]

    M., Thakkar, O., and Thakurta, A

    Amid, E., Ganesh, A., Mathews, R., Ramaswamy, S., Song, S., Steinke, T., Suriyakumar, V. M., Thakkar, O., and Thakurta, A. Public data-assisted mirror descent for private model training. In International Conference on Machine Learning, pp.\ 517--535. PMLR, 2022

    work page 2022

  5. [5]

    A., Jia, H., Travers, A., Zhang, B., Lie, D., and Papernot, N

    Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C. A., Jia, H., Travers, A., Zhang, B., Lie, D., and Papernot, N. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pp.\ 141--159. IEEE, 2021

    2021

  6. [6]

    Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate

    Bu, Z., Jin, X., Vinzamuri, B., Ramakrishna, A., Chang, K.-W., Cevher, V., and Hong, M. Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate. arXiv preprint arXiv:2410.22086, 2024

    arXiv

  7. [7]

    S., Tarun, A

    Chundawat, V. S., Tarun, A. K., Mandal, M., and Kankanhalli, M. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.\ 7210--7217, 2023

    work page 2023

  8. [8]

    F., Choquette-Choo, C

    Cooper, A. F., Choquette-Choo, C. A., Bogen, M., Jagielski, M., Filippova, K., Liu, K. Z., Chouldechova, A., Hayes, J., Huang, Y., Mireshghallah, N., et al. Machine unlearning doesn't do what you think: Lessons for generative ai policy, research, and practice. arXiv preprint arXiv:2412.06966, 2024

    arXiv 2024

Show all 47 references
  1. [9]

    GitHub , I

    Doe 1 v. GitHub , I. 4:22-cv-06823. N.D. Cal., 2022

    2022

  2. [10]

    and Russinovich, M

    Eldan, R. and Russinovich, M. Who's harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238, 2023

    2023 arXiv

  3. [11]

    Simplicity prevails: Rethinking negative preference optimization for llm unlearning

    Fan, C., Liu, J., Lin, L., Jia, J., Zhang, R., Mei, S., and Liu, S. Simplicity prevails: Rethinking negative preference optimization for llm unlearning. arXiv preprint arXiv:2410.07163, 2024

    2024

  4. [12]

    Influence diagnostics under self-concordance

    Fisher, J., Liu, L., Pillutla, K., Choi, Y., and Harchaoui, Z. Influence diagnostics under self-concordance. In International Conference on Artificial Intelligence and Statistics, pp.\ 10028--10076. PMLR, 2023

    2023

  5. [13]

    Practical unlearning for large language models

    Gao, C., Wang, L., Weng, C., Wang, X., and Zhu, Q. Practical unlearning for large language models. arXiv preprint arXiv:2407.10223, 2024

    2024 arXiv

  6. [14]

    The pile: An 800gb dataset of diverse text for language modeling

    Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  7. [15]

    An investigation into neural net optimization via hessian eigenvalue density

    Ghorbani, B., Krishnan, S., and Xiao, Y. An investigation into neural net optimization via hessian eigenvalue density. In International Conference on Machine Learning, pp.\ 2232--2241. PMLR, 2019

    2019

  8. [16]

    Ginart, A., Guan, M., Valiant, G., and Zou, J. Y. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019

    2019

  9. [17]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  10. [18]

    Jogging the memory of unlearned models through targeted relearning attacks

    Hu, S., Fu, Y., Wu, S., and Smith, V. Jogging the memory of unlearned models through targeted relearning attacks. In ICML 2024 Workshop on Foundation Models in the Wild, 2024

    2024

  11. [19]

    R., Liu, S., and Chang, S

    Ji, J., Liu, Y., Zhang, Y., Liu, G., Kompella, R. R., Liu, S., and Chang, S. Reversing the forget-retain objectives: An efficient llm unlearning framework from logit difference. arXiv preprint arXiv:2406.08607, 2024

    2024 arXiv

  12. [20]

    and Liu, Y

    Klochkov, Y. and Liu, Y. Revisiting inverse hessian vector products for calculating influence functions. arXiv preprint arXiv:2409.17357, 2024

    2024 arXiv

  13. [21]

    Koh, P. W. and Liang, P. Understanding black-box predictions via influence functions. In International conference on machine learning, pp.\ 1885--1894. PMLR, 2017

    2017

  14. [22]

    Limitations of the empirical fisher approximation for natural gradient descent

    Kunstner, F., Hennig, P., and Balles, L. Limitations of the empirical fisher approximation for natural gradient descent. Advances in neural information processing systems, 32, 2019

    2019

  15. [23]

    D., Dombrowski, A.-K., Goel, S., Phan, L., et al

    Li, N., Pan, A., Gopal, A., Yue, S., Berrios, D., Gatti, A., Li, J. D., Dombrowski, A.-K., Goel, S., Phan, L., et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218, 2024

    2024 arXiv

  16. [24]

    Y., Wang, Y., Flanigan, J., and Liu, Y

    Liu, C. Y., Wang, Y., Flanigan, J., and Liu, Y. Large language model unlearning via embedding-corrupted prompts. arXiv preprint arXiv:2406.07933, 2024

    2024 arXiv

  17. [25]

    Fixing weight decay regularization in adam

    Loshchilov, I., Hutter, F., et al. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101, 5: 0 5, 2017

    2017 arXiv

  18. [26]

    Unlearning via RMU is mostly shallow

    MacLaurin, D. Unlearning via RMU is mostly shallow. https://www.lesswrong.com/posts/6QYpXEscd8GuE7BgW/unlearning-via-rmu-is-mostly-shallow, 2024. Accessed: 2024-01-31

    2024

  19. [27]

    C., and Kolter, J

    Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z. C., and Kolter, J. Z. Tofu: A task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06121, 2024

    2024 arXiv

  20. [28]

    New insights and perspectives on the natural gradient method

    Martens, J. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21 0 (146): 0 1--76, 2020

    2020

  21. [29]

    Nemirovskij, A. S. and Yudin, D. B. Problem complexity and method efficiency in optimization. 1983

    1983

  22. [30]

    Polyak, B. T. Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics, 4 0 (5): 0 1--17, 1964

    1964

  23. [31]

    Schraudolph, N. N. Fast curvature matrix-vector products for second-order gradient descent. Neural computation, 14 0 (7): 0 1723--1738, 2002

    2002

  24. [32]

    Detecting pretraining data from large language models

    Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, L. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789, 2023

    2023 arXiv

  25. [33]

    A., and Zhang, C

    Shi, W., Lee, J., Huang, Y., Malladi, S., Zhao, J., Holtzman, A., Liu, D., Zettlemoyer, L., Smith, N. A., and Zhang, C. MUSE: Machine unlearning six-way evaluation for language models . arXiv preprint arXiv:2407.06460, 2024

    2024 arXiv

  26. [34]

    and Valpola, H

    Tarvainen, A. and Valpola, H. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017

    2017

  27. [35]

    S., and Smith, V

    Thaker, P., Hu, S., Kale, N., Maurya, Y., Wu, Z. S., and Smith, V. Position: Llm unlearning benchmarks are weak measures of progress. arXiv preprint arXiv:2410.02879, 2024 a

    2024 arXiv

  28. [36]

    Guardrail baselines for unlearning in llms

    Thaker, P., Maurya, Y., and Smith, V. Guardrail baselines for unlearning in llms. arXiv preprint arXiv:2403.03329, 2024 b

    2024 arXiv

  29. [37]

    Unrolling sgd: Understanding factors influencing machine unlearning

    Thudi, A., Deza, G., Chandrasekaran, V., and Papernot, N. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pp.\ 303--319. IEEE, 2022

    2022

  30. [38]

    OpenAI , I

    Tremblay v. OpenAI , I. 23-cv-03416-amo. N.D. Cal., 2023

    2023

  31. [39]

    Y., Pang, J., Liu, Q., Shah, A

    Wang, Y., Wei, J., Liu, C. Y., Pang, J., Liu, Q., Shah, A. P., Bao, Y., Liu, Y., and Wei, W. Llm unlearning via loss adjustment with only forget data. arXiv preprint arXiv:2410.11143, 2024

    2024 arXiv

  32. [40]

    Gru: Mitigating the trade-off between unlearning and retention for large language models

    Wang, Y., Wang, Q., Liu, F., Huang, W., Du, Y., Du, X., and Han, B. Gru: Mitigating the trade-off between unlearning and retention for large language models. arXiv preprint arXiv:2503.09117, 2025

    2025 arXiv

  33. [41]

    Large language model unlearning

    Yao, Y., Xu, X., and Liu, Y. Large language model unlearning. arXiv preprint arXiv:2310.10683, 2023

    2023 arXiv

  34. [42]

    Zhang, G., Li, L., Nado, Z., Martens, J., Sachdeva, S., Dahl, G., Shallue, C., and Grosse, R. B. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. Advances in neural information processing systems, 32, 2019

    2019

  35. [43]

    Negative preference optimization: From catastrophic collapse to effective unlearning

    Zhang, R., Lin, L., Bai, Y., and Mei, S. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868, 2024 a

    2024 arXiv

  36. [44]

    Catastrophic failure of LLM unlearning via quantization

    Zhang, Z., Wang, F., Li, X., Wu, Z., Tang, X., Liu, H., He, Q., Yin, W., and Wang, S. Catastrophic failure of LLM unlearning via quantization. arXiv preprint arXiv:2410.16454, 2024 b

    2024 arXiv

  37. [45]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  38. [46]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  39. [47]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.