Pith. sign in

REVIEW 4 major objections 5 minor 46 references

Towards Evaluation for Real-World LLM Unlearning

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that LLM unlearning can be evaluated without the retrained model by correcting a Kolmogorov-Smirnov statistic over core-token confidence scores using a validation set.

desk verdict A useful critique of existing unlearning metrics with a novel evaluation pipeline, but the load-bearing distribution-correction heuristic is unproven and the KS-based p-value is statistically unjustified. read the letter →

arxiv 2508.01324 v1 pith:ACNU46FF submitted 2025-08-02 cs.AI

Ke Miao , Yuke Hu , Xiaochen Li , Wenjie Bao , Zhihao Liu , Zhan Qin , Kui Ren This is my paper
classification cs.AI
keywords LLMunlearningevaluationKolmogorov-Smirnovtestcoretokenconfidencescoresdistributioncorrectionvalidationsetmachinemetricsretrainedmodelindependence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's aim is to make LLM unlearning verifiable where it actually happens: in deployed models that no one will retrain just to check them. It argues that existing evaluation metrics fail on three counts: several need the retrained model as a gold standard, text-similarity and multiple-choice scores can be gamed by irrelevant wording or general reasoning, and scores shift after unrelated updates. To fix this, the paper proposes DCUE, which scores only confidence assigned to answer-bearing "core tokens", corrects the difference between the original and unlearned models using a validation set drawn from the fine-tuning domain, and reports a Kolmogorov-Smirnov p-value. Across multiple model architectures and datasets, DCUE achieves the paper's own ideals of practicality, exactness, and stability under post-processing. It also finds that six current unlearning methods, GA, GD, IDK, DPO, NPO, and SimNPO, remain far from the level of a truly retrained model.

What carries the argument

The load-bearing object is the corrected Kolmogorov-Smirnov statistic on core-token confidence scores (CTCS), where CTCS are the next-token probabilities a model gives to the minimal subset of tokens that actually carry an answer. The paper builds empirical cumulative distribution functions of CTCS for the original and unlearned models on both the forget set and a validation set, computes the KS statistics $S^f_{o,u}$ and $S^v_{o,u}$, and estimates the retained-data shift as $\delta_S \approx \min\{S^v_{o,u}, S^f_{o,u}\}$. That estimate is subtracted from $S^f_{o,u}$ to form $S_{\mathrm{corr}}$, and the p-value of $S_{\mathrm{corr}}$ is the evaluation result.

What would settle it

Take a case where the retained data shifts the model on the validation set in a direction opposite to its effect on the forget set, compute DCUE's corrected p-values, and compare them with p-values obtained from a genuinely retrained model; if the two disagree sharply, the approximation $\delta_S \approx \min\{S^v_{o,u}, S^f_{o,u}\}$ has failed.

Watch

Extended reading notes

Core claim

DCUE's central claim is that the unattainable comparison against a retrained model $M_r$ can be replaced by a corrected comparison against the original open-source model $M_o$, as long as a validation set $D_v$ from the fine-tuning domain is available and unused. On the forget set $D_f$, DCUE compares the empirical cumulative distribution functions of core-token confidence scores for $M_o$ and the unlearned model $M_u$; on $D_v$, the same comparison estimates the distributional shift produced by fine-tuning on the retained data. Subtracting the smaller of the two observed shifts from the forget-set shift gives a corrected statistic, $$S_{\mathrm{corr}} = S^f_{o,u} - \min\{S^v_{o,u}, S^f_{o,u}\},$$ and the p-value from the two-sample Kolmogorov-Smirnov test on this statistic is the unlearning score. The paper reports that on its test cases this metric assigns the best possible score to a retrained model and the worst to an untouched model, stays stable under further unlearning and fine-tuning, and does not require access to $M_r$.

Load-bearing premise

The load-bearing bet is that the shift caused by fine-tuning on the kept data is captured by the smaller of two directly observable shifts, and that a validation set from the fine-tuning domain is available but unused; if either fails, the corrected score is unreliable.

Editorial extensions

If this is right

  • Data owners and auditors can check whether a model forgot requested data using only the original model, a validation set, and the unlearned model, with no retraining required.
  • Evaluations can avoid being fooled by non-answer wording, because only probabilities on answer-bearing tokens enter the score.
  • Unlearning methods can be compared by a score that stays stable when the model is later fine-tuned on unrelated data or unlearns other samples.
  • Under DCUE, current unlearning methods look far from complete, so reported unlearning should be treated as partial until proven otherwise.
  • Future unlearning algorithms can use DCUE's p-value as a deployment-time signal or acceptance criterion rather than depending on benchmark-only comparisons.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same validation-set correction could be applied to other leakage signals, such as membership-inference scores or hidden-state probes, to make them independent of a retrained model.
  • Beyond the paper: a natural hardening of the method is to require p-value stability across multiple validation draws, since the current score is a single p-value derived from one validation split.
  • Beyond the paper: core-token extraction currently relies on an auxiliary language model, so replacing that step with a deterministic extractor would make DCUE fully auditable.
  • Beyond the paper: the method's assumption that validation data resembles retained data suggests that in settings where the fine-tuning split is unknown, cross-validation over the forget set itself could serve as a fallback.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that existing LLM unlearning evaluation metrics (text similarity, multiple-choice accuracy, prediction probability, MIA) are impractical, inexact, or non-robust in real-world settings, and proposes a new metric called DCUE. DCUE extracts core-token confidence scores (CTCS) via LLM prompts, uses a validation set Dv to correct distributional drift caused by retained data, and then applies a Kolmogorov-Smirnov test to a corrected statistic Scorr defined in Eq. (8). Experiments on Phi-1.5B, LLaMA2-7B, and Qwen2.5-7B with TOFU and MUSE-News claim that DCUE achieves perfect practicality, exactness, and robustness scores, and that existing unlearning methods (GA, GD, IDK, DPO, NPO, SimNPO) remain far from complete unlearning.

Significance. If the statistical construction were sound, DCUE would be a practically valuable contribution: it removes the dependence on an inaccessible retrained model, addresses the known token-level noise of text-similarity metrics, and offers a unified protocol for third-party verification of unlearning. The paper's critique of existing metrics in Section 3.2 is clear and well-illustrated, and the manual validation of core-token extraction (97% and 96% precision across models) is a concrete strength. However, the central approximation in Eq. (6) is heuristic, the corrected statistic Scorr is not a two-sample KS statistic, and the validation experiments cover only the favorable endpoints Mt and Mr. These are load-bearing gaps: until the null distribution of Scorr is established and the approximation is tested on intermediate unlearned models, the claimed exactness and the quantitative p-values in Table 3 are not reliable. The paper is therefore promising but requires substantial revision.

major comments (4)
  1. [§4.3, Eq. (8)] The min correction makes Scorr identically zero whenever Sv_o,u ≥ Sf_o,u, so DCUE then reports p = 1 and certifies complete unlearning regardless of Mu's actual internal state. This is not a remote edge case: if Dv is drawn from data close to Dr, fine-tuning on Dr can shift the model's validation-set CTCS more than memorization of Df shifts its forget-set CTCS, giving Sv_o,t > Sf_o,t for a model that has not unlearned at all. The validation in §4.5 only tests Mu = Mt and Mu = Mr, the two endpoints for which the approximation is most favorable, and does not include intermediate unlearned models such as those in Table 3. The exactness+ and exactness− scores of 1.0000 in Table 2 therefore do not establish exactness for the general case the metric is designed for.
  2. [§4.4, Appendix A] Feeding Scorr into the two-sample KS survival function is not justified. Scorr = Sf_o,u − min{Sv_o,u, Sf_o,u} is a min-corrected difference of two KS statistics, not a KS statistic computed from the maximum difference between two ECDFs. The null distribution of the two-sample KS statistic, used in Eq. (9) and Appendix A, does not apply to this corrected statistic. Consequently, the values RDCUE reported in Table 3 and the approximate p-values in Figure 5 are not calibrated p-values. The paper needs either a derived null distribution for Scorr or a permutation/bootstrap calibration; without that, the quantitative comparisons among unlearning methods are not statistically interpretable.
  3. [§4.5] The approximation δS ≈ min{Sv_o,u, Sf_o,u} in Eq. (6) is introduced purely as a heuristic ('The intuition behind this approximation is as follows...') and is not derived from any distributional assumption. The numerical validation is limited to two models, one dataset (TOFU), 100 draws of Dv, and only two choices of Mu (Mt and Mr); no error bars or sensitivity analyses are reported. This is a load-bearing point because the entire distribution-correction mechanism and every subsequent exactness claim depend on Eq. (6). At minimum, the authors should validate the approximation on the intermediate unlearned models studied in Table 3 and on Dv distributions that are closer to Dr than to Df.
  4. [§5.1, Table 2] The robustness experiments use Mr as Mu ('Considering Mr represents the gold standard... we use Mr as Mu in subsequent experiment of robustness'). Mr is already perfectly unlearned, so post-processing it with unlearning on other data or fine-tuning on new data is not representative of applying post-processing to a partially unlearned model from Table 3. This weakens the claim that DCUE is robust in realistic deployment, where the model being evaluated will typically be in an intermediate unlearning state. Additionally, the fact that DCUE's robustness scores are exactly 1.0000 in every configuration (Tables 2 and 4) suggests possible saturation rather than meaningful measurement of robustness; the authors should report experiments where the metric is expected to change and show that it tracks those changes.
minor comments (5)
  1. [§4.3] The statement 'It is uncommon for δS to equal Sf_o,u' is not quantified or supported; the paper should provide the empirical frequency of this event in the §4.5 simulation.
  2. [§5.2] The ablation results in Figures 6 and 7 are reported as single deterministic numbers without error bars or repeated runs; adding variability measures would strengthen the claim that the observed differences are meaningful.
  3. [Table 3] The p-value formatting is inconsistent: some rows use scientific notation without parentheses (e.g., Mt and Mr), while others use parentheses with mixed forms such as '6.88e2' and '1.55e7'; the table should use a uniform notation and define the base value for the multiples.
  4. [General] There are typographical and notation issues: 'PostProul' appears to be a typo for 'PostPro_ul'; 'CT CSd m' should be typeset as a single variable; and 'Qween' in Table 4 should be 'Qwen'.
  5. [§4.2] The core-token extraction precision is manually reviewed on only the first 200 tokens for each dataset; reporting inter-annotator agreement or confidence intervals would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DCUE's min-correction is an explicit, empirically checked heuristic rather than a hidden identity, and no load-bearing self-citation is present.

full rationale

The paper's central construction is Eq. (5)–(8): it defines the retrained-model comparison Sf_r,u = Sf_o,u − δS and then approximates δS by min{Sv_o,u, Sf_o,u}. This is an explicit proxy, not a hidden identity: Scorr is a deterministic function of the two observable KS statistics and is directly checked against Mr-based p-values in §4.5. The fact that Scorr = 0 whenever Sv_o,u ≥ Sf_o,u means the metric can saturate at 'complete unlearning' for models whose forget-set distribution is close to the original model; that is a validity/exactness limitation of the approximation, not a circular derivation, because the paper does not claim to derive the approximation from first principles. The §4.5 validation covers only Mt and Mr endpoints on TOFU, so the exactness and ranking claims for intermediate unlearned models (Table 3) are not fully established; again, this is an evidentiary gap rather than circularity. No uniqueness theorem from the authors is invoked, and the only prior work by the authors in the reference list ([13]) appears in a general enumeration of unlearning methods and is not load-bearing for DCUE. Thus no step in the derivation chain reduces by construction to its own inputs.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central results rest on one hand-chosen correction rule and several domain assumptions about data access and token semantics. No fitted numeric parameters are introduced, but the ad hoc approximation in Eq. (6) is the main unvalidated load-bearing element.

assumptions (5)
  • ad hoc to paper The distributional shift delta_S caused by retained data Dr on an unseen dataset is approximated by min{Sv_{o,u}, Sf_{o,u}}.
    Eq. (6) in Section 4.3. This is a heuristic with no derivation and is the core of the distribution correction; the paper only provides intuition and a limited simulation.
  • domain assumption A validation set Dv sampled from the fine-tunable dataset and excluded from fine-tuning can estimate the shift caused by Dr on Df.
    Section 4.1 and 4.3. If Dv does not match the distribution of Dt, the correction may be biased; the paper claims a strict match is not required, but this is not substantiated.
  • domain assumption Core token confidence scores (CTCS) reflect the model's retention of target knowledge.
    Section 4.1. Relies on the LLM-based core token extraction being accurate; manual check on 200 tokens shows 96-97% precision, but extraction is stochastic and this precision may not hold at scale.
  • standard math The KS test p-value based on Scorr and sample size |CTCSf_u| is a valid quantification of unlearning degree.
    Section 4.4. The KS test is standard, but applying it to a corrected statistic requires independence assumptions that are not discussed.
  • domain assumption The original open-source model Mo is publicly accessible to the evaluator.
    Introduction and Section 4.1. A real-world data owner requesting deletion may only have access to the deployed Mu, not to Mo or a validation set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Evaluation for Real-World LLM Unlearning." pith.science (2026). https://pith.science/paper/ACNU46FF

@misc{pith2026250801324,
  author       = {Pith},
  title        = {Pith review of: Towards Evaluation for Real-World LLM Unlearning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ACNU46FF}},
  note         = {Machine review of arXiv:2508.01324}
}
read the original abstract

This paper analyzes the limitations of existing unlearning evaluation metrics in terms of practicality, exactness, and robustness in real-world LLM unlearning scenarios. To overcome these limitations, we propose a new metric called Distribution Correction-based Unlearning Evaluation (DCUE). It identifies core tokens and corrects distributional biases in their confidence scores using a validation set. The evaluation results are quantified using the Kolmogorov-Smirnov test. Experimental results demonstrate that DCUE overcomes the limitations of existing metrics, which also guides the design of more practical and reliable unlearning algorithms in the future.

Figures

Figures reproduced from arXiv: 2508.01324 by the authors.

Figure 1
Figure 1. Example illustrating the limitation of Evaluation based on Text Similarity. Q: How has Yun’s father influenced her leadership works? A. It discourages her from writing. B. It focuses solely on technical skills. C. It provides practical examples of leadership. D. It has no influence on her works [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Blueprint of ideal metric in real-world settings. It contains three key properties: Practicality [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. The workflow of DCUE. It first extracts CTCS of [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Validation of the approximation strategy on Phi-1.5B and LLaMA2-7B models. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Experimental results of ablation on Phi-1.5B model. practicality exactness+ exactness− robustnessul robustnessft robustnessmix 0.0 0.2 0.4 0.6 0.8 1.0 score DCUE No core token No validate dataset [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 16 canonical work pages

  1. [1]

    Sulla determinazione empirica di una legge didistribuzione

    Kolmogorov An. Sulla determinazione empirica di una legge didistribuzione. Giorn Dell’inst Ital Degli Att, 4:89–91, 1933

    work page 1933

  2. [2]

    Machine unlearning

    Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141–159. IEEE, 2021

    2021

  3. [3]

    Extracting training data from large language models

    Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-V oss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650, 2021

    2021

  4. [4]

    Unlearn what you want to forget: Efficient unlearning for llms, 2023

    Jiaao Chen and Diyi Yang. Unlearn what you want to forget: Efficient unlearning for llms, 2023. URL https://arxiv.org/abs/2310.20150

    arXiv 2023

  5. [5]

    Simplicity prevails: Rethinking negative preference optimization for llm unlearning, 2024

    Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. Simplicity prevails: Rethinking negative preference optimization for llm unlearning, 2024. URL https://arxiv.org/abs/2410.07163

    arXiv 2024

  6. [6]

    Lawbench: Benchmarking legal knowledge of large language models

    Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. Lawbench: Benchmarking legal knowledge of large language models. arXiv preprint arXiv:2309.16289, 2023

    arXiv 2023

  7. [7]

    Ticketed learning–unlearning schemes

    Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Ayush Sekhari, and Chiyuan Zhang. Ticketed learning–unlearning schemes. In The Thirty Sixth Annual Conference on Learning Theory, pages 5110–5139. PMLR, 2023

    work page 2023

  8. [8]

    Making ai forget you: Data deletion in machine learning

    Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019

    2019

Show all 46 references
  1. [9]

    Adaptive machine unlearning

    Varun Gupta, Christopher Jung, Seth Neel, Aaron Roth, Saeed Sharifi-Malvajerdi, and Chris Waites. Adaptive machine unlearning. In M. Ranzato, A. Beygelzimer, Y . Dauphin, P.S. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information Processing Systems, volume 34, ...

    2021

  2. [10]

    Fantastic copyrighted beasts and how (not) to generate them

    Luxi He, Yangsibo Huang, Weijia Shi, Tinghao Xie, Haotian Liu, Yue Wang, Luke Zettlemoyer, Chiyuan Zhang, Danqi Chen, and Peter Henderson. Fantastic copyrighted beasts and how (not) to generate them. arXiv preprint arXiv:2406.14526, 2024

    2024 arXiv

  3. [11]

    Foundation models and fair use

    Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A Lemley, and Percy Liang. Foundation models and fair use. arXiv preprint arXiv:2303.15715, 2023. 10

    2023 arXiv

  4. [12]

    Jogging the memory of unlearned llms through targeted relearning attacks, 2024

    Shengyuan Hu, Yiwei Fu, Zhiwei Steven Wu, and Virginia Smith. Jogging the memory of unlearned llms through targeted relearning attacks, 2024. URL https://arxiv.org/abs/ 2406.13356

    2024 arXiv

  5. [13]

    Eraser: Machine unlearning in mlaas via an inference serving-aware approach

    Yuke Hu, Jian Lou, Jiaqi Liu, Feng Lin, Zhan Qin, and Kui Ren. Eraser: Machine unlearning in mlaas via an inference serving-aware approach. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS ’24, page 3883–3897, 2024

    2024

  6. [14]

    Editing models with task arithmetic, 2023

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic, 2023

    2023

  7. [15]

    Knowledge sanitization of large language models,

    Yoichi Ishibashi and Hidetoshi Shimodaira. Knowledge sanitization of large language models,

  8. [16]

    Approximate data deletion from machine learning models

    Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pages 2008–2016. PMLR, 2021

    2008

  9. [17]

    Knowledge unlearning for mitigating privacy risks in language models

    Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint arXiv:2210.01504, 2022

    2022 arXiv

  10. [18]

    Rwku: Benchmarking real-world knowledge unlearning for large language models, 2024

    Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. Rwku: Benchmarking real-world knowledge unlearning for large language models, 2024. URL https://arxiv.org/abs/2406.10890

    2024 arXiv

  11. [19]

    Privacy adhering machine un-learning in nlp, 2022

    Vinayshekhar Bannihatti Kumar, Rashmi Gangadharaiah, and Dan Roth. Privacy adhering machine un-learning in nlp, 2022. URL https://arxiv.org/abs/2212.09573

    2022 arXiv

  12. [20]

    Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm- Burger, Rassin Lababidi, Lennart Justen, Andrew B

    Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm- Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang...

    2024 arXiv

  13. [21]

    Large language models in finance: A survey

    Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. Large language models in finance: A survey. In Proceedings of the fourth ACM international conference on AI in finance , pages 374–382, 2023

    2023

  14. [22]

    Continual learning and private unlearning, 2022

    Bo Liu, Qiang Liu, and Peter Stone. Continual learning and private unlearning, 2022

    2022

  15. [23]

    Federated unlearning

    Gaoyang Liu, Xiaoqiang Ma, Yang Yang, Chen Wang, and Jiangchuan Liu. Federated unlearning. arXiv preprint arXiv:2012.13891, 2020

    2012 arXiv

  16. [24]

    Quark: Controllable text generation with reinforced unlearning

    Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591–27609, 2022

    2022

  17. [25]

    Eight methods to evaluate robust unlearning in llms, 2024

    Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight methods to evaluate robust unlearning in llms, 2024. URL https://arxiv.org/abs/2402. 16835

    2024

  18. [26]

    Zico Kolter

    Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and J. Zico Kolter. Tofu: A task of fictitious unlearning for llms. ArXiv, abs/2401.06121, 2024. URL https: //api.semanticscholar.org/CorpusID:266933371. 11

    2024 arXiv

  19. [27]

    Silo language models: Isolating legal risk in a nonparametric datastore

    Sewon Min, Suchin Gururangan, Eric Wallace, Weijia Shi, Hannaneh Hajishirzi, Noah A Smith, and Luke Zettlemoyer. Silo language models: Isolating legal risk in a nonparametric datastore. arXiv preprint arXiv:2308.04430, 2023

    2023 arXiv

  20. [28]

    Can sensitive information be deleted from llms? objectives for defending against extraction attacks, 2023

    Vaidehi Patil, Peter Hase, and Mohit Bansal. Can sensitive information be deleted from llms? objectives for defending against extraction attacks, 2023. URL https://arxiv.org/abs/ 2309.17410

    2023 arXiv

  21. [29]

    In-context unlearning: Language models as few shot unlearners

    Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.07579, 2023

    2023 arXiv

  22. [30]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2023

    2023

  23. [31]

    Remember what you want to forget: Algorithms for machine unlearning

    Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, 34:18075–18086, 2021

    2021

  24. [32]

    Smith, and Chiyuan Zhang

    Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models, 2024. URL https://arxiv.org/abs/2407.06460

    2024 arXiv

  25. [33]

    N. W. Smirnoff. On the estimation of the discrepancy between empirical curves of distribution for two independent samples. Bulletin de l’Université de Moscou, 1939

    1939

  26. [34]

    Chatgpt: Optimizing language models for dialogue

    OpenAI Team. Chatgpt: Optimizing language models for dialogue. https://openai.com/ blog/chatgpt, 2022. Accessed: 2023-08-20

    2022

  27. [35]

    Guardrail baselines for unlearning in llms, 2024

    Pratiksha Thaker, Yash Maurya, Shengyuan Hu, Zhiwei Steven Wu, and Virginia Smith. Guardrail baselines for unlearning in llms, 2024. URL https://arxiv.org/abs/2403. 03329

    2024

  28. [36]

    Large language models in medicine.Nature medicine, 29(8):1930–1940, 2023

    Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine.Nature medicine, 29(8):1930–1940, 2023

    1930

  29. [37]

    Towards efficient and effective unlearning of large language models for recommendation,

    Hangyu Wang, Jianghao Lin, Bo Chen, Yang Yang, Ruiming Tang, Weinan Zhang, and Yong Yu. Towards efficient and effective unlearning of large language models for recommendation,

  30. [38]

    Depn: Detecting and editing privacy neurons in pretrained language models

    Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. Depn: Detecting and editing privacy neurons in pretrained language models. arXiv preprint arXiv:2310.20138, 2023

    2023 arXiv

  31. [39]

    URL https://arxiv.org/abs/2403.03536

    arXiv

  32. [40]

    Evaluating reading comprehension exercises generated by llms: A showcase of chatgpt in education applications

    Changrong Xiao, Sean Xin Xu, Kunpeng Zhang, Yufang Wang, and Lei Xia. Evaluating reading comprehension exercises generated by llms: A showcase of chatgpt in education applications. In Proceedings of the 18th workshop on innovative use of NLP for building educational applicatio...

    2023

  33. [41]

    Deltagrad: Rapid retraining of machine learning models

    Yinjun Wu, Edgar Dobriban, and Susan Davidson. Deltagrad: Rapid retraining of machine learning models. In International Conference on Machine Learning , pages 10355–10366. PMLR, 2020

    2020

  34. [42]

    Composing parameter-efficient modules with arithmetic operations, 2023

    Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He. Composing parameter-efficient modules with arithmetic operations, 2023. URL https://arxiv.org/abs/2306.14870

    2023 arXiv

  35. [43]

    Unlearning bias in language models by partitioning gradients

    Charles Yu, Sullam Jeoung, Anish Kasi, Pengfei Yu, and Heng Ji. Unlearning bias in language models by partitioning gradients. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6032–6048, 2023

    2023

  36. [44]

    For the given blank filling question ‘{blankfilling_question}’, the reference material is ‘{answer}’. Your response should only include answers separated by spaces:

    Zhiwei Zhang, Fali Wang, Xiaomin Li, Zongyu Wu, Xianfeng Tang, Hui Liu, Qi He, Wenpeng Yin, and Suhang Wang. Does your llm truly unlearn? an embarrassingly simple approach to recover unlearned knowledge, 2024. URL https://arxiv.org/abs/2410.16454. 12 A Kolmogorov-Smirnov Test ...

    2024 arXiv

  37. [45]

    Negative preference optimization: From catastrophic collapse to effective unlearning, 2024

    Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning, 2024

    2024

  38. [2024]

    URL https://arxiv.org/abs/2309.11852

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.