Unlearning vs. Obfuscation: Are We Truly Removing Knowledge?

Guangzhi Sun; Mark Gales; Potsawee Manakul; Xiao Zhan

REVIEW 4 major objections 5 minor 1 cited by

Unlearning vs. Obfuscation: Are We Truly Removing Knowledge?

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes DF-MCQ, which flattens answer distributions over multiple-choice questions, and claims this achieves genuine knowledge removal rather than obfuscation.

desk verdict A useful probing framework and a cheap refusal-inducing method, but the paper overclaims that DF-MCQ truly removes knowledge when the evidence is largely consistent with a name-triggered suppression shortcut. read the letter →

arxiv 2505.02884 v2 pith:G5Q2WRPQ submitted 2025-05-05 cs.LG cs.AI

Guangzhi Sun , Potsawee Manakul , Xiao Zhan , Mark Gales This is my paper · ORCID

classification cs.LGcs.AI

keywords machineunlearninglargelanguagemodelsobfuscationknowledgeremovalprobingevaluationdistributionflatteningmultiple-choicequestionsprivacy

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Many LLM unlearning methods are really obfuscation: they add wrong or irrelevant facts to suppress the target knowledge, leaving the original association in place. The paper defines unlearning as the uncertainty state of a model that never saw the fact, with refusal behaviour ("I do not have information") as its observable signature, and builds probing questions, Yes-No and multiple choice, that expose whether a method merely added distracting edges. It then introduces DF-MCQ, which fine-tunes the model so its probability over MCQ answer choices about the target person is flat, while a retain loss preserves other knowledge. On the WPU benchmark, DF-MCQ reaches 92.72% refusal on open-ended questions and maximum five-choice entropy of 1.61 on every MCQ probing subset, which the paper presents as evidence that the knowledge itself has been removed. If correct, this gives privacy-sensitive applications a way to test for true unlearning instead of trusting surface-level evasion.

What carries the argument

The central mechanism is distribution flattening over a confined MCQ output space. Each automatically generated question $X_i$ has a small set of answer letters $c$; DF-MCQ minimizes $L_{\text{unlearn}} = \sum_i D_{\text{KL}}[P_\theta(c|X_i) \,\|\, \hat P(c|X_i)]$ with $\hat P$ uniform, which in a space whose probabilities sum to almost one is equivalent to flattening the whole relevant output distribution, and combines it with $L_{\text{retain}} = \sum_j D_{\text{KL}}[P_\theta(c|X_j) \,\|\, P_{\theta_{\text{orig}}}(c|X_j)]$ to prevent a global uniform-output shortcut. The evaluation machinery is the probing framework itself: open-ended refusal rate, Yes-No entropy and accuracy on reference/in-training/out-of-training splits, and MCQ accuracy and entropy, which together are designed to reveal whether an unlearning method has removed the edge or just added distractors.

What would settle it

Take a DF-MCQ-unlearned model and ask about the target without using the name, e.g. "Where was the 19th-century German historian born who wrote about palaeography?" or paraphrase the MCQ; if the correct answer resurfaces, the knowledge was suppressed, not removed. The same test on a model genuinely retrained without the target passages would stay silent, and the behaviour difference would settle whether DF-MCQ truly unlearns.

Watch

Extended reading notes

Core claim

The paper's central claim is that DF-MCQ removes targeted personal knowledge rather than masking it. Viewing model knowledge as a graph of facts (subject, relation, object), the paper defines genuine unlearning by the equality $H_\theta(Y_i|X_i,R_i;\mathcal{D}) \approx H_\theta(Y_i|X_i,R_i;\mathcal{D}\setminus F_i)$: the model should be exactly as uncertain as one trained without the fact, and should therefore refuse open-ended questions rather than answer them. Obfuscation methods fail this definition because they only add competing edges; the original edge still exists, so probes that rule out the distractors recover the truth. DF-MCQ avoids adding edges by minimizing $D_{\text{KL}}[P_\theta(c|X_i)\,\|\,\hat{P}(c|X_i)]$ over automatically generated MCQs, pushing the choice-letter distribution to uniform, and pairs this with a retain loss on MCQs about other people so the flattening does not become a global shortcut. The reported behaviour, 92.72% refusal on open-ended forget-set questions, 77.6% accuracy on reference Yes-No probes with 0.65 entropy, and the maximum 1.61 entropy on every MCQ subset, supports the paper's conclusion that the model treats questions about the target as unanswerable, which is what unlearning should look like.

Load-bearing premise

The evaluation assumes that high output entropy and refusal behaviour prove the knowledge is gone; the paper does not rule out the model simply learning a shortcut that makes it say "I do not know" or answer uniformly whenever the target person's name appears, while the underlying fact remains stored.

Editorial extensions

If this is right

Open-ended ROUGE-L scores alone cannot certify unlearning; evaluations should include Yes-No and MCQ probes because these expose whether the original fact is still reachable.
A model unlearned with DF-MCQ will refuse instead of producing a plausible wrong answer, avoiding the new false facts that obfuscation methods inject into the knowledge graph.
The forget-set behaviour survives a benign SFT attack and continual unlearning of additional targets, so the method can be used incrementally.
DF-MCQ transfers across instruction-tuned base models: Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct show similar refusal rates, 92.72% and 88.17%, and low forget-set ROUGE-L.
Obfuscation methods' privacy protection degrades on MCQ probes, where WHP+ still reaches 36.73% accuracy and sometimes nearly matches the original model; DF-MCQ drops to 18.86% with maximum entropy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported exact uniform entropy of 1.61 on every MCQ subset is also exactly what a target-name-triggered shortcut would produce; behavioural flattening does not by itself prove the internal fact representation was deleted.
A decisive test would compare DF-MCQ with a model retrained without the target passages, probing both with paraphrased or name-free questions, or with activation-level probes; until then, "removal" is an interpretation of the behavioural evidence.
If the shortcut reading is correct, DF-MCQ is a stronger form of suppression rather than a different kind of operation, consistent with the paper's own remark that there is no clear boundary between obfuscation and unlearning.
The same flattening idea might extend to non-person facts, but the paper's limitations list restricts the current evidence to person-centric facts and models under 10B parameters, so generalization to multi-hop or procedural knowledge is untested.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should look at this paper if you work on LLM unlearning. The probing framework is the real contribution: three question types (open-ended, yes/no, MCQ) that separate hiding from removal, plus a clear demonstration that the obfuscation baseline WHP+ collapses into an always-“yes” shortcut on yes/no probes. That pattern, with the 0.84 correlation between obfuscation efficacy and yes-rate, is a solid empirical finding. The DF-MCQ method itself is simple and practical: flatten the model’s distribution over automatically generated MCQs with KL divergence, keep a retain loss on other people’s MCQs, and the model ends up refusing open-ended questions about the target. It is cheap (LoRA, 15 minutes on an A100) and works on two model families. I would use the probing suite in my own evaluations.

But the central claim—“effectively removing knowledge”—is not supported by the experiments. The two headline numbers are both direct consequences of the training objective. The 92.72% refusal rate is tuned: Section 6.2 says the learning rate is adjusted until the model refuses to answer. The MCQ entropy of 1.61 is exactly what Eq. (4) minimizes; and the fact that it is 1.61 across all five subsets suggests the model learned a simple policy of outputting uniform over the letters whenever the target name appears. That is suppression, not deletion. The paper never constructs the retrained oracle that Eq. (1) defines as the reference for unlearning, so we never see whether the entropy actually matches the “never trained on the fact” model. The SFT attack is also non-discriminating: it only fine-tunes on other individuals, so it cannot show whether the target association is recoverable. The absence of error bars is a minor issue relative to these.

The authors do acknowledge the shortcut concern implicitly by testing the retain set, which shows the flattening is largely tied to target names. That is consistent with a name-triggered shortcut rather than knowledge erasure. I think the paper would be strengthened substantially by one of three additions: a retrained oracle baseline, a relearning test on target facts, or a representation-level probe. As it stands, the paper is a good framework plus a useful method, with an overstated interpretation.

Verdict for peer review: yes, send it out. The probing framework and the DF-MCQ method are useful enough to merit referee time, but the authors should be pushed to either add the missing baselines or soften the removal claim to suppression. I would not cite the central claim; I would cite the probing suite and the WHP+ shortcut finding.

Referee Report

4 major / 5 minor

Summary. The paper argues that many existing LLM unlearning methods are better characterized as obfuscation, because they add distracting or false knowledge rather than removing the original fact. It proposes an evaluation framework with three probe types (open-ended, yes/no, and multiple-choice questions) and introduces DF-MCQ, a method that flattens the model's predictive distribution over automatically generated MCQs via a KL-divergence loss. Experiments on the WPU forget-2 benchmark with Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct report that DF-MCQ achieves a 92.72% refusal rate on open-ended forget-set questions, near-random entropy on yes/no and MCQ probes, and less degradation on retain sets than the obfuscation baselines NPO and WHP+. The paper concludes that DF-MCQ achieves true knowledge removal rather than obfuscation.

Significance. If the central claim is correct, the paper would make a useful contribution by sharpening the distinction between unlearning and obfuscation and by providing a concrete method with a low-cost training recipe and an open-source implementation. The probing framework is a reasonable step beyond open-ended ROUGE-L evaluation, and the decision to compare against WHP+ and NPO on a standard benchmark is appropriate. However, the main evidence for 'true removal' is currently weak: the headline refusal rate is a tuned training target, and the near-uniform MCQ entropy is the optimization objective itself. The paper's contribution is therefore best assessed as a promising evaluation framework and method proposal whose central empirical claim requires independent verification before it can support the strong conclusion stated in the abstract.

major comments (4)

[6.2 Model and Training] The reported refusal rate of 92.72% in Table 2 is not independent evidence of knowledge removal because the learning rate is explicitly 'adjusted such that the model refuses to answer.' This makes refusal a tuned training target rather than an emergent property of forgetting. The authors should either fix the training protocol in advance, report the tuning process and its variance, or provide a separate test where refusal is not optimized for.
[7.3 MCQ Probing Questions and Eq. (4)] The near-uniform MCQ entropy in Table 7 and the exact value 1.61 for every subset in Table 10 is precisely what the DF-MCQ loss in Eq. (4) minimizes, so this entropy cannot serve as independent evidence of true removal. The paper needs a diagnostic that is not the training objective itself, such as a retrained-oracle baseline from Eq. (1), a representation-level probe, or a relearning/attack experiment on the target facts.
[3 Unlearning and Obfuscation, Eq. (1)] The formal definition of unlearning in Eq. (1) requires comparing the unlearned model with a model trained on D\F, but no such retrained oracle is ever constructed or evaluated in the experiments. Without this baseline, the claim that DF-MCQ achieves 'the same level of uncertainty as one trained on the dataset excluding the fact' is unsupported. Adding this baseline, or at least a lower-bound oracle, is necessary to substantiate the central claim.
[4 Distribution Flattening MCQ, Eq. (5)] The retain loss in Eq. (5) only preserves behavior on MCQs about other people, so it does not exclude a target-name-triggered shortcut: whenever the prompt contains the target person, the model could learn to output a refusal or a uniform distribution without deleting the underlying fact. The control in Section 7.2 with distractors from the MCQ training set reduces but does not eliminate this possibility. A control using target-name prompts that are unrelated to the unlearned facts, or a relearning test on the target facts, would help separate suppression from deletion.

minor comments (5)

[Tables 2–10] No error bars, confidence intervals, or repeated-run statistics are reported anywhere in the paper, despite the fact that the DF-MCQ learning rate is tuned to achieve refusal. Reporting variance across seeds or subsets would strengthen the empirical claims.
[7.1 Open-Ended Questions, Fig. 5] The Pearson correlation coefficient of 0.84 in Fig. 5 is reported without a p-value or sample-size information; given the small number of settings in Fig. 5(a), this should be interpreted cautiously or supplemented with a significance test.
[Figures and text] The same Figure 3 caption and example block appears twice in the manuscript, once after Section 5.2 and again inside Section 7.1. This duplicate should be removed.
[References] The reference list contains a duplicate: Xu et al. 2025a and Xu et al. 2025b are the same arXiv preprint 'Relearn: Unlearning via learning for large language models.' One entry should be removed.
[Table 5] The caption contains a typo: 'comapred' should be 'compared.'

Circularity Check

2 steps flagged · score 6.0 of 10

DF-MCQ's headline evidence is largely its own training objective: MCQ entropy is the DF-MCQ loss, and refusal behavior is hand-tuned via learning rate; the claimed 'random choice-level uncertainty' does not independently establish knowledge removal.

fitted input called prediction [Section 4, Eq. (4); Section 7.3, Table 7; Abstract]
"The unlearning loss is defined as Eqn. (4) below. Lunlearn = NX i=1 DKL h Pθ(c|Xi)|| ˆP(c|Xi) i ... Pθ is the output distribution over the choices and ˆP is the flat distribution over the choices ... DF-MCQ assigns almost equal probability to all options, subject to certain priors ... Therefore, compared to obfuscation, DF-MCQ better protects the privacy when a malicious query contains a range of options."

The headline 'random choice-level uncertainty' on MCQ probes is precisely the quantity optimized by Eq. (4), which minimizes KL divergence from the model's MCQ distribution to a flat distribution. Table 7's entropy 1.61 is the maximum entropy for 5 choices and is the direct fixed point of that objective, not a measurement of the retrained-model entropy Hθ(Y|X;D\F) required by Eq. (1). Since no model trained on D\F is ever constructed, the near-uniform probe entropy is the trained behavior itself rather than an independent test of knowledge deletion; the result reduces to the loss by construction.
fitted input called prediction [Section 6.2; Section 7.1, Table 2]
"We prompt the LLM to generate a passage about the target person at the end of each epoch, and the learning rate is adjusted such that the model refuses to answer and respond with 'I do not have information'. ... DF-MCQ almost always refuses to answer the question, hence minimizing the possibility of text overlapping and yielding the lowest ROUGE-L among counterparts."

Section 6.2 states that the learning rate is adjusted until the model refuses. The reported 92.72% refusal rate (Table 2) is therefore a tuned hyperparameter target, not an emergent property that independently confirms unlearning. The paper presents refusal as the 'expected behaviour of unlearning' (Section 7.1), but because the training recipe is explicitly tuned to produce refusals, the observation is forced by construction and cannot discriminate between genuine deletion and a learned target-name-triggered 'I do not know' response.

full rationale

The paper's central evidence for 'true knowledge removal' is the near-uniform MCQ entropy and the high refusal rate. Both are internal to the training recipe rather than independent confirmations: DF-MCQ's loss directly pushes the MCQ distribution to uniform, and the refusal behavior is obtained by adjusting the learning rate until it appears. The Eq. (1) definition of unlearning requires matching the entropy of a model trained without the fact, but no such retrained oracle is ever built, so the measured flatness is not shown to equal Hθ(Y|X;D\F). The retain loss (Eq. 5) only preserves behavior on other people's MCQs, which does not exclude a target-name-triggered shortcut: whenever the target person appears, the model can output a uniform distribution or 'I do not know' without deleting the underlying edge. This is especially consistent with the exact 1.61 entropy on all five subsets in Table 10. Some evidence is less directly circular: yes/no probe entropy is not optimized in that format and retain-set performance is not a training objective, which is why the score is 6 rather than 8 or 10. No load-bearing self-citation chain was found; the circularity is internal to the method's own loss and tuning choices.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claims rest on five domain assumptions and two unreported hyperparameters. The most consequential are the refusal-as-evidence assumption and the claim that flattening a finite MCQ output space flattens the entire output space. No new physical or conceptual entities are introduced; DF-MCQ is a training objective, not an entity.

free parameters (2)

Learning rate for DF-MCQ fine-tuning = not reported; manually adjusted to induce refusal
Section 6.2 states the learning rate is adjusted until the model refuses to answer, so the headline refusal result depends on a hand-tuned hyperparameter.
LoRA rank = not reported; higher rank used for Qwen-2.5-7B
The paper notes that Qwen-2.5 requires a higher LoRA rank to achieve refusal, but exact ranks are not listed, making the method sensitive to an unreported setting.

assumptions (5)

domain assumption LLM knowledge can be represented as a knowledge graph of (subject, relation, object) triplets, and unlearning can be measured by conditional entropy over objects.
Section 3 Eq. (1) defines the unlearning effect in this graph form; real LLM knowledge is not explicitly stored as triplets.
domain assumption An instruction-tuned LLM that lacks knowledge will refuse or say it does not know, so refusal probability is a valid proxy for unlearning.
Section 3 Eq. (2) makes refusal the indication of unlearning; in practice models may refuse for many reasons unrelated to knowledge absence.
ad hoc to paper Flattening the distribution over MCQ choice letters is effectively flattening the entire output space for the target question.
Section 7.1 Discussion claims this because the sum of choice-letter probabilities is close to 1; this is not formally derived and underlies why DF-MCQ should remove knowledge rather than merely reduce confidence.
domain assumption Automatically generated MCQs from LLM-sampled passages cover the target person's knowledge relevant to probing.
Section 6.1 generates 300-400 MCQs from 20 LLM passages; if these miss salient facts, flattening will not touch them.
domain assumption Yes/No and MCQ probing categories reliably indicate whether a knowledge edge exists.
Section 5.2-5.3 assumes a yes answer or correct choice reveals the retained edge, but output probabilities can be affected by prior biases, wording, and refusal behavior.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Unlearning vs. Obfuscation: Are We Truly Removing Knowledge?." pith.science (2026). https://pith.science/paper/G5Q2WRPQ

@misc{pith2026250502884,
  author       = {Pith},
  title        = {Pith review of: Unlearning vs. Obfuscation: Are We Truly Removing Knowledge?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G5Q2WRPQ}},
  note         = {Machine review of arXiv:2505.02884}
}

read the original abstract

Unlearning has emerged as a critical capability for large language models (LLMs) to support data privacy, regulatory compliance, and ethical AI deployment. Recent techniques often rely on obfuscation by injecting incorrect or irrelevant information to suppress knowledge. Such methods effectively constitute knowledge addition rather than true removal, often leaving models vulnerable to probing. In this paper, we formally distinguish unlearning from obfuscation and introduce a probing-based evaluation framework to assess whether existing approaches genuinely remove targeted information. Moreover, we propose DF-MCQ, a novel unlearning method that flattens the model predictive distribution over automatically generated multiple-choice questions using KL-divergence, effectively removing knowledge about target individuals and triggering appropriate refusal behaviour. Experimental results demonstrate that DF-MCQ achieves unlearning with over 90% refusal rate and a random choice-level uncertainty that is much higher than obfuscation on probing questions.

Figures

Figures reproduced from arXiv: 2505.02884 by the authors.

**Figure 2.** Illustration of the proposed distribution flattening MCQ (DF-MCQ) method. For questions in the forget [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Probing questions to distinguish between obfuscation and unlearning. Open-ended questions (Left) are [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Examples from the forget set and the hard [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: The rate of answering Yes (Yes rate) against [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

DECAF: De-Clustering for Adaptive Representational Unlearning
cs.LG 2026-07 conditional novelty 5.0 of 10

DECAF is a forget-only unlearning method that adds input noise, suppresses the forget-class probability, and diversifies outputs, achieving 0.10% forget accuracy and 79.4% retain accuracy on CIFAR-10/ResNet-18 while d...

Reference graph

Works this paper leans on

25 extracted references · 10 canonical work pages · cited by 1 Pith paper

[1]

George-Octavian Barbulescu and Peter Triantafillou. 2024. To each (textual sequence) its own: Improving memorized-data unlearning in large language models. arXiv preprint arXiv:2405.03097

arXiv 2024
[2]

Deniz Bayazit, Negar Foroutan, Zeming Chen, Gail Weiss, and Antoine Bosselut. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.376 Discovering knowledge-critical subnetworks in pretrained language models . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6549--6583, Miami, Florida, USA. Association for Comp...

doi:10.18653/v1/2024.emnlp-main.376 2024
[3]

Jiaao Chen and Diyi Yang. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.738 Unlearn what you want to forget: Efficient unlearning for LLM s . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12041--12052, Singapore. Association for Computational Linguistics

doi:10.18653/v1/2023.emnlp-main.738 2023
[4]

Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vuli ć . 2024. Undial: Self-distillation with adjusted logits for robust unlearning in large language models. arXiv:2402.10052

arXiv 2024
[5]

Ronen Eldan and Mark Russinovich. 2023. Who’s harry potter? approximate unlearning in llms. arXiv:2310.02238

arXiv 2023
[6]

XiaoHua Feng, Chaochao Chen, Yuyuan Li, and Zibin Lin. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.566 Fine-grained pluggable gradient ascent for knowledge unlearning in language models . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10141--10155, Miami, Florida, USA. Association for Computational L...

work page doi:10.18653/v1/2024.emnlp-main.566 2024
[7]

Shengyuan Hu, Yiwei Fu, Zhiwei Steven Wu, and Virginia Smith. 2025. Unlearning or obfuscating? jogging the memory of unlearned llms via benign relearning. In ICLR

work page 2025
[8]

Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023 a . https://openreview.net/forum?id=6t0Kwf8-jrj Editing models with task arithmetic . In The Eleventh International Conference on Learning Representations

work page 2023

Show all 25 references

[9]

Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023 b . https://openreview.net/forum?id=6t0Kwf8-jrj Editing models with task arithmetic . In The Eleventh International Conference on Learning Representations

2023
[10]

Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2023. https://doi.org/10.18653/v1/2023.acl-long.805 Knowledge unlearning for mitigating privacy risks in language models . In Proceedings of the 61st Annual Meeting of the As...

2023 doi
[11]

Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. 2025. Rethinking machine unlearning for large language models. Nature Machine Intelligence, pages 1--14

2025
[12]

Yujian Liu, Yang Zhang, Tommi Jaakkola, and Shiyu Chang. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.495 Revisiting who`s harry potter: Towards targeted unlearning from a causal intervention perspective . In Proceedings of the 2024 Conference on Empirical Methods in Natu...

2024 doi
[13]

Lipton, and J

Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. 2024. https://openreview.net/forum?id=B41hNBoWLo Tofu: A task of fictitious unlearning for llms . In First Conference on Language Modeling

2024
[14]

Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. 2022. Memory-based model editing at scale. In International Conference on Machine Learning, pages 15817--15831. PMLR

2022
[15]

Yash Sinha, Murari Mandal, and Mohan Kankanhalli. 2025. https://openreview.net/forum?id=J9Ofr1PmvX Un STAR : Unlearning with self-taught anti-sample reasoning for LLM s

2025
[16]

Bichen Wang, Yuzhe Zi, Yixin Sun, Yanyan Zhao, and Bing Qin. 2024. Rkld: Reverse kl-divergence-based knowledge distillation for unlearning personal information in large language models. In NAACL

2024
[17]

Lingzhi Wang, Tong Chen, Wei Yuan, Xingshan Zeng, Kam-Fai Wong, and Hongzhi Yin. 2023. https://doi.org/10.18653/v1/2023.acl-long.740 KGA : A general machine unlearning framework based on knowledge gap alignment . In Proceedings of the 61st Annual Meeting of the Association for...

2023 doi
[18]

Yaxuan Wang, Jiaheng Wei, Chris Yuhao Liu, Jinlong Pang, Quan Liu, Ankit Parag Shah, Yujia Bao, Yang Liu, and Wei Wei. 2025. Llm unlearning via loss adjustment with only forget data. In ICLR

2025
[19]

Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.174 DEPN : Detecting and editing privacy neurons in pretrained language models . In Proceedings of the 2023 Conference on Empirical Metho...

2023 doi
[21]

Haoming Xu, Ningyuan Zhao, Liming Yang, Sendong Zhao, Shumin Deng, Mengru Wang, Bryan Hooi, Nay Oo, Huajun Chen, and Ningyu Zhang. 2025 b . Relearn: Unlearning via learning for large language models. arXiv preprint arXiv:2502.11190

2025 arXiv
[22]

Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2024 a . Large language model unlearning. arXiv:2310.10683

2024 arXiv
[23]

Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2024 b . https://openreview.net/forum?id=8Dy42ThoNe Large language model unlearning . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

2024
[24]

Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. https://openreview.net/forum?id=MXLBXjQkmb Negative preference optimization: From catastrophic collapse to effective unlearning . In First Conference on Language Modeling

2024
[25]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...
[26]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

George-Octavian Barbulescu and Peter Triantafillou. 2024. To each (textual sequence) its own: Improving memorized-data unlearning in large language models. arXiv preprint arXiv:2405.03097

arXiv 2024

[2] [2]

Deniz Bayazit, Negar Foroutan, Zeming Chen, Gail Weiss, and Antoine Bosselut. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.376 Discovering knowledge-critical subnetworks in pretrained language models . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6549--6583, Miami, Florida, USA. Association for Comp...

doi:10.18653/v1/2024.emnlp-main.376 2024

[3] [3]

Jiaao Chen and Diyi Yang. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.738 Unlearn what you want to forget: Efficient unlearning for LLM s . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12041--12052, Singapore. Association for Computational Linguistics

doi:10.18653/v1/2023.emnlp-main.738 2023

[4] [4]

Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vuli ć . 2024. Undial: Self-distillation with adjusted logits for robust unlearning in large language models. arXiv:2402.10052

arXiv 2024

[5] [5]

Ronen Eldan and Mark Russinovich. 2023. Who’s harry potter? approximate unlearning in llms. arXiv:2310.02238

arXiv 2023

[6] [6]

XiaoHua Feng, Chaochao Chen, Yuyuan Li, and Zibin Lin. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.566 Fine-grained pluggable gradient ascent for knowledge unlearning in language models . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10141--10155, Miami, Florida, USA. Association for Computational L...

work page doi:10.18653/v1/2024.emnlp-main.566 2024

[7] [7]

Shengyuan Hu, Yiwei Fu, Zhiwei Steven Wu, and Virginia Smith. 2025. Unlearning or obfuscating? jogging the memory of unlearned llms via benign relearning. In ICLR

work page 2025

[8] [8]

Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023 a . https://openreview.net/forum?id=6t0Kwf8-jrj Editing models with task arithmetic . In The Eleventh International Conference on Learning Representations

work page 2023