Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

By optimizing next-token distributions rather than whole responses, DiPO builds preference pairs from the model's own logits and reports a TOFU-10% forget quality of 0.86—nearly double the 0.45 of the strongest baseline—while preserving mod

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 11:23 UTC pith:JI3QHG6T

load-bearing objection DiPO is a genuinely new distribution-level unlearning loss with strong TOFU numbers, but the consistency proof hinges on an unstated stop-gradient and the MUSE privacy leak is a real gap. the 4 major comments →

arxiv 2510.04773 v2 pith:JI3QHG6T submitted 2025-10-06 cs.LG cs.AI

Distribution Preference Optimization: A Fine-grained Perspective for LLM Unlearning

Kai Qin , Jiaqi Wu , Jianxiang He , Haoyuan Sun , Yifei Zhao , Xu Wang , Bin Liang , Yongzhe Chang
show 3 more authors
Cheng Li Tiantian Zhang Houde Liu
This is my paper
classification cs.LG cs.AI
keywords LLM unlearningdistribution-level preference optimizationnegative preference optimizationlogit modulationsequence KL divergenceTOFU benchmarkMUSE benchmarkforget quality
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

DiPO (Distribution Preference Optimization) claims that LLM unlearning can be moved from the response level to the next-token probability distribution level. It derives a DPO-style preference loss whose preferred and dispreferred distributions are constructed by amplifying or suppressing the model's own high-confidence logits, so no auxiliary model or hand-written 'I don't know' answers are needed. The paper proves the loss is consistent with the intended unlearning direction, and reports that on TOFU-10% forget quality reaches 0.86 versus 0.45 for the strongest baseline, with model utility roughly preserved (0.57 versus retrain's 0.62). A reader should care because it suggests a general, stable, and domain-knowledge-free route to targeted forgetting.

Core claim

The paper's central claim is that a valid pair of preference distributions for unlearning can be constructed intrinsically from the current model's logits: isolate a 'memory vector' of top-k high-confidence token logits, add it to get a memory-enhancing distribution πm, and subtract it to get a forgetting-promoting distribution πf. The DiPO loss then maximizes the gap between Sequence KL divergences—moving the policy away from the dispreferred distribution and toward the preferred one—and the paper proves this direction is consistent with the loss gradient. Empirically, DiPO attains a TOFU-10% forget quality of 0.86, nearly doubling the strongest baseline's 0.45, while maintaining model util

What carries the argument

The central object is the DiPO loss, a DPO-style Bradley-Terry objective over distribution-level returns: L_DiPO = -E log σ[β(D_SeqKL(π_l||π_θ) - D_SeqKL(π_w||π_θ)) + β(D_SeqKL(π_w||π_ref) - D_SeqKL(π_l||π_ref))], where D_SeqKL is the sum of per-timestep KL divergences. The required preference pair comes from a 'memory vector': top-k high-confidence logits are isolated, and π_m = softmax(z + αm), π_f = softmax(z - αm). Adding or subtracting this vector shapes the distribution toward memorization or forgetting, and the same pair, with roles reversed, forms the retain loss. This machinery turns the finite vocabulary into a complete, intrinsic set of alternative tokens, removing the need for ha

Load-bearing premise

The load-bearing assumption is that the preferred and dispreferred distributions, although built from the current model's own logits at each step, are treated as constants in the gradient; if gradients are allowed to flow through that construction, the paper's proof that the loss pushes toward πw and away from πl no longer applies.

What would settle it

Take the DiPO implementation on TOFU-10% and compare two variants: one where gradient flow is cut (stop-gradient) before πw, πl, and C are formed, and one where it is not. If the second variant's FQ and MU trajectory is materially worse, or if the total derivative sign contradicts the partial-derivative analysis, then the reported unlearning gains are not explained by the theorem as stated.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • DiPO's forget objective alone, without any retain loss, can reach FQ around 0.51 with MU around 0.18 on TOFU-10%, suggesting the mechanism is stable and useful when retain data is unavailable.
  • The same preference-pair construction works for both forget and retain objectives by reversing roles, so no domain-specific prompt engineering or auxiliary model is needed.
  • At the final epoch, DiPO keeps FQ between 0.84 and 0.89 on TOFU, so it does not depend on fragile early stopping.
  • On TOFU-10%, DiPO's 0.86 forget quality exceeds AltPO's 0.58 and ULD's 0.48 without their TOFU-specific augmentations.
  • On MUSE, DiPO achieves the best verbatim-forgetting and knowledge-retention scores and retains utility as the forget set grows and under sequential unlearning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: DiPO's logit-modulation recipe resembles a differentiable form of contrastive decoding; a testable extension is to make the amplification factor α and the top-k rate adaptive per token, which could improve privacy-leakage metrics such as PrivLeak.
  • Beyond the paper: the theoretical consistency proof treats πw, πl, and C as constants, even though they are constructed from the current policy's logits; whether implementations detach these from the computation graph is an unstated implementation detail, and checking it would settle whether the proven gradient direction is actually the update being applied.
  • Beyond the paper: because the method requires only output logits, it should transfer to other generative and classification settings with a finite output space, such as unlearning in multimodal or retrieval-augmented models; this is an untested extrapolation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces DiPO, an LLM unlearning method that operates on next-token probability distributions rather than on full responses. It constructs preferred and dispreferred distributions (π_m, π_f) by adding or subtracting a top-k 'memory vector' derived from the current policy's logits (Eq. 19), then applies a Bradley-Terry style logistic loss over differences of Sequence KL divergences (Eq. 14). Forget and retain objectives are formed by reversing the roles of π_m and π_f (Eqs. 20–21). The authors give a theoretical consistency argument in §4.2 and report experiments on TOFU and MUSE, including a forget quality of 0.86 on TOFU-10% with good model utility and stable training curves.

Significance. If the theoretical and empirical claims hold, DiPO would be a useful contribution to LLM unlearning: it avoids the need to construct response-level preferred alternatives, which is a known bottleneck for methods like NPO and AltPO, and it reports substantially higher TOFU forget quality than the optimization-based baselines considered. The paper is also candid about limitations on MUSE privacy leakage. However, the central theory currently rests on an unstated gradient-detachment assumption and an unjustified substitution of the current policy for the optimal policy, and the key empirical comparisons lack variance reporting. With clarifications and a corrected proof, the method could be significant; in its present form, the main claims are not fully supported.

major comments (4)
  1. [§4.2, Eq. (19) and Algorithm 1] The gradient analysis treats x1, x2, and C as constants with respect to θ in Eqs. (15)–(18), but Eq. (19) constructs π_m and π_f from z_t, the current policy's logits, and Algorithm 1 generates them 'from π_θ' with no stated stop-gradient. If z_t participates in the computation graph, then π_l and π_w are functions of θ, so ∂L/∂x1 and ∂L/∂x2 do not determine dL/dθ, and the sign analysis does not prove the claimed 'push away from π_l / pull toward π_w' dynamics. C in Eq. (16) is likewise not constant. The theorem is only valid if these distributions are detached, an assumption the paper never states. This is load-bearing for Theorem 4.1 and for the actual update implemented in practice.
  2. [Appendix B.2, Eq. (26)] The proof of Theorem 4.1 derives returns R^{π_w}(x,y,π*_θ) and R^{π_l}(x,y,π*_θ) using the optimal policy π*_θ in Eqs. (23)–(24), but then constructs the MLE objective in Eq. (26) by replacing π*_θ with the current π_θ. This substitution is not justified. In DPO the analogous step has a clean interpretation because the optimal policy is expressed through the current policy under a fixed reward; here the reward and the preference distributions themselves depend on the evolving π_θ. The proof needs an explicit statement of why π_θ may stand in for π*_θ, or a revised derivation.
  3. [Tables 2 and 3] The headline results, e.g. FQ 0.86 vs. NPO+GD 0.45 on TOFU-10%, are reported as averages over five seeds but without standard deviations, error bars, or significance tests. Table 3 further states that AltPO and DiPO are reported from the 'best-performing seed' rather than average performance. This makes it difficult to assess whether the improvements are stable or an artifact of seed selection. Please report mean±std over all seeds for every method, or justify the best-seed protocol and show its worst-case behavior.
  4. [Table 4 and Appendix A] On MUSE, DiPO's PrivLeak is 98.1, far from Retrain's 0.0 and comparable to GA+GD's 108.1. Appendix A correctly concedes that current preference modeling is insufficient for MIA-style privacy leakage. Additionally, DiPO's VM-f of 31.67 is above Retrain's 20.8, indicating incomplete verbatim forgetting relative to several baselines. The paper's broader claim of a 'strong trade-off' should be qualified to the utility and stability dimensions; the privacy dimension substantially limits the significance of the MUSE results.
minor comments (4)
  1. [Algorithm 1] The algorithm input and step 5/6 say 'top-p logit filtering', while Section 4.3 and Appendix D.2 describe 'top-k filtering' with rate p_k. Please align the terminology.
  2. [Table 4] The DiPO row reads '31.6753.22' with no delimiter; it should be '31.67 53.22'.
  3. [§D.5.1] Configurations 3 (GA+DiPO(r)) and 4 (NPO+DiPO(r)) both say 'uses the standard Gradient Descent loss' before listing different objectives. This appears to be a copy-paste error; the intended losses should be labeled correctly.
  4. [Table 1 caption] Typo: 'Groud truth' should be 'Ground truth'.

Circularity Check

1 steps flagged

DiPO's 'consistency with desired unlearning direction' is a self-referential partial-derivative statement: the preference distributions are built from the model's own logits, yet §4.2 treats them as constants.

specific steps
  1. self definitional [Section 4.2 (Eqs. 15–18) vs. Section 4.3 (Eq. 19) and Algorithm 1]
    "we can construct the memory-enhancing distribution πm and the forgetting-promoting distribution πf by adding or subtracting this memory vector, scaled by a factor α: πm(·|x,y<t)=softmax(zt+αmt), πf(·|x,y<t)=softmax(zt−αmt).(19)... Note that x1 and x2 depend on the trainable policy πθ, while C is treated as a constant... Since ∂L/∂x1<0, minimizing L via gradient descent increases x1=DSeqKL(πl||πθ), effectively pushing πθ away from πl."

    The preferred/dispreferred distributions are not external targets: they are constructed from z_t, the current policy's own logits, and Algorithm 1 generates them 'from πθ' without stating any stop-gradient. The gradient proof differentiates L only through x1=DSeqKL(πl||πθ) and x2=DSeqKL(πw||πθ), and declares C constant. But under Eq. (19), πl, πw, and C all depend on θ when gradients flow into z_t. The total derivative then contains unanalyzed terms through ∂πl/∂θ and ∂πw/∂θ, so ∂L/∂x1<0 does not imply that gradient descent actually increases x1. The claimed 'consistency with the desired unlearning direction' therefore restates the sign of a partial derivative under an unstated constancy assumption, rather than proving the actual update dynamics of the loss as constructed.

full rationale

The TOFU/MUSE empirical results are external benchmark evaluations and are not themselves circular; the paper also honestly acknowledges in Appendix A that DiPO's PrivLeak scores remain weak (e.g., 98.1 vs. Retrain 0.0 on MUSE), which is a limitation rather than a circularity. Self-citations are minor (e.g., ref. [50] in the related-work list) and are not load-bearing. The central circularity is narrower: the theoretical consistency claim in §4.2 treats πw, πl, and C as constants with respect to θ, while §4.3/Eq. (19) constructs these distributions from the current policy's logits. Unless a stop-gradient is silently applied, the proof does not establish the claimed 'push away from πl / toward πw' behavior; the theorem reduces to the sign of the loss's own partial derivatives under an assumption that is nowhere stated. Because the empirical SOTA claim has independent benchmark content, the overall score is moderate (4) rather than higher.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

The central derivation depends on a chain of existing preference-optimization assumptions plus an unstated stop-gradient treatment of the self-constructed preference distributions. The only free parameters are hand-set hyperparameters, and α is missing entirely.

free parameters (4)
  • β (β_f, β_r) = 0.05
    Preference strength in DiPO loss; set to 0.05 in D.3.3, not derived.
  • λ (retain weight) = 1
    Weight on retain loss; set to 1 following common practice.
  • α (memory-vector scale) = not reported
    Controls how much top-k logits are amplified/suppressed in Eq. (19); never specified in the paper.
  • p_k (top-k fraction) = 0.05
    Fraction of high-confidence tokens used to build memory vector; set in D.2.
axioms (5)
  • domain assumption Bradley-Terry model of preference between distribution-level returns
    Used in Eq. (25) to convert return differences into a logistic probability; standard in DPO but not validated for this constructed preference setting.
  • standard math TDPO formulation of text generation as MDP with closed-form optimal policy (Eq. 7)
    Borrowed from [51]; the paper relies on it to express reward via log-policy ratios.
  • ad hoc to paper The current policy πθ can be substituted for the optimal policy π*θ in the return expression when constructing the MLE objective
    In Section B.2, Rπ(x,y,πθ) replaces π*θ with πθ without justification; this is DPO-style but not proven and affects the derived loss.
  • ad hoc to paper π_w, π_l and C are treated as constants during gradient analysis
    Section 4.2 differentiates only the second argument of D_SeqKL; requires stop-gradient on constructed distributions, which is not stated in Algorithm 1.
  • domain assumption The top-k high-confidence tokens in the forget set carry the knowledge to be unlearned, and suppressing them yields non-sensitive alternatives
    The memory-vector construction (Eq. 19, Section 4.3) assumes this to make π_f a valid forgetting target.
invented entities (1)
  • Memory vector m_t no independent evidence
    purpose: Selects top-k high-confidence logits from current model output to build π_m and π_f
    It is an internal construct derived from the model's logits; it makes no falsifiable prediction outside the method.

pith-pipeline@v1.3.0-alltime-deepseek · 19210 in / 15667 out tokens · 113127 ms · 2026-08-04T11:23:07.423099+00:00 · methodology

0 comments
read the original abstract

As Large Language Models (LLMs) demonstrate remarkable capabilities learned from vast corpora, concerns regarding data privacy and safety are receiving increasing attention. LLM unlearning, which aims to remove the influence of specific data while preserving overall model utility, is becoming an important research area. One of the mainstream unlearning classes is optimization-based methods, which achieve forgetting directly through fine-tuning, exemplified by Negative Preference Optimization (NPO). However, NPO's effectiveness is limited by its inherent lack of explicit positive preference signals. Attempts to introduce such signals by constructing preferred responses often necessitate domain-specific knowledge or well-designed prompts, fundamentally restricting their generalizability. In this paper, we shift the focus to the distribution-level, directly targeting the next-token probability distribution instead of entire responses, and derive a novel unlearning algorithm termed \textbf{Di}stribution \textbf{P}reference \textbf{O}ptimization (DiPO). We show that the requisite preference distribution pairs for DiPO, which are distributions over the model's output tokens, can be constructed by selectively amplifying or suppressing the model's high-confidence output logits, thereby effectively overcoming NPO's limitations. We theoretically prove the consistency of DiPO's loss function with the desired unlearning direction. Extensive experiments demonstrate that DiPO achieves a strong trade-off between model utility and forget quality. Notably, DiPO attains the highest forget quality on the TOFU benchmark, and maintains leading scalability and sustainability in utility preservation on the MUSE benchmark.

Figures

Figures reproduced from arXiv: 2510.04773 by Bin Liang, Cheng Li, Haoyuan Sun, Houde Liu, Jianxiang He, Jiaqi Wu, Kai Qin, Tiantian Zhang, Xu Wang, Yifei Zhao, Yongzhe Chang.

Figure 1
Figure 1. Figure 1: Construction of memory-enhancing distribution πm and forgetting-promoting dis￾tribution πf by a memory vector filtered from origin logits. Our approach to constructing preference pairs (πw,πl) from the model’s logits zt focuses on mod￾ulating a small subset of high-probability tokens: If these tokens correspond to undesirable informa￾tion, suppressing their logits naturally steers the model towards alterna… view at source ↗
Figure 2
Figure 2. Figure 2: Performance analysis on TOFU at the best epoch over five seeds. (a) FQ vs. MU on [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Robustness analysis on MUSE and DiPO’s internal mechanisms. (a) Scalability and [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Training curves for only-forget configuration on TOFU-10%, with GA and NPO curves [PITH_FULL_IMAGE:figures/full_fig_p020_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning

    cs.LG 2026-05 unverdicted novelty 6.0

    Distinguishable Deletion unifies knowledge erasure and refusal for LLM unlearning via an energy index that enforces boundaries during training and enables refusal at inference.

Reference graph

Works this paper leans on

70 extracted references · 21 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Unlearning bias in language models by partitioning gradients

    Charles Yu, Sullam Jeoung, Anish Kasi, Pengfei Yu, and Heng Ji. Unlearning bias in language models by partitioning gradients. InFindings of the Association for Computational Linguistics: ACL 2023, pages 6032–6048, 2023

    2023

  2. [2]

    Continual learning and private unlearning

    Bo Liu, Qiang Liu, and Peter Stone. Continual learning and private unlearning. InConference on Lifelong Learning Agents, pages 243–254. PMLR, 2022

    2022

  3. [3]

    Who’s harry potter? approximate unlearning for llms

    Ronen Eldan and Mark Russinovich. Who’s harry potter? approximate unlearning for llms. 2023

    2023

  4. [4]

    Knowledge unlearning for mitigating privacy risks in language models.arXiv preprint arXiv:2210.01504, 2022

    Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models.arXiv preprint arXiv:2210.01504, 2022

    Pith/arXiv arXiv 2022

  5. [5]

    Depn: Detecting and editing privacy neurons in pretrained language models.arXiv preprint arXiv:2310.20138, 2023

    Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. Depn: Detecting and editing privacy neurons in pretrained language models.arXiv preprint arXiv:2310.20138, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Towards safer large language models through machine unlearning.arXiv preprint arXiv:2402.10058, 2024

    Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. Towards safer large language models through machine unlearning.arXiv preprint arXiv:2402.10058, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Identifying and mitigating the security risks of generative ai.Foundations and Trends® in Privacy and Security, 6(1):1–52, 2023

    Clark Barrett, Brad Boyd, Elie Bursztein, Nicholas Carlini, Brad Chen, Jihye Choi, Amrita Roy Chowdhury, Mihai Christodorescu, Anupam Datta, Soheil Feizi, et al. Identifying and mitigating the security risks of generative ai.Foundations and Trends® in Privacy and Security, 6(1):1–52, 2023

    2023

  8. [8]

    European Parliament and Council of the European Union. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/EC (General Data Protection Regulation). Official Journal of ...

    2016

  9. [9]

    Assembly bill no

    California State Assembly. Assembly bill no. 375 (Chapter 55, statutes of 2018). an act to add title 1.81.5 (commencing with section 1798.100) to part 4 of division 3 of the civil code, relating to privacy. (California Consumer Privacy Act of 2018). California Legislature, 2017–2018 Regular Session, June 2018. Approved by Governor and filed with Secretary...

    2018

  10. [10]

    Large language model unlearning.Advances in Neural Information Processing Systems, 37:105425–105475, 2024

    Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning.Advances in Neural Information Processing Systems, 37:105425–105475, 2024

    2024

  11. [11]

    Negative preference optimization: From catastrophic collapse to effective unlearning

    Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. InFirst Conference on Language Modeling, 2024

    2024

  12. [12]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  13. [13]

    TOFU: A task of fictitious unlearning for LLMs

    Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and J Zico Kolter. TOFU: A task of fictitious unlearning for LLMs. InFirst Conference on Language Modeling, 2024

    2024

  14. [14]

    Smith, and Chiyuan Zhang

    Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, and Chiyuan Zhang. MUSE: Machine unlearning six- way evaluation for language models. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  15. [15]

    A survey of machine unlearning.arXiv preprint arXiv:2209.02299, 2022

    Thanh Tam Nguyen, Thanh Trung Huynh, Zhao Ren, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. A survey of machine unlearning.arXiv preprint arXiv:2209.02299, 2022. 10

    Pith/arXiv arXiv 2022

  16. [16]

    Towards making systems forget with machine unlearning

    Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pages 463–480. IEEE, 2015

    2015

  17. [17]

    Unrolling sgd: Understanding factors influencing machine unlearning

    Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. In2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pages 303–319. IEEE, 2022

    2022

  18. [18]

    Approximate data deletion from machine learning models

    Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. InInternational Conference on Artificial Intelligence and Statistics, pages 2008–2016. PMLR, 2021

    2008

  19. [19]

    Kga: A general machine unlearning framework based on knowledge gap alignment.arXiv preprint arXiv:2305.06535, 2023

    Lingzhi Wang, Tong Chen, Wei Yuan, Xingshan Zeng, Kam-Fai Wong, and Hongzhi Yin. Kga: A general machine unlearning framework based on knowledge gap alignment.arXiv preprint arXiv:2305.06535, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    Are we making progress in unlearning? findings from the first neurips unlearning competition.arXiv preprint arXiv:2406.09073, 2024

    Eleni Triantafillou, Peter Kairouz, Fabian Pedregosa, Jamie Hayes, Meghdad Kurmanji, Kairan Zhao, Vincent Dumoulin, Julio Jacques Junior, Ioannis Mitliagkas, Jun Wan, et al. Are we making progress in unlearning? findings from the first neurips unlearning competition.arXiv preprint arXiv:2406.09073, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Eternal sunshine of the spotless net: Selective forgetting in deep networks

    Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9304–9312, 2020

    2020

  22. [22]

    Machine unlearning

    Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In2021 IEEE symposium on security and privacy (SP), pages 141–159. IEEE, 2021

    2021

  23. [23]

    Model sparsity can simplify machine unlearning.Advances in Neural Information Processing Systems, 36:51584–51605, 2023

    Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning.Advances in Neural Information Processing Systems, 36:51584–51605, 2023

    2023

  24. [24]

    Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation.arXiv preprint arXiv:2310.12508, 2023

    Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation.arXiv preprint arXiv:2310.12508, 2023

    Pith/arXiv arXiv 2023

  25. [25]

    Towards un- bounded machine unlearning.Advances in neural information processing systems, 36:1957– 1987, 2023

    Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards un- bounded machine unlearning.Advances in neural information processing systems, 36:1957– 1987, 2023

    1957

  26. [26]

    Making ai forget you: Data deletion in machine learning.Advances in neural information processing systems, 32, 2019

    Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning.Advances in neural information processing systems, 32, 2019

    2019

  27. [27]

    Erasing concepts from diffusion models

    Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2426–2436, 2023

    2023

  28. [28]

    Forget-me- not: Learning to forget in text-to-image diffusion models

    Gong Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me- not: Learning to forget in text-to-image diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1755–1764, 2024

    2024

  29. [29]

    Fast federated machine unlearning with nonlinear functional theory

    Tianshi Che, Yang Zhou, Zijie Zhang, Lingjuan Lyu, Ji Liu, Da Yan, Dejing Dou, and Jun Huan. Fast federated machine unlearning with nonlinear functional theory. InInternational conference on machine learning, pages 4241–4268. PMLR, 2023

    2023

  30. [30]

    Federated unlearning with gradient descent and conflict mitigation

    Zibin Pan, Zhichao Wang, Chi Li, Kaiyan Zheng, Boqi Wang, Xiaoying Tang, and Junhua Zhao. Federated unlearning with gradient descent and conflict mitigation. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 19804–19812, 2025

    2025

  31. [31]

    Efficient model updates for approximate un- learning of graph-structured data

    Eli Chien, Chao Pan, and Olgica Milenkovic. Efficient model updates for approximate un- learning of graph-structured data. InThe Eleventh International Conference on Learning Representations, 2022. 11

    2022

  32. [32]

    Certified edge unlearning for graph neural networks

    Kun Wu, Jie Shen, Yue Ning, Ting Wang, and Wendy Hui Wang. Certified edge unlearning for graph neural networks. InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2606–2617, 2023

    2023

  33. [33]

    Machine unlearning for recommendation systems: An insight

    Bhavika Sachdeva, Harshita Rathee, Sristi, Arun Sharma, and Witold Wydma´nski. Machine unlearning for recommendation systems: An insight. InInternational Conference On Innovative Computing And Communication, pages 415–430. Springer, 2024

    2024

  34. [34]

    Simplicity prevails: Rethinking negative preference optimization for llm unlearning.arXiv preprint arXiv:2410.07163, 2024

    Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. Simplicity prevails: Rethinking negative preference optimization for llm unlearning.arXiv preprint arXiv:2410.07163, 2024

    arXiv 2024

  35. [35]

    Alternate preference optimization for unlearning factual knowledge in large language models.arXiv preprint arXiv:2409.13474, 2024

    Anmol Mekala, Vineeth Dorna, Shreya Dubey, Abhishek Lalwani, David Koleczek, Mukund Rungta, Sadid Hasan, and Elita Lobo. Alternate preference optimization for unlearning factual knowledge in large language models.arXiv preprint arXiv:2409.13474, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Soul: Unlocking the power of second-order optimization for llm unlearning.arXiv preprint arXiv:2404.18239, 2024

    Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. Soul: Unlocking the power of second-order optimization for llm unlearning.arXiv preprint arXiv:2404.18239, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher

    Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 7210–7217, 2023

    2023

  38. [38]

    Reversing the forget-retain objectives: An efficient llm unlearning framework from logit difference.Advances in Neural Information Processing Systems, 37:12581–12611, 2024

    Jiabao Ji, Yujian Liu, Yang Zhang, Gaowen Liu, Ramana Kompella, Sijia Liu, and Shiyu Chang. Reversing the forget-retain objectives: An efficient llm unlearning framework from logit difference.Advances in Neural Information Processing Systems, 37:12581–12611, 2024

    2024

  39. [39]

    Unlearn what you want to forget: Efficient unlearning for llms

    Jiaao Chen and Diyi Yang. Unlearn what you want to forget: Efficient unlearning for llms. arXiv preprint arXiv:2310.20150, 2023

    Pith/arXiv arXiv 2023

  40. [40]

    Machine unlearning of pre-trained large language models.arXiv preprint arXiv:2402.15159, 2024

    Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. Machine unlearning of pre-trained large language models.arXiv preprint arXiv:2402.15159, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Knowledge sanitization of large language models

    Yoichi Ishibashi and Hidetoshi Shimodaira. Knowledge sanitization of large language models. arXiv preprint arXiv:2309.11852, 2023

    Pith/arXiv arXiv 2023

  42. [42]

    Revisiting who’s harry pot- ter: Towards targeted unlearning from a causal intervention perspective.arXiv preprint arXiv:2407.16997, 2024

    Yujian Liu, Yang Zhang, Tommi Jaakkola, and Shiyu Chang. Revisiting who’s harry pot- ter: Towards targeted unlearning from a causal intervention perspective.arXiv preprint arXiv:2407.16997, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Guardrail baselines for unlearning in llms.arXiv preprint arXiv:2403.03329, 2024

    Pratiksha Thaker, Yash Maurya, Shengyuan Hu, Zhiwei Steven Wu, and Virginia Smith. Guardrail baselines for unlearning in llms.arXiv preprint arXiv:2403.03329, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    In-context unlearning: Language models as few shot unlearners.arXiv preprint arXiv:2310.07579, 2023

    Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners.arXiv preprint arXiv:2310.07579, 2023

    Pith/arXiv arXiv 2023

  45. [45]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  46. [46]

    Orpo: Monolithic preference optimization without reference model.arXiv preprint arXiv:2403.07691, 2024

    Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without reference model.arXiv preprint arXiv:2403.07691, 2024

    Pith/arXiv arXiv 2024

  47. [47]

    Simpo: Simple preference optimization with a reference-free reward.Advances in Neural Information Processing Systems, 37:124198–124235, 2024

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward.Advances in Neural Information Processing Systems, 37:124198–124235, 2024

    2024

  48. [48]

    Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints.arXiv preprint arXiv:2309.16240, 2023

    Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints.arXiv preprint arXiv:2309.16240, 2023. 12

    Pith/arXiv arXiv 2023

  49. [49]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. InInternational Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR, 2024

    2024

  50. [50]

    Generalizing offline alignment theoretical paradigm with diverse divergence constraints

    Haoyuan Sun, Yuxin Zheng, Yifei Zhao, Yongzhe Chang, and Xueqian Wang. Generalizing offline alignment theoretical paradigm with diverse divergence constraints. InICML 2024 Workshop on Models of Human Feedback for AI Alignment, 2024

    2024

  51. [51]

    Token- level direct preference optimization.arXiv preprint arXiv:2404.11999, 2024

    Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token- level direct preference optimization.arXiv preprint arXiv:2404.11999, 2024

    Pith/arXiv arXiv 2024

  52. [52]

    Contrastive decoding: Open-ended text generation as optimization.arXiv preprint arXiv:2210.15097, 2022

    Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization.arXiv preprint arXiv:2210.15097, 2022

    Pith/arXiv arXiv 2022

  53. [53]

    Dola: Decoding by contrasting layers improves factuality in large language models.arXiv preprint arXiv:2309.03883, 2023

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models.arXiv preprint arXiv:2309.03883, 2023

    Pith/arXiv arXiv 2023

  54. [54]

    Dexperts: Decoding-time controlled text generation with experts and anti-experts.arXiv preprint arXiv:2105.03023, 2021

    Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. Dexperts: Decoding-time controlled text generation with experts and anti-experts.arXiv preprint arXiv:2105.03023, 2021. A Limitations Despite DiPO demonstrating strong unlearning capabilities, certain limitations warrant discussion. First, similar to ...

    Pith/arXiv arXiv 2021

  55. [56]

    I don’t know

    for unlearning [11] (distinguish from standard DPO). It requires a dataset of simple, template- based alternative responses Da (e.g. yidk = “I don’t know”) and formulates the forget loss to prefer yidk over the original forget responsey f : LDPO(θ) =− 1 β E(x f ,y f )∼D f ,yidk ∼Da[logσ(β πθ (yidk |x f ) πref(yidk |x f ) −β πθ (y f |x f ) πref(y f |x f ) ...

  56. [57]

    It is set to the log-probability corresponding to the k-th rank when tokens are sorted by log-probability in descending order, wherek=max(1,⌊p k · |V|⌋)

    Rank-based Threshold (τk):This ensures at least a minimum number of tokens are kept. It is set to the log-probability corresponding to the k-th rank when tokens are sorted by log-probability in descending order, wherek=max(1,⌊p k · |V|⌋)

  57. [58]

    The final threshold used for filtering is the minimum of these two: τ=min(τ k,τ rel)

    Relative Threshold (τrel):This adapts to the sharpness of the distribution and is calculated relative to the maximum log-probability:τ rel =max(s t ) +log(pk). The final threshold used for filtering is the minimum of these two: τ=min(τ k,τ rel). The set St then comprises all tokens whose log-probability is greater than or equal to this final threshold (St...

  58. [59]

    Verbatim text: Original text excerpts from news articles used to assess the prevention of verbatim memorization

  59. [60]

    D.4.2 Evaluation Metrics MUSE evaluates unlearning across six criteria

    Knowledge set: Question-answer (QA) pairs derived from the original news texts to evaluate the removal of factual knowledge. D.4.2 Evaluation Metrics MUSE evaluates unlearning across six criteria. We highlight key metrics reflecting data owner and deployer concerns as applied to the NEWS corpus: Data Owner Focused Metrics

  60. [61]

    Quantified by VerbMem-f, which measures the ROUGE-L F1 score between model-generated continuations and true continuations fromD forget

    No Verbatim Memorization (VerbMem-f): Assesses if the unlearned model ( funlearn) avoids reproducing exact text sequences from the Dforget of news articles. Quantified by VerbMem-f, which measures the ROUGE-L F1 score between model-generated continuations and true continuations fromD forget. VerbMem-f(f,D forget):= 1 |Dforget| ∑ x∈Dforget ROUGE-L(f(x [:l]...

  61. [62]

    Quantified by KnowMem-f, averaging ROUGE scores between model answers and ground-truth answers for QA pairs derived fromD forget

    No Knowledge Memorization (KnowMem-f): Measures if funlearn can no longer answer questions whose answers are exclusively found in the Dforget of news articles. Quantified by KnowMem-f, averaging ROUGE scores between model answers and ground-truth answers for QA pairs derived fromD forget

  62. [63]

    No Privacy Leakage (PrivLeak): Evaluates if the inclusion of news articles from Dforget in the original training data (Dtrain) can be inferred from funlearn. Measured byPrivLeak, which compares the Area Under the ROC Curve (AUC) of a Membership Inference Attack (MIA) on funlearn against that on a perfectly retrained model (fretrain), discriminating betwee...

  63. [64]

    This is typically measured using theKnowMem-rmetric applied toD retain: KnowMem-r(f unlearn,D retain)

    Utility Preservation (KnowMem-r): Quantifies how well funlearn maintains its performance on the Dretain of news articles. This is typically measured using theKnowMem-rmetric applied toD retain: KnowMem-r(f unlearn,D retain)

  64. [65]

    Scalability: Assesses how unlearning methods perform with increasing sizes of Dforget within the NEWS corpus

  65. [66]

    Sustainability: Evaluates performance under sequential unlearning requests involving different sets of news articles. D.4.3 Hyperparameter Implementation Following the setup of MUSE [14], we use LLaMA-2 7B as the original model, which was released before the collected BBC news articles to prevent potential data leakage. For baseline methods, we set the ba...

  66. [67]

    The combined objective is then expressed asL=L DiPO-f(θ) +λLDiPO-r(θ)

    DiPO (ours):This is the configuration presented as our main result in the paper, using the LDiPO-r by reversing the roles of the preference distributions of LDiPO-f on the retain set. The combined objective is then expressed asL=L DiPO-f(θ) +λLDiPO-r(θ)

  67. [68]

    DiPO(f)+GD:This configuration utilizes the standard Gradient Descent loss Equation (30) on the retain set: min θ L(θ) =min θ (LDiPO-f(θ) +γLGD(θ)) =min θ LDiPO-f(θ) +λE(xr,yr)∼Dr [−logπ θ (yr|xr)]

  68. [69]

    GA+DiPO(r):This configuration utilizes the standard Gradient Descent loss Equation (30) on the retain set: min θ L(θ) =min θ (LGA(θ) +λLDiPO-r(θ)) =min θ E(x f ,y f )∼D f [logπ θ (y f |x f )] +λLDiPO-r(θ)

  69. [70]

    Additionally, we discuss DiPO-Forget (using only LDiPO-f without any retain loss)

    NPO+DiPO(r):This configuration utilizes the standard Gradient Descent loss Equation (30) on the retain set: min θ L(θ) =min θ (LNPO(θ) +λLDiPO-r(θ)) For these settings, the final results are presented in Table 5. Additionally, we discuss DiPO-Forget (using only LDiPO-f without any retain loss). This setup simulates scenarios where retain data might be una...

  70. [2018]

    This bill enacted the CCPA