Pith. sign in

REVIEW 2 major objections 5 minor 19 references

The standard test for machine unlearning can certify models that still hold the deleted information.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 14:05 UTC pith:P5TREXAN

load-bearing objection A careful empirical paper that exposes a real flaw in trained-probe unlearning evaluation, but it ships without code/data and one advertised theorem is missing. the 2 major comments →

arxiv 2607.19442 v1 pith:P5TREXAN submitted 2026-07-21 cs.LG cs.AI

Unlearning as Distribution Restoration: A Controlled Counterfactual Study, a Validated Selective Screen, and the Limits of Oracle-Free Certification

Sen Yang , Yuen-Hei Yeung This is my paper
classification cs.LG cs.AI
keywords machine unlearningdistribution restorationretraining oracletrained-probe criterionheld-out evaluationcounterfactual identifiabilityselective certificationLLM unlearning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that the standard way of evaluating machine unlearning—checking how close an unlearned model is to a retrained oracle on the same question phrasings used during training—can certify models that still retain the forgotten knowledge. In a controlled testbed with a matched retraining reference, candidates that this criterion rates as adequate score held-out paraphrases of the forget facts 2.82 nats below the never-learned level, meaning the knowledge survives rephrasing. The paper recasts good unlearning as restoration to the retrained distribution, proposes a base-anchored held-out rank screen as a necessary test, and shows that absolute certification thresholds fail even the reference model itself. It also proves an identifiability limit: an oracle-free forget threshold exists only for genuinely counterfactual facts, with the TOFU benchmark as the predicted boundary case.

Core claim

The central claim is that matching the retrained model on trained probes is not evidence of restoration. Candidates that the trained-probe oracle-KL criterion rates as adequate score held-out forget facts a mean of -2.82 nats (cluster CI [-3.16, -2.48]) below the never-learned level, meaning they retain paraphrase-recoverable knowledge of the forget set while matching the reference on the phrasings the criterion happens to query. Consequently, benchmarks that treat this criterion as ground truth are systematically misleading. The paper's constructive alternative is to evaluate unlearning as restoration to a matched retraining reference, using base-anchored held-out deltas and a round-trip re

What carries the argument

The carrying instrument is a base-anchored held-out rank screen: for a candidate model, compute base-anchored deltas (the candidate's NLL minus the base model's NLL) on the forget facts and on never-learned probe facts, then run a one-sided exact Mann–Whitney rank test to exclude candidates whose forget deltas fall significantly below their own probe deltas—a sign of residual knowledge. This is paired with a round-trip residual: reacquire the forget set into the candidate and measure the KL divergence back to the injected model, used as a ranking signal. An identifiability theorem, built on a counterfactual-exchangeability assumption that the oracle's forget threshold equals a known function

Load-bearing premise

The whole oracle-free framework rests on the counterfactual-exchangeability assumption that the retrained oracle's forget threshold can be read off the injected model's distribution of never-learned probe scores; if retained data can reconstruct the forgotten fact, this assumption fails and no oracle-free selector can be consistent.

What would settle it

In any cell of the paper's testbed, find a candidate that the trained-probe oracle-KL criterion rates adequate but whose held-out forget-NLL gap from the never-learned level is within the replication tolerance (≤0.93 nats) of zero; the paper predicts a gap of roughly -2.82 nats, so such a candidate would directly contradict its central negative result. Alternatively, show that the matched retraining reference itself fails the paper's base-anchored screen in a non-trivial share of cells, which would undermine the screen's validity as a necessary test.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Benchmarks that use trained-probe oracle-KL as the ground-truth unlearning metric inherit a systematic flaw: they may certify methods that merely suppress phrasing-level evidence while the knowledge survives rephrasing.
  • Absolute retain and round-trip thresholds should be replaced by reference-relative bars set by the retraining replication noise; otherwise even the retraining reference itself fails them in nearly every configuration.
  • Base-anchored held-out screens are valuable as necessary tests—they reject under-forgetting with high sensitivity—but they are not sufficiency certificates and must be paired with other axes.
  • Oracle-free selection is only sound for counterfactual facts; for inferable facts, where retained data can reconstruct the forgotten knowledge, the retraining reference is unavoidable.
  • Forward-only certification is not adversarially sound: a fixed-magnitude logit-suppression attack defeats the full forward battery in a measurable fraction of cells, so relearning axes or white-box access are required for stronger guarantees.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The roughly 2.8-nat gap implies that the field's default metric measures phrasing-level matching rather than knowledge deletion; a direct testable extension is that unlearning methods optimized on trained probes will systematically fail on paraphrase-based adversarial probes, so rephrasing attacks should become a standard stress test.
  • The identifiability boundary suggests a practical deployment rule: estimate the counterfactual headroom (the paper's G-hat, which tracks true headroom at rank correlation 0.97 on a graded axis) before trusting any oracle-free selector; for inferable facts, abstain or demand the reference.
  • The round-trip residual—reacquire and compare—could transfer to other deletion settings beyond language models, such as image or tabular unlearning, wherever a re-acquisition protocol is feasible, and could serve as a general test of whether a method truly deleted versus merely hid.
  • The drift-seam detection property implies that routing-based suppression leaves a detectable distributional trace; a sophisticated adversary could plausibly attenuate that seam by also adjusting probe-side responses, which points to adaptive challenge panels as the next test.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper studies machine unlearning evaluation in a controlled nonce-fact testbed with a matched retraining reference (M_oracle). It reports three main empirical findings: (1) the common trained-probe oracle-KL criterion can favor candidates that retain held-out forget knowledge, quantified as a −2.82-nat gap between candidates' held-out forget NLLs and their never-learned probe NLLs; (2) fixed absolute retain/round-trip thresholds are mis-scaled, since the injected model and even the retraining reference fail them in most of 45 model–seed cells; and (3) a base-anchored held-out rank screen is a validated necessary test with measured sensitivity on a sealed challenge panel, while a damage-relative recalibration yields a selective partial positive in 15/45 cells. The paper also presents an identifiability theorem delimiting when oracle-free selection is possible, with TOFU as the predicted boundary case, and reports that a fixed logit-suppression attack defeats the full forward battery in 12/45 cells, scoping the method as an empirical selective test rather than a formal certificate.

Significance. If the central falsification holds, the field's default ground-truth metric for unlearning—matching a retrained oracle on trained probes—is systematically misleading, and benchmarks built on it inherit the flaw. The paper's strengths are substantial: a controlled dose-matched testbed with a matched retraining reference, independent redraws quantifying replication noise, a sealed known-label challenge panel, frozen preregistered recalibration rules, cluster-bootstrap confidence intervals, every cell reported, and an unusually candid limitations section. The method also ships a finite-sample certifiability bound (Proposition 1) and an identifiability theorem (Theorem 1) that together clarify when oracle-free selection is even possible. The empirical apparatus is reproducible in principle, though the promised audit-pool hash is deferred. The main risk is that the headline residual-knowledge gap is not yet shown to be significantly larger than the baseline F-versus-P gap exhibited by the matched reference itself; if that baseline gap is substantial, the −2.82-nat interpretation is confounded by fact-set difficulty rather than retained knowledge.

major comments (2)
  1. [Section 3.1 / Section 4 (ΔF/ΔP definitions)] The headline −2.82-nat claim compares trained-probe-adequate candidates' held-out forget deltas to their own never-learned probe deltas. The interpretation as retainable knowledge assumes that, had the candidate not learned F, its ΔF distribution would match its ΔP distribution. The manuscript justifies this by asserting M0 is 'symmetric' in F and P, but M0 is not the relevant counterfactual for a candidate derived from Minj; the matched reference Mor is. The reference is available in all 45 cells, yet the paper reports only that it passes the screen in 43/45 (Section 5), not the reference's own F-versus-P delta gap. A one-sided rank test with m=4, n=16 can have low power, so 'passing' does not quantify the gap. Please report the reference's median and distribution of ΔF−ΔP per cell, and show that the −2.82 gap for trained-probe-adequate candidates is significantly larger than the refere
  2. [Abstract; §1 contribution 5; §6; §11] The paper repeatedly advertises a 'finite-query impossibility argument' (also 'finite-query impossibility boundary' and, in Section 11, 'our finite-query argument'), and Section 6 refers to 'the finite-query impossibility argument of Section 9.' Section 9 contains Theorem 1, Lemma 1, and Proposition 2 on identifiability, and Proposition 1 in Section 4 is a finite-sample certifiability bound for the rank test; none of these is a query-complexity impossibility theorem. Either supply the promised argument/proof or revise the abstract and contributions to describe what is actually proved: identifiability limits and a measured adversarial evasion rate. This is a discrepancy between promises and content that should be resolved before publication.
minor comments (5)
  1. [Appendix A] The appendix states that 'we publish a cryptographic hash of the audit pool with this paper,' but then says the hash and repository link 'will be added to this section in a revision.' The sealed-pool claim is therefore not currently verifiable. Include the hash or a timestamped commitment in the version of record.
  2. [Section 9 vs Section 4] The theoretical selector in Lemma 1 uses a threshold on q_a(w) against Γ(ν_w), while the implemented screen in Section 4 is a one-sided rank test on base-anchored deltas. Clarify how the rank test operationalizes the threshold condition, especially with only m=4 forget items and n=16 probes, and how the p-value floor of Proposition 1 maps to the Γ-based cutoff.
  3. [Proposition 1] The statement and proof of Proposition 1 assume no ties in the rank lattice, but the implementation audit notes that ties push the code onto a normal approximation. State explicitly that the exact-lattice bound applies only in the tied-free case, and whether the 1/4845 floor was observed for all 238 candidates that attained it.
  4. [Table 2] The 0.80-vs-5.17 comparison is carefully caveated, but it is computed only in the 15 cells where the recalibrated certificate does not abstain. Please state whether the trained-probe best's 5.17-nat gap is also representative of the full 45 cells, or explain why it is not reportable there.
  5. [Section 5] The claim that the absolute bar fails 'its own reference' is supported by pass rates, but the reference's absolute retain-NLL and round-trip residual distributions are only summarized by medians and pass counts. Reporting per-cell values or a scatter plot would make the threshold-mis-scaling diagnosis more transparent.

Circularity Check

1 steps flagged

One disclosed partly-definitional supporting comparison; the central falsification and screen validation are independent, empirically benchmarked results.

specific steps
  1. self definitional [Section 7, 'Selection quality (same-cell)' and Table 2; certification rule (ii)]
    "That the recalibrated pick scores best on the recalibrated axes is partly definitional (the selector optimizes those axes), so we do not present 0.80 vs 5.17 as a head-to-head selection benchmark"

    Certification rule (ii) requires |med(ΔF(C))−med(ΔF(reference))|≤δf, so any non-abstaining pick is within δf of the reference on the forget-equivalence axis by construction. The later report that the recalibrated pick lies 0.80 nats from the reference 'within retraining noise' therefore restates the inclusion criterion rather than measuring an independent outcome. The paper itself flags this as 'partly definitional' and demotes it to a supporting comparison; the central falsification (Section 3.1) and the screen validation (Section 6) do not rely on this number.

full rationale

The main empirical claims are not circular. Section 3.1's −2.82-nat finding is a direct measurement: candidates rated adequate by the trained-probe criterion are scored on held-out forget paraphrases against never-learned probes, and the gap is an observed quantity. The interpretation that this reflects retained knowledge depends on Assumption 3 (CE), but CE is explicitly labeled a strong counterfactual invariance assumption, not a derived result; the paper's honest-scope paragraph acknowledges when it fails. The forgery-qualification screen is validated on a sealed, known-label challenge panel with the reference run through the battery, which is external validation rather than self-justification. The absolute-bar audit is likewise an empirical audit of the reference and injected model. The only step that reduces by construction is the Section 7 'selection quality' comparison: a certified pick is defined to satisfy the forget-equivalence tolerance, so reporting that the pick is within retraining noise on that axis is a restatement of the selection rule. The paper openly calls this 'partly definitional' and does not make the recalibration load-bearing, so it is a minor, disclosed definitional element rather than a hidden circularity. I also note two non-circular omissions that the manuscript itself flags: the advertised finite-query impossibility theorem is not stated as a theorem (Section 9 contains only the identifiability theorem, and Section 11 gives a scope), and the audit-pool hash is deferred to a revision. These are missing-support issues, not circularity. Overall, no self-citation chain is load-bearing, no fitted parameter is renamed as a prediction, and no uniqueness result is imported from the authors' prior work; the central contributions are benchmarked against externally constructed references and never-learned probes.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 2 invented entities

The empirical core rests on the testbed design (nonce facts, matched retraining reference, sealed probe pools) plus standard statistics; the theoretical boundary rests on three explicitly stated assumptions (TR, RS, CE), two of which are ad hoc to this framework. The recalibrated certificate's tolerances (δ_f, ε_f, round-trip floor) are data-fitted from three reference redraws. No new physical entities are postulated; the matched reference and probe pools are constructed artifacts with reproducible construction rules but no external falsifiable handle.

free parameters (6)
  • per-family replication tolerance δ_f = 0.82–2.92 nats (median 0.93) per family
    P90 per-fact held-out delta gap between three independent reference redraws; used as the forget-equivalence tolerance in the §7 recalibrated certificate. Fitted to the same reference draws it later certifies against; certification count drops from 15/45 to 4/45 cells under the global minimum tolerance.
  • reference-noise margins ε_f = draw-to-draw spread of retain-NLL and round-trip residual per family
    Shift applied to cross-draw references in §7 conditions (iii)-(iv); estimated from the same three redraws and therefore uncertain (noted in §11).
  • round-trip floor per cell = median floor 0.92 nats (Minj self-closure)
    The re-acquire-into-injected-model residual subtracted as common-mode budget drift in the recalibrated bar (§5).
  • absolute thresholds for retain/round-trip = 0.5 nats
    The frozen absolute bar retired by its own reference (§5); kept because the headline count (only 3/820 candidates certify) depends on it.
  • logit-suppression attack magnitude = −10 on forget-answer logits
    Fixed-magnitude adversarial probe used to measure the forward-only boundary in §6; the resulting 12/45 evasion rate is magnitude-dependent.
  • significance level α and candidate pool size K = α=0.05, K=16/20 per cell
    Holm-corrected α and pool size enter Proposition 1's combinatorial requirement C(m+n,m) ≥ K/α; standard conventions but chosen by hand for the testbed.
axioms (6)
  • standard math Standard rank-test and multiple-testing facts: one-sided exact Mann-Whitney p-value lattice, Holm step-down, Clopper-Pearson bounds.
    Used in Proposition 1, the screen, and the 12/45 upper bound; standard and checkable.
  • domain assumption Matched retraining reference is a valid counterfactual: replaying the injection stream without F yields a model that never saw F and has comparable retain competence.
    The entire audit compares candidates to M_oracle; if the reference construction leaks F or mismatches retain competence, all reference-relative numbers shift. §2 acknowledges bf16 nondeterminism but treats the reference as the gold standard.
  • domain assumption Nonce facts under continued pretraining create clean counterfactual structure, with held-out paraphrase NLLs as a sensitive residual-knowledge surface.
    The −2.82 nats finding and the screen's 45/45 rejection rely on held-out phrasings being unlearnable-but-detectable; §3.1 and the challenge panel are its only evidence.
  • ad hoc to paper Assumptions TR (threshold separation and fixed tie-breaking) and RS (residual soundness): adequate sets are separated by any observable equality of X, and the residual-minimizer among qualified candidates is ε-close to the oracle.
    Explicit assumptions in §9; Theorem 1 is a conditional equivalence under them. The paper labels them strong and notes they are not consequences of oracle-free observability.
  • ad hoc to paper Counterfactual exchangeability (CE): τ⋆(w)=Γ(NLL_Mor on forgotten)=Γ(NLL_Minj on never-learned probes)=Γ(ν_w).
    Assumption 3 in §9; makes the never-learned reference a valid proxy for the oracle's forget threshold. The paper's own Proposition 2 shows it fails for inferable facts; TOFU is the predicted failure, so the nonce testbed's validity rests on CE holding there.
  • domain assumption Re-acquisition is a valid restoration probe: fine-tuning a candidate on F to convergence measures how successfully the forgotten distribution was restored, with residual KL to Minj as the summary.
    The round-trip certificate's meaning; §5 shows the floor is dominated by reacquisition budget drift and one family has intrinsic path-dependence, so the shared-floor subtraction is itself an assumption.
invented entities (2)
  • matched retraining reference M_oracle no independent evidence
    purpose: Gold-standard counterfactual: identical stream with F removed, used as the restoration target and as the audit standard for all criteria.
    Constructed for this paper; no falsifiable handle outside the paper's protocol, though the construction rules in §2/Appendix B make it reproducible.
  • never-learned probe pool (n=16 per cell) and sealed audit pool no independent evidence
    purpose: Reference distribution for base-anchored delta ranking; sealed challenge panel with known labels used once for instrument validation.
    Internal artifacts; a cryptographic hash is promised but not present in the text, and the pool is withheld to preserve one-time use.

pith-pipeline@v1.3.0-alltime-deepseek · 15433 in / 22695 out tokens · 188240 ms · 2026-08-01T14:05:33.286888+00:00 · methodology

0 comments
read the original abstract

Machine unlearning is commonly evaluated by matching a retrained oracle on trained probes. In a controlled nonce-fact testbed with a matched retraining reference, we find this criterion can favor methods that retain held-out knowledge: candidates it rates adequate score held-out forget facts $-2.82$ nats below the never-learned level (cluster CI $[-3.16,-2.48]$). We recast unlearning as restoration to the matched reference and audit oracle-free screens and certificate-style criteria across 45 model-seed cells spanning five open architecture families. The reference itself falsifies an absolute retain/round-trip certificate: the injected model, which retains the retain set by construction, fails the fixed retain threshold in 41/45 cells and its own round trip in 31/45, and the reference fully certifies in only 1/45. A base-anchored held-out screen remains strong as a selective necessary test: on a sealed challenge suite it rejects the injected model in 45/45 cells, accepts the reference in 44/45, and partially detects entity-routing suppression (35/45); it is a necessary test with measured sensitivity, not a sufficiency certificate. A damage-relative recalibration anchored to the reference's own operating point certifies a small subset in 15/45 cells; where it does not abstain, its picks lie within retraining noise (0.80 nats) on the axes it optimizes, while the common trained-probe criterion sits 5.17 nats away (a supporting comparison, not a head-to-head benchmark). A fixed-magnitude logit-suppression attack defeats the full forward battery in 12/45 cells, so forward-only certification is not sound; our method is an empirical selective test for methods-as-produced. An identifiability theorem delimits which facts admit an oracle-free forget threshold at all, with TOFU as the predicted boundary case.

Figures

Figures reproduced from arXiv: 2607.19442 by Sen Yang, Yuen-Hei Yeung.

Figure 1
Figure 1. Figure 1: Diagnostic (pooled candidates, 45 matrix cells). Suppression methods (GA/NPO) over-forget [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 17 linked inside Pith

  1. [3]

    Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer

    URLhttps://arxiv.org/ abs/2012.07805. Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In2022 IEEE Symposium on Security and Privacy (SP),

    Pith/arXiv arXiv 2012

  2. [8]

    Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens van der Maaten

    URLhttps://arxiv.org/abs/ 2308.07707. Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens van der Maaten. Certified data removal from machine learning models. InProceedings of the 37th International Conference on Machine Learning,

    Pith/arXiv arXiv

  3. [11]

    Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D

    URLhttps://arxiv.org/abs/2302.09880. Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann- Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning.arXiv preprint arXiv:2403.03218,

    Pith/arXiv arXiv

  4. [12]

    Yujian Liu, Yang Zhang, Tommi Jaakkola, and Shiyu Chang

    URLhttps://arxiv.org/abs/ 2403.03218. Yujian Liu, Yang Zhang, Tommi Jaakkola, and Shiyu Chang. Revisiting who’s harry potter: Towards targeted unlearning from a causal intervention perspective.arXiv preprint arXiv:2407.16997,

    Pith/arXiv arXiv

  5. [13]

    Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C

    URL https://arxiv.org/abs/2407.16997. Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. TOFU: A task of fictitious unlearning for LLMs. InConference on Language Modeling (COLM),

    Pith/arXiv arXiv

  6. [15]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov

    URLhttps://arxiv.org/abs/2111.09832. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt.arXiv preprint arXiv:2202.05262, 2022a. URLhttps://arxiv.org/abs/2202.05262. 13 Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer.arXiv preprint a...

    Pith/arXiv arXiv 2007

  7. [16]

    Jonas Peters, Dominik Janzing, and Bernhard Schölkopf.Elements of Causal Inference: Founda- tions and Learning Algorithms

    URLhttps://arxiv.org/abs/2007.02923. Jonas Peters, Dominik Janzing, and Bernhard Schölkopf.Elements of Causal Inference: Founda- tions and Learning Algorithms. MIT Press,

    Pith/arXiv arXiv 2007

  8. [18]

    Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A

    URLhttps://arxiv.org/abs/2103.03279. Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, and Chiyuan Zhang. MUSE: Machine unlearning six-way evaluation for language models. InInternational Conference on Learning Representations (ICLR),

    Pith/arXiv arXiv

  9. [19]

    Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S

    URLhttps: //arxiv.org/abs/1610.05820. Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. InProceedin...

    Pith/arXiv arXiv

  10. [20]

    Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue

    URLhttps: //arxiv.org/abs/2203.05482. Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. Machine unlearning of pre-trained large language models.arXiv preprint arXiv:2402.15159,

    Pith/arXiv arXiv

  11. [21]

    Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei

    URLhttps: //arxiv.org/abs/2402.15159. Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. InConference on Language Modeling (COLM),

    Pith/arXiv arXiv

  12. [2017]

    Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh

    URLhttps://mitpress.mit.edu/9780262037310/ elements-of-causal-inference/. Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. InAdvances in Neural Information Processing Systems,

    arXiv

  13. [2019]

    Lucas Bourtoule, Varun Chandrasekaran, Christopher A

    URLhttps://arxiv.org/abs/1812.03789. Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In2021 IEEE Symposium on Security and Privacy (SP),

    Pith/arXiv arXiv

  14. [2020]

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi

    URLhttps://arxiv.org/abs/1911.03030. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. InInternational Conference on Learning Representations (ICLR),

    Pith/arXiv arXiv 1911

  15. [2021]

    URLhttps://arxiv.org/abs/1912.03817. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In30th USENIX Security Symposium,

    Pith/arXiv arXiv 1912

  16. [2022]

    Vineeth Dorna, Anmol Mekala, Wenlong Zhao, Andrew McCallum, Zachary C

    URLhttps://arxiv.org/abs/2112.03570. Vineeth Dorna, Anmol Mekala, Wenlong Zhao, Andrew McCallum, Zachary C. Lipton, J. Zico Kolter, and Pratyush Maini. Openunlearning: Accelerating llm unlearning via unified benchmarking of methods and metrics.arXiv preprint arXiv:2506.12618,

    Pith/arXiv arXiv

  17. [2023]

    Jack Foster, Stefan Schoepf, and Alexandra Brintrup

    URLhttps://arxiv.org/abs/2310.02238. Jack Foster, Stefan Schoepf, and Alexandra Brintrup. Fast machine unlearning without retraining through selective synaptic dampening.arXiv preprint arXiv:2308.07707,

    Pith/arXiv arXiv

  18. [2024]

    Ronen Eldan and Mark Russinovich

    URLhttps://arxiv.org/abs/2411.12103. Ronen Eldan and Mark Russinovich. Who’s harry potter? approximate unlearning in llms.arXiv preprint arXiv:2310.02238,

    Pith/arXiv arXiv

  19. [2025]

    Jai Doshi and Asa Cooper Stickland

    URLhttps://arxiv.org/abs/2506.12618. Jai Doshi and Asa Cooper Stickland. Does unlearning truly unlearn? a black box evaluation of llm unlearning methods.arXiv preprint arXiv:2411.12103,

    arXiv