Pith. sign in

REVIEW 4 major objections 6 minor 41 references

Standard vs. Modular Sampling: Best Practices for Reliable LLM Unlearning

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Standard LLM unlearning practice—single-type neighbor sets and 1:1 forget-retain pairing—is suboptimal; entity-aligned MELU sampling is more stable and effective, the paper argues.

desk verdict A useful, honestly reported empirical sweep of retain-set sampling for LLM unlearning, but two of the three headline 'best practices' are not supported by the paper's own data: the 1:1 inefficiency claim is confounded by an epoch-budget mismatch, and the diverse-neighbor claim contradicts the Balanced result. read the letter →

arxiv 2509.05316 v2 pith:2ZIIXDED submitted 2025-08-29 cs.LG cs.AI

Praveen Bushipaka , Lucia Passaro , Tommaso Cucinotta This is my paper
classification cs.LGcs.AI
keywords LLMunlearningmachineretainsetcompositionneighborsets1:1samplingcyclicMELUentity-level
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether two de facto standards of LLM unlearning are actually good practice: building the retain set from a single kind of neighbor data, and pairing forget and retain samples in a strict 1:1 ratio. On the Wikipedia Person Unlearning benchmark, extended with indirect and syntactically similar neighbors plus a dedicated test set, the authors evaluate three unlearning algorithms (Gradient Difference, DPO, NPO) across seven data and sampling configurations. They find that no single neighbor set is optimal—direct neighbors favor forgetting for gradient-based methods, indirect neighbors protect model utility, and neither alone balances both—and that 1:1 sampling (sequential or random) barely forgets anything, with forget efficacy around 0.35 across all three algorithms. Their positive proposal is Modular Entity-Level Unlearning (MELU), which cycles each forgotten entity only over retain samples of the same entity; it outperforms cyclic sampling under DPO, raising forget efficacy by 12 points while holding model utility near baseline. If the findings hold, unlearning practitioners should abandon 1:1 sampling and construct retain sets with multiple neighbor types.

What carries the argument

Modular Entity-Level Unlearning (MELU), a sampling strategy that pairs each forget target only with its own retain samples—direct and indirect neighbors of that entity, plus randomly assigned general-knowledge samples—instead of cycling forget samples against the whole retain pool. It carries the paper's main positive claim. The paper attributes MELU's stability to a more consistent learning signal: low-variance gradients from related forget-retain pairs, versus the high-variance gradients of cyclic sampling. The neighbor-set taxonomy (direct, indirect, syntactic) is the second load-bearing structure, organizing the data-practice comparison.

What would settle it

Regenerate the indirect-neighbor retain set and the test set with a model outside the LLaMA family (or with the target model itself) and rerun the seven settings: if MELU's 12-point DPO forget-efficacy advantage over cyclic disappears, or the single-neighbor rankings change, the results hinge on the same-family data-generation assumption. A cheaper check is to run cyclic and MELU under DPO across several random seeds and verify that the per-target FE≥0.9 count (3 vs 1) and the roughly 0.82 vs 0.70 forget-efficacy gap replicate; the paper's own note that 1:1 random reaches 0.79 forget efficacy

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that retain-set composition and forget-retain pairing are first-order levers in LLM unlearning, not implementation details. Concretely: (1) a single neighbor set is suboptimal—direct neighbors favor forget efficacy for gradient-based methods, indirect neighbors favor model utility, and the balanced mix behaves differently per algorithm; (2) the standard 1:1 sampling (sequential and random) leaves forget efficacy near 0.35 across all three algorithms, far below the 0.63–0.97 achieved by cyclic and MELU; (3) MELU, which pairs each forget entity only with its own direct and indirect retain samples plus randomly assigned general-knowledge samples, is mo

Load-bearing premise

The indirect neighbor set and test set rest on the assumption that LLaMA 3.3 70B and LLaMA 3.1 8B, being the same model family, share similar pre-training knowledge; if that fails, the retain data and evaluation are biased toward the target model and the forget-efficacy and model-utility comparisons do not measure what they claim.

Editorial extensions

If this is right

  • Unlearning benchmarks should stop defaulting to a single neighbor set; the paper shows direct-only and indirect-only retain sets push the forget-efficacy/model-utility trade-off in opposite directions.
  • Practitioners should treat 1:1 forget-retain sampling as a known-weak baseline: it under-uses the retain set and leaves forget efficacy near pre-unlearning levels for GD, DPO, and NPO alike.
  • MELU is a drop-in replacement for cyclic sampling that requires no new algorithm or compute, and it improves per-target stability and forget efficacy, especially with preference-based methods.
  • Per-entity evaluation matters: targets with high memorization scores resist all settings, so aggregate forget-efficacy/model-utility numbers alone can hide where unlearning fails.
  • With proper sampling (cyclic or MELU), general model utility stays stable on MMLU, whereas changes in data composition produce inconsistent MMLU shifts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If MELU's advantage really comes from lower-variance gradients, the same entity-aligned principle should transfer to sequential unlearning and to other model families; a direct test is measuring per-batch gradient variance under cyclic versus MELU.
  • The 1:1 result may be a repetition or epoch effect rather than an inherent ceiling: the paper notes that 1:1 random reaches 0.79 forget efficacy at 100 epochs, so the practical claim is that 1:1 is inefficient at standard epoch budgets, not that it cannot work.
  • Because the indirect neighbors were generated under the same-family knowledge assumption, the indirect-versus-direct rankings are the least portable result; regenerating the neighbor and test data with a cross-family generator is the natural external-validity check.
  • Selecting only five indirect entities per target leaves the balanced setting fairly thin; knowledge-graph-defined connections, as the authors themselves suggest, could change the balanced-set results.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper investigates two aspects of LLM unlearning practice: retain-set composition (direct, indirect, and balanced neighbor sets) and sampling strategies (1:1 sequential/random, cyclic, and the proposed Modular Entity-Level Unlearning, MELU). Using an extension of the WPU dataset with 20 forget entities, the authors add an indirect-neighbor set and a test set generated with LLaMA 3.3 70B, then evaluate Gradient Difference, DPO, and NPO on LLaMA 3.1 8B. The paper claims three best practices: (1) diverse neighbor sets balance forget efficacy (FE) and model utility (MU-T); (2) 1:1 sampling is inefficient and yields poor results; (3) MELU provides more stable and effective unlearning than cyclic sampling. The contributions are a systematic comparison of these practices and an open-source implementation.

Significance. If the claims held, the paper would give practitioners concrete, low-cost guidance on retain-set design and sampling, an area where the field currently relies on heuristics. The manuscript is transparent: code is released, the data-construction pipeline is described in detail, per-entity results are reported, and limitations are stated explicitly. However, the load-bearing claims overreach the evidence. The 1:1 versus cyclic/MELU comparison is confounded by an 18x difference in retain-set exposures, and the Balanced setting does not actually support the 'diverse neighbor sets' claim—the paper's own results contradict it. Without repeated seeds or significance tests, the stability claims are not established. The work is a useful starting point but currently reads as an exploratory study rather than a validated set of best practices.

major comments (4)
  1. [§4.2, §5.2, footnote 6] The claim that 'standard 1:1 sampling methods are inefficient and yield poor results' (abstract; §1) is not supported because the comparison is not controlled for total retain updates. In §4.2, 1:1 seq/random draw |Dr|=|Df|=98 retain samples per epoch, while Cyclic and MELU use the full 1801-sample retain pool. With 4 epochs, 1:1 sees ~392 retain exposures versus ~7204 for Cyclic/MELU—an ~18x difference. Footnote 6 reports that 1:1 random reaches FE 0.79 and MU-T 0.78 at epoch 100, close to Cyclic/MELU at 4 epochs. The inefficiency conclusion is therefore an artifact of the fixed epoch budget, not a property of the sampling scheme. To support claim (2), the authors need either equal retain-exposure comparisons or a principled compute/epoch budget with ablations over epochs.
  2. [§5.1, Table 1, §6] The conclusion 'By including a diverse range of neighbors - we can improve the balance between forget efficacy and model utility' (§6) is contradicted by Table 1. For GD, Balanced FE 0.97/MU-T 0.55 versus Direct 0.96/0.34 and Indirect 0.89/0.65: Balanced improves FE over Indirect but degrades MU-T relative to Indirect; it does not dominate. For DPO, Balanced FE 0.57/MU-T 0.79 versus Direct 0.57/0.76 and Indirect 0.71/0.77: Balanced does not improve FE over Direct and has lower FE than Indirect. For NPO, Balanced 0.61/0.74 is not Pareto-better than Indirect 0.60/0.78. Section 5.1 itself states 'Balanced fails to achieve better FE and MU-T.' The claim that diverse neighbor sets balance the trade-off needs a defined aggregation or Pareto criterion, or the conclusion must be revised.
  3. [§4.2, §5.2, Appendix A.4] All results are based on a single run per configuration; no seeds, error bars, or significance tests are reported. This is especially load-bearing for the 'stability' claims about MELU. For example, in NPO the difference between Cyclic FE 0.63/MU-T 0.78 and MELU 0.65/0.79 is within likely run-to-run variation for an 8B model with LoRA. The per-target 'improvements' (e.g., Amy Clampitt FE +20%) are also single draws. Without variance estimates, the paper cannot support the 'more stable and effective' wording in claim (3). At minimum, the authors should report repeated seeds and dispersion; ideally, a paired test across seeds for the per-target comparisons.
  4. [§4.1, §7 Limitations] The indirect-neighbor set and the test set are generated with LLaMA 3.3 70B, and the unlearning target is LLaMA 3.1 8B. The paper assumes 'models of the same family would likely share similar pre-training knowledge.' This is acknowledged as a limitation, but it is not merely a generalizability caveat: it affects the validity of the Nind retain set and MU-T evaluation for every comparison. If the 3.3-generation data reflects family-shared knowledge, then the retain/test sets are not independent probes of unlearning quality. The authors should test the assumption explicitly (e.g., compare with a non-family model or a human-authored subset) or at least re-frame all conclusions as family-specific.
minor comments (6)
  1. [Figure 5] Typo: 'Siene' should be 'Seine.' Also, the figure caption does not explain what 'iconic tower' refers to; consider clarifying.
  2. [Table 1] The table lacks clear grouping and a caption describing how FE and MU-T are computed. The row labels 'GA' and 'NPO' appear without distinguishing the unlearning setting from the data-practice rows.
  3. [§5.2] The sentence 'we already achieve better stability (FE and MU-T) with cyclic and MELU with the same number of epochs' is correct but does not address the confound raised in footnote 6. Please clarify that the comparison is under a fixed epoch budget and state whether that budget is meant to be the intended practical regime.
  4. [§2.2] The description of syntactic similarity uses an example 'When was Benedetto Varchi born?' and 'When was Donald Trump born?' but the paper later notes it could not build a full Ns set. This should be cross-referenced so the reader does not assume Ns was included in the experiments.
  5. [§6] The statement 'we couldn't answer why these setups provides more stability' is an honest acknowledgment, but it contradicts the causal explanation in §5.2 ('high-variance gradients... lower variance per batch'). Please either provide evidence for the variance explanation or label it as a hypothesis.
  6. [Appendix A.4] PPL-F and PPL-T columns contain values like '3.09 × 10^82' for GD. These are not meaningful for readers; consider clipping or reporting in log space.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical comparison with acknowledged limitations, not a derivation that reduces to its inputs.

full rationale

The paper does not present a formal derivation or a fitted-parameter-then-prediction chain; it is an empirical study of sampling practices. The main claims—that diverse neighbor sets help, that 1:1 sampling is inefficient at fixed epoch budgets, and that MELU is more stable—are supported by experiments rather than by definition. The 1:1 sampling result is confounded by the fact that 1:1 uses |Dr|=|Df|=98 retain samples per epoch while Cyclic uses all 1801 retain samples, and the paper's own footnote 6 shows 1:1 catches up given 100 epochs; however, this is an experimental design/validity concern, not a circular reduction: the outcome is not equivalent to the input definition, and the paper explicitly qualifies its claim to 'the same number of epochs.' The sister-model assumption (LLaMA 3.3 70B generating Nind and test data for a LLaMA 3.1 8B target) is a dataset-validity limitation that the authors acknowledge in Section 7; it does not make the evaluation circular because the unlearning method is not defined in terms of that data. There are no load-bearing self-citations: the authors cite prior benchmarks and methods but do not rely on their own prior work to justify the central premise. No ansatz is smuggled via citation, and no known result is merely renamed. The paper's internal inconsistencies and confounds are correctness risks, but they do not constitute circularity under the stated criteria.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's empirical claims rest on standard unlearning objectives, a same-family model assumption for data generation, and several hand-set hyperparameters. No fitted constants or invented entities are introduced.

free parameters (4)
  • unlearning_epochs = 4
    The conclusion that 1:1 sampling is inefficient is specific to this epoch count; at 100 epochs DPO 1:1 random reaches FE 0.79 (Section 5.2 footnote 6).
  • balanced neighbor ratio = 1:1 direct-to-indirect per entity
    The Balanced setting selects an equal number of indirect samples as direct samples per entity; other ratios might produce different trade-offs, so the failure of Balanced may be an artifact of this choice.
  • LoRA rank/alpha for unlearning = rank=8, alpha=16
    Chosen hyperparameters, not swept or justified; they affect the optimization trajectory.
  • beta, alpha, gamma in DPO/NPO = beta=0.1, alpha=1, gamma=1
    Set per prior work, not tuned to this dataset; could influence the relative performance of sampling methods.
assumptions (4)
  • domain assumption Unlearning objective L = -Lf + lambda Lr (Section 2.1) captures the intended tradeoff.
    The paper builds on this objective from [13], standard in the field.
  • domain assumption LLaMA 3.3 70B shares pre-training knowledge with LLaMA 3.1 8B (Section 4.1).
    Used to generate Nind and test data; if false, evaluation invalid.
  • ad hoc to paper Edit distance at 40% similarity indicates syntactic neighbor quality (Section 4.1).
    This threshold/interpretation is not validated against actual syntactic similarity metrics.
  • domain assumption WPU forget_20 subset is representative of unlearning difficulty (Section 4.1).
    Only 20 entities are used; results may not generalize to forget_2 or forget_100.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Standard vs. Modular Sampling: Best Practices for Reliable LLM Unlearning." pith.science (2026). https://pith.science/paper/2ZIIXDED

@misc{pith2026250905316,
  author       = {Pith},
  title        = {Pith review of: Standard vs. Modular Sampling: Best Practices for Reliable LLM Unlearning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2ZIIXDED}},
  note         = {Machine review of arXiv:2509.05316}
}
read the original abstract

A conventional LLM Unlearning setting consists of two subsets -"forget" and "retain", with the objectives of removing the undesired knowledge from the forget set while preserving the remaining knowledge from the retain. In privacy-focused unlearning research, a retain set is often further divided into neighbor sets, containing either directly or indirectly connected to the forget targets; and augmented by a general-knowledge set. A common practice in existing benchmarks is to employ only a single neighbor set, with general knowledge which fails to reflect the real-world data complexities and relationships. LLM Unlearning typically involves 1:1 sampling or cyclic iteration sampling. However, the efficacy and stability of these de facto standards have not been critically examined. In this study, we systematically evaluate these common practices. Our findings reveal that relying on a single neighbor set is suboptimal and that a standard sampling approach can obscure performance trade-offs. Based on this analysis, we propose and validate an initial set of best practices: (1) Incorporation of diverse neighbor sets to balance forget efficacy and model utility, (2) Standard 1:1 sampling methods are inefficient and yield poor results, (3) Our proposed Modular Entity-Level Unlearning (MELU) strategy as an alternative to cyclic sampling. We demonstrate that this modular approach, combined with robust algorithms, provides a clear and stable path towards effective unlearning.

Figures

Figures reproduced from arXiv: 2509.05316 by the authors.

Figure 1
Figure 1. Types of Neighbor sets and their connections to the forget sample [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. MELU setting. In cyclic, each entities ( [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. FE and MU-T comparison results. Top row provides the data practices [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Pipeline in creating Indirect connections for retain and test set [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Generation of Test set samples for the Direct Neighbors [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Composition of Forget-Retain samples per target 0 20 40 60 80 100 Total Connections Najaf Daryabandari Ann Brashares Ted Kooser Jorge Semprún Alfred Vogel Paul Gerhardt Giovanni Battista Casti Heinrich Ritter Philip Stanhope, 5th Earl Stanhope Adrienne Monnier Theopomp…
Figure 8
Figure 8. Figure 8: General Model Utility (MMLU) across all experiments. Baseline accuracy is [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Token diversity and per-entity metrics [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: MU-T across various data practices. We find Gradient Based method [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Memorization Scores of Each Entity [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 27 canonical work pages

  1. [1]

    In: Proceedingsofthe34thInternationalConferenceonNeuralInformationProcessing Systems

    Brown, T.B., Mann, B., Ryder, e.a.: Language models are few-shot learners. In: Proceedingsofthe34thInternationalConferenceonNeuralInformationProcessing Systems. NIPS ’20, Curran Associates Inc., Red Hook, NY, USA (2020)

    work page 2020

  2. [2]

    In: The Eleventh International Con- ference on Learning Representations (2023),https://openreview.net/forum?id= TatRHT_1cK

    Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., Zhang, C.: Quantifying memorization across neural language models. In: The Eleventh International Con- ference on Learning Representations (2023),https://openreview.net/forum?id= TatRHT_1cK

    work page 2023

  3. [3]

    In: 30th USENIX Security Symposium (USENIX Security 21)

    Carlini, N., Tramèr, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, Ú., Oprea, A., Raffel, C.: Ex- tracting training data from large language models. In: 30th USENIX Security Symposium (USENIX Security 21). pp. 2633–2650. USENIX Association (Aug 2021), https://www.usenix.org/conference/usenixsecuri...

    work page 2021

  4. [4]

    In: Che, W., Nabende, J., Shutova, E., Pilehvar, M.T

    Chang, H., Lee, H.: Which retain set matters for LLM unlearning? a case study on entity unlearning. In: Che, W., Nabende, J., Shutova, E., Pilehvar, M.T. (eds.) Findings of the Association for Computational Linguistics: ACL

    work page

  5. [5]

    In: Che, W., Nabende, J., Shutova, E., Pilehvar, M.T

    Choi, M., Rim, D., Lee, D., Choo, J.: Opt-out: Investigating entity-level un- learning for large language models via optimal transport. In: Che, W., Nabende, J., Shutova, E., Pilehvar, M.T. (eds.) Proceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Pa- pers). pp. 28280–28297. Association for Computatio...

    work page doi:10.18653/v1/2025.acl-long.1371 2025

  6. [6]

    Chollet, F., Knoop, M., Kamradt, G., Landers, B.: Arc prize 2024: Technical report (2025), https://arxiv.org/abs/2412.04604

    arXiv 2024

  7. [7]

    com/locuslab/open-unlearning (2025), accessed: February 27, 2025

    Dorna, V., Mekala, A., Zhao, W., McCallum, A., Kolter, J.Z., Maini, P.: OpenUn- learning: A unified framework for llm unlearning benchmarks.https://github. com/locuslab/open-unlearning (2025), accessed: February 27, 2025

    work page 2025

  8. [8]

    Fan, C., Liu, J., Lin, L., Jia, J., Zhang, R., Mei, S., Liu, S.: Simplicity prevails: Rethinking negative preference optimization for LLM unlearning (2025),https: //openreview.net/forum?id=Pd3jVGTacT

    work page 2025

Show all 41 references
  1. [9]

    In: 2020 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR)

    Golatkar, A., Achille, A., Soatto, S.: Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks . In: 2020 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR). pp. 9301–9309. IEEE Computer Society, Los Alamitos, CA, USA (Jun 2020). https:...

    2020

  2. [10]

    Grattafiori, A., Dubey, A., et al, A.J.: The llama 3 herd of models (2024),https: //arxiv.org/abs/2407.21783

    2024 arXiv

  3. [11]

    In: International Con- ference on Learning Representations (2022),https://openreview.net/forum?id= nZeVKeeFYf9

    Hu, E.J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. In: International Con- ference on Learning Representations (2022),https://openreview.net/forum?id= nZeVKeeFYf9

    2022

  4. [12]

    In: Rogers, A., Boyd-Graber, J., Okazaki, N

    Jang, J., Yoon, D., Yang, S., Cha, S., Lee, M., Logeswaran, L., Seo, M.: Knowl- edge unlearning for mitigating privacy risks in language models. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Proceedings of the 61st Annual Meet- ing of the Association for Computational Li...

    2023 doi

  5. [13]

    In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024), https://openreview.net/forum?id=tYdR1lTWqh

    Ji, J., Liu, Y., Zhang, Y., Liu, G., Kompella, R.R., Liu, S., Chang, S.: Reversing the forget-retain objectives: An efficient LLM unlearning framework from logit differ- ence. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024), https://open...

    2024

  6. [14]

    In: The Thirty-eight Conference on Neural Information Processing Sys- tems Datasets and Benchmarks Track (2024),https://openreview.net/forum? id=wOmtZ5FgMH Standard vs

    Jin, Z., Cao, P., Wang, C., He, Z., Yuan, H., Li, J., Chen, Y., Liu, K., Zhao, J.: RWKU: Benchmarking real-world knowledge unlearning for large language models. In: The Thirty-eight Conference on Neural Information Processing Sys- tems Datasets and Benchmarks Track (2024),http...

    2024

  7. [15]

    In: Sec- ond Conference on Language Modeling (2025),https://openreview.net/forum? id=Kd97lfFfTu

    Krishnan, A., Reddy, S., Mosbach, M.: Not all data are unlearned equally. In: Sec- ond Conference on Language Modeling (2025),https://openreview.net/forum? id=Kd97lfFfTu

    2025

  8. [16]

    In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies

    Li, J., Galley, M., Brockett, C., Gao, J., Dolan, B.: A diversity-promoting objective function for neural conversation models. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. ...

    2016 doi

  9. [17]

    In: Proceedings of the 41st Inter- national Conference on Machine Learning

    Li, N., Pan, A., Gopal, A., Yue, S., Berrios, e.a.: The WMDP benchmark: Measur- ing and reducing malicious use with unlearning. In: Proceedings of the 41st Inter- national Conference on Machine Learning. Proceedings of Machine Learning Re- search, vol. 235, pp. 28525–28550. PM...

    2024

  10. [18]

    In: Text Summarization Branches Out

    Lin, C.Y.: ROUGE: A package for automatic evaluation of summaries. In: Text Summarization Branches Out. pp. 74–81. Association for Computational Linguis- tics, Barcelona, Spain (Jul 2004),https://aclanthology.org/W04-1013/

    2004

  11. [19]

    Liu, B., Liu, Q., Stone, P.: Continual learning and private unlearning (2022), https://arxiv.org/abs/2203.12817

    2022 arXiv

  12. [20]

    Nature Machine Intelligence 7, 181–194 (2025)

    Liu, S., Yao, Y., Jia, J., et al.: Rethinking machine unlearning for large language models. Nature Machine Intelligence 7, 181–194 (2025). https://doi.org/10. 1038/s42256-025-00985-0, https://doi.org/10.1038/s42256-025-00985-0

    2025 doi

  13. [21]

    In: Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Process- ing

    Liu, Y., Zhang, Y., Jaakkola, T., Chang, S.: Revisiting who‘s harry potter: To- wards targeted unlearning from a causal intervention perspective. In: Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Process- ing. pp. 8708–8731. Association for Comp...

    2024 doi

  14. [22]

    In: First Conference on Language Modeling (2024), https://openreview.net/forum?id=B41hNBoWLo

    Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z.C., Kolter, J.Z.: TOFU: A task of fictitious unlearning for LLMs. In: First Conference on Language Modeling (2024), https://openreview.net/forum?id=B41hNBoWLo

    2024

  15. [23]

    In: Proceedings of the 31st International Con- ference on Computational Linguistics

    Mekala, A., Dorna, V., Dubey, S., Lalwani, A., Koleczek, D., Rungta, M., Hasan, S., Lobo, E.: Alternate preference optimization for unlearning factual knowl- edge in large language models. In: Proceedings of the 31st International Con- ference on Computational Linguistics. pp....

    2025

  16. [24]

    Transactions on Machine Learning Research (2025),https://openreview

    Miranda, M., Ruzzetti, E.S., Santilli, A., Zanzotto, F.M., Bratières, S., Rodolà, E.: Preserving privacy in large language models: A survey on current threats and solu- tions. Transactions on Machine Learning Research (2025),https://openreview. net/forum?id=Ss9MTTN7OL

    2025

  17. [25]

    In: Rosen- thal, S., Rosá, A., Ghosh, D., Zampieri, M

    Premptis, I., Lymperaiou, M., Filandrianos, G., Menis Mastromichalakis, O., Voulodimos, A., Stamou, G.: AILS-NTUA at SemEval-2025 task 4: Parameter- efficient unlearning for large language models using data chunking. In: Rosen- thal, S., Rosá, A., Ghosh, D., Zampieri, M. (eds....

    2025

  18. [26]

    Qiu, X., Shen, W.F., Chen, Y., Kurmanji, M., Cancedda, N., Stenetorp, P., Lane, N.D.: How data inter-connectivity shapes llms unlearning: A structural unlearning perspective (2025), https://arxiv.org/abs/2406.16810 16 P.Bushipaka et al

    2025 arXiv

  19. [27]

    In: Pro- ceedings of the 37th International Conference on Neural Information Processing Systems

    Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C.D., Finn, C.: Direct preference optimization: your language model is secretly a reward model. In: Pro- ceedings of the 37th International Conference on Neural Information Processing Systems. NIPS ’23, Curran Associa...

    2023

  20. [28]

    In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)

    Reimers, N., Gurevych, I.: Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 3...

    2019

  21. [29]

    In: The Twelfth International Conference on Learning Representations (2024),https://openreview.net/forum? id=kmn0BhQk7p

    Staab, R., Vero, M., Balunovic, M., Vechev, M.: Beyond memorization: Violating privacy via inference with large language models. In: The Twelfth International Conference on Learning Representations (2024),https://openreview.net/forum? id=kmn0BhQk7p

    2024

  22. [30]

    org/abs/2410.02879

    Thaker, P., Hu, S., Kale, N., Maurya, Y., Wu, Z.S., Smith, V.: Position: Llm unlearning benchmarks are weak measures of progress (2025), https://arxiv. org/abs/2410.02879

    2025 arXiv

  23. [31]

    In: Oh,A.H.,Agarwal,A.,Belgrave,D.,Cho,K.(eds.)AdvancesinNeuralInformation Processing Systems (2022),https://openreview.net/forum?id=u3vEuRr08MT

    Tirumala, K., Markosyan, A.H., Zettlemoyer, L., Aghajanyan, A.: Memorization without overfitting: Analyzing the training dynamics of large language models. In: Oh,A.H.,Agarwal,A.,Belgrave,D.,Cho,K.(eds.)AdvancesinNeuralInformation Processing Systems (2022),https://openreview.n...

    2022

  24. [32]

    In: Proceedings of the 31st International Conference on Neural Information Processing Systems

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. p. 6000–6010. NIPS’17, Curran Associates Inc., Red...

    2017

  25. [33]

    In: The Thirteenth International Conference on Learning Representations (2025),https: //openreview.net/forum?id=huo8MqVH6t

    Wang, Q., Zhou, J.P., Zhou, Z., Shin, S., Han, B., Weinberger, K.Q.: Rethink- ing LLM unlearning objectives: A gradient perspective and go beyond. In: The Thirteenth International Conference on Learning Representations (2025),https: //openreview.net/forum?id=huo8MqVH6t

    2025

  26. [34]

    Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., Chen, W.: Mmlu-pro: A more robust and challenging multi-task language understanding benchmark (2024), https://arxiv.org/a...

    2024 arXiv

  27. [35]

    In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024),https:// openreview.net/forum?id=8Dy42ThoNe

    Yao, Y., Xu, X., Liu, Y.: Large language model unlearning. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024),https:// openreview.net/forum?id=8Dy42ThoNe

    2024

  28. [36]

    In: The Thirteenth International Con- ference on Learning Representations (2025),https://openreview.net/forum?id= Q1MHvGmhyT

    Yuan, X., Pang, T., Du, C., Chen, K., Zhang, W., Lin, M.: A closer look at ma- chine unlearning for large language models. In: The Thirteenth International Con- ference on Learning Representations (2025),https://openreview.net/forum?id= Q1MHvGmhyT

    2025

  29. [37]

    (eds.) Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., Choi, Y.: HellaSwag: Can a machine really finish your sentence? In: Korhonen, A., Traum, D., Màrquez, L. (eds.) Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics. pp. 4791–4800. Associati...

    2019 doi

  30. [38]

    In: First Conference on Language Modeling (2024), https://openreview.net/forum?id=MXLBXjQkmb Standard vs

    Zhang, R., Lin, L., Bai, Y., Mei, S.: Negative preference optimization: From catas- trophic collapse to effective unlearning. In: First Conference on Language Modeling (2024), https://openreview.net/forum?id=MXLBXjQkmb Standard vs. Modular Sampling: Best Practices for Reliable...

    2024

  31. [39]

    name": name of the person,

    Zhang, S., Hu, Y., Bian, G.: Research on string similarity algorithm based on levenshtein distance. In: 2017 IEEE 2nd Advanced Information Technology, Electronic and Automation Control Conference (IAEAC). pp. 2247–2251 (2017). https://doi.org/10.1109/IAEAC.2017.8054419 A Appen...

    2017

  32. [41]

    ROUGE-L provides the longest sequence overlap and the verbatim memory of the Unlearned Model(M ; θ∗)

    score between the generated responseg(x; θ∗) and the ground-truth answer y, written asROU GE− L(g(x; θ∗), y). ROUGE-L provides the longest sequence overlap and the verbatim memory of the Unlearned Model(M ; θ∗). CosineSimilarity(CS) measuresthesemanticsimilarityofthemodel’sout...

  33. [2025]

    5966–5982

    pp. 5966–5982. Association for Computational Linguistics, Vienna, Aus- tria (Jul 2025).https://doi.org/10.18653/v1/2025.findings-acl.310, https: //aclanthology.org/2025.findings-acl.310/

    2025 doi

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.