Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

BLUR: A Benchmark for LLM Unlearning Robust to Forget-Retain Overlap

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that standard LLM unlearning benchmarks mislead because their forget and retain queries barely overlap, and shows that under BLUR's combined queries and benign relearning, existing unlearning methods lose retain quality…

desk verdict BLUR is a useful benchmark resource, but the headline claim of significant performance drops under forget-retain overlap is not yet supported because the combined-query evaluation lacks a base-model baseline and error bars. read the letter →

arxiv 2506.15699 v1 pith:YMOCHQNC submitted 2025-05-28 cs.LG cs.AI

Shengyuan Hu , Neil Kale , Pratiksha Thaker , Yiwei Fu , Steven Wu , Virginia Smith This is my paper
classification cs.LGcs.AI
keywords machineunlearningLLMforget-retainoverlapbenchmarkrelearningattackRouge-Lrecallgradientascentrobustevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that current LLM unlearning benchmarks paint a false picture, because their forget and retain queries are nearly unrelated and therefore easy for an unlearning method to keep apart. It introduces BLUR, a benchmark that adds realistic overlap: combined queries that mix one forget question with one retain question, forget keywords inserted into retain multiple-choice questions, and relearning datasets of low, medium, and high relevance to the forget set. Evaluated across four existing unlearning testbeds, BLUR finds that retain quality drops sharply on the combined queries, with 2-3x reductions on WHP and RWKU, and that benign relearning recovers a substantial share of supposedly unlearned knowledge. The paper's central empirical claim is that this kind of robust evaluation changes method rankings, with simple gradient ascent matching or beating more recent approaches like NPO and SCRUB. A fair reader should care because if the claim holds, reported unlearning results overstate how ready these methods are for deployment.

What carries the argument

The central mechanism is the forget-retain overlap perturbation, implemented in BLUR as three evaluation instruments: combined queries formed by concatenating one forget question with one retain question; insertion of unlearn-set keywords into incorrect multiple-choice answers of retain questions; and relearning datasets D_hi, D_mid, and D_low, of high, medium, and low relevance to the forget set, used to finetune the unlearned model. These instruments are paired with Rouge-L recall, the length-normalized longest-common-subsequence similarity between base model answers and unlearned model answers, which the paper uses to quantify both forget quality and retain quality.

What would settle it

Run a matched-length control on the same models: build combined queries from two retain questions with the same total length as the forget+retain pairs and measure Rouge-L recall on both. If retain+retain pairs show a similar drop, the 2-3x degradation in BLUR is a length artifact; if they do not, the drop is caused by forget-retain overlap.

Watch

Extended reading notes

Core claim

BLUR's central discovery is that the prevailing evaluation protocol hides a failure mode: unlearning methods that score well on disjoint forget and retain sets degrade when the two kinds of knowledge appear in the same query or when the model is finetuned on benign text related to the forget set. Across TOFU, WHP, WMDP, and RWKU, concatenating a retain question to a forget question reduces Rouge-L recall on the retain portion relative to the same retain question asked alone, with the sharpest drops on WHP and RWKU. Relearning on text of increasing relevance to the forget set produces increasing Rouge-L recovery of forgotten answers, and even nonsensical Lorem Ipsum text causes some recovery. On the retain-MCQ perturbation, gradient-based methods are largely robust to swapped-in forget keywords, while a representation-perturbation method (RMU) is not. The authors conclude that robust evaluation can rank simple gradient ascent above more recent methods, and that benchmark design, not just method design, determines measured unlearning progress.

Load-bearing premise

The central empirical result assumes that Rouge-L recall between base and unlearned model answers is a fair measure of retain quality, so the lower scores on longer combined queries reflect genuine forget-retain interference rather than artifacts of output length or truncation.

Editorial extensions

If this is right

  • Benchmark numbers obtained with disjoint forget and retain sets should not be read as deployment readiness; BLUR shows that simply appending a retain question to a forget question can cut measured retain quality by 2-3x.
  • Unlearning methods are vulnerable to benign relearning: finetuning on innocuous text, even text with no semantic content, partially restores supposedly forgotten knowledge, and more relevant relearn text restores more.
  • Robustness to forget-retain overlap is method-specific: gradient-based methods withstand forget-keyword insertion into retain MCQs, while representation-perturbation methods such as RMU do not.
  • On a benchmark that includes overlap, simpler baselines such as gradient ascent can match or beat newer methods such as NPO and SCRUB, so recent progress claims need to be re-examined under this evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because combined queries are simple concatenations and generation length is not controlled, part of the reported Rouge-L drop may be a length artifact; a retain+retain concatenation control of the same total length would separate genuine interference from truncation effects.
  • Editorial inference: the benchmark structure suggests a practical acceptance rule for future unlearning methods: retain quality on combined queries should be close to retain quality on isolated queries, and forget knowledge should not resurface after relearning on unrelated text; BLUR ships the data needed to test both.
  • Editorial inference: the keyword-insertion test could be extended beyond multiple-choice questions, since distractor keywords might also derail open-ended generations on retain topics, a failure mode the current accuracy-based metric would miss.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces BLUR, a benchmark for evaluating LLM unlearning under forget-retain overlap. It augments four existing unlearning benchmarks (TOFU, WHP, WMDP, RWKU) with two kinds of stress tests: (i) combined evaluation queries that concatenate forget and retain questions or insert unlearned keywords into retain MCQs, and (ii) relearning datasets of varying relevance to the forget set. The authors evaluate six unlearning methods (GA, GA+KL, NPO, NPO+KL, SCRUB, and RMU where applicable). The main empirical claims are that retain quality drops substantially on combined queries, that benign relearning recovers forgotten knowledge roughly in proportion to relearn-set relevance, and that simple methods such as gradient ascent often match or outperform more recent methods. The paper concludes that existing benchmarks overstate unlearning effectiveness.

Significance. BLUR addresses an important gap: current unlearning benchmarks evaluate forget and retain sets in isolation, while real deployments mix them. The dataset construction is a useful contribution, and the release of code and data is a strength. The crossover queries in Appendix C are a particularly valuable extension. If the central results are confirmed, they would change how the community compares unlearning methods. However, the main quantitative evidence currently rests on a metric comparison without a necessary baseline, and the claims of 'significant' differences are not supported by any variance or statistical testing. The resource itself is likely to be used, but the paper's headline conclusions need revision.

major comments (4)
  1. [Section 4.1, Table 5] The central comparison of retain quality on combined queries is uncontrolled. The metric is defined as len(LCS(RetAns_base, RetAns_un))/len(RetAns_base) for retain-only queries and len(LCS(RetAns_base, ComAns_un))/len(RetAns_base) for combined queries, but the paper never reports the corresponding base-model baseline len(LCS(RetAns_base, ComAns_base))/len(RetAns_base). Combined queries are roughly twice as long and contain two separate questions, so the base model itself may produce longer or truncated outputs that yield lower LCS against its own retain-only answer. The paper's own perplexity numbers for WMDP show the base model behaves differently on combined queries (2.1232 vs 1.3758 for retain-only), so the query format is not neutral. Without this baseline, the observed 2-3x drop on WHP and RWKU may be partly an artifact of the metric or of output truncation, not evidence of an unlearning-specific failure. This is load-bearing because the abstract's claim of significant drops rests on this table. The authors should report the base-model combined-query score, control generation length/truncation, or supplement with a judgment-based evaluation as in Appendix C.
  2. [Sections 4.1 and 4.3, Tables 5 and 8] The paper repeatedly uses the word 'significantly' (e.g., 'significantly lower' retain quality, 'significantly worse' performance) without any error bars, multiple seeds, or statistical tests. Tables 5 and 8 report single runs per method and dataset. This is a particular problem for the comparative claim that 'simple approaches perform better on average than more recent methods,' because method rankings can flip with different hyperparameter choices or random seeds. The authors should provide standard deviations across seeds and, where appropriate, paired significance tests for the primary quantitative comparisons.
  3. [Section 5, Limitations and Section 4.1] The limitations section concedes that the combined forget-retain queries are 'simple concatenations' and that more realistic variants are left to future work. Since the main quantitative evidence for the paper's central claim comes from these concatenated queries, the representativeness of the stress test is asserted rather than demonstrated. The crossover questions in Appendix C are a stronger test, but they are only constructed for WHP and are not integrated into the main quantitative results. The authors should either extend the crossover-style queries to the other datasets or add an explicit base-model baseline to the concatenation experiment so that the simple-concatenation results can be interpreted as an upper bound on format-induced degradation.
  4. [Sections 4.1-4.3] The main text provides no hyperparameter settings (learning rates, number of epochs, KL regularization weights) for GA, GA+KL, NPO, NPO+KL, SCRUB, or RMU, nor generation parameters such as maximum output length and temperature for the generations used in Tables 5, 7, and 8. Since the paper is a benchmark paper whose findings depend on these method configurations, these details must appear in the main text or a clearly referenced appendix; pointing to code alone is not sufficient for reproducibility in a scientific record.
minor comments (6)
  1. [Introduction] There is a typo in the second paragraph: 'undesriable data' should be 'undesirable data'.
  2. [Table 5] The numeric formatting in the table is inconsistent, with stray spaces in entries such as '1 .5342' and '1.4915 1 .5342 1 .5397'; these should be cleaned up for readability.
  3. [Table 8 caption] The caption spells the metric as 'Rogue-L recall'; this should be 'ROUGE-L recall' for consistency with the rest of the paper.
  4. [Section 4.2] The text says 'We show the results in Table 10 for WMDP' but the corresponding in-main-text table is labeled Table 7; the cross-reference appears to be off by one relative to the appendix tables.
  5. [Section 3.2] The 'C-Value method' for keyword extraction is mentioned without a citation; a reference should be provided.
  6. [Table 5] The column headers for retain quality ('Retain vs Retain' and 'Retain vs Combined') are ambiguous without an explicit statement that the numerator in both cases uses the base-model answer to the retain-only question; please clarify in the caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: BLUR's empirical benchmark results are self-contained external evaluations, with no fitted parameter, derived quantity, or self-cited theorem that reduces to its own input.

full rationale

BLUR is an empirical benchmark paper rather than a derivation. Its central claims are comparative measurements of ROUGE-L recall and perplexity for five unlearning methods on original queries, combined forget/retain queries, and relearning datasets. These results come from running external base models and unlearning algorithms (Table 5, Table 8, Appendix A), so no predicted quantity is defined in terms of the fitted outcome. The paper does cite the authors' prior work, e.g., 'Following Thaker et al. [9], we choose QA pairs from the Forget10 Split' and 'GPT generated relearn text from Hu et al. [7]' (Sections 3.2 and 3.3), but these citations motivate dataset construction and methodology rather than prove the empirical findings; the reported drops in retain quality are independently produced by the paper's own evaluations. The limitation that the combined queries 'are simple concatenations of forget and retain questions' (Section 5) is an acknowledged validity constraint about representativeness, not a circularity: the measured effect is still an external experimental observation, not an artifact of an equation that equates input and output. No uniqueness theorem is imported from the authors, no ansatz is smuggled in via citation, and no known result is merely renamed. Accordingly, the circularity score is 0.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The benchmark's findings rest on methodological assumptions about how to measure unlearning, rather than on fitted physical parameters. The main unsupported premises are that behavioral prompts reveal unlearning, that concatenated and keyword-swapped queries capture realistic overlap, that relearning on benign text is a meaningful threat, and that Rouge-L and perplexity are adequate metrics. No new physical entities are introduced. The only hand-chosen quantities are the experimental hyperparameters and dataset construction choices, which are not fully specified in the main text.

free parameters (2)
  • Unlearning method hyperparameters (learning rate, epochs, KL weight, etc.) for GA, GA+KL, NPO, NPO+KL, SCRUB, RMU = Not reported in main text; deferred to appendix/code
    These hand-chosen hyperparameters directly determine forget/retain trade-offs and the reported drops on BLUR. Without multiple seeds or a hyperparameter search, results may not generalize beyond the chosen settings.
  • Relearn finetuning budget and learning rate for relearning experiments = Not reported in main text
    Relearning success depends on the finetuning strength; without these values, the correlation between relevance level and recovery cannot be reproduced from the text alone.
assumptions (5)
  • domain assumption Behavioral query evaluation is a valid proxy for unlearning success.
    Section 3.1 establishes the standard evaluation setup using forget and retain queries, assuming prompting captures whether knowledge was removed.
  • ad hoc to paper Concatenating forget and retain queries and injecting unlearned keywords into MMLU choices captures realistic forget-retain overlap.
    Section 3.2 constructs BLUR's combined queries this way; the paper itself notes these are simple concatenations and calls for more complex variants, so the representativeness is assumed.
  • domain assumption Benign relearning on related but non-answering text is a meaningful threat model for deployed unlearned models.
    Section 3.3 assumes finetuning on Dhi, Dmid, and Dlow reflects realistic downstream modifications; evidence from prior work motivates this.
  • domain assumption Rouge-L recall and perplexity using the base model as reference are informative retain-quality and forget-quality metrics.
    Section 4.1 defines the metrics; their validity for longer combined outputs and without truncation control is assumed.
  • domain assumption LLM-as-judge ratings by o3-mini provide valid measurements for crossover questions.
    Appendix C uses LLM-as-Judge for the 50 crossover queries without calibration against human ratings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BLUR: A Benchmark for LLM Unlearning Robust to Forget-Retain Overlap." pith.science (2026). https://pith.science/paper/YMOCHQNC

@misc{pith2026250615699,
  author       = {Pith},
  title        = {Pith review of: BLUR: A Benchmark for LLM Unlearning Robust to Forget-Retain Overlap},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YMOCHQNC}},
  note         = {Machine review of arXiv:2506.15699}
}
abstract

Machine unlearning has the potential to improve the safety of large language models (LLMs) by removing sensitive or harmful information post hoc. A key challenge in unlearning involves balancing between forget quality (effectively unlearning undesirable information) and retain quality (maintaining good performance on other, general tasks). Unfortunately, as we show, current LLM unlearning benchmarks contain highly disparate forget and retain sets -- painting a false picture of the effectiveness of LLM unlearning methods. This can be particularly problematic because it opens the door for benign perturbations, such as relearning attacks, to easily reveal supposedly unlearned knowledge once models are deployed. To address this, we present $\texttt{BLUR}$: a benchmark for LLM unlearning that provides more realistic scenarios of forget-retain overlap. $\texttt{BLUR}$ significantly expands on existing unlearning benchmarks by providing extended evaluation tasks, combined forget/retain queries, and relearning datasets of varying degrees of difficulty. Despite the benign nature of the queries considered, we find that the performance of existing methods drops significantly when evaluated on $\texttt{BLUR}$, with simple approaches performing better on average than more recent methods. These results highlight the importance of robust evaluation and suggest several important directions of future study. Our benchmark is publicly available at: https://huggingface.co/datasets/forgelab/BLUR

Figures

Figures reproduced from arXiv: 2506.15699 by the authors.

Figure 1
Figure 1. (Left) BLUR provides two key components to measure robustness to forget/retain overlap: combined forget/retain set queries for evaluation, and a suite of relearning data of varying degrees of difficulty. (Right) Despite the benign nature of these perturbations, we find that the performance of existing methods is significantly worse when incorporating the forget/retain overlap present in BLUR via relearning or combin… view at source ↗
Figure 2
Figure 2. LLM-as-Judge evaluation for forget and retain knowledge memorization evaluated on the complex combined forget-retain query. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem

    cs.LG 2026-07 accept novelty 7.0 of 10

    SUITE defines the forget-retain boundary at semantic, syntactic and lexical levels; training on it plus JensUn++ yields near-complete forgetting with minimal retain and utility loss.

Reference graph

Works this paper leans on

31 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [1]

    Towards making systems forget with machine unlearning

    Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In IEEE symposium on Security and Privacy, 2015

    work page 2015

  2. [2]

    Making ai forget you: Data deletion in machine learning

    Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in Neural Information Processing Systems, 2019

    work page 2019

  3. [3]

    Machine unlearning

    Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In IEEE Symposium on Security and Privacy, 2021

    work page 2021

  4. [4]

    Hard to forget: Poisoning attacks on certified machine unlearning

    Neil G Marchant, Benjamin IP Rubinstein, and Scott Alfeld. Hard to forget: Poisoning attacks on certified machine unlearning. In AAAI Conference on Artificial Intelligence, 2022

    work page 2022

  5. [5]

    Reconstruction attacks on machine unlearning: Simple models are vulnerable

    Martin Bertran, Shuai Tang, Michael Kearns, Jamie Morgenstern, Aaron Roth, and Zhi- wei Steven Wu. Reconstruction attacks on machine unlearning: Simple models are vulnerable. arXiv preprint arXiv:2405.20272, 2024

    arXiv 2024

  6. [6]

    Learn what you want to unlearn: Unlearning inversion attacks against machine unlearning

    Hongsheng Hu, Shuo Wang, Tian Dong, and Minhui Xue. Learn what you want to unlearn: Unlearning inversion attacks against machine unlearning. arXiv preprint arXiv:2404.03233, 2024

    arXiv 2024

  7. [7]

    Unlearning or obfuscating? jogging the memory of unlearned llms via benign relearning

    Shengyuan Hu, Yiwei Fu, Steven Wu, and Virginia Smith. Unlearning or obfuscating? jogging the memory of unlearned llms via benign relearning. In International Conference on Learning Representations, 2025

    work page 2025

  8. [8]

    Fast yet effective machine unlearning

    Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems, 2023

    work page 2023

Show all 31 references
  1. [9]

    Position: Llm unlearning benchmarks are weak measures of progress

    Pratiksha Thaker, Shengyuan Hu, Neil Kale, Yash Maurya, Zhiwei Steven Wu, and Virginia Smith. Position: Llm unlearning benchmarks are weak measures of progress. In IEEE Conference on Secure and Trustworthy Machine Learning, 2025

    2025

  2. [10]

    Eight methods to evaluate robust unlearning in llms

    Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402.16835, 2024

    2024 arXiv

  3. [11]

    Muse: Machine unlearning six-way evaluation for language models

    Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models. arXiv preprint arXiv:2407.06460, 2024

    2024 arXiv

  4. [12]

    Rwku: Benchmarking real-world knowledge unlearning for large language models

    Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. Rwku: Benchmarking real-world knowledge unlearning for large language models. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2024

    2024

  5. [13]

    Tofu: A task of fictitious unlearning for llms

    Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. Tofu: A task of fictitious unlearning for llms. In Conference on Language Modeling, 2024

    2024

  6. [14]

    Who’s harry potter? approximate unlearning in llms

    Ronen Eldan and Mark Russinovich. Who’s harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238, 2023

    2023 arXiv

  7. [15]

    The wmdp benchmark: Measuring and reducing malicious use with unlearning

    Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. InInternational Conference on Machine Learning, 2024

    2024

  8. [16]

    An adversarial perspective on machine unlearning for ai safety.arXiv preprint arXiv:2409.18025, 2024

    Jakub Łucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tramèr, and Javier Rando. An adversarial perspective on machine unlearning for ai safety.arXiv preprint arXiv:2409.18025, 2024. 10

    2024 arXiv

  9. [17]

    Do unlearning methods remove information from language model weights? arXiv preprint arXiv:2410.08827, 2024

    Aghyad Deeb and Fabien Roger. Do unlearning methods remove information from language model weights? arXiv preprint arXiv:2410.08827, 2024

    2024 arXiv

  10. [18]

    General data protection regulation (gdpr)

    Formerly Data Protection. General data protection regulation (gdpr). Intersoft Consulting, Accessed in October, 24(1), 2018

    2018

  11. [19]

    Adaptive machine unlearning

    Varun Gupta, Christopher Jung, Seth Neel, Aaron Roth, Saeed Sharifi-Malvajerdi, and Chris Waites. Adaptive machine unlearning. Advances in Neural Information Processing Systems, 34: 16319–16330, 2021

    2021

  12. [20]

    Arcane: An efficient architecture for exact machine unlearning

    Haonan Yan, Xiaoguang Li, Ziyao Guo, Hui Li, Fenghua Li, and Xiaodong Lin. Arcane: An efficient architecture for exact machine unlearning. In IJCAI, volume 6, page 19, 2022

    2022

  13. [21]

    Recommendation unlearning

    Chong Chen, Fei Sun, Min Zhang, and Bolin Ding. Recommendation unlearning. InProceedings of the ACM Web Conference 2022, pages 2768–2777, 2022

    2022

  14. [22]

    Making recommender systems forget: Learning and unlearning for erasable recommendation

    Yuyuan Li, Chaochao Chen, Xiaolin Zheng, Junlin Liu, and Jun Wang. Making recommender systems forget: Learning and unlearning for erasable recommendation. Knowledge-Based Systems, 283:111124, 2024

    2024

  15. [23]

    Towards scalable exact machine unlearning using parameter-efficient fine-tuning

    Somnath Basu Roy Chowdhury, Krzysztof Choromanski, Arijit Sehanobish, Avinava Dubey, and Snigdha Chaturvedi. Towards scalable exact machine unlearning using parameter-efficient fine-tuning. arXiv preprint arXiv:2406.16257, 2024

    2024 arXiv

  16. [24]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021

    2021

  17. [25]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024

    2024

  18. [26]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017

    2017 arXiv

  19. [27]

    Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the embedding space

    Leo Schwinn, David Dobre, Sophie Xhonneux, Gauthier Gidel, and Stephan Günnemann. Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the embedding space. Advances in Neural Information Processing Systems, 37:9086–9116, 2024

    2024

  20. [28]

    Eternal sunshine of the spotless net: Selective forgetting in deep networks

    Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020

    2020

  21. [29]

    Negative preference optimization: From catastrophic collapse to effective unlearning

    Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. In Conference on Language Modeling, 2024

    2024

  22. [30]

    Towards un- bounded machine unlearning

    Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards un- bounded machine unlearning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  23. [31]

    the One Ring

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.