Pith. sign in

REVIEW 3 major objections 4 minor 44 references

Towards Robust Evaluation of Unlearning in LLMs via Data Transformations

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A model that passes an unlearning test in Q&A can still retrieve the same fact in cloze, multiple-choice, or analogy formats, so unlearning benchmarks must evaluate multiple formats.

desk verdict Solid empirical resource with one clean Cloze result; the MCQA-family numbers need calibration before the broader claims are trusted. read the letter →

arxiv 2411.15477 v1 pith:RJ3O6UWS submitted 2024-11-23 cs.CL cs.AIcs.CYcs.LG

Abhinav Joshi , Shaswati Saha , Divyaksh Shukla , Sriram Vema , Harsh Jhamtani , Manas Gaur , Ashutosh Modi This is my paper · ORCID
classification cs.CLcs.AIcs.CYcs.LG
keywords machineunlearninglargelanguagemodelsdatatransformationsTOFUbenchmarkevaluationrobustnessclozetestmultiple-choicequestionansweringknowledgeretention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the standard way to measure machine unlearning in large language models is unreliable because it tests knowledge in one input format, usually question-answer. The authors transform facts from the TOFU benchmark into five additional formats—multiple-choice, cloze, analogy, odd-one-out, and comprehension—and find that an 'unlearned' model can look as if it has forgotten a fact in Q&A while still retrieving it in another format. That matters because unlearning is the mechanism behind the right to be forgotten and behind safety guarantees, and a format-specific pass creates false confidence. The paper concludes that unlearning benchmarks should evaluate across multiple formats, a claim it supports with two base models and three unlearning algorithms.

What carries the argument

The central object is a multi-format evaluation suite built by transforming each TOFU fact into five additional task formats: MCQA (choose an option; scored by success rate over next-token probabilities of option ids), cloze (fill a masked entity; scored by length-normalized sequence probability), analogy (complete a relational analogy from in-context examples; scored by success rate), odd-one-out (identify which fact comes from the other set; scored by success rate), and comprehension (answer or choose with the fact supplied in the context; scored by ROUGE or success rate). The same underlying knowledge is probed through different surface structures, so comparing the unlearned model's scores on each format against the pretrained and fine-tuned models isolates whether forgetting generalizes beyond the training format. This comparison mechanism is what carries the paper's argument: format-conditional gaps, not the default-format gap, reveal the reliability of unlearning.

What would settle it

Fine-tune a control model on the retain set only—so it has never seen the forget set—and score it on all five formats; if the control shows the same format-to-format gaps as the unlearned model, then the gaps are intrinsic to the formats and the multi-format leak verdict collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLM unlearning quality is format-dependent, not a single latent property. After applying the TOFU benchmark's standard unlearning algorithms (gradient ascent, gradient difference, and KL minimization) to a model fine-tuned on fictional-author facts, the paper measures the forget set through the original Q&A format and five transformed versions that encode the same facts. It reports that the unlearned model's score relative to the pretrained and fine-tuned baselines varies across formats: the model looks successfully unlearned under Q&A (score dropping toward the pretrained level) but remains closer to the fine-tuned level under cloze and other formats, indicating residual knowledge. The paper's conclusion is that evaluation over a single format 'may lead to unreliable improvements' and that unlearning benchmarks 'should consider evaluation over multiple formats.'

Load-bearing premise

The five transformed formats all probe the same underlying facts with comparable difficulty, so a higher post-unlearning score in Cloze or MCQA must mean residual knowledge rather than an easier or differently measuring question type.

Editorial extensions

If this is right

  • Unlearning benchmarks should score the forget set in several formats; a model that passes in the default Q&A format should not be certified as forgotten until it also matches the pretrained model on cloze, MCQA, analogy, odd-one-out, and comprehension.
  • Current unlearning methods (gradient ascent, gradient difference, KL minimization) are likely over-reporting success: their apparent forgetting in the default format is contradicted by higher residual scores in alternative formats across both Llama2-7B and Phi-1.5.
  • Format-sensitive evaluation also exposes collateral damage: in Comprehension-Q&A, the unlearned model performs below the pretrained model even on retained facts, meaning unlearning can degrade the model's ability to read and extract information from context.
  • The released five-format dataset with matching forget/retain splits gives practitioners a drop-in way to audit unlearning quality beyond the original TOFU Q&A pairs.
  • Reported single-number unlearning scores should be disaggregated by format or aggregated with a format-consistent measure, since different formats imply different conclusions about the same model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the cross-format gaps give a practical privacy-attack recipe—probing a model with cloze or MCQA prompts can expose residual knowledge that Q&A checklists miss, so unlearning audits should use adversarial format searches.
  • Editorial inference: because pretrained-model scores vary by more than an order of magnitude across formats (cloze sequence probability near 0.003 versus Q&A ROUGE near 0.40), some apparent leak signal may be format sensitivity rather than retained fact; a metric normalizing each format by its never-learned baseline would separate the two.
  • Editorial inference: applying the same multi-format logic to other unlearning targets (personally identifiable information, copyrighted text, hazardous knowledge) would likely show the same format-dependence, and unlearning methods could be made more dependable by training them to match the pretrained model on every format, not just the training format.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes an evaluation scheme for LLM unlearning that goes beyond the single-format (Q&A) evaluation used in the TOFU benchmark. The authors build five transformed versions of TOFU facts—MCQA, Cloze, Analogy, Odd-one-out, and Comprehension—and evaluate two open-weight models (Llama2-7b and Phi-1.5) after applying three unlearning algorithms (gradient ascent, gradient difference, and KL minimization). Their main empirical finding is that the apparent success of unlearning depends strongly on the evaluation format: for example, KL minimization reduces Q&A forget ROUGE from 0.93 to 0.51 at step 30, while the Cloze forget sequence probability stays at 0.0187, close to the fine-tuned value of 0.0181 and far above the pretrained value of 0.0032. The paper concludes that evaluation over a single format can be unreliable and that unlearning benchmarks should consider multiple formats. The authors release code and data.

Significance. If the central empirical claim is sustained, the paper provides a useful caution for the machine unlearning community: benchmark scores are format-dependent, and a method that looks successful in the default Q&A format may leave information retrievable in other formats. The main contribution is a new evaluation resource—five transformed formats over TOFU, with manual verification of the transformations and standard evaluation metrics—which the authors make publicly available. The paper also evaluates two base models and three unlearning methods, and includes qualitative examples that illustrate format-dependent behavior. However, the stronger interpretation that non-default format scores reveal residual knowledge of forgotten facts is not fully established, because the paper lacks the Target-LLM (retain-only) calibration that would separate true leakage from format bias or retain-set similarity. The resource is still valuable as a benchmark extension, and the recommendation to evaluate over multiple formats is reasonable even if the current evidence is not conclusive about the underlying cause of the discrepancies.

major comments (3)
  1. [Section 4.2, Figure 2, Table 3] The paper defines the ideal unlearned model as behaving like the pretrained model, but in the TOFU setup the correct counterfactual is the Target-LLM introduced in Figure 1, i.e., a model fine-tuned only on the retain set. The Target-LLM is never evaluated. Consequently, the central interpretation that a high Cloze or MCQA score after unlearning indicates residual knowledge of forget facts is not established. In Table 3, the KL-minimized Llama2 model has a Cloze forget sequence probability of 0.0187 at step 30 versus 0.0032 for the pretrained model, but this gap could be explained by the model's retained knowledge of the other 180 authors, by general linguistic priors in the Cloze prompt, or by the specific masking scheme. Please add Target-LLM scores for all formats and both base models, and base the 'recall of forgotten information' interpretation on comparisons to that baseline.
  2. [Section 4.2, Table 3] The MCQA results are contaminated by strong format bias. The pretrained Llama2-7b, which has never seen TOFU, achieves 0.59 on the MCQA-4 forget items (chance 0.25), 0.72 on MCQA-2, and 0.91 on Comprehension-MCQA-4, the latter sometimes exceeding the fine-tuned model's 0.71. This shows that next-token selection among option IDs is driven by option ordering, lexical overlap, and prompt structure, not only by factual knowledge. Therefore, the difference between unlearned and fine-tuned MCQA success rates cannot be interpreted as a clean measure of fact recall. Please report chance-calibrated metrics, per-option-position accuracy, or option-randomized variants, and interpret the MCQA columns with this bias in mind.
  3. [Section 4.2, Tables 3-4, Table 1] All experimental numbers come from a single run of fine-tuning and unlearning, with no confidence intervals or multiple seeds. This is particularly problematic for the Odd-one-out retain split, which contains only 13 samples in the Forget01 setting and 66 in the Forget05 setting (Table 1). As a result, differences such as the Cloze forget value 0.0187 for KL minimization versus 0.0049 for gradient difference at step 30 in Table 3 could be within noise. Please provide error bars via multiple seeds or bootstrap resampling, and clearly state the limited sample size for Odd-one-out in the main text.
minor comments (4)
  1. [Section 3] The notation in the problem setup is inconsistent: 'Dtrain = Dretain ∪ Df orget' contains a stray space in 'Df orget', and the symbol '∼' for the transformation 'fi(Dp)' is nonstandard and should be replaced with a functional definition of the data-generation process.
  2. [Abstract and Section 5] The number of new formats is inconsistent. The abstract says '5 new formats', Section 3 says three formats (MCQA, Cloze, Analogy) plus two additional ones (Odd-one-out and Comprehension), while the Discussion says '5 new variants ... including Q&A, MCQA, Cloze, Analogy, Comprehension, and Odd-One-Out', which lists six items including the default Q&A. Please clarify that Q&A is the original TOFU format and the five new formats are the transformations.
  3. [Appendix B] The Cloze sequence-probability formula is typeset incorrectly: 'P (x1, x2, . . . , xn) = n vuut Qn i=1 P (xi)' should be written as the n-th root of the product of token probabilities. As printed, the equation is difficult to read and may confuse readers about the normalization.
  4. [Figure 1 caption] The caption says 'The spider plot shows a performance comparison', but Figure 1 contains the pipeline diagram and a spider plot embedded in the same figure. Since different formats use different metrics and the Cloze axis is 10x scaled, a separate main-text figure with labeled axes would be clearer than relying on the spider plot in the pipeline diagram.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical evaluation study whose claim is supported by external model comparisons, not by construction or self-citation.

full rationale

This is an empirical benchmarking paper, not a derivation. The central claim is that unlearning quality in LLMs depends on input format, supported by measuring pretrained, finetuned, and unlearned models on transformed versions of the TOFU dataset. No parameter is fitted to a subset of data and then renamed as a prediction; no uniqueness theorem or load-bearing result is imported from the authors' own prior work; and the authors do not cite themselves as the authority for any premise. The transformed datasets are constructed from the same TOFU facts, but that is the experimental manipulation being studied, not a circular derivation: the conclusion that single-format evaluation is unreliable is inferred from observed performance gaps across formats and from comparisons with external baselines (Pretrained-LLM, Finetuned-LLM, Target-LLM). Concerns such as uncalibrated format sensitivity, option-order bias in MCQA, and small dynamic range in Cloze scores are validity threats to the empirical inference, but they do not make the argument circular, because no step assumes the conclusion it purports to establish. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no free parameters and no invented entities. It relies on the TOFU unlearning setup as a domain assumption and, more critically, on the assumption that the five constructed formats are equally sensitive probes of the same facts, which is not calibrated in the paper.

assumptions (3)
  • domain assumption The TOFU unlearning setup, including finetuning on 4k Q&A pairs and the forget/retain splits, is a valid proxy for machine unlearning in LLMs.
    The paper adopts TOFU's setup without revalidation, and all conclusions are conditioned on this benchmark's definition of unlearning (Section 3).
  • domain assumption Each transformed format measures the same underlying factual knowledge with comparable sensitivity to the default Q&A format.
    The interpretation of performance gaps as retention differences relies on this assumption; if formats differ in difficulty or elicitation pattern, gaps may reflect format effects rather than residual knowledge (Section 3, Figures 2 and 3).
  • domain assumption Manual verification of GPT-generated transformations guarantees the transformed items contain the same facts and valid labels.
    Appendix A states manual inspection was performed, but no inter-annotator agreement or verification protocol is reported, so label errors could bias specific formats.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Robust Evaluation of Unlearning in LLMs via Data Transformations." pith.science (2026). https://pith.science/paper/RJ3O6UWS

@misc{pith2026241115477,
  author       = {Pith},
  title        = {Pith review of: Towards Robust Evaluation of Unlearning in LLMs via Data Transformations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RJ3O6UWS}},
  note         = {Machine review of arXiv:2411.15477}
}
read the original abstract

Large Language Models (LLMs) have shown to be a great success in a wide range of applications ranging from regular NLP-based use cases to AI agents. LLMs have been trained on a vast corpus of texts from various sources; despite the best efforts during the data pre-processing stage while training the LLMs, they may pick some undesirable information such as personally identifiable information (PII). Consequently, in recent times research in the area of Machine Unlearning (MUL) has become active, the main idea is to force LLMs to forget (unlearn) certain information (e.g., PII) without suffering from performance loss on regular tasks. In this work, we examine the robustness of the existing MUL techniques for their ability to enable leakage-proof forgetting in LLMs. In particular, we examine the effect of data transformation on forgetting, i.e., is an unlearned LLM able to recall forgotten information if there is a change in the format of the input? Our findings on the TOFU dataset highlight the necessity of using diverse data formats to quantify unlearning in LLMs more reliably.

Figures

Figures reproduced from arXiv: 2411.15477 by the authors.

Figure 1
Figure 1. The pipeline of using open-weight LLMs to train/finetune over new information (Finetuned-LLM). Later, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Performance of Llama2-7b on different proposed formats of TOFU [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance of Llama2-7b on our formats of TOFU [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Input prompt formats for the MCQA evaluation of autoregressive open-weight models (e.g., [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Input prompt formats for the Cloze test evaluation of autoregressive open-weight models (e.g., [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Input prompt formats for the MCQA-based Analogy detection evaluation of autoregressive open-weight [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Input prompt formats for the MCQA-based odd-one-out evaluation of autoregressive open-weight models [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Input prompt formats for the reading comprehension evaluation of autoregressive open-weight models [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Qualitative Examples for evaluation over different formats. Llama2 gave the correct output for the MCQA [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Qualitative Examples for evaluation over different formats. Llama2 gave the correct answer for MCQA [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Qualitative Examples for evaluation over different formats. We evaluate on the same fictitious author as [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Qualitative Examples for evaluation over different formats. Llama2 gives the correct answer for the Cloze [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Qualitative Examples for evaluation over different formats. Llama2 generated incorrect responses for all [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Qualitative Examples for evaluation over different formats. Phi1.5 generated the correct output for [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: Qualitative Examples for evaluation over different formats. Phi1.5 generated correct answers for the [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]
Figure 16
Figure 16. Figure 16: Qualitative Examples for evaluation over different formats. Phi1.5 gave the correct output for MCQA but [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]
Figure 17
Figure 17. Figure 17: Qualitative Examples for evaluation over different formats. Phi1.5 gave incorrect responses to all the [PITH_FULL_IMAGE:figures/full_fig_p018_17.png]
Figure 18
Figure 18. Figure 18: Performance of Phi-1.5 on different proposed formats of TOFU forget dataset on the base, fine-tuned, and unlearned model (with gradient-diff algorithm). Performance measures the ability of the language model to retrieve the author’s information from the forget set. In…
Figure 19
Figure 19. Figure 19: Performance of Phi-1.5 on the created formats of TOFU retain dataset on the base, fine-tuned, and unlearned model (with gradient-diff algorithm). In contrast to Fig.18, here the performance measures the ability of the language model to retrieve information from the re…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 13 canonical work pages

  1. [1]

    Clark Barrett, Brad Boyd, Elie Bursztein, Nicholas Carlini, Brad Chen, Jihye Choi, Amrita Roy Chowdhury, Mihai Christodorescu, Anupam Datta, Soheil Feizi, et al. 2023. Identifying and mitigating the security risks of generative ai. Foundations and Trends in Privacy and Security , 6(1):1--52

    work page 2023

  2. [2]

    Alexander Becker and Thomas Liebig. 2022. Evaluating Machine Unlearning via Epistemic Uncertainty . arXiv preprint arXiv:2208.10836

    arXiv 2022

  3. [3]

    BIG bench authors. 2023. https://openreview.net/forum?id=uyTL5Bvosj Beyond the imitation game: Quantifying and extrapolating the capabilities of language models . Transactions on Machine Learning Research

    2023

  4. [4]

    Alberto Blanco-Justicia, Najeeb Jebreel, Benet Manzanares, David Sánchez, Josep Domingo-Ferrer, Guillem Collell, and Kuan Eeik Tan. 2024. https://arxiv.org/abs/2404.02062 Digital forgetting in large language models: A survey of unlearning methods . Preprint, arXiv:2404.02062

    arXiv 2024

  5. [5]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  6. [6]

    Yinzhi Cao and Junfeng Yang. 2015. https://doi.org/10.1109/SP.2015.35 Towards making systems forget with machine unlearning . In 2015 IEEE Symposium on Security and Privacy, pages 463--480

    doi:10.1109/sp.2015.35 2015

  7. [7]

    Jiaao Chen and Diyi Yang. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.738 Unlearn what you want to forget: Efficient unlearning for LLM s . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12041--12052, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-main.738 2023

  8. [8]

    Min Chen, Weizhuo Gao, Gaoyang Liu, Kai Peng, and Chen Wang. 2023. Boundary unlearning: Rapid forgetting of deep networks via shifting the decision boundary. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7766--7775

    work page 2023

Show all 44 references
  1. [9]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sashank Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vi...

    2024

  2. [10]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. https://api.semanticscholar.org/CorpusID:3922816 Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge . ArXiv, abs/1803.05457

    2018 arXiv

  3. [11]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  4. [12]

    Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vulić. 2024. https://arxiv.org/abs/2402.10052 Unmemorization in large language models via self-distillation and deliberate imagination . Preprint, arXiv:2402.10052

    2024 arXiv

  5. [13]

    Aditya Golatkar, Alessandro Achille, and Stefano Soatto. 2020. Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks . In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2020

  6. [14]

    Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021 a . https://openreview.net/forum?id=dNy_RKzJacY Aligning AI With Shared Human Values . In International Conference on Learning Representations

    2021

  7. [15]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021 b . https://openreview.net/forum?id=d7KBjmI3GmQ Measuring Massive Multitask Language Understanding . In International Conference on Learning Representations

    2021

  8. [16]

    Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. 2023. An overview of Catastrophic AI Risks . arXiv preprint arXiv:2306.12001

    2023 arXiv

  9. [17]

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. https://arxiv.org/abs/2212.04089 Editing models with task arithmetic . Preprint, arXiv:2212.04089

    2023 arXiv

  10. [18]

    Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. 2023. https://openreview.net/forum?id=0jZH883i34 Model Sparsity Can Simplify Machine Unlearning . In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  11. [19]

    Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B

    Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang,...

    2024 arXiv

  12. [20]

    Yuanzhi Li, S \'e bastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023. Textbooks Are All You Need II: phi-1.5 technical report . arXiv preprint arXiv:2309.05463

    2023 arXiv

  13. [21]

    ROUGE : A Package for Automatic Evaluation of Summaries

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 " ROUGE : A Package for Automatic Evaluation of Summaries" . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  14. [22]

    Bo Liu, Qiang Liu, and Peter Stone. 2022. https://proceedings.mlr.press/v199/liu22a.html Continual learning and private unlearning . In Proceedings of The 1st Conference on Lifelong Learning Agents, volume 199 of Proceedings of Machine Learning Research, pages 243--254. PMLR

    2022

  15. [23]

    Varshney, Mohit Bansal, Sanmi Koyejo, and Yang Liu

    Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, Kush R. Varshney, Mohit Bansal, Sanmi Koyejo, and Yang Liu. 2024. https://arxiv.org/abs/2402.08787 Rethinking Machine Unlearning for Large L...

    2024 arXiv

  16. [24]

    Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. 2022. https://openreview.net/forum?id=5HaIds3ux5O QUARK : Controllable text generation with reinforced unlearning . In Advances in Neural Information Processing Systems

    2022

  17. [25]

    Lipton, and J

    Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. 2024. https://arxiv.org/abs/2401.06121 TOFU: A Task of Fictitious Unlearning for LLMs . Preprint, arXiv:2401.06121

    2024 arXiv

  18. [26]

    A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories

    Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. https://doi.org/10.18653/v1/N16-1098 "A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories" . In Proceedings of the...

    2016 doi

  19. [27]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners . OpenAI blog, 1(8):9

    2019

  20. [28]

    Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. https://doi.org/10.18653/v1/P18-2124 Know what you don ' t know: Unanswerable questions for SQ u AD . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages ...

    2018 doi

  21. [29]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://doi.org/10.18653/v1/D16-1264 SQ u AD : 100,000+ questions for machine comprehension of text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-...

    2016 doi

  22. [30]

    Joshua Robinson and David Wingate. 2023. https://openreview.net/forum?id=yKbprarjc5B Leveraging Large Language Models for Multiple Choice Question Answering . In The Eleventh International Conference on Learning Representations

    2023

  23. [31]

    Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. https://arxiv.org/abs/1610.05820 Membership Inference Attacks against Machine Learning Models . Preprint, arXiv:1610.05820

    2017 arXiv

  24. [32]

    Nianwen Si, Hao Zhang, Heyu Chang, Wenlin Zhang, Dan Qu, and Weiqiang Zhang. 2023. https://arxiv.org/abs/2311.15766 Knowledge unlearning for llms: Tasks, methods, and challenges . Preprint, arXiv:2311.15766

    2023 arXiv

  25. [33]

    Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. 2022. Unrolling SGD: Understanding factors influencing Machine Unlearning . In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pages 303--319. IEEE

    2022

  26. [34]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://api.semanticscholar.org...

    2023 arXiv

  27. [35]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. https://arxiv.org/abs/2406.01574 MMLU-Pro: A More Robust and...

    2024 arXiv

  28. [36]

    Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. 2021. Machine Unlearning of Features and Labels . arXiv preprint arXiv:2108.11577

    2021 arXiv

  29. [37]

    Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V. Le. 2024. https://arxiv.org/abs/2403.18802 Long-form Factuality in Large Language Models . Preprint, arXiv:2403.18802

    2024 arXiv

  30. [38]

    ANALOGICAL - A Novel Benchmark for Long Text Analogy Evaluation in Large Language Models

    Thilini Wijesiriwardene, Ruwan Wickramarachchi, Bimal Gajera, Shreeyash Gowaikar, Chandan Gupta, Aman Chadha, Aishwarya Naresh Reganti, Amit Sheth, and Amitava Das. 2023. https://doi.org/10.18653/v1/2023.findings-acl.218 " ANALOGICAL - A Novel Benchmark for Long Text Analogy E...

    2023 doi

  31. [39]

    On the Relationship between Sentence Analogy Identification and Sentence Structure Encoding in Large Language Models

    Thilini Wijesiriwardene, Ruwan Wickramarachchi, Aishwarya Naresh Reganti, Vinija Jain, Aman Chadha, Amit Sheth, and Amitava Das. 2024. https://aclanthology.org/2024.findings-eacl.31 "On the Relationship between Sentence Analogy Identification and Sentence Structure Encoding in...

    2024

  32. [40]

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, W...

    2023 arXiv

  33. [41]

    Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2024. https://arxiv.org/abs/2310.10683 Large Language Model Unlearning . Preprint, arXiv:2310.10683

    2024 arXiv

  34. [42]

    Unlearning Bias in Language Models by Partitioning Gradients

    Charles Yu, Sullam Jeoung, Anish Kasi, Pengfei Yu, and Heng Ji. 2023. https://doi.org/10.18653/v1/2023.findings-acl.375 "Unlearning Bias in Language Models by Partitioning Gradients" . In Findings of the Association for Computational Linguistics: ACL 2023, pages 6032--6048, To...

    2023 doi

  35. [43]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  36. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.