REVIEW 5 major objections 6 minor 1 cited by
Step-by-Step Reasoning Attack: Revealing 'Erased' Knowledge in Large Language Models
T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Step-by-step prompts recover 'erased' knowledge from unlearned LLMs
desk verdict The attack idea is a reasonable extension of existing jailbreaking work, but the paper's own tables contradict its abstract and the keyword metric counts denials as leakage, so the headline numbers don't hold. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is SLEEK, a black-box attack pipeline built on the idea of decomposing a target fact into reasoning steps and re-asking each step as its own question. It works by (i) prompting a support LLM to 'think step by step' about a forget-set question, (ii) extracting intermediate facts from that trace and generating a question per fact, (iii) deduplicating questions with agglomerative clustering on sentence-transformer embeddings, (iv) human validation and iterative expansion of the question set, and (v) probing the unlearned model and classifying each response as direct, indirect, implied, or irrelevant. The classification scheme plus keyword-presence and GPT scoring is what converts a response into a measured unlearning failure or a measured suppression of retained knowledge.
What would settle it
Have human raters judge whether each keyword-positive response in SLEEK actually entails the target fact, or check a control condition where the unlearned model is asked about a fact whose keywords appear in a wrong answer. If keyword-positive responses are mostly non-entailing or fabricated, the reported 62.5% retrieval rate would be an artifact of the scoring rule rather than a measure of knowledge retention.
Extended reading notes
Core claim
SLEEK claims to demonstrate that 'erased' knowledge remains recoverable through step-by-step reasoning. The attack generates reasoning traces from a support LLM, decomposes them into individual knowledge points, and creates one question per point; the question set is then deduplicated by embedding clustering, filtered by human annotators, expanded iteratively, and used to probe the unlearned model. Responses are categorized as direct, indirect, implied, or irrelevant, with leakage detected through keyword presence and a GPT-based inference score for implied answers. The headline results are that 62.5% of adversarial prompts retrieved forgotten Harry Potter facts from WHP-unlearned Llama, 50% exposed unfair suppression of retained knowledge, and RMU failed every probe, while UNSTAR resisted best yet still leaked in some categories.
Load-bearing premise
The attack's success metric assumes that when the unlearned model's output contains a keyword such as 'Hogwarts' or 'Gryffindor', the model genuinely knows the forgotten fact, rather than merely echoing vocabulary without the underlying knowledge.
Editorial extensions
If this is right
- Unlearning evaluations that only re-ask the original forget-set questions will miss most residual knowledge, because SLEEK recovers it through rephrased and inferred queries.
- Deployers should treat indirect mentions and logical implications as leakage events, not as acceptable partial forgetting.
- Refusal-based and hallucination-based unlearning methods (OPT-OUT and WHP in this study) can appear safe on direct prompts while still leaking through background knowledge.
- Retain-set probing matters as much as forget-set probing: the attack found 50% suppression of retained knowledge in WHP-unlearned Llama, meaning unlearning can silently damage model utility.
- An adversary with only API access plus a support LLM can run the full attack, so black-box exposure alone is not a safeguard.
Reading between the lines
- A natural extension would turn SLEEK into a standard red-team harness for unlearned models before deployment, since it needs no weights or gradients.
- The keyword-presence metric could overstate leakage when the unlearned model generates keyword-rich but factually wrong text; human or entailment-based validation of responses would give a truer retention rate.
- The same reasoning-decomposition trick might transfer to other input modalities, such as retrieving erased facts through translation, summarization, or multilingual paraphrases, though the paper does not test those.
- If the mechanism is that unlearning never removes latent traces, defense may require corrupting or randomizing internal representations rather than filtering outputs; that is an inference, not a claim the paper makes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces SLEEK, a black-box attack that uses step-by-step reasoning to probe LLMs after machine unlearning. The attack generates adversarial questions from a support LLM's chain-of-thought decomposition of forget-set and retain-set facts, clusters and human-filters them, and then evaluates the unlearned model's responses using keyword presence and a GPT-based score. The authors report high success rates, e.g., 62.5% retrieval of Harry Potter facts from WHP-unlearned Llama, and conclude that current unlearning methods fail to reliably erase knowledge and may unfairly suppress retained knowledge.
Significance. If the central claim were established, the work would be relevant to the LLM unlearning community: a systematic black-box attack that recovers supposedly erased knowledge across multiple unlearning methods and two LLMs would sharpen evaluation standards and highlight the limits of current suppression-based unlearning. The paper is clearly written, the attack pipeline is described in sufficient detail to be reproduced, and the authors make a genuine effort to include human-in-the-loop validation of generated questions, iterative expansion, and comparisons across four unlearning methods. However, the evaluation metric is not a valid measure of knowledge retention, the headline numbers are internally inconsistent, and the authors' own Limitations section (page 17) concedes that the analysis relies on specific domains and that human oversight introduces subjectivity. As presented, the quantitative evidence does not support the paper's central claims.
major comments (5)
- [§5.2, Algorithm 2] The Keyword Presence Score in Algorithm 2 (Step 3) labels a response as a successful retrieval whenever a keyword from a human-curated list (Appendix A.1) appears. The WHP example in §5.2 is a direct counterexample: the response "I don't have information on a school called 'Hogwarts'..." contains both "Hogwarts" and "Harry Potter" and would be scored as a Direct hit, yet the response explicitly denies the fact and even confabulates the author as "Michael Coleman". The metric therefore measures surface-level lexical overlap, not knowledge retention. Since Tables 1–3 and the abstract's headline figures are computed with this metric, the quantitative foundation of the central claim is not established. The human-in-the-loop validation in §4.1 filters generated questions, not model responses, so it does not correct this flaw.
- [Abstract, Table 3] The abstract states that "62.5% successfully retrieved forgotten Harry Potter facts from WHP-unlearned Llama", but Table 3 reports 62.5% in the WHP Llama retain-set Direct row, which corresponds to unfair suppression of retained knowledge, not retrieval of forgotten facts. The forget-set Direct rate for WHP Llama is 27.5%. The headline number is therefore misattributed to the wrong quantity, and the actual forget-set retrieval rates are materially lower. This internal inconsistency must be resolved before any of the paper's quantitative conclusions can be interpreted.
- [§5.2 (RMU)] The claim that "RMU consistently exhibits a 100% failure rate across all question types" is not supported by any table or per-question data in the paper. The single illustrative response about Platform 9 3/4 is ungrammatical repetition ("a heart, a bird, a fish...") that contains none of the Harry Potter keywords from Appendix A.1. Under the paper's own keyword-presence metric, this response would not be counted as a success. Either the 100% figure is based on a different, undescribed scoring rule, or it is unsupported. This claim is central to the paper's ranking of unlearning methods and needs a data table or a precise definition.
- [Table 2, §5.2 (UNSTAR)] The UNSTAR discussion in §5.2 attributes results to both Llama and Mistral, including "0% failure for Llama and Mistral" in the Indirect category and "a maximum of 17.5% failure in the Forget Set (Llama, Implied prompts)". However, Table 2 only reports results for Mistral; the 17.5% value appears in the Mistral Implied cell, not in any Llama row. The conclusion that UNSTAR is "the most robust technique" is therefore not backed by the displayed data for one of the two LLMs the text claims to evaluate.
- [§4.1, Appendix A.1] The attack construction has a circularity problem: probing questions are generated by decomposing the target answer with a support LLM that has full knowledge of the forget set (e.g., "Where did Harry Potter study?" is decomposed into steps that explicitly name Hogwarts), and the keyword list in Appendix A.1 (Step 5) is derived from the same entities and relationships. Success is then defined as the presence of those very keywords in the unlearned model's output. This procedure partially guarantees leakage by construction: any model that talks about Hogwarts, Voldemort, or Gryffindor—even to deny their existence or to hallucinate—will be scored as a success. To support the claim that step-by-step reasoning "recalls erased content", the evaluation must measure whether the response actually entails the target fact (e.g., human-annotated entailment labels or a follow-up question whose correct answer is not one of the seed keywords).
minor comments (6)
- [§1 (Introduction)] The Introduction says "comprehensive evaluations of three state-of-the-art unlearning techniques (WHP, RMU, OPT-OUT, UNSTAR)", but the parenthetical lists four methods; the count should be four.
- [§3 (Preliminaries, Eq. (1))] The retention condition in Eq. (1) is written as "M(q, ϕ′) ̸= a" for retain-set questions, which is identical in form to the forgetting condition and contradicts the prose that the model "should continue to provide correct answers". The inequality should be an equality.
- [§5.2 and Figure 1 caption] There are several typographical inconsistencies in method names: "OPT-O UT" appears in §5.2, "S LEEK" appears in the Figure 1 caption and in Appendix A.2, and "LLaMa" and "Llama" are used interchangeably. These should be standardized.
- [Appendix A.1, Step 4] Step 4 says "Cluster based on computed embeddings of responses and filter", but Algorithm 1 (Step 2) clusters the generated questions, not the responses; the description should be corrected to match the algorithm.
- [Appendix A.2] The GPT score prompt does not specify the GPT model version, decoding parameters, or any validation against human judgments (e.g., inter-annotator agreement). Since the GPT score is the only metric for Implied questions, this missing detail limits reproducibility.
- [Tables 1–3] All tables report single-point success rates without confidence intervals, standard deviations, or significance tests. Given that question generation and model decoding are stochastic, the differences between methods and conditions may not be reliable without variance estimates.
Circularity Check
The 62.5% retrieval figure is partially an artifact of a keyword metric built from the same target-fact terms the attack claims to recover, but the qualitative finding is supported by independent examples.
-
self definitional
[Section 4.1; Algorithm 2 Step 3; Appendix A.1 Steps 3 and 5]
"Questions are categorized based on the presence of specific keywords in the responses, which the adversary uses to detect traces of the forgotten knowledge. ... Step 5: Human in the Loop Validation. "Hogwarts", "Voldemort", "Quidditch", "Philosopher's Stone", "Gryffindor", "Ravenclaw", "Dumbledore", "Snape", "McGonagall", "Hagrid", "Filch""
The attack's success metric for Direct and Indirect questions is the Keyword Presence Score, which counts how often keywords from a human-curated list appear in the unlearned model's response. That list (Appendix A.1 Step 5) is composed of the very entities that define the forgotten facts, e.g., "Hogwarts" for the target fact "Harry Potter studied at Hogwarts." The probing questions are also generated by decomposing the target fact and creating a question for each knowledge point (Appendix A.1 Steps 2-3), so the answer terms are the same strings used as detection keywords.
full rationale
This paper makes no formal derivation claim; it is an empirical attack evaluation. The load-bearing quantitative step is the Keyword Presence Score, which defines attack success as the occurrence of target-fact keywords in model outputs. Because the keyword list is extracted from the same knowledge used to generate the probing questions, this metric is partially self-definitional: it can count denials and hallucinations as leakage. The WHP example in Section 5.2 illustrates exactly this failure mode. Nevertheless, the central qualitative claim - that step-by-step reasoning can expose incomplete unlearning - does not rest solely on this metric. The paper provides concrete response transcripts showing genuine leakage, such as OPT-OUT listing "Harry Potter, Ron Weasley, and Hermione Granger" when asked about Gryffindor students, and RMU producing incoherent text that indicates failed erasure. UNSTAR, a method by the same authors, is evaluated as a baseline, but its reported robustness is a tabulated result rather than an imported self-citation used to justify the attack's premise. There is no uniqueness theorem, no fitted parameter renamed as a prediction, and no ansatz smuggled in via citation. The circularity is confined to the numerical success metric, which inflates the reported rates but does not fully determine the overall conclusion. Hence a moderate score of 3 is appropriate.
Assumptions & free parameters
free parameters (3)
- Agglomerative clustering distance threshold multiplier =
0.15
- Keyword list K =
Human-curated (e.g., 'Hogwarts', 'Gryffindor')
- GPT score threshold =
0 or 1, no calibration
assumptions (3)
- domain assumption Keyword presence in a model response implies residual knowledge of the forgotten fact.
- domain assumption GPT-based scoring of implied questions reliably measures logical inference of the target fact.
- domain assumption A support LLM with all knowledge can generate questions that fairly test the unlearned model.
Cite this review
Pith. "Pith review of Step-by-Step Reasoning Attack: Revealing 'Erased' Knowledge in Large Language Models." pith.science (2026). https://pith.science/paper/OGWT6YNV
@misc{pith2026250617279,
author = {Pith},
title = {Pith review of: Step-by-Step Reasoning Attack: Revealing 'Erased' Knowledge in Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/OGWT6YNV}},
note = {Machine review of arXiv:2506.17279}
}
read the original abstract
Knowledge erasure in large language models (LLMs) is important for ensuring compliance with data and AI regulations, safeguarding user privacy, mitigating bias, and misinformation. Existing unlearning methods aim to make the process of knowledge erasure more efficient and effective by removing specific knowledge while preserving overall model performance, especially for retained information. However, it has been observed that the unlearning techniques tend to suppress and leave the knowledge beneath the surface, thus making it retrievable with the right prompts. In this work, we demonstrate that \textit{step-by-step reasoning} can serve as a backdoor to recover this hidden information. We introduce a step-by-step reasoning-based black-box attack, Sleek, that systematically exposes unlearning failures. We employ a structured attack framework with three core components: (1) an adversarial prompt generation strategy leveraging step-by-step reasoning built from LLM-generated queries, (2) an attack mechanism that successfully recalls erased content, and exposes unfair suppression of knowledge intended for retention and (3) a categorization of prompts as direct, indirect, and implied, to identify which query types most effectively exploit unlearning weaknesses. Through extensive evaluations on four state-of-the-art unlearning techniques and two widely used LLMs, we show that existing approaches fail to ensure reliable knowledge removal. Of the generated adversarial prompts, 62.5% successfully retrieved forgotten Harry Potter facts from WHP-unlearned Llama, while 50% exposed unfair suppression of retained knowledge. Our work highlights the persistent risks of information leakage, emphasizing the need for more robust unlearning strategies for erasure.
Figures
Forward citations
Cited by 1 Pith paper
-
A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
A structured literature survey concluding that reasoning capabilities do not automatically make LLMs more trustworthy and can introduce new vulnerabilities in safety, robustness, and privacy.
Reference graph
Works this paper leans on
-
[1]
Aghyad Deeb and Fabien Roger. Do unlearning methods remove information from language model weights? arXiv preprint arXiv:2410.08827,
-
[3]
Who’s harry potter? approximate unlearning for llms
Ronen Eldan and Mark Russinovich. Who’s harry potter? approximate unlearning for llms. Phillip Guo, Aaquib Syed, Abhay Sheshadri, Aidan Ewart, and Gintare Karolina Dziugaite. Mechanistic unlearning: Robust knowledge unlearning and editing via mechanistic localization. arXiv preprint arXiv:2410.12949,
-
[4]
Intrinsic evaluation of unlearning using parametric knowledge traces
Yihuai Hong, Lei Yu, Haiqin Yang, Shauli Ravfogel, and Mor Geva. Intrinsic evaluation of unlearning using parametric knowledge traces. arXiv preprint arXiv:2406.11614,
-
[5]
Jogging the memory of unlearned llms through targeted relearning attacks
Shengyuan Hu, Yiwei Fu, Zhiwei Steven Wu, and Virginia Smith. Jogging the memory of unlearned llms through targeted relearning attacks. arXiv preprint arXiv:2406.13356,
-
[6]
On effects of steering latent representation for large language model unlearning
Dang Huu-Tien, Trung-Tin Pham, Hoang Thanh-Tung, and Naoya Inoue. On effects of steering latent representation for large language model unlearning. arXiv preprint arXiv:2408.06223,
-
[7]
Knowledge unlearning for mitigating privacy risks in language models
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint arXiv:2210.01504,
-
[8]
10 Preprint. Under review. Jinghan Jia, Jiancheng Liu, Yihua Zhang, Parikshit Ram, Nathalie Baracaldo, and Sijia Liu. Wagle: Strategic weight attribution for effective and modular unlearning in large language models. arXiv preprint arXiv:2410.17509,
-
[11]
An adversarial perspective on machine unlearning for ai safety
Jakub Łucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tram `er, and Javier Rando. An adversarial perspective on machine unlearning for ai safety. arXiv preprint arXiv:2409.18025,
Show all 25 references
-
[12]
Eight methods to evaluate robust unlearning in llms
Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402.16835,
-
[14]
Can sensitive information be deleted from llms? objectives for defending against extraction attacks
Vaidehi Patil, Peter Hase, and Mohit Bansal. Can sensitive information be deleted from llms? objectives for defending against extraction attacks. arXiv preprint arXiv:2309.17410,
-
[15]
In-context unlearning: Language models as few shot unlearners
Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.07579,
-
[16]
Latent adversarial training improves robustness to persistent harmful behaviors in llms
Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, et al. Latent adversarial training improves robustness to persistent harmful behaviors in llms. arXiv preprint arXiv:2407.15549,
-
[17]
Unstar: Unlearning with self-taught anti-sample reasoning for llms
Yash Sinha, Murari Mandal, and Mohan Kankanhalli. Unstar: Unlearning with self-taught anti-sample reasoning for llms. arXiv preprint arXiv:2410.17050,
-
[18]
Under review
11 Preprint. Under review. Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, et al. Tamper-resistant safeguards for open-weight llms. arXiv preprint arXiv:2408.00761,
-
[19]
Guardrail baselines for unlearning in llms
Pratiksha Thaker, Yash Maurya, Shengyuan Hu, Zhiwei Steven Wu, and Virginia Smith. Guardrail baselines for unlearning in llms. arXiv preprint arXiv:2403.03329,
-
[20]
Rkld: Reverse kl-divergence- based knowledge distillation for unlearning personal information in large language models
Bichen Wang, Yuzhe Zi, Yixin Sun, Yanyan Zhao, and Bing Qin. Rkld: Reverse kl-divergence- based knowledge distillation for unlearning personal information in large language models. arXiv preprint arXiv:2406.01983,
-
[22]
Machine unlearning of pre-trained large language models
Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. Machine unlearning of pre-trained large language models. arXiv preprint arXiv:2402.15159,
-
[23]
A closer look at machine unlearning for large language models
Xiaojian Yuan, Tianyu Pang, Chao Du, Kejiang Chen, Weiming Zhang, and Min Lin. A closer look at machine unlearning for large language models. arXiv preprint arXiv:2410.08109,
-
[24]
Negative preference optimization: From catastrophic collapse to effective unlearning
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868,
-
[25]
Harry Potter and the Philosopher 's Stone
as the final output. A.3 Generated Questions for Attack and their Types Direct. What is the significance of Platform 9 3/4 for Harry? What does Harry learn about from Rubeus Hagrid? What is the name of the school where Harry spent six years? Who places Harry into a house durin...
1991
-
[2020]
Depn: Detecting and editing privacy neurons in pretrained language models
Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. Depn: Detecting and editing privacy neurons in pretrained language models. arXiv preprint arXiv:2310.20138,
-
[2022]
Large language model unlearning via embedding-corrupted prompts
Chris Liu, Yaxuan Wang, Jeffrey Flanigan, and Yang Liu. Large language model unlearning via embedding-corrupted prompts. Advances in Neural Information Processing Systems, 37: 118198–118266, 2024a. Yujian Liu, Yang Zhang, Tommi Jaakkola, and Shiyu Chang. Revisiting who’s harry...
2024 arXiv
-
[2023]
The wmdp benchmark: Measuring and reducing malicious use with unlearning
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218,
-
[2024]
Undial: Self-distillation with adjusted logits for robust unlearning in large language models.arXiv preprint arXiv:2402.10052,
Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vuli´c. Undial: Self-distillation with adjusted logits for robust unlearning in large language models.arXiv preprint arXiv:2402.10052,
-
[2025]
Tofu: A task of fictitious unlearning for llms
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. Tofu: A task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06121,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.