REVIEW 2 major objections 6 minor 24 references
Prompt Attacks Reveal Superficial Knowledge Removal in Unlearning Methods
T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Prepending a fixed Hindi filler passage restores 57.3% accuracy on WMDP-bio questions that the ELM unlearning method was supposed to have removed, and logit and probe analyses show the knowledge was suppressed rather than erased.
desk verdict A competent empirical audit with a plausible core finding, but the headline Hindi-filler recovery is a post-hoc selected template that needs held-out validation before being treated as a robust measurement. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing tool is a three-channel evaluation suite: output-token accuracy on the original and rephrased prompts, logit accuracy (argmax over the A/B/C/D tokens), and linear probes trained on residual-stream activations from each layer. This suite lets the paper separate three failure modes: refusing to answer, genuinely losing knowledge, and suppressing knowledge behind an output-level gate. The specific attack that exposes ELM is a fixed Hindi filler passage prepended to the unchanged English question; because the filler is semantically unrelated, its effect is attributed to disrupting the suppression pattern rather than to added information.
What would settle it
Run a pre-registered version of the experiment: fix the Hindi filler template and several control filler templates (English, Latin, another low-resource language) before looking at any results, evaluate on a held-out half of WMDP-bio never used to choose the templates, and require the Hindi condition to beat both controls and the original-format condition by a margin that survives a multiple-comparison correction. If the recovery drops to near the roughly 30% unlearned baseline for new filler texts or held-out questions, the claim that ELM's knowledge is retrievably stored would be weakened. A second independent check: train linear probes on one subset and test on another; if probe accuracy stays high for ELM with Hindi filler but not for RMU, the representation-level interpretation is supported.
Extended reading notes
Core claim
The central claim is that unlearning effectiveness is method-specific and that at least one widely used method, ELM (Erasure of Language Memory), does not truly unlearn WMDP-bio knowledge: it suppresses it. The evidence is that the same model that scores near chance when asked in the original format answers correctly when the original question is prefixed with Hindi filler content (57.3% output accuracy on Zephyr-7B, with smaller recoveries on Mistral-7B and Llama-3 variants), and that the recovered accuracy is mirrored in the logits over the four multiple-choice tokens and in linear probes of the residual stream. Because output accuracy and logit accuracy are highly correlated across methods, the paper argues the model is not hiding knowledge through refusal or formatting games. The authors' interpretation is that ELM suppresses knowledge at the output level while the information remains encoded internally, whereas RMU and TAR show more robust removal, at the cost of formatting failures or degraded general capability.
Load-bearing premise
The headline result depends on the Hindi filler template being a genuine attack rather than the single best-performing variant selected from many conditions measured on the same test questions, because the paper reports no pre-specified attack template and no correction for multiple comparisons.
Editorial extensions
If this is right
- ELM-unlearned models retain retrievable WMDP-bio knowledge; the paper's released evaluation framework lets other researchers test further unlearning methods against the same prompting techniques.
- Output-token and logit accuracies move together across most methods, so unlearning evaluations can rely on either signal without worrying that models are concealing knowledge by changing answer format.
- RMU and TAR resist the tested prompt attacks, but RMU frequently fails to emit a valid option token and TAR, GradDiff, RepNoise, and RMU+LAT show reduced tinyMMLU accuracy, indicating robustness may partly be collateral capability loss.
- Standard unlearning evaluations that reuse the original question format are insufficient; robustness should be measured across prompt variations and checked against general-capability retention.
- The Hindi-filler recovery holds across three model families (Zephyr, Mistral, Llama 3), so the effect is not an artifact of a single architecture.
Reading between the lines
- The 57.3% figure is best-of-many: the Hindi filler condition was selected after the fact because it stood out among a large set of rephrasings and translations measured on the same WMDP-bio questions. A pre-registered replication with an independent attack template and a held-out question split is needed before treating that number as a stable property of ELM.
- If the effect is tokenization-driven, as the paper hints in its discussion, then other low-resource-language filler texts or adversarially chosen token sequences should produce similar recovery; scanning a grid of filler templates with a multiple-comparison correction would test this directly.
- The logit-output correlation implies that for most methods, logit-based scoring will not reveal hidden knowledge; probe analyses at mid-to-late layers are a more promising detector of residual knowledge than either surface metric, and could be adopted as a standard extra evaluation.
- The pattern across methods suggests a capability-robustness frontier: ELM keeps general capabilities but is attackable, while TAR and RepNoise look robust largely because unlearning degraded them. Future unlearning methods may need explicit objectives for both representation erasure and output consistency.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents an empirical evaluation of eight machine unlearning methods (RMU, ELM, TAR, RMU+LAT, GradDiff, PB&J, RR, RepNoise) on the WMDP-bio benchmark across Zephyr, Mistral, and Llama-3 model families. The authors measure output-level and logit-level accuracy under a battery of prompt modifications (filler text in English/Latin/Hindi, rephrasing as conversation/poem, variable substitution, term removal, translation) and use linear probes on the residual stream. They find that most methods remain at low accuracy under all prompt variants, but prepending a specific Hindi filler paragraph to ELM-unlearned models recovers 57.3% accuracy (with consistent but smaller recoveries across all ELM checkpoints tested), while RMU and TAR remain robust. They conclude that ELM suppresses rather than removes targeted knowledge, and that output-formatting effects do not explain the results because logit and output accuracies are similar.
Significance. If the findings hold, the paper provides a concrete, reproducible prompt attack that defeats ELM unlearning and a useful toolkit for evaluating unlearning robustness; the replication across four ELM checkpoints and the release of the evaluation framework are strengths. The paper is transparent about reporting all per-condition results in appendices, which makes the post-hoc selection of the Hindi filler condition visible. The main caveat is that the headline 57.3% is a best-of-many score without correction or held-out validation, so the quantitative magnitude and the generality of 'Hindi filler' as a category remain uncertain. With additional validation, the work would be a meaningful contribution to unlearning evaluation.
major comments (2)
- [Section 2 / Appendix A] The headline recovery rate of 57.3% for ELM is the maximum over many prompt conditions tested on the same WMDP-bio questions, and the paper states that Hindi filler text 'stood out' among all rephrasings. Because the same test set was used to select the condition and to report the recovery rate, the 57.3% figure is a best-of-many estimate with no multiple-comparison correction and no held-out confirmation. The appendix also reveals that the 'Hindi filler' is not random filler but a single, semantically specific Devanagari paragraph about placeholder text, so the language label is confounded with template-specific content and length. I request a validation experiment with several fresh, pre-specified filler templates (and matched English/Latin controls) on the same ELM checkpoints, or a clear restriction of the claim to the exact template used.
- [Section 3 / Figure 3] The probe analysis is central to the claim that ELM 'suppresses knowledge at the output level without truly removing it from internal representations,' but the methodology is under-specified. The paper does not report the probe training set size, the train/test split, regularization, number of runs, or whether the probe questions overlap with the accuracy-evaluation questions. If probes are trained and evaluated on the same WMDP-bio items, the high probe accuracy under Hindi filler could reflect overfitting to the evaluation set rather than a general representational finding. Please provide these details and, if necessary, evaluate probes on a held-out split.
minor comments (6)
- [Appendix A] The description says 'Randomly generated filler text' but the displayed English, Latin, and Hindi filler texts are fixed, hand-authored paragraphs; please revise the wording to state that these are fixed templates.
- [Table 1] The duplicate rows 'translated_bengali' and 'translated_hindi' appear twice in Table 1; please check whether these are repeated runs and label them accordingly.
- [Table 2] In the hindi_filler_text row, the logit accuracy for wrong-format answers is reported as 0.1875, which is below chance and based on fewer than 2% of trials; this should be flagged as a small-sample estimate.
- [Figure 4 caption] The chance-adjustment (rescaling 0.25 to 0) is only explained in the caption; please introduce it in the main text so the y-axis values are interpretable.
- [Section 2, last paragraph] The statement that 'unlearning effectiveness does not vary significantly across model families' is not supported by any statistical test; the tables show similar qualitative patterns, but 'significantly' should be removed or supported.
- [Section 1, Methodology] The threat model is described as 'black-box' while granting access to logits; since logits are typically considered white-box information, please clarify the threat model terminology.
Circularity Check
No significant circularity; the central results are empirical measurements against external benchmarks and public checkpoints, not derivations from fitted inputs.
full rationale
This paper is an empirical evaluation, not a derivation chain. It measures output and logit accuracy of existing unlearned checkpoints (RMU, ELM, TAR, etc.) on the public WMDP-bio and tinyMMLU benchmarks using lm-evaluation-harness. No parameter is fitted to the target result and then renamed a prediction; the 57.3% Hindi-filler recovery is a measured outcome on a fixed model and benchmark, not a quantity constructed from its own inputs. The linear probes are diagnostic tools and do not feed back into the reported headline accuracies. The logit-output correlation is an independent measurement of the same model behavior, not an identity assumed in advance. The main methodological concern is that the Hindi filler template was selected after inspecting many rephrasings on the same test set, which limits generalizability and inflates the reported effect, but this is a selection-bias or external-validity issue, not circularity: the paper does not define ELM vulnerability in terms of the Hindi filler prompt, nor does it cite its own prior work to supply a load-bearing premise. The claims are self-contained against public checkpoints and external benchmarks, so no circular step is present.
Assumptions & free parameters
assumptions (5)
- domain assumption WMDP-bio multiple-choice accuracy measures presence of the targeted knowledge in the model.
- domain assumption Public unlearning checkpoints such as cais/Zephyr_RMU, baulab/elm-zephyr-7b-beta, and the LLM-GAT checkpoints faithfully implement the named methods.
- domain assumption Logit-based accuracy over A/B/C/D tokens and output-based accuracy are comparable measures of knowledge.
- domain assumption Linear probes trained on residual stream activations can decode whether knowledge is represented internally.
- standard math The lm-evaluation-harness computes correct next-token logits for the evaluated models.
Cite this review
Pith. "Pith review of Prompt Attacks Reveal Superficial Knowledge Removal in Unlearning Methods." pith.science (2026). https://pith.science/paper/K2W6CGJR
@misc{pith2026250610236,
author = {Pith},
title = {Pith review of: Prompt Attacks Reveal Superficial Knowledge Removal in Unlearning Methods},
year = {2026},
howpublished = {\url{https://pith.science/paper/K2W6CGJR}},
note = {Machine review of arXiv:2506.10236}
}
read the original abstract
In this work, we demonstrate that certain machine unlearning methods may fail under straightforward prompt attacks. We systematically evaluate eight unlearning techniques across three model families using output-based, logit-based, and probe analysis to assess the extent to which supposedly unlearned knowledge can be retrieved. While methods like RMU and TAR exhibit robust unlearning, ELM remains vulnerable to specific prompt attacks (e.g., prepending Hindi filler text to the original prompt recovers 57.3% accuracy). Our logit analysis further indicates that unlearned models are unlikely to hide knowledge through changes in answer formatting, given the strong correlation between output and logit accuracy. These findings challenge prevailing assumptions about unlearning effectiveness and highlight the need for evaluation frameworks that can reliably distinguish between genuine knowledge removal and superficial output suppression. To facilitate further research, we publicly release our evaluation framework to easily evaluate prompting techniques to retrieve unlearned knowledge.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[3]
Who’s harry potter? approximate unlearning in llms
Raz Eldan and Mark Russinovich. Who’s harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238,
-
[5]
6 Ryan Greenblatt, Fabien Roger, Dmitrii Krasheninnikov, and David Krueger
URL https://zenodo.org/records/12608602. 6 Ryan Greenblatt, Fabien Roger, Dmitrii Krasheninnikov, and David Krueger. Stress-testing capability elicitation with password-locked models. arXiv preprint arXiv:2405.19550,
-
[7]
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D
URL https://arxiv.org/ abs/2310.06825. Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bh...
-
[9]
Alexander Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell
URL https://openreview.net/forum?id=J5IRyTKZ9s. Alexander Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402.16835,
-
[10]
Wang, Colin Raffel, and Jonathan Frankle
Pratyush Maini, Himanshu Jain, Ho-Chiang Shen, Rohan Tian, Moitreya Mazeika, Tomas Olausson, Haizi Jang, Logan Cabrera, Jane Kim, Zhangir S. Wang, Colin Raffel, and Jonathan Frankle. Tofu: A task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06121,
-
[11]
McKinney, Anvith Thudi, Juhan Bae, Tara Rezaei Kheirkhah, Nicolas Papernot, Sheila A
Lev E. McKinney, Anvith Thudi, Juhan Bae, Tara Rezaei Kheirkhah, Nicolas Papernot, Sheila A. McIlraith, and Roger Baker Grosse. Gauss-newton unlearning for the llm era. In ICML 2025 Workshop on Machine Unlearning for Generative AI (MUGen) ,
work page 2025
-
[12]
URL https: //openreview.net/forum?id=VFfttnDvW6. Appendix F (“Unlearning with Projections”) de- scribes PullBack & proJect (PB&J). Vaidehi Patil, Peter Hase, and Mohit Bansal. Can sensitive information be deleted from LLMs? objectives for defending against extraction attacks. arXiv preprint arXiv:2309.17410,
-
[14]
arXiv:2402.14992. 7 Domenic Rosati, Jan Wehner, Kai Williams, Łukasz Bartoszcze, David Atanasov, Robie Gonzales, Subhabrata Majumdar, Carsten Maple, Hassan Sajjad, and Frank Rudzicz. Representation noising: A defence mechanism against harmful finetuning,
Show all 24 references
-
[15]
Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A
URL https://arxiv.org/abs/2407.15549. Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, and Chiyuan Zhang. MUSE: Machine unlearning Six-Way evaluation for language models. arXiv preprint arXiv:2407.06460,
-
[16]
UnUnlearning: Unlearning is not sufficient for content regulation in advanced generative AI.arXiv preprint arXiv:2407.00106,
Ilia Shumailov, Jamie Hayes, Eleni Triantafillou, Guillermo Ortiz-Jimenez, Nicolas Papernot, Matthew Jagielski, Itay Yona, Heidi Howard, and Eugene Bagdasaryan. UnUnlearning: Unlearning is not sufficient for content regulation in advanced generative AI.arXiv preprint arXiv:2407.00106,
-
[17]
URL https: //arxiv.org/abs/2408.00761. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zeph...
-
[18]
Brown, and Francis Rhys Ward
Teun van der Weij, Felix Hofstätter, Ollie Jaffe, Samuel F. Brown, and Francis Rhys Ward. AI sandbagging: Language models can strategically underperform on evaluations. arXiv preprint arXiv:2406.07358,
-
[19]
Gilbert, Yonatan Dagan, Peter Liao, Katherine Meng, Yuchen Yang, Michael Liao, Jianfeng Wu, Eric Wang, and Alborz Geramifard
Alexander Wei, Jiang Hu, Yixuan Weng, Zhangir Chi, Nguyet King, Stephen Macke, Besmira Nushi, Ece Kamar, Thomas K. Gilbert, Yonatan Dagan, Peter Liao, Katherine Meng, Yuchen Yang, Michael Liao, Jianfeng Wu, Eric Wang, and Alborz Geramifard. Jailbroken: How does llm behavior ch...
-
[20]
In-context learning can re-learn forbidden tasks
Sophie Xhonneux, David Dobre, Jian Tang, Gauthier Gidel, and Dhanya Sridhar. In-context learning can re-learn forbidden tasks. arXiv preprint arXiv:2402.05723,
-
[21]
Zheng-Xin Yong, Cristina Menghini, and Stephen H. Bach. Low-resource languages jailbreak GPT-4. arXiv preprint arXiv:2310.02446,
-
[22]
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J
URL https://arxiv.org/abs/2408.10682. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043,
-
[23]
8 A Models and Datasets Tested We focus on two primary benchmarks: • WMDP (The Weapons of Mass Destruction Proxy) Li et al
URL https://arxiv.org/abs/2406.04313. 8 A Models and Datasets Tested We focus on two primary benchmarks: • WMDP (The Weapons of Mass Destruction Proxy) Li et al
-
[24]
[2024], a subset of 100 data points selected from MMLU (Massive Multitask Language Understanding) Hendrycks et al
, with a specific focus on the biosecurity domain to assess harmful knowledge removal • tinyMMLU Polo et al. [2024], a subset of 100 data points selected from MMLU (Massive Multitask Language Understanding) Hendrycks et al
2024
-
[25]
Right Format
to assess overall model capabilities and potential side effects of unlearning We evaluate multiple unlearned model checkpoints, including: • Zephyr_RMU (base model: Zephyr-7B-beta) • ELM models (Gandikota et al. [2024]) – ELM Zephyr-7B-Beta (base model: Zephyr-7B-beta) – ELM M...
2024
-
[2021]
The elicitation game: Evaluating capability elicitation techniques
Felix Hofstätter, Teun van der Weij, Jayden Teoh, Henning Bartsch, and Francis Rhys Ward. The elicitation game: Evaluating capability elicitation techniques. arXiv preprint arXiv:2502.02180,
-
[2022]
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, Kush R
URL https://arxiv.org/abs/2203.12817. Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, Kush R. Varshney, Mohit Bansal, Sanmi Koyejo, and Yang Liu. Rethinking machine unlearning for large la...
-
[2023]
Erasing conceptual knowledge from language models
Rohit Gandikota, Sheridan Feucht, Samuel Marks, and David Bau. Erasing conceptual knowledge from language models. arXiv preprint arXiv:2410.02760,
-
[2024]
Zora Che, Stephen Casper, Robert Kirk, Anirudh Satheesh, Stewart Slocum, Lev E
URL https://github.com/meta-llama/llama3/blob/ main/MODEL_CARD.md. Zora Che, Stephen Casper, Robert Kirk, Anirudh Satheesh, Stewart Slocum, Lev E. McKinney, Rohit Gandikota, Aidan Ewart, Domenic Rosati, Zichu Wu, Zikui Cai, Bilal Chughtai, Yarin Gal, Furong Huang, and Dylan Ha...
-
[2025]
Does unlearning truly unlearn? a black box evaluation of llm unlearning methods
Jai Doshi and Asa Cooper Stickland. Does unlearning truly unlearn? a black box evaluation of llm unlearning methods. arXiv preprint arXiv:2411.12103,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.