{"id":"06c91c34-26a8-40cd-9789-53f10d13b6cd","arxiv_id":"2506.10236","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Prepending a Hindi filler paragraph to WMDP-bio questions restores 57.3% accuracy in ELM-unlearned models, showing the unlearning is superficial output suppression rather than true knowledge removal.","lead":"This paper tests whether eight machine unlearning methods really delete dangerous knowledge from language models by trying simple prompt rewrites, translations, and filler text. It finds one popular method, ELM, lets supposedly deleted biosecurity knowledge come back when a Hindi filler paragraph is prepended, recovering 57.3% accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hindi filler recovery is a post-hoc selection from many prompt conditions; no pre-specified or held-out filler template confirms that ELM's vulnerability generalizes beyond the single Devanagari paragraph.","rationale":"The abstract's central empirical claim is that a prompt attack recovers 57.3% on ELM, and Section 3 interprets this as evidence of superficial knowledge removal. This claim is load-bearing because the rest of the paper's framing, that unlearning evaluations overstate removal, depends on the recovery being a real, robust property of ELM rather than an artifact of a single prompt template chosen after inspecting many conditions. The reader's weakest assumption identifies exactly this issue: the Hindi filler was selected post hoc, with no multiple-comparison correction and no independent template. I agree. The cross-checkpoint consistency in Tables 6-9 is real mitigating evidence: the same selected template lifts five ELM checkpoints from near-baseline to 0.47-0.59, so the effect is not a single-model fluke. However, that consistency does not address template selection; if the one Hindi paragraph happens to be an unusually effective prompt (e.g., because its content or tokenization disrupts ELM's suppression), the same template would consistently work across models while other Hindi fillers would not. The abstract itself hedges with 'specific prompt attacks,' but the Discussion generalizes to 'prompt manipulations' broadly. A concrete, pre-registered replication with multiple new Hindi fillers would settle whether the vulnerability is language-level or template-level. Because the current evidence is suggestive but not definitive, the reader's CONDITIONAL verdict is appropriate; no verdict change is warranted.","tokens_in":17774,"tokens_out":9713,"duration_ms":122351,"concrete_test":"Using the released baulab/elm-zephyr-7b-beta checkpoint and the paper's lm-evaluation-harness script, fix a priori a set of 10 new Hindi filler passages: 5 from unrelated Devanagari sources (e.g., random Wikipedia articles or news paragraphs) and 5 Hindi translations of the English filler text in Appendix A. Prepend each to the original WMDP-bio questions, compute output and logit accuracy, and compare to the paper's 0.5727. Pre-register a threshold, e.g., require mean accuracy above 0.45 across all 10 templates and at least 8 of 10 above 0.45. If the effect replicates broadly, the vulnerability generalizes; if only the original template clears 0.50, the headline should be reframed as a single-template artifact and the Discussion's generalization withdrawn.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim rests on one prompt template: Hindi filler text recovers 57.3% on ELM (abstract; Section 2). Section 2 states this template 'stood out' among all rephrasings, but the methods do not pre-specify it, and the same WMDP-bio questions used to select the condition are used to report the recovery rate. Appendix A shows the Hindi filler is not a random sample of Hindi text but a single, semantically specific Devanagari paragraph about placeholder text; the English and Latin fillers differ in length and content, so the 'Hindi' label may be a proxy for template-specific content rather than language. Tables 6-9 show the selected template consistently lifts ELM checkpoints to roughly 0.47-0.59 from 0.29-0.34 baselines, which mitigates pure chance, but it does not establish that other Hindi fillers would work. The Discussion's broader claim that ELM remains vulnerable to prompt manipulations extrapolates from this single unvalidated template. If replication with fresh Hindi fillers fails, the central conclusion reduces to 'one specially constructed template can recover some accuracy,' which would not support the claim that unlearning evaluations generally overstate removal.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an empirical evaluation of eight machine unlearning methods (RMU, ELM, TAR, RMU+LAT, GradDiff, PB&J, RR, RepNoise) on the WMDP-bio benchmark across Zephyr, Mistral, and Llama-3 model families. The authors measure output-level and logit-level accuracy under a battery of prompt modifications (filler text in English/Latin/Hindi, rephrasing as conversation/poem, variable substitution, term removal, translation) and use linear probes on the residual stream. They find that most methods remain at low accuracy under all prompt variants, but prepending a specific Hindi filler paragraph to ELM-unlearned models recovers 57.3% accuracy (with consistent but smaller recoveries across all ELM checkpoints tested), while RMU and TAR remain robust. They conclude that ELM suppresses rather than removes targeted knowledge, and that output-formatting effects do not explain the results because logit and output accuracies are similar.","tokens_in":17999,"tokens_out":6321,"duration_ms":74324,"significance":"If the findings hold, the paper provides a concrete, reproducible prompt attack that defeats ELM unlearning and a useful toolkit for evaluating unlearning robustness; the replication across four ELM checkpoints and the release of the evaluation framework are strengths. The paper is transparent about reporting all per-condition results in appendices, which makes the post-hoc selection of the Hindi filler condition visible. The main caveat is that the headline 57.3% is a best-of-many score without correction or held-out validation, so the quantitative magnitude and the generality of 'Hindi filler' as a category remain uncertain. With additional validation, the work would be a meaningful contribution to unlearning evaluation.","major_comments":[{"comment":"The headline recovery rate of 57.3% for ELM is the maximum over many prompt conditions tested on the same WMDP-bio questions, and the paper states that Hindi filler text 'stood out' among all rephrasings. Because the same test set was used to select the condition and to report the recovery rate, the 57.3% figure is a best-of-many estimate with no multiple-comparison correction and no held-out confirmation. The appendix also reveals that the 'Hindi filler' is not random filler but a single, semantically specific Devanagari paragraph about placeholder text, so the language label is confounded with template-specific content and length. I request a validation experiment with several fresh, pre-specified filler templates (and matched English/Latin controls) on the same ELM checkpoints, or a clear restriction of the claim to the exact template used.","section":"Section 2 / Appendix A"},{"comment":"The probe analysis is central to the claim that ELM 'suppresses knowledge at the output level without truly removing it from internal representations,' but the methodology is under-specified. The paper does not report the probe training set size, the train/test split, regularization, number of runs, or whether the probe questions overlap with the accuracy-evaluation questions. If probes are trained and evaluated on the same WMDP-bio items, the high probe accuracy under Hindi filler could reflect overfitting to the evaluation set rather than a general representational finding. Please provide these details and, if necessary, evaluate probes on a held-out split.","section":"Section 3 / Figure 3"}],"minor_comments":[{"comment":"The description says 'Randomly generated filler text' but the displayed English, Latin, and Hindi filler texts are fixed, hand-authored paragraphs; please revise the wording to state that these are fixed templates.","section":"Appendix A"},{"comment":"The duplicate rows 'translated_bengali' and 'translated_hindi' appear twice in Table 1; please check whether these are repeated runs and label them accordingly.","section":"Table 1"},{"comment":"In the hindi_filler_text row, the logit accuracy for wrong-format answers is reported as 0.1875, which is below chance and based on fewer than 2% of trials; this should be flagged as a small-sample estimate.","section":"Table 2"},{"comment":"The chance-adjustment (rescaling 0.25 to 0) is only explained in the caption; please introduce it in the main text so the y-axis values are interpretable.","section":"Figure 4 caption"},{"comment":"The statement that 'unlearning effectiveness does not vary significantly across model families' is not supported by any statistical test; the tables show similar qualitative patterns, but 'significantly' should be removed or supported.","section":"Section 2, last paragraph"},{"comment":"The threat model is described as 'black-box' while granting access to logits; since logits are typically considered white-box information, please clarify the threat model terminology.","section":"Section 1, Methodology"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly written and the empirical sweep is useful, but it reads more like a workshop contribution than a journal article in its current form. The post-hoc selection issue is the main barrier; I would ask the authors to either run a small confirmatory study with fresh filler templates or substantially weaken the abstract's wording. There is also a minor data-quality issue (duplicate rows in Table 1) that should be fixed. The paper's scope (unlearning evaluation) fits the journal, but the methodology needs the additional validation described."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful, well-scoped empirical audit, and the ELM vulnerability is real in the narrow sense that a single Devanagari filler template consistently lifts accuracy across every ELM checkpoint they tested. But the headline 57.3% is a post-hoc selection from many prompt conditions on the same test set, with no held-out attack template or multiple-comparison correction. So the paper's central claim should be framed as an existence proof for 'specific prompt attacks', not as evidence that ELM is broadly vulnerable to Hindi filler text.\n\nWhat's genuinely new: the systematic comparison of eight unlearning methods across three model families, with output, logit, and probe analysis, on public checkpoints, with code released. That is a solid contribution. The contrast between RMU/TAR on one side and ELM on the other is non-obvious and worth knowing, especially since TAR's robustness is accompanied by degraded tinyMMLU performance. The logit/output correlation analysis is a reasonable negative result against the formatting-suppression hypothesis, though it is informal.\n\nSoft spots, in order of importance. First, the selection problem. The paper admits the Hindi filler 'stood out' after testing many rephrasings, and the same WMDP-bio questions were used both to select and to report the recovery rate. The replication across ELM checkpoints (Tables 6-9, 14) mitigates the worry that it is one lucky run, but the template itself is a single, semantically specific Devanagari paragraph about placeholder text, not a random sample of Hindi. Without a second filler template or a pre-specified one, we cannot tell whether this is a language effect or a template-specific/tokenization effect. Second, the probe analysis is underreported: no probe architecture, training data, or accuracy numbers in the main text, just the curves in Figure 3. That weakens the claim that internal representations retain knowledge. Third, the abstract's implication that unlearning evaluations generally overstate removal extrapolates from ELM; RMU and TAR look robust here, so the paper's own data are more nuanced.\n\nWho this is for: anyone working on unlearning evaluation or safety benchmarks. The systematic comparison and public framework make it a legitimate reference point, even if the headline number needs replication with fresh templates.\n\nRecommendation: send to peer review. The selection issue is fixable with a pre-specified or held-out filler template, and the probe details are fixable with a proper methods section. The empirical core is solid enough to warrant referee time.","headline":"A competent empirical audit with a plausible core finding, but the headline Hindi-filler recovery is a post-hoc selected template that needs held-out validation before being treated as a robust measurement.","tokens_in":18550,"tokens_out":2539,"would_cite":true,"duration_ms":30020,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prepending a fixed Hindi filler passage restores 57.3% accuracy on WMDP-bio questions that the ELM unlearning method was supposed to have removed, and logit and probe analyses show the knowledge was suppressed rather than erased.","keywords":["machine unlearning","prompt attacks","ELM","RMU","WMDP benchmark","logit analysis","representation probing","large language models"],"falsifier":"Run a pre-registered version of the experiment: fix the Hindi filler template and several control filler templates (English, Latin, another low-resource language) before looking at any results, evaluate on a held-out half of WMDP-bio never used to choose the templates, and require the Hindi condition to beat both controls and the original-format condition by a margin that survives a multiple-comparison correction. If the recovery drops to near the roughly 30% unlearned baseline for new filler texts or held-out questions, the claim that ELM's knowledge is retrievably stored would be weakened. A second independent check: train linear probes on one subset and test on another; if probe accuracy stays high for ELM with Hindi filler but not for RMU, the representation-level interpretation is supported.","tokens_in":17586,"feed_emoji":"🧠","tokens_out":7607,"duration_ms":83456,"temperature":0.7,"pith_summary":"The paper asks whether machine unlearning actually erases knowledge or merely teaches a model to suppress it in familiar prompt formats, and it answers by stress-testing eight unlearning methods with rephrasings, translations, and filler text on the WMDP-bio benchmark. Its central finding is that one method, ELM, remains vulnerable to a simple attack: prepending a fixed block of Hindi filler text to the original question lifts accuracy on supposedly unlearned biology multiple-choice questions to 57.3%, roughly double the unlearned baseline. Logit-level scoring and layer-by-layer probing show the recovered knowledge exists in the model's internal representations, so ELM appears to hide knowledge at the output rather than remove it. The paper also finds that robust-looking methods such as RMU and TAR either fail to answer in the expected format or lose general capability, suggesting current unlearning involves a trade-off between preservation and true removal. If correct, the result means standard unlearning evaluations that only try the original question format can substantially overstate how much knowledge has been deleted.","feed_headline":"A Hindi filler prompt restores 57.3% accuracy to an unlearned model","feed_subtitle":"ELM models suppress risky knowledge at output instead of erasing it; standard evals miss the recovery.","key_machinery":"The load-bearing tool is a three-channel evaluation suite: output-token accuracy on the original and rephrased prompts, logit accuracy (argmax over the A/B/C/D tokens), and linear probes trained on residual-stream activations from each layer. This suite lets the paper separate three failure modes: refusing to answer, genuinely losing knowledge, and suppressing knowledge behind an output-level gate. The specific attack that exposes ELM is a fixed Hindi filler passage prepended to the unchanged English question; because the filler is semantically unrelated, its effect is attributed to disrupting the suppression pattern rather than to added information.","core_discovery":"The central claim is that unlearning effectiveness is method-specific and that at least one widely used method, ELM (Erasure of Language Memory), does not truly unlearn WMDP-bio knowledge: it suppresses it. The evidence is that the same model that scores near chance when asked in the original format answers correctly when the original question is prefixed with Hindi filler content (57.3% output accuracy on Zephyr-7B, with smaller recoveries on Mistral-7B and Llama-3 variants), and that the recovered accuracy is mirrored in the logits over the four multiple-choice tokens and in linear probes of the residual stream. Because output accuracy and logit accuracy are highly correlated across methods, the paper argues the model is not hiding knowledge through refusal or formatting games. The authors' interpretation is that ELM suppresses knowledge at the output level while the information remains encoded internally, whereas RMU and TAR show more robust removal, at the cost of formatting failures or degraded general capability.","pith_inferences":["The 57.3% figure is best-of-many: the Hindi filler condition was selected after the fact because it stood out among a large set of rephrasings and translations measured on the same WMDP-bio questions. A pre-registered replication with an independent attack template and a held-out question split is needed before treating that number as a stable property of ELM.","If the effect is tokenization-driven, as the paper hints in its discussion, then other low-resource-language filler texts or adversarially chosen token sequences should produce similar recovery; scanning a grid of filler templates with a multiple-comparison correction would test this directly.","The logit-output correlation implies that for most methods, logit-based scoring will not reveal hidden knowledge; probe analyses at mid-to-late layers are a more promising detector of residual knowledge than either surface metric, and could be adopted as a standard extra evaluation.","The pattern across methods suggests a capability-robustness frontier: ELM keeps general capabilities but is attackable, while TAR and RepNoise look robust largely because unlearning degraded them. Future unlearning methods may need explicit objectives for both representation erasure and output consistency."],"forward_implications":["ELM-unlearned models retain retrievable WMDP-bio knowledge; the paper's released evaluation framework lets other researchers test further unlearning methods against the same prompting techniques.","Output-token and logit accuracies move together across most methods, so unlearning evaluations can rely on either signal without worrying that models are concealing knowledge by changing answer format.","RMU and TAR resist the tested prompt attacks, but RMU frequently fails to emit a valid option token and TAR, GradDiff, RepNoise, and RMU+LAT show reduced tinyMMLU accuracy, indicating robustness may partly be collateral capability loss.","Standard unlearning evaluations that reuse the original question format are insufficient; robustness should be measured across prompt variations and checked against general-capability retention.","The Hindi-filler recovery holds across three model families (Zephyr, Mistral, Llama 3), so the effect is not an artifact of a single architecture."],"supporting_citations":[{"why":"Supplies the WMDP-bio benchmark used to measure unlearning and the RMU method that serves as a robust comparison.","marker":"Li et al. [2024]"},{"why":"Provides the ELM-unlearned models (Erasure of Language Memory) that the paper shows remain vulnerable to Hindi filler prompts.","marker":"Gandikota et al. [2024]"},{"why":"Source of the prompting techniques and the hypothesis that unlearning may suppress knowledge at the output level rather than remove it, which the paper tests.","marker":"Doshi and Stickland [2024]"},{"why":"Defines the RMU+LAT method and the threat model for latent adversarial training, one of the robust baselines.","marker":"Sheshadri et al. [2024]"},{"why":"Defines Tamper Attack Resistance (TAR), the method the paper finds robust but with degraded general capabilities.","marker":"Tamirisa et al. [2025]"},{"why":"Supplies the language-model evaluation harness used for logit-based multiple-choice scoring across all methods.","marker":"Gao et al. [2024]"}],"fun_headline_variants":["Hindi filler prompt restores 57% of supposedly unlearned knowledge","Simple Hindi prefix bypasses unlearning, recovers 57% accuracy","Unlearning exposed: Hindi filler reveals knowledge still in model","Prompt attacks show unlearning only hides, not erases, knowledge"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline result depends on the Hindi filler template being a genuine attack rather than the single best-performing variant selected from many conditions measured on the same test questions, because the paper reports no pre-specified attack template and no correction for multiple comparisons.","fun_headline_variants_meta":{"raw":{"variants":["Hindi filler prompt restores 57% of supposedly unlearned knowledge","Simple Hindi prefix bypasses unlearning, recovers 57% accuracy","Unlearning exposed: Hindi filler reveals knowledge still in model","Prompt attacks show unlearning only hides, not erases, knowledge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00065,"raw_usage":{"total_tokens":2961,"prompt_tokens":904,"completion_tokens":2057,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":1982}},"tokens_in":520,"tokens_out":2057,"duration_ms":18758,"temperature":1.0,"reasoning_tokens":1982,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:31:22.140971+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a pre-registered version of the experiment: fix the Hindi filler template and several control filler templates (English, Latin, another low-resource language) before looking at any results, evaluate on a held-out half of WMDP-bio never used to choose the templates, and require the Hindi condition to beat both controls and the original-format condition by a margin that survives a multiple-comparison correction. If the recovery drops to near the roughly 30% unlearned baseline for new filler texts or held-out questions, the claim that ELM's knowledge is retrievably stored would be weakened. A second independent check: train linear probes on one subset and test on another; if probe accuracy stays high for ELM with Hindi filler but not for RMU, the representation-level interpretation is supported.","supporting_citations":[{"cited_title":"Right Format","cited_arxiv_id":null,"evidence_quote":"Provides the ELM-unlearned models (Erasure of Language Memory) that the paper shows remain vulnerable to Hindi filler prompts."}],"review_version":1}