{"id":"59e56f10-f914-4420-a5d9-757ba9e4abfd","arxiv_id":"2502.05209","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Model tampering attacks, especially few-shot fine-tuning, reliably re-elicit unlearned capabilities in Llama-3-8B and can bound the success of held-out input-space attacks.","lead":"This paper tests whether tampering with a model's internal weights or activations reveals hidden dangerous knowledge more reliably than standard input-based jailbreaks. It finds that few-shot fine-tuning undoes state-of-the-art unlearning and can provide conservative estimates of input-space attack success.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The predictive claim rests on 65 models that are only 8 checkpoints from 8 unlearning runs of one base model; the apparent correlations may not survive independent model sampling, and the held-out AISI and jailbreak results already show weakened signal.","rationale":"The reader's weakest assumption is that the five held-out input-space attacks proxy unforeseen attacks. My concern is adjacent but more specific: even for the held-out attacks tested, the statistical evidence for predictiveness is weakened by non-independent checkpoints and narrow model diversity. The paper's own UK AISI results (pruning r=0.66, benign r=0.59, best fine-tune r=-0.16; weak cross-task residuals) and jailbreak results (no clear correlation with n=9) are consistent with this concern. The low-dimensional subspace result, 89% of variance in three principal components, is descriptive and may also be inflated by shared training runs, but the actionable claim for evaluation practice is the predictive and conservative-estimate claim. The fine-tuning reversal result, where 16 or fewer steps recover over 30 points of WMDP-Bio performance, is strong and largely independent of the correlation analysis; it alone supports the paper's practical message about open-weight risks. The paper is transparent about non-independence and the proxy assumption, and it releases the model suite, so the right response is to keep the verdict CONDITIONAL pending an independent-checkpoint analysis rather than to reject. The concrete test above directly settles whether the correlations survive when checkpoints are not treated as independent samples.","tokens_in":25497,"tokens_out":4078,"duration_ms":40075,"concrete_test":"Recompute the unlearning-score-weighted Pearson correlations in Figure 5 in three ways: (1) using only the best-checkpoint model from each of the 8 unlearning methods plus TAR-v2 (n=9); (2) using only the final checkpoint of each run; and (3) within-method centered correlations, subtracting each method's mean attack-success vector from its 8 checkpoints to remove between-method variation. If the embedding, latent, pruning, or benign-fine-tune correlations drop below about 0.5 or change sign in any of these analyses, the predictive claim is an artifact of non-independent checkpoints rather than a general property of model robustness. As a second check, repeat the Section C.1 worst-case regression with one checkpoint per method; if including model tampering attacks no longer improves R2 by the reported 0.05-0.1, the claim that tampering attacks improve held-out estimation is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All 65 unlearning models are 8 evenly spaced checkpoints from 8 training runs of Llama-3-8B-Instruct. The central empirical claim that model tampering attack success predicts, and conservatively bounds, held-out input-space attack success is estimated on these non-independent checkpoints, weighted by unlearning score. Correlations such as r=0.87 (pruning) and r=0.84 (benign fine-tune) in Figure 5 may be inflated by within-run trajectory correlations and by method-level differences, for example methods that produce dysfluent models failing all tasks. The held-out UK AISI evaluation (Figures 6 and 10) shows the signal largely disappears: only pruning (r=0.66) and benign fine-tuning (r=0.59) correlate, while best fine-tune r=-0.16; cross-task residual correlations are weak (Figure 12). The jailbreaking replication has only 9 models and no clear correlation (Figure 17, best fine-tune r=0.32, p=0.40). Thus the 'predict' half of the central claim is not yet established for genuinely new models or attacks. The 'conservative estimate' half is better supported, since fine-tuning attacks exceed the best input-space attack on 64 of 65 models, but this may simply reflect that weight-space attacks are a strictly more powerful threat model rather than a bound on unknown input-space attacks. The paper acknowledges non-independence and the proxy assumption, but the abstract and methodology contributions state the predictive relationship as a main finding.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes adding model tampering attacks (perturbations to latent activations or model weights) to the standard input-space attack toolbox for LLM capability evaluation. It pits 8 unlearning methods and 9 refusal-finetuned Llama-3-8B-Instruct models against 5 input-space and 6 model tampering attacks, measuring WMDP-Bio accuracy recovery and StrongReject compliance rates. The main claims are: (1) attack robustness lies on a low-dimensional subspace; (2) model tampering attack success can empirically predict and conservatively bound the success of held-out input-space attacks; and (3) state-of-the-art unlearning can be undone within 16 fine-tuning steps. The paper releases 64 unlearned models and includes proprietary held-out attacks from the UK AI Security Institute and Haize Labs' Cascade as additional validation.","tokens_in":25744,"tokens_out":7289,"duration_ms":64360,"significance":"If the central claims hold, the paper makes a valuable methodological contribution: it provides the first systematic comparison of input-space and weight/latent-space capability elicitation, releases a reusable unlearning model suite, and offers practical efficiency evidence for model tampering attacks. The low-dimensional subspace finding and the masking of input-space bounds by fine-tuning attacks are relevant to safety evaluations of open-weight models. The paper is also commendable for explicitly acknowledging non-i.i.d. data, reporting a held-out proprietary attack evaluation, and releasing artifacts. However, the strongest contribution is the benchmark and the conservative-estimate observation; the predictive claim is currently weaker than the abstract states, for reasons detailed in the major comments.","major_comments":[{"comment":"The Abstract and Figure 7 state that state-of-the-art unlearning can be reversed within 16 gradient steps using 128 or fewer examples, but the hyperparameter table in Appendix A.4 lists configurations with 400 examples and up to 50 gradient steps (Full-1: 400 examples, 25 steps; LoRA-1, LoRA-2, Benign-1: 400 examples, 50 steps). Either the \"within 16 steps\" claim applies only to a subset of rows in Table 4 and must be stated as such, or the efficiency figures in Table 3 are incomplete. This is load-bearing for contribution (3), so the manuscript should be corrected and the affected claims re-examined.","section":"Appendix A.4, Table 4; §4.1.3"},{"comment":"The predictive claim that model tampering attacks \"empirically predict\" held-out input-space attacks is estimated on 65 non-independent points: eight evenly spaced checkpoints from each of eight unlearning runs of Llama-3-8B-Instruct plus one additional TAR model. Weighted Pearson correlations such as r=0.87 (pruning) and r=0.84 (benign fine-tuning) can be inflated by within-run trajectory correlations and by method-level differences, for example dysfluent GradDiff and TAR models failing all tasks. The footnote disclaiming formal p-values is appropriate, but the abstract and contribution 3 present the predictive relationship without this caveat. The paper should report analyses that hold out entire unlearning runs or method families, use one checkpoint per run, or bootstrap over independent runs, and show whether the correlations survive.","section":"§4.1.3, Figure 5"},{"comment":"The held-out proprietary evaluations substantially weaken the predictive claim. For the UK AISI attack, only pruning (r=0.66) and benign fine-tuning (r=0.59) correlate with attack success, best fine-tuning correlates negatively (r=-0.16), and cross-task residual correlations are weak (Figure 12). In the jailbreaking replication with 9 models, no clear correlation appears (Figure 17, best fine-tune r=0.32, p=0.40). Since these are the closest available tests to \"unforeseen\" input-space attacks, the current evidence supports a narrower claim: certain tampering attacks are informative for some held-out attacks on the same task, not a general predictive relationship. The paper should temper the contribution or provide additional evidence for generalization to new models and new attacks.","section":"§4.1.3, Figures 6, 10, 12; §4.2, Figure 17"},{"comment":"The attack suite was selected after many-shot and translation attacks were found to be consistently unsuccessful and were excluded from the reported results. If these exclusions were made after inspecting outcomes, the reported correlations and the \"64 of 65 models\" conservative-estimate percentages in Figure 5 are conditional on a favorable selection and may overstate the relationship. Please report the full set of attempted attacks, state the selection rule, and show whether the main correlations and the 64/65 result are robust under alternative attack suites.","section":"Appendix A.4, \"Excluded attacks\""},{"comment":"The unlearning score in Eq. (1) uses hand-picked utility weights (1/4, 1/4, and 1/2 for MMLU, AGIEval, and MT-Bench), and the PCA in §4.1.2 weights rows by sqrt(S_unlearn). The low-dimensional subspace and the unlearning-score-weighted correlations both depend on this aggregation. A short ablation varying the utility weights, the PCA row-weighting exponent, or using unweighted rows would establish that the subspace and predictive results are not artifacts of the chosen aggregation.","section":"Equation (1), §4.1.2"},{"comment":"The phrase \"conservative estimates\" is used for fine-tuning attacks exceeding the best input-space attack on 64 of 65 models. Because fine-tuning modifies weights rather than inputs, this may simply reflect a strictly more powerful threat model rather than a bound on unknown input-space attacks. The Discussion's caveat about possible limitations in the mechanistic similarity of input-space and tampering attacks (Section 5) should be integrated into the formulation of the contribution, and the estimand (maximum over the five tested input-space attacks) should be defined explicitly.","section":"§4.1.3, contribution 3"}],"minor_comments":[{"comment":"The list of contributions states that 64 models are released, while Section 3 and the experiments use 65 models (8 checkpoints from 8 methods plus one TAR-v2 model); please make the count consistent.","section":"§1, Contribution 4"},{"comment":"Please state in the Table 4 caption which rows correspond to the attacks summarized in Figure 7 so that the \"16 steps, 128 examples\" statement can be verified against the hyperparameters.","section":"Appendix A.4, Table 4"},{"comment":"The K-FADE method is cited as \"McKinney et al.\" without a year or venue in the bibliography; please add the full reference.","section":"References"},{"comment":"The caption of Figure 3(b) should clarify that the displayed values are eigenvalue-weighted principal component loadings rather than raw loadings, since the current label alone is ambiguous.","section":"Figure 3"},{"comment":"The text moves from correlations to \"conservative estimates\" without explicitly defining the estimand; please define it as the maximum over the five tested input-space attacks and note that it is not a bound over all possible future attacks.","section":"§4.1.3"},{"comment":"The full correlation matrix shows negative correlations for the adversarial fine-tuning attacks, which complicates the predictive story; a main-text sentence explaining this would help readers interpret the contrast with the positive correlations in Figure 5.","section":"Figure 13"}],"recommendation":"major_revision","confidential_remarks":"The manuscript header states that it is already published in TMLR (07/2025); the editor should confirm that this submission is a revision or that prior publication is disclosed, although this does not affect my technical assessment. The benchmark and released model suite are genuinely useful, but I would not accept the predictive claim in its current abstract-level form until the non-independence issue, the held-out AISI/jailbreak results, and the Table 4 contradiction with the \"16 steps\" claim are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The strongest and most durable finding is that fine-tuning undoes unlearning: within 16 steps, most state-of-the-art unlearning methods lose over 30 points of WMDP-Bio protection, and even benign fine-tuning partly recovers the knowledge. That result is simple, reproducible, and directly relevant to anyone governing open-weight models. It alone justifies the paper.\n\nThe rest is more mixed. The low-dimensional robustness subspace (3 PCs explain 89% of variance) is a nice empirical observation, and the release of 64 unlearned models is a genuine service. The conservative-bounding property of fine-tuning attacks—they beat the best input-space attack on 64 of 65 models—is real, though it may partly just mean weight-space attacks are a strictly more powerful threat model.\n\nWhat I would not take at face value is the 'predict' half of the central claim. The correlations in Figure 5 (r=0.87 pruning, r=0.84 benign fine-tuning) come from 65 models that are really 8 checkpoints from 8 unlearning runs of one base model. Within-run trajectory correlation inflates these numbers. The paper admits this in a footnote, but the abstract still presents the predictive relationship as a main finding. When tested on a genuinely held-out proprietary attack from UK AISI, the signal mostly disappears: only pruning (r=0.66) and benign fine-tuning (r=0.59) correlate, and best fine-tuning is r=-0.16. The jailbreak replication has only 9 models and no clear correlation (r=0.32, p=0.40). Cross-task results are also weak. So the predictive claim survives only as a conservative-bounding claim, not as a robust way to forecast unknown input-space attacks.\n\nThe paper is honest about most of this. It flags the proxy assumption, the non-independence of the points, and the fact that many-shot and translation attacks were dropped after failing. That curation is a concern, but it is disclosed. The unlearning score weights are hand-picked, and there are no multiple seeds or error bars—minor weaknesses given the breadth of experiments.\n\nBottom line: this deserved peer review and it got it. Cite the fine-tuning result freely; cite the predictive claim with care. I'd bring it to reading group.","headline":"The fine-tuning-undoes-unlearning result is solid and important, but the predictive claim about held-out input-space attacks is weaker than the abstract implies and survives only on the conservative-bounding half.","tokens_in":26367,"tokens_out":2920,"would_cite":true,"duration_ms":25715,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that attacks modifying a model's weights or latent activations reveal worst-case LLM capabilities that prompt-only evaluations miss, and that these tampering attacks can predict and conservatively bound held-out…","keywords":["model tampering attacks","LLM capability evaluation","machine unlearning","jailbreaking","fine-tuning attacks","latent-space attacks","robustness subspace","WMDP-Bio"],"falsifier":"Take the released 65 unlearned models and run a genuinely novel input-space attack, for example a multi-turn jailbreak in a low-resource language that was not part of the attack suite. If its success rate correlates near zero with the model tampering attacks and exceeds the best fine-tuning attack on most models, the predictive claim fails; separately, a utility-preserving unlearning method whose WMDP-Bio accuracy rises by less than 10 points after 16 gradient steps on 128 examples would refute the reversal claim.","tokens_in":25263,"feed_emoji":"🧠","tokens_out":8106,"duration_ms":68542,"temperature":0.7,"pith_summary":"Standard LLM risk evaluations probe a model with inputs—jailbreak prompts, adversarial suffixes—and record which ones elicit harmful behavior. This paper argues that such input-only evaluations are systematically incomplete, because they cannot see what an attacker who can modify the model itself would find. To close that gap, the authors pit 8 unlearning methods and 9 safety-tuned models against 5 input-space attacks and 6 model tampering attacks: embedding perturbations, latent perturbations, weights-and-activations pruning, benign fine-tuning, LoRA fine-tuning, and full-parameter fine-tuning. They claim three things: resilience across attack types collapses onto a low-dimensional subspace; tampering attack success predicts and generally over-estimates the success of held-out input-space attacks; and state-of-the-art unlearning can be reversed within a few gradient steps. If true, any evaluation that only changes inputs understates worst-case capability, especially for open-weight models.","feed_headline":"Fine-tuning in 16 steps defeats top LLM unlearning","feed_subtitle":"Weight and activation attacks out-predict prompt jailbreaks; input-only evals understate worst-case risk.","key_machinery":"The mechanism that carries the argument is the model tampering attack suite: embedding-space perturbations, latent-space perturbations, weights-and-activations pruning, benign fine-tuning, LoRA fine-tuning, and full-parameter fine-tuning, all rendered universal so they work for any prompt. Fine-tuning attacks take 1–16 gradient steps, sometimes with as few as 64 examples, and they act on the model's internals rather than searching the prompt space. The paper also uses weighted principal component analysis over attack-success vectors, weighted by unlearning score, to show that attacks share a low-dimensional robustness structure, and linear regression on attack subsets to show that tampering attacks add predictive signal for worst-case input-space failure.","core_discovery":"The paper's central claim is that model tampering attacks—modifications to latent activations or weights rather than to prompts—reveal worst-case LLM capabilities that input-space evaluations miss, and do so efficiently enough to serve as routine complements. Empirically, it reports that for 65 Llama-3-8B-Instruct derived models unlearned on WMDP-Bio, attack successes live in a low-dimensional subspace: three principal components explain 89% of the variance. It also reports that embedding, latent, pruning, and benign fine-tuning attack successes correlate with the success of held-out input-space attacks, while the stronger of LoRA and full fine-tuning attacks empirically exceeds the best input-space attack on 64 of 65 models. Finally, it reports that all utility-preserving unlearning methods can be attacked to recover more than 30 WMDP-Bio percentage points within 16 gradient steps and at most 128 examples, sometimes in a single step. The paper treats the held-out standard attacks as proxies for unforeseen input-space attacks, and notes weaker but still positive correlations with a proprietary, externally developed input-space attack.","pith_inferences":["Editorial inference: if the low-dimensional robustness subspace holds beyond the one model family tested, evaluators could select a small battery of tampering and input attacks spanning the principal components rather than enumerating attacks.","Editorial inference: the weak cross-task correlations with the proprietary external evaluation suggest tampering attacks indicate vulnerability within a task distribution, not global capability deletion; safety cases should combine tampering probes with task-specific benchmarks.","Editorial inference: because benign fine-tuning on ordinary text also resurfaces unlearned knowledge, the result implies that even non-adversarial downstream adaptation of open-weight models can restore supposedly removed capabilities, a deployment-relevant consequence beyond the paper's attack framing.","Editorial inference: a testable extension is to check whether the same low-dimensional robustness pattern appears across model sizes and architectures; if it does, tampering attacks could become a cheap pre-deployment screening tool."],"forward_implications":["Standard risk evaluations that only manipulate prompts should be supplemented with latent-space and weight-space attacks, because the paper shows input-only results understate worst-case capability.","Fine-tuning attacks can serve as a conservative upper-bound proxy: the stronger of LoRA and full fine-tuning beat the best input-space attack on 64 of 65 unlearned models.","Unlearning methods should be stress-tested against few-step fine-tuning before deployment, since all utility-preserving methods recovered more than 30 percentage points of WMDP-Bio performance within 16 gradient steps.","Because three principal components explain most attack-success variance, a small diverse set of attacks can approximate full-suite robustness measurements.","Refusal-tuned models show the same pattern: model tampering, especially fine-tuning and multi-turn attack algorithms, jailbroke all 9 safety-tuned models tested."],"supporting_citations":[{"why":"Supplies the GCG gradient-guided attack used as one of the held-out input-space baselines.","marker":"Zou et al. (2023b)"},{"why":"Defines WMDP-Bio, the dual-use biology benchmark and forget-set data that all unlearning experiments target.","marker":"Li et al. (2024b)"},{"why":"Supplies the latent-space perturbation attack and the latent adversarial training defense used in the benchmark.","marker":"Sheshadri et al. (2024)"},{"why":"Establishes few-shot fine-tuning as an attack that removes safety alignment, the basis for the weight-space attacks here.","marker":"Qi et al. (2023)"},{"why":"Prior evidence that pruning and low-rank modification expose safety brittleness, which the paper extends with more attacks and subspace analysis.","marker":"Wei et al. (2024)"},{"why":"Supplies the weights-and-activations pruning attack.","marker":"Sun et al. (2023)"},{"why":"Supplies the PAIR black-box input-space attack used as a held-out baseline.","marker":"Chao et al. (2024)"},{"why":"Supplies the BEAST perplexity-guided input-space attack used as a held-out baseline.","marker":"Sadasivan et al. (2024)"},{"why":"Supplies the AutoPrompt gradient-guided input-space attack.","marker":"Shin et al. (2020)"}],"fun_headline_variants":["Weight attacks reveal hidden LLM capabilities","Model tampering beats prompt jailbreaks","Unlearning undone in 16 gradient steps","Input-space evals understate LLM risk","Activation attacks predict held-out jailbreaks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's main claim depends on the assumption that the five held-out prompt attacks stand in for all future, unseen prompt attacks; if a genuinely new attack exploits a different mechanism, the observed correlations and conservative upper bounds may not hold.","fun_headline_variants_meta":{"raw":{"variants":["Weight attacks reveal hidden LLM capabilities","Model tampering beats prompt jailbreaks","Unlearning undone in 16 gradient steps","Input-space evals understate LLM risk","Activation attacks predict held-out jailbreaks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000351,"raw_usage":{"total_tokens":1944,"prompt_tokens":1008,"completion_tokens":936,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":871}},"tokens_in":624,"tokens_out":936,"duration_ms":8024,"temperature":1.0,"reasoning_tokens":871,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T14:42:42.225900+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the released 65 unlearned models and run a genuinely novel input-space attack, for example a multi-turn jailbreak in a low-resource language that was not part of the attack suite. If its success rate correlates near zero with the model tampering attacks and exceeds the best fine-tuning attack on most models, the predictive claim fails; separately, a utility-preserving unlearning method whose WMDP-Bio accuracy rises by less than 10 points after 16 gradient steps on 128 examples would refute the reversal claim.","supporting_citations":[],"review_version":1}