{"id":"5e05e1f6-7278-49bf-bd59-148af973db2c","arxiv_id":"2505.18588","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"CKU freezes neurons identified as important for useful knowledge during gradient-based unlearning, reporting lower jailbreak success rates with minimal utility loss.","lead":"This paper proposes a method called Constrained Knowledge Unlearning (CKU) that aims to make large language models forget harmful knowledge while preserving useful abilities. It matters because it offers a potential way to harden AI chatbots against jailbreak attacks, though the paper contains a sign inconsistency in its core equations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unlearning update in Eqs. (6)-(7) is inconsistent with gradient ascent: as written, it increases the likelihood of harmful responses, so the reported safety gains cannot follow from the described method.","rationale":"I read the paper in good faith: the central claim is that CKU improves jailbreak resistance with negligible utility loss, and the method must actually reduce the likelihood of harmful responses. Following the formal specification, however, the update rule does the opposite. Equations (2), (6), and (7) together describe either a gradient ascent on a positive log-likelihood (which would strengthen harmful behavior) or a minimization of a hinge-like loss that is never described as such. This is an internal inconsistency, not a disagreement with external consensus: the paper's own equations make the mechanism fail. The reader's explicit weakest_assumption about neuron localization is a separate empirical concern; even if freezing SNIP-selected neurons preserves useful knowledge perfectly, the sign error prevents the described training from delivering safety gains. I therefore agree with the rejection, though my emphasis differs from the reader's weakest-assumption framing. The paper has useful exploratory analysis of NLR and layer selection, but the core algorithm is not reproducibly specified, and no code is released. A corrected sign or optimizer description, along with code and confidence intervals, would be needed before the claims can be evaluated.","tokens_in":16166,"tokens_out":6319,"duration_ms":50499,"concrete_test":"Run the update rule exactly as printed on Llama2-7B-Chat: compute L_f from Eq. (6) on 10 AdvBench harmful prompts, take one gradient-ascent step with the Appendix A learning rate 4e-6, and measure the average log-likelihood of the harmful responses before and after. If the log-likelihood increases (or ASR on simplified GCG does not decrease), the formulation as written is antithetical to unlearning, and the paper must either change the sign in Eq. 6 or replace gradient ascent with gradient descent to match the reported results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines unlearning as gradient ascent, θ ← θ + η ∇_θ L_unlearn (Eq. 2), and states that maximizing L_unlearn progressively diminishes the model's reliance on the targeted data. Section 4.3 defines L_f = (1/|D_f|) Σ_{(x,y)∈D_f} Σ_{i=1}^{|y|} log p(y_i | T(x), y_<i) (Eq. 6). Because each p(y_i | ·) is a probability in (0,1), every log term is negative, so L_f is a negative quantity. Applying gradient ascent to L_f drives L_f upward (toward zero), increasing the conditional probability of the harmful response y given the jailbroken prompt T(x). That is the opposite of unlearning. The regularized objective in Eq. 7, L = max(0, λ + L_f), could lower the harmful-response likelihood only if it were minimized, but that would contradict the 'gradient ascent' description in §3.1, the update formula in Eq. 2, and the Figure 2 caption ('Update parameters using gradient ascent'). No code or checkpoints are provided to disambiguate. Thus the paper's formal specification of the core training step cannot reproduce the reported ASR reductions, and the central claim is unsupported as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Constrained Knowledge Unlearning (CKU), a safety-alignment method that identifies neurons in MLP layers associated with useful knowledge via SNIP scoring, freezes the top 80% of those neurons by pruning their gradients, and then performs unlearning on harmful prompt-response pairs using a regularized loss. The authors evaluate CKU on Llama2-7B-Chat and Llama3-8B-Instruct against AIM, GCG, AutoDAN, and decoding-exploitation attacks, reporting the lowest attack success rates in most settings and utility degradation of only about 0.1-0.2 points on average benchmarks. They also study the effects of neuron locking rate, the choice of unlearning layers, and the regularization weight λ.","tokens_in":16466,"tokens_out":5584,"duration_ms":50009,"significance":"If the results were reproducible, CKU would be a simple and practically useful contribution to jailbreak defense, and the layer- and neuron-level analysis could inform knowledge editing and model pruning. The empirical scope is broad: two base models, four attack families, and several utility benchmarks. However, the core training objective is specified inconsistently with the stated unlearning goal: as written, the gradient-ascent update on the loss in Eq. (6) would increase the likelihood of harmful responses, so the reported safety gains cannot be attributed to the described method. In addition, key hyperparameters are selected on the same safety and utility metrics that are later reported as the main results, which inflates the apparent advantage over baselines. No code or checkpoints are provided to resolve the ambiguity.","major_comments":[{"comment":"Equation (6) defines L_f as the average log-likelihood of the harmful target responses. Since each log-probability is negative, maximizing L_f via the gradient-ascent update in Eq. (2) drives the model to assign higher probability to the harmful tokens, which is the opposite of unlearning. The regularized objective in Eq. (7), L = max(0, λ + L_f), behaves consistently with the text only if it is minimized (driving L_f below -λ), but the paper repeatedly states that unlearning uses gradient ascent, including the caption of Figure 2. As written, the training procedure cannot produce the reported reductions in attack success rate; without code or checkpoints, the reader cannot tell whether the implemented method used gradient descent on Eq. (7), in which case the paper's formal description is incorrect.","section":"§3.1, §4.3, §4.4, Figure 2"},{"comment":"The values NLR=0.8 and λ=1.5 are selected in Figures 4 and 6 by sweeping exactly the same metrics—GCG ASR and average benchmark accuracy—that are later reported as the main results in Tables 1 and 2. This is test-set hyperparameter selection: the reported 'best' numbers are not independent of the selection criterion, and no held-out validation or nested procedure is described. The claim of a superior safety-utility balance is therefore overstated.","section":"§5.2 and §6.4"},{"comment":"The central assumption that the top 80% of neurons ranked by SNIP on Alpaca encode useful knowledge while harmful knowledge resides in the remaining 20% is not directly tested. Table 3 only compares random selection with SNIP ranking on the final safety and utility metrics; it does not verify that harmful knowledge is localized outside the frozen set. If useful and harmful knowledge share neurons, the gradient masking would either prevent unlearning or preserve harmful associations, so the mechanism underlying CKU remains unvalidated.","section":"§4.1-4.2 and Table 3"}],"minor_comments":[{"comment":"Equation (4) reuses the symbol I(W, x) for an expectation over the calibration dataset; a different symbol such as Ī(W) would avoid confusion with the per-sample score in Eq. (3).","section":"§4.1, Eq. (4)"},{"comment":"Figure 3 is referenced in the text, but the actual plot is not included in the manuscript; only the caption is present, making the 'only_mlp achieves the best' result unverifiable.","section":"Figure 3"},{"comment":"The text reports an 'improvement of more than threefold' at NLR=0.8 and 'more than fourfold' for layers 8-12, but Table 1 shows Llama2 GCG ASR dropping from 11.54% to 4.23%, which is about a 2.7x reduction; clarify what ratio is being reported.","section":"§5.2 and §5.3"},{"comment":"The decoding-exploitation columns labeled 'w/o sys. prompt' and 'w/ sys. prompt' are not defined in the main text; specify what system prompt was used in the 'with' condition and why it matters for ASR.","section":"Table 1"},{"comment":"The reported differences between CKU and several baselines (e.g., Safe Unlearning and Circuit Break) are often within a few tenths of a percent, yet the text claims CKU 'consistently' outperforms baselines; confidence intervals or significance tests are needed to support this claim.","section":"Tables 1-2"}],"recommendation":"reject","confidential_remarks":"The sign error in Eqs. (6)-(7) is a fundamental inconsistency in the core method description; even if the authors correct the sign, the hyperparameter selection procedure on the evaluation metrics would still need to be addressed. Without code or checkpoints, the reported results cannot be attributed to the method as described, so I do not see a path to acceptance within the current manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has a solid, interesting idea -- freeze neurons that encode useful knowledge and unlearn harmful knowledge only in the rest -- but the core training loss in Eqs. (6)-(7) has a sign error that makes the described method increase the likelihood of harmful responses rather than decrease it. The reported safety gains cannot follow from the method as written. This is the main reason I'd reject the paper in its current form.\n\nWhat's genuinely new: the combination of SNIP-based neuron importance scoring with gradient-masked unlearning. Prior unlearning defenses like Eraser or Safe Unlearning don't freeze knowledge-related neurons. The layer-selection analysis (which MLP layers matter most) is also a fresh empirical contribution. The experimental setup is thorough: two models, four jailbreak attacks, several utility benchmarks, and CKU does show the lowest ASR in most cells if the numbers are real.\n\nThe soft spots, in order of weight:\n\n1. The sign problem. Section 3.1 sets up gradient ascent on L_unlearn. Eq. (6) defines L_f as the average log-likelihood of harmful responses. Since every log-probability is negative, maximizing L_f drives it toward zero, which pushes the conditional probability of the harmful response up, not down. The regularizer in Eq. (7) doesn't fix this -- with a positive λ it just caps the increase. So the described algorithm would make the model more likely to emit harmful text, which is exactly the opposite of what Table 1 shows. Either the equations are missing a minus sign, or the experiments used a different objective than the one described. No code is provided to disambiguate.\n\n2. Hyperparameter selection. NLR=0.8 and λ=1.5 are chosen by sweeping the same safety and utility metrics that then become the headline results. That makes the 'best balance' claim weaker, though it's a common practice.\n\n3. Engineering details. No code, no seeds for the main runs (only for the identification step), and no error bars. For a defense paper where gains are often a few ASR points, this matters.\n\nIf the sign is a typo, the fix is easy and the method is probably worth publishing. But as written, the central claim is unsupported. I'd send it to review rather than desk-reject because the idea has merit and a referee could quickly verify the corrected objective. The authors need to fix the loss, release code, and report variance.\n\nRecommendation: reject in current form; invite resubmission after fixing the core objective and providing code.","headline":"The core unlearning objective has a sign error that reverses its effect, so the paper's central claim is unsupported as written, though the idea is worth a second look.","tokens_in":16960,"tokens_out":4934,"would_cite":false,"duration_ms":40526,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By freezing the top 80% of useful-knowledge neurons during unlearning, CKU claims jailbreak attacks lose most of their power while general ability stays nearly intact.","keywords":["safety alignment","knowledge unlearning","jailbreak defense","neuron importance scoring","SNIP","gradient masking","MLP layers","attack success rate"],"falsifier":"Take a harmful behavior that causal tracing places in the top-80% of neurons CKU locks, unlearn it with CKU, and run the AIM or GCG attack on that behavior; if the attack still succeeds, the frozen neurons retained the harmful association, which would be evidence that the claimed separation between useful and harmful knowledge failed.","tokens_in":15957,"feed_emoji":"🧠","tokens_out":11115,"duration_ms":85320,"temperature":0.7,"pith_summary":"The paper argues that jailbreak attacks succeed because aligned models still carry harmful knowledge, and that refusal-oriented training does not remove it. Its proposed fix, Constrained Knowledge Unlearning (CKU), scores neurons in the MLP layers for usefulness, freezes the top 80% of them during unlearning, and applies gradient ascent on harmful responses so the remaining neurons forget that content. On Llama2-7B-Chat and Llama3-8B-Instruct, CKU reports the best overall defense among the tested baselines across AIM, GCG, AutoDAN, and decoding-exploitation attacks, with the lowest attack success rate in most configurations, while average accuracy on MT-Bench and five NLP benchmarks stays within a few hundredths of a point of the base model. The authors also report that unlearning on just MLP layers 8 to 12 at a neuron locking rate of 0.8 yields a roughly fourfold safety gain at a 0.15% utility cost. If the claim holds, safety alignment can be improved by deleting specific knowledge rather than adding refusal behavior.","feed_headline":"Freezing 80% of useful neurons blocks jailbreaks, keeps accuracy","feed_subtitle":"Selective unlearning in MLP layers cuts jailbreak success rates while benchmark scores barely move.","key_machinery":"The load-bearing object is the SNIP neuron-importance score, $I(W,x)=|W \\odot \\nabla_W L(x)|$ averaged over the Alpaca identification dataset, which ranks each MLP neuron by how much it contributes to predicting useful knowledge. From those ranks, the top $p\\%$ of neurons in each layer are designated knowledge-related neurons (KRNs) and locked by setting their gradient to zero during unlearning. Unlearning is gradient ascent on harmful prompt-response pairs with a next-token log-likelihood loss, capped by $\\max(0, \\lambda + L_f)$ so the update stops once enough forgetting has occurred. The mechanism's job is to confine harmful-knowledge removal to the unfrozen 20% of neurons while the frozen 80% preserve general capability; the paper's key result is that this split at NLR 0.8 and layers 8-12 gives large safety gains with negligible utility loss.","core_discovery":"The paper's central claim is that an aligned LLM can be made substantially more resistant to jailbreaks by unlearning harmful knowledge in a constrained way: locate neurons that support useful knowledge, freeze them during unlearning, and let gradient ascent erase the harmful associations from the rest of the MLP weights. CKU operationalizes this by scoring each neuron with a first-order SNIP importance estimate averaged over the Alpaca dataset, ranking neurons within each MLP layer, and keeping the top 80% fixed by zeroing their backpropagation gradients during unlearning. The unlearning step maximizes the harmful-response log-likelihood $L_f = \\frac{1}{|D_f|}\\sum_{(x,y)\\in D_f}\\sum_{i=1}^{|y|} \\log p(y_i \\mid T(x), y_{<i})$ under the regularizer $\\max(0, \\lambda + L_f)$, so training stops once enough forgetting has occurred. The paper reports that this yields the best average safety-utility balance: attack success rates drop substantially relative to the base model and are the lowest in most configurations across two Llama chat models, while utility stays close to the original.","pith_inferences":["A direct implication the authors leave implicit: the same gradient-masking recipe could be redirected at any unwanted knowledge type, such as copyrighted text or private personal data, by swapping the identification and unlearning datasets, giving a general selective-forgetting tool.","The choice of Alpaca as the proxy for useful knowledge draws a testable boundary: if the identification data changes to a domain where harmful and useful topics overlap heavily, the frozen neuron set should include harmful knowledge and the safety gain should shrink.","Because CKU only updates the unfrozen 20% of MLP neurons, an adaptive attacker who knows the mask could concentrate adversarial suffixes in that subspace; whether CKU withstands such an attack is untested in the paper.","The authors' own limitation note says effectiveness varies across domains and that nuanced or ambiguous requests may still be handled poorly, so the honest scope of the claim is a strong safety-utility trade-off on the tested jailbreak families rather than universal robustness."],"forward_implications":["If CKU's central claim is correct, safety alignment can be achieved by targeted knowledge erasure rather than additional refusal training, which should make aligned models less vulnerable to jailbreak prompts that exploit retained harmful knowledge.","The neuron locking rate is load-bearing: the paper reports that locking the top 80% of SNIP-ranked neurons gives the largest safety improvement, while locking too few or too many disrupts the balance.","Unlearning can be concentrated in a subset of MLP layers: layers 8-12 of Llama2-7B-Chat gave a roughly fourfold safety gain at about 0.15% average accuracy loss, so full-model fine-tuning is not required for the reported gains.","Neuron selection is not interchangeable: random selection produced higher attack success rates and lower utility than SNIP ranking in the paper's comparison.","Because CKU removes harmful knowledge rather than only suppressing outputs, its reported ASR gains on AdvExtent suggest the defense may carry over to unseen harmful topics beyond the training set."],"supporting_citations":[{"why":"Supplies the SNIP connection-sensitivity scoring used to rank which MLP neurons carry useful knowledge.","marker":"(Lee et al., 2019)"},{"why":"Establishes that transformer feed-forward layers act as key-value memories, motivating the focus on MLP layers.","marker":"(Geva et al., 2021)"},{"why":"Provides the neuron-level importance aggregation used to build the knowledge-related neuron set.","marker":"(Michel et al., 2019)"},{"why":"Supplies the gradient-ascent unlearning objective CKU adapts for harmful-content removal.","marker":"(Chen and Yang, 2023)"},{"why":"Eraser is the closest prior jailbreak defense by unlearning; it provides a main baseline, the AdvExtent test set, and the regularization motivation.","marker":"(Lu et al., 2024)"},{"why":"Provides the GCG attack and the string-matching ASR evaluation procedure used in the safety experiments.","marker":"(Zou et al., 2023b)"},{"why":"Llama2-7B-Chat is one of the two base models on which CKU is evaluated.","marker":"(Touvron et al., 2023)"}],"fun_headline_variants":["Freeze 80% of useful neurons to stop jailbreaks, keep accuracy","CKU unlearns harmful knowledge while freezing 80% useful neurons","Selective neuron freezing cuts jailbreaks without hurting utility","Freeze useful neurons, unlearn harm: LLM safety up, accuracy steady"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that scoring neurons on Alpaca and freezing the top 80% separates useful knowledge from harmful knowledge, so harmful content mostly lives in the unfrozen 20% and can be removed without disturbing what the model needs.","fun_headline_variants_meta":{"raw":{"variants":["Freeze 80% of useful neurons to stop jailbreaks, keep accuracy","CKU unlearns harmful knowledge while freezing 80% useful neurons","Selective neuron freezing cuts jailbreaks without hurting utility","Freeze useful neurons, unlearn harm: LLM safety up, accuracy steady"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000945,"raw_usage":{"total_tokens":4032,"prompt_tokens":941,"completion_tokens":3091,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":3013}},"tokens_in":557,"tokens_out":3091,"duration_ms":18696,"temperature":1.0,"reasoning_tokens":3013,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:28:43.117621+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a harmful behavior that causal tracing places in the top-80% of neurons CKU locks, unlearn it with CKU, and run the AIM or GCG attack on that behavior; if the attack still succeeds, the frozen neurons retained the harmful association, which would be evidence that the claimed separation between useful and harmful knowledge failed.","supporting_citations":[],"review_version":1}