{"id":"ea989f53-8b2e-45f8-925e-867ba8c1e724","arxiv_id":"2506.07596","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TwinBreak removes safety alignment by pruning the parameters whose activations differ most between paired harmful and harmless prompts, achieving high attack success on many open-source LLMs.","lead":"This paper introduces TwinBreak, a method that disables the safety guardrails of open-source language models by finding and cutting a small set of internal parameters that react strongly to harmful prompts. It reports high jailbreak success across 16 models from five vendors using only minutes of compute and little loss of normal performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing random-pruning control: the claim that TwinBreak isolates safety-specific neurons is untested against equivalent random ablation in the same layers, so the safety-specificity conclusion is not yet established.","rationale":"The reader's conditional verdict is appropriate, and the stress-test confirms the weakest point the reader flagged: the activation-difference selection is never tested against a null model of arbitrary pruning in the same layers. The paper is well-executed in other respects, with extensive ablations, cross-model generalization across 16 LLMs, and evidence that the 50-token switch to the unpruned model is not the source of the attack (Tab. 6, line 7). Still, the central claim of safety-specific parameter localization requires a baseline showing that the specific neurons chosen by activation differences are doing the work. Tab. 6 ablates which layers and how many neurons, but never 'which neurons' by random selection. This matters because refusal behavior is known to be fragile; the directional-ablation comparison [4] shows that a single direction can mediate refusal, so moderate pruning in the MLP could plausibly disable refusal while leaving broad benchmark accuracy intact. The no-utility-exclusion ablation (ASR 0%, incoherent output) shows the attack is not purely arbitrary, but it does not separate safety-critical neurons from generally influential ones. I therefore keep the reader's verdict as CONDITIONAL, with the added condition that the authors run a random or magnitude-based pruning control. No other concern outweighs this one.","tokens_in":37524,"tokens_out":6587,"duration_ms":85513,"concrete_test":"On LLaMA-2 7B, run TwinBreak's exact five-iteration pruning schedule and 50-token pruned prefix, but replace the activation-difference-based neuron ranking in each Gate/Up layer with (a) random selection and (b) selection of the lowest-magnitude output neurons. Report ASR on HarmBench, AdvBench, and StrongREJECT plus the five utility benchmarks. If either baseline reaches within roughly 5 points of TwinBreak's ASR at comparable utility, the safety-specificity claim fails; if both baselines yield much lower ASR or much worse utility, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that activation differences between twin prompts isolate parameters specific to the safety mechanism (Sec. 3.3, Alg. 1). The ablations in Tab. 6 vary layer targets, prompt similarity, token aggregation, and pruning rates, but they never compare TwinBreak's neuron selection against random selection of the same number of neurons in the same Gate/Up layers. Without this control, the high ASR could be explained by a much weaker hypothesis: any moderate perturbation of MLP neurons in mid-decoder layers disrupts the fragile refusal behavior, while average accuracy on five benchmarks remains roughly intact because refusal is a single, sensitive behavior. The utility exclusion result (line 2 of Tab. 6) shows that pruning the top activation-difference neurons without the 0.1% utility exclusion destroys coherence, so selection matters for keeping the model fluent; but it does not show the selected neurons are safety-specific rather than merely influential for general computation. The absence of a random or magnitude-based baseline is the most load-bearing gap: it separates 'safety is a sparse, localized component' from 'moderate unstructured pruning of MLP layers breaks safety before it breaks utility.' The paper's own comparison with directional ablation [4] shows refusal can be mediated by a single direction, so an unstructured-perturbation explanation is not vacuous.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces TwinBreak, a white-box jailbreaking method that removes safety alignment from decoder-only LLMs by pruning parameters identified through activation differences between harmful prompts and manually crafted harmless 'twin' prompts. The method iteratively prunes the top 1% of the most divergent parameters in MLP Gate and Up layers (excluding the first and last decoder blocks), while reserving a small set of 'utility' parameters from pruning, then generates the first 50 tokens with the pruned model before switching to the unpruned model. The authors present the TwinPrompt dataset of 100 twin pairs and report high attack success rates (89-98% in the abstract) with minimal utility loss across 16 LLMs from five vendors, comparing favorably to directional ablation and set-difference baselines.","tokens_in":37740,"tokens_out":7510,"duration_ms":80622,"significance":"If the central claim is correct, TwinBreak would be a notable advance: it is computationally lightweight (a few minutes on a 7B model), uses a small one-time dataset, and appears to generalize across model families and scales. The paper's strengths include a broad empirical study (16 models, four evaluation datasets, five utility benchmarks), a detailed hyperparameter ablation (Tab. 6 and Tab. 20), and a new dataset (TwinPrompt) that could be reused by the community. The comparison with two close white-box baselines is informative. However, the safety-specificity claim needs a random-pruning control, and several headline claims—89-98% ASR, hyperparameter insensitivity, and minimal utility loss—are not fully supported by the reported evidence as written.","major_comments":[{"comment":"The abstract's claim of '89% to 98% success rates across 16 LLMs' is not directly supported by the reported data. Tab. 8 reports StrongREJECT mean scores between 0.674 and 0.814 for the 16 models, not binary ASR values; the binary ASR tables (Tabs. 2-5) cover only four models. Labeling the StrongREJECT mean score as 'ASR' in Tab. 8 conflates two different metrics. The authors should either report binary ASR for all 16 models or revise the abstract and Table to accurately reflect the metric used.","section":"Abstract, Sec. 4.6, Tab. 8"},{"comment":"The central claim that twin-prompt activation differences isolate safety-specific parameters is not tested against a random-pruning baseline. The ablations in Tab. 6 vary targeted layers, prompt similarity, token aggregation, and pruning rates, but no experiment compares TwinBreak's parameter selection to random selection of the same number of parameters in the same Gate/Up layers. Without this control, the high ASR could be explained by the general fragility of refusal behavior to moderate MLP perturbation. Adding a random-pruning control (same fraction, layers, and iterations) is essential to support the specificity claim.","section":"Sec. 3.3, Alg. 1, Tab. 6"},{"comment":"The claim that TwinBreak is 'independent of any sensitive hyperparameters' is contradicted by the appendix. Tab. 9's note reports that pruning rates of 0.001 (LLaMA 3.1) and 0.002 (Qwen 2.5) were used instead of the default 0.01, and utility retention had to be raised to 1% for LLaMA 2 with dataset sizes 60 and 70. These per-model adjustments indicate hyperparameter sensitivity; the authors should either justify these as minor tuning or weaken the claim accordingly.","section":"Sec. 7.3, Tab. 9, Sec. 4.3"},{"comment":"The claim of 'minimal utility loss' is overstated. While average degradation over five benchmarks is modest, Tab. 24 shows substantial per-benchmark drops, e.g., RTE -14.4% for LLaMA 2 70B, -14.8% for Gemma 2 2B, and -21.6% for Qwen 2.5 3B. The paper should discuss worst-case per-benchmark degradation and clarify that the utility preservation in the attack is achieved by switching to the unpruned model after 50 tokens, which limits the degradation to the initial generation phase.","section":"Sec. 4.2, Fig. 4, Tab. 24"},{"comment":"The default attack uses the pruned model only for the first 50 output tokens before switching to the unpruned model. The paper should clarify that TwinBreak does not permanently remove safety alignment unless the pruned model is used for the full response, and should report full-response ASRs for all 16 models (currently only Qwen 32B and LLaMA 3.3 70B are shown in Sec. 4.3, and Tab. 21 covers only four models).","section":"Sec. 4.3, n_out_pr, Tab. 21"}],"minor_comments":[{"comment":"The word 'unpuned' should be 'unpruned'.","section":"Sec. 4.2"},{"comment":"The word 'producting' should be 'producing'.","section":"Sec. 3.4"},{"comment":"The word 'bechmarks' should be 'benchmarks'.","section":"App. 9.5"},{"comment":"The header 'RuntimeTr Val' is unclear; it should be split into 'Runtime' and 'Val' columns.","section":"Sec. 4.5, Tab. 7"},{"comment":"The conclusion states 'across four LLMs', which is inconsistent with the abstract's '16 LLMs'; the wording should be aligned.","section":"Sec. 6"},{"comment":"The term 'twin' is used throughout but never formally defined; a short definition in Sec. 3.3 would improve clarity.","section":"Sec. 3.3"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical study with a useful new dataset and a practical attack, but the paper's central scientific claim requires a random-pruning control. The headline claims (89-98% ASR, hyperparameter insensitivity, minimal utility loss) are currently overstated relative to the reported tables. With the missing control and the presentation fixes, the paper could become a strong contribution suitable for the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: TwinBreak is a credible, carefully evaluated white-box jailbreak that pairs harmful prompts with structurally and content-similar harmless twins, then iteratively prunes MLP Gate/Up parameters in middle decoder blocks. It achieves high ASRs across a wide model zoo with modest utility loss. The contribution over Arditi et al. and Wei et al. is real but incremental: the twin-prompt dataset, the targeted layer restriction, the utility-parameter exclusion, and iterative pruning. The comparisons to directional ablation and set difference show a better ASR/utility/runtime trade-off, and the full-pruned inference experiments rule out the 50-token-switch as a confound. Held-out generalization to AdvBench, JailbreakBench, StrongREJECT, and the HarmBench split is properly done, and the ablation study is unusually thorough.\n\nThe soft spots, in order of importance. First, there is no random-pruning or magnitude-based baseline in the same layers. The paper claims the selected neurons are safety-specific, but without pruning the same fraction of random Gate/Up parameters you cannot rule out that any moderate mid-MLP perturbation breaks refusal before breaking utility. The existing ablations (Down layer, attention, non-twin prompts) show selection matters for coherence, especially the no-utility-exclusion run, but they do not establish that the specific set is uniquely safety-related. I would ask for that control in Tab. 6. Second, the abstract's \"89% to 98% across 16 LLMs\" is not backed by the 16-model table, which reports StrongREJECT mean scores (0.674–0.814), not binary ASR; the binary ASR tables cover only four models. That is an overstatement. Third, the per-model pruning rates (0.001 for LLaMA 3.1, 0.002 for Qwen) weaken the strong claim of hyperparameter independence; it should be qualified. Fourth, no repeated runs or confidence intervals—less concerning given deterministic pruning, but variance would help. Code and dataset are promised, not shipped; fine for review, but it matters for reproduction.\n\nThe citation pattern is honest, and the \"first\" claims are about the specific combination, which is fair given prior work. This is a useful, competent attack paper that deserves serious refereeing. The missing controls and abstract overreach are fixable in revision.","headline":"TwinBreak is a solid, well-ablated white-box jailbreak with a genuine new twist (twin prompts) and broad evaluation, but the safety-specificity claim needs a random-pruning control and the abstract overstates the 16-model numbers.","tokens_in":38347,"tokens_out":1951,"would_cite":true,"duration_ms":25904,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that safety alignment in open-weight LLMs is a sparse, pruneable component: comparing a harmful prompt with a nearly identical harmless twin isolates the safety parameters, and pruning them alone disables refusal…","keywords":["LLM jailbreaking","safety alignment removal","parameter pruning","activation differences","twin prompts","white-box attack","RLHF robustness","large language models"],"falsifier":"Prune exactly the same fraction, locations, and iterations as TwinBreak but choose the parameters uniformly at random instead of by activation difference, and measure attack success on HarmBench with the paper's evaluation setup. If random pruning of roughly 5% of Gate and Up parameters in the middle decoder blocks reaches the same 89 to 98 percent attack success rate, then the twin-prompt ranking adds nothing and the safety-sparsity claim fails.","tokens_in":37259,"feed_emoji":"🔓","tokens_out":9127,"duration_ms":105283,"temperature":0.7,"pith_summary":"This paper claims that the safety alignment of a decoder-only LLM is not a distributed property but a sparse set of parameters that can be found and removed. The method, TwinBreak, pairs each harmful prompt with a harmless twin that is nearly identical in wording and content, records the internal activations both prompts produce, and prunes the parameters whose activations diverge most. Five rounds of pruning, each cutting the top 1% of divergent parameters in the MLP Gate and Up layers while skipping the first and last decoder blocks, reportedly disables the refusal mechanism across 16 open-weight models from five vendors, with attack success rates of 89 to 98 percent and only a few percentage points of utility loss. A sympathetic reader would care because the result suggests that a low-resource attacker with no training data, no gradients, and a few minutes of compute can permanently strip safety alignment from a released model. If the claim is right, safety alignment in open-weight LLMs is a fragile, removable component rather than something woven through the whole network.","feed_headline":"Five percent of a model's neurons can switch off its safety guard","feed_subtitle":"Pairing harmful prompts with harmless twins finds the safety circuitry; 89 to 98 percent of attack attempts succeed.","key_machinery":"The central object is the twin prompt: a harmful prompt paired with a harmless prompt matched in grammar and content. The carrying mechanism is the activation-difference ranking, which feeds each padded pair through the model for exactly one output-token generation, takes the last six input-token positions, sorts them by the L2 norm of the activation difference, averages the top five, and ranks parameters by this score. The same procedure over harmless-to-harmless pairs marks the top 0.1% of parameters as utility parameters that must not be pruned. Five iterations then prune the top 1% of remaining safety-ranked parameters in the Gate and Up projections (the feed-forward projections that gate and expand hidden features) of every decoder block except the first and last, and the jailbroken response is generated with the pruned model for the first 50 tokens before switching back to the unpruned model.","core_discovery":"On the paper's own terms, TwinBreak treats the LLM's refusal behavior as an embedded backdoor. For each of the 100 harmful prompts in the TwinPrompt dataset, the authors manually craft a harmless twin that keeps the same grammar and topic, so the only meaningful difference is whether the safety mechanism should fire. While the model generates a single output token, activations are collected in the MLP Gate and Up layers of the middle decoder blocks; ranking parameters by the absolute activation difference between harmful and harmless twins isolates the safety-critical subset, and the top 1% is pruned per round for five rounds, after excluding the top 0.1% of parameters flagged as utility-critical by harmless-to-harmless comparisons. The experiments measure attack success with a safety classifier across evaluation datasets and report that the pruned models answer 89 to 98 percent of harmful prompts while utility benchmarks drop by about one to five percentage points on average; the pruning phase itself takes about three to five minutes on a 7B model.","pith_inferences":["Beyond the paper: the same activation-difference probe could be used in reverse, to transplant safety parameters back into a fine-tuned model or to certify whether a released model still contains its safety circuitry.","Beyond the paper: if RLHF learned safety as a sparse set of features, then other behavior-specific mechanisms, such as sycophancy, hallucination triggers, or refusal exceptions, might be isolable by the same twin-prompt construction.","Beyond the paper: the 50-token-pruned-then-unpruned inference protocol suggests that only the first tokens of a response bear the refusal decision; a testable extension would be whether masking safety parameters only during the first few tokens reproduces the full attack success rate.","Beyond the paper: a defense could randomize or replicate safety parameters across layers so that no small prunable subset exists; the paper itself suggests entangled safety as future work but does not evaluate it."],"forward_implications":["If the central claim is correct, safety alignment in open-weight decoder-only LLMs is concentrated in a small, identifiable subset of parameters rather than distributed through the whole model.","An attacker with read/write access to a released model can permanently disable its refusal mechanism in minutes, without training data, gradients, or expensive GPU clusters.","Because the same pruned model answers unseen harmful prompts from several benchmark families, the removal transfers across prompts and across model vendors and sizes from 1B to 72B parameters.","The twin-prompt structure is load-bearing: replacing twin pairs with unrelated harmless prompts lowers the reported attack success rate and increases utility degradation, and direct comparisons show TwinBreak outperforms both prior activation-ablation and set-difference pruning methods."],"supporting_citations":[{"why":"Supplies the HarmBench harmful prompts from which TwinPrompt's 100 twin pairs are built and the validation split is drawn.","marker":"[39]"},{"why":"AdvBench serves as an unseen validation dataset for testing how well the pruned model generalizes to new harmful prompts.","marker":"[75]"},{"why":"StrongREJECT is used as a validation benchmark and its fine-tuned evaluator scores the quality and effectiveness of jailbroken responses.","marker":"[57]"},{"why":"Directional ablation is the closest baseline that manipulates activations from harmful and harmless prompts and is compared against throughout the evaluation.","marker":"[4]"},{"why":"Set difference is the closest pruning-based baseline, computing safety-versus-utility parameter scores that TwinBreak outperforms.","marker":"[67]"},{"why":"BadNets supplies the backdoor analogy: a trigger pattern can induce a specific misbehavior in a network, which motivates locating safety parameters.","marker":"[27]"},{"why":"Neural Cleanse motivates targeted pruning of backdoor-critical parameters, the strategy TwinBreak transfers to LLM safety alignment.","marker":"[65]"},{"why":"LlamaGuard3 is the safety classifier used to label generated responses as unsafe or safe when computing attack success rates.","marker":"[45]"}],"fun_headline_variants":["Twin prompts reveal and prune a model's safety neurons","Pairing harmful and harmless prompts locates 1% safety-critical weights","Removing 1% of neurons disables LLM safety in 89-98% of attacks","TwinBreak: pruning 1% safety neurons yields 89-98% jailbreak success"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing bet is that the activation gap between a harmful prompt and its near-identical harmless twin, measured during a single output-token generation over the top five of the last six input positions, points exactly at the parameters that enforce refusals.","fun_headline_variants_meta":{"raw":{"variants":["Twin prompts reveal and prune a model's safety neurons","Pairing harmful and harmless prompts locates 1% safety-critical weights","Removing 1% of neurons disables LLM safety in 89-98% of attacks","TwinBreak: pruning 1% safety neurons yields 89-98% jailbreak success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000401,"raw_usage":{"total_tokens":2111,"prompt_tokens":982,"completion_tokens":1129,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":1042}},"tokens_in":598,"tokens_out":1129,"duration_ms":11492,"temperature":1.0,"reasoning_tokens":1042,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:30:43.228168+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Prune exactly the same fraction, locations, and iterations as TwinBreak but choose the parameters uniformly at random instead of by activation difference, and measure attack success on HarmBench with the paper's evaluation setup. If random pruning of roughly 5% of Gate and Up parameters in the middle decoder blocks reaches the same 89 to 98 percent attack success rate, then the twin-prompt ranking adds nothing and the safety-sparsity claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"StrongREJECT is used as a validation benchmark and its fine-tuned evaluator scores the quality and effectiveness of jailbroken responses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Directional ablation is the closest baseline that manipulates activations from harmful and harmless prompts and is compared against throughout the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Set difference is the closest pruning-based baseline, computing safety-versus-utility parameter scores that TwinBreak outperforms."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Neural Cleanse motivates targeted pruning of backdoor-critical parameters, the strategy TwinBreak transfers to LLM safety alignment."},{"cited_title":"Llama guard 3 8b","cited_arxiv_id":null,"evidence_quote":"LlamaGuard3 is the safety classifier used to label generated responses as unsafe or safe when computing attack success rates."}],"review_version":1}