{"id":"fd4ae076-489f-4285-ae9b-acf8b19372ef","arxiv_id":"2505.23556","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Refusal in Gemma-2-2B and Llama-3.1-8B is mediated by a small set of SAE features, harm features causally activate refusal features, and adversarial jailbreaks suppress those refusal features.","lead":"This paper finds the sparse internal features that make two aligned chat models refuse harmful requests. It shows that harm and refusal are stored as separate feature sets, that refusal features sit downstream of harm features, and that jailbreaks work by suppressing the refusal features.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The AS-substitution in Alg. 1 / Appendix A.1 is unvalidated: replacing the true corrupted input with activation-steered clean activations turns attribution patching into attribution of the steering intervention itself, so F* and the harm-to-refusal separation built on it may be artifacts of that…","rationale":"I agree with the reader's weakest-assumption diagnosis, and I would not move the verdict. The paper has genuine independent support: open-sourced code, two models, behavioral validation of the selected features, and honest limitations. The AS substitution is not internally inconsistent, but it is an unvalidated approximation at the root of the pipeline. Because F0 is constrained to be refusal-direction-aligned and the substituted corrupted state is exactly the refusal-direction-ablated clean state, the attribution in Eq. 9 is close to measuring the sensitivity of the steering intervention rather than the natural causal effect of harmful content on refusal. The selection of F* therefore cannot, by itself, support the strong claim that harm and refusal are separately encoded with a specific FH-to-FR dependency; that claim requires the substitution to be faithful or the feature sets to be re-derived from true activation patching. The proposed concrete test would settle this with modest compute. If it passes, the central claims are substantially strengthened; if it fails, the paper's mechanistic conclusions need reframing as properties of the steering intervention rather than of natural refusal behavior. Hence the reader's CONDITIONAL verdict remains appropriate and unchanged.","tokens_in":17890,"tokens_out":9923,"duration_ms":100498,"concrete_test":"Take 100 harmful prompts from HarmBench/CATQA and construct length-matched benign counterparts by replacing harmful nouns/verbs with benign ones while keeping token count fixed. For each pair, run brute-force activation patching: replace z_clean with z_corrupt from the benign input, keep y_corrupt as the first token of the benign completion, and compute IE scores via Eq. 9 with the same N=10 budget. Compare this F*_true with F*_AS from Alg. 1 on three quantities: (i) Jaccard overlap of the top-20 feature sets; (ii) Spearman rank correlation of the IE scores; (iii) jailbreak rate after negatively scaling each set. If overlap is below 50% or rank correlation below roughly 0.5, the AS substitution changes the selected mediators and the central claims should be treated as conditional on the steering intervention rather than validated for natural harmful inputs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on F*, the feature set selected by CosSim+AP (Alg. 1). In Appendix A.1, the paper replaces the standard corrupted activations z_corrupt|d_corrupt with z_AS(d_clean), i.e. the activations obtained by applying activation steering to the clean harmful input, and defines y_corrupt as the first output token of AS(d_clean). It asserts that this 'effectively replicates the patching effects' but gives no ablation. This matters because F0 is already the top K0 features per layer closest to the refusal direction V_R (Eq. 5), and z_AS(d_clean) is obtained by projecting out V_R from z_clean. Hence the difference (z_corrupt - z_clean) used in Eq. 9 lies almost entirely along V_R, and the integrated-gradient score measures each feature's contribution to reproducing the steering intervention, not the natural harmful-vs-harmless counterfactual that AP is designed to estimate. If the steering intervention is not causally equivalent to a corrupted harmless input, F* may be a set of features that are merely most responsive to steering. The downstream conclusions in Sec. 4.2 (harm and refusal as separate feature sets with harm causally upstream of refusal) and Sec. 4.3 (jailbreaks suppress refusal features) inherit this dependency. The behavioral check that clamping F* induces jailbreak shows some causal relevance, but it does not establish minimality or the specific FH-to-FR architecture. An independent validation of the substitution on length-matched inputs is therefore the load-bearing missing step.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies refusal in Gemma-2-2B and Llama-3.1-8B using sparse autoencoders. It proposes a hybrid method, CosSim+AP, which restricts attribution patching to the K0 features closest to the refusal direction and then selects the top K* features by integrated-gradient importance. The authors validate the selected features by clamping their activations and measuring HarmBench jailbreak scores, and they report controls on coherence and reasoning. They then claim that harm and refusal are encoded by separate feature sets (F_H upstream of F_R), that adversarial paraphrases and suffixes suppress F_R, and that linear probes trained on A(F_R) generalize better to out-of-distribution adversarial instructions than dense probes. Code is released.","tokens_in":18261,"tokens_out":6867,"duration_ms":65390,"significance":"If the causal claims hold, this would be a valuable step toward a sparse, intervention-validated account of refusal, with practical payoff for OOD safety classification. The paper's strengths include the use of two model families, publicly available SAEs, behavioral intervention checks, coherence/reasoning evaluation, and an open-source implementation. The main limitation is that the central attribution step—using activation-steered activations as a stand-in for a corrupted input—is not validated, and several support analyses are correlational or lack matched controls. Consequently, the headline separation of harm and refusal and the jailbreak mechanism are currently plausible but not established.","major_comments":[{"comment":"The replacement of corrupted activations by activation-steered clean activations is the load-bearing step of the method. Since F0 is defined by cosine similarity to V*_R (Eq. 5) and z_corrupt is obtained by projecting V*_R out of z_clean, the difference (z_corrupt - z_clean) in the integrated-gradient term is dominated by the steering direction. The importance scores then measure how well each candidate feature reproduces the steering intervention, not the harmful-vs-harmless counterfactual that attribution patching targets, and the paper's assertion that this 'effectively replicates the patching effects' is not accompanied by an ablation. Because F* feeds every downstream claim in Sections 4.2 and 4.3, this is not a presentation issue. I request a validation experiment comparing CosSim+AP with standard attribution patching on length-matched harmful/harmless input pairs, reporting feature-set overlap and the resulting jailbreak faithfulness after clamping; if the approximation fails, the causal-separation and jailbreak-mechanism conclusions need to be rederived.","section":"Appendix A.1 and Eq. (9) (Algorithm 1)"},{"comment":"The labels F_R and F_H are assigned by cross-category frequency, not by an independent causal criterion. The observation that clamping F_common produces larger jailbreak rates is compatible with F_common features having higher average activation, occupying more influential layers, or differing in feature count; the same-category comparison of F_specific,j vs F_common does not control for these quantities. The random-feature control in Table 1 is 100 times larger than F_H, which is not a matched baseline. Please add controls for feature count, layer distribution, and activation magnitudes, use a matched-size random set, and report per-feature effects with variance; without this, the claim that LLMs 'distinctly encode harm and refusal as separate feature sets' is not quantitatively supported.","section":"Sec. 4.2 and Table 1"},{"comment":"The conclusion that adversarial jailbreaks operate by suppressing refusal features is inferred from activation differences (Tab. 2, Fig. 4) and is therefore correlational at the level of the latent variables. To establish that suppression of F_R is the causal mechanism, the paper should intervene during generation: for example, clamping or holding F_R fixed while appending the suffix or paraphrase and measuring whether the jailbreak effect is blocked, or scaling F_R down without the adversarial input and showing that it reproduces the jailbreak. Without such a mediation experiment, the mechanism is a plausible conjecture but not a demonstrated finding.","section":"Sec. 4.3"}],"minor_comments":[{"comment":"The notation 'arg top' is nonstandard; please define it precisely, for example as top-K selection by descending score with ties broken deterministically.","section":"Eq. (6) and Algorithm 1"},{"comment":"The Random row contains the malformed entries '0.5 0. 1.0 1.0'; please fix the formatting in the Average and Vanilla columns.","section":"Table 4"},{"comment":"Please clarify the status of y_corrupt: is it the first token sampled from AS(d_clean), and is Eq. (9) applied to log-probabilities of a single token? The multi-token nature of refusal makes this choice material.","section":"Appendix A.1"},{"comment":"The entries for Kissane et al. (2024a/b) and Lee et al. (2025) are listed as 'Accessed: ...' without URLs; please complete the bibliographic information.","section":"References"},{"comment":"No confidence intervals or standard errors are reported for the jailbreak scores; given the small feature sets (7-10 features), some measure of variance would help the reader assess stability.","section":"Figs. 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the AS-substitution is, in my reading, the central issue and is not addressed by the paper's existing ablations. The paper has many strengths—two model families, public SAEs, behavioral validation, and coherence/reasoning checks—but the headline causal claims should not be accepted until the approximation in Appendix A.1 is validated against standard attribution patching or removed. I also recommend asking for matched random-feature controls and a direct mediation experiment for the jailbreak mechanism."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for the note. I read the paper and mostly share your take, with maybe a bit more credit on the empirical side and equal concern about the appendix substitution.\n\nWhat's new: the CosSim+AP method, which restricts attribution patching to features already aligned with the refusal direction, then uses integrated gradients to pick a small set per sample. Applied to Gemma-2-2B and Llama-3.1-8B with open SAEs, it finds compact feature sets (7-10 features) that, when clamped, induce jailbreak behavior with modest coherence and reasoning cost. The paper then uses those sets to argue that harm and refusal are separate feature groups, that harm features sit upstream of refusal features, that jailbreak suffixes suppress refusal features, and that a probe trained on refusal features generalizes better to OOD adversarial instructions. Code is open. That is a genuinely useful package, and the intervention results are suggestive.\n\nNow the soft spots. The load-bearing issue is the AS-substitution in Appendix A.1. The paper replaces the corrupted input activations with activation-steered activations from the clean input and asserts this 'effectively replicates the patching effects,' but gives no ablation. Since the candidate pool F0 is already top-K by cosine similarity to the refusal direction, and steering projects out that same direction, the integrated-gradient scores largely measure how well each feature reproduces the steering intervention, not the natural harmful-vs-harmless counterfactual. The behavioral check that clamping F* causes jailbreaks shows the features are causally relevant, but it does not validate the substitution or the specific FH-to-FR architecture. That matters because the Sec. 4.2 'clear causal effect' claim rests on that architecture. This is fixable: run the same selection with length-matched harmful/harmless pairs, or at least show that AS(d_clean) and corrupted activations are close in distribution for the features considered.\n\nOther soft spots: no error bars in the key tables; the OOD probe evaluation only uses jailbroken adversarial samples, which biases the comparison in favor of the sparse probe; and the common/specific feature split relies on a small K* with no sensitivity analysis. The limitations section is honest about some of this, and the appendix has useful detail on metric failures and SAE transfer. The citation pattern is fine.\n\nBottom line: the paper deserves a serious referee. The method is novel enough and the results plausible enough that the unvalidated substitution should be the main request, not a desk reject. If they tighten that, it becomes a solid contribution.","headline":"A solid, novel causal-feature study of refusal with a fixable but load-bearing validation gap in the attribution-patching substitution.","tokens_in":18765,"tokens_out":3618,"would_cite":true,"duration_ms":30126,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In two open LLMs, refusal is carried by a small set of sparse autoencoder features that sit downstream of separately encoded harm features.","keywords":["sparse autoencoders","mechanistic interpretability","refusal behavior","jailbreak","attribution patching","activation steering","harm features","out-of-distribution probing"],"falsifier":"Recompute the top feature set F* on a sample of prompts using genuine harmful/harmless input pairs, so attribution patching uses actual corrupted activations, and compare the recovered features and jailbreak behavior with the steered-activation results; if the sets differ materially, the substitution is the source of the reported features. Independently re-label a sample of jailbreak outputs with human judges to check that the classifier scores match.","tokens_in":17703,"feed_emoji":"🛡️","tokens_out":8768,"duration_ms":90220,"temperature":0.7,"pith_summary":"This paper claims that refusal in instruction-tuned language models is not a diffuse property but is carried by a small, identifiable set of latent features, and that these features can be found and causally tested. Working on Gemma-2-2B and Llama-3.1-8B with sparse autoencoders, the authors isolate a minimal set F_R (7 features for Gemma, 10 for Llama) that directly mediates refusal, alongside a separate set F_H that encodes harmful content and acts as an upstream trigger. They report that intervening on F_H suppresses F_R and lowers the probability of refusal, that successful jailbreaks—both paraphrased and suffix-based—reduce F_R activation, and that a probe trained only on F_R generalizes to out-of-distribution adversarial prompts better than a dense probe. If correct, this gives a concrete, intervention-tested picture of where safety behavior lives in the model and how attacks disable it.","feed_headline":"LLM refusal is driven by a tiny set of causal features","feed_subtitle":"Sparse-autoencoder study shows jailbreaks suppress these refusal features rather than adding harmful ones.","key_machinery":"The machinery is a two-stage feature search the authors call CosSim+AP, applied to sparse autoencoder (SAE) features. SAEs decompose each residual-stream activation into a sparse set of feature directions; the paper first filters candidate features to those with highest cosine similarity to a difference-in-means refusal direction, then ranks them by attribution patching with integrated gradients to obtain a minimal per-sample set F*. Comparing F* across harm categories separates the shared refusal set F_R from behavior-specific harm features F_H. The causal test is activation clamping: scaling a feature set's activations by a constant and observing whether jailbreak or refusal behavior follows.","core_discovery":"The paper's central claim is that LLMs encode harm and refusal as separate feature sets, with harmful features exerting a clear causal effect on refusal features. Concretely, the authors define a small shared feature set F_R that is directly responsible for refusing, and category-specific harm features F_H that sit upstream: when F_H is suppressed, F_R activation drops and the model stops refusing. Adversarial jailbreaks are then explained as suppression of F_R: both benign paraphrases of harmful prompts and optimized suffix tokens reduce F_R activation, and the reduction is larger on prompts where the jailbreak succeeds. Finally, a linear probe trained only on F_R activations, rather than dense activations, is the signal that keeps working on adversarial out-of-distribution instructions.","pith_inferences":["A testable extension: training sparse autoencoders on chat-model activations rather than base-model activations could yield an even smaller and more causal refusal set; the paper's transfer experiments suggest the improvement would be modest but measurable.","If the harm-to-refusal chain is real, raising the feature budget K* could expose intermediate stages (for example, legality or consequence features) between raw harmful concepts and the final refusal response.","The steered-activation substitution, if it holds up, would make attribution patching practical for long, unstructured prompts generally, not just short templated input pairs.","The out-of-distribution probing result implies a practical recipe for robust safety classifiers: train on a small interpretable feature set, provided a faithful SAE for the target model exists."],"forward_implications":["Safety interventions could target the small F_R set directly instead of steering entire activation layers, reducing collateral damage to reasoning and coherence.","Monitoring F_R activation could serve as an early signal of an in-progress jailbreak attempt, since successful attacks lower it.","Because harm and refusal are separate feature sets, a model can recognize harmful content without refusing; safety depends on the downstream refusal features firing.","Classifiers built on refusal-feature activations rather than dense representations should transfer better to unseen jailbreak styles.","The same minimal-feature pipeline can be applied to other safety-relevant behaviors such as sycophancy or bias with the same intervention-testing protocol."],"supporting_citations":[{"why":"Supplies the refusal direction and activation-steering baseline that the feature search is designed to approximate, plus the initial refusal measurement protocol.","marker":"Arditi et al. (2024)"},{"why":"Extends attribution patching to SAE features with integrated gradients, the ranking technique adapted here.","marker":"Marks et al. (2025)"},{"why":"Provides the attribution patching method and its linear approximation used to score feature importance.","marker":"Syed et al. (2024)"},{"why":"Establishes the prior result that upstream SAE features causally activate the refusal direction, which the paper refines into separate harm and refusal sets.","marker":"Lee et al. (2025)"},{"why":"Provides the open Gemma sparse autoencoder whose features carry the Gemma half of the experiments.","marker":"Lieberum et al. (2024)"},{"why":"Provides the open Llama sparse autoencoder whose features carry the Llama half of the experiments.","marker":"He et al. (2024)"},{"why":"Supplies the jailbreak classifier used as the metric for whether an output complies with a harmful instruction, along with one of the harmful datasets.","marker":"Mazeika et al. (2024)"},{"why":"Supplies the adversarial paraphrase dataset used to study how benign rephrasings suppress refusal features.","marker":"Jiang et al. (2024)"},{"why":"Supplies the adversarial suffix dataset used to study token-level suppression of refusal features.","marker":"Basani and Zhang (2024)"},{"why":"Supplies the behavior categories used to separate common refusal features from behavior-specific harm features.","marker":"Bhardwaj et al. (2024)"}],"fun_headline_variants":["Sparse features explain LLM refusal and jailbreak bypass","Refusal features are few, and jailbreaks suppress them","A handful of features gates LLM refusal","Jailbreaks work by muting a tiny set of refusal features","Sparse autoencoders reveal the causal core of refusal"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that substituting activation-steered activations for genuinely corrupted activations in attribution patching faithfully reproduces the counterfactual state, and that the jailbreak classifier correctly labels jailbreaks; if either gives way, the claimed causal feature sets could be artifacts of the measurement.","fun_headline_variants_meta":{"raw":{"variants":["Sparse features explain LLM refusal and jailbreak bypass","Refusal features are few, and jailbreaks suppress them","A handful of features gates LLM refusal","Jailbreaks work by muting a tiny set of refusal features","Sparse autoencoders reveal the causal core of refusal"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000599,"raw_usage":{"total_tokens":2744,"prompt_tokens":830,"completion_tokens":1914,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":446,"completion_tokens_details":{"reasoning_tokens":1834}},"tokens_in":446,"tokens_out":1914,"duration_ms":13889,"temperature":1.0,"reasoning_tokens":1834,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:43:18.405829+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the top feature set F* on a sample of prompts using genuine harmful/harmless input pairs, so attribution patching uses actual corrupted activations, and compare the recovered features and jailbreak behavior with the steered-activation results; if the sets differ materially, the substitution is the source of the reported features. Independently re-label a sample of jailbreak outputs with human judges to check that the classifier scores match.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the prior result that upstream SAE features causally activate the refusal direction, which the paper refines into separate harm and refusal sets."}],"review_version":1}