{"id":"243f90b7-e3c0-423c-86c9-07ddf4673148","arxiv_id":"2506.17052","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Top-K attention heads with the highest cosine similarity to a concept vector form a sparse module whose scalar scaling can strengthen or suppress that concept in LLMs and vision transformers.","lead":"Researchers propose a simple way to find which attention heads in a transformer encode a chosen concept, such as 'French', 'safety', or a cat class, by comparing each head's output with the concept's vector and taking the top few heads. Scaling those heads up or down with one scalar then weakens or amplifies the concept, enabling effects like jailbreaking a chatbot or suppressing an image classifier's target label.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No control modules: SAMI effects are never compared with random-head or bottom-K interventions under the same scalar protocol, so topK selection is not yet shown to be causal.","rationale":"The paper has real strengths: the French module qualitatively switches response language, the ViT scalar sweep drives target-label error to zero while other labels remain largely intact, and the SAE-based modules are visualized with TransformerLens and ViT-Prisma for reproducibility. These make the method plausible. The load-bearing weak point is not the cosine-similarity metric per se but the absence of any counterfactual control in the intervention experiments. Because SAMI is the only causal evidence, the topK selection and the observed effect could be conflated with generic head scaling. The reader's weakest assumption identifies the same issue: cosine similarity is never validated against random heads or alternative attributions. The HarmBench Qwen overclaim (79.2 for ORTHO vs 78.0 for the module) and the single-model-pair stability claim are secondary; they affect headline numbers but not the core localization argument. A control-module experiment is cheap and decisive, so the appropriate verdict remains CONDITIONAL rather than ACCEPT or REJECT.","tokens_in":19454,"tokens_out":4698,"duration_ms":54233,"concrete_test":"Reproduce the four headline evaluations (HarmBench ASR in Table 2, GSM8K in Table 1, FQuAD French-to-English ratio in Section 4.1, and ViT target-label error in Figure 9) with control modules of K randomly sampled attention heads, and separately with the bottom-K heads by Eq. 3 score, using the same per-task scalar grid search and at least 10 random head sets per model. Compare the best result for each control against the reported topK result. If any control matches the topK effect within, say, 10% relative on the target metric, the SAMD selection is not demonstrated to be causal; if all controls are clearly weaker, the module-specificity concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the topK heads selected by Eq. 3 form the concept module and that SAMI (Eq. 4) causally controls the concept. The decisive evidence is the intervention results in Sections 4.1-4.4. However, none of these experiments includes a control module. The scalar s is chosen by grid search on the same benchmark used for reporting (Table 1, Table 2, Figure 9, Sections 4.2-4.3), and no random-head or bottom-K module is evaluated under the same protocol. If scaling any K heads with a similar negative scalar also raises HarmBench ASR, degrades French, or reduces target-label accuracy, then the observed effects are generic perturbation artifacts, and cosine-similarity selection is not load-bearing. This concern applies to the strongest results: the FQuAD language switch (85.35% to 1.66%) and the ViT label suppression to 0% target error show that intervening on discovered heads changes behavior, but without a control they do not show that these particular heads are the concept-encoding ones. Appendix E correctly states that causality and stability are not analyzed, but the abstract and conclusion still assert causal controllability, so the missing control is a genuine gap in the argument.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Scalable Attention Module Discovery (SAMD), a method that represents a concept as a vector and scores each attention head by the average cosine similarity between that head's output contribution and the concept vector over a positive dataset; the top-K heads form the 'concept module.' It then proposes Scalar Attention Module Intervention (SAMI), which multiplies the output magnitude of the selected heads by a scalar s to diminish or amplify the concept. Experiments cover SAE-derived concepts on Gemma-2-2B(-IT) (including a French-language module evaluated on FQuAD), a reasoning module evaluated on GSM8K with Llama-3.1-8B-Instruct and Gemma-7B-Base, a safety module evaluated on HarmBench with three aligned LLMs, and label-recognition modules in a ViT-B/32 evaluated on ImageNet. The paper claims that only 3-10 attention heads encode these diverse concepts, that modules are stable across post-training, and that SAMI can jailbreak (+72.7% ASR), improve GSM8K (+1.6%), and suppress ViT target-label accuracy to 0%.","tokens_in":19619,"tokens_out":4919,"duration_ms":47997,"significance":"If the central claim is correct, SAMD would be a valuable, low-cost attribution tool: it requires a single forward pass per input, operates on arbitrary concepts across LLMs and ViTs, leverages public SAE features, and produces sparse modules that are amenable to a simple scalar intervention. The paper's strengths include its cross-domain scope, the use of open libraries (TransformerLens, ViT-Prisma), and the concrete qualitative demonstrations that negative intervention on the discovered 'French' module switches output language and that positive intervention on a 'dog' module causes repetition of dog-related content. However, the load-bearing quantitative claims are currently undermined by the absence of control interventions and by grid-search-based scalar selection on the evaluation benchmarks, so the causal interpretation of the modules is not yet established. The paper also overstates the HarmBench comparison for one model and internally concedes in Appendix E that causality and stability are not analyzed.","major_comments":[{"comment":"The paper never compares the top-K cosine-selected modules with control modules (e.g., random sets of K heads or bottom-K heads) under the same SAMI protocol. Because SAMI changes the magnitude of K attention-head contributions by a scalar, a generic perturbation on any K heads may produce large behavioral changes; without such controls, the headline results (HarmBench 71.1% ASR, GSM8K +1.6%, FQuAD 85.35% to 1.66%, ViT target-label 0%) do not establish that Eq. (3) selects concept-encoding heads. Please add random-head and bottom-K baselines with the identical scalar search and report the comparison.","section":"Sections 4.1-4.4, Eq. (3)-(4)"},{"comment":"The intervention scalars for the central quantitative results (s=1.4/1.2 for GSM8K and s=-1.7/-0.7/-0.8 for HarmBench) are selected by grid search on the same benchmark on which the results are reported, with no held-out split or explicit selection rule. The reported improvements are therefore maxima over the searched grid rather than out-of-sample predictions. Please either fix the scalars on a validation set and report test performance, or present the full sweep curves together with the selection protocol.","section":"Sections 4.2-4.3, Tables 1-2"},{"comment":"The text asserts that the safety module intervention is 'more powerful than the vector-based ORTHO intervention,' but for Qwen7B the safety module reaches 78.0% ASR versus 79.2% for ORTHO. The claim as stated is contradicted by the table; please qualify the comparison (e.g., by model) or correct the text.","section":"Section 4.3, Table 2"},{"comment":"The GSM8K gains (+0.83 and +2.35 percentage points for the two models) are reported without variance estimates, multiple evaluation seeds, or any significance assessment, and the baseline appears to be a single run. Given that the scalar is chosen by grid search, the reader cannot distinguish a true amplification effect from selection noise. Please report standard errors over evaluation seeds or an equivalent perturbation analysis.","section":"Section 4.2, Table 1"},{"comment":"The limitations section explicitly states that 'we do not emphasize causality' and that the discovered module 'could be either overcomplete or incomplete,' yet the abstract and Section 4.4 describe SAMI as 'facilitating jailbreaking' and 'disabling' target-label recognition, and the conclusion states that SAMI 'allows direct control over model prediction and generation.' These causal claims should be reconciled with the stated limitation, either by softening the language or by adding the causal controls requested above.","section":"Appendix E"}],"minor_comments":[{"comment":"The threshold for the positive dataset Dp ('activation above 80% of the maximum strength') is stated without a sensitivity analysis; please indicate whether the discovered modules and the FQuAD result are stable to reasonable variations of this threshold.","section":"Section 4.1"},{"comment":"The x-axis label 'Intervention coefficient' is ambiguous; the sweep procedure is described in words, but the figure should state whether larger values correspond to stronger negative or positive steering and give the units or scale of the coefficient.","section":"Figure 26"},{"comment":"The caption says '4 labels with the lowest/highest generalization error after intervention' but the four panels are not identified; please name the labels and state which two are lowest and which two are highest.","section":"Figure 9"},{"comment":"The evaluation protocol for the DR baseline and the Gemma7B row is not fully specified (e.g., whether the same HarmBench harness and system-prompt settings are used for all rows); please add a footnote describing the exact evaluation setup.","section":"Table 2"},{"comment":"The term 'concept-agnostic' is used although the method requires a positive dataset Dp and a chosen concept-vector construction; consider 'concept-general' to avoid overstatement.","section":"Abstract and Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's qualitative results and fixed-scalar French experiment give it genuine promise, but the quantitative headline claims are currently selection-on-evaluation artifacts in part. The requested control-module experiments and held-out scalar selection are well-defined and within scope, so I recommend major revision rather than rejection. No concerns about attribution or citation practices beyond the overstatement noted in the report."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful, honest interpretability paper that proposes a cheap pipeline—represent a concept as a vector, score every attention head by cosine similarity of its residual contribution, take the top K, then scale those heads with one scalar. It works surprisingly well in the experiments where the protocol is clean, and the authors are upfront in Appendix E that causality and stability are not established. The main gap is evaluation, not method: no control modules, and the headline numbers are fits, not predictions.\n\nWhat is actually new is the specific combination and the breadth. The individual pieces (residual stream decomposition, SAE/difference-in-means concept vectors, logit-lens-style cosine scoring) are all prior art, but putting them together as a concept-agnostic attention-module discovery plus single-scalar intervention is not something I've seen done this cleanly. The French experiment is the strongest result: fixed s=-1 drops FQuAD French responses from 85.35% to 1.66%, beating a swept SAE steering baseline. The ViT result is also convincing in spirit: sweeping s on the discovered 3-head module drives target-label accuracy to zero while leaving other labels roughly intact. The module locations (later layers for language, final layers for vision) are consistent with prior work.\n\nSoft spots, in order. First and most important: there are no control interventions. Nothing is compared against random-head or bottom-K modules under the same scalar protocol. If scaling any K heads with a comparable negative scalar also raises HarmBench ASR or degrades GSM8K, then cosine-similarity selection is not load-bearing and the central claim weakens. Second, the headline HarmBench and GSM8K scalars are grid-searched on the same benchmark used for reporting, so those numbers are post-hoc fits. The authors should hold out a portion of the benchmark or pre-register the scalar. Third, the abstract says SAMI is \"more powerful\" than ORTHO, but Table 2 shows Qwen ORTHO at 79.2 versus their 78.0, so that comparison is overstated. Fourth, the post-training stability observation rests on one model pair; fine as a suggestive case study, not as a general claim.\n\nBottom line: the mechanism is plausible and the pipeline is genuinely useful as a cheap attribution/control tool. It deserves a serious referee, but the referee should insist on control modules and a non-fitted scalar selection protocol before the causal language in the abstract can be supported.","headline":"Useful, honest pipeline paper whose main results are undercut by missing control modules and grid-searched scalars, but worth refereeing for revision.","tokens_in":20214,"tokens_out":2304,"would_cite":true,"duration_ms":23373,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A transformer concept such as 'safety' or 'reasoning' can be localized to 3-10 attention heads and controlled by one scalar.","keywords":["attention head attribution","concept vectors","sparse autoencoders","intervention","transformer interpretability","jailbreaking","vision transformers","cosine similarity"],"falsifier":"Take the same safety and reasoning modules but select K heads with the lowest cosine similarity instead of the highest, apply SAMI with the same scalar, and compare HarmBench attack success and GSM8K accuracy; if low-similarity heads produce comparable effects, the cosine criterion is not doing the selection work. A stronger version is to replace SAMD with causal patching per head and check whether the heads whose ablation most changes concept output coincide with the top-K cosine heads.","tokens_in":19161,"feed_emoji":"🎯","tokens_out":6770,"duration_ms":62982,"temperature":0.7,"pith_summary":"The paper claims that an arbitrary concept in a transformer—'French', 'safety', 'reasoning', or an ImageNet label—can be localized to a small set of attention heads by averaging the cosine similarity between a vector representation of the concept and each head's additive contribution to the residual stream. Across all concepts and models tested, the selected module is sparse, containing 3-10 heads, and scaling exactly those heads' contributions by a single scalar weakens or amplifies the concept. The authors report that suppressing the 'safety' module raises attack success on HarmBench by 72.7 points, amplifying the 'reasoning' module improves GSM8K accuracy by about 1.6 points, and scaling down a label module in a vision transformer drives target-label recognition to 0%. If this picture is right, concept-level control over large transformers reduces to knowing which handful of attention heads to rescale, and much of the machinery of activation steering and prompting becomes an indirect way of doing that.","feed_headline":"Three to ten attention heads carry a concept; one scalar steers it","feed_subtitle":"Rescaling discovered heads lifts math scores 1.6%, raises jailbreak success 72.7 points, and maps concepts in both language and vision.","key_machinery":"The load-bearing object is the residual-stream decomposition of a transformer layer into per-head additive contributions $a_{l,h}(p)$, so each head can be assigned a score by the average cosine similarity between $a_{l,h}(p)$ and a concept vector $v_c$ (SAMD, Eq.~3). The top-$K$ heads form the attention module, and Scalar Attention Module Intervention (SAMI) rescales exactly those heads' contributions by one scalar $s$ at the attention output projection (Eq.~4). Because the scalar acts as a coefficient on existing weights, the whole intervention costs no retraining, no prompt edit, and (at inference) only a tiny change to output matrices.","core_discovery":"The paper's central discovery is that attention-head modules found by top-K cosine similarity between head contributions and a concept vector are sparse and causally controllable: 3-10 heads encode each concept, and multiplying only those heads' output magnitudes by a scalar s produces concept-level behavior changes in both language and vision transformers. The same module locations reappear before and after post-training, supporting the view that concept knowledge is already present in the base model; safety modules concentrate in middle layers, output-language modules in later layers, and image-label modules in final layers. The intervention consistently beats or matches vector-steering baselines while touching about 0.1% of model weights, and it works without prompt changes or optimization.","pith_inferences":["If cosine ranking is genuine, attention heads act like an overcomplete linear basis for concepts; a testable follow-up is whether modules for different concepts overlap, and whether scaling one module measurably moves another.","The same pipeline could probe concepts the paper does not include, such as social bias, hallucination tendency, or persona, by supplying the corresponding contrastive dataset; whether 3-10 heads still suffice is open.","The paper's own caveat that its modules may be overcomplete or incomplete suggests a minimality test: iteratively drop the lowest-similarity head from the module and check how much intervention strength must grow to keep the same behavioral effect.","Because SAMI's scalar is set by grid search, a natural extension is to make the module scale a learned parameter during fine-tuning, freezing the discovered heads and optimizing s directly."],"forward_implications":["Concept localization becomes a one-forward-pass, training-free operation: average cosine similarities against a concept vector and take the top-K heads.","Only about 0.1% of weights—10 heads for safety, 5 for reasoning—need to be rescaled to move benchmark behavior: +72.7 points on HarmBench attack success and about +1.6 points on GSM8K.","The same scalar intervention carries over to vision: rescaling 3 heads removes a target ImageNet label's recognizability to 0% while leaving other labels mostly intact.","Module stability before and after post-training implies that aligning a model does not rewrite where concepts live; it changes what the model does with them.","Because SAMI is equivalent to rescaling output-projection weights, it composes cheaply with existing inference pipelines and can be applied per concept without accumulating model copies."],"supporting_citations":[{"why":"Supplies the residual-stream decomposition in which each attention head contributes an additive vector, the framework SAMD scores.","marker":"[18]"},{"why":"Establishes the hypothesis that higher cosine similarity between feature vectors implies higher semantic similarity, which SAMD relies on.","marker":"[20]"},{"why":"Provides the sparse-autoencoder features with human-readable concept annotations used as concept vectors in the SAE experiments.","marker":"[21]"},{"why":"Gives the contrastive refusal-direction concept vector and the orthogonalization baseline that SAMI is compared against on safety.","marker":"[26]"},{"why":"Defines the GSM8K benchmark and prompts used both to derive the reasoning concept vector and to measure the +1.6% gain.","marker":"[17]"},{"why":"Provides the HarmBench standard-behaviors set on which safety-module suppression raises attack success rate by 72.7 points.","marker":"[16]"},{"why":"Inspires the method of comparing residual-stream contributions with a vector to attribute meaning to components.","marker":"[27]"}],"fun_headline_variants":["One scalar on 3-10 heads: +72.7% jailbreak, +1.6% math","Sparse attention heads: single scalar steers concepts in LLMs","3-10 heads encode a concept; one scalar adjusts them","Single scalar boosts math 1.6% and jailbreak 72.7%","Stable concept heads pre/post training; one scalar controls"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that higher cosine similarity between a head's contribution and a concept vector reliably identifies the heads that encode that concept; the paper asserts this link from earlier representation work rather than validating it against random heads or causal attribution.","fun_headline_variants_meta":{"raw":{"variants":["One scalar on 3-10 heads: +72.7% jailbreak, +1.6% math","Sparse attention heads: single scalar steers concepts in LLMs","3-10 heads encode a concept; one scalar adjusts them","Single scalar boosts math 1.6% and jailbreak 72.7%","Stable concept heads pre/post training; one scalar controls"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001477,"raw_usage":{"total_tokens":5965,"prompt_tokens":1005,"completion_tokens":4960,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":4858}},"tokens_in":621,"tokens_out":4960,"duration_ms":33180,"temperature":1.0,"reasoning_tokens":4858,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:12:49.511013+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same safety and reasoning modules but select K heads with the lowest cosine similarity instead of the highest, apply SAMI with the same scalar, and compare HarmBench attack success and GSM8K accuracy; if low-similarity heads produce comparable effects, the cosine criterion is not doing the selection work. A stronger version is to replace SAMD with causal patching per head and check whether the heads whose ablation most changes concept output coincide with the top-K cosine heads.","supporting_citations":[{"cited_title":"Daniel Freeman, Theodore R","cited_arxiv_id":null,"evidence_quote":"Establishes the hypothesis that higher cosine similarity between feature vectors implies higher semantic similarity, which SAMD relies on."},{"cited_title":"HarmBench: A standardized evaluation framework for automated red teaming and robust refusal","cited_arxiv_id":null,"evidence_quote":"Provides the HarmBench standard-behaviors set on which safety-module suppression raises attack success rate by 72.7 points."},{"cited_title":"interpreting gpt: the logit lens","cited_arxiv_id":null,"evidence_quote":"Inspires the method of comparing residual-stream contributions with a vector to attribute meaning to components."}],"review_version":2}