{"id":"dd7884de-c45e-4afe-b54b-370be40439d2","arxiv_id":"2607.07128","paper_version":1,"verdict":"CONDITIONAL","confidence":"UNKNOWN","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":6,"one_line_summary":"Sparse interventions on 8–64 neurons distributed across layers can activate task behavior in instruction-tuned LLMs, outperforming first-order linear steering approaches by modeling nonlinear neuron interactions.","lead":"This paper shows that modifying the activity of as few as 8–64 individual neurons (0.01% of a model) can switch a language model into performing a specific task it wasn't explicitly prompted to do. The method, called DSI, accounts for nonlinear interactions between neurons that simpler linear steering methods miss, and the neuron-level view enables decomposing tasks into shared and task-specific components.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"DSI's advantage over simpler baselines may stem from per-neuron scaling rather than iterative optimization or gradient robustification, leaving the core methodological contribution under-isolated.","rationale":"The reader correctly identifies that the paper lacks systematic sensitivity analysis and that initialization quality is untested. However, the more fundamental concern is that the ablation design conflates two variables (iterative refinement and per-neuron scaling), which means the paper has not actually demonstrated that its core methodological contribution—iterative optimization of sparse neuron sets accounting for nonlinear interactions—is necessary. The reader's focus on initialization quality and optimization hyperparameters is a secondary concern; even with perfect optimization, if per-neuron scaling alone explains the performance gap, the central claim about the importance of modeling nonlinear interactions through iterative refinement is weakened. The paper's Figure 2 does show nonlinear interaction effects exist (e.g., pairwise interventions in Figure 2b), but showing that effects exist is different from showing that the iterative procedure is needed to capture them effectively. The experimental evidence in Table 1 is consistent with DSI working, but does not isolate why it works. This is a correctable experimental gap: adding a per-neuron-scaling baseline without iterative set refinement would clarify the contribution. The verdict remains CONDITIONAL because the method clearly works (substantial accuracy gains over baseline), but the explanation for why it works—and whether the iterative machinery is necessary—is not yet established. The single-token evaluation limitation and qualitative case studies, which the reader notes, are additional but less central concerns.","tokens_in":39229,"tokens_out":722,"duration_ms":462266,"concrete_test":"Add an ablation: keep the neuron set fixed at the first-order top-n selection (no iterative set refinement), but optimize per-neuron scaling factors (e.g., via L-BFGS on the n scaling factors, or even a single coordinate-descent sweep). Compare accuracy to full DSI and the 'no iter.' baseline on all 12 tasks. If this intermediate baseline matches DSI within confidence intervals, the iterative set refinement is not the load-bearing component.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that DSI outperforms linear steering by accounting for nonlinear neuron interactions via iterative refinement (Section 3.2). However, the ablation 'no iter.' (Table 1) differs from full DSI in two ways simultaneously: (1) it removes iterative refinement, and (2) it replaces per-neuron scaling factors with a single joint scalar s ∈ [0.01, 50]. This conflates the contribution of iterative optimization with the contribution of per-neuron scaling. The 'no iter.' baseline is thus a strawman: a linear steering method with a uniform scaling factor is expected to perform poorly given the nonlinear, neuron-specific effects demonstrated in Figure 2a, where optimal intervention strength varies substantially across neurons. If the 'no iter.' baseline were granted per-neuron scaling (e.g., via a closed-form solution or a few steps of coordinate descent on the scaling factors), the gap between it and full DSI might narrow considerably. The paper does not test this intermediate baseline, so it is unclear whether the iterative optimization of neuron sets (the core algorithmic contribution) is necessary, or whether simply allowing per-neuron scaling on the first-order-selected neurons would achieve comparable results. This matters because the iterative ZeroFPR procedure (50 steps, 10 restarts) is the main computational cost and methodological complexity of DSI.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"The paper introduces Distributed Sparse Interventions (DSI), a method for identifying sparse sets of neurons distributed across layers and attention heads that can activate task-relevant behavior in instruction-tuned language models. DSI proceeds by (1) computing activation differences between k-shot and 0-shot prompts, (2) selecting an initial neuron set via a robustified (LRP-modified) gradient estimate, and (3) iteratively refining both the neuron set and per-neuron scaling factors using the ZeroFPR optimizer. The method is evaluated on 12 tasks across three models (Qwen 3 8B, Gemma 3 4B, Llama 3.2 3B), showing that intervening on as few as 8–64 neurons can substantially improve 0-shot task accuracy. Two case studies investigate task composition by decomposing neuron sets into shared (copying) and task-specific components via set operations.","tokens_in":39377,"tokens_out":2254,"duration_ms":194134,"significance":"The paper addresses a genuine gap in the model steering literature: most prior work operates on global directions in activation space, assuming approximate linearity and additivity. The demonstration in Figure 2 that neuron-level interventions exhibit nonlinear, neuron-specific effects—including interaction effects between neuron pairs—is a valuable empirical contribution. The finding that tasks can be activated by intervening on 0.01–0.04% of neurons, distributed across layers and heads, is striking and well-supported by the experimental design (train/test splits, 3 models × 12 tasks, 95% confidence intervals). The set-based perspective on task composition (Section 5) is a novel conceptual angle. Code is reportedly available on GitHub, supporting reproducibility.","major_comments":[{"comment":"The ablation design in Section 4.1 conflates two methodological changes, undermining the paper's attribution of performance gains to iterative refinement. The 'no iter.' baseline (Table 1) differs from full DSI in two ways simultaneously: (1) it removes iterative refinement of the neuron set, and (2) it replaces per-neuron scaling factors with a single joint scalar s ∈ [0.01, 50]. The paper states (Section 4.1) that 'both gradient robustification and iterative refinement contribute consistently to improved intervention performance,' but this claim cannot be evaluated because the contribution of iterative set refinement is entangled with the contribution of per-neuron scaling. Figure 2a already shows that optimal intervention strength varies substantially across neurons, so a uniform-scaling baseline is expected to perform poorly regardless of whether iterative set refinement helps. An中间基","section":null}],"minor_comments":[{"comment":"Section 3.2, Algorithm 1: The notation s₀ ← 1(topₙ(e)) is introduced without explicit definition of the indicator function. Readers unfamiliar with this convention may need to infer it from context.","section":null},{"comment":"Table 1: The ranges (e.g., '0.18–0.97') are described as 'minimum and maximum accuracy across tasks' but it is unclear whether these are per-task accuracies or per-condition accuracies. Clarifying this would help interpretation.","section":null},{"comment":"Figure 3: The x-axis uses a log-like spacing for neuron set sizes (8, 16, 32, ..., 2048) but is labeled linearly. This could mislead readers about the rate of performance change with set size.","section":null},{"comment":"Section 5.1: The case study reports that the 'specific' neuron set (9 neurons for present-past) achieves 0.37 accuracy on the present-past task, compared to 0.85 for the full 32-neuron set. This is a substantial drop, and the text acknowledges that the specific set 'induce[s] less copy behaviour' but does not fully discuss why the task-specific neurons alone are insufficient. This weakens the decomposition narrative.","section":null},{"comment":"Appendix F: The optimization uses 10 perturbed restarts with noise scale 0.3 and a step size multiplier of 1.5. No sensitivity analysis to these hyperparameters is provided. A brief note on robustness would strengthen the methodological contribution.","section":null},{"comment":"The paper does not compare DSI against existing steering baselines (e.g., function vectors [49], contrastive activation addition [43], LoFiT [54]) in terms of task accuracy. While the ablations against simplified versions of DSI are informative, a direct comparison to established methods would better contextualize the reported gains.","section":null},{"comment":"Section 4, footnote 2: The compute time (~108h on MI300A APUs) is noted, but the per-task or per-intervention-set-size compute cost is not broken down. This would help readers assess practical scalability.","section":null}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the ablation design is well-founded and is the primary reason for the major revision recommendation. The paper's core empirical finding (that sparse neuron-level interventions can activate tasks) is solid, but the methodological claim about the necessity of iterative refinement is not adequately isolated. The authors may be able to address this with a relatively modest additional experiment, so I would encourage giving them the opportunity to revise rather than rejecting outright. The LRP gradient rules cited from overlapping author groups [1, 2, 4, 42] are standard tools and do not raise a circularity concern."},"author_rebuttal":null,"desk_editor":{"model":"glm-5.2","letter":"The paper you should know about: DSI identifies tiny sets of neurons (8–64, roughly 0.01% of the model) that, when you additively intervene on them, activate ICL task behavior in 0-shot prompts. The method combines LRP-modified gradients for initialization with iterative sparse optimization (ZeroFPR) to refine both which neurons and how much to scale each one. Across 3 models and 12 tasks, it consistently beats no-intervention baselines and two ablations. That part is solid and the experimental scope is reasonable for a first pass — 50/50 splits, confidence intervals, three model families, ablations for both gradient robustification and iterative refinement. Figure 2 directly demonstrates the nonlinear and pairwise interaction effects that motivate going beyond first-order linear steering, which is the right evidence to show. The set-based decomposition in Section 5 (intersecting task neuron sets with copy-task neuron sets) is a genuinely nice conceptual move, even if the case studies are mostly qualitative. Code is reportedly released, though no commit hash is given. The stress-test concern about the ablation design is the real soft spot and it lands. The 'no iter.' baseline differs from full DSI in two ways simultaneously: it removes iterative refinement AND collapses per-neuron scaling to a single joint scalar. So the gap between 'no iter.' and full DSI could be driven by per-neuron scaling alone, not by the iterative optimization of neuron membership — which is the actual algorithmic contribution. An intermediate baseline (first-order neuron selection + per-neuron scaling, no iterative set refinement) would isolate this. The paper doesn't run it. This doesn't sink the paper — the method clearly works and the nonlinear effects are real — but it does mean the core methodological claim is under-isolated. The evaluation being single-token prediction on relatively simple ICL tasks is a known limitation the authors acknowledge. The case studies on task decomposition have mixed quantitative results (the 'specific' neuron sets often fail to activate the target transformation), which slightly undercuts the compositionality story. Overall: a solid methods paper for the mechanistic interpretability crowd. The neuron-level perspective and set operations are a useful lens. The ablation gap is the thing I'd push the authors to fix in revision. Deserves a serious referee.","headline":"DSI: sparse neuron-level steering that works, but the ablation design conflates two improvements at once","tokens_in":40022,"tokens_out":554,"would_cite":true,"duration_ms":116951,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Editing 0.01% of neurons activates task behavior in language models","keywords":["language models","model steering","neuron-level interventions","sparse representations","interpretability","task composition","nonlinear interactions","in-context learning"],"falsifier":"If one could show that randomly chosen neuron sets of the same size, when paired with the same iterative optimization procedure, achieve comparable task accuracy to DSI-selected sets, then the contribution of the neuron-selection mechanism (as opposed to the optimization procedure itself) would be undermined.","tokens_in":39462,"feed_emoji":"🧠","tokens_out":1119,"duration_ms":221214,"temperature":0.7,"pith_summary":"This paper introduces Distributed Sparse Interventions (DSI), a method that identifies tiny sets of neurons—as few as 8 out of roughly 100,000—distributed across layers and attention heads, whose additive modification activates a specific task in an instruction-tuned language model. The central claim is that task-relevant computation lives in a sparse, nonlinear, neuron-specific basis rather than in dense global directions in activation space, and that by iteratively re-evaluating intervention effects (rather than relying on a single first-order gradient estimate) one can find these small neuron sets and steer model behavior with high fidelity. The paper demonstrates this across 12 tasks and three model families, showing that 8–64 neurons often match or exceed 10-shot in-context-learning accuracy, and that both gradient robustification and iterative refinement are necessary because neuron interactions produce substantial nonlinear effects that linear steering methods miss. A set-based perspective on the identified neurons further allows decomposing tasks into shared subcomponents (e.g., input-copying neurons shared between translation and tense-switching) and task-specific transformation neurons, offering a compositional view of how tasks are structured at the neuron level.","feed_headline":"Editing 0.01% of neurons activates task behavior in language models","feed_subtitle":"Sparse, neuron-level interventions that model nonlinear interactions can steer LLMs with as few as 8 neurons, outperforming linear steering.","key_machinery":"DSI works in four steps: (1) compute average activation differences between 10-shot and 0-shot prompts for every neuron; (2) estimate each neuron's expected effect on the correct output logit using a first-order Taylor approximation with LRP-modified (robustified) gradients; (3) select the top-n neurons by expected effect as a starting point and iteratively refine both which neurons are selected and their per-neuron intervention scaling using ZeroFPR, a nonconvex sparse optimization algorithm that re-evaluates expected effects at each step to capture nonlinear interactions; (4) apply the refined per-neuron interventions additively during the forward pass on 0-shot inputs. The activation-diff","core_discovery":"The paper's central discovery is that task behavior in large language models can be activated by intervening on a remarkably sparse set of individual neurons—distributed across layers and attention heads—provided that nonlinear interactions between neurons are explicitly modeled through iterative refinement of intervention effects, rather than approximated by a single first-order gradient. This finding challenges the prevailing assumption that effective steering requires modifying global directions across all neurons in a layer, and shows that the native neuron basis contains sufficiently sparse task-relevant structure to support fine-grained behavioral control.","pith_inferences":["The paper tests single-token prediction only; if sparse neuron sets identified for single-token tasks do not transfer to multi-token generation (where the model must maintain task behavior across positions), the practical scope of DSI would be narrower than the results suggest.","The set-based decomposition into copy and transform neurons is demonstrated on two tasks with rigid input-output structure; whether richer, open-ended tasks exhibit similarly decomposable neuron-set structure remains an open question that the paper's case studies only begin to address.","If the LRP-modified gradient initialization were replaced with a fundamentally different attribution method, the iterative optimizer might converge to different neuron sets—raising the question of whether the identified neurons are stable features of the model or artifacts of the search procedure."],"forward_implications":["If sparse neuron sets can reliably activate tasks, then targeted suppression of those same sets could selectively disable capabilities—such as translation or sentiment analysis—without retraining, offering a surgical alternative to coarse methods like fine-tuning or safety filters.","The decomposition of tasks into shared neuron sets (e.g., copy neurons) and task-specific neuron sets suggests a natural hierarchy: if higher-level reasoning tasks decompose similarly, one could map compositional structure of model capabilities by analyzing overlaps and differences in neuron sets across tasks.","The finding that interventions distribute across many layers and heads, rather than concentrating in predefined layers, implies that task representations are genuinely distributed—challenging the common practice of steering at a single chosen layer.","If the iterative refinement is essential because first-order approximations miss nonlinear interactions, then other interpretability methods that rely on single-pass gradient estimates (e.g., attribution patching) may systematically misidentify the most causally important components."],"fun_headline_variants":["Editing 0.01% of neurons activates task behavior in language models","Sparse neuron interventions steer LLM behavior by modeling nonlinear interactions","As few as 8 neurons can activate task behavior in language models","Distributed sparse interventions localize task computations to 0.01% of neurons","Nonlinear neuron-level interventions outperform global steering in LLMs"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The method depends on a first-order gradient approximation (even if robustified) to provide a good enough starting point for the iterative optimizer to find effective sparse neuron sets; if this initialization were poor, the 50-step optimization with 10 restarts might not escape local optima, and the paper does not systematically test how sensitive the results are to initialization quality or optimization settings.","fun_headline_variants_meta":{"raw":{"variants":["Editing 0.01% of neurons activates task behavior in language models","Sparse neuron interventions steer LLM behavior by modeling nonlinear interactions","As few as 8 neurons can activate task behavior in language models","Distributed sparse interventions localize task computations to 0.01% of neurons","Nonlinear neuron-level interventions outperform global steering in LLMs"]},"model":"glm-5.2","effort":"low","cost_usd":0.0,"raw_usage":{"total_tokens":637,"prompt_tokens":562,"completion_tokens":75,"prompt_tokens_details":null},"tokens_in":562,"tokens_out":75,"duration_ms":68694,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T19:19:43.015667+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If one could show that randomly chosen neuron sets of the same size, when paired with the same iterative optimization procedure, achieve comparable task accuracy to DSI-selected sets, then the contribution of the neuron-selection mechanism (as opposed to the optimization procedure itself) would be undermined.","supporting_citations":[],"review_version":1}