Pith. sign in

REVIEW 1 major objections 7 minor 56 references

Distributed Sparse Interventions in Language Models

T0 review · 1 major / 7 minor · reviewed 2026-07-09 · glm-5.2

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Editing 0.01% of neurons activates task behavior in language models

desk verdict DSI: sparse neuron-level steering that works, but the ablation design conflates two improvements at once read the letter →

arxiv 2607.07128 v1 pith:GFF6SE6U submitted 2026-07-08 cs.LG

Maximilian S. Ernst (1 , 2 , 3) , Lorenz Linhardt (3 , 4) , Aaron Peikert (2) , Oliver Eberle (3 , 4) ((1) Max Planck School of Cognition
show 4 more authors
(2) Center for Lifespan Psychology Max Planck Institute for Human Development (3) Machine Learning Group Technische Universität Berlin (4) Berlin Institute for the Foundations of Learning Data (BIFOLD))
This is my paper · ORCID
classification cs.LG
keywords languagemodelsmodelsteeringneuron-levelinterventionssparserepresentationsinterpretabilitytaskcompositionnonlinearinteractionsin-contextlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Distributed Sparse Interventions (DSI), a method that identifies tiny sets of neurons—as few as 8 out of roughly 100,000—distributed across layers and attention heads, whose additive modification activates a specific task in an instruction-tuned language model. The central claim is that task-relevant computation lives in a sparse, nonlinear, neuron-specific basis rather than in dense global directions in activation space, and that by iteratively re-evaluating intervention effects (rather than relying on a single first-order gradient estimate) one can find these small neuron sets and steer model behavior with high fidelity. The paper demonstrates this across 12 tasks and three model families, showing that 8–64 neurons often match or exceed 10-shot in-context-learning accuracy, and that both gradient robustification and iterative refinement are necessary because neuron interactions produce substantial nonlinear effects that linear steering methods miss. A set-based perspective on the identified neurons further allows decomposing tasks into shared subcomponents (e.g., input-copying neurons shared between translation and tense-switching) and task-specific transformation neurons, offering a compositional view of how tasks are structured at the neuron level.

What carries the argument

DSI works in four steps: (1) compute average activation differences between 10-shot and 0-shot prompts for every neuron; (2) estimate each neuron's expected effect on the correct output logit using a first-order Taylor approximation with LRP-modified (robustified) gradients; (3) select the top-n neurons by expected effect as a starting point and iteratively refine both which neurons are selected and their per-neuron intervention scaling using ZeroFPR, a nonconvex sparse optimization algorithm that re-evaluates expected effects at each step to capture nonlinear interactions; (4) apply the refined per-neuron interventions additively during the forward pass on 0-shot inputs. The activation-diff

What would settle it

If one could show that randomly chosen neuron sets of the same size, when paired with the same iterative optimization procedure, achieve comparable task accuracy to DSI-selected sets, then the contribution of the neuron-selection mechanism (as opposed to the optimization procedure itself) would be undermined.

Watch

Extended reading notes

Core claim

The paper's central discovery is that task behavior in large language models can be activated by intervening on a remarkably sparse set of individual neurons—distributed across layers and attention heads—provided that nonlinear interactions between neurons are explicitly modeled through iterative refinement of intervention effects, rather than approximated by a single first-order gradient. This finding challenges the prevailing assumption that effective steering requires modifying global directions across all neurons in a layer, and shows that the native neuron basis contains sufficiently sparse task-relevant structure to support fine-grained behavioral control.

Load-bearing premise

The method depends on a first-order gradient approximation (even if robustified) to provide a good enough starting point for the iterative optimizer to find effective sparse neuron sets; if this initialization were poor, the 50-step optimization with 10 restarts might not escape local optima, and the paper does not systematically test how sensitive the results are to initialization quality or optimization settings.

Editorial extensions

If this is right

  • If sparse neuron sets can reliably activate tasks, then targeted suppression of those same sets could selectively disable capabilities—such as translation or sentiment analysis—without retraining, offering a surgical alternative to coarse methods like fine-tuning or safety filters.
  • The decomposition of tasks into shared neuron sets (e.g., copy neurons) and task-specific neuron sets suggests a natural hierarchy: if higher-level reasoning tasks decompose similarly, one could map compositional structure of model capabilities by analyzing overlaps and differences in neuron sets across tasks.
  • The finding that interventions distribute across many layers and heads, rather than concentrating in predefined layers, implies that task representations are genuinely distributed—challenging the common practice of steering at a single chosen layer.
  • If the iterative refinement is essential because first-order approximations miss nonlinear interactions, then other interpretability methods that rely on single-pass gradient estimates (e.g., attribution patching) may systematically misidentify the most causally important components.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper tests single-token prediction only; if sparse neuron sets identified for single-token tasks do not transfer to multi-token generation (where the model must maintain task behavior across positions), the practical scope of DSI would be narrower than the results suggest.
  • The set-based decomposition into copy and transform neurons is demonstrated on two tasks with rigid input-output structure; whether richer, open-ended tasks exhibit similarly decomposable neuron-set structure remains an open question that the paper's case studies only begin to address.
  • If the LRP-modified gradient initialization were replaced with a fundamentally different attribution method, the iterative optimizer might converge to different neuron sets—raising the question of whether the identified neurons are stable features of the model or artifacts of the search procedure.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 7 minor

Summary. The paper introduces Distributed Sparse Interventions (DSI), a method for identifying sparse sets of neurons distributed across layers and attention heads that can activate task-relevant behavior in instruction-tuned language models. DSI proceeds by (1) computing activation differences between k-shot and 0-shot prompts, (2) selecting an initial neuron set via a robustified (LRP-modified) gradient estimate, and (3) iteratively refining both the neuron set and per-neuron scaling factors using the ZeroFPR optimizer. The method is evaluated on 12 tasks across three models (Qwen 3 8B, Gemma 3 4B, Llama 3.2 3B), showing that intervening on as few as 8–64 neurons can substantially improve 0-shot task accuracy. Two case studies investigate task composition by decomposing neuron sets into shared (copying) and task-specific components via set operations.

Significance. The paper addresses a genuine gap in the model steering literature: most prior work operates on global directions in activation space, assuming approximate linearity and additivity. The demonstration in Figure 2 that neuron-level interventions exhibit nonlinear, neuron-specific effects—including interaction effects between neuron pairs—is a valuable empirical contribution. The finding that tasks can be activated by intervening on 0.01–0.04% of neurons, distributed across layers and heads, is striking and well-supported by the experimental design (train/test splits, 3 models × 12 tasks, 95% confidence intervals). The set-based perspective on task composition (Section 5) is a novel conceptual angle. Code is reportedly available on GitHub, supporting reproducibility.

major comments (1)
  1. The ablation design in Section 4.1 conflates two methodological changes, undermining the paper's attribution of performance gains to iterative refinement. The 'no iter.' baseline (Table 1) differs from full DSI in two ways simultaneously: (1) it removes iterative refinement of the neuron set, and (2) it replaces per-neuron scaling factors with a single joint scalar s ∈ [0.01, 50]. The paper states (Section 4.1) that 'both gradient robustification and iterative refinement contribute consistently to improved intervention performance,' but this claim cannot be evaluated because the contribution of iterative set refinement is entangled with the contribution of per-neuron scaling. Figure 2a already shows that optimal intervention strength varies substantially across neurons, so a uniform-scaling baseline is expected to perform poorly regardless of whether iterative set refinement helps. An中间基
minor comments (7)
  1. Section 3.2, Algorithm 1: The notation s₀ ← 1(topₙ(e)) is introduced without explicit definition of the indicator function. Readers unfamiliar with this convention may need to infer it from context.
  2. Table 1: The ranges (e.g., '0.18–0.97') are described as 'minimum and maximum accuracy across tasks' but it is unclear whether these are per-task accuracies or per-condition accuracies. Clarifying this would help interpretation.
  3. Figure 3: The x-axis uses a log-like spacing for neuron set sizes (8, 16, 32, ..., 2048) but is labeled linearly. This could mislead readers about the rate of performance change with set size.
  4. Section 5.1: The case study reports that the 'specific' neuron set (9 neurons for present-past) achieves 0.37 accuracy on the present-past task, compared to 0.85 for the full 32-neuron set. This is a substantial drop, and the text acknowledges that the specific set 'induce[s] less copy behaviour' but does not fully discuss why the task-specific neurons alone are insufficient. This weakens the decomposition narrative.
  5. Appendix F: The optimization uses 10 perturbed restarts with noise scale 0.3 and a step size multiplier of 1.5. No sensitivity analysis to these hyperparameters is provided. A brief note on robustness would strengthen the methodological contribution.
  6. The paper does not compare DSI against existing steering baselines (e.g., function vectors [49], contrastive activation addition [43], LoFiT [54]) in terms of task accuracy. While the ablations against simplified versions of DSI are informative, a direct comparison to established methods would better contextualize the reported gains.
  7. Section 4, footnote 2: The compute time (~108h on MI300A APUs) is noted, but the per-task or per-intervention-set-size compute cost is not broken down. This would help readers assess practical scalability.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity found; DSI's derivation is self-contained with proper train/test separation and LRP rules used as gradient tools, not load-bearing claims.

full rationale

The paper's derivation chain is self-contained. DSI identifies sparse neuron interventions using activation differences between 10-shot and 0-shot prompts (Eq. 3), a robustified gradient for first-order effect estimation (Eq. 2), and iterative refinement via ZeroFPR optimization (Algorithm 1). The LRP-modified gradient rules (Appendix D, Table 4) are cited from prior work by overlapping author groups [1, 2, 4, 42], but these serve as a computational tool for gradient estimation—they are not the central claim and do not define the intervention in terms of its own output. The method is trained on 100 0-shot and 100 10-shot prompts from a training split and evaluated on 200 held-out 0-shot prompts from a separate test split, so the 'prediction' (task activation on test data) is not fitted to the evaluation data by construction. The ablation baselines ('no iter.' and 'no rob.') differ from full DSI in conflated ways (per-neuron scaling vs. joint scaling), which is a correctness/experimental design concern rather than a circularity issue. The skeptic's attack targets ablation isolation, not circular reasoning. No step in the derivation reduces to its own inputs by definition or self-citation.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

No new entities, particles, forces, or dimensions are postulated. DSI operates entirely within the existing architecture of transformer language models.

free parameters (6)
  • n (neuron set size) = 8 to 2048 (swept)
    The number of neurons to intervene on is a hyperparameter swept across values; results are reported as a function of n.
  • t (number of optimization steps) = 50
    Maximum number of ZeroFPR iterations, chosen by the authors (Appendix F).
  • Number of perturbed restarts = 10
    Number of random perturbation runs for escaping local minima (Appendix F).
  • Perturbation noise scale = 0.3
    Standard deviation of Gaussian noise for perturbed starting values (Appendix F).
  • Step size multiplier for exploration = 1.5
    Factor applied to initial step size to encourage exploration beyond first step (Appendix F).
  • k (number of ICL examples for activation differences) = 10
    Number of in-context examples used to compute activation differences; fixed at 10 throughout.
assumptions (4)
  • domain assumption Task-relevant behavior can be activated by additive interventions on neuron activations (a ← a + δ) rather than requiring structural replacement.
    Section 3, notation paragraph. The paper intervenes by adding δ to neuron activations during the forward pass, assuming additive modification is sufficient to elicit task behavior.
  • domain assumption The last token position is the relevant position for capturing task-relevant computations.
    Section 4, Experimental setup: 'we consider the outputs of all attention heads in each layer at the last token position.' This assumes task information is concentrated at the last token.
  • domain assumption LRP-modified gradients provide a more robust estimate of intervention effects than raw gradients in transformer models.
    Section 3.1, citing [23]. The identity rule, half rule, and LayerNorm rule (Appendix D, Table 4) are assumed to improve effect estimation without independent verification in this paper.
  • domain assumption Neurons in the native basis (before projection to residual stream) carry task-relevant structure accessible for intervention.
    Section 4, Experimental setup. The paper intervenes on attention head outputs before residual stream projection, assuming this basis is sufficiently informative despite potential superposition [13].

how reviews work

0 comments
Cite this review

Pith. "Pith review of Distributed Sparse Interventions in Language Models." pith.science (2026). https://pith.science/paper/GFF6SE6U

@misc{pith2026260707128,
  author       = {Pith},
  title        = {Pith review of: Distributed Sparse Interventions in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GFF6SE6U}},
  note         = {Machine review of arXiv:2607.07128}
}
read the original abstract

Language models perform a wide range of tasks at varying levels of abstraction with the capacity to flexibly infer tasks from context, execute multiple tasks simultaneously, and select among competing tasks. To study the role of model components in task behaviour, their causal influence can be investigated through interventions. Prior work on model steering has largely focused on interventions along global directions in activation space, modeling task representations as approximately linear and additive. By studying interventions at the neuron level, we find substantial, neuron-specific nonlinear effects on model outputs that are not captured by current steering approaches. We introduce Distributed Sparse Interventions (DSI), an intervention approach that considers nonlinearities and interactions between neurons across layers to identify sparse sets of neurons that elicit task-relevant computations. Across a range of tasks, we demonstrate that DSI can activate task behaviour in instruction-tuned language models by localising and intervening on as few as 0.01% of neurons, highlighting the effectiveness of sparse, distributed interventions in the neuron basis. Additionally, adopting a set-based perspective enables computations over the identified neuron sets, offering insights into the roles of individual neurons by analysing their effects across tasks. Through sparse interventions, DSI enables fine-grained control over model behaviour, localisation of task-relevant neuron sets, and furthers our understanding of task composition.

Figures

Figures reproduced from arXiv: 2607.07128 by the authors.

Figure 1
Figure 1. Schematic overview of how distributed sparse interventions are identified and applied [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Value of the correct output logit of Llama 3.2 (3B) for the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. 0-shot test set accuracy for interventions across set sizes for extractive (top row) and [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: b contains the resulting accuracies and a qualitative example for the set size of [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 4
Figure 4. Figure 4: Task decomposition into neuron sets shared with a [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Example system with causal interventions [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: 0-shot test set accuracy for interventions across set sizes for extractive (top row) and [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: 0-shot test set accuracy for interventions across set sizes for extractive (top row) and [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Absolute intervention strength for the first-order and iterative interventions, summed over [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Overlap of neuron sets identified by first-order approximation and the iterative algorithm. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Number of intervened-on model components (attention heads), identified by DSI, per [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Number of intervened-on neurons, identified by DSI, within layers and heads for Qwen 3 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 13
Figure 13. Figure 13: Number of intervened-on neurons, identified by DSI, within layers and heads for Gemma [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 12
Figure 12. Figure 12: Number of intervened-on neurons, identified by DSI, within layers and heads for Llama [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 56 canonical work pages

  1. [1]

    Achtibat, S

    R. Achtibat, S. M. V . Hatefi, M. Dreyer, A. Jain, T. Wiegand, S. Lapuschkin, and W. Samek. AttnLRP: Attention-aware layer-wise relevance propagation for transformers. InProceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 135–168. PMLR, 2024. URL https://proceedings.mlr.pres...

    work page 2024

  2. [2]

    A. Ali, T. Schnake, O. Eberle, G. Montavon, K.-R. Müller, and L. Wolf. XAI for transformers: Better explanations through conservative propagation. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 435–451. PMLR,

    work page

  3. [3]

    URL https://proceedings.mlr.press/v162/ali22a.html

    work page

  4. [4]

    Language Model Circuits Are Sparse in the Neuron Basis

    A. Arora, Z. Wu, J. Steinhardt, and S. Schwettmann. Language model circuits are sparse in the neuron basis. arXiv preprint arXiv:2601.22594, 2026

    work page Pith review arXiv 2026

  5. [5]

    Arras, J

    L. Arras, J. Arjona-Medina, M. Widrich, G. Montavon, M. Gillhofer, K.-R. Müller, S. Hochreiter, and W. Samek. Explaining and Interpreting LSTMs, page 211–238. Springer-Verlag, Berlin, Heidelberg, 2022. ISBN 978-3-030-28953-9. URL https://doi.org/10.1007/978-3-030-28954-6_11

    work page doi:10.1007/978-3-030-28954-6_11 2022

  6. [6]

    S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, and W. Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130140, 2015

    work page 2015

  7. [7]

    Balduzzi, M

    D. Balduzzi, M. Frean, L. Leary, J. P. Lewis, K. W.-D. Ma, and B. McWilliams. The shattered gradients problem: if resnets are the answer, then what is the question? In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 342–350. JMLR.org, 2017

    work page 2017

  8. [8]

    D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017

    work page 2017

Show all 56 references
  1. [9]

    Beaglehole, A

    D. Beaglehole, A. Radhakrishnan, E. Boix-Adsera, and M. Belkin. Toward universal steering and monitoring of AI models. Science, 391(6787):787–792, 2026

    2026

  2. [10]

    Bricken, A

    T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2(5):6, 2023

    2023

  3. [11]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Ch...

    1901

  4. [12]

    D. Dai, L. Dong, Y . Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages...

    2022 doi

  5. [13]

    Eberhardt and R

    F. Eberhardt and R. Scheines. Interventions and causal inference. Philosophy of science, 74(5):981–995, 2007

    2007

  6. [14]

    Elhage, T

    N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022

    2022 arXiv

  7. [15]

    U. Evci, T. Gale, J. Menick, P. S. C. Rivadeneira, and E. Elsen. Rigging the lottery: Making all tickets winners. In International Conference of Machine Learning, 2020. 10

    2020

  8. [16]

    Frankle and M

    J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum? id=rJl-b3RcF7

    2019

  9. [17]

    Gemma 3 technical report

    Gemma Team. Gemma 3 technical report. 2025. URL https://goo.gle/Gemma3Report

    2025

  10. [18]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  11. [19]

    Hendel, M

    R. Hendel, M. Geva, and A. Globerson. In-context learning creates task vectors. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9318–9333, 2023

    2023

  12. [20]

    Hernandez, A

    E. Hernandez, A. S. Sharma, T. Haklay, K. Meng, M. Wattenberg, J. Andreas, Y . Belinkov, and D. Bau. Linearity of relation decoding in transformer language models. arXiv preprint arXiv:2308.09124, 2023

    2023 arXiv

  13. [21]

    Honovich, U

    O. Honovich, U. Shaham, S. R. Bowman, and O. Levy. Instruction induction: From few examples to natural language task descriptions. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vo...

    1935 doi

  14. [22]

    Huben, H

    R. Huben, H. Cunningham, L. R. Smith, A. Ewart, and L. Sharkey. Sparse autoencoders find highly inter- pretable features in language models. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  15. [23]

    Ilharco, M

    G. Ilharco, M. T. Ribeiro, M. Wortsman, S. Gururangan, L. Schmidt, H. Hajishirzi, and A. Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022

    2022 arXiv

  16. [24]

    F. R. Jafari, O. Eberle, A. Khakzar, and N. Nanda. Relp: Faithful and efficient circuit discovery in language models via relevance patching. arXiv preprint arXiv:2508.21258, 2025

    2025

  17. [25]

    A. K. Lampinen, Y . Li, E. Hosseini, S. Bhardwaj, and M. Shanahan. Linear representations in language models can change dramatically over a conversation. arXiv preprint arXiv:2601.20834, 2026

    2026

  18. [26]

    Lample, A

    G. Lample, A. Conneau, M. Ranzato, L. Denoyer, and H. Jégou. Word translation without parallel data. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum? id=H196sainb

    2018

  19. [27]

    J. Li, Y . Hou, M. Sachan, and R. Cotterell. What do language models learn in context? the structured task hypothesis. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12365–12379, 2024

    2024

  20. [28]

    L. Li, P. Dong, Z. Tang, X. Liu, Q. Wang, W. Luo, W. Xue, Q. Liu, X. Chu, and Y . Guo. Discovering sparsity allocation for layer-wise pruning of large language models. InThe Thirty-eighth Annual Conference on Neu- ral Information Processing Systems, 2024. URL https://openrevie...

    2024

  21. [29]

    Y . Li, D. I. Campbell, S. C. Chan, and A. K. Lampinen. Just-in-time and distributed task representations in language models. In Mechanistic Interpretability Workshop at NeurIPS 2025 , 2025. URL https: //openreview.net/forum?id=KC13tBPbQg

    2025

  22. [30]

    Lindsey, A

    J. Lindsey, A. Templeton, J. Marcus, T. Conerly, J. Batson, and C. Olah. Sparse crosscoders for cross- layer features and model diffing.https://transformer-circuits.pub/2024/crosscoders/index. html, 2024. Transformer Circuits Thread

    2024

  23. [31]

    Lippl, T

    S. Lippl, T. McGee, K. Lopez, Z. Pan, P. Zhang, S. Ziadi, O. Eberle, and I. Momennejad. Algorithmic primitives and compositional geometry of reasoning in language models, 2026. URL https://arxiv. org/abs/2510.15987

    2026

  24. [32]

    D. C. Liu and J. Nocedal. On the limited memory BFGS method for large scale optimization.Mathematical programming, 45(1):503–528, 1989

    1989

  25. [33]

    S. Liu, H. Ye, L. Xing, and J. Zou. In-context vectors: making in context learning more effective and controllable through latent space steering. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024

    2024

  26. [34]

    Marks, C

    S. Marks, C. Rager, E. J. Michaud, Y . Belinkov, D. Bau, and A. Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. In The Thirteenth International Conference on Learning Representations, 2025. 11

    2025

  27. [35]

    Montavon, A

    G. Montavon, A. Binder, S. Lapuschkin, W. Samek, and K.-R. Müller. Layer-wise relevance propagation: an overview. Explainable AI: interpreting, explaining and visualizing deep learning, pages 193–209, 2019

    2019

  28. [36]

    N. Nanda. Attribution patching: Activation patching at industrial scale. URL: https://www. neelnanda. io/mechanistic-interpretability/attribution-patching, 15:19, 2023

    2023

  29. [37]

    Nanda and J

    N. Nanda and J. Bloom. Transformerlens. https://github.com/TransformerLensOrg/ TransformerLens, 2022

    2022

  30. [38]

    Nikankin, A

    Y . Nikankin, A. Reusch, A. Mueller, and Y . Belinkov. Arithmetic without algorithms: Language models solve math with a bag of heuristics. In The Thirteenth International Conference on Learning Representa- tions, 2024

    2024

  31. [39]

    B. A. Olshausen and D. J. Field. Sparse coding of sensory inputs. Current Opinion in Neurobiology, 14(4):481–487, 2004. ISSN 0959-4388. doi: https://doi.org/10.1016/j.conb.2004.07.007. URL https: //www.sciencedirect.com/science/article/pii/S0959438804001035

    2004 doi

  32. [40]

    B. A. Olshausen et al. Principles of image representation in visual cortex. The visual neurosciences, 2: 1603–1615, 2003

    2003

  33. [41]

    Panigrahi, N

    A. Panigrahi, N. Saunshi, H. Zhao, and S. Arora. Task-specific skill localization in fine-tuned language models. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023

    2023

  34. [42]

    Qwen3 technical report, 2025

    Qwen Team. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388

    2025 arXiv

  35. [43]

    Rezaei Jafari, G

    F. Rezaei Jafari, G. Montavon, K.-R. Müller, and O. Eberle. MambaLRP: Explaining selective state space sequence models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, ...

    2024 doi

  36. [44]

    Rimsky, N

    N. Rimsky, N. Gabrieli, J. Schulz, M. Tong, E. Hubinger, and A. Turner. Steering llama 2 via contrastive activation addition. In L.-W. Ku, A. Martins, and V . Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: L...

    2024 doi

  37. [45]

    Smilkov, N

    D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825, 2017

    2017 arXiv

  38. [46]

    Socher, A

    R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In D. Yarowsky, T. Baldwin, A. Korhonen, K. Livescu, and S. Bethard, editors, Proceedings of the 2013 Conference on Emp...

    2013

  39. [47]

    L. Stella. ProximalAlgorithms.jl: Proximal algorithms for nonsmooth optimization in Julia. URL https://github.com/JuliaFirstOrder/ProximalAlgorithms.jl

  40. [48]

    Sundararajan, A

    M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research, pages 3319–3328. PMLR, 06–11 Aug 2017. URL https://proceedings.mlr.pr...

    2017

  41. [49]

    Themelis, L

    A. Themelis, L. Stella, and P. Patrinos. Forward-backward envelope for the sum of two nonconvex functions: Further properties and nonmonotone linesearch algorithms. SIAM Journal on Optimization, 28 (3):2274–2303, 2018. doi: 10.1137/16M1080240

    2018 doi

  42. [50]

    E. Todd, M. Li, A. S. Sharma, A. Mueller, B. C. Wallace, and D. Bau. Function vectors in large language models. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=AwyxtyMwaG

    2024

  43. [51]

    Vargas-Barroso, J

    V . Vargas-Barroso, J. F. Watson, A. Navas-Olive, A. Schlögl, and P. Jonas. Developmental emergence of sparse and structured synaptic connectivity in the hippocampal ca3 memory circuit. Nature Communica- tions, 2026. 12

    2026

  44. [52]

    X. Wang, K. Wen, Z. Zhang, L. Hou, Z. Liu, and J. Li. Finding skill neurons in pre-trained transformer- based language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11132–11152, 2022

    2022

  45. [53]

    E. B. Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209–212, 1927

    1927

  46. [54]

    Xiong, Z

    Z. Xiong, Z. Cai, J. Cooper, A. Ge, V . Papageorgiou, Z. Sifakis, A. Giannou, Z. Lin, L. Yang, S. Agarwal, G. Chrysos, S. Oymak, K. Lee, and D. Papailiopoulos. Everything everywhere all at once: LLMs can in-context learn multiple tasks in superposition. In Forty-second Interna...

    2025

  47. [55]

    Q: input 1\nA: output1\n\nQ: input2\nA: output2\n\n[...]Q: inputk\nA: outputk\n\nQ: input\nA:

    F. Yin, X. Ye, and G. Durrett. Lofit: Localized fine-tuning on LLM representations. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 9474–9506. Curran Associates, ...

    2024

  48. [56]

    How are you feeling today?

    and a maximum of 200 generated tokens. During the autoregressive answering process, we intervene on the model on each forward pass. Some special characters (such as emotes) that could not be correctly displayed were stripped. Prompt 1: How are you feeling today? Baseline: I’m ...

    2014

Pith tools

Reviewed July 9, 2026 · model on record in the stance chip above.