Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Stochastic Parameter Decomposition

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Stochastic Parameter Decomposition learns, per rank-one piece of a network's weights, how much that piece can be randomly ablated without changing outputs, and this suffices to recover known mechanisms in every toy model tested.

desk verdict A genuine, clearly-written method advance over APD, but the robustness claim is only as strong as the beta3 selection rule, which still leans on known ground truth. read the letter →

arxiv 2506.20790 v2 pith:IEORHBU5 submitted 2025-06-25 cs.LG cs.AI

Lucius Bushnaq , Dan Braun , Lee Sharkey This is my paper
classification cs.LGcs.AI MSC 68T07
keywords stochasticparameterdecompositionlinearmechanisticinterpretabilitycausalimportanceablatabilityrank-onefactorizationmediationanalysisnetwork
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Stochastic Parameter Decomposition (SPD), a method for splitting a trained network's weights into a sum of rank-one subcomponents (each a single outer product of two vectors) so that on any given input only a small number of them are causally required. The central move is to replace gradient attributions with a learned causal importance function that predicts, for each subcomponent, how much it can be randomly ablated without changing the network's output, and then to train the network to behave identically under such random ablations while penalizing any subcomponent for staying important. If the method works as claimed, it removes the two barriers that kept linear parameter decomposition impractical, computational cost and sensitivity to the number-of-active-components hyperparameter, and extends the approach to deeper and more complex models. Across every toy model tested, including two new ones that defeated the earlier attribution-based method, SPD recovers the known ground-truth mechanisms with near-perfect cosine alignment between learned and true components and no parameter shrinkage.

What carries the argument

The load-bearing object is the learned causal importance function: for each rank-one subcomponent $U^l_{:,c}V^l_{c,:}$ of weight matrix $W^l$, a small MLP maps the subcomponent's inner activation $h^l_c(x)=\sum_j V^l_{c,j}a^l_j(x)$ to an importance value $g^l_c(x)\in[0,1]$ through a leaky hard sigmoid. The importance value sets the lower end of a uniform mask $m^l_c(x,r)=g^l_c(x)+(1-g^l_c(x))r$ with $r\sim U(0,1)$, so a subcomponent predicted unimportant is aggressively and randomly ablated, while a predicted-important one stays near full strength. Three objectives compete: faithfulness (the subcomponents sum to the original weights), stochastic reconstruction (the randomly masked network reproduces the target's outputs, both globally and layerwise), and minimality (the $\ell^p$ magnitude of the importance values is penalized). The mechanism works because the stochastic masks probe every joint ablation pattern with some probability, so a subcomponent that is genuinely load-bearing on an input cannot afford a high ablation probability and keeps its importance high, whereas superfluous subcomponents are driven to importance zero.

What would settle it

Construct a small network with a known mechanism whose causal importance genuinely depends on activity elsewhere in the network, for example a gated circuit where a downstream layer decides whether an upstream rank-one component matters on a given input. Apply SPD with its per-subcomponent, own-activation-only importance MLPs: if the learned importance values systematically disagree with brute-force measurement of how much each subcomponent can actually be ablated on inputs where the cross-layer dependence binds, the expressivity assumption behind the method gives way.

Watch

Extended reading notes

Core claim

The paper claims that a neural network can be decomposed into its causal mechanisms without computing any attributions. For each weight matrix $W^l$ the method learns a set of rank-one subcomponents $\sum_c U^l_{:,c}V^l_{c,:}$ together with a small learned predictor that outputs an importance value $g^l_c(x)\in[0,1]$ for each subcomponent on each input; importance is defined operationally as the complement of ablatability, the amount by which the subcomponent can be randomly masked while leaving the output unchanged. On every input the subcomponents are masked by random amounts between their predicted importance and full strength, and the masked network is trained to reproduce the target output while a minimality penalty drives all importance values toward zero. The demonstrated result is that this stochastic ablation objective aligns subcomponents with ground-truth mechanisms: in toy models of superposition, compressed computation, and cross-layer distributed representations the learned decompositions match the known mechanisms up to small error, with mean max cosine similarity of 1.000 and mean L2 ratio close to 1, and without the shrinkage the attribution-based predecessor exhibited. In the compressed-computation model the method also reverses a mechanistic conclusion of the earlier method, finding that the MLP output matrix is one high-rank component rather than one per input feature.

Load-bearing premise

The method assumes that a subcomponent's ablatability on a given input is predictable from a single scalar, that subcomponent's own inner activation, and the authors concede that arbitrary models will likely require more expressive importance functions that take in more of the network.

Editorial extensions

If this is right

  • SPD decomposes a three-layer residual MLP that the attribution-based method could not handle, with masks showing one subcomponent per input feature and a single output component spread across all three layers.
  • In every toy model tested the learned subcomponents recover the ground-truth mechanisms with mean max cosine similarity 1.000 and mean L2 ratio near 1, indicating correct orientation and no shrinkage of magnitudes.
  • On the Toy Model of Compressed Computation SPD finds that the MLP output matrix is a single rank-50 component, contradicting the earlier attribution-based decomposition's claim of one component per input feature.
  • SPD decomposes an identity matrix inserted into a superposition model into exactly $m_1$ rank-one subcomponents that sum to it, demonstrating the method does not suffer the feature-splitting failure of activation-space dictionary learning.
  • Because it needs no estimate of how many components are active per datapoint and lets gradients reach every subcomponent on every step, the authors expect SPD to scale to larger, non-toy models such as language models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the local-importance assumption survives scaling, the practical bottleneck of the approach shifts to clustering: SPD produces per-layer rank-one pieces, and a general algorithmic grouping step, which the toy-model study avoids because ground truth names the clusters, will decide whether the method works on real networks.
  • The learned importance values double as a basis-free causal map: the masks specify which parameter vectors matter for which inputs, so SPD outputs could serve as circuit hypotheses that activation-patching or causal-scrubbing tests would then verify, without choosing units in advance.
  • A stress test the paper does not run: applying SPD to the embedding and unembedding matrices of a small transformer, where the one-component-per-token expectation can be checked directly and would give the first evidence the method transfers beyond toy MLPs.
  • The reported mean L2 ratios slightly exceed 1 (up to 1.031), a mild magnitude overshoot opposite in sign to the predecessor's shrinkage; whether this bias grows with depth and couples to the importance penalty $\beta_3$ is left open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Stochastic Parameter Decomposition (SPD), a method for decomposing a neural network's weight matrices into rank-one subcomponents that sum to the original weights (faithfulness), are active as sparsely as possible per input (minimality), and are simple by construction. SPD trains a per-subcomponent causal importance function that predicts how ablatable each subcomponent is on a given input, and uses stochastically sampled masks to encourage the masked model to match the target model. The full loss is Eq. (8), combining a faithfulness term, two stochastic reconstruction losses, and an importance-minimality penalty with coefficient beta3. The paper evaluates SPD on several toy models with known ground-truth mechanisms (Toy Model of Superposition, TMS with a hidden identity, Toy Model of Compressed Computation, and two- and three-layer Cross-Layer Distributed Representations), reporting MMCS and ML2R near 1 for the superposition models and qualitative agreement for the others. The authors claim SPD is more scalable and robust to hyperparameters than Attribution-based Parameter Decomposition (APD), and release code and training details.

Significance. If the empirical claims hold, SPD is a meaningful step for linear parameter decomposition and mechanistic interpretability: it replaces gradient-based attributions with a directly trained causal importance model, avoids the top-k hyperparameter of APD, and is demonstrated on slightly larger and deeper toy models than APD could handle. The paper is commendably concrete: it releases code, provides pseudocode (Algorithm 1), reports hyperparameters in Appendix A.4, and includes the authors' own admission of unresolved issues, notably an unexplained skew in the two-layer model (Section 3.4, Figure 10) and the lack of a general clustering procedure. The central advertised property, however, is robustness to hyperparameters, and the evidence for that property is currently entangled with knowledge of the ground truth, as detailed in the major comments. The significance is therefore conditional: the method has clear promise, but the paper does not yet establish the robustness claim it advertises.

major comments (4)
  1. [§2.3.3, Eq. (7); §3.3, Fig. 8; Appendix A.3] The central empirical claim that SPD identifies ground-truth mechanisms is not yet separated from knowledge of the ground truth. Section 3.3, Figure 8 shows that when beta3 is too small, the learned masks become non-monosemantic, but the decomposition remains faithful in the sense that the masked model reconstructs the target output; the Appendix A.3 heuristics 'Negligible performance loss' and 'Other sanity checks' do not rule out this regime, because the bad regime has importance values equal to 1 everywhere and no reconstruction failure. Appendix A.4.1 then reports beta3=3e-3 with p=1 for TMS5-2/TMS5-2+ID and beta3=1e-4 with p=2 for TMS40-10/TMS40-10+ID, a 30-fold variation, and the 'Recovering known mechanisms' heuristic explicitly uses the known decomposition to select beta3. Consequently, the MMCS=1.000 results in Table 1 may reflect selecting hyperparameters from the answer rather than a robust property of SPD. Please quantify the beta3 working range for at least one model and provide a ground-truth-free selection rule, for example a validation criterion that penalizes non-minimality without access to known mechanisms.
  2. [§3.4, Fig. 10] The two-layer Cross-Layer Distributed Representations result is presented as a success, yet Figure 10 shows a systematic skew toward higher values on the Y-axis, and the text states 'We currently do not understand the source of this discrepancy' and notes sensitivity to random seeds. Because the paper's central claim includes that SPD succeeds on this model, the unexplained systematic error and seed sensitivity should be resolved or quantified. Please report the number of seeds, the spread of the neuron-contribution agreement and MMCS across seeds, and whether the skew persists after hyperparameter search; without this, the reported result may reflect a favorable run rather than a stable property of SPD.
  3. [§3, Eq. (9)] The MMCS metric in Eq. (9) takes a maximum over all C subcomponents. When C is large, as in the 2-layer and 3-layer models where C=400 and C=500 (Appendix A.4.2), random or spurious subcomponents can inflate the max, and the metric does not penalize the presence of extra subcomponents. Table 1 reports MMCS=1.000 to three decimal places for four models, which is strong; please report the full distribution of cosine similarities, the number of non-negligible subcomponents learned per model, and a precision/recall-style statistic that penalizes spurious subcomponents, so the quantitative claim is not driven by the max over many candidates.
  4. [§5, footnote 6] The identification argument in Section 5, footnote 6 asserts that in an idealized setting, any beta3 infinitesimally larger than zero makes the desired decomposition the global optimum, but this claim is not proved and is stated conditionally on several idealizations. The toy experiments do not establish robustness to beta3; they establish that a few hand-picked beta3 values recover known mechanisms. Please either prove the global-optimum claim for the specific linear and ReLU toy models, or replace it with a clearly delimited empirical claim about the beta3 working range, and report how the quantitative metrics vary across beta3 values for each model.
minor comments (5)
  1. [§2.3.1 and Appendix A.2] The causal importance function is defined as outputting values in [0,1], but the leaky hard sigmoids in Appendix A.2 deliberately produce outputs outside this range; the text should clarify how the mask sampling in Eq. (3) and the importance penalty in Eq. (7) treat out-of-range values.
  2. [§3.2] The sentence 'APD fails to learn ground truth mechanisms in this model' is stated without a quantitative comparison or a reference to a figure/table; please add a quantitative comparison or move the claim to a discussion section.
  3. [§2.3.2, Eq. (5)] The statement that Eq. (5) is equivalent to Eq. (4) 'if the subcomponents sum to the original weights' should be expanded: the equivalence also requires that the layerwise losses are combined with weights that match the sampling distribution, and the current wording is too terse.
  4. [§3.3] The reference to 'unpublished work, forthcoming' should be replaced by a citation or removed, and the 'WandB report here' links should include stable URLs or DOIs.
  5. [Figure 8] The caption and text mix 'left'/'right' with 'left two columns' and 'right two columns'; please standardize the direction labels so the too-low, just-right, and too-high regimes are unambiguous.

Circularity Check

1 steps flagged · score 4.0 of 10

SPD's headline recovery results are partly fitted inputs: Appendix A.3 recommends choosing the sparsity coefficient β3 using the same known ground-truth mechanisms whose recovery is then reported as the headline result.

  1. fitted input called prediction [Appendix A.3, Heuristics for Hyperparameter Selection]
    "Recovering known mechanisms: If some of the ground truth mechanisms in the target model are already known, we can restrict hyperparameters such that they recover those mechanisms. For example, in a language model, the embedding matrix mechanisms are usually known: Each vocabulary element should be assigned one mechanism. If none of a model's mechanisms are known to start with, we could insert known mechanisms into it."

    The paper's headline quantitative result—MMCS = 1.000 and ML2R ≈ 1 in Table 1, read as evidence that SPD 'seems to identify known ground-truth mechanisms up to a small error'—is obtained after selecting β3, the only coefficient in Equation 8 that drives sparsity, using exactly those known ground-truth mechanisms. The Appendix A.3 heuristic explicitly tells the user to narrow hyperparameters until known mechanisms are recovered. Since a too-small β3 produces faithful but non-minimal decompositions (g≈1 everywhere) that still pass the other A.3 sanity checks, the selection against ground truth is load-bearing: the reported recovery is not an independent prediction but a hyperparameter chosen to produce it.

full rationale

SPD is not circular in the strong sense: it optimizes the loss in Equation 8 and compares learned subcomponents to external ground-truth weight columns, and the loss does not directly encode the target mechanisms. The central derivation (Section 2) is self-contained: faithfulness, stochastic reconstruction, and importance-minimality are defined independently of the benchmark metrics. However, the paper's robustness and identification claims are weakened by a fitted-input step. Appendix A.3's 'Recovering known mechanisms' heuristic selects the sparsity coefficient β3 by checking whether the decomposition recovers known mechanisms; the same known mechanisms are then used to compute the MMCS/ML2R results in Table 1. Because β3 is the sole term in Equation 8 pushing toward minimality, a too-small value collapses the learned causal importance to g≈1, making the stochastic reconstruction losses trivially satisfied while the decomposition remains faithful but non-minimal. The paper acknowledges this failure mode in Section 3.3 ('When the importance loss coefficient β3 is too small...', calling it 'a poor decomposition') and shows only three β3 values in Figure 8 without quantifying a working range; the per-model β3 values in Appendix A.4.1 differ by more than an order of magnitude (3e-3 vs 1e-4). Consequently, the claim in Section 5 that SPD is 'considerably more scalable and robust to hyperparameters than APD' is not supported by a ground-truth-free selection rule, and the recovery demonstration is partly a re-statement of the tuning criterion. Self-citations to Braun et al. [2025] provide context and a baseline but are not load-bearing for SPD's own derivation. Overall, the circularity is limited to hyperparameter selection rather than the method's mathematical core, so a moderate score of 4 is appropriate.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several domain assumptions about the sufficiency of rank-one, per-layer subcomponents and the expressiveness of the importance functions, plus a set of free hyperparameters tuned per model.

free parameters (5)
  • beta3 (importance minimality coefficient) = 3e-3 (TMS5-2, TMS5-2+ID); 1e-4 (TMS40-10, TMS40-10+ID); 1e-5 (1- and 2-layer residual MLPs); 0.5e-5 (3-layer)
    Tuned per model; Appendix A.4 lists model-specific values, and Section 3.3 shows results are sensitive to beta3.
  • C (subcomponents per layer) = 20 (TMS); 100 (1-layer); 400 (2-layer); 500 (3-layer)
    Chosen per model; enough to cover rank of each matrix plus slack.
  • p in L_importance-minimality = 1 (TMS5-2, TMS5-2+ID); 2 (others)
    Authors say they expect similar results with many settings; values are model-specific.
  • S (mask samples per step) = 1
    Set to 1 for all experiments; authors find it sufficient without evidence for general cases.
  • learning rate = 1e-3 (TMS); 2e-3 (1-layer); 1e-3 (2- and 3-layer)
    Model-specific optimizer hyperparameter.
assumptions (4)
  • domain assumption Rank-one subcomponents within a single layer can represent the network's mechanisms
    Section 2.1 introduces this decomposition; cross-layer clustering is left to future work, so mechanisms spanning layers are not directly identified.
  • domain assumption A subcomponent's causal importance can be predicted from its own inner activation alone
    Section 2.3.3; the authors note more expressive importance functions may be needed for arbitrary models.
  • ad hoc to paper The stochastic objective (Eq. 8) has a global optimum at the ground-truth decomposition
    No proof is given; empirical evidence is provided for toy models, with seed sensitivity in multi-layer models (Section 3.4).
  • domain assumption The hard-sigmoid gate with leaky slopes and S=1 suffices for stable training
    Appendix A.2 justifies leaky sigmoids; the sufficiency of S=1 is an empirical choice (Section 2.4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Stochastic Parameter Decomposition." pith.science (2026). https://pith.science/paper/IEORHBU5

@misc{pith2026250620790,
  author       = {Pith},
  title        = {Pith review of: Stochastic Parameter Decomposition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IEORHBU5}},
  note         = {Machine review of arXiv:2506.20790}
}
read the original abstract

A key step in reverse engineering neural networks is to decompose them into simpler parts that can be studied in relative isolation. Linear parameter decomposition -- a framework that has been proposed to resolve several issues with current decomposition methods -- decomposes neural network parameters into a sum of sparsely used vectors in parameter space. However, the current main method in this framework, Attribution-based Parameter Decomposition (APD), is impractical on account of its computational cost and sensitivity to hyperparameters. In this work, we introduce \textit{Stochastic Parameter Decomposition} (SPD), a method that is more scalable and robust to hyperparameters than APD, which we demonstrate by decomposing models that are slightly larger and more complex than was possible to decompose with APD. We also show that SPD avoids other issues, such as shrinkage of the learned parameters, and better identifies ground truth mechanisms in toy models. By bridging causal mediation analysis and network decomposition methods, this demonstration opens up new research possibilities in mechanistic interpretability by removing barriers to scaling linear parameter decomposition methods to larger models. We release a library for running SPD and reproducing our experiments at https://github.com/goodfire-ai/spd/tree/spd-paper.

Figures

Figures reproduced from arXiv: 2506.20790 by the authors.

Figure 1
Figure 1. Results of running SPD on TMS5−2. Top row: Plots of (left to right) the columns of the weight matrix of the target model; the sum of the SPD parameter components; and individual parameter components. Although this run of SPD used 20 subcomponents, only 6 subcomponents are shown, ordered by the sum of the norms of each of the columns of their (rank-one) weight matrices. The first five have learned one direction each,… view at source ↗
Figure 2
Figure 2. Plots of (left to right) the columns of the input weight matrix [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Plots of (left to right) the weights in the hidden identity matrix [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Plots of (left to right) the TMS5−2+ID networks parametrized by: The target model parameters; the sum of all parameter subcomponents found by SPD the decomposition of the model; and the seven individual subcomponents of non-negligible size. We see that SPD finds five s…
Figure 5
Figure 5. Figure 5: The architecture of the Toy Model of Compressed Computation. It uses a [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Toy Model of Compressed Computation: Similarity between target model weights and SPD [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Toy Model of Compressed Computation: Similarity between target model weights and [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Causal importance values of each subcomponent (clipped between [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: The architecture of one of our two Toy models of Cross-Layer Distributed representations. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Toy Model of Distributed Representations (Two Layers): Similarity between target [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Toy Model of Distributed Representations (Three Layers): Similarity between target model [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Toy Model of Distributed Representations (Two Layers): Causal importance values of each [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Toy Model of Distributed Representations (Three Layers): Causal importance val [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Toy Model of Distributed Representations (Two Layers): Similarity between target [PITH_FULL_IMAGE:figures/full_fig_p027_14.png]
Figure 15
Figure 15. Figure 15: Toy Model of Distributed Representations (Three Layers): Similarity between target [PITH_FULL_IMAGE:figures/full_fig_p028_15.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Individual Parameters in Weight-Sparse Transformers Appear Interpretable

    cs.LG 2026-07 conditional novelty 6.5 of 10

    An automated LLM pipeline finds that 12–31% of nonzero weights in weight-sparse transformers admit short, held-out-validated descriptions of when they matter, far above dense controls.

  2. Targeted Recovery of Weight-Space Mechanisms From Neural Networks

    cs.LG 2026-06 conditional novelty 6.0 of 10

    A targeted decomposition method recovers the weight-space mechanisms behind specific inputs at low FLOPs, enabling focused ablation and rewiring of a 12-block transformer.

Reference graph

Works this paper leans on

45 extracted references · 19 canonical work pages · cited by 2 Pith papers

  1. [1]

    Sanity checks for saliency maps

    Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_file...

    work page 2018

  2. [2]

    Compressed computation is (probably) not computation in superposition

    Jai Bhagat, Sara Molas Medina, Giorgi Giglemiani, and Stefan Heimersheim. Compressed computation is (probably) not computation in superposition. https://www.lesswrong.com/posts/ZxFchCFJFcgysYsT9/compressed-computation-is-probably-not-computation-in, 2025

    work page 2025

  3. [3]

    Interpretability in parameter space: Minimizing mechanistic description length with attribution-based parameter decomposition, 2025

    Dan Braun, Lucius Bushnaq, Stefan Heimersheim, Jake Mendel, and Lee Sharkey. Interpretability in parameter space: Minimizing mechanistic description length with attribution-based parameter decomposition, 2025. URL https://arxiv.org/abs/2501.14926

    arXiv 2025

  4. [4]

    Towards monosemanticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Ch...

    work page 2023

  5. [5]

    Steven Cao, Victor Sanh, and Alexander M. Rush. Low-complexity probing via finding subnetworks, 2021. URL https://arxiv.org/abs/2104.03514

    arXiv 2021

  6. [6]

    Causal scrubbing: a method for rigorously testing interpretability hypotheses [redwood research], December 2022

    Lawrence Chan, Adrià Garriga-Alonso, Nicholas Goldowsky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. Causal scrubbing: a method for rigorously testing interpretability hypotheses [redwood research], December 2022. URL https://www.alignmentforum.org/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-ri...

    work page 2022

  7. [7]

    A is for absorption: Studying feature splitting and absorption in sparse autoencoders, 2024

    David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, and Joseph Bloom. A is for absorption: Studying feature splitting and absorption in sparse autoencoders, 2024. URL https://arxiv.org/abs/2409.14507

    arXiv 2024

  8. [8]

    Identifying Sparsely Active Circuits Through Local Loss Landscape Decomposition

    Brianna Chrisman, Lucius Bushnaq, and Lee Sharkey. Identifying sparsely active circuits through local loss landscape decomposition, 2025. URL https://arxiv.org/abs/2504.00194

    work page Pith review arXiv 2025

Show all 45 references
  1. [9]

    Towards automated circuit discovery for mechanistic interpretability

    Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adri\` a Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Inform...

    2023

  2. [10]

    Are neural nets modular? inspecting functional modularity through differentiable weight masks

    R \'o bert Csord \'a s, Sjoerd van Steenkiste, and J \"u rgen Schmidhuber. Are neural nets modular? inspecting functional modularity through differentiable weight masks. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=7uVcpu-gMD

    2021

  3. [11]

    Sparse autoencoders find highly interpretable features in language models

    Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=F76bwRSLeK

    2024

  4. [12]

    Toy models of superposition, 2022

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposi...

    2022

  5. [13]

    Fong and Andrea Vedaldi

    Ruth C. Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, October 2017. doi:10.1109/iccv.2017.371. URL http://dx.doi.org/10.1109/ICCV.2017.371

    2017 doi

  6. [14]

    Scaling and evaluating sparse autoencoders, 2024

    Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, 2024. URL https://arxiv.org/abs/2406.04093

    2024 arXiv

  7. [15]

    Causal abstractions of neural networks

    Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2024 a . Curran Associates Inc. ISBN 9781713845393

    2024

  8. [16]

    Finding alignments between interpretable causal variables and distributed neural representations

    Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. Finding alignments between interpretable causal variables and distributed neural representations. In Francesco Locatello and Vanessa Didelez, editors, Proceedings of the Third Conference on Causal...

    2024

  9. [17]

    Mathematical models of computation in superposition, 2024

    Kaarel Hänni, Jake Mendel, Dmitry Vaintrob, and Lawrence Chan. Mathematical models of computation in superposition, 2024. URL https://arxiv.org/abs/2408.05451

    2024 arXiv

  10. [18]

    Tanh penalty in dictionary learning

    Adam Jermyn, Adly Templeton, Joshua Batson, and Trenton Bricken. Tanh penalty in dictionary learning. https://transformer-circuits.pub/2024/feb-update/index.html#: :text=handle

    2024

  11. [19]

    Auto-encoding variational bayes, 2013

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2013. URL https://arxiv.org/abs/1312.6114

    2013 arXiv

  12. [20]

    Atp*: An efficient and scalable method for localizing llm behaviour to components, 2024

    János Kramár, Tom Lieberum, Rohin Shah, and Neel Nanda. Atp*: An efficient and scalable method for localizing llm behaviour to components, 2024. URL https://arxiv.org/abs/2403.00745

    2024 arXiv

  13. [21]

    Sparse autoencoders do not find canonical units of analysis, 2025

    Patrick Leask, Bart Bussmann, Michael Pearce, Joseph Bloom, Curt Tigges, Noura Al Moubayed, Lee Sharkey, and Neel Nanda. Sparse autoencoders do not find canonical units of analysis, 2025. URL https://arxiv.org/abs/2502.04878

    2025 arXiv

  14. [22]

    David K. Lewis. Counterfactuals. Blackwell, Malden, Mass., 1973

    1973

  15. [23]

    Sparse crosscoders for cross-layer features and model diffing, October 2024

    Jack Lindsay, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christopher Olah. Sparse crosscoders for cross-layer features and model diffing, October 2024. URL https://transformer-circuits.pub/2024/crosscoders/index.html

    2024

  16. [24]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https://arxiv.org/abs/1711.05101

    2019 arXiv

  17. [25]

    Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller

    Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models, 2024. URL https://arxiv.org/abs/2403.19647

    2024 arXiv

  18. [26]

    Uncovering model processing strategies with non-negative per-example fisher factorization, 2025

    Michael Matena and Colin Raffel. Uncovering model processing strategies with non-negative per-example fisher factorization, 2025. URL https://arxiv.org/abs/2310.04649

    2025 arXiv

  19. [27]

    The hydra effect: Emergent self-repair in language model computations, 2023

    Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations, 2023. URL https://arxiv.org/abs/2307.15771

    2023 arXiv

  20. [28]

    SAE feature geometry is outside the superposition hypothesis

    Jake Mendel. SAE feature geometry is outside the superposition hypothesis. Alignment Forum, 2024. URL https://www.alignmentforum.org/posts/MFBTjb2qf3ziWmzz6/sae-feature-geometry-is-outside-the-superposition-hypothesis

    2024

  21. [29]

    Skeletonization: A technique for trimming the fat from a network via relevance assessment

    Michael C Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume 1. Morgan-Kaufmann, 1988. URL https://proceedings.neurips.cc/paper_...

    1988

  22. [30]

    Missed causes and ambiguous effects: Counterfactuals pose challenges for interpreting neural networks, 2024

    Aaron Mueller. Missed causes and ambiguous effects: Counterfactuals pose challenges for interpreting neural networks, 2024. URL https://arxiv.org/abs/2407.04690

    2024 arXiv

  23. [31]

    The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability, 2024

    Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Arnab Sen Sharma, Jiuding Sun, Eric Todd, David Bau, and Yonatan Belinkov. The quest for the right mediator: A history, survey, and theoretical grounding...

    2024

  24. [32]

    Taking features out of superposition with sparse autoencoders, Dec 2022

    Lee Sharkey, Dan Braun, and Beren Millidge. Taking features out of superposition with sparse autoencoders, Dec 2022. URL https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj/interim-research-report-taking-features-out-of-superposition

    2022

  25. [33]

    Michaud, Stephen Casper, Max Tegmark, William Saunders, David Bau, Eric Todd, Atticus Geiger, Mor Geva, Jesse Hoogland, Daniel Murfet, and Tom McGrath

    Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, Stella Biderman, Adria Garriga-Alonso, Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi Schoots, ...

    2025 arXiv

  26. [34]

    Deep inside convolutional networks: Visualising image classification models and saliency maps, 2014

    Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps, 2014. URL https://arxiv.org/abs/1312.6034

    2014 arXiv

  27. [35]

    Axiomatic attribution for deep networks, 2017

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks, 2017

    2017

  28. [36]

    Attribution patching outperforms automated circuit discovery

    Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen, editors, Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpret...

    2024 doi

  29. [37]

    true features

    Demian Till. Do sparse autoencoders find "true features"?, February 2024. URL https://www.lesswrong.com/posts/QoR8noAB3Mp2KBA4B/do-sparse-autoencoders-find-true-features

    2024

  30. [38]

    Investigating gender bias in language models using causal mediation analysis

    Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Ne...

    2020

  31. [39]

    Differentiation and specialization of attention heads via the refined local learning coefficient, 2024

    George Wang, Jesse Hoogland, Stan van Wingerden, Zach Furman, and Daniel Murfet. Differentiation and specialization of attention heads via the refined local learning coefficient, 2024. URL https://arxiv.org/abs/2410.02984

    2024 arXiv

  32. [40]

    Interpretability in the wild: a circuit for indirect object identification in gpt-2 small

    Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022

    2022 arXiv

  33. [41]

    Algebraic geometry and statistical learning theory, volume 25

    Sumio Watanabe. Algebraic geometry and statistical learning theory, volume 25. Cambridge university press, 2009

    2009

  34. [42]

    David S. Watson. Conceptual challenges for interpretable machine learning. Synthese, 200: 0 65, 2022. doi:10.1007/s11229-022-03485-5. URL https://doi.org/10.1007/s11229-022-03485-5

    2022 doi

  35. [43]

    Addressing feature suppression in saes, Feb 2024

    Benjamin Wright and Lee Sharkey. Addressing feature suppression in saes, Feb 2024. URL https://www.alignmentforum.org/posts/3JuSjTZyMzaSeTxKk/addressing-feature-suppression-in-saes

    2024

  36. [44]

    Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors

    Zeyu Yun, Yubei Chen, Bruno Olshausen, and Yann LeCun. Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors. In Eneko Agirre, Marianna Apidianaki, and Ivan Vuli \'c , editors, Proceedings of Deep Learning ...

    2021 doi

  37. [45]

    Can subnetwork structure be the key to out-of-distribution generalization?, 2021

    Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, and Aaron Courville. Can subnetwork structure be the key to out-of-distribution generalization?, 2021. URL https://arxiv.org/abs/2106.02890

    2021 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.