REVIEW 3 major objections 5 minor 29 references
Sparse Weight Decomposition for Efficient Circuit Extraction
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Sparse Weight Decomposition rewrites a dense transformer's weight matrices as products of two sparse factors, turning each shared coordinate into an individually addressable, ablatable circuit unit without training any auxiliary…
desk verdict Honest, well-controlled empirical paper that repurposes DSF for circuit extraction; the main caveat is that all circuit conclusions rest on an approximate factorization whose task-level fidelity is never directly tested. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the sparse read–write bottleneck unit: the rank-one path $A_{:,i}B_{i,:}$ formed by one column of the sparse factor $A$ and one row of the sparse factor $B$, so unit $i$ reads inputs only through the support of $A_{:,i}$ and writes outputs only through the support of $B_{i,:}$. The two factors are fit under a total nonzero budget $K$ by alternating $\ell_0$-constrained regressions — the Double Sparse Factorization heuristic with ADMM inner updates, optionally weighted by the calibration Gram matrix $G = \mathbb{E}[h^\top h]$ — so the unit surface lives directly in the pretrained weights rather than in learned activation features. This machinery does three jobs at once: it partitions a dense map into independently ablatable additive paths, it makes circuit cost countable (each selected unit's active edges are simply the nonzeros of its read and write vectors), and it supplies the control that isolates sparsity from factorization, since dense SVD or random-orthogonal reparameterizations of the same matrix lack the edge savings.
What would settle it
Replay a circuit extracted from $AB$ on the original dense model by materializing each selected unit's rank-one update $\alpha A_{:,i}B_{i,:}$ directly in $W$: if a held-out task input changes the dense model's task margin against the direction predicted from $AB$ while average cross-entropy is matched, the extracted circuit is an artifact of the factorization rather than a property of the pretrained model.
Extended reading notes
Core claim
The central claim is that a dense pretrained linear projection can be reparameterized as $W \approx AB$ with both factors sparse, and that the intermediate coordinates of this factorization form a usable causal unit surface for circuit extraction. Each bottleneck unit $i$ is the rank-one path $A_{:,i}B_{i,:}$ with sparse read and write edges, so its scalar activation $z_i = h A_{:,i}$ can be scored by positive first-order task-margin attribution, and keeping or mean-ablating it changes only that unit's term in the sum $\sum_i z_i B_{i,:}$. Because the factors are fit directly to the checkpoint — minimizing the activation-weighted reconstruction error $\mathbb{E}\|hW - hAB\|_2^2$ subject to an $\ell_0$ budget on factor entries — no separate replacement network is trained. The paper reports that at matched pre-pruning cross-entropy, SWD reaches the same held-out sufficiency and necessity thresholds with fewer active read/write edges and selected units than Transcoder and VPD baselines on GPT-2, Qwen2.5, and Qwen3.5-27B; that fixed-support fine-tuning extends the approach to replacing all 48 attention and MLP matrices of GPT-2 Small at competitive loss; that exact dense factorizations (SVD, random-orthogonal) do not reproduce the edge savings, isolating sparse read/write structure as the cause; and that a zero-data variant with $G = I$ still produces useful task circuits.
Load-bearing premise
Circuit conclusions are drawn from interventions on the approximate product $AB$, not on the original dense weights $W$, and matching average cross-entropy does not guarantee that $AB$ agrees with $W$ on the particular inputs where the task behavior lives.
Editorial extensions
If this is right
- Circuit analysis of dense transformers no longer requires training sparse autoencoders, Transcoders, or replacement modules; the same scoring–selection–ablation workflow applies directly to factorized weights, using under 1% of the data.
- Sparse read/write connectivity, not factorization alone, is what buys compact circuits: exact SVD and random-orthogonal reparameterizations of the same matrix need more active edges to hit the same sufficiency and necessity targets.
- The method scales: single-matrix replacement reaches matched fidelity on Qwen3.5-27B, and fixed-support fine-tuning extends SWD to replacing all 48 attention and MLP matrices of GPT-2 Small at loss competitive with sparse pretraining.
- The zero-data variant (Frobenius objective, no calibration activations) still yields units that support task circuits, enabling per-step mechanistic analysis of circuit structure during training.
- SWD bottleneck units support targeted edits and attention diagnostics: a single unit's read direction can implement a more local weight edit than rank-4 LoRA, and an ablated query unit can sharply redirect attention in a head where reconstruction is faithful.
Reading between the lines
- If $AB$ is faithful on the inputs where the task actually lives, SWD units become a cheap circuit-hypothesis generator: circuits found on $AB$ could be re-verified on the dense model by materializing the same sparse rank-one updates in $W$, a two-stage validation the paper does not run.
- The sparse-connectivity result suggests post-hoc factorization may serve as a fast proxy for sparse pretraining's interpretability benefits, letting researchers test circuit hypotheses on existing checkpoints before committing to training a weight-sparse model.
- A testable extension is trajectory analysis: run the zero-data factorization at successive checkpoints of a single training run and ask whether the same bottleneck units appear, consolidate, or vanish as task skill emerges; the paper leaves this explicitly to future work.
- Because matched cross-entropy only controls average drift, a natural next experiment is cross-method circuit agreement on the same task — whether SWD and Transcoder select the same computational paths when both are faithful — which would separate circuits that reflect the model from circuits that reflect the representation.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Sparse Weight Decomposition (SWD), which factorizes a dense weight matrix as W ≈ A B with sparse factors A and B, and treats the shared intermediate coordinates as addressable bottleneck units for circuit extraction. The authors evaluate replacement fidelity (CE delta, KL, activation relative MSE) and circuit cost-quality tradeoffs (sufficiency and necessity as functions of selected units and active edges) on GPT-2 Small, Qwen2.5 0.5B-3B, and Qwen3.5-27B, comparing against Transcoder, VPD, sparse pretraining, and exact dense SVD/Random-B controls. They report that SWD reaches matched replacement CE with less than 1% of the data used by trained baselines, reaches circuit targets with fewer active edges, remains competitive after fixed-support fine-tuning for full-model replacement on GPT-2 Small, and has a zero-data variant.
Significance. The paper is a carefully executed empirical study of a practical idea: reusing double sparse factorization as a cheap parameter-space source of circuit units. The experimental protocol has real strengths: pre-pruning CE is matched within 0.001, exact dense controls (full-rank SVD and Random-B) are used to separate the effect of sparse edges from factorization alone, zero-ablation robustness is checked, non-matched baselines are reported honestly, and the code and model links are provided. If the central claim holds, SWD would substantially lower the cost of obtaining causally testable units from dense checkpoints, and the method's applicability to attention and MLP projections is broader than activation-space replacements. The main risk is that the causal claims are made about the dense model while all interventions are performed on the approximate factorization, and the active-edge metric is partly built into the comparison with activation-space baselines.
major comments (3)
- [Sections 3.2-3.6, Section 5] Section 5 states that matched cross-entropy and reconstruction control average drift but cannot guarantee agreement with the original dense model on every input; yet all circuit interventions in Sections 3.3-3.6 are performed on the approximate replacement AB, never on the dense weight W. Because W = AB + R, the same top-k keep/ablate protocols can be applied to the dense model by holding the residual term R fixed and intervening only on the AB term's unit activations. I request a direct experiment comparing task margins and sufficiency/necessity curves of SWD circuits in AB versus in this dense-model implementation, evaluated on the circuit train/test splits. Without such a check, the abstract's claim that SWD 'reparameterizes' pretrained projections and yields circuits of the dense model is not established; the circuits may be properties of the factorization.
- [Section 3.2, Table 8] The matching rule in Section 3.2 only constrains pre-pruning CE delta (at most 0.001) on FineWeb-Edu, while Suff and NecDrop are defined relative to each method's own unpruned replacement Munpruned. This does not imply equal task margins Qτ(Munpruned; D) on the circuit splits; for example, Table 8 lists Qwen3.5-27B checkpoints with CE deltas of 0.001847 (VPD-Recon-CI) and 0.001121 (TC48k), which are outside the stated tolerance but are still used in circuit comparisons as 'references.' If Q_unpruned differs across methods, the same relative sufficiency or necessity drop does not correspond to the same absolute task behavior. Please report absolute held-out task margins for every method and either match Q_unpruned, normalize appropriately, or explicitly exclude outside-tolerance baselines from the 'matched-fidelity' language.
- [Section 3.2, Section 3.5, Table 1] The active-edge cost C_edge(S) counts nonzero read/write weights of selected units, but Transcoder and VPD units have dense parameter-side read/write vectors by construction: Table 1 classifies them as activation-sparse or input-dependent-selection methods, not weight-sparse methods. Comparing C_edge between SWD and these baselines therefore partly measures the representation choice rather than circuit quality; the exact dense controls in Section 3.5 confirm that dense factorizations need more edges, but they do not make Transcoder edges commensurable with SWD edges. The central abstract claim of 'fewer active read/write edges... than learned sparse baselines' should be supported by a normalized cost meaningful across parameter-space and activation-space methods, or the selected-unit axis should be made primary for cross-paradigm comparisons.
minor comments (5)
- [Table 1 footnote] The footnote 'This greater data use does not consistently reduce circuit cost' appears to say the opposite of what is meant; it should say 'lower data use' or 'data advantage.'
- [Section 3.6] The sentence 'zero-data SWD remains closer to W throughout the all sparsity levels' contains a typo and should read 'at all sparsity levels.'
- [Appendix C.5, Figure 6] TC48k and VPD-Recon-CI are outside the 0.001 CE-matching tolerance on Qwen3.5-27B; the main-text Figure 6 caption and legend should mark them as non-matched references rather than leaving this information only in the appendix.
- [Appendix E, Section 3.3 takeaway] The main-text takeaway states a broad SWD advantage, but Appendix E reports mixed results on the full-MLP surface, with the standard MLP Transcoder competitive in several selected-unit and sufficiency comparisons; the main-text claims should be qualified to the single-matrix surface.
- [Section 3.7.2, Appendix I] The targeted editing study uses a factorization fitted on WikiText-2 calibration data, whereas the main circuit protocol uses FineWeb-Edu; please specify whether unit c205 is drawn from the same factorization as the main experiments and whether the different calibration distribution affects unit selection.
Circularity Check
No circularity: SWD factors are fit to reconstruction, circuits are scored on held-out task data, and the sole self-citation is a non-load-bearing caveat.
full rationale
The derivation chain is self-contained. In Section 2.1, SWD fits A and B by minimizing activation-weighted reconstruction error on FineWeb-Edu calibration data, with no circuit-task labels in the objective. Bottleneck units are then scored by positive first-order task-margin attribution on a separate circuit-train split, and the fixed top-k prefixes are evaluated on held-out circuit-test data (Section 3.2), so the sufficiency and necessity curves are genuine out-of-sample measurements rather than refittings of the target. The comparison is controlled by requiring pre-pruning CE deltas within 0.001 and by reporting method-relative scores, which prevents a method from winning merely through a low unpruned task baseline. The active-edge advantage is measured structurally and is checked against exact dense SVD and Random-B controls (Section 3.5) and a zero-ablation robustness test (Appendix C.3), so it is not an artifact of the mean-ablation convention. The only self-citation is Sheng and Fu (2026) in Section 5, used to note that circuit claims can vary with reporting choices; that caveat is not load-bearing for any derivation. The paper's own limitation that matched cross-entropy does not guarantee per-input agreement between AB and W is a fidelity and validation concern, not a circularity: no prediction in the paper is defined in terms of its own output or fitted to the quantity it claims to predict.
Assumptions & free parameters
free parameters (5)
- sparsity level s =
0.5 and 0.75 in main matched comparisons; 0.125-0.875 in zero-data sweep
- factor-specific density split =
0.16 for square factor, 0.25 for rectangular factor
- calibration data size =
16,384 tokens (GPT-2 s=0.5/0.75); 1,024-2,048 tokens (Qwen s=0.5); 1.05M tokens (Qwen s=0.75)
- DSF outer iterations =
40 (GPT-2), 8 (Qwen)
- fixed-support fine-tuning budget =
16.38M tokens for GPT-2 full-model SWD-FT
assumptions (5)
- domain assumption DSF heuristic approximately solves the l0-constrained activation-weighted reconstruction problem.
- domain assumption Minimizing the Gram-weighted reconstruction error on calibration data transfers to held-out behavioral fidelity.
- domain assumption The replacement AB is faithful enough on task-relevant inputs that circuits found in AB are informative about the dense model.
- domain assumption Positive first-order task-margin attribution ranks units well for ablation.
- domain assumption Mean ablation is a valid reference intervention for sufficiency and necessity.
invented entities (1)
-
SWD bottleneck unit
Cite this review
Pith. "Pith review of Sparse Weight Decomposition for Efficient Circuit Extraction." pith.science (2026). https://pith.science/paper/NYI4AXYR
@misc{pith2026260803913,
author = {Pith},
title = {Pith review of: Sparse Weight Decomposition for Efficient Circuit Extraction},
year = {2026},
howpublished = {\url{https://pith.science/paper/NYI4AXYR}},
note = {Machine review of arXiv:2608.03913}
}
read the original abstract
Dense pretrained transformers do not naturally expose interpretable units for circuit extraction. Existing approaches obtain such units by learning auxiliary sparse representations or training sparse models, incurring substantial additional computation while potentially introducing a fidelity gap between the representation being analyzed and the original pretrained model. We propose Sparse Weight Decomposition (SWD), which reparameterizes pretrained linear projections by factorizing each weight matrix into two sparse factors whose shared intermediate coordinates serve as individually addressable circuit units. Without training a separate replacement network, this parametric representation supports the same scoring, selection, and ablation circuit extraction workflow used for methods that learn sparse features. Across single-matrix replacements, SWD matches the held-out fidelity achieved by Transcoder and other strong baselines while using less than 1% of the data that those baselines use to train their replacements. For matched replacement fidelity, SWD reaches the same circuit sufficiency and necessity targets with fewer active read/write edges and selected units across tasks on GPT-2, Qwen2.5, and Qwen3.5-27B. We further show that SWD remains effective for full-model replacement of all attention and MLP weight matrices after fine-tuning the nonzero factor values. Finally, SWD also features a zero-data variant, allowing broader use of mechanistic interpretability analysis (e.g., per-step analysis).
Figures
Figures from the paper (34 more)
Reference graph
Works this paper leans on
-
[3]
Dan Braun, Lucius Bushnaq, Stefan Heimersheim, Jake Mendel, and Lee Sharkey
URLhttps://openreview.net/forum?id=DwiwOcK1B7. Dan Braun, Lucius Bushnaq, Stefan Heimersheim, Jake Mendel, and Lee Sharkey. Interpretability in parameter space: Minimizing mechanistic description length with attribution-based parameter decomposition.arXiv preprint arXiv:2501.14926,
-
[6]
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models.arXiv preprint arXiv:2309.08600,
-
[9]
carapace length of 80–85millimetres
For each curve, the x-axis is the target threshold and the y-axis is the minimum active edges needed to reach it (lower-right is better). Left: sufficiency. Right: necessity drop. Transcoder and VPD-Recon-CI use the fixed CE-matched 6.29M- and 4.19M-token checkpoints, respectively. 34 Sparse Weight Decomposition for Efficient Circuit Extraction SWD s0.75 ...
work page 2026
-
[10]
Bold marks the token or phrase most relevant to each tentative semantic hypoth- esis. 38 Sparse Weight Decomposition for Efficient Circuit Extraction Overall, the semantic audit gives the GreaterThan circuit a more concrete interpretation. Most dis- played bottleneck units respond to numbers, years, quantities, or measurements in task-independent text, co...
work page 2022
-
[11]
Michael Hanna, Ollie Liu, and Alexandre Variengien
URLhttps://arxiv.org/ abs/2511.13653. Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT-2 compute greater-than? inter- preting mathematical abilities in a pre-trained language model. InAdvances in Neural Information Processing Systems,
-
[12]
Quantized Sparse Weight Decomposition for Neural Network Compression
URLhttps: //transformer-circuits.pub/2025/attention-qk/index.html. Andrey Kuzmin, Mart van Baalen, Markus Nagel, and Arash Behboodi. Quantized sparse weight decomposition for neural network compression.arXiv preprint arXiv:2207.11048,
work page Pith review arXiv 2025
-
[14]
URLhttps://aclanthology.org/2024. blackboxnlp-1.19/. Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content,
work page 2024
-
[15]
URLhttps://openreview. net/forum?id=I4e82CIDxv. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT.arXiv preprint arXiv:2202.05262,
Show all 29 references
-
[17]
URLhttps://transformer-circuits.pub/2022/ in-context-learning-and-induction-heads/index.html. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jia...
2022
-
[18]
Qwen Team
URLhttps://arxiv.org/abs/2412.15115. Qwen Team. Qwen3.5: Accelerating productivity with native multimodal agents, February
-
[19]
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter
URLhttps://arxiv.org/abs/2607.18921. Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. InInternational Conference on Learning Representations, volume 2024, pp. 4942–4964,
2024 arXiv
-
[20]
Attribution patching outperforms automated circuit discovery.arXiv preprint arXiv:2310.10348,
Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery.arXiv preprint arXiv:2310.10348,
-
[23]
Fred Zhang and Neel Nanda
URLhttps://arxiv.org/abs/2312.05821. Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. InInternational Conference on Learning Representations,
-
[25]
19 Sparse Weight Decomposition for Efficient Circuit Extraction A NOTATION Table 3 consolidates the recurring notation used across the main text and appendices
URLhttps://arxiv.org/abs/ 2110.01235. 19 Sparse Weight Decomposition for Efficient Circuit Extraction A NOTATION Table 3 consolidates the recurring notation used across the main text and appendices. We typeset scalars and indices in ordinary italic, vectors in bold lowercase, ...
-
[26]
Dτ,train,Dτ,test Training and held-out evaluation splits for circuit taskτ
Dcal Distribution of calibration activations used to fit SWD. Dτ,train,Dτ,test Training and held-out evaluation splits for circuit taskτ. zi,µi,ai Activation of uniti, its train-split mean used for ablation, and its task- attribution score. For SWD,z i =hA :,i. Sk,S eff Sk is ...
2025
-
[27]
SWD exposes bottleneck units, Transcoders expose hidden features, and the VPD variants expose parameter components
B.3 UNITSCORING ANDCIRCUITSELECTION Candidate units are scored under the unpruned replacement. SWD exposes bottleneck units, Transcoders expose hidden features, and the VPD variants expose parameter components. For the sparse-pretrained model, which has no explicit replacement...
2025
-
[28]
These task examples are separate from the FineWeb- Edu data above and are not counted as replacement fitting or training data
and auto- circuit (Conmy et al., 2023; Miller et al., 2024). These task examples are separate from the FineWeb- Edu data above and are not counted as replacement fitting or training data. The training split is used to estimate unit scores and mean-ablation values; the test spl...
2023
-
[31]
Finally, we remove q266 from the reconstructed Q slice and recompute the head
Each cell showsz 266(ht)z64(hu)γh 266,64 for one causally valid query–key pair; the signed mean is+4.26. Finally, we remove q266 from the reconstructed Q slice and recompute the head. Attention shifts sharply toward the first token: relative to the intact reconstructed pattern...
2024
-
[1999]
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Janos Kramar, Anca Dragan, Rohin Shah, and Neel Nanda
doi: 10.1038/44565. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Janos Kramar, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on Gemma
-
[2004]
Transcoders find interpretable LLM feature circuits.arXiv preprint arXiv:2406.11944,
Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable LLM feature circuits.arXiv preprint arXiv:2406.11944,
-
[2011]
Vladimir Boza and Vladimir Macko
doi: 10.1561/2200000016. Vladimir Boza and Vladimir Macko. Two sparse matrices are better than one: Sparsifying neural networks with double sparse factorization. InInternational Conference on Learning Representa- tions,
-
[2018]
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt
URLhttps:// openreview.net/forum?id=ByOfBggRZ. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: A circuit for indirect object identification in GPT-2 small.arXiv preprint arXiv:2211.00593,
-
[2020]
URL https://distill.pub/2020/circuits/zoom-in/
doi: 10.23915/distill.00024.001. URL https://distill.pub/2020/circuits/zoom-in/. Catherine Olsson, Nelson Elhage, Neel Nanda, et al. In-context learning and induction heads.Trans- former Circuits Thread,
2020 doi
-
[2021]
pub/2021/framework/index.html
URLhttps://transformer-circuits. pub/2021/framework/index.html. Nelson Elhage, Tristan Hume, Catherine Olsson, et al. Toy models of superposition.Trans- former Circuits Thread,
2021
-
[2022]
Gabriel Franco and Mark Crovella
URLhttps://transformer-circuits.pub/2022/ toy_model/index.html. Gabriel Franco and Mark Crovella. Sparse attention decomposition applied to circuit tracing,
2022
-
[2023]
Lucius Bushnaq, Dan Braun, and Lee Sharkey
URLhttps: //transformer-circuits.pub/2023/monosemantic-features. Lucius Bushnaq, Dan Braun, and Lee Sharkey. Stochastic parameter decomposition.arXiv preprint arXiv:2506.20790,
2023 arXiv
-
[2024]
Elias Frantar and Dan Alistarh
URLhttps://arxiv.org/abs/2410.00340. Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. InProceedings of the 40th International Conference on Machine Learning, volume 202 ofProceedings of Machine Learning Research. PMLR,
-
[2025]
Adithya Bhaskar, Alexander Wettig, Dan Friedman, and Danqi Chen
URLhttps: //transformer-circuits.pub/2025/attribution-graphs/methods.html. Adithya Bhaskar, Alexander Wettig, Dan Friedman, and Danqi Chen. Finding transformer circuits with edge pruning. InAdvances in Neural Information Processing Systems (NeurIPS),
2025
-
[2026]
Arthur Conmy, Augustine N
URL https://www.goodfire.ai/research/interpreting-lm-parameters. Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adria Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability.arXiv preprint arXiv:2304.14997,
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.