Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A data-free cosine-similarity match between sparse-autoencoder directions traces how features are born, transformed, or carried through a transformer, and the resulting flow graph improves model steering by targeting multiple features…

desk verdict Honest, well-scoped extension of SAE feature matching to multi-module flow graphs; the deactivation evidence is real, but the steering gains are confounded and need controls before the paper's headline claim holds. read the letter →

arxiv 2502.03032 v3 pith:IZTHVNI3 submitted 2025-02-05 cs.LG cs.CL

Daniil Laptev , Nikita Balagansky , Yaroslav Aksenov , Daniil Gavrilov This is my paper · ORCID
classification cs.LGcs.CL MSC 68T0768T50
keywords sparseautoencodersfeatureflowcross-layerinterpretabilitycosinesimilaritymatchingmodelsteeringmechanisticcircuitsLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the inner geometry of a large language model can be mapped as a flow of concepts without running any data through the model at inference time. The key move is to measure cosine similarity between the decoder directions learned by sparse autoencoders trained at different layers and modules, then chain these matches backward through the network to build a flow graph for any chosen feature. The authors claim these graphs describe how features originate, persist, transform, or vanish, and that they double as practical steering handles: suppressing or amplifying a whole set of causally related features across layers controls generation themes better than editing one feature at one layer. If correct, this gives a lightweight, interpretable, and partially causal control layer for LLMs that needs no activation statistics.

What carries the argument

The central object is the cosine-similarity matching operator between SAE decoder matrices, $T^{(A\to B)} = I_{x>0}\big(\mathrm{top}_k\big({W^{(A)}_{\mathrm{dec}}}^\top W^{(B)}_{\mathrm{dec}}\big)\big)$, which maps features trained at position $A$ to the most similar features at position $B$ under unit-norm columns. For each residual feature at layer $L$, the authors compare its embedding against the SAE decoder columns of the previous residual, the MLP output, and the attention output, assigning the feature to groups ('From RES', 'From MLP', 'From ATT', and their combinations) according to which of those matched predecessors is active; composing these backward matches over consecutive layers yields a flow graph. The same matched directions are used for steering by rescaling hidden states, $h \leftarrow h + a(r-1)v$, which subtracts or amplifies the feature direction. The flow graph is what carries the argument: it converts pairwise geometric matches into a causal-looking computational pathway that can be validated by deactivation and then used for intervention.

What would settle it

Measure the deactivation success of top-1 cosine matches on a set of features whose true causal predecessors are known by construction, for example a synthetic transformer where layer $L{+}1$ features are defined as sparse linear combinations of layer $L$ features. If the top-1 cosine predecessor is not the planted one, or deactivating it rarely removes the target feature, the geometric-correspondence claim fails.

Watch

Extended reading notes

Core claim

The paper argues that the per-layer feature dictionaries learned by sparse autoencoders are compatible enough, layer to layer and module to module, that a pure geometry check—maximum cosine similarity between decoder columns—can stand in for activation statistics and reveal the computational ancestry of a feature. Matching backward from a residual-stream feature at layer $L$ to the residual, MLP, and attention SAEs at layer $L-1$, and composing these matches over consecutive layers, produces a flow graph showing a feature persisting unchanged, being refined by the MLP, or appearing for the first time (features with no active predecessor are labeled 'From nowhere'). The authors then treat the matched predecessors as intervention targets: removing a predecessor's decoder direction from the hidden state is expected to remove the target feature, and doing this at several layers at once steers generation thematically. Their deactivation results (top-1 cosine matches succeed about 65% of the time, on par with Pearson-correlation matching and below an exhaustive-search ceiling of 73%) and their steering results (cumulative, multi-layer interventions beat single-layer steering at small rescaling coefficients) are offered as evidence that these geometric flow graphs describe real causal structure rather than mere visual similarity.

Load-bearing premise

The method assumes that when a decoder column of one SAE is the most cosine-similar to a column of another SAE, the two columns name the same underlying concept and the earlier one causes the later one; the paper's own tests show this link fails for a large share of features, with deactivation succeeding only about 65% of the time and many features having no matched active predecessor at all.

Editorial extensions

If this is right

  • A feature's causal predecessors can be identified from SAE weights alone, without collecting activation statistics, and this matches the accuracy of Pearson-correlation-based matching.
  • Deactivating the single most similar predecessor is informative for causal analysis: it deactivates the target feature roughly 65% of the time, and random selection among the top-5 candidates performs markedly worse.
  • Intervening on features across several layers (cumulative steering) outperforms single-layer steering at small steering coefficients and reduces sensitivity to the rescaling hyperparameter.
  • Flow graphs explain steering side effects: the 'London' feature steers toward fashion themes because its earlier-layer ancestors already carry fashion-related semantics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An immediate testable extension is to replace the many-to-one top-1 match with the many-to-many top-k match everywhere and compare deactivation success, since the paper uses top-5 only to define 'inactive' rather than to steer.
  • Because the 'From nowhere' group is large in early layers (up to roughly half of features) and the attention SAEs were trained by the authors themselves, the flow graphs are likely to get denser and more accurate as SAE quality improves; the qualitative three-part layer segmentation may partly reflect under-trained attention features.
  • If the geometric-correspondence assumption is right, the same flow graphs should transfer to other diagnostics such as targeted ablation or circuit pruning, and should agree with crosscoder-style shared-feature decompositions where those are available.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a data-free method for matching sparse autoencoder (SAE) features across consecutive layers and modules of a transformer by taking the cosine similarity between decoder columns. The matches are composed into 'flow graphs' that are meant to show how features originate, propagate, or vanish through the residual stream, MLP, and attention. The authors validate the matching with co-activation statistics, test causal dependence by deactivating matched predecessor features in hidden states, and use the resulting graphs to steer generation toward or away from target topics. They claim the first multi-layer steering via SAE features and position the method as a causal, cross-layer interpretability and control tool.

Significance. The matching step is simple, parameter-free, and requires no activation data, which is a real practical advantage over correlation-based matching. The deactivation experiments are a genuine attempt at causal validation: top-1 cosine matching beats random selection and is comparable to Pearson correlation and to an exhaustive-search ceiling (Table 1). If the steering results were controlled for total intervention strength and feature-selection bias, the paper would provide a useful cross-layer interpretability and control method. As it stands, the steering evidence is confounded, so the paper's third contribution — that flow graphs improve steering — is not yet established.

major comments (4)
  1. [Section 5.3, Figure 10, Appendix B.2] The comparison between cumulative and single-layer steering is confounded by total intervention magnitude. In the cumulative strategy, a rescaling or activation coefficient is applied at every layer from 0 to l, so with exponential decay alpha = -0.05 the coefficients sum to roughly 15 times the layer-0 coefficient, whereas the single-layer baseline perturbs one layer only. The cumulative feature set is also manually curated: Appendix B states that features are added only if their Neuronpedia interpretation 'remains consistent and relevant', and thresholds t(R)=0.5, t(M)=0.15, alpha, and s are tuned on a small generated batch. Without a control that steers the same number of randomly selected features at the same layers, or features matched by direct cosine similarity without intermediate flow-graph edges, Figure 10 cannot be read as evidence that the flow-graph structure itself improves steering.
  2. [Section 5.2, Table 1, Appendix C.2] The causal-precursor assumption is only partially validated. Top-1 cosine deactivation succeeds in 65% of cases, and the exhaustive search achieves only 73%, meaning that for about a third of target features the matched predecessor is not a sufficient causal parent on its own. Section C.2 further states that in most cases deactivating all predecessors is required to deactivate the target. The paper should report feature-level diagnostics — for example, the distribution of activation change, success rates by layer and module, and the fraction of target features whose top-1 match is active when the target is active — to bound how often a flow-graph edge corresponds to the main causal pathway. The 'From nowhere' group can be roughly half of features in early layers (Figure 5), so the claim that flow graphs describe how features 'originate, propagate, or vanish' needs an explicit statement of coverage and failure modes.
  3. [Section 5.1 and Appendix D] The attention branch of the flow graph rests on SAEs whose quality the authors themselves doubt. Section 5.1 says the rarity of attention groups 'could stem from our SAE training rather than an inherent property of Gemma', and Appendix D says 'our attention SAEs for Gemma were perhaps not trained well enough'. Because attention is one of the three module types used in both the group-distribution analysis and the steering procedure, the paper should report attention-SAE quality metrics (reconstruction loss, l0, or downstream interpretability) and show that the main conclusions are robust either to excluding attention or to using higher-quality attention SAEs.
  4. [Section 3.2 and Section 5.1] The group-separability validation (Mann-Whitney U tests summarized in Figure 4) shows that score distributions differ between groups, but it does not establish that the top-1 cosine match is the correct individual predecessor. A feature-level evaluation is needed — for example, comparing the cosine-ranked predecessor against the Pearson-correlation-ranked predecessor per target feature and measuring agreement and activation correlation, in addition to group-level comparisons. This would directly test the load-bearing assumption that geometric alignment of independently trained SAE spaces implies semantic and causal correspondence.
minor comments (5)
  1. [Appendix C.3] The cross-reference 'Figure 15(a)' should be 'Figure 17(a)' for the plot of the number of selected features, and the caption 'Score is a total metric measured as Behavioral × Cumulative' appears to be a typo for 'Behavioral × Coherence'.
  2. [Figure 10] The text refers to a green line for the initial-feature baseline, but the caption only describes 'One-layer' and 'Cumulative'; please add the color or label to the legend.
  3. [Section 3.2] The description 'topk(·) zeroes out values below the kth order statistic' should specify that the threshold is the k-th largest value and how ties are handled.
  4. [Appendix B.2] The variable s* is used in the linear-scaling formula before it is introduced; please define it at first use (it is later set to 1).
  5. [Section 5.1 and Appendix C.1] The 'From nowhere' category is described as evidence of high-entropy early processing, but it could also reflect matching failure; the text should make explicit that both interpretations are possible, as Appendix C.1 acknowledges.

Circularity Check

2 steps flagged · score 6.0 of 10

Deactivation 'top-1 is informative' result re-measures the cosine-similarity matching criterion by construction; steering coefficients and features are tuned on the same test completions, so the flow-graph steering gains are partially fitted.

  1. self definitional [Section 3.2 (feature matching), Section A.2 (feature rescaling), Section 5.2 (deactivation results)]
    "We find the matched feature index as j = arg max_k ( f · W(B)dec:,k ). Then we say that F(A)i corresponds to F(B)j. ... Setting r = 0 removes the selected features from the existing linear combination, which is (up to SAE reconstruction error) the same as setting those features' activations to zero."

    The predecessor is selected as the decoder column v with maximal cosine similarity to the target f. The deactivation intervention removes a·v from the hidden state. Under the paper's own linear SAE approximation (Appendix A.2), the induced change in the target activation is proportional to f·v — exactly the score used for matching. Therefore the top-1 vs random comparison in Figure 6 is forced by construction: top-1 must give the largest projection removal. The reported 'top1 is informative for causal analysis' evidence re-states the matching criterion rather than independently confirming causality.

  2. fitted input called prediction [Appendix B (Distribution of steering coefficient; Flow graph building), Section 5.3 (Model steering)]
    "We set α = −0.05 and s∗ = 1, based on generating a small batch of test completions and manually checking the trade-off between coherence and theme intensity. ... If the semantic meaning of a feature remains consistent and relevant, we add it to our target collection; otherwise, we continue searching until we have a satisfactory set of features."

    The features added to the flow graph are manually selected for semantic relevance and the steering coefficients are tuned on 'a small batch of test completions'; the Section 5.3 steering scores are then reported for these same completions. The cumulative strategy also applies the rescaling at every layer, so the apparent advantage over single-layer steering conflates graph structure with total intervention strength and with manual curation. The demonstration is therefore a fitted/selected result rather than an out-of-sample prediction of flow-graph utility.

full rationale

The flow-graph construction itself is data-free and parameter-free: T(A→B) = I_{x>0} topk(W(A)⊤dec W(B)dec) is a fixed function of pretrained SAE weights, so the graph edges do not reduce to the downstream validations. The paper also compares against permutation and Pearson baselines, which gives the matching independent content. However, two load-bearing demonstrations are partially circular. (1) Deactivation validation: the predecessor is defined as the maximally cosine-similar decoder column, and deactivation removes exactly that column from the hidden state; under the linear SAE approximation used in Appendix A.2, the activation-change metric is monotone in the same cosine similarity, so top-1-vs-random is guaranteed by construction and cannot independently establish causality. (2) Steering: feature sets are manually curated for semantic relevance and coefficients α, s∗ are tuned on 'a small batch of test completions', after which the same-style completions are scored to claim improved steering; this is fitting to the evaluation, and the cumulative-vs-single comparison is confounded by total intervention strength. These issues make the causal and steering claims partially circular, while the core cosine-similarity matching remains a legitimate construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No new physical entities are introduced. The 'flow graph' is a method artifact, not an invented entity. The main load-bearing assumptions are the geometric alignment of SAE spaces and the linearity of the intervention, both partly validated but partly contradicted by the paper's own results. Free parameters appear chiefly in the steering loop and graph thresholds.

free parameters (5)
  • Steering scale s = 8 and 3 in activation experiments
    Manually chosen based on a small batch of test completions; Appendix B.2.
  • Exponential decay alpha = -0.05 and -0.25
    Chosen manually to distribute steering coefficients across layers; affects results; Appendix B.2.
  • Graph thresholds t(R), t(M) = 0.5 and 0.15
    Hand-chosen cutoffs for building flow graphs; Appendix B.2.
  • Matching top-k = k=5 for top-k matching, k=1 for top1
    Design choice; k=5 used for deactivation experiments, k=1 is the main method.
  • Layer-specific high/low thresholds = unspecified
    Section 3.3 states thresholds are specific for each layer but does not give their values, leaving an unstated parameter in the classification.
assumptions (5)
  • domain assumption Linear representation hypothesis
    States that features are linear directions in hidden state space; invoked in Section 2.1.
  • domain assumption Geometric alignment of SAE decoder spaces across positions
    Cosine similarity between decoder columns assumes SAEs are trained on aligned hidden states; the paper flags this in Section 2.3 and notes dimension mismatches for attention SAEs.
  • domain assumption Linear rescaling removes a feature from the hidden state
    The transformation h <- h + (r-1)(a * V^T) is treated as equivalent to zeroing activations, stated as true only up to SAE reconstruction error in Appendix A.2.
  • domain assumption gpt4o-mini judge produces valid behavioral and coherence scores
    The steering evaluation relies on LLM-as-judge scores without human validation; Appendix B.1.
  • ad hoc to paper The matched predecessor is the relevant causal parent of the target feature
    The whole classification into 'From RES', 'From MLP', etc. assumes the active matched predecessor explains the target's activation; the large 'From nowhere' group and partial deactivation success contradict this, acknowledged in Section C.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Analyze Feature Flow to Enhance Interpretation and Steering in Language Models." pith.science (2026). https://pith.science/paper/IZTHVNI3

@misc{pith2026250203032,
  author       = {Pith},
  title        = {Pith review of: Analyze Feature Flow to Enhance Interpretation and Steering in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IZTHVNI3}},
  note         = {Machine review of arXiv:2502.03032}
}
read the original abstract

We introduce a new approach to systematically map features discovered by sparse autoencoder across consecutive layers of large language models, extending earlier work that examined inter-layer feature links. By using a data-free cosine similarity technique, we trace how specific features persist, transform, or first appear at each stage. This method yields granular flow graphs of feature evolution, enabling fine-grained interpretability and mechanistic insights into model computations. Crucially, we demonstrate how these cross-layer feature maps facilitate direct steering of model behavior by amplifying or suppressing chosen features, achieving targeted thematic control in text generation. Together, our findings highlight the utility of a causal, cross-layer interpretability framework that not only clarifies how features develop through forward passes but also provides new means for transparent manipulation of large language models.

Figures

Figures reproduced from arXiv: 2502.03032 by the authors.

Figure 1
Figure 1. Schematic illustration of inner-layer matching. We select a feature with index i on the SAE trained at the layer output. Its embedding f, which is the ith column of this SAE’s decoder weight, is compared to every column of other SAEs on the same layer (after the MLP and attention blocks, as well as with the SAE on the residual stream before some layer). These comparisons indicate the feature’s source. See Section 3.… view at source ↗
Figure 2
Figure 2. An illustration of the resulting flow graph, which we also use in the deactivation experiment (section 5.2). As a starting point, we select the feature on the 24th-layer residual with index 14548. For a detailed explanation of this graph, see Appendix E. (the layer input). MLP and attention modules read from RL−1 and their outputs produce RL. We pick a feature from the SAE trained on RL with embed￾ding f ∈ R d . Let… view at source ↗
Figure 3
Figure 3. Example of cosine similarity vs. simultaneous activa￾tion with a predecessor (350 features were sampled per layer). “From MLP” and “From RES” groups are notably different: high s (M) and low s (R) suggest simultaneous activation with an MLP￾module match. Cosine similarity serves as a good proxy for shared semantic and mechanistic properties. h ← h + a(r − 1)v, where a is the predecessor’s activation strength, v its … view at source ↗
Figures from the paper (19 more)
Figure 4
Figure 4. Figure 4: Percentage of statistically significant differences between groups for each module’s similarity scores. AO means module P is active in only one group, AB means active in both, and IB means inactive in both. For MLP, two groups differ in s (R) only 87% of the time when …
Figure 6
Figure 6. Figure 6: Deactivation methods compared. Group labels show which active predecessors were deactivated. The random approach underperforms, suggesting that choosing the top1 feature is al￾ready meaningful for causal analysis. language, and TinyStories has the most natural and simp…
Figure 7
Figure 7. Figure 7: Impact of different r values on deactivation success, with rescaling of all available predecessors. When r < 1, the activation change grows nonlinearly, indicating alternative causal pathways still convey information. Relative loss change measured as (Lnew− Lold)/Lold …
Figure 9
Figure 9. Figure 9: Deactivating the “Scientific concepts and entities” theme. The dashed black line shows the default generation score. Red points mark the best layer for each r in the single-layer method. Larger r boosts performance but shifts the optimal layer earlier. features identif…
Figure 10
Figure 10. Figure 10: Comparison of best deactivation scores. The green line indicates deactivation using only the initial feature set. Interven￾tions on layers detected by our method (orange, blue) perform better across different r values, suggesting additional discovered features reduce …
Figure 11
Figure 11. Figure 11: Activation of specific topics. We compare single-layer steering and cumulative approaches with three rescaling strategies (Appendix B). Activating multiple similar features amplifies a topic’s presence but may degrade overall text coherence [PITH_FULL_IMAGE:figures/f…
Figure 12
Figure 12. Figure 12: (a) Percentage of feature groups obtained for each dataset. (b) Distribution of scores for layers 8 and 18. We observe a clear distinction between groups, which additionally indicates the validity of the proposed method. From nowhere From RES From MLP From ATT From RE…
Figure 13
Figure 13. Figure 13: Probability of group A (row) to appear in group B (column), aggregated over all layers. For example, if we take the “From ATT” group, then with a probability of 0.45, features from this group would appear in the “From RES & ATT” group. High scores for the “From nowher…
Figure 14
Figure 14. Figure 14: Percentage of statistically significant differences between groups with respect to a certain score. random permutation top-1 top-k 0 10 20 30 40 50 60 Percent of features per category From nowhere From RES From MLP From ATT From RES & MLP From RES & ATT From MLP & ATT…
Figure 15
Figure 15. Figure 15: (a) Percentage of features per each method. There was a total of 13106 activated features, and for every feature, four matching strategies were applied. We see that top5 method detects many more combined groups than other methods, especially “From RES & MLP”. (b) Prob…
Figure 16
Figure 16. Figure 16: From each flow graph, we select features on a particular layer l and perform steering with the four different strategies. Bars represent the best result for each layer among all scores s. In some cases, steering on a layer other than 12 may improve results. features. …
Figure 17
Figure 17. Figure 17: (a) Amount of features selected for activation of “Research methodology and experimentation” theme. Vertical lines represent the placement of the initially selected features. (b) Results for steering of selected features. Score is a total metric measured as Behavioral…
Figure 18
Figure 18. Figure 18: Feature group identification comparison (Section 5.1) between top1 cosine similarity and Pearson correlation. While correlation better captures predecessors with under-trained embeddings, it exhibits stronger dataset dependence and sparsity sensitivity. Correlation-ba…
Figure 19
Figure 19. Figure 19: (a) Distribution of groups for Llama Scope. We observe a clear distinction from Gemma Scope results ( [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]
Figure 20
Figure 20. Figure 20: Flow graph for the 12/res/14455 feature. As reported in Chalnev et al. (2024), steering of that feature might produce themes related to fashion, and we clearly observe that our flow graph captures this semantics in the earlier layers. E. Examples of flow graphs In thi…
Figure 21
Figure 21. Figure 21: Flow graph for the 12/res/4230 feature. In this case, we observe that the second half of the model is closely related to wedding and marriage ceremonies. We believe that the “official” aspect in the interpretation of features in earlier layers is closely related to th…
Figure 22
Figure 22. Figure 22: Two SAEs with a learned transition matrix T can be seen as a transcoder from layer t to layer t + 1. Wedding and marriage graph. We have observed in our experiments that steering feature 12/res/4230 with interpretation “terms related to weddings and marriage ceremonie…
Figure 23
Figure 23. Figure 23: Explained variance of the various permutation variants. Cosine similarity between decoders’ vectors (Ix>0 top 1W(14)⊤ dec W(15) dec ) performs best. See Appendix F for more details. 1 2 3 4 5 k 0.50 0.55 0.60 0.65 0.70 0.75 0.80 Explained Variance Layer 15-16 Residual…
Figure 24
Figure 24. Figure 24: Comparison of various k in topk operator and different weights of the SAE. Cosine similarity (Ix>0 top 1W(14)⊤ dec W(15) dec ) performs best. See Appendix F for more details. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_24.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FaithfulSAE: Towards Capturing Faithful Features with Sparse Autoencoders without External Dataset Dependencies

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Training sparse autoencoders on a language model's own generated text can improve seed stability and downstream probing relative to training on web text.

  2. Cross-Layer Discrete Concept Discovery for Interpreting Language Models

    cs.LG 2025-06 reject novelty 5.0 of 10

    CLVQ-VAE maps lower-layer transformer activations to higher-layer ones through a discrete codebook, yielding concept vectors evaluated with probe ablation and human annotation.

Reference graph

Works this paper leans on

32 extracted references · 12 canonical work pages · cited by 2 Pith papers

  1. [1]

    Mechanistic permutability: Match features across layers

    Balagansky, N., Maksimov, I., and Gavrilov, D. Mechanistic permutability: Match features across layers. In The Thirteenth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=MDvecs7EvO

    work page 2024

  2. [2]

    Evolution of sae features across layers in llms, 2024

    Balcells, D., Lerner, B., Oesterle, M., Ucar, E., and Heimersheim, S. Evolution of sae features across layers in llms, 2024. URL https://arxiv.org/abs/2410.08869

    arXiv 2024

  3. [3]

    E., Hume, T., Carter, S., Henighan, T., and Olah, C

    Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Hatfield-Dodds, Z., Tamkin, A., Nguyen, K., McLean, B., Burke, J. E., Hume, T., Carter, S., Henighan, T., and Olah, C. Towards monosemanticity: Decomposing languag...

    2023

  4. [4]

    Batchtopk sparse autoencoders

    Bussmann, B., Leask, P., and Nanda, N. Batchtopk sparse autoencoders. arXiv preprint arXiv: 2412.06410, 2024

    arXiv 2024

  5. [5]

    Improving steering vectors by targeting sparse autoencoder features, 2024

    Chalnev, S., Siu, M., and Conmy, A. Improving steering vectors by targeting sparse autoencoder features, 2024. URL https://arxiv.org/abs/2411.02193

    arXiv 2024

  6. [6]

    N., Lynch, A., Heimersheim, S., and Garriga-Alonso, A

    Conmy, A., Mavor-Parker, A. N., Lynch, A., Heimersheim, S., and Garriga-Alonso, A. Towards automated circuit discovery for mechanistic interpretability. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    work page 2023

  7. [7]

    Sparse autoencoders find highly interpretable features in language models, 2023

    Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models, 2023. URL https://arxiv.org/abs/2309.08600

    arXiv 2023

  8. [8]

    Transcoders find interpretable llm feature circuits

    Dunefsky, J., Chlenski, P., and Nanda, N. Transcoders find interpretable llm feature circuits. arXiv preprint arXiv: 2406.11944, 2024

    arXiv 2024

Show all 32 references
  1. [9]

    and Li, Y

    Eldan, R. and Li, Y. Tinystories: How small can language models be and still speak coherent english?, 2023. URL https://arxiv.org/abs/2305.07759

    2023 arXiv

  2. [10]

    A mathematical framework for transformer circuits, 2021

    Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph†, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kapla...

    2021

  3. [11]

    J., Liao, I., Gurnee, W., and Tegmark, M

    Engels, J., Michaud, E. J., Liao, I., Gurnee, W., and Tegmark, M. Not all language model features are one-dimensionally linear. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=d63a4AM4hb

    2025

  4. [12]

    D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J

    Gao, L., la Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=tcsZt9ZNKD

    2025

  5. [13]

    Automatically identifying local and global circuits with linear computation graphs

    Ge, X., Zhu, F., Shu, W., Wang, J., He, Z., and Qiu, X. Automatically identifying local and global circuits with linear computation graphs. arXiv preprint arXiv: 2405.13868, 2024

    2024 arXiv

  6. [14]

    Gemma 2: Improving open language models at a practical size, 2024

    Gemma Team . Gemma 2: Improving open language models at a practical size, 2024. URL https://arxiv.org/abs/2408.00118

    2024 arXiv

  7. [15]

    Accelerating sparse autoencoder training via layer-wise transfer learning in large language models

    Ghilardi, D., Belotti, F., Molinari, M., and Lim, J. Accelerating sparse autoencoder training via layer-wise transfer learning in large language models. In Belinkov, Y., Kim, N., Jumelet, J., Mohebbi, H., Mueller, A., and Chen, H. (eds.), Proceedings of the 7th BlackboxNLP Wor...

    2024 doi

  8. [16]

    and Tegmark, M

    Gurnee, W. and Tegmark, M. Language models represent space and time, 2024. URL https://arxiv.org/abs/2310.02207

    2024 arXiv

  9. [17]

    Finding neurons in a haystack: Case studies with sparse probing

    Gurnee, W., Nanda, N., Pauly, M., Harvey, K., Troitskii, D., and Bertsimas, D. Finding neurons in a haystack: Case studies with sparse probing. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=JYs1R9IMJr

    2023

  10. [18]

    Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders

    He, Z., Shu, W., Ge, X., Chen, L., Wang, J., Zhou, Y., Liu, F., Guo, Q., Huang, X., Wu, Z., Jiang, Y.-G., and Qiu, X. Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders. arXiv preprint arXiv: 2410.20526, 2024

    arXiv

  11. [19]

    Random open problems

    Jermyn, A., Batson, J., and Olah, C. Random open problems. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/jan-update/index.html#open-problems

    2024

  12. [20]

    Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2

    Lieberum, T., Rajamanoharan, S., Conmy, A., Smith, L., Sonnerat, N., Varma, V., Kram'ar, J., Dragan, A., Shah, R., and Nanda, N. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP,...

    2024 doi arXiv

  13. [21]

    Sparse crosscoders for cross-layer features and model diffing, 2024

    Lindsey, J., Templeton, A., Marcus, J., Conerly, T., Batson, J., and Olah, C. Sparse crosscoders for cross-layer features and model diffing, 2024. URL https://transformer-circuits.pub/2024/crosscoders/index.html

    2024

  14. [22]

    and Frey, B

    Makhzani, A. and Frey, B. k-sparse autoencoders, 2014. URL https://arxiv.org/abs/1312.5663

    2014 arXiv

  15. [23]

    and Tegmark, M

    Marks, S. and Tegmark, M. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets, 2024. URL https://arxiv.org/abs/2310.06824

    2024 arXiv

  16. [24]

    J., Belinkov, Y., Bau, D., and Mueller, A

    Marks, S., Rager, C., Michaud, E. J., Belinkov, Y., Bau, D., and Mueller, A. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.ne...

    2025

  17. [25]

    The hydra effect: Emergent self-repair in language model computations, 2023

    McGrath, T., Rahtz, M., Kramar, J., Mikulik, V., and Legg, S. The hydra effect: Emergent self-repair in language model computations, 2023. URL https://arxiv.org/abs/2307.15771

    2023 arXiv

  18. [26]

    Linguistic regularities in continuous space word representations

    Mikolov, T., Yih, W.-t., and Zweig, G. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies , pp.\ 746--751, Atlanta,...

    2013

  19. [27]

    B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L

    Penedo, G., Kydl \' c ek, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchma...

    2024

  20. [28]

    Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders

    Rajamanoharan, S., Lieberum, T., Sonnerat, N., Conmy, A., Varma, V., Kramár, J., and Nanda, N. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv: 2407.14435, 2024

    2024 arXiv

  21. [29]

    L., McDougall, C., MacDiarmid, M., Freeman, C

    Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C., and H...

    2024

  22. [30]

    Towards universality: Studying mechanistic similarity across language model architectures

    Wang, J., Ge, X., Shu, W., Tang, Q., Zhou, Y., He, Z., and Qiu, X. Towards universality: Studying mechanistic similarity across language model architectures. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=2J18i8T0oI

    2025

  23. [31]

    Zhang, Y., Luo, Y., Yuan, Y., and Yao, A. C.-C. Autonomous data selection with language models for mathematical texts, 2024. URL https://arxiv.org/abs/2402.07625

    2024 arXiv

  24. [32]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.