Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Inference-Time Decomposition of Activations (ITDA): A Scalable Approach to Interpreting Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ITDAs are greedy activation dictionaries that train in roughly 1% of an SAE's time and data, reach comparable reconstruction on some models, and yield a Jaccard-based representation similarity index that outperforms CKA, SVCCA, and…

desk verdict A well-written, honest introduction of a cheap SAE alternative whose headline Jaccard layer-matching claim needs a token-statistics control before it is believable. read the letter →

arxiv 2505.17769 v2 pith:INK6UH3L submitted 2025-05-23 cs.LG

Patrick Leask , Neel Nanda , Noura Al Moubayed This is my paper
classification cs.LG
keywords sparseautoencodersdictionarylearningmatchingpursuitrepresentationsimilarityJaccardLLMinterpretabilitycross-modelcomparisoninference-timedecomposition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sparse autoencoders (SAEs) decompose LLM activations into interpretable latents but are costly to train and do not transfer across models. This paper introduces Inference-Time Decomposition of Activations (ITDA), which builds a dictionary from real activations by greedily adding any activation whose matching-pursuit reconstruction error exceeds a threshold, so each atom carries a natural prompt-token label. ITDAs train on roughly 1% of an SAE's tokens and time, making them feasible on Llama-3.1 70B and 405B with a single consumer GPU, with reconstruction performance comparable to SAEs on Pythia models though worse on Gemma-2. Because dictionary labels are shared across models, the paper defines a representation similarity index as the Jaccard similarity of two dictionaries and reports it outperforms CKA, SVCCA, and relative representation methods on the Kornblith layer-matching benchmark. The pith is that cheap, label-anchored dictionaries can serve both interpretability and cross-model comparison.

What carries the argument

The central object is the ITDA dictionary: a set of normalized residual-stream activations, each collected from a specific prompt and token position at a fixed layer, plus a matching-pursuit solver that decomposes a new activation into an $L_0$-sparse combination of dictionary atoms. The dictionary is built greedily by processing a training corpus, reconstructing each activation with the current dictionary, and adding the activation whenever the mean-squared reconstruction error exceeds a threshold $\tau$, so dictionary size is set by the threshold rather than by a fixed architecture. The paper's key identity for cross-model comparison is the Jaccard index $S(M_0, M_1) := |D_0 \cap D_1| / |D_0 \cup D_1|$, where atoms are matched by their shared prompt-token labels rather than by their vector values. This label-based overlap is what carries the representation-similarity claim.

What would settle it

Train ITDAs on two GPT-2 small instances with deliberately disjoint training corpora (for example, code versus Wikipedia) and rerun the layer-matching benchmark; if Jaccard accuracy drops to near chance, the similarity signal comes from overlapping training text rather than from aligned representations. A second check is to swap in a different tokenizer for one instance and see whether layer matching degrades.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a dictionary of raw activations, selected greedily by reconstruction error and decomposed by matching pursuit, can replace a learned sparse autoencoder for many interpretability purposes at a fraction of the training cost, and its prompt-token labels make dictionaries from different models directly comparable. The authors claim that ITDAs match SAE reconstruction quality on some models (Pythia) while incurring a penalty on others (Gemma-2), that ITDAs find interpretable and monosemantic latents in Llama-3.1 70B and 405B, and that a plain Jaccard index over dictionary labels achieves state-of-the-art accuracy on layer matching between randomly initialized GPT-2 instances. This is the claim a sympathetic reader is asked to accept: one percent of the data and compute buys a decomposition that is competitive enough for many uses and uniquely transferable across models.

Load-bearing premise

The load-bearing premise is that dictionary atoms labelled by their source prompt-token remain comparable across models, so Jaccard overlap of those labels measures representational similarity; this premise is only demonstrated for GPT-2 small and medium with random initializations and could fail when tokenizers, training data, or dictionary thresholds differ.

Editorial extensions

If this is right

  • ITDA training requires around 1.2 million tokens and minutes of compute, so interpretability tools become available for very large models such as Llama-3.1 70B and 405B on a single consumer GPU.
  • On the Kornblith layer-matching benchmark, ITDA Jaccard similarity reaches 0.88 on GPT-2 small and 0.89 on GPT-2 medium, above the relative-representation baseline (0.87 and 0.78) and well above CKA and SVCCA.
  • ITDA dictionaries can distinguish model instances from architectures: Jaccard similarity is 0.56–0.59 within GPT-2 small or medium and 0.46–0.47 across them.
  • Because each ITDA atom is labelled by its source prompt and token, the sparse codes are interpretable without a separate feature-dashboard step, and dictionary differences can be read as model differences.
  • Compared with SAEs, ITDA reconstruction is comparable to ReLU SAEs on Pythia but worse than TopK SAEs and substantially worse on Gemma-2, so the paper positions ITDA as a complement rather than a replacement where modern SAEs already exist.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Jaccard signal is real, ITDA dictionaries are a cheap substrate for model diffing: comparing base and chat-fine-tuned versions of the same model could expose newly appearing or disappearing features with no SAE retraining.
  • The label-overlap metric depends on tokenizer and training data; a cross-tokenizer or cross-corpus control would tell whether the advantage reflects shared features or shared surface tokens.
  • The greedy threshold construction makes dictionary size a cost-quality knob, so ITDA could also serve as a fast exploratory stage that decides where expensive SAEs are worth training.
  • The method should transfer to other decomposition sites, such as attention outputs or MLP layers, since it only assumes that activations live in a space where cosine correlation is meaningful.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Inference-Time Decomposition of Activations (ITDA), a dictionary-learning method that builds a sparse dictionary of model activations by greedily adding activations that are worst reconstructed by matching pursuit, and labels each atom with its source prompt-token. The authors claim ITDAs train about 100x faster and on 1% of the data relative to sparse autoencoders (SAEs), enabling dictionaries for Llama-3.1 70B and 405B on a single consumer GPU. They report that ITDAs reach reconstruction performance similar to SAEs on some Pythia models but are substantially worse on Gemma-2 2B. The central novelty is a representation-similarity index: the Jaccard overlap of two ITDA dictionaries' prompt-token labels, which is shown to outperform CKA, SVCCA, and relative representations on a layer-matching benchmark for GPT-2 small and medium.

Significance. If the representation-similarity claim holds, ITDAs would be a valuable low-cost tool for cross-model interpretability, model diffing, and studying very large models where SAE training is prohibitive. The paper is honest about the reconstruction trade-off and provides a reproducible algorithm plus public code. The Llama-3.1 70B/405B case study demonstrates a real scalability benefit. However, the headline contribution---state-of-the-art layer matching via Jaccard overlap of token labels---is not yet validated against a plausible confound: shared token statistics rather than representational geometry could drive the overlap. The reconstruction comparison is also under-specified in terms of dictionary sizes and sparsity levels. The significance of the paper therefore hinges on whether the Jaccard result survives the missing controls.

major comments (3)
  1. [5.1 Model Instance Layer Similarity, Eq. (8)] The Jaccard index in Eq. (8) is computed over sets of prompt-token labels assigned by Algorithm 1, but Algorithm 1 selects atoms based on reconstruction loss of individual activations. An atom can therefore appear in two models' dictionaries either because both representations genuinely share a hard-to-reconstruct direction or because the corresponding token is an outlier in the shared input distribution (e.g., high norm, rare frequency, unusual context) in both models. The GPT-2 instances in the layer-matching benchmark share a tokenizer and training corpus, so the latter pathway is fully confounded with the former. The paper provides no control such as a token-frequency or activation-norm baseline dictionary, a permutation test of labels, or a setting with different tokenizers or non-overlapping training prompts. Since the state-of-the-art claim in Table 1 rests entirely on this Jaccard score, the result is currently underdetermined.
  2. [4.1 Reconstruction Performance, Fig. 7 and Table 2] The comparison with SAEs is not matched for dictionary size or sparsity: ITDAs are cropped to a fixed size while SAEs have fixed dictionary sizes (Table 2), but the paper does not report the actual L0 values or dictionary sizes used for either method in Fig. 7. Nor does it report the ITDA dictionary sizes for Gemma-2 2B, where performance is reported as substantially worse. This makes it impossible to separate the effect of the method from the effect of dictionary capacity. Please report the matched settings, or explicitly frame the claim as 'similar reconstruction performance at the tested dictionary sizes and L0.'
  3. [5.1, Table 1] The headline layer-matching results in Table 1 are not reproducible as reported because the ITDA hyperparameters (threshold τ, L0, dictionary size, training data) for the GPT-2 small/medium dictionaries are not given. The Jaccard similarity is sensitive to all of these, and the reported gap over the Relative method (0.88 vs 0.87 for GPT-2 Small; 0.89 vs 0.78 for GPT-2 Medium) could change under different hyperparameter choices. The paper should state the hyperparameters for the Table 1 dictionaries and report variance across the five random initialisations.
minor comments (5)
  1. [Section 4.1, Eq. (7)] The notation H, Horig, and H0 in Eq. (7) is used before being defined; please define these quantities in the text.
  2. [Section 5.2] The sentence 'The Jaccard similarity between different instances of the same model architecture ranges from 0.56 to 0.59; whereas the Jaccard similarity between different model architectures range from 0.46 to 0.47 (Figure 11)' refers to a figure that appears only in the appendix; please fix the cross-reference and add error bars or confidence intervals, since the ranges overlap in a way that may not be statistically significant.
  3. [Section 3.3] The sentence 'However, comparison of randomly sampled dictionaries, as is done with SAE decoders in (Lan et al., 2024).' is a sentence fragment and should be completed.
  4. [Section 4.2] The link in 'at [redacted]' for the released Llama ITDAs must be filled in before publication, as the current text is not usable by readers.
  5. [Appendix A.3.4] The caption for Figure 12 in the appendix refers to 'A.4.2' while the figure is numbered 12; please align the cross-references.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the ITDA reconstruction and layer-matching results are evaluated against external benchmarks, and the self-citations are background context rather than load-bearing premises.

full rationale

The core ITDA construction (Algorithm 1) is a deterministic dictionary-learning procedure: each training activation is sparse-coded by matching pursuit (Algorithm 2, Eq. 5), and the activation is added to the dictionary only if its reconstruction loss exceeds threshold tau (Eq. 6). Reconstruction quality is then measured with the external SAEBench CE-loss score (Eq. 7) on Pythia and Gemma-2, which is a held-out evaluation rather than a restatement of the training objective. The representation-similarity claim is defined by Eq. 8, the Jaccard index of per-layer dictionary label sets, and scored on the Kornblith layer-matching benchmark (Eq. 9); neither equation encodes the expected layer identity, so the reported accuracy is an empirical result rather than a tautology. Self-citations (e.g., Leask et al. 2025 for SAE training-time and hyperparameter background; Bussmann et al. 2024, Karvonen et al. 2024b, and Kantamneni et al. 2025 for comparison methods and metrics) support context and comparison choices, but they are not load-bearing for the main derivation, which is benchmarked against external SAEs and CKA, SVCCA, and relative-representation baselines. The main risk with the Jaccard layer-matching result is a confound: a shared tokenizer and shared training prompts can make token-label overlap reflect input statistics rather than representational alignment, but that is a validity or interpretation concern, not a circular reduction of the claimed prediction to its inputs.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The main tuned quantities are τ and L0, which are hand-chosen hyperparameters. No new theoretical entities are introduced. The Jaccard similarity measure is a new use of an existing concept, not an invented entity. The key unstated assumption is that the prompt-token overlap of dictionaries is a faithful proxy for representation similarity, which is plausible but only tested on GPT-2 instances.

free parameters (2)
  • reconstruction loss threshold τ = not specified globally
    Controls dictionary size and reconstruction quality; chosen by hand for each model/site. The paper does not report the exact values used or a principled selection method.
  • sparsity level L0 = 40 for Llama 70B example, varies elsewhere
    Number of matching pursuit atoms per activation; chosen by hand, directly affects the reconstruction and the dictionary quality.
assumptions (3)
  • domain assumption The relative representation assumption: angles between activations are approximately preserved across models (Moschella et al., 2022), which underlies the use of Jaccard similarity on dictionaries.
    Stated in §3.1. The Jaccard similarity of dictionaries only reflects model similarity if the dictionary construction is stable under the symmetries that relate representations of different models. The paper does not verify this beyond the GPT-2 experiments.
  • domain assumption Matching pursuit with a fixed dictionary provides a faithful approximation of the optimal sparse code for the purpose of comparing dictionaries.
    Used throughout §3.2 and in the dictionary construction. Matching pursuit is greedy and does not guarantee the optimal sparse code, but this is an accepted approximation in sparse coding.
  • domain assumption The tokenizer and prompt-token labels are shared enough across models to make Jaccard overlap meaningful.
    For GPT-2 instances this is true by construction. For cross-architecture comparisons (GPT-2 small vs medium) the tokenizer differs, and the paper does not test across tokenizers. This assumption is load-bearing for the model-similarity experiment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Inference-Time Decomposition of Activations (ITDA): A Scalable Approach to Interpreting Large Language Models." pith.science (2026). https://pith.science/paper/INK6UH3L

@misc{pith2026250517769,
  author       = {Pith},
  title        = {Pith review of: Inference-Time Decomposition of Activations (ITDA): A Scalable Approach to Interpreting Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/INK6UH3L}},
  note         = {Machine review of arXiv:2505.17769}
}
read the original abstract

Sparse autoencoders (SAEs) are a popular method for decomposing Large Langage Models (LLM) activations into interpretable latents. However, due to their substantial training cost, most academic research uses open-source SAEs which are only available for a restricted set of models of up to 27B parameters. SAE latents are also learned from a dataset of activations, which means they do not transfer between models. Motivated by relative representation similarity measures, we introduce Inference-Time Decomposition of Activations (ITDA) models, an alternative method for decomposing language model activations. To train an ITDA, we greedily construct a dictionary of language model activations on a dataset of prompts, selecting those activations which were worst approximated by matching pursuit on the existing dictionary. ITDAs can be trained in just 1% of the time required for SAEs, using 1% of the data. This allowed us to train ITDAs on Llama-3.1 70B and 405B on a single consumer GPU. ITDAs can achieve similar reconstruction performance to SAEs on some target LLMs, but generally incur a performance penalty. However, ITDA dictionaries enable cross-model comparisons, and a simple Jaccard similarity index on ITDA dictionaries outperforms existing methods like CKA, SVCCA, and relative representation similarity metrics. ITDAs provide a cheap alternative to SAEs where computational resources are limited, or when cross model comparisons are necessary. Code available at https://github.com/pleask/itda.

Figures

Figures reproduced from arXiv: 2505.17769 by the authors.

Figure 1
Figure 1. Stylized system-level diagram of inference-time decomposition of activations. To decompose a target token in a prompt (1), we compute its activation at a certain point in the LLM (2), for example after layer 8 in GPT-2. We use matching pursuit to decompose the activation into a sparse code of other activations collected from the same point in the LLM (3). Our dictionary atoms are labeled with the prompt and token fo… view at source ↗
Figure 2
Figure 2. Heatmap of similarity indexes between layers in in￾stances of GPT-2 small with random initialisations, metric average across all pairs of model. Each axis represents the ordered layers of the model. The dark blue colors represent pairs of layers that, on average, have low similarity scores, whilst the yellow colors represent pairs of layers that, on average, have high similarity scores. Note that these are averages,… view at source ↗
Figure 3
Figure 3. Activation distribution for latent 50. The peak at 4.33 corresponds to prompts starting with “Q: How”. Activations of 0 are omitted for legibility. Latent 16990: Responds tokens relating to surprise. The input string from which this activation was taken, with relevant token highlighted, is “web . You truly realize how to bring a problem to light and make it crucial . Many more people should really have a look at thi… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Activation distribution for latent 16990 Latent 17002: Responds tokens relating to surprise. The input string from which this activation was taken, with relevant token highlighted, is “the deadlines is our priority and we often acquire strict actions to fulfil our guar…
Figure 5
Figure 5. Figure 5: Activation distribution for latent 17002 17 [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Performance of Pythia-70m SAEs and ITDAs when trained on limited numbers of tokens. Here, we do not crop the size of the dictionary, as on smaller numbers of training tokens the dictionary does not achieve the minimum size. (a) CE Loss Score (b) Automated Interpretabil…
Figure 7
Figure 7. Figure 7: CE loss score and automated interpretability scores. CE loss score is defined in ??, see Karvonen et al. (2024b) for details of the automated interpretability metric. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: Overall and top-1 linear probing scores. Overall score measures whether concepts can be probed from the activations of all the latents, whilst top-1 score measures whether the concepts can be probed from the single most active latent. 24 [PITH_FULL_IMAGE:figures/full_…
Figure 9
Figure 9. Figure 9: Top 5 sparse probing scores. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]
Figure 10
Figure 10. Figure 10: Heatmap of similarity indexes between layers in instances of GPT-2 medium with random initialisations, metric average across all pairs of model. See [PITH_FULL_IMAGE:figures/full_fig_p026_10.png]
Figure 11
Figure 11. Figure 11: Model similarity between ITDAs trained on instances of GPT-2 small and medium. Note the higher similarity within model architectures. A.4.2. LAYER CONVERGENCE DURING TRAINING Raghu et al. (2017) demonstrate using SVCCA that, in a convnet and resnet trained on CIFAR-10…
Figure 12
Figure 12. Figure 12: ITDA similarity between layers every 10k steps during training compared to their final state. Layers increase from the first layer in dark blue to the last layer in light yellow. Note the convergence of early layers to 1 before the end of training. 27 [PITH_FULL_IMAG…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Interpreting Large Text-to-Image Diffusion Models with Dictionary Learning

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Sparse autoencoders and ITDA produce comparably interpretable and steerable features in FLUX.1, outperforming MLP neurons on an automated visual interpretability metric.

Reference graph

Works this paper leans on

31 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    21 Inference-Time Decomposition of Activations (ITDA) Algorithm 2Matching Pursuit (MP) with Normalized Dictionary Input:Normalized dictionaryD∈R m×n, batch of signalsX∈R B×n, number of nonzero coefficientsL Output:Coefficient matrixC∈R B×m Initialize the residuals:R←X Initialize the coefficients:C←0 B×m forℓ= 1toLdo Compute correlations:Corr←R D T For eac...

    work page 2023

  2. [4]

    Transcoders find interpretable llm feature circuits.NeurIPS 2024,

    Dunefsky, J., Chlenski, P., and Nanda, N. Transcoders find interpretable llm feature circuits.NeurIPS 2024,

    work page 2024

  3. [7]

    The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027,

    Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027,

    arXiv

  4. [8]

    Sparse autoencoders can interpret randomly initialized transform- ers.arXiv preprint arXiv:2501.17727,

    Heap, T., Lawson, T., Farnik, L., and Aitchison, L. Sparse autoencoders can interpret randomly initialized transform- ers.arXiv preprint arXiv:2501.17727,

    arXiv

  5. [11]

    Saebench: A comprehensive benchmark for sparse autoencoders, December 2024a

    Karvonen, A., Rager, C., Lin, J., Tigges, C., Bloom, J., Chanin, D., Lau, Y .-T., Farrell, E., Conmy, A., Mc- Dougall, C., Ayonrinde, K., Wearden, M., Marks, S., and Nanda, N. Saebench: A comprehensive benchmark for sparse autoencoders, December 2024a. URL https:// www.neuronpedia.org/sae-bench/info. Ac- cessed: 2025-01-20. Karvonen, A., Wright, B., Rager...

    work page 2025

  6. [13]

    Lindsey, J., Templeton, A., Marcus, J., Conerly, T., Batson, J., and Olah, C

    URL https: //arxiv.org/abs/2408.05147. Lindsey, J., Templeton, A., Marcus, J., Conerly, T., Batson, J., and Olah, C. Sparse crosscoders for cross-layer fea- tures and model diffing.Transformer Circuits Thread,

    arXiv

  7. [17]

    Automati- cally interpreting millions of features in large language models.arXiv preprint arXiv:2410.13928,

    Paulo, G., Mallen, A., Juang, C., and Belrose, N. Automati- cally interpreting millions of features in large language models.arXiv preprint arXiv:2410.13928,

    arXiv

  8. [19]

    Open problems in mechanistic interpretability.arXiv preprint arXiv:2501.16496,

    Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., Bushnaq, L., Goldowsky-Dill, N., Heimersheim, S., Or- tega, A., Bloom, J., et al. Open problems in mechanistic interpretability.arXiv preprint arXiv:2501.16496,

    arXiv

Show all 31 references
  1. [20]

    G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ram ´e, A., et al

    Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ram ´e, A., et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,

    arXiv

  2. [21]

    This provides a gradient for training unlike the L0-norm, but suppresses latent activations harming reconstruction performance (Rajamanoharan et al., 2025)

    use the L1-norm S(f) :=||f|| 1 as an approximation to the L0-norm for the sparsity penalty. This provides a gradient for training unlike the L0-norm, but suppresses latent activations harming reconstruction performance (Rajamanoharan et al., 2025). Furthermore, the L1 penalty ...

    2023

  3. [22]

    separate the selection of dictionary elements used for reconstruction, with the estimation of the coefficients of these dictionary values. This results in the following architecture: πgate(x) :=Wgate(x−b d) +bgate (11) f(x) :=I[π gate(x)>0]⊙ReLU(W mag(x−b d) +bmag)(12) ˆx(f(x)...

    2025

  4. [23]

    replace the standard ReLU activation function with the JumpReLU activation, defined as JumpReLUθ(z) :=zH(z−θ)(16) where H is the Heaviside step function, and θ is a learned parameter for each SAE latent, below which the activation is set to zero. JumpReLU SAEs are trained usin...

    2024

  5. [24]

    How Idris Elba’s ’Luther’ Puts Us in the Mind set of a Renegade Detective. “Luther

    impose a TopK constraint over the activations of entire batches during training. I.e. for a desired sparsity k and a batch size B, all activations not within the top B·k in a batch are zeroed. During training, a threshold value is learned, which replaces the BatchTopK activati...

    2020

  6. [28]

    A.3.3. MATCHINGPURSUIT The Matching Pursuit algorithms due to Mallat & Zhang (1993) is described in Algorithm 2, with the algorithm for iteratively constructing the dictionary of activations in Algorithm

    1993

  7. [30]

    Robot-assisted laparoscopic renal artery aneurysm repair with selective arterial clamping. Renal artery aneurysms represent a rare clinical

    Sequence 7110 Token 30:“Robot-assisted laparoscopic renal artery aneurysm repair with selective arterial clamping. Renal artery aneurysms represent a rare clinical” 20 Inference-Time Decomposition of Activations (ITDA) Atom Index Activation Atom Prompt 9371 5.0731 val of blade...

    1946

  8. [31]

    Note the higher similarity within model architectures

    Model similarity between ITDAs trained on instances of GPT-2 small and medium. Note the higher similarity within model architectures. A.4.2. LAYERCONVERGENCE DURINGTRAINING Raghu et al. (2017) demonstrate using SVCCA that, in a convnet and resnet trained on CIFAR-10 (Krizhevsk...

    2017

  9. [50]

    Activations of 0 are omitted for legibility

    The peak at 4.33 corresponds to prompts starting with “Q: How”. Activations of 0 are omitted for legibility. Latent 16990: Responds tokens relating to surprise. The input string from which this activation was taken, with relevant token highlighted, is “web . You truly realize ...

    1999

  10. [1997]

    In-context learning and induction heads.arXiv preprint arXiv:2209.11895,

    Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y ., Chen, A., et al. In-context learning and induction heads.arXiv preprint arXiv:2209.11895,

    arXiv

  11. [2006]

    Understanding intermediate layers using linear classifier probes.arXiv preprint arXiv:1610.01644,

    Alain, G. Understanding intermediate layers using linear classifier probes.arXiv preprint arXiv:1610.01644,

    arXiv

  12. [2008]

    Are sparse autoencoders useful? a case study in sparse probing.arXiv preprint arXiv:2502.16681,

    Kantamneni, S., Engels, J., Rajamanoharan, S., Tegmark, M., and Nanda, N. Are sparse autoencoders useful? a case study in sparse probing.arXiv preprint arXiv:2502.16681,

    arXiv

  13. [2009]

    Quantifying feature space universality across large language models via sparse autoencoders.arXiv preprint arXiv:2410.06981,

    Lan, M., Torr, P., Meek, A., Khakzar, A., Krueger, D., and Barez, F. Quantifying feature space universality across large language models via sparse autoencoders.arXiv preprint arXiv:2410.06981,

    arXiv

  14. [2010]

    R., Todd, E., Brinkmann, J., Juang, C., Pal, K., Rager, C., Mueller, A., Marks, S., Sharma, A

    Fiotto-Kaufman, J., Loftus, A. R., Todd, E., Brinkmann, J., Juang, C., Pal, K., Rager, C., Mueller, A., Marks, S., Sharma, A. S., et al. Nnsight and ndif: Democratizing access to foundation model internals.arXiv preprint arXiv:2407.14561,

    arXiv

  15. [2014]

    Mallat, S

    URLhttps://arxiv.org/abs/1312.5663. Mallat, S. G. and Zhang, Z. Matching pursuits with time- frequency dictionaries.IEEE Transactions on signal processing, 41(12):3397–3415,

    arXiv

  16. [2017]

    Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders.arXiv preprint arXiv:2407.14435,

    Rajamanoharan, S., Lieberum, T., Sonnerat, N., Conmy, A., Varma, V ., Kram´ar, J., and Nanda, N. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders.arXiv preprint arXiv:2407.14435,

    arXiv

  17. [2020]

    SAEBench metrics were created for evaluating SAEs, which limits their applicability to ITDAs

    or p-annealing SAE (Karvonen et al., 2024b). SAEBench metrics were created for evaluating SAEs, which limits their applicability to ITDAs. In particular, Spurious Correlation Removal and Targeted Probe Perturbation make assumptions about the decoder weight matrix of SAEs that ...

    2025

  18. [2021]

    Erhan, D., Courville, A., Bengio, Y ., and Vincent, P

    https://transformer- circuits.pub/2021/framework/index.html. Erhan, D., Courville, A., Bengio, Y ., and Vincent, P. Why does unsupervised pre-training help deep learning? In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 2...

    2021

  19. [2022]

    Relative representations enable zero-shot latent space communication.arXiv preprint arXiv:2209.15430,

    Moschella, L., Maiorca, V ., Fumero, M., Norelli, A., Lo- catello, F., and Rodol`a, E. Relative representations enable zero-shot latent space communication.arXiv preprint arXiv:2209.15430,

    arXiv

  20. [2023]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  21. [2024]

    Sparse autoencoders find highly interpretable features in language models.ICLR 2024,

    Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models.ICLR 2024,

    2024

  22. [2025]

    and Manning, C

    Hewitt, J. and Manning, C. D. A structural probe for finding syntax in word representations. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Papers), pp...

    2019

  23. [7000]

    1 . Field of the Invention \n The present invention relates to a camera system for transmitting and receiving data to and from a camera by obtaining information /hlon

    Token 30 is chosen as it includes contextual information whilst keeping the prompt relatively short and readable. Activations less than 0.0001 are omitted. The text displayed in the third column is the prompt corresponding to the atom’s activation with the specific token highl...

    1926

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.