{"id":"faccbb94-e3cb-4792-b142-fd49b18747cf","arxiv_id":"2505.17769","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"ITDA dictionaries, built greedily from poorly reconstructed activations and decomposed by matching pursuit, match some SAE reconstruction performance at 100x lower training cost and enable SOTA cross-model layer similarity via Jaccard overlap.","lead":"This paper introduces a cheap way to break down a large language model's internal activations into a dictionary of interpretable units, called Inference-Time Decomposition of Activations (ITDA). It matters because ITDAs can be trained in about 1% of the time and data of sparse autoencoders, making interpretability feasible on 70B and 405B models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Jaccard layer-matching may be driven by shared token/activation statistics rather than representational similarity; the paper provides no control separating these, so the SOTA representation-similarity claim is not yet established.","rationale":"The reader's weakest assumption identifies the same load-bearing point: the cross-model Jaccard comparison depends on prompt-token labels and has only been demonstrated on GPT-2 instances sharing a tokenizer and training corpus. My concern sharpens this by specifying a mechanism: the overlap could be inflated by token-level activation-magnitude outliers that are selected at the same layer in both models even if their representation geometry is not otherwise aligned. This is the single most load-bearing concern because the representation-similarity result is the paper's main claimed new capability and is structurally independent of the reconstruction comparison. The reconstruction claims are honestly caveated and the code release is a positive step, so I would not reject the paper; the conditional verdict already in place is appropriate. The concrete test—a norm-based dictionary baseline on the same layer-matching benchmark—would settle whether the Jaccard signal reflects representational similarity or shared token statistics. If the baseline matches the reported accuracy, the central claim would need substantial revision; if not, the current claim is strengthened. Therefore the reader's CONDITIONAL verdict stands unchanged, with the added condition that this control be run.","tokens_in":20740,"tokens_out":7543,"duration_ms":90594,"concrete_test":"Recompute Table 1 (§5.1) with a norm-only baseline: at each layer of each GPT-2 instance, build a dictionary from the top-N training-token positions by activation L2 norm (same N as the ITDA dictionary at that layer) and compute the same Jaccard layer-matching accuracy. If this baseline approaches the reported ITDA scores (0.88 small / 0.89 medium), the layer-matching signal is carried by token-level activation-magnitude statistics, not by the ITDA reconstruction-based selection. If it does not, a second check is to permute atom labels within each ITDA dictionary (same size, random labels) and confirm the score is not a size/frequency artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The SOTA representation-similarity claim rests on Eq. 8: the Jaccard index of the sets of source prompt-token labels of per-layer ITDA dictionaries. Algorithm 1 selects, for each model and layer, the token positions whose activations are worst reconstructed by the current dictionary, but the matching step then ignores all activation geometry and counts only whether the same tokenized position was selected in both dictionaries. In the §5.1 benchmark, the five GPT-2 small/medium instances share a tokenizer and (through the Mistral pipeline) the same training corpus, so a token position can enter two dictionaries for either of two reasons: (a) the two models' layer-i representations genuinely share the same hard-to-reconstruct directions, or (b) the token is an outlier in the shared input distribution at that layer in both models—large norm, rare frequency, unusual context—regardless of representational alignment. The paper offers no control separating (a) from (b): no norm/frequency-selected baseline, no permutation test, no experiment with different tokenizers or non-overlapping training prompts. Because the claimed advantage over CKA, SVCCA, and relative representations is exactly this layer-matching score (Table 1), a demonstration that the score is a token-statistics artifact would remove the main new contribution. This is a correctness risk rather than an internal inconsistency: the construction is well specified and reproducible, but its interpretation is underdetermined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Inference-Time Decomposition of Activations (ITDA), a dictionary-learning method that builds a sparse dictionary of model activations by greedily adding activations that are worst reconstructed by matching pursuit, and labels each atom with its source prompt-token. The authors claim ITDAs train about 100x faster and on 1% of the data relative to sparse autoencoders (SAEs), enabling dictionaries for Llama-3.1 70B and 405B on a single consumer GPU. They report that ITDAs reach reconstruction performance similar to SAEs on some Pythia models but are substantially worse on Gemma-2 2B. The central novelty is a representation-similarity index: the Jaccard overlap of two ITDA dictionaries' prompt-token labels, which is shown to outperform CKA, SVCCA, and relative representations on a layer-matching benchmark for GPT-2 small and medium.","tokens_in":20982,"tokens_out":4510,"duration_ms":49035,"significance":"If the representation-similarity claim holds, ITDAs would be a valuable low-cost tool for cross-model interpretability, model diffing, and studying very large models where SAE training is prohibitive. The paper is honest about the reconstruction trade-off and provides a reproducible algorithm plus public code. The Llama-3.1 70B/405B case study demonstrates a real scalability benefit. However, the headline contribution---state-of-the-art layer matching via Jaccard overlap of token labels---is not yet validated against a plausible confound: shared token statistics rather than representational geometry could drive the overlap. The reconstruction comparison is also under-specified in terms of dictionary sizes and sparsity levels. The significance of the paper therefore hinges on whether the Jaccard result survives the missing controls.","major_comments":[{"comment":"The Jaccard index in Eq. (8) is computed over sets of prompt-token labels assigned by Algorithm 1, but Algorithm 1 selects atoms based on reconstruction loss of individual activations. An atom can therefore appear in two models' dictionaries either because both representations genuinely share a hard-to-reconstruct direction or because the corresponding token is an outlier in the shared input distribution (e.g., high norm, rare frequency, unusual context) in both models. The GPT-2 instances in the layer-matching benchmark share a tokenizer and training corpus, so the latter pathway is fully confounded with the former. The paper provides no control such as a token-frequency or activation-norm baseline dictionary, a permutation test of labels, or a setting with different tokenizers or non-overlapping training prompts. Since the state-of-the-art claim in Table 1 rests entirely on this Jaccard score, the result is currently underdetermined.","section":"5.1 Model Instance Layer Similarity, Eq. (8)"},{"comment":"The comparison with SAEs is not matched for dictionary size or sparsity: ITDAs are cropped to a fixed size while SAEs have fixed dictionary sizes (Table 2), but the paper does not report the actual L0 values or dictionary sizes used for either method in Fig. 7. Nor does it report the ITDA dictionary sizes for Gemma-2 2B, where performance is reported as substantially worse. This makes it impossible to separate the effect of the method from the effect of dictionary capacity. Please report the matched settings, or explicitly frame the claim as 'similar reconstruction performance at the tested dictionary sizes and L0.'","section":"4.1 Reconstruction Performance, Fig. 7 and Table 2"},{"comment":"The headline layer-matching results in Table 1 are not reproducible as reported because the ITDA hyperparameters (threshold τ, L0, dictionary size, training data) for the GPT-2 small/medium dictionaries are not given. The Jaccard similarity is sensitive to all of these, and the reported gap over the Relative method (0.88 vs 0.87 for GPT-2 Small; 0.89 vs 0.78 for GPT-2 Medium) could change under different hyperparameter choices. The paper should state the hyperparameters for the Table 1 dictionaries and report variance across the five random initialisations.","section":"5.1, Table 1"}],"minor_comments":[{"comment":"The notation H, Horig, and H0 in Eq. (7) is used before being defined; please define these quantities in the text.","section":"Section 4.1, Eq. (7)"},{"comment":"The sentence 'The Jaccard similarity between different instances of the same model architecture ranges from 0.56 to 0.59; whereas the Jaccard similarity between different model architectures range from 0.46 to 0.47 (Figure 11)' refers to a figure that appears only in the appendix; please fix the cross-reference and add error bars or confidence intervals, since the ranges overlap in a way that may not be statistically significant.","section":"Section 5.2"},{"comment":"The sentence 'However, comparison of randomly sampled dictionaries, as is done with SAE decoders in (Lan et al., 2024).' is a sentence fragment and should be completed.","section":"Section 3.3"},{"comment":"The link in 'at [redacted]' for the released Llama ITDAs must be filled in before publication, as the current text is not usable by readers.","section":"Section 4.2"},{"comment":"The caption for Figure 12 in the appendix refers to 'A.4.2' while the figure is numbered 12; please align the cross-references.","section":"Appendix A.3.4"}],"recommendation":"major_revision","confidential_remarks":"The main new contribution of this paper is the Jaccard-based representation similarity index; the reconstruction results are honestly reported as mixed and are, on their own, only an incremental demonstration of a cheaper but lower-fidelity SAE substitute. The Jaccard confound (shared token statistics vs. representational similarity) is the key issue and needs to be addressed with concrete controls before the paper can be considered for publication. If the authors can provide those controls, the paper would be a useful contribution to the interpretability and representation-similarity literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the core method is genuinely useful and honestly reported: ITDAs are greedy dictionaries of raw activations paired with prompt-token labels, decomposed with matching pursuit. They train in minutes on a consumer GPU, scale to Llama-3.1 405B, reconstruct activations comparably to ReLU SAEs on Pythia, and the authors openly say they are worse on Gemma-2 and worse than TopK SAEs. That is a fair, non-overhyped contribution for anyone who needs cheap interpretability on very large models. Second, the more interesting claim, that Jaccard overlap of ITDA dictionaries is a SOTA representation-similarity index, has a load-bearing confound. All GPT-2 small and medium instances in the layer-matching benchmark share the same tokenizer and, through the Mistral pipeline, the same training corpus. Dictionary atoms are selected as worst-reconstructed activations, so a token position can enter both dictionaries because it is a statistical outlier in the shared input distribution, not because the two models represent it similarly. The paper provides no control: no random-dictionary baseline, no norm/frequency-selected selection, no permutation test, no experiment with different tokenizers or non-overlapping prompts. That control is necessary because the Jaccard score's layer-matching advantage over CKA, SVCCA, and relative representations is the paper's main new result. The reconstruction claim does not share this problem: it is a straightforward inference-time optimization against an explicit dictionary, and the comparison, while imperfect (unmatched dictionary sizes, different training data), is honestly caveated. The interpretability examples are cherry-picked but labeled as such, and the authors rightly warn against over-reading SAEBench metrics applied to ITDAs. This is a clear-thinking paper, well-situated in the literature, with code that mostly reproduces the experiments, though the repo lacks a commit hash and exact data specification. My read is that the stress-test concern stands: the layer-matching metric is underdetermined until the token-statistics confound is removed. That does not make the paper unserious; it makes the central similarity claim unproven. This paper deserves a serious referee: the method is cheap, reproducible, and fills a real gap for 70B-405B interpretability. I would send it to review with the explicit requirement that the authors add a token-statistics control and matched-baseline reconstruction comparisons. I would cite it for the ITDA method itself, but not yet for the representation-similarity result. Bring it to reading group; the confound discussion is worth having.","headline":"A well-written, honest introduction of a cheap SAE alternative whose headline Jaccard layer-matching claim needs a token-statistics control before it is believable.","tokens_in":21520,"tokens_out":1609,"would_cite":true,"duration_ms":15236,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ITDAs are greedy activation dictionaries that train in roughly 1% of an SAE's time and data, reach comparable reconstruction on some models, and yield a Jaccard-based representation similarity index that outperforms CKA, SVCCA, and…","keywords":["sparse autoencoders","dictionary learning","matching pursuit","representation similarity","Jaccard similarity","LLM interpretability","cross-model comparison","inference-time decomposition"],"falsifier":"Train ITDAs on two GPT-2 small instances with deliberately disjoint training corpora (for example, code versus Wikipedia) and rerun the layer-matching benchmark; if Jaccard accuracy drops to near chance, the similarity signal comes from overlapping training text rather than from aligned representations. A second check is to swap in a different tokenizer for one instance and see whether layer matching degrades.","tokens_in":20510,"feed_emoji":"🧩","tokens_out":7659,"duration_ms":46335,"temperature":0.7,"pith_summary":"Sparse autoencoders (SAEs) decompose LLM activations into interpretable latents but are costly to train and do not transfer across models. This paper introduces Inference-Time Decomposition of Activations (ITDA), which builds a dictionary from real activations by greedily adding any activation whose matching-pursuit reconstruction error exceeds a threshold, so each atom carries a natural prompt-token label. ITDAs train on roughly 1% of an SAE's tokens and time, making them feasible on Llama-3.1 70B and 405B with a single consumer GPU, with reconstruction performance comparable to SAEs on Pythia models though worse on Gemma-2. Because dictionary labels are shared across models, the paper defines a representation similarity index as the Jaccard similarity of two dictionaries and reports it outperforms CKA, SVCCA, and relative representation methods on the Kornblith layer-matching benchmark. The pith is that cheap, label-anchored dictionaries can serve both interpretability and cross-model comparison.","feed_headline":"Cheap activation dictionaries beat CKA, SVCCA at LLM layer matching","feed_subtitle":"ITDAs train in minutes on 1% of SAE data, and their Jaccard overlap outscores prior similarity metrics.","key_machinery":"The central object is the ITDA dictionary: a set of normalized residual-stream activations, each collected from a specific prompt and token position at a fixed layer, plus a matching-pursuit solver that decomposes a new activation into an $L_0$-sparse combination of dictionary atoms. The dictionary is built greedily by processing a training corpus, reconstructing each activation with the current dictionary, and adding the activation whenever the mean-squared reconstruction error exceeds a threshold $\\tau$, so dictionary size is set by the threshold rather than by a fixed architecture. The paper's key identity for cross-model comparison is the Jaccard index $S(M_0, M_1) := |D_0 \\cap D_1| / |D_0 \\cup D_1|$, where atoms are matched by their shared prompt-token labels rather than by their vector values. This label-based overlap is what carries the representation-similarity claim.","core_discovery":"On the paper's own terms, the discovery is that a dictionary of raw activations, selected greedily by reconstruction error and decomposed by matching pursuit, can replace a learned sparse autoencoder for many interpretability purposes at a fraction of the training cost, and its prompt-token labels make dictionaries from different models directly comparable. The authors claim that ITDAs match SAE reconstruction quality on some models (Pythia) while incurring a penalty on others (Gemma-2), that ITDAs find interpretable and monosemantic latents in Llama-3.1 70B and 405B, and that a plain Jaccard index over dictionary labels achieves state-of-the-art accuracy on layer matching between randomly initialized GPT-2 instances. This is the claim a sympathetic reader is asked to accept: one percent of the data and compute buys a decomposition that is competitive enough for many uses and uniquely transferable across models.","pith_inferences":["If the Jaccard signal is real, ITDA dictionaries are a cheap substrate for model diffing: comparing base and chat-fine-tuned versions of the same model could expose newly appearing or disappearing features with no SAE retraining.","The label-overlap metric depends on tokenizer and training data; a cross-tokenizer or cross-corpus control would tell whether the advantage reflects shared features or shared surface tokens.","The greedy threshold construction makes dictionary size a cost-quality knob, so ITDA could also serve as a fast exploratory stage that decides where expensive SAEs are worth training.","The method should transfer to other decomposition sites, such as attention outputs or MLP layers, since it only assumes that activations live in a space where cosine correlation is meaningful."],"forward_implications":["ITDA training requires around 1.2 million tokens and minutes of compute, so interpretability tools become available for very large models such as Llama-3.1 70B and 405B on a single consumer GPU.","On the Kornblith layer-matching benchmark, ITDA Jaccard similarity reaches 0.88 on GPT-2 small and 0.89 on GPT-2 medium, above the relative-representation baseline (0.87 and 0.78) and well above CKA and SVCCA.","ITDA dictionaries can distinguish model instances from architectures: Jaccard similarity is 0.56–0.59 within GPT-2 small or medium and 0.46–0.47 across them.","Because each ITDA atom is labelled by its source prompt and token, the sparse codes are interpretable without a separate feature-dashboard step, and dictionary differences can be read as model differences.","Compared with SAEs, ITDA reconstruction is comparable to ReLU SAEs on Pythia but worse than TopK SAEs and substantially worse on Gemma-2, so the paper positions ITDA as a complement rather than a replacement where modern SAEs already exist."],"supporting_citations":[{"why":"Supplies the matching pursuit algorithm used for both inference-time sparse coding and dictionary construction.","marker":"Mallat & Zhang, 1993"},{"why":"Motivates relative representation measures and anchor-based dictionaries, which ITDA extends with learned label-bearing anchors.","marker":"Moschella et al., 2022"},{"why":"Defines the CKA baseline and the layer-matching benchmark used to score similarity indices.","marker":"Kornblith et al., 2019"},{"why":"Defines the SVCCA baseline and the layer-convergence experiment that the paper replicates with ITDA dictionaries.","marker":"Raghu et al., 2017"},{"why":"Provides the TopK SAE architecture and training setup that serves as a key reconstruction and interpretability baseline.","marker":"Gao et al., 2025"},{"why":"Establishes the ReLU SAE and dashboard-style interpretation that ITDA is compared against for reconstruction and interpretability.","marker":"Bricken et al., 2023"},{"why":"Supplies the SAEBench evaluation suite used to measure CE loss scores and automated interpretability.","marker":"Karvonen et al., 2024a"},{"why":"Provides the Pythia models on which ITDA and SAE reconstruction are compared.","marker":"Biderman et al., 2023"},{"why":"Provides the GPT-2 small and medium models used in the layer-matching and model-similarity experiments.","marker":"Radford et al., 2019"},{"why":"Provides the Llama-3.1 70B and 405B models on which the paper demonstrates large-scale ITDA training.","marker":"Dubey et al., 2024"}],"fun_headline_variants":["ITDAs train on 1% of SAE data to match reconstruction on some models","Greedy activation dictionaries give cross-model comparability at low cost","Jaccard index on ITDA dictionaries beats CKA and SVCCA for layer alignment","Activation dictionaries train in 1% the time of sparse autoencoders","One-percent-cost activation dictionaries enable LLM interpretation and transfer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that dictionary atoms labelled by their source prompt-token remain comparable across models, so Jaccard overlap of those labels measures representational similarity; this premise is only demonstrated for GPT-2 small and medium with random initializations and could fail when tokenizers, training data, or dictionary thresholds differ.","fun_headline_variants_meta":{"raw":{"variants":["ITDAs train on 1% of SAE data to match reconstruction on some models","Greedy activation dictionaries give cross-model comparability at low cost","Jaccard index on ITDA dictionaries beats CKA and SVCCA for layer alignment","Activation dictionaries train in 1% the time of sparse autoencoders","One-percent-cost activation dictionaries enable LLM interpretation and transfer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000589,"raw_usage":{"total_tokens":2810,"prompt_tokens":1033,"completion_tokens":1777,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":649,"completion_tokens_details":{"reasoning_tokens":1676}},"tokens_in":649,"tokens_out":1777,"duration_ms":11936,"temperature":1.0,"reasoning_tokens":1676,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:40:16.717710+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train ITDAs on two GPT-2 small instances with deliberately disjoint training corpora (for example, code versus Wikipedia) and rerun the layer-matching benchmark; if Jaccard accuracy drops to near chance, the similarity signal comes from overlapping training text rather than from aligned representations. A second check is to swap in a different tokenizer for one instance and see whether layer matching degrades.","supporting_citations":[],"review_version":1}