REVIEW 7 cited by
Sparse Autoencoders Do Not Find Canonical Units of Analysis
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
A common goal of mechanistic interpretability is to decompose the activations of neural networks into features: interpretable properties of the input computed by the model. Sparse autoencoders (SAEs) are a popular method for finding these features in LLMs, and it has been postulated that they can be used to find a \textit{canonical} set of units: a unique and complete list of atomic features. We cast doubt on this belief using two novel techniques: SAE stitching to show they are incomplete, and meta-SAEs to show they are not atomic. SAE stitching involves inserting or swapping latents from a larger SAE into a smaller one. Latents from the larger SAE can be divided into two categories: \emph{novel latents}, which improve performance when added to the smaller SAE, indicating they capture novel information, and \emph{reconstruction latents}, which can replace corresponding latents in the smaller SAE that have similar behavior. The existence of novel features indicates incompleteness of smaller SAEs. Using meta-SAEs -- SAEs trained on the decoder matrix of another SAE -- we find that latents in SAEs often decompose into combinations of latents from a smaller SAE, showing that larger SAE latents are not atomic. The resulting decompositions are often interpretable; e.g. a latent representing ``Einstein'' decomposes into ``scientist'', ``Germany'', and ``famous person''. Even if SAEs do not find canonical units of analysis, they may still be useful tools. We suggest that future research should either pursue different approaches for identifying such units, or pragmatically choose the SAE size suited to their task. We provide an interactive dashboard to explore meta-SAEs: https://metasaes.streamlit.app/
Forward citations
Cited by 7 Pith papers
-
Verbalizable Representations Form a Global Workspace in Language Models
Language models represent their current reasoning in a small, readable set of verbalizable vectors (the J-space) that functions like a global workspace.
-
Sign-Aware Gated Sparse Autoencoders: Modeling Anticorrelated Features with Bi-Jump-ReLU Activations
SA-GSAE with Bi-Jump-ReLU enables one latent to encode both polarities of anticorrelated features, Pareto-dominating or matching full-width gated SAEs while reducing dead latents by up to 500x on some LLM hookpoints.
-
Making Interpretable Discoveries from Unstructured Data: A High-Dimensional Multiple Hypothesis Testing Approach
A new framework combines AI-derived concept embeddings with high-dimensional selective inference to enable statistically principled, interpretable discovery from unstructured data in empirical economics.
-
Teach Old SAEs New Domain Tricks with Boosting
Training a small secondary sparse autoencoder on the reconstruction error of a pretrained SAE improves domain-specific reconstruction and language-model perplexity without hurting general performance.
-
Stochastic Parameter Decomposition
SPD uses stochastic masking and a learned causal importance function to decompose neural network parameters into sparsely active rank-one subcomponents, recovering ground-truth mechanisms in toy models where APD struggled.
-
Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEs
A position paper proposing feature consistency, measured by PW-MCC, as a core SAE evaluation criterion, with evidence that TopK SAEs achieve high consistency on LLM activations.
-
Superposition Without Interference? Towards Isolated Interventions via Almost Orthogonal Features in Language Models
Orthogonality-regularizing a language model's sparse-autoencoder features modestly improves the model's ability to swap a named entity during generation, without hurting math-reasoning accuracy.
Discussion (0). Sign in to comment.