REVIEW 9 cited by
Interpretability as Compression: Reconsidering SAE Explanations of Neural Activations with MDL-SAEs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Sparse Autoencoders (SAEs) have emerged as a useful tool for interpreting the internal representations of neural networks. However, naively optimising SAEs for reconstruction loss and sparsity results in a preference for SAEs that are extremely wide and sparse. We present an information-theoretic framework for interpreting SAEs as lossy compression algorithms for communicating explanations of neural activations. We appeal to the Minimal Description Length (MDL) principle to motivate explanations of activations which are both accurate and concise. We further argue that interpretable SAEs require an additional property, "independent additivity": features should be able to be understood separately. We demonstrate an example of applying our MDL-inspired framework by training SAEs on MNIST handwritten digits and find that SAE features representing significant line segments are optimal, as opposed to SAEs with features for memorised digits from the dataset or small digit fragments. We argue that using MDL rather than sparsity may avoid potential pitfalls with naively maximising sparsity such as undesirable feature splitting and that this framework naturally suggests new hierarchical SAE architectures which provide more concise explanations.
Forward citations
Cited by 9 Pith papers
-
Sparse Autoencoders Trained on the Same Data Learn Different Features
Seed variation alone causes sparse autoencoders to learn different, often equally interpretable feature sets, so SAE features are pragmatic decompositions rather than ground-truth units.
-
Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition
Attribution-based Parameter Decomposition splits a network's parameters into faithful, minimal, and simple components and recovers ground-truth mechanisms in toy models of superposition and compressed computation.
-
Decoder-Preserving Sparse Autoencoders: Which Readouts Survive Sparse Compression?
A new SAE objective penalizes disagreement between ridge prediction operators, preserving more linear readouts at equal reconstruction error.
-
FaithfulSAE: Towards Capturing Faithful Features with Sparse Autoencoders without External Dataset Dependencies
Training sparse autoencoders on a language model's own generated text can improve seed stability and downstream probing relative to training on web text.
-
Transcoders Beat Sparse Autoencoders for Interpretability
Skip transcoders beat sparse autoencoders on both reconstruction fidelity and automated interpretability scores for transformer MLP layers.
-
Stable and Steerable Sparse Autoencoders with Weight Regularization
L2 weight regularization in TopK SAEs increases cross-seed feature overlap and roughly doubles measured steering success on Pythia-70M, at the cost of collapsing most latents to zero.
-
Evaluating SAE interpretability without explanations
SAE latent interpretability can be scored directly from activation examples via intruder detection and embedding clustering, with LLM scores correlating strongly with human scores.
-
Train One Sparse Autoencoder Across Multiple Sparsity Budgets to Preserve Interpretability and Accuracy
HierarchicalTopK trains a single sparse autoencoder that reconstructs transformer activations well at many sparsity levels, matching or beating separate per-level models.
-
Evaluating Explanations: An Explanatory Virtues Framework for Mechanistic Interpretability -- The Strange Science Part I.ii
The paper introduces an Explanatory Virtues Framework and argues, via a qualitative rubric, that Compact Proofs are the most promising method for mechanistic interpretability.
Discussion (0). Continue with ORCID to comment.