REVIEW 13 cited by
Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Sparse autoencoders (SAEs) are a popular method for interpreting concepts represented in large language model (LLM) activations. However, there is a lack of evidence regarding the validity of their interpretations due to the lack of a ground truth for the concepts used by an LLM, and a growing number of works have presented problems with current SAEs. One alternative source of evidence would be demonstrating that SAEs improve performance on downstream tasks beyond existing baselines. We test this by applying SAEs to the real-world task of LLM activation probing in four regimes: data scarcity, class imbalance, label noise, and covariate shift. Due to the difficulty of detecting concepts in these challenging settings, we hypothesize that SAEs' basis of interpretable, concept-level latents should provide a useful inductive bias. However, although SAEs occasionally perform better than baselines on individual datasets, we are unable to design ensemble methods combining SAEs with baselines that consistently outperform ensemble methods solely using baselines. Additionally, although SAEs initially appear promising for identifying spurious correlations, detecting poor dataset quality, and training multi-token probes, we are able to achieve similar results with simple non-SAE baselines as well. Though we cannot discount SAEs' utility on other tasks, our findings highlight the shortcomings of current SAEs and the need to rigorously evaluate interpretability methods on downstream tasks with strong baselines.
Forward citations
Cited by 13 Pith papers
-
Laguerre Geometry for Interpreting Large Language Models
LLM concepts are Laguerre–Voronoi cells; Geometric Lens reads the exact cell of any hidden vector by isolating residual piecewise-linear flow from cross-token attention transport.
-
SynthSAEBench: Evaluating Sparse Autoencoders on Scalable Realistic Synthetic Data
SynthSAEBench-16k is a large-scale synthetic SAE benchmark with ground-truth features on which all tested SAE architectures fall short, and on which Matching Pursuit SAEs gain reconstruction by exploiting superposition noise.
-
Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders
Incorrect L0 makes sparse autoencoders mix correlated features rather than disentangling them, and a decoder projection metric can identify the correct L0.
-
TRACE: Training and Inference-Time Interpretability Analysis for Language Models
TRACE is a modular training-time interpretability toolkit that, combined with the ABSynth synthetic corpus generator, reveals phased emergence of syntax and semantics during transformer training.
-
Position: Use Sparse Autoencoders to Discover Unknowns
Sparse autoencoders are best used to discover unknown concepts, not to act on known concepts.
-
Fine-Grained Interpretation of Political Opinions in Large Language Models
Four-dimensional political concept vectors learned from LLM internals can detect and partially steer political leanings better than a single left-right axis.
-
TRACE for Tracking the Emergence of Semantic Representations in Transformers
Using Hessian curvature, intrinsic dimensionality, and linguistic probes on a synthetic frame-semantic corpus, the paper claims a coordinated intersection-based phase transition in small transformers, though the marke...
-
Inference-Time Decomposition of Activations (ITDA): A Scalable Approach to Interpreting Large Language Models
ITDA dictionaries, built greedily from poorly reconstructed activations and decomposed by matching pursuit, match some SAE reconstruction performance at 100x lower training cost and enable SOTA cross-model layer simil...
-
Screening of Biosecurity Features in Metagenomic Data with Evo 2 Probes
Frozen Evo 2 layer-26 embeddings carry linearly accessible AMR and virulence signals that simple probes read out with high held-out AUC, including on simulated short reads.
-
Stable and Steerable Sparse Autoencoders with Weight Regularization
L2 weight regularization in TopK SAEs increases cross-seed feature overlap and roughly doubles measured steering success on Pythia-70M, at the cost of collapsing most latents to zero.
-
When Benchmarks Lie: Evaluating Malicious Prompt Classifiers Under True Distribution Shift
Standard train/test evaluation overstates prompt-attack classifier generalization by 8+ AUC points; leave-one-dataset-out reveals dataset shortcuts that CV hides.
-
BlueGlass: A Framework for Composite AI Safety
BlueGlass provides composite AI safety infrastructure; its case studies on object-detection VLMs reveal dataset trade-offs, a decoder-layer phase transition in probe accuracy, and SAE-discovered concepts including spu...
-
Cross-Layer Discrete Concept Discovery for Interpreting Language Models
CLVQ-VAE maps lower-layer transformer activations to higher-layer ones through a discrete codebook, yielding concept vectors evaluated with probe ablation and human annotation.
Discussion (0). Sign in to comment.