REVIEW 7 cited by
Efficient Dictionary Learning with Switch Sparse Autoencoders
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Sparse autoencoders (SAEs) are a recent technique for decomposing neural network activations into human-interpretable features. However, in order for SAEs to identify all features represented in frontier models, it will be necessary to scale them up to very high width, posing a computational challenge. In this work, we introduce Switch Sparse Autoencoders, a novel SAE architecture aimed at reducing the compute cost of training SAEs. Inspired by sparse mixture of experts models, Switch SAEs route activation vectors between smaller "expert" SAEs, enabling SAEs to efficiently scale to many more features. We present experiments comparing Switch SAEs with other SAE architectures, and find that Switch SAEs deliver a substantial Pareto improvement in the reconstruction vs. sparsity frontier for a given fixed training compute budget. We also study the geometry of features across experts, analyze features duplicated across experts, and verify that Switch SAE features are as interpretable as features found by other SAE architectures.
Forward citations
Cited by 7 Pith papers
-
Sign-Aware Gated Sparse Autoencoders: Modeling Anticorrelated Features with Bi-Jump-ReLU Activations
SA-GSAE with Bi-Jump-ReLU enables one latent to encode both polarities of anticorrelated features, Pareto-dominating or matching full-width gated SAEs while reducing dead latents by up to 500x on some LLM hookpoints.
-
Incorporating Hierarchical Semantics in Sparse Autoencoder Architectures
A two-level mixture-of-experts sparse autoencoder models parent and child concepts together, improving reconstruction and reducing feature redundancy on Gemma 2-2B activations compared to flat top-k SAEs.
-
HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation
Attention outputs in transformers occupy a subspace with about 60% effective rank, and starting sparse dictionaries inside that subspace reduces dead features from 87% to below 1%.
-
Insights into a radiology-specialised multimodal large language model with sparse autoencoders
Applying Matryoshka sparse autoencoders to a radiology-specialised multimodal LLM reveals a minority of interpretable clinical features, while steering them produces unreliable and often off-target report changes.
-
Evaluate Bias without Manual Test Sets: A Concept Representation Perspective for LLMs
BiasLens uses concept activation vectors and sparse autoencoders to estimate LLM bias from internal representations, reporting moderate to strong agreement with behavioral bias metrics in a small evaluation.
-
Interpreting CFD Surrogates through Sparse Autoencoders
Sparse autoencoders trained on frozen MeshGraphNets embeddings yield feature dictionaries whose mesh-space activations align with high-vorticity regions better than embedding-norm, PCA, or random baselines.
-
Sparsification and Reconstruction from the Perspective of Representation Geometry
Sparse encoding appears to stratify and compress feature representations, but the claimed causal link between cluster separation and reconstruction is not supported.
Discussion (0). Continue with ORCID to comment.