Pith. sign in

REVIEW 10 cited by

Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Small

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2409.04478 v1 pith:HPG2X7MD submitted 2024-09-05 cs.LG cs.AIcs.NE

Maheep Chaudhary , Atticus Geiger This is my paper · ORCID
classification cs.LGcs.AIcs.NE
keywords featuressaesgpt-2smallanalysisautoencodersbaselinecity
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

A popular new method in mechanistic interpretability is to train high-dimensional sparse autoencoders (SAEs) on neuron activations and use SAE features as the atomic units of analysis. However, the body of evidence on whether SAE feature spaces are useful for causal analysis is underdeveloped. In this work, we use the RAVEL benchmark to evaluate whether SAEs trained on hidden representations of GPT-2 small have sets of features that separately mediate knowledge of which country a city is in and which continent it is in. We evaluate four open-source SAEs for GPT-2 small against each other, with neurons serving as a baseline, and linear features learned via distributed alignment search (DAS) serving as a skyline. For each, we learn a binary mask to select features that will be patched to change the country of a city without changing the continent, or vice versa. Our results show that SAEs struggle to reach the neuron baseline, and none come close to the DAS skyline. We release code here: https://github.com/MaheepChaudhary/SAE-Ravel

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Where You Measure Decides What You Measure: Position Selection in Ablation-Based SAE Evaluation

    cs.LG 2026-08 accept novelty 7.0 of 10

    In ablation-based SAE evaluation, the measurement token is selected by the dictionary under test, and holding that token fixed collapses most of the variance that is usually attributed to differences between dictionaries.

  2. Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effects

    cs.LG 2026-07 conditional novelty 7.0 of 10

    SAE features can be interpretable and causally useful yet still lack a stable one-dimensional logit direction for steering, with value-like features more structured than pointer-like ones.

  3. MIB: A Mechanistic Interpretability Benchmark

    cs.LG 2025-04 conditional novelty 7.0 of 10

    MIB, a multi-task benchmark for mechanistic interpretability, reports that attribution and mask optimization methods excel at circuit localization, and DAS outperforms sparse autoencoder features at causal variable lo...

  4. Less is Enough: Synthesizing Diverse Data in LLM Feature Space with Sparse Autoencoders

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Coverage of sparse-autoencoder-identified task features predicts post-training performance and can guide synthesis of small, high-impact datasets (2,000 vs. 300,000 samples).

  5. Sparse Autoencoders, Again?

    cs.LG 2025-06 conditional novelty 6.0 of 10

    VAEase gates the VAE decoder input by the encoder's variance, combining sparse-autoencoder adaptive sparsity with a hyperparameter-free loss; a global-minimizer theorem says active latent dimensions recover per-manifo...

  6. Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words

    cs.CL 2025-01 conditional novelty 6.0 of 10

    PS-Eval, a Word-in-Context-based benchmark, shows that SAEs optimized for MSE-L0 do not necessarily extract better word-meaning features, and that separation improves in deeper layers and attention outputs.

  7. LLM Scheming Inversely Scales with Pretraining Language Coverage

    cs.AI 2026-06 reject novelty 5.0 of 10

    A Qwen3 model exhibits higher scheming scores in low-resource languages than in English and Chinese, suggesting alignment does not transfer uniformly across languages.

  8. Discovering Chunks in Neural Embeddings for Interpretability

    cs.LG 2025-02 conditional novelty 5.0 of 10

    Recurring 'chunks' in neural embeddings can be extracted, predict input patterns, and be perturbed to steer a model's outputs.

  9. Steering Language Model Refusal with Sparse Autoencoders

    cs.LG 2024-11 conditional novelty 5.0 of 10

    Boosting one SAE 'refusal' feature in Phi-3 Mini and Llama 3.1 raises refusal rates on unsafe and safe prompts alike while sharply reducing MMLU, TruthfulQA, and GSM8K accuracy.

  10. Perspectives for Direct Interpretability in Multi-Agent Deep Reinforcement Learning

    cs.AI 2025-02 unverdicted novelty 4.0 of 10

    A perspective paper that advocates direct, post hoc interpretability for multi-agent deep reinforcement learning and offers a taxonomy of where those methods might apply.

Pith tools