Pith. sign in

REVIEW 4 major objections 8 minor 1 cited by

FaithfulSAE: Towards Capturing Faithful Features with Sparse Autoencoders without External Dataset Dependencies

T0 review · 4 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that training sparse autoencoders on a language model's own synthetic text yields features that are more faithful to the model's internal structure, more stable across training seeds, and less contaminated by…

desk verdict Genuinely new data-source idea for SAEs; the BOS-sampling coverage problem is real and limits the faithfulness claim. read the letter →

arxiv 2506.17673 v1 pith:VELJK2NT submitted 2025-06-21 cs.LG cs.AIcs.CL

Seonglae Cho , Harryn Oh , Donghyun Lee , Luis Eduardo Rodrigues Vieira , Andrew Bermingham , Ziad El Sayed This is my paper
classification cs.LGcs.AIcs.CL
keywords sparseautoencodersmechanisticinterpretabilityfaithfulfeaturesout-of-distributiondatasyntheticdatasetgenerationfeaturestabilitySAEprobingFakeRatio
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sparse autoencoders (SAEs) turn a language model's internal activations into sparse, supposedly interpretable features, but they can be unstable across random seeds and can learn artifacts of the external text they were trained on. This paper proposes FaithfulSAE, which trains the SAE on a synthetic corpus generated by the model itself from a bare start-of-sequence token, so the training distribution stays inside what the model can actually produce. The experiments show that on non-instruction-tuned models this increases cross-seed Shared Feature Ratio compared with instruction-tuned datasets, and across seven models it lowers the Fake Feature Ratio (latents that fire on random inputs) in 5 of 7 cases and beats web-trained SAEs on classification probing in 12 of 18 cases. The authors argue that external web and instruction data are out-of-distribution for the model and therefore cause hallucinated features. If the claim holds, interpretability no longer requires access to the model's training corpus or the open web.

What carries the argument

The load-bearing object is the Faithful Dataset, produced by unconditional sampling from only the BOS token, intended to match the model's next-token distribution and thus its activation manifold. The evaluation machinery is feature matching: decoder weight vectors from two SAEs trained with different seeds are matched one-to-one by cosine similarity, yielding the Shared Feature Ratio $SFR$, while the Fake Feature Ratio $FFR$ counts latents whose activation frequency on random token sequences exceeds a threshold $\tau_f = 0.1$. The TopK SAE with scaling-law hyperparameters is the architecture carrying the training runs.

What would settle it

Compute the Fake Feature Ratio of a FaithfulSAE on a large held-out set of naturally occurring prompts, and compare it with a web-trained SAE on the same set; if FaithfulSAE is not lower there, or if its probing advantage disappears on tasks whose text is absent from its synthetic corpus, the central claim fails.

Watch

Extended reading notes

Core claim

The central claim is that SAE unreliability is driven by training on out-of-distribution data, and that using the model's own generated text removes that driver and produces features that better reflect model-internal structure. On the paper's metrics, FaithfulSAE does not uniformly win: web-trained SAEs often achieve higher Shared Feature Ratio, which the authors attribute to the diversity of web corpora, but FaithfulSAE wins on downstream probing and on Fake Feature Ratio. The paper therefore reframes 'faithfulness' away from raw dictionary similarity toward reconstruction quality, probing performance, and absence of spuriously activated latents. The recommended practice is to generate a model-specific synthetic dataset and train TopK SAEs on it, eliminating the external dataset dependency.

Load-bearing premise

The approach depends on the assumption that sampling from a bare start-of-sequence token covers the same range of internal activations the model produces in real use; the paper's own table shows first-token coverage as low as 0.40 percent for Gemma 2B, so this coverage is the point most worth testing.

Editorial extensions

If this is right

  • On non-instruction-tuned models, FaithfulSAE training yields higher cross-seed Shared Feature Ratio than SAEs trained on FLAN, Open-Instruct, or Alpaca instruction data, supporting the out-of-distribution explanation.
  • Across six model architectures and three probing tasks, FaithfulSAE reconstructions beat web-trained SAEs in 12 of 18 accuracy comparisons and were consistently best on CoLA.
  • In 5 of the 7 models tested, FaithfulSAE has a lower Fake Feature Ratio, meaning fewer latents fire spuriously on random token sequences.
  • SAE training can proceed with only the target model and no external text, removing a dependency for inspecting models whose training data is undisclosed or scarce.
  • FaithfulSAE does not beat web-trained SAEs on Shared Feature Ratio in most models, so the paper's claim is specifically about faithfulness and probing quality rather than raw dictionary stability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the causal story is right, perturbing the model's own samples with synthetic texts from an unrelated domain should degrade Shared Feature Ratio and raise Fake Feature Ratio; that is a direct test of the out-of-distribution explanation.
  • The paper's own appendix shows first-token coverage decays exponentially and reaches only 0.40 percent for one model, so BOS-only sampling likely under-covers the activation manifold; higher-temperature sampling or diverse seed prompts are a natural extension that could also push FaithfulSAE past web-trained SAEs on Shared Feature Ratio.
  • The data-independence is the sharpest practical consequence: for proprietary or domain-specific models whose training corpus is unavailable or sensitive, self-generated text may be the only viable SAE training distribution.
  • The probing gains could partly reflect train/evaluation distribution overlap rather than a genuinely better feature dictionary; causal tests that silence or amplify individual features would separate those explanations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper proposes FaithfulSAE, a method for training sparse autoencoders (SAEs) on a synthetic dataset generated by the target LLM itself, via unconditional sampling from the beginning-of-sequence (BOS) token, rather than on external web or instruction-tuned corpora. The central claim is that this 'Faithful dataset' better reflects the model's internal activation distribution, yielding SAEs that are more stable across seeds (higher Shared Feature Ratio), more faithful in downstream probing tasks, and less prone to 'Fake Features'. The authors evaluate on Pythia 1.4B/2.8B, GPT-2 Small, LLaMA 3.2 1B/3B, LLaMA 3.1 8B, and Gemma 2B, comparing against The Pile, FineWeb, and instruction datasets (FLAN, OpenInstruct, Alpaca). They report that FaithfulSAEs outperform instruction-dataset SAEs on SFR, achieve better probing accuracy in 12/18 comparisons, and show lower Fake Feature Ratio in 5/7 models, while acknowledging lower SFR relative to web-based datasets.

Significance. The paper addresses a real and timely problem: SAE training data can introduce artifacts that are not intrinsic to the model, and the field lacks a principled way to choose training data when the pretraining corpus is not public. The proposed idea of using the model's own generations is conceptually appealing and 'encapsulates' the model. If the faithfulness claims held, the method would remove dependence on external corpora for interpretability, which is a genuine advance. The authors also provide open-source code and trained models, and they report multiple complementary metrics across seven model architectures, which is a strength. However, the significance is currently tempered by the paper's own evidence that the Faithful dataset has low first-token coverage (e.g., 0.40% for Gemma 2B, KL 2.24) and that FaithfulSAEs underperform web-trained SAEs on the primary SFR metric. The positive evidence comes from probing and Fake Feature Ratio, which are indirect and, in the case of probing, not significance-tested. The contribution is therefore a promising empirical hypothesis rather than an established method.

major comments (4)
  1. [§3.1, Table 1, Appendix C] The central assumption that unconditional BOS-only sampling captures the model's activation distribution is not supported by the paper's own statistics. Table 1 reports first-token coverage as low as 0.40% (Gemma 2B) and KL divergences up to 2.2392, and Appendix C shows that the first-token frequency distribution decays exponentially, leaving long-tail tokens essentially unsampled. Since the SAE is trained on sequences whose initial tokens come from a narrow slice, the resulting dictionary may reflect a narrow manifold. The authors themselves suggest (Section 5.4) that a flat initial distribution or higher temperature could improve coverage, which is an admission that the current sampling is coverage-limited. This is load-bearing: the paper's claim that FaithfulSAEs 'better capture model-internal features' would not follow if the training distribution is unrepresentative. To support the claim, the authors should either demonstrate that the activation distribution induced by the Faithful dataset matches the model's general activation distribution (e.g., via activation-space statistics compared with web data) or re-run the full comparison with a sampling scheme that covers the BOS distribution more fully and show that the qualitative conclusions persist.
  2. [Table 4, §5.3] The paper's primary faithfulness metric, Shared Feature Ratio, is lower for FaithfulSAEs than for web-trained SAEs on every model considered (e.g., GPT-2: 0.5258 vs 0.5405 and 0.5209; LLaMA 3B: 0.1835 vs 0.2222 and 0.2248; LLaMA 8B: 0.0914 vs 0.1066 and 0.0936). The explanation that web datasets are 'sufficiently diverse' effectively concedes that the Faithful dataset is less representative than web data. Yet the abstract and conclusion claim that FaithfulSAEs are more faithful; this is an overstatement given the paper's own chosen metric. The conclusion should be revised to explicitly acknowledge that the SFR evidence supports the method only against instruction datasets, and that the case against web datasets rests on probing and Fake Feature Ratio.
  3. [§3.5, Table 7] The Fake Feature Ratio is defined as the proportion of features activating on more than tau_f of 'randomly generated token sequences' (treated as OOD). This measure conflates dataset coverage with feature quality: a feature that genuinely responds to rare but meaningful tokens might appear 'fake' simply because the random sequences do not exercise the same activation space as the training distribution. The paper does not validate that the flagged features are actually hallucinated, e.g., by ablating them and showing no behavioral effect or by inspecting their top activating contexts. Without such validation, the lower FFR of FaithfulSAEs in 5/7 models is difficult to interpret and cannot bear the weight of the faithfulness claim.
  4. [§5.6, Table 6] The probing advantage (12/18 cases) is reported without any measure of variance or statistical significance. Many of the reported differences are small (e.g., GPT-2 SST-2: 0.7746 vs 0.7723 and 0.7500; LLaMA 1B SST-2: Faithful is worse than both baselines). Since probing is the strongest positive evidence for the central claim, the authors should report per-seed results, confidence intervals, or significance tests (e.g., paired tests across seeds or bootstrap) to establish that the 12/18 result is not noise. This is essential given that the paper's own SFR results go the other direction.
minor comments (8)
  1. [Abstract] The abstract states the method 'eliminates the dependency on external datasets,' but the evaluation still relies on external probing datasets (SST-2, CoLA, AG News, Yelp). Clarify that the independence is restricted to SAE training data.
  2. [§6] The term 'encapsulation' in the conclusion is introduced without definition; please define it or remove it.
  3. [General] Figure and table numbering is inconsistent: Figure 5 is referenced in the text for LLaMA 8B but also appears in the appendix; Table 5 is hyperparameters while Table 6 is probing results; Table 7 is FFR. Renumber and ensure all in-text citations match.
  4. [Table 4] Table 4 column order (Pile, Faithful, FineWeb) differs from the order in the text and figure; make it consistent.
  5. [§3.5] The definition of activation frequency for Fake Feature Ratio should specify whether it is the fraction of random sequences where the feature is active (i.e., latent > 0) or something else, and over how many tokens per sequence.
  6. [§4] The paper states that tau_f = 0.1 follows Smith et al. (2025), but the cited source is a blog post; include a formal citation if available.
  7. [§5.4] The claim that KL divergence values 'stay below 2' is true except for Gemma 2B, but the sentence structure is ambiguous; clarify that Gemma 2B is the exception.
  8. [Appendix C] The appendix sections are misnamed; 'C.1 SAE Probing' contains a table but the appendix is about the Faithful dataset, causing confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found; the central claims rest on external benchmarks and fixed thresholds, not on a fitted parameter or self-citation chain.

full rationale

The paper's derivation chain is not circular. The Faithful dataset is defined independently as unconditional BOS-only samples from the target model (Section 3.1), and its quality is checked against token-coverage and KL-divergence statistics (Table 1, Appendix C) rather than assumed. The main claims—higher cross-seed Shared Feature Ratio than instruction datasets, better SAE probing accuracy in 12/18 cases, and lower Fake Feature Ratio in 5/7 models—are empirical comparisons evaluated with external test sets (SST-2, CoLA, AG News, Yelp) and fixed thresholds (tau_s=0.7, tau_f=0.1) that are either hand-set or borrowed from prior work, not tuned to force the reported outcome. No parameter is fitted to a subset and then renamed as a prediction. The paper contains no load-bearing self-citation: its references to prior SAE work (Paulo and Belrose; Heap et al.; Smith et al.) are external, and the authors' own released code and datasets are artifacts, not uniqueness assumptions. The admitted limitations—lower SFR than web-trained SAEs and first-token coverage as low as 0.40% with KL up to 2.24—are correctness and coverage risks, not circularity, because the paper explicitly reports them as weaknesses rather than redefining them away. Overall, the conclusions are not forced by the definitions or by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The method rests on a small number of domain assumptions about what 'faithful' means and how to measure it; no mathematical derivation is involved. Thresholds and the Fake Feature definition are the main hand-set elements.

free parameters (2)
  • SFR matching threshold tau_s = 0.7
    Set by hand in Section 4; all Shared Feature Ratio results are reported at this single threshold with no sensitivity analysis.
  • Fake Feature threshold tau_f = 0.1
    Adopted from Smith et al. (2025), cited in Section 3.5; no robustness check is performed.
assumptions (4)
  • ad hoc to paper Unconditional sampling from the BOS token captures the model's internal activation distribution.
    Section 3.1 defines the Faithful Dataset this way; Table 1 and Appendix C show low first-token coverage, so this premise is only partially satisfied.
  • ad hoc to paper Shared Feature Ratio via Hungarian matching on decoder weights with threshold 0.7 is a valid proxy for whether SAEs capture the same model-internal features.
    This metric is the primary evidence of stability in Section 3.4, but there is no external validation that SFR corresponds to faithfulness.
  • ad hoc to paper Features that activate frequently on random OOD token sequences are hallucinated and do not represent model internals.
    This is the operational definition in Section 3.5; no evidence is provided that such features misrepresent internal activations beyond their activation frequency.
  • domain assumption Web datasets (The Pile, FineWeb) are within the pretraining distribution for the tested models, so they are not OOD.
    Used in Section 5.3 to explain why FaithfulSAEs underperform on SFR; this is asserted rather than directly verified per model.
invented entities (1)
  • Fake Feature
    purpose: A latent SAE dimension that activates frequently on OOD random text, claimed to be hallucinated and to misrepresent model-internal activations.
    Defined only by activation frequency on OOD samples; no independent validation that such dimensions are hallucinated rather than legitimate rare or context-specific features.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FaithfulSAE: Towards Capturing Faithful Features with Sparse Autoencoders without External Dataset Dependencies." pith.science (2026). https://pith.science/paper/VELJK2NT

@misc{pith2026250617673,
  author       = {Pith},
  title        = {Pith review of: FaithfulSAE: Towards Capturing Faithful Features with Sparse Autoencoders without External Dataset Dependencies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VELJK2NT}},
  note         = {Machine review of arXiv:2506.17673}
}
read the original abstract

Sparse Autoencoders (SAEs) have emerged as a promising solution for decomposing large language model representations into interpretable features. However, Paulo and Belrose (2025) have highlighted instability across different initialization seeds, and Heap et al. (2025) have pointed out that SAEs may not capture model-internal features. These problems likely stem from training SAEs on external datasets - either collected from the Web or generated by another model - which may contain out-of-distribution (OOD) data beyond the model's generalisation capabilities. This can result in hallucinated SAE features, which we term "Fake Features", that misrepresent the model's internal activations. To address these issues, we propose FaithfulSAE, a method that trains SAEs on the model's own synthetic dataset. Using FaithfulSAEs, we demonstrate that training SAEs on less-OOD instruction datasets results in SAEs being more stable across seeds. Notably, FaithfulSAEs outperform SAEs trained on web-based datasets in the SAE probing task and exhibit a lower Fake Feature Ratio in 5 out of 7 models. Overall, our approach eliminates the dependency on external datasets, advancing interpretability by better capturing model-internal features while highlighting the often neglected importance of SAE training datasets.

Figures

Figures reproduced from arXiv: 2506.17673 by the authors.

Figure 1
Figure 1. Fake Feature Ratio for SAEs trained on Faith [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Shared Feature Ratio (SFR) comparison be [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Shared Feature Ratio by model and dataset. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Cross-Entropy difference between SAEs trained on different datasets. Colors represent training datasets: [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Faithful SAE representation for LLaMa 8B. This figure shows the SAE’s reconstruction of the LLaMa 8B [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: SAE Probing performance comparison between FaithfulSAE and Web-based SAEs with different types of [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Faithful SAE representation for GPT-2. This figure visualizes the SAE model’s ability to reconstruct [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Faithful SAE representation for LLaMA 1B. This figure demonstrates the SAE’s performance in [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Faithful SAE representation for LLaMA 3B. This figure highlights the SAE’s reconstruction quality for [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Faithful SAE representation for Gemma 2B. This figure shows the SAE’s reconstruction of the Gemma [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: This figure compares the token distribution of the generated dataset for GPT-2 with the model’s expected [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: This figure compares the token distribution of the generated dataset for LLaMA 1B with the model’s [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: This comparison shows the token distribution of LLaMA 3B’s generated dataset versus the model’s [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: This figure visualizes how well the generated dataset represents LLaMA 8B’s token distribution. [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: This visualization compares the generated token distribution with the original model for Gemma 2B. [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 16
Figure 16. Figure 16: This figure shows the token distribution for the generated Pythia 1.4B dataset, comparing it to the model’s [PITH_FULL_IMAGE:figures/full_fig_p017_16.png]
Figure 17
Figure 17. Figure 17: This figure shows the token distribution for the generated Pythia 2.8B dataset, comparing it to the model’s [PITH_FULL_IMAGE:figures/full_fig_p017_17.png]
Figure 18
Figure 18. Figure 18: This figure shows the token distribution for the generated Pythia 6.9B dataset, comparing it to the model’s [PITH_FULL_IMAGE:figures/full_fig_p018_18.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Stable SAE features dominate functional signal while unstable features concentrate in reproducible subspaces, allowing more stable SAEs via cross-seed pooling.

Reference graph

Works this paper leans on

53 extracted references · 27 canonical work pages · cited by 1 Pith paper

  1. [1]

    Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2018. https://doi.org/10.1162/tacl_a_00034 Linear algebraic structure of word senses, with applications to polysemy . Transactions of the Association for Computational Linguistics, 6:483--495

    doi:10.1162/tacl_a_00034 2018

  2. [2]

    Pearce, and Lee Sharkey

    Kola Ayonrinde, Michael T. Pearce, and Lee Sharkey. 2024. https://arxiv.org/abs/2410.11179 Interpretability as compression: Reconsidering sae explanations of neural activations with mdl-saes . Preprint, arXiv:2410.11179

    arXiv 2024

  3. [3]

    Nikita Balagansky, Ian Maksimov, and Daniil Gavrilov. 2025. https://openreview.net/forum?id=MDvecs7EvO Mechanistic permutability: Match features across layers . In The Thirteenth International Conference on Learning Representations

    work page 2025

  4. [4]

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. 2023. https://proceedings.mlr.press/v202/biderman23a.html Pythia: A suite for analyzing large language models across trai...

    work page 2023

  5. [5]

    Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G Dimakis. 2017. https://proceedings.mlr.press/v70/bora17a/bora17a.pdf Compressed sensing using generative models . In Proceedings of the 34th International Conference on Machine Learning (ICML), volume 70 of Proceedings of Machine Learning Research, pages 537--546, Sydney, Australia. PMLR

    work page 2017

  6. [6]

    Trenton Bricken, Joshua Batson, Adly Templeton, Adam Jermyn, Tom Henighan, and Chris Olah. 2023 a . https://transformer-circuits.pub/2023/may-update/index.html#simple-factorization Features as the simplest factorization . Part of the May 2023 Circuits Updates by the Anthropic interpretability team

    work page 2023

  7. [7]

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, and 6 others. 2023 b . https://transformer-circuits.pub/2023/monosemantic-featu...

    work page 2023

  8. [8]

    Bart Bussmann, Patrick Leask, and Neel Nanda. 2024. https://openreview.net/forum?id=d4dpOCqybL Batchtopk sparse autoencoders . In NeurIPS 2024 Workshop on Scientific Methods for Understanding Deep Learning

    work page 2024

Show all 53 references
  1. [9]

    Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. 2019. https://doi.org/10.23915/distill.00015 Activation atlas . Distill

    2019 doi

  2. [10]

    D.L. Donoho. 2006. https://doi.org/10.1109/TIT.2006.871582 Compressed sensing . IEEE Transactions on Information Theory, 52(4):1289--1306

    2006

  3. [11]

    Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. 2024. https://openreview.net/forum?id=J6zHcScAo0 Transcoders find interpretable LLM feature circuits . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  4. [12]

    Michael Elad. 2010. https://doi.org/10.1007/978-1-4419-7011-4 Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing , 1 edition. Springer New York

    2010 doi

  5. [13]

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. https://transform...

    2022

  6. [14]

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, and ...

    2021

  7. [15]

    Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah A. Smith. 2015. https://doi.org/10.3115/v1/P15-1144 Sparse overcomplete word vector representations . In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th Inte...

    2015 doi

  8. [16]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2021. https://arxiv.org/abs/2101.00027 The pile: An 800gb dataset of diverse text for language modeling . C...

    2021 arXiv

  9. [17]

    Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. https://openreview.net/forum?id=tcsZt9ZNKD Scaling and evaluating sparse autoencoders . In The Thirteenth International Conference on Learning Rep...

    2024

  10. [18]

    Gabriel Goh, Nick Cammarata †, Chelsea Voss †, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. 2021. https://doi.org/10.23915/distill.00030 Multimodal neurons in artificial neural networks . Distill

    2021 doi

  11. [19]

    Aaron Gokaslan and Vanya Cohen. 2019. Openwebtext corpus. https://skylion007.github.io/OpenWebTextCorpus/

    2019

  12. [20]

    Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. 2023. https://openreview.net/forum?id=JYs1R9IMJr Finding neurons in a haystack: Case studies with sparse probing . Transactions on Machine Learning Research

    2023

  13. [21]

    Thomas Heap, Tim Lawson, Lucy Farnik, and Laurence Aitchison. 2025. https://arxiv.org/abs/2501.17727 Sparse autoencoders can interpret randomly initialized transformers . Preprint, arXiv:2501.17727

    2025

  14. [22]

    Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. 2023. https://openreview.net/forum?id=F76bwRSLeK Sparse autoencoders find highly interpretable features in language models . In The Twelfth International Conference on Learning Representations

    2023

  15. [23]

    Harold W. Kuhn. 1955. https://web.eecs.umich.edu/ pettie/matching/Kuhn-hungarian-assignment.pdf The hungarian method for the assignment problem . Naval Research Logistics (NRL), 52

    1955

  16. [24]

    Daniil Laptev, Nikita Balagansky, Yaroslav Aksenov, and Daniil Gavrilov. 2025. https://arxiv.org/abs/2502.03032 Analyze feature flow to enhance interpretation and steering in language models . Preprint, arXiv:2502.03032

    2025 arXiv

  17. [25]

    Patrick Leask, Bart Bussmann, Michael T Pearce, Joseph Isaac Bloom, Curt Tigges, Noura Al Moubayed, Lee Sharkey, and Neel Nanda. 2025. https://openreview.net/forum?id=9ca9eHNrdH Sparse autoencoders do not find canonical units of analysis . In The Thirteenth International Confe...

    2025

  18. [26]

    Yuanzhi Li and Ronen Eldan. 2024. https://openreview.net/forum?id=yiPtWSrBrN Tinystories: How small can language models be and still speak coherent english

    2024

  19. [27]

    Bo Liu, Li-Ming Zhan, Zexin Lu, Yujie Feng, Lei Xue, and Xiao-Ming Wu. 2024. https://aclanthology.org/2024.lrec-main.720/ How good are LLM s at out-of-distribution detection? In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resou...

    2024

  20. [28]

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. https://proceedings.mlr.press/v202/longpre23a.html The flan collection: Designing data and methods for effective instruction tuning . I...

    2023

  21. [29]

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. https://doi.org/10.23915/distill.00024.001 Zoom in: An introduction to circuits . Distill. Https://distill.pub/2020/circuits/zoom-in

    2020 doi

  22. [30]

    Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. 2017. https://doi.org/10.23915/distill.00007 Feature visualization . Distill

    2017 doi

  23. [31]

    Olshausen and David J

    Bruno A. Olshausen and David J. Field. 1997. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision Research, 37(23):3311--3325

    1997

  24. [32]

    Gonçalo Paulo and Nora Belrose. 2025. https://arxiv.org/abs/2501.16615 Sparse autoencoders trained on the same data learn different features . Preprint, arXiv:2501.16615

    2025 arXiv

  25. [33]

    Guilherme Penedo, Hynek Kydl \' c ek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. https://openreview.net/forum?id=n6SCkn2QaG The fineweb datasets: Decanting the web for the finest text data at scale . In The Thirt...

    2024

  26. [34]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf Language models are unsupervised multitask learners . OpenAI. Accessed: 2024-11-15

    2019

  27. [35]

    Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, Janos Kramar, and Neel Nanda. 2025. https://openreview.net/forum?id=mMPaQzgzAN Jumping ahead: Improving reconstruction fidelity with jumpre LU sparse autoencoders

    2025

  28. [36]

    Ludwig Schubert, Chelsea Voss, Nick Cammarata, Gabriel Goh, and Chris Olah. 2021. https://doi.org/10.23915/distill.00024.005 High-low frequency detectors . Distill

    2021 doi

  29. [37]

    Lee Sharkey, Dan Braun, and Beren Millidge. 2022. https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj/interim-research-report-taking-features-out-of-superposition Interim research report: Taking features out of superposition with sparse autoencoders . AI Alignment Forum, po...

    2022

  30. [38]

    Lewis Smith, Senthooran Rajamanoharan, Arthur Conmy, Callum McDougall, Tom Lieberum, János Kramár, Rohin Shah, and Neel Nanda. 2025. Negative results for saes on downstream tasks and deprioritising sae research. https://www.lesswrong.com/posts/4uXCAJNuPKtKBsi28/sae-progress-up...

    2025

  31. [39]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. https://aclanthology.org/D13-1170/ Recursive deep models for semantic compositionality over a sentiment treebank . In Proceedings of the 2013 Conference on Em...

    2013

  32. [40]

    Alessandro Stolfo, Ben Peng Wu, and Mrinmaya Sachan. 2025. https://openreview.net/forum?id=Zlx6AlEoB0 Antipodal pairing and mechanistic signals in dense SAE latents . In ICLR 2025 Workshop on Building Trust in Language Models and Applications

    2025

  33. [41]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. https://crfm.stanford.edu/2023/03/13/alpaca.html Alpaca: A Strong , Replicable Instruction - Following Model

    2023

  34. [42]

    Gemma Team. 2024 a . https://arxiv.org/abs/2408.00118 Gemma 2: Improving open language models at a practical size . Preprint, arXiv:2408.00118

    2024 arXiv

  35. [43]

    Llama Team. 2024 b . https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  36. [44]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua ...

    2024

  37. [45]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Advances in Ne...

    2017

  38. [46]

    Smith, Iz Beltagy, and Hannaneh Hajishirzi

    Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. https://openreview.net/forum?id=w4zZNC4ZaV How far can camels go? exploring the state of instruction...

    2023

  39. [47]

    Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. https://doi.org/10.1162/tacl_a_00290 Neural network acceptability judgments . Transactions of the Association for Computational Linguistics, 7:625--641

    2019 doi

  40. [48]

    Dai, and Quoc V Le

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022 a . https://openreview.net/forum?id=gEZrGCozdqR Finetuned language models are zero-shot learners . In International Conference on Learning Representations

    2022

  41. [49]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022 b . https://openreview.net/forum?id=yzkSU...

    2022

  42. [50]

    Linyi Yang, Yaoxian Song, Xuan Ren, Chenyang Lyu, Yidong Wang, Jingming Zhuo, Lingqiao Liu, Jindong Wang, Jennifer Foster, and Yue Zhang. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.276 Out-of-distribution generalization in natural language processing: Past, present, and...

    2023 doi

  43. [51]

    Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. https://proceedings.neurips.cc/paper_files/paper/2015/file/250cf8b51c773f3f8dc8b4be867a9a02-Paper.pdf Character-level convolutional networks for text classification . In Advances in Neural Information Processing Systems, volume 28...

    2015

  44. [52]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  45. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.