Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Train One Sparse Autoencoder Across Multiple Sparsity Budgets to Preserve Interpretability and Accuracy

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single sparse autoencoder can serve every sparsity budget at once.

desk verdict A simple, cheap training trick that mostly delivers what it promises, but the Pareto claim overreaches the evidence. read the letter →

arxiv 2505.24473 v2 pith:G2ZKLKWA submitted 2025-05-30 cs.LG cs.AI

Nikita Balagansky , Yaroslav Aksenov , Daniil Laptev , Vadim Kurochkin , Gleb Gerasimov , Nikita Koryagin , Daniil Gavrilov This is my paper
classification cs.LGcs.AI
keywords sparseautoencodershierarchicaltrainingobjectiveTopKactivationinterpretabilityexplainedvarianceParetofrontierfeaturehierarchiesneuralnetwork
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes HierarchicalTopK, a training objective that makes one sparse autoencoder reconstruct a Transformer's hidden representations well at every sparsity level up to a chosen maximum, instead of only at the single level it was trained for. The claim is that this single model lands on the Pareto frontier of sparsity versus explained variance, matching or beating separately trained TopK, BatchTopK, and JumpReLU autoencoders while remaining interpretable at the largest sparsity. If true, practitioners no longer need to train a new autoencoder for each sparsity budget, and can choose the budget at inference time. The experiments are carried out on layer-12 activations of a 2-billion-parameter language model using about a billion tokens of web text.

What carries the argument

The central object is Equation (1), the hierarchical loss: $L_{\mathrm{hierarchical}} = \frac{1}{|J|} \sum_{j \in J} L^j_{\mathrm{rec}}$, where $L^j_{\mathrm{rec}}$ is the mean-squared error using only the top-$j$ active latents. Computed as a cumulative sum of decoder embeddings scaled by activations, it forces the learned ordering of features to be a hierarchy: earlier, higher-activation features must be individually useful while later features refine the reconstruction. This ordering is also what preserves interpretability, because the top-ranked latents stay meaningful at every budget. A subsampled version, evaluating the loss every eighth index, gives the paper's reported eight-fold reduction in theoretical FLOPs without changing reconstruction quality.

What would settle it

Construct a synthetic input distribution where the optimal single feature and the optimal pair of features are different. Train a HierarchicalTopK SAE and a standard TopK SAE with $k=2$; if the fixed top-2 prefix of the hierarchical model has higher FVU than the separately trained $k=2$ model, the Pareto-optimality claim fails.

Watch

Extended reading notes

Core claim

Standard sparse autoencoders trained with a fixed top-k activation only optimise reconstruction for that k. HierarchicalTopK instead averages the reconstruction loss over all prefixes $j \le K$ of the activation-sorted latent list: $\hat{x}_j = \sum_{i \in \mathrm{top}_j} l_i e_i + b_{\mathrm{dec}}$, so a single decoder is pressed into being a good $j$-sparse reconstruction for every $j$. The paper reports that a single HierarchicalTopK model with $K=128$ matches or surpasses separately trained TopK and BatchTopK models at $k \in \{32, 64, 128\}$ in fraction of unexplained variance, keeps a stable AutoInterp interpretability score when moving from $\ell_0=32$ to $\ell_0=128$, and avoids the drop into 'almost dead' features that fixed-$k$ models show when evaluated at smaller $\ell_0$.

Load-bearing premise

The method assumes that sorting features once by activation magnitude gives the best possible reconstruction for every smaller sparsity budget, so one fixed ranking can serve all budgets.

Editorial extensions

If this is right

  • A single autoencoder trained once can be evaluated at any $\ell_0 \le K$, so sparsity can be selected after training based on the downstream compute or fidelity requirement.
  • Replacing a sweep of separately trained SAEs with one HierarchicalTopK model cuts training cost roughly by the number of budgets in the sweep.
  • The method's Pareto-frontier result means that, for the tested setting, there is no reconstruction-quality penalty for demanding flexibility from one model.
  • Higher sparsity budgets no longer imply a loss of interpretability, because the hierarchical model's AutoInterp score at $\ell_0=128$ stays close to its score at $\ell_0=32$.
  • Approximating the loss by evaluating every 8th sparsity level gives nearly the same Pareto frontier at one-eighth the theoretical loss computation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the activation-magnitude ordering is a stable feature hierarchy, the same cumulative-reconstruction objective could produce progressive summaries of a representation, letting downstream tasks read only the top $m$ features with predictable fidelity.
  • Beyond the paper: the method's advantage should depend on how stable the optimal feature ordering is across contexts; on representations where the best single feature and best two-feature set use different features, the fixed-order assumption would be testable and could break.
  • Beyond the paper: the subsampling result suggests an even sparser index set (e.g., powers of two) might suffice for many budgets, a cheap ablation the paper did not run.
  • Beyond the paper: the claims are established for one model and one dataset; extending to other architectures and layers would show whether the Pareto gain is a property of the objective or of this particular representation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes HierarchicalTopK, a sparse-autoencoder training objective that trains one SAE to reconstruct inputs at every sparsity level j <= K by minimizing the average reconstruction error over the top-j prefixes of a single sorted activation list (Eq. (1)). The method is evaluated on Gemma-2 2B representations using FineWeb data, with FVU, AutoInterp, and dead-feature metrics. The authors report that a single HierarchicalTopK model matches or surpasses TopK and BatchTopK SAEs trained individually at k in {32, 64, 128}, preserves interpretability at higher l0, and can be subsampled to every 8th sparsity level without performance loss.

Significance. If the claims hold, the method is practically valuable: it offers a single model whose sparsity can be adjusted at inference time, with an efficient fused-kernel implementation and apparently no loss of reconstruction quality or interpretability relative to per-level trained baselines. The objective is simple and clearly specified, and the paper includes a minimal PyTorch implementation that makes the method easy to reproduce. The main limitation is that the central 'Pareto-optimal' claim depends on an unexamined nested-support assumption, and the experimental evidence is single-run without variance estimates or a comparison to the closest prior structural method (Matryoshka SAEs). These issues are fixable, so the contribution is conditionally significant.

major comments (3)
  1. [§2, Eq. (1)] The definition x_hat_j = sum_{i in top_j} l_i(x) e_i + b_dec fixes the support of every j-sparse reconstruction to be a prefix of the same pre-sorted activation list. Equation (1) therefore minimizes the average error only over this nested family of reconstructions. For a general non-orthogonal decoder dictionary, the MSE-optimal j-sparse support need not be a prefix of the MSE-optimal (j+1)-sparse support, so the hierarchical loss enforces a structural constraint rather than computing the true unconstrained Pareto frontier. The headline claim that HierarchicalTopK 'achieves Pareto-optimal trade-offs' at every l0 is thus not implied by the objective and is not tested: there is no oracle comparison (e.g., OMP with the same decoder), no comparison to Matryoshka SAEs, and no diagnostic for support nestedness. Section 6's limitations list only evaluation scope and interpretability proxies, not this structural assumption. I would ask for either an experiment that isolates the cost of the prefix constraint or a reformulated, weaker claim.
  2. [§3.2–3.3, Figs. 2–3] All reported FVU curves are based on a single training run; no error bars, standard deviations, or repeated-seed results are provided. The central quantitative claim is that the Hierarchical model 'matches or surpasses' separately trained TopK and BatchTopK models across l0, but in several regions of Figures 2 and 3 the differences are small, so without variance estimates the comparison is not statistically supported. Please add repeated-seed results (at least 3–5 seeds) for the main Pareto-frontier comparison or report confidence intervals.
  3. [§3.2 and §4] The experimental comparison includes TopK, BatchTopK, and JumpReLU, but not Matryoshka SAEs, which are the closest prior approach for learning multi-level/nested features with a single model. Matryoshka SAEs (Bussmann et al., 2025) and hierarchical dictionaries (Ayonrinde et al., 2024) are cited in Related Work but never used as baselines. Since the paper's contribution is specifically a single SAE spanning multiple sparsity budgets, omitting the most directly competing method leaves the relative advantage unquantified and makes the 'Pareto-optimal' claim incomplete.
minor comments (4)
  1. [§2, first paragraph] The roles of D and h are easy to confuse: D is called the dictionary size and h the hidden dimension, but in the equations W_enc maps from R^h to R^D and Table 2 uses h for the model/residual-stream dimension. Please introduce a distinct symbol such as d_model to clarify the input dimension.
  2. [Appendix C] The method is called FlexSAE in Listing 1 and the kernel table uses 'Fused Hierarchical', while the main text consistently uses HierarchicalTopK. Please unify the terminology.
  3. [Figure 7 caption] The caption says 'e1 denotes the top-1 activation', but e_i are decoder embedding vectors, not activations. Please rephrase as 'the top-1 decoder direction' or similar.
  4. [Throughout] Capitalization is inconsistent (TopK vs. TOPK) and several figure labels render 'TopK' with an unintended space as 'T opK'; please normalize these formatting issues.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HierarchicalTopK is an empirically evaluated training objective; no prediction is derived from fitted inputs and no load-bearing self-citation chain is present.

full rationale

The paper proposes a training objective, Eq. (1), that averages per-prefix reconstruction losses over sparsity levels, and then evaluates the resulting single SAE against separately trained TopK, BatchTopK, and JumpReLU baselines using FVU and AutoInterp scores. These evaluation metrics are external to the training objective in the sense that the hierarchical loss is not fitted to the reported FVU values or interpretability scores; rather, the same MSE-style reconstruction loss is used for training all models, and the comparison is an empirical benchmark. Although FVU is a normalized form of the per-prefix MSE, this is a standard alignment between training objective and evaluation metric, not a fitted parameter being renamed as a prediction. The related-work citations, including Matryoshka SAE and TopK, are external prior art and are not used as an author-specific uniqueness theorem or as the sole justification for the central claim. The skeptic's concern that the prefix assumption (a single activation ranking being optimal for all budgets) is unverified and that the Pareto-front claim is over-strong is a validity and robustness critique, not a circularity: the claim is supported by experiments rather than being equivalent to the definition of the objective by construction. No step in the derivation reduces to its own inputs, so the paper is not circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The ledger contains no fitted free parameters and no invented entities. Training hyperparameters such as K=128, learning rate, and batch size are design choices, not parameters fitted to the evaluation target. The main axiomatic burdens are the validity of the activation ordering for all budgets, the representativeness of the single model/dataset setup, and the use of an automated interpretability proxy.

assumptions (4)
  • domain assumption Independent additivity: individual latents contribute independently to reconstruction, making isolated feature analysis valid (Ayonrinde et al., 2024).
    Invoked in Section 1 to justify why low-k reconstructions should be interpretable; if false, the interpretability preservation claim loses its foundation.
  • ad hoc to paper The top-j prefix of the sorted TopK activations is the correct j-sparse reconstruction ordering for every j <= K.
    Equation (1) optimizes cumulative reconstructions x_hat_j over the same sorted activation order; the method assumes this order simultaneously serves all budgets.
  • domain assumption Gemma-2 2B layer-12 residual-stream activations on a 1B-token FineWeb sample are representative for SAE sparsity generalization.
    All experiments use one model, one layer, and one dataset; the authors themselves list this as a limitation in Section 6.
  • domain assumption AutoInterp detection score is a sufficient proxy for human interpretability.
    Interpretability claims in Section 3.6 rest on automated scores; the authors acknowledge in Section 6 that user studies are needed to validate semantic alignment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Train One Sparse Autoencoder Across Multiple Sparsity Budgets to Preserve Interpretability and Accuracy." pith.science (2026). https://pith.science/paper/G2ZKLKWA

@misc{pith2026250524473,
  author       = {Pith},
  title        = {Pith review of: Train One Sparse Autoencoder Across Multiple Sparsity Budgets to Preserve Interpretability and Accuracy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G2ZKLKWA}},
  note         = {Machine review of arXiv:2505.24473}
}
read the original abstract

Sparse Autoencoders (SAEs) have proven to be powerful tools for interpreting neural networks by decomposing hidden representations into disentangled, interpretable features via sparsity constraints. However, conventional SAEs are constrained by the fixed sparsity level chosen during training; meeting different sparsity requirements therefore demands separate models and increases the computational footprint during both training and evaluation. We introduce a novel training objective, \emph{HierarchicalTopK}, which trains a single SAE to optimise reconstructions across multiple sparsity levels simultaneously. Experiments with Gemma-2 2B demonstrate that our approach achieves Pareto-optimal trade-offs between sparsity and explained variance, outperforming traditional SAEs trained at individual sparsity levels. Further analysis shows that HierarchicalTopK preserves high interpretability scores even at higher sparsity. The proposed objective thus closes an important gap between flexibility and interpretability in SAE design.

Figures

Figures reproduced from arXiv: 2505.24473 by the authors.

Figure 1
Figure 1. Left: SAE trained on a single k. Right: SAE trained on all k ≤ K. A sparse autoencoder (SAE) is defined as l = σ [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of an SAE with Hierarchical acti [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Pareto frontier for SAEs with BatchTopK, TopK, and Hierarchical activation functions. Red dots denote [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Number of features with activation frequency [PITH_FULL_IMAGE:figures/full_fig_p003_5.png]
Figure 4
Figure 4. Figure 4: We test simple heuristics to reduce the compu [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 6
Figure 6. Figure 6: AutoInterp Score (Paulo et al., 2024). TopK and BatchTopK scores are obtained from two separate SAEs trained with k = 32 and k = 128; the Hierarchical model uses a single SAE trained on all k ≤ 128. Hierarchical activation preserves the interpretability level of SAEs t…
Figure 7
Figure 7. Figure 7: Cosine similarity of feature embeddings in the reconstruction sum. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Latent-feature distributions for SAEs trained with [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: TopK versus JumpReLU inference. We did not find a significant difference between JumpReLU and fixed [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning

    cs.LG 2026-06 unverdicted novelty 6.5 of 10

    SFT supplies entangled compositional traces of atomic skills and routing modules; RL identifies those modules and enables recombination on novel compositions outside the SFT support.

Reference graph

Works this paper leans on

14 extracted references · 4 canonical work pages · cited by 1 Pith paper

  1. [1]

    Pearce, and Lee Sharkey

    Kola Ayonrinde, Michael T. Pearce, and Lee Sharkey. 2024. https://arxiv.org/abs/2410.11179 Interpretability as compression: Reconsidering sae explanations of neural activations with mdl-saes . Preprint, arXiv:2410.11179

    arXiv 2024

  2. [2]

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, and 6 others. 2023. Towards monosemanticity: Decomposing language models with d...

    2023

  3. [3]

    Bart Bussmann, Patrick Leask, and Neel Nanda. 2024. Batchtopk sparse autoencoders. arXiv preprint arXiv: 2412.06410

    arXiv 2024

  4. [4]

    Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda. 2025. https://arxiv.org/abs/2503.17547 Learning multi-level features with matryoshka sparse autoencoders . Preprint, arXiv:2503.17547

    arXiv 2025

  5. [5]

    Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2025. https://openreview.net/forum?id=tcsZt9ZNKD Scaling and evaluating sparse autoencoders . In The Thirteenth International Conference on Learning Representations

    work page 2025

  6. [6]

    Google DeepMind Gemma Team. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv: 2408.00118

    arXiv 2024

  7. [7]

    Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Callum McDougall, Kola Ayonrinde, Matthew Wearden, Arthur Conmy, Samuel Marks, and Neel Nanda. 2025. Saebench: A comprehensive benchmark for sparse autoencoders in language model interpretability. arXiv preprint arXiv: 2503.09532

    arXiv 2025

  8. [8]

    Alireza Makhzani and Brendan J. Frey. 2013. k-sparse autoencoders. International Conference on Learning Representations

    work page 2013

Show all 14 references
  1. [9]

    OpenAI. 2021. Introducing triton: Open-source gpu programming for neural networks. https://openai.com/index/triton/ . Accessed: 2021-07-24

    2021

  2. [10]

    Gonçalo Paulo, Alex Mallen, Caden Juang, and Nora Belrose. 2024. Automatically interpreting millions of features in large language models. arXiv preprint arXiv: 2410.13928

    2024 arXiv

  3. [11]

    Guilherme Penedo, Hynek Kydlicek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. https://openreview.net/forum?id=n6SCkn2QaG The fineweb datasets: Decanting the web for the finest text data at scale . In The Thirty-ei...

    2024

  4. [12]

    Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. 2024. https://arxiv.org/abs/2407.14435 Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders . Preprint, arXiv:2407.14435

    2024 arXiv

  5. [13]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  6. [14]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.