Pith. sign in

REVIEW 4 major objections 6 minor 13 references

Distribution-Aware Feature Selection for SAEs

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Distribution-aware pre-filtering makes sparse autoencoders better at concept isolation and less prone to feature absorption, without sacrificing much reconstruction quality.

desk verdict The Sampled-SAE candidate-pool idea is useful, but the L2-norm mechanism is arithmetically wrong and L2-norm equals squared-L2 in ranking, so the central story needs correction. read the letter →

arxiv 2508.21324 v1 pith:7BD7GSBC submitted 2025-08-29 cs.LG

Narmeen Oozeer , Nirmalendu Prakash , Michael Lan , Alice Rigg , Amirali Abdullah This is my paper
classification cs.LG
keywords sparseautoencodersinterpretabilityBatchTopKfeatureselectionactivationlotteryPythia-160Mabsorptionprobing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the way BatchTopK sparse autoencoders pick which features stay active is flawed: letting every feature compete on raw activation magnitude creates an "activation lottery" in which rare high-magnitude spikes crowd out features that fire consistently across many tokens. To fix this, the authors introduce Sampled-SAE, which scores each feature over the batch, keeps only the top K*l features as a candidate pool, and then applies BatchTopK inside that pool. On the residual stream of layer 6 of Pythia-160M, L2-norm and squared-L2 scoring with small l improve k-sparse probing accuracy and reduce feature absorption compared with BatchTopK, while reaching 2-3 times higher density of frequently activating features at comparable reconstruction error (FVU). The paper does not claim one setting is best everywhere: BatchTopK still wins on raw reconstruction and the autointerp-versus-FVU frontier, and the choice of l is a trade-off. The contribution is a tunable family that reframes batch-level feature selection as distribution-aware.

What carries the argument

The candidate pool: a two-stage gating mechanism on the batch activation matrix Z (m features x B tokens). First score each column (feature) with a batch-level function s; keep the top K*l columns as a candidate set; then apply BatchTopK to the masked matrix to select K activations. The expansion factor l is the dial: small l forces global consistency, large l drifts toward token-specific selection, and l = n/k recovers BatchTopK. The scoring functions tested—L2-norm, squared-L2, entropy, and uniform—are what make the selection "distribution-aware."

What would settle it

Measure the activation frequency of features that actually enter the candidate pool under L2-norm and squared-L2 scoring on a real Pythia-160M batch at low l. If the selected features are not predominantly mid-frequency, the proposed mechanism is not what drives the downstream gains. The same test could be run on the synthetic data once training instabilities are fixed: compute bucket recovery rates for HF+LA vs LF+HA features.

Watch

Extended reading notes

Core claim

The central claim is that restricting the candidate pool before batch-level TopK selection changes which features an SAE learns, and that this can be done to promote consistent mid-frequency features over rare spikes. Sampled-SAE computes a per-feature score across the batch (L2 norm, squared-L2, entropy, or uniform), keeps the top K*l features, and only then selects K activations across the batch. At l=1, every token draws from the same K globally influential features; at l=n/k, the method reduces to BatchTopK. Empirically, L2-norm and squared-L2 variants at low l (3-5) achieve higher sparse-probing accuracy and lower absorption fraction than BatchTopK at modestly higher FVU, and increase t

Load-bearing premise

The load-bearing premise is that L2-norm and squared-L2 scoring favor consistently active mid-frequency features over rare high-magnitude spikes; the paper states this, but the column-norm arithmetic does not obviously guarantee it, since a feature at magnitude 100 on 5% of samples can outscore one at magnitude 10 on 50%.

Editorial extensions

If this is right

  • On Pythia-160M layer 6, L2-norm and squared-L2 Sampled-SAEs with l=3-5 beat BatchTopK on k-sparse probing accuracy and absorption fraction while keeping FVU within a few points.
  • The fraction of features active on more than 10% of tokens can be increased 2-3x at similar FVU, addressing the underutilization of dense features.
  • BatchTopK is a special case (l=n/k), so the Sampled-SAE family gives a single knob to trade global consistency against fine-grained reconstruction; no single l optimizes all interpretability metrics.
  • Different scoring functions systematically learn different types of features — L2-norm and squared-L2 favor compositional structures like variables and HTML tags, entropy captures discriminative patterns, uniform defaults to syntactic elements.
  • BatchTopK shows higher cross-seed feature consistency (MMCS 0.277) than Sampled-SAE variants (≈0.176), meaning distribution-aware selection trades away some seed stability for task-specific gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism holds, interpretability pipelines could use l as a regularizer against polysemantic spikes, and the probing gains suggest concept isolation improves even when autointerp scores stay flat.
  • The scoring-arithmetic caveat applies: L2 norms as defined favor high-magnitude rare features over moderate-frequency ones, so the claimed mid-frequency bias may actually come from training dynamics interacting with the mask rather than from the score itself; a direct frequency analysis of the pool would settle it.
  • Because Sampled-SAE features match BatchTopK features only weakly (MMCS ≈0.72 cross-scoring, and unique features with similarity <0.15), conclusions about what a model 'represents' may depend on which scoring is chosen; cross-seed instability amplifies this caution.
  • A testable extension is to replace L2 norms with theoretically grounded scores such as leverage or ridge leverage scores, or to make l adaptive per batch, which could push the density-interpretability frontier further; the paper leaves this open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Sampled-SAE, a two-stage sparse autoencoder that first scores the columns of a batch activation matrix (via L2 norm, squared-L2, entropy, or uniform scoring), forms a candidate pool of size Kℓ, and then applies BatchTopK within that restricted pool. The authors argue that this promotes consistent mid-frequency features over rare high-magnitude spikes, and they report trade-offs on Pythia-160M layer 6: lower ℓ values improve probing accuracy and reduce feature absorption compared to BatchTopK, at some cost in FVU, while no single ℓ dominates all metrics. The synthetic experiment is described but results are explicitly deferred.

Significance. If the empirical findings hold, the paper makes a useful contribution by reframing BatchTopK as a tunable family and by demonstrating that candidate-pool selection can produce interpretability-relevant trade-offs. The use of a uniform-score baseline, multiple SAEBench metrics, and explicit limitations sections are strengths. However, the central mechanistic claim—that L2-norm and squared-L2 scoring favor consistent mid-frequency features—is contradicted by the arithmetic in §3.1, and the incomplete synthetic experiment removes the planned ground-truth validation. The paper therefore needs substantial revision before its conclusions can be accepted.

major comments (4)
  1. [§3.1, Eq. (2) and feature scoring definitions] The stated mechanism is contradicted by the definition. The text claims L2-norm 'emphasizes features with consistent batch-level activation—a feature firing at magnitude 10 across 50% of samples scores higher than one firing at magnitude 100 on 5% of samples.' But for a batch factor B, the first has norm sqrt(B·0.5·100) ≈ 7.07√B, while the second has sqrt(B·0.05·10000) ≈ 22.4√B. Moreover, L2 norm is a strictly increasing function of squared-L2, so with constant λ the two scorings induce identical top-Kℓ candidate pools. Yet Figures 3–6 report different numeric results for L2-norm and squared-L2, and Table 3 lists 'Leverage' separately. This internal inconsistency means the implementation must deviate from the written definition, or the distinct panels are not meaningful. The mechanistic explanation for downstream gains is unsupported.
  2. [§4 and A.2] The synthetic experiment is explicitly incomplete: 'We are actively troubleshooting Sampled-SAE training instabilities. Once satisfactory reconstructions are obtained, we will evaluate...' No bucket recovery rate or activation fidelity results are reported. Since this experiment was designed to test whether scoring strategies recover features across the frequency-magnitude spectrum, its absence removes the only controlled ground-truth evidence for the core claim. The paper should either complete this experiment or explicitly state that the mechanism remains a hypothesis.
  3. [Figures 3–6 and Table 3] Most real-data results appear to be single runs with no error bars or significance tests. Probing accuracy, absorption fraction, and density are plotted as point clouds with no reported variance across seeds or training runs. Table 1 reports n=2 for MMCS and says standard deviation is 'close to zero,' which is not a substitute for error bars on the main trade-off figures. Without this, it is impossible to assess whether the claimed improvements over BatchTopK are within noise.
  4. [§5, Feature Density (>10%)] The feature-density metric is partly circular with the selection mechanism. The L2-norm and squared-L2 scorings preferentially select features with high total activation energy, which correlates with activation frequency; measuring the fraction of features active on >10% of tokens will favor these selections by construction. The Uniform baseline also improves density under small ℓ (Figure 4d), indicating that the candidate-pool restriction itself—not distribution-aware scoring—may drive part of the gain. The paper should control for this by matching FVU/sparsity and by reporting density for features that would have been excluded by the pool, or by using a less selection-aligned metric.
minor comments (6)
  1. [§3, Architecture] The orientation of Z is stated incorrectly: 'The m columns of Z represent features and the B rows represent input samples' should be 'the rows represent features and the columns represent samples,' since Z = WencX + b with Z ∈ R^{m×B}.
  2. [Throughout] Notation is inconsistent: ℓ appears as both 'l' and 'ℓ', and K/k are used interchangeably (e.g., Eq. (3) uses Kℓ while §3 Candidate Selection uses ℓ·k). This makes the candidate-pool size and the recovery of BatchTopK at ℓ = n/k confusing.
  3. [Figure 2 caption] Typo: 'outperfrormed' should be 'outperformed.' Also the caption says 'It is outperfrormed by the features with lower l values,' but the comparison is between architectures, not features.
  4. [Figure 4 caption] The caption says 'Points correspond to different sparsity levels (ℓ)' but ℓ is the candidate pool expansion factor, not a sparsity level. Rephrase to avoid confusion.
  5. [Table 3] The table lists 'SampledSAE (Leverage, ℓ=5)' while the text and other tables use 'Squared-ℓ' or 'L2-norm.' This inconsistency, plus the identical numerical FVU and accuracy for L2-norm and Leverage, reinforces the concern in Major Comment 1.
  6. [§6, Limitations] The limitation paragraph says 'added hyperparameter s' but should be 'ℓ.' Also, the reference to Paulo and Belrose appears twice as 2024 and 2025 with the same arXiv number; consolidate.

Circularity Check

2 steps flagged · score 3.0 of 10

Empirical probing/absorption results are not circular, but the L2-norm/squared-L2 distinction collapses by monotonicity and the density gain partly measures the selection objective itself.

  1. renaming known result [Section 3.1, equations (2)-(3) and the bullet definitions of ℓ₂-norm and Squared-L2]
    "ℓ₂-norm: s_j = ‖Z_{:,j}‖₂ computes the column ℓ₂ norm ... This emphasizes features with consistent batch-level activation. Squared-L2: s_j = Σ_b Z_{bj}² + λ rewards total activation energy across the batch. Unlike the L2-norm, Squared-L2 uses squared magnitudes, making it sensitive to both frequency and high intensity."

    Top-K selection depends only on the ranking of the scores, and the ℓ₂ norm is a strictly increasing function of the squared ℓ₂ norm. Therefore the candidate pool c = TopK(q, Kℓ) in Eq. (3) is identical for the 'L2-norm' and 'Squared-L2' scorings. The paper nevertheless presents them as two distinct strategies with different mechanistic biases and gives them separate panels and results. The claimed difference between the two scorings reduces, by the paper's own equations, to the same column ranking by construction.

  2. self definitional [Section 3.1 scoring rationale and Section 5 'Feature Density (>10%)']
    "This emphasizes features with consistent batch-level activation—a feature firing at magnitude 10 across 50% of samples scores higher than one firing at magnitude 100 on 5% of samples. ... SampledSAE variants, particularly L2-norm and Squared-ℓ scoring strategies, achieve substantially higher feature densities compared to BatchTopK (Figure 4). This suggests that distribution-aware candidate selection promotes consistent mid-frequency features over rare high-magnitude spikes."

    The feature-density metric counts the fraction of features active on more than 10% of tokens, i.e. it measures 'consistent batch-level activation'. The candidate pool is formed by ranking columns by the very score the paper asserts to reward exactly that property, so the higher density of frequently-active features is partly a restatement of the selection objective rather than an independent empirical discovery. In addition, the quoted arithmetic is inverted by the paper's own formula: a magnitude-100 feature on 5% of samples has ℓ₂ norm ≈ 22.4 vs ≈ 7.1 for magnitude 10 on 50%, so the mechanistic claim is not even consistent with the stated scoring rule.

full rationale

The core probing, absorption, and autointerp comparisons are empirical outcomes of training full SAEs and are not obtained by fitting parameters to those metrics, so they are not circular in the usual sense. There are, however, two load-bearing weaknesses. First, L2-norm and squared-L2 define identical rankings, so reporting them as distinct scoring functions with different biases is internally inconsistent; any apparent difference between those panels cannot be caused by the scoring rule as written. Second, the paper's stated mechanism for all real-data gains is that L2-norm/squared-L2 favor consistent mid-frequency features over rare spikes, but the given formula ranks the rare spike higher; the density result then partially measures the property the selection rule was designed to enforce, and the causal explanation is unsupported. The synthetic experiment that was designed to validate the frequency-magnitude mechanism is explicitly reported as incomplete ('We are actively troubleshooting Sampled-SAE training instabilities'), so the central mechanism remains unverified. No load-bearing self-citation chain or fitted-input-called-prediction pattern is present. Overall score 3: partial circularity/self-definitional issues around the density metric and the L2/squared-L2 identity, while the main empirical claims retain independent content.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities beyond the conceptual 'activation lottery'. Its main assumptions are domain-level: the scoring functions are meaningful, high-frequency features are valuable, and autointerp is a valid proxy. The free parameters are the candidate pool multiplier and the ridge term, both hand-set rather than fitted.

free parameters (2)
  • Candidate pool multiplier (l) = swept over {1,3,4,5,10,20,30,40,50,60,70,100}
    Controls the size of the candidate feature pool. Explored as a grid, not fitted to any target metric, but it is the key tunable hyperparameter introduced by the paper.
  • Ridge parameter (lambda) for Squared-L2 = 0.01
    Added for numerical stability in the Squared-L2 scoring function. Hand-chosen, no sensitivity analysis.
assumptions (3)
  • domain assumption Batch-level column statistics (L2 norm, squared-L2, entropy) are meaningful proxies for feature utility in sparse coding.
    The entire method rests on the premise that these scores rank features by usefulness for reconstruction and interpretability. No theoretical justification is given; the Limitations section explicitly calls this 'unproven choice of scoring function'.
  • domain assumption High-frequency features (active in over 10% of tokens) are more interpretable or valuable than rare high-magnitude features.
    This motivates the 'activation lottery' problem and the focus on feature density. Cited from Sun et al. but not independently validated here; the paper's Limitations state this is unproven.
  • domain assumption Automated interpretability (autointerp) is a valid measure of SAE feature interpretability.
    Used as a primary evaluation metric throughout. The Limitations section acknowledges its known limitations, and the paper relies on it for Pareto comparisons.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Distribution-Aware Feature Selection for SAEs." pith.science (2026). https://pith.science/paper/7BD7GSBC

@misc{pith2026250821324,
  author       = {Pith},
  title        = {Pith review of: Distribution-Aware Feature Selection for SAEs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7BD7GSBC}},
  note         = {Machine review of arXiv:2508.21324}
}
abstract

Sparse autoencoders (SAEs) decompose neural activations into interpretable features. A widely adopted variant, the TopK SAE, reconstructs each token from its K most active latents. However, this approach is inefficient, as some tokens carry more information than others. BatchTopK addresses this limitation by selecting top activations across a batch of tokens. This improves average reconstruction but risks an "activation lottery," where rare high-magnitude features crowd out more informative but lower-magnitude ones. To address this issue, we introduce Sampled-SAE: we score the columns (representing features) of the batch activation matrix (via $L_2$ norm or entropy), forming a candidate pool of size $Kl$, and then apply Top-$K$ to select tokens across the batch from the restricted pool of features. Varying $l$ traces a spectrum between batch-level and token-specific selection. At $l=1$, tokens draw only from $K$ globally influential features, while larger $l$ expands the pool toward standard BatchTopK and more token-specific features across the batch. Small $l$ thus enforces global consistency; large $l$ favors fine-grained reconstruction. On Pythia-160M, no single value optimizes $l$ across all metrics: the best choice depends on the trade-off between shared structure, reconstruction fidelity, and downstream performance. Sampled-SAE thus reframes BatchTopK as a tunable, distribution-aware family.

Figures

Figures reproduced from arXiv: 2508.21324 by the authors.

Figure 1
Figure 1. Comparison of BatchTopK to Sampled-SAE feature selection strategies: (a) BatchTopK [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Pareto frontier analyses of SAE architectures: (a) Probing accuracy versus Absorption; [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Probing accuracy (top-5 SAE features) vs. fraction of variance unexplained (FVU) for [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Fraction of SAE features active on more than 10% of tokens vs. fraction of variance [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Automated Interpretability score vs. fraction of variance unexplained (FVU) for Sampled [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Absorption fraction vs. fraction of variance unexplained (FVU) for SampledSAE under [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: We train for 10,000 synthetic samples with training parameters same as the real data [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 4 canonical work pages

  1. [9]

    Saebench: A comprehensive benchmark for sparse autoencoders in language model interpretability

    Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Callum McDougall, Kola Ayonrinde, et al. Saebench: A comprehensive benchmark for sparse autoencoders in language model interpretability. arXiv preprint arXiv:2503.09532,

    arXiv

  2. [11]

    Sparse autoencoders trained on the same data learn different features

    Gonçalo Paulo and Nora Belrose. Sparse autoencoders trained on the same data learn different features. arXiv preprint arXiv:2501.16615,

    arXiv

  3. [12]

    activation lottery

    defaults to high-frequency 12 Table 3: Comparison across datasets. Winners are selected by highest probe accuracy; ties are bolded. ℓ is shown inline with the architecture. FVU is lower-is-better. Dataset Architecture (with ℓ) FVU ↓ SAE acc ↑ 1 SampledSAE (Entropy, ℓ=3) 0.308 0 .6822 SampledSAE (L2-norm, ℓ=5) 0.047 0.7434 SampledSAE (Leverage, ℓ=5) 0.047 ...

    work page 2021

  4. [13]

    High frequency latents are features, not bugs

    Xiaoqing Sun, Joshua Engels, and Max Tegmark. High frequency latents are features, not bugs. In ICLR 2025 Workshop on Sparsity in LLMs,

    work page 2025

  5. [2004]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027,

    arXiv

  6. [2009]

    Interpretability in parameter space: Minimizing mechanistic description length with attribution-based parameter decomposition

    Dan Braun, Lucius Bushnaq, Stefan Heimersheim, Jake Mendel, and Lee Sharkey. Interpretability in parameter space: Minimizing mechanistic description length with attribution-based parameter decomposition. arXiv preprint arXiv:2501.14926,

    arXiv

  7. [2015]

    Online row sampling

    Michael B Cohen, Sam Elder, Cameron Musco, Christopher Musco, and Madalina Persu. Online row sampling. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques (APPROX/RANDOM 2016),

    work page 2016

  8. [2016]

    Finding neurons in a haystack: Case studies with sparse probing

    Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610,

    arXiv

Show all 13 references
  1. [2017]

    Sparse autoen- coders find highly interpretable features in language models

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600,

    arXiv

  2. [2020]

    Improving dictionary learning with gated sparse autoencoders

    10 Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Improving dictionary learning with gated sparse autoencoders. arXiv preprint arXiv:2404.16014,

    arXiv

  3. [2023]

    Bart Bussmann, Joseph Jermyn, and Nix Robertson

    URL https://transformer-circuits.pub/2023/monosemantic-features/index.html . Bart Bussmann, Joseph Jermyn, and Nix Robertson. Batchtopk: A simple improvement for topk sparse autoencoders. GitHub repository,

    2023

  4. [2024]

    David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, Satvik Golechha, and Joseph Bloom

    URL https://github.com/bartbussmann/ BatchTopK. David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, Satvik Golechha, and Joseph Bloom. A is for absorption: Studying feature splitting and absorption in sparse autoencoders. arXiv preprint arXiv:2409.14507,

  5. [2025]

    Automatically interpreting millions of features in large language models

    Gonçalo Paulo, Alex Mallen, Caden Juang, and Nora Belrose. Automatically interpreting millions of features in large language models. arXiv preprint arXiv:2410.13928,

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.