Pith. sign in

REVIEW 4 major objections 5 minor 31 references

Beyond Black Boxes: Enhancing Interpretability of Transformers Trained on Neural Data

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A sparse autoencoder inserted into a brain-decoding transformer keeps performance while turning internal units into selective, causal encoders of visual features.

desk verdict First real application of TopK SAEs to transformer latents from neural population data, with a clever biological validation and honest limitations, but the causal ablation claim is undercut by circular selection and predicted-label receptive fields. read the letter →

arxiv 2506.14014 v1 pith:QEVTCL2S submitted 2025-06-16 q-bio.QM

Laurence Freeman , Philip Shamash , Vinam Arora , Caswell Barry , Tiago Branco , Eva Dyer This is my paper
classification q-bio.QM
keywords sparseautoencodersmechanisticinterpretabilityneuraldecodingtransformercalciumimagingvisualcortexTopKsparsityfeatureablation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that transformer models trained on neural population data can be made interpretable without losing their predictive power. It inserts a TopK sparse autoencoder between the final cross-attention latents and the output layer of POYO+, a transformer trained to decode visual stimuli from calcium imaging in mouse visual cortex. The authors report that decoding accuracy is largely preserved, that individual SAE units become sharply tuned to orientation, temporal frequency, and genetic cell type, and that ablating those units produces feature-specific drops in accuracy. The value, if true, is a non-invasive route from black-box neural decoders to mechanistic hypotheses about what information is encoded and where.

What carries the argument

The load-bearing object is the TopK sparse autoencoder, an overcomplete dictionary of 192 units (expansion factor 3) that retains only the $k$ largest activations per input, with no sparsity penalty in the loss. After training to reconstruct the 64-dimensional output of POYO+'s final cross-attention layer, it is inserted between that layer and the linear projection head, so that every prediction passes through a sparse code. Two auxiliary instruments carry the interpretation: activation maps, which weight each trial's softmax class probabilities by the absolute latent activation to give each unit a $5\times8$ receptive field over temporal frequency and orientation, and binary masks that zero out selected latents during inference to test causal necessity. The TopK mechanism is what produces selectivity; the masks are what turn selectivity into causal claims.

What would settle it

Recompute the activation maps with the true presented orientation and temporal frequency, encoded as one-hot labels, instead of the model's softmax probabilities; if the median selectivity gap (0.4 versus 0.9 occupied bins) and the 0-degree ablation specificity disappear, the claim that SAE latents genuinely encode stimulus features is not supported.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a TopK sparse autoencoder placed after the transformer's final cross-attention output untangles entangled latent dimensions into units that are biologically readable and causally actionable. Concretely, the authors show three things: the SAE-injected model retains more than 40 percent accuracy on orientation and temporal-frequency decoding versus roughly 45 percent for baseline, with chance at 12.5 percent and 20 percent respectively; running-speed prediction falls from $R^2\approx0.57$ to $\approx0.52$; SAE latents occupy a median of 0.4 of the 40 stimulus bins compared to 0.9 for baseline latents, and cluster into sharply tuned receptive fields; and ablating latents associated with the 0-degree orientation selectively reduces 0-degree decoding, while ablating SST-associated latents drops SST Cre-line decoding from 0.73 to 0.56 without affecting VIP decoding. The paper reads the first two findings as evidence that sparsity produces interpretable units rather than reconstruction artifacts, and the ablation results as evidence that the units are causally responsible for the features they seem to encode.

Load-bearing premise

The interpretability conclusion rests on treating the model's own predicted stimulus probabilities as the labels that define each latent's receptive field, even though the model is correct only about 45 percent of the time.

Editorial extensions

If this is right

  • A brain-decoding transformer can be made interpretable without much accuracy loss: orientation and temporal-frequency decoding drop only from roughly 45 percent to above 40 percent, while running-speed $R^2$ falls from about 0.57 to about 0.52.
  • SAE latents are markedly more selective than baseline latents, occupying a median of 0.4 stimulus bins versus 0.9, so each unit corresponds to a narrower, more interpretable feature.
  • Ablating latents tuned to the 0-degree orientation selectively impairs 0-degree decoding while leaving other orientations largely intact, showing that encoding of that stimulus is sparse and causal.
  • Ablating SST-associated latents drops SST Cre-line decoding from 0.73 to 0.56 without affecting VIP decoding, showing that cell-type information is separable enough to be surgically removed.
  • Targeting latents associated with emergent features such as genetic background offers a route to remove sensitive or unwanted information from neural decoders without retraining the model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the receptive fields are built from the model's own softmax labels rather than the true presented stimulus, the reported tuning could partly reflect model confusions; recomputing with ground-truth labels would clarify whether the selectivity is genuine.
  • The paper's own ablation result leaves SST decoding at 0.56 rather than chance, consistent with its admission of residual entanglement; a stricter test would ablate every unit with any significant SST association and ask whether decoding collapses to chance.
  • The same sparse-decomposition approach could transfer to other neural decoders and to closed-loop brain-computer interfaces, but that extension is untested and would need to show that ablations remain safe and selective under distribution shift.
  • The analysis is confined to drifting gratings in primary visual cortex, a deliberately low-dimensional and well-understood stimulus space; generalizing the interpretability claim to natural movies or multi-region recordings is an open empirical question.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper integrates a TopK sparse autoencoder (SAE) into the POYO+ transformer, which was trained on calcium imaging from the Allen Brain Observatory Visual Coding dataset. The authors claim that (1) inserting the SAE preserves downstream decoding performance, (2) individual SAE latents develop sharply tuned 'receptive fields' for stimulus orientation and temporal frequency and also align with genetic Cre-line identity, and (3) targeted ablations of SAE latents produce causal, feature-specific effects on prediction accuracy. The methods center on training the SAE on POYO+ latent activations, constructing activation maps from latent activations weighted by the model's own softmax predictions, clustering latents, decoding Cre-line labels with logistic regression, and zeroing selected latents to measure changes in decoding accuracy. The results show a roughly 5% drop in classification accuracy and a 0.05 drop in running-speed R², an SAE reconstruction R² of 0.885 at TopK=20, sparser activation maps for SAE latents than baseline latents, a VIP/SST contrast in logistic-regression coefficients, and selective ablation effects for 0° orientation and SST decoding.

Significance. If the central claims were established, this would be a valuable contribution to mechanistic interpretability for transformer models in neuroscience: it would show that a TopK-SAE can be inserted into a neural decoder with only modest performance loss, that sparse latents can be mapped to biologically interpretable variables, and that targeted ablations can selectively remove feature information. The paper uses a large, real dataset, draws on a state-of-the-art model (POYO+), and reports reconstruction/sparsity trade-offs across multiple SAE checkpoints. The main strengths are the clear architecture choice (per-sample TopK), the use of multiple independently trained SAEs in several analyses, and the attempt to link latent structure to known biological properties of VIP and SST interneurons. However, the interpretability and causal claims currently rest on two methodological problems: activation maps are built from the model's own predicted labels rather than ground-truth stimuli, and ablation targets are selected on the same data used to evaluate the ablation effects.

major comments (4)
  1. [Section 2.4, Eq. (4); Section 2.6; Figure 6] The activation maps used to define 'receptive fields' and to select ablation targets are computed from the model's own softmax predictions (P_ori, P_temp) rather than from ground-truth stimulus labels. At the reported ~45% decoding accuracy, these predicted labels can be systematically biased, so a latent's activation map may reflect the model's confusion structure rather than true tuning to the presented stimulus. In addition, the same drifting-grating trials appear to be used both to select the top 0°-tuned latents and to measure the post-ablation drop in 0° accuracy. This selection-on-evaluation bias can make a feature-specific drop appear causal even if the selected latents carry no true stimulus information. I recommend constructing activation maps from ground-truth labels (or from a calibration split) and selecting ablation targets on a separate data split from the one used to evaluate ablation effects; reporting results under nested cross-validation would address the concern.
  2. [Section 3.1 and Figure 2A-B] The claim that the SAE 'preserves' decoding performance is not supported by any error bars, confidence intervals, or significance tests. The reported ~5% drop in accuracy and ~0.05 drop in running-speed R² could be within session-to-session or checkpoint-to-checkpoint variability. Please report per-session or per-checkpoint variability and provide a formal statistical comparison (e.g., bootstrap confidence intervals or a mixed-effects test) between baseline and SAE-augmented models.
  3. [Section 3.5 and Appendix 5.4.4] The SST ablation analysis selects latents using bootstrap p-values computed on the full dataset and then measures SST decoding accuracy on that same dataset after zeroing those latents. This is a further instance of selection-on-evaluation: selecting units that most strongly predict SST on a dataset will tend to reduce SST decoding when removed on that dataset, even if those units are not specifically required for SST information. The same concern applies to the Cre-line ablation claims in Section 3.3. Please use a held-out selection set or nested cross-validation so that the evaluated decoding performance is independent of the target-selection procedure.
  4. [Section 2.4, Eq. (4)] The definition of the activation map is under-specified: the subscript i is used for both the latent index and the temporal-frequency bin in P_temp_n,i; the denominator appears to normalize the activation weights but does not normalize the joint distribution over orientation and frequency bins; and the 'high-confidence' trial filtering is described only in words. Since this map underpins the selectivity, clustering, and ablation conclusions, the exact normalization and the step from P_joint to A_i should be written out unambiguously, ideally with a small worked example.
minor comments (5)
  1. [Appendix 5.2 vs. Section 3.1] The hyperparameter sweep in Appendix 5.2 restricts TopK values to {15, 20}, while Section 3.1 states that TopK was swept from 5 to 30 retained activations; please reconcile these descriptions.
  2. [Section 3.2] The text says '1–8Hz' when describing drifting-grating temporal frequencies, but the stimulus set includes 15 Hz; please correct the range.
  3. [Discussion] There are several typos, including 'intergrating' in the Discussion, 'plateou' in Appendix 5.3, and 'focussed' in Section 2.3; a proofreading pass is needed.
  4. [Section 2.4] The term 'receptive field' is used for activation maps built from model predictions; this term has a specific neurophysiological meaning, and its use here may overstate what the maps represent. Consider using 'activation map' or 'stimulus selectivity map' consistently.
  5. [References] The TopK-SAE method is attributed to Makhzani and Frey [13], but the per-sample TopK variant used here is more commonly associated with later dictionary-learning work; citing the specific TopK-SAEs from the language-model-interpretability literature would be more precise.

Circularity Check

3 steps flagged · score 6.0 of 10

Receptive-field maps and ablation targets are constructed from the model's own predictions on the same trials used for evaluation, so the stimulus-selectivity and causal-ablation claims are partly self-referential.

  1. self definitional [Section 2.4, Eq. (4); interpreted in Section 3.2]
    "We compute class probabilities as P ori n,j =σ(y ori(n))j , where σ denotes the softmax function. We then form a joint stimulus–activation distribution by weighting each trial’s class probabilities by the magnitude of the corresponding latent activation: P joint i,j = ... We interpret each A i as the 'receptive field' of that latent, allowing direct comparison across all D=192 SAE dimensions."

    The 'receptive field' is defined using the model's own softmax logits, not the ground-truth stimulus orientation or temporal frequency. Therefore the later statement that SAE latents are 'selectively tuned to specific stimuli' is, by construction, a statement about co-activation with the model's predicted classes. At roughly 45% decoding accuracy, the maps can equally reflect systematic model confusions; the link to the actual stimulus is an assumption imported into the definition, making the selectivity result self-referential rather than an independent measurement.

  2. fitted input called prediction [Section 2.6 (Ablation Experiments) and Section 3.5]
    "For trained features (orientation, temporal frequency), we selected the class combination with the highest activation in their respective bins from the activation maps Ai (see Section 2.4). ... By comparing performance before and after ablation on both trained and emergent tasks, we (1) verify that the targeted latents are indeed causally necessary for encoding the feature of interest."

    Ablation targets are selected from activation maps that are built from the model's own predicted class probabilities (Eq. 4), and post-ablation accuracy is measured on the same trials that generated those maps. Units called '0°-selective' are, by construction, units whose activity is high when the model emits high 0°-probability logits; zeroing them on the same trials necessarily lowers the 0° logits that were used for selection. The reported feature-specific drop is therefore partly an artifact of selecting on the evaluation signal rather than an independent causal demonstration.

1 more flagged steps
  1. fitted input called prediction [Section 2.6, Appendix 5.4.4, and Section 3.5 (Fig. 6C)]
    "For emergent features (Cre-line, region), we fit multinomial logistic regressions on each latent Z i and selected those with statistically significant positive coefficients (p <0.01) for each class (See Appendix 5.4.4). ... This intervention significantly reduced SST Cre-line decoding accuracy (from 0.73 to 0.56, p <0.05) without affecting VIP decoding."

    The SST-associated latents are selected by fitting per-latent logistic regressions to the SST labels on the full dataset, and the ablation's effect is then measured as SST decoding accuracy on that same dataset. Selecting the features with the most stable positive SST coefficients makes a subsequent drop in SST decodability statistically forced; the evaluation does not use held-out selection. Thus the 'causal, feature-specific' conclusion for emergent features is a fitted-input-called-prediction: the selection and the evaluation share the same labels and trials.

full rationale

The SAE-preserves-performance claim (Section 3.1) is not circular: it is a direct empirical comparison of baseline versus SAE-augmented decoding, and the reconstruction R2 is an external fidelity metric. Similarly, the Cre-line decodability comparison and the VIP/SST directional-selectivity analysis are independent of the circularity concerns. However, the two central interpretability claims are partly self-referential. The 'receptive fields' of Section 2.4 are constructed from the model's own softmax predictions rather than ground-truth stimuli, so the claim that latents are selectively tuned to orientation and temporal frequency is, by construction, a claim about co-activation with the model's predicted classes. The ablation analyses inherit this problem: orientation/temporal-frequency targets are chosen from these same model-derived maps, and SST targets are chosen by fitting logistic regressions to the same labels and trials used to evaluate ablation outcome. Selection on the evaluation signal makes the subsequent 'causal, feature-specific' drops partly forced. Because substantial independent content remains, the score is 6 rather than higher.

Assumptions & free parameters 5 free parameters · 7 assumptions · 0 invented entities

The central claim rests on the validity of POYO+ as a trained neural decoder, the Allen dataset as ground truth for visual stimuli and Cre-line labels, and a set of hand-chosen SAE hyperparameters and significance thresholds. No new physical or biological entities are postulated. The main ad hoc assumptions are that model-predicted softmax labels can stand in for true stimulus labels when constructing receptive fields, and that ablating units selected on the same data reveals causal necessity.

free parameters (5)
  • TopK value k = 20
    Selected via Bayesian sweep balancing reconstruction R2 (0.885) and sparsity; central to the sparsity/interpretability claim.
  • SAE expansion factor gamma = 3
    Fixed expansion factor for SAE latent dimensionality (64 to 192); no ablation over gamma is reported.
  • High-confidence threshold for activation maps = 0.5
    Used in Section 2.4 to include trials in activation-map construction; hand-chosen threshold.
  • Significance thresholds for latent-class association and ablation selection = alpha=0.05, p<0.01
    Appendix 5.4.4 sets alpha=0.05 for latent-class significance and p<0.01 for ablation target selection; arbitrary thresholds without multiple-comparison correction.
  • Minimum activation threshold for clustering inclusion = 0.1
    Appendix 5.5 excludes latents with sum of absolute values below 0.1; hand-chosen to focus on meaningful units.
assumptions (7)
  • domain assumption POYO+ final cross-attention latents are a valid substrate for mechanistic interpretation.
    Section 2.1 assumes the post-cross-attention latents carry task-relevant information suitable for SAE analysis.
  • domain assumption The Allen Brain Observatory calcium imaging data and POYO+ decoder provide a faithful model of visual cortical population activity.
    Sections 2.3 and 5.1 rely on prior work [15,16] for dataset and model validity.
  • domain assumption VISp neurons have orientation and temporal frequency tuning representable by linear decoding.
    Section 1 cites refs [17-19] to justify the choice of visual cortex; selectivity comparisons assume these tunings are the right reference frame.
  • domain assumption VIP interneurons show stronger directional selectivity than SST interneurons.
    Section 3.4 uses this from ref [21] as the biological benchmark for the negative-coefficient analysis.
  • ad hoc to paper Model-predicted class probabilities can be used as labels to construct latent receptive fields.
    Section 2.4 weights latent activations by softmax logits; this assumes model predictions are a valid proxy for true stimulus identity.
  • ad hoc to paper Ablating latents selected on the same data used to measure ablation effects identifies causal necessity.
    Section 2.6 selects targets from activation maps and logistic regressions fit on the same drifting-grating samples used for evaluation; this assumes no selection bias.
  • ad hoc to paper An SAE trained on all five stimulus paradigms transfers to the drifting-gratings-only analysis.
    Appendix 5.1.1 says SAEs were fit on the full dataset while analyses use drifting gratings; the transfer is assumed, not tested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Black Boxes: Enhancing Interpretability of Transformers Trained on Neural Data." pith.science (2026). https://pith.science/paper/QEVTCL2S

@misc{pith2026250614014,
  author       = {Pith},
  title        = {Pith review of: Beyond Black Boxes: Enhancing Interpretability of Transformers Trained on Neural Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QEVTCL2S}},
  note         = {Machine review of arXiv:2506.14014}
}
read the original abstract

Transformer models have become state-of-the-art in decoding stimuli and behavior from neural activity, significantly advancing neuroscience research. Yet greater transparency in their decision-making processes would substantially enhance their utility in scientific and clinical contexts. Sparse autoencoders offer a promising solution by producing hidden units that respond selectively to specific variables, enhancing interpretability. Here, we introduce SAEs into a neural decoding framework by augmenting a transformer trained to predict visual stimuli from calcium imaging in the mouse visual cortex. The enhancement of the transformer model with an SAE preserved its original performance while yielding hidden units that selectively responded to interpretable features, such as stimulus orientation and genetic background. Furthermore, ablating units associated with a given variable impaired the model's ability to process that variable, revealing how specific internal representations support downstream computations. Together, these results demonstrate that integrating SAEs with transformers combines the power of modern deep learning with the interpretability essential for scientific understanding and clinical translation.

Figures

Figures reproduced from arXiv: 2506.14014 by the authors.

Figure 1
Figure 1. Integrating a Sparse Autoencoder into the POYO+ architecture. Raw calcium fluo￾rescence traces recorded across multiple channels are encoded by POYO+, passed through a Sparse Autoencoder (SAE) in order to constrain the latent residual stream to a sparse representation, re￾constructed to the original latent dimensionality, and then linearly projected to predict downstream variables such as pupil location and running … view at source ↗
Figure 2
Figure 2. Decoding and Reconstruction Performance Comparison between Baseline and Sparse Autoencoder Models. (A) Classification accuracy for orientation and temporal frequency decoding tasks. (B) Prediction of running speed (regression task) also demonstrates slightly suppressed performance when incorporating the SAE model compared to baseline latents. Dotted lines represent chance levels. (C) Sparsity analysis of SAE latent … view at source ↗
Figure 3
Figure 3. Sparse Autoencoder Latents Exhibit Increased Task Selectivity Compared to Baseline Latents. (A) Average activation map of 14 SAE latents clustered via UMAP and HDBSCAN, showing selective activation across orientation (angular coordinates) and temporal frequency (radial coordinates). (B) Average activation map of 18 baseline latents using the same clustering method, exhibiting broader, less selective activation. (C) … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: SAE Latents Maintain Genetic Cre-line Decoding with Enhanced Interpretability. (A) Confusion matrix showing Cre-line classification accuracy from baseline latents, with strong performance for inhibitory neuron lines SST and VIP using logistic regression. (B) Decoding a…
Figure 5
Figure 5. Figure 5: VIP-Associated SAE Latents Exhibit Stronger Directional Selectivity Than SST Latents. Logistic regression coefficients linking SAE latents to orientation decoding were analyzed across eight independently trained SAE checkpoints. We compared VIP- and SST-associated la￾t…
Figure 6
Figure 6. Figure 6: Ablation of Specific SAE Latents Demonstrates Causal and Selective Encoding of Task and Emergent Properties. (A) Ablation of SAE latents selectively encoding the 0 ◦ orientation stimulus across twelve independently trained SAEs. Ablation predominantly impairs decoding …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 26 canonical work pages

  1. [1]

    and Polosukhin, I.Attention is all you need.Advances in Neural Information Processing Systems, 30, 2017

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł. and Polosukhin, I.Attention is all you need.Advances in Neural Information Processing Systems, 30, 2017

    work page 2017

  2. [2]

    A unified, scalable framework for neural population decoding.Advances in Neural Information Processing Systems, 36:44937–44956, 2023

    Azabou, M., Arora, V ., Ganesh, V ., Mao, X., Nachimuthu, S., Mendelson, M., Richards, B., Perich, M., Lajoie, G., and Dyer, E. A unified, scalable framework for neural population decoding.Advances in Neural Information Processing Systems, 36:44937–44956, 2023

    work page 2023

  3. [3]

    The mythos of model interpretability.Communications of the ACM, 61(10), pp.36–43

    Lipton, Z.C., 2018. The mythos of model interpretability.Communications of the ACM, 61(10), pp.36–43

    work page 2018

  4. [4]

    Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.Nature Machine Intelligence, 1(5), pp.206–215

    Rudin, C., 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.Nature Machine Intelligence, 1(5), pp.206–215

    work page 2019

  5. [5]

    and Müller, K.R., 2017

    Samek, W., Wiegand, T. and Müller, K.R., 2017. Explainable artificial intelligence: Understand- ing, visualizing and interpreting deep learning models.arXiv preprint arXiv:1708.08296

    arXiv 2017

  6. [6]

    and Elhadad, N., 2015

    Caruana, R., Lou, Y ., Gehrke, J., Koch, P., Sturm, M. and Elhadad, N., 2015. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission.Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp.1721–1730

    work page 2015

  7. [7]

    Mohsenzadeh Khaligh-Razavi and Nikolaus Kriegeskorte

    S. Mohsenzadeh Khaligh-Razavi and Nikolaus Kriegeskorte. Deep supervised, but not un- supervised, models may explain IT cortical representation.PLoS Computational Biology, 10(11):e1003915, 2014

    work page 2014

  8. [8]

    and Bengio, Y .Understanding intermediate layers using linear classifier probes.2017

    Alain, G. and Bengio, Y .Understanding intermediate layers using linear classifier probes.2017. https://openreview.net/forum

    work page 2017

Show all 31 references
  1. [9]

    and Sharkey, L

    Cunningham, H., Ewart, A., Riggs, L., Huben, R. and Sharkey, L. Sparse autoencoders find highly interpretable features in language models.arXiv preprint arXiv:2309.08600, 2023

    2023 arXiv

  2. [10]

    L., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y ., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Tamkin, A., Nguyen, K., McLean, B., Burke, J

    Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N. L., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y ., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Tamkin, A., Nguyen, K., McLean, B., Burke, J. E., Hume, T., Carter, S., Henigha...

    2023

  3. [11]

    and Deja, K

    Cywi´nski, B. and Deja, K. SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoders.arXiv preprint arXiv:2501.18052, 2025

    2025 arXiv

  4. [12]

    Olshausen and David J

    Bruno A. Olshausen and David J. Field. Sparse coding with an overcomplete basis set: A strategy employed by V1?Vision Research, 37(23):3311–3325, 1997

    1997

  5. [13]

    and Frey, B.K-sparse autoencoders.arXiv preprint arXiv:1312.5663, 2013

    Makhzani, A. and Frey, B.K-sparse autoencoders.arXiv preprint arXiv:1312.5663, 2013

    2013 arXiv

  6. [14]

    Adams, E., Bai, L., Lee, M., Yu, Y ., and AlQuraishi, M.From Mechanistic Interpretability to Mechanistic Biology: Training, Evaluating, and Interpreting Sparse Autoencoders on Protein Language Models.bioRxiv, pp.2025–02, 2025

    2025

  7. [15]

    X., Arora, V ., Knight, I

    Azabou, M., Pan, K. X., Arora, V ., Knight, I. J., Dyer, E. L., and Richards, B. A.Multi-session, multi-task neural decoding from distinct cell–types and brain regions. InProceedings of the Thirteenth International Conference on Learning Representations, 2025

    2025

  8. [16]

    E., Lecoq, J

    de Vries, S. E., Lecoq, J. A., Buice, M. A., Groblewski, P. A., Ocker, G. K., Oliver, M., Feng, D., Cain, N., Ledochowitsch, P., Millman, D., and Roll, K.A large-scale standardized physio- logical survey reveals functional organization of the mouse visual cortex.Nature Neurosc...

    2020

  9. [17]

    Hubel, D. H. and Wiesel, T. N.Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex.Journal of Physiology, 160(1):106–154, 1962. 10

    1962

  10. [18]

    L.Spatial structure and symmetry of simple-cell receptive fields in macaque primary visual cortex.Journal of Neurophysiology, 88(1):455–463, 2002

    Ringach, D. L.Spatial structure and symmetry of simple-cell receptive fields in macaque primary visual cortex.Journal of Neurophysiology, 88(1):455–463, 2002

    2002

  11. [19]

    J., and Movshon, J

    Carandini, M., Heeger, D. J., and Movshon, J. A.Linearity and normalization in simple cells of the macaque primary visual cortex.Journal of Neuroscience, 25(44):10027–10036, 2005

    2005

  12. [20]

    Jaegle, A., Borgeaud, S., Alayrac, J. B., Doersch, C., Ionescu, C., Ding, D., Koppula, S., Zoran, D., Brock, A., Shelhamer, E., and Hénaff, O.Perceiver io: A general architecture for structured inputs & outputs.arXiv preprint arXiv:2107.14795v3, 2021

    2021 arXiv

  13. [21]

    J., Ocker, G

    Millman, D. J., Ocker, G. K., Caldejon, S., Kato, I., Larkin, J. D., Lee, E. K., Luviano, J., Nayan, C., Nguyen, T. V ., North, K., and Seid, S. VIP interneurons in mouse primary visual cortex selectively enhance responses to weak but specific stimuli.eLife, 9:e55130, 2020. 5 ...

    2020

  14. [22]

    ,D(B) by sampling N datapoints with replacement fromD

    Generate B bootstrap samples D(1), . . . ,D(B) by sampling N datapoints with replacement fromD

  15. [23]

    For each bootstrap sample D(b), train a logistic regression classifier to obtain class-specific coefficientsθ (b) c

  16. [24]

    For each latent dimensionjand classc, collect the bootstrapped coefficient values: Ccj = n θ(b) cj oB b=1

  17. [25]

    Compute the empirical mean: ¯θcj = 1 B BX b=1 θ(b) cj . 13

  18. [26]

    Compute a95%confidence interval: CI95% cj = [Percentile2.5(Ccj),Percentile 97.5(Ccj)]

  19. [27]

    receptive fields

    Estimate a two-sided p-value by calculating the proportion of bootstrap samples whose sign disagrees with the mean: pcj = 1 B BX b=1 I h θ(b) cj · ¯θcj <0 i . 5.4.5 Latent Significance Criterion We define a latent unitjas significantly associated with classcif: pcj < α, where ...

  20. [28]

    Each Ai,j matrix was flattened into a 40-dimensional vector and standardized via z-scoring

    Preprocessing:Latents with negligible overall activation (sum of absolute values <0.1 ) were excluded to focus analysis on meaningful units. Each Ai,j matrix was flattened into a 40-dimensional vector and standardized via z-scoring

  21. [29]

    UMAP parameters were set ton neighbors = 15, min_dist= 0.1, using correlation distance as the metric

    Dimensionality Reduction:Uniform Manifold Approximation and Projection (UMAP) was applied to reduce the activation vectors to two dimensions. UMAP parameters were set ton neighbors = 15, min_dist= 0.1, using correlation distance as the metric

  22. [30]

    HDBSCAN was configured with a minimum cluster size of 5 and minimum samples of 1, using Euclidean distance

    Clustering:HDBSCAN was then run on the 2D UMAP embeddings to discover clusters of latents with similar receptive fields. HDBSCAN was configured with a minimum cluster size of 5 and minimum samples of 1, using Euclidean distance

  23. [31]

    We visualized both the UMAP embeddings colored by cluster identity and summarized the average activation patterns within each cluster

    Evaluation and Visualization:Cluster quality was assessed using silhouette scores when multiple clusters were found. We visualized both the UMAP embeddings colored by cluster identity and summarized the average activation patterns within each cluster. This analysis provided an...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.