{"id":"8e8086cd-ea1a-4241-966c-89764ed762e0","arxiv_id":"2506.14014","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Inserting a TopK sparse autoencoder into the POYO+ neural decoder preserves performance while yielding latent units selective for orientation, temporal frequency, and genetic background, and ablating them causally impairs the corresponding decoding.","lead":"A team trained a sparse autoencoder on the hidden states of a transformer that decodes visual stimuli from mouse brain calcium imaging, and showed the added module keeps most decoding accuracy while producing individual units that respond to specific features like stimulus orientation and mouse genetic line. These units can be removed to selectively disable the model's ability to use a given feature, a step toward interpretable and steerable brain-computer interfaces.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ablation and receptive-field analyses select targets using the model's own predictions on the same data used for evaluation; the causal, feature-specific claim may be an artifact of selection-on-evaluation-data.","rationale":"The reader's weakest assumption identified the use of model-predicted class probabilities instead of ground-truth stimuli in the receptive-field construction. This is a real problem, but the more load-bearing issue is that the same circularity extends into the ablation experiments and is compounded by selecting targets on the exact data used for evaluation. The ablation results in Fig. 6 are the primary evidence for the paper's causal claim (3), and the selection method in §2.6 does not support a causal interpretation. I agree with the reader that the paper's conditional verdict is appropriate: the concerns are serious but addressable with held-out validation, ground-truth labels, and released code. If the proposed test shows the ablation effects vanish, the central claim would need substantial revision, potentially warranting rejection; if the effects persist under held-out selection and permutation controls, the causal claim would be materially strengthened. The paper has independent support in the consistency across independently trained SAE checkpoints and the specificity of the SST/VIP decoding results, which is why I do not recommend outright rejection at this stage. However, the current manuscript does not establish causality for the ablation results, and the interpretability maps are built on a questionable proxy. The verdict should remain CONDITIONAL pending the described control analyses, and the reader's emphasis on the predicted-label proxy is partially correct but should be broadened to the selection-on-evaluation circularity.","tokens_in":11183,"tokens_out":3792,"duration_ms":45417,"concrete_test":"Split the drifting-grating trials into disjoint selection and evaluation sets by recording session. Compute activation maps (Eq. 4) using ground-truth orientation labels on the selection set only, select the top k latents for 0° orientation there, ablate them, and measure 0° accuracy on the held-out evaluation set. Repeat with selection based on the model's softmax predictions, and with a permutation control (same number of latents selected from shuffled 0° labels). If the 0°-specific accuracy drop disappears or shrinks substantially under held-out or permutation conditions, the causal claim in §3.5 is an artifact of same-data selection and predicted-label weighting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (3) — that targeted ablations of SAE units yield causal, feature-specific effects — rests on a selection procedure in §2.6 that is circular in two ways. First, for orientation and temporal frequency, ablation targets are chosen from activation maps A_i computed in §2.4 using the model's own softmax predictions (Eq. 4), not the ground-truth stimulus labels. At ≈45% decoding accuracy, these maps can reflect systematic model confusions rather than genuine tuning to the true stimulus. Second, the same trials are used to select the top 0°-tuned latents and to measure the post-ablation drop in 0° accuracy. Selecting units that are most predictive of the model's 0° outputs on a dataset will, by construction, tend to reduce those outputs when ablated on that same dataset, even if the units carry no true stimulus information. The same issue affects the SST ablation in §3.5: latents are selected via logistic-regression p-values computed on the full dataset, then decoding accuracy is measured on that same dataset after ablation. This selection-on-evaluation bias, combined with the predicted-label proxy, means the causal feature-specific effects reported in Fig. 6 are not established. The receptive-field claim (2) inherits the same predicted-label problem, as the reader noted. Without held-out selection and ground-truth labels, the interpretability conclusions may reflect the model's internal error structure rather than neural coding of the stimulus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper integrates a TopK sparse autoencoder (SAE) into the POYO+ transformer, which was trained on calcium imaging from the Allen Brain Observatory Visual Coding dataset. The authors claim that (1) inserting the SAE preserves downstream decoding performance, (2) individual SAE latents develop sharply tuned 'receptive fields' for stimulus orientation and temporal frequency and also align with genetic Cre-line identity, and (3) targeted ablations of SAE latents produce causal, feature-specific effects on prediction accuracy. The methods center on training the SAE on POYO+ latent activations, constructing activation maps from latent activations weighted by the model's own softmax predictions, clustering latents, decoding Cre-line labels with logistic regression, and zeroing selected latents to measure changes in decoding accuracy. The results show a roughly 5% drop in classification accuracy and a 0.05 drop in running-speed R², an SAE reconstruction R² of 0.885 at TopK=20, sparser activation maps for SAE latents than baseline latents, a VIP/SST contrast in logistic-regression coefficients, and selective ablation effects for 0° orientation and SST decoding.","tokens_in":11555,"tokens_out":4973,"duration_ms":50248,"significance":"If the central claims were established, this would be a valuable contribution to mechanistic interpretability for transformer models in neuroscience: it would show that a TopK-SAE can be inserted into a neural decoder with only modest performance loss, that sparse latents can be mapped to biologically interpretable variables, and that targeted ablations can selectively remove feature information. The paper uses a large, real dataset, draws on a state-of-the-art model (POYO+), and reports reconstruction/sparsity trade-offs across multiple SAE checkpoints. The main strengths are the clear architecture choice (per-sample TopK), the use of multiple independently trained SAEs in several analyses, and the attempt to link latent structure to known biological properties of VIP and SST interneurons. However, the interpretability and causal claims currently rest on two methodological problems: activation maps are built from the model's own predicted labels rather than ground-truth stimuli, and ablation targets are selected on the same data used to evaluate the ablation effects.","major_comments":[{"comment":"The activation maps used to define 'receptive fields' and to select ablation targets are computed from the model's own softmax predictions (P_ori, P_temp) rather than from ground-truth stimulus labels. At the reported ~45% decoding accuracy, these predicted labels can be systematically biased, so a latent's activation map may reflect the model's confusion structure rather than true tuning to the presented stimulus. In addition, the same drifting-grating trials appear to be used both to select the top 0°-tuned latents and to measure the post-ablation drop in 0° accuracy. This selection-on-evaluation bias can make a feature-specific drop appear causal even if the selected latents carry no true stimulus information. I recommend constructing activation maps from ground-truth labels (or from a calibration split) and selecting ablation targets on a separate data split from the one used to evaluate ablation effects; reporting results under nested cross-validation would address the concern.","section":"Section 2.4, Eq. (4); Section 2.6; Figure 6"},{"comment":"The claim that the SAE 'preserves' decoding performance is not supported by any error bars, confidence intervals, or significance tests. The reported ~5% drop in accuracy and ~0.05 drop in running-speed R² could be within session-to-session or checkpoint-to-checkpoint variability. Please report per-session or per-checkpoint variability and provide a formal statistical comparison (e.g., bootstrap confidence intervals or a mixed-effects test) between baseline and SAE-augmented models.","section":"Section 3.1 and Figure 2A-B"},{"comment":"The SST ablation analysis selects latents using bootstrap p-values computed on the full dataset and then measures SST decoding accuracy on that same dataset after zeroing those latents. This is a further instance of selection-on-evaluation: selecting units that most strongly predict SST on a dataset will tend to reduce SST decoding when removed on that dataset, even if those units are not specifically required for SST information. The same concern applies to the Cre-line ablation claims in Section 3.3. Please use a held-out selection set or nested cross-validation so that the evaluated decoding performance is independent of the target-selection procedure.","section":"Section 3.5 and Appendix 5.4.4"},{"comment":"The definition of the activation map is under-specified: the subscript i is used for both the latent index and the temporal-frequency bin in P_temp_n,i; the denominator appears to normalize the activation weights but does not normalize the joint distribution over orientation and frequency bins; and the 'high-confidence' trial filtering is described only in words. Since this map underpins the selectivity, clustering, and ablation conclusions, the exact normalization and the step from P_joint to A_i should be written out unambiguously, ideally with a small worked example.","section":"Section 2.4, Eq. (4)"}],"minor_comments":[{"comment":"The hyperparameter sweep in Appendix 5.2 restricts TopK values to {15, 20}, while Section 3.1 states that TopK was swept from 5 to 30 retained activations; please reconcile these descriptions.","section":"Appendix 5.2 vs. Section 3.1"},{"comment":"The text says '1–8Hz' when describing drifting-grating temporal frequencies, but the stimulus set includes 15 Hz; please correct the range.","section":"Section 3.2"},{"comment":"There are several typos, including 'intergrating' in the Discussion, 'plateou' in Appendix 5.3, and 'focussed' in Section 2.3; a proofreading pass is needed.","section":"Discussion"},{"comment":"The term 'receptive field' is used for activation maps built from model predictions; this term has a specific neurophysiological meaning, and its use here may overstate what the maps represent. Consider using 'activation map' or 'stimulus selectivity map' consistently.","section":"Section 2.4"},{"comment":"The TopK-SAE method is attributed to Makhzani and Frey [13], but the per-sample TopK variant used here is more commonly associated with later dictionary-learning work; citing the specific TopK-SAEs from the language-model-interpretability literature would be more precise.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript does not mention code or data availability. Since the validity of the central claims depends on details of target selection and evaluation splits, releasing code or at least a precise data-processing pipeline would be important for any revised version. The fit of the paper is reasonable for an applied machine-learning or computational-neuroscience venue, but the current analysis does not yet establish the interpretability and causal claims as stated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Freeman et al. take the TopK sparse autoencoder trick from LLM interpretability and point it at POYO+, a transformer trained on Allen Brain Observatory calcium imaging. The new thing here is the application: nobody has run SAEs on transformer latents from neural population data before, and the paper backs that up with a sensible survey of the relevant SAE literature. The work is also honest about its own limitations — the discussion says residual entanglement remains — and the VIP versus SST directional selectivity result is a genuinely clever biological validation that holds across eight independently trained checkpoints. That alone makes the paper worth a look.\n\nWhere it gets soft is the 'receptive field' construction in Section 2.4 and the ablation selection in Section 2.6. The activation maps are built by weighting the model's own softmax predictions, not ground truth orientation or temporal frequency. At roughly 45% decoding accuracy, those maps can encode the model's confusions rather than the stimulus. And the ablation targets are selected and evaluated on the same data, so the 0-degree-specific drop in accuracy is partly built into the selection procedure. This is not a fatal flaw for the whole paper — the reconstruction fidelity, the Cre-line decoding preservation, and the VIP/SST result stand independently — but it does mean the headline claim of causal, feature-specific ablations is over-stated as written.\n\nOther soft spots: the key numbers (45% vs 40%, R2 of 0.885) come without error bars or significance tests, and no code is released. A revision should fix selection-on-evaluation by selecting ablation targets on a held-out split and validating the receptive fields against ground-truth stimulus labels.\n\nBottom line: this is a useful proof-of-concept that deserves a serious referee. The central application is new, the biological validation is clever, and the methodological flaws are addressable. I'd send it out.","headline":"First real application of TopK SAEs to transformer latents from neural population data, with a clever biological validation and honest limitations, but the causal ablation claim is undercut by circular selection and predicted-label receptive fields.","tokens_in":12011,"tokens_out":1556,"would_cite":false,"duration_ms":17626,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A sparse autoencoder inserted into a brain-decoding transformer keeps performance while turning internal units into selective, causal encoders of visual features.","keywords":["sparse autoencoders","mechanistic interpretability","neural decoding","transformer","calcium imaging","visual cortex","TopK sparsity","feature ablation"],"falsifier":"Recompute the activation maps with the true presented orientation and temporal frequency, encoded as one-hot labels, instead of the model's softmax probabilities; if the median selectivity gap (0.4 versus 0.9 occupied bins) and the 0-degree ablation specificity disappear, the claim that SAE latents genuinely encode stimulus features is not supported.","tokens_in":1782,"feed_emoji":"🧠","tokens_out":1984,"duration_ms":71412,"temperature":0.7,"pith_summary":"The paper tries to establish that transformer models trained on neural population data can be made interpretable without losing their predictive power. It inserts a TopK sparse autoencoder between the final cross-attention latents and the output layer of POYO+, a transformer trained to decode visual stimuli from calcium imaging in mouse visual cortex. The authors report that decoding accuracy is largely preserved, that individual SAE units become sharply tuned to orientation, temporal frequency, and genetic cell type, and that ablating those units produces feature-specific drops in accuracy. The value, if true, is a non-invasive route from black-box neural decoders to mechanistic hypotheses about what information is encoded and where.","feed_headline":"Sparse autoencoder exposes what a brain-decoding transformer sees","feed_subtitle":"A sparse bottleneck keeps decoding accuracy while giving each unit a sharp stimulus tuning that can be surgically removed.","key_machinery":"The load-bearing object is the TopK sparse autoencoder, an overcomplete dictionary of 192 units (expansion factor 3) that retains only the $k$ largest activations per input, with no sparsity penalty in the loss. After training to reconstruct the 64-dimensional output of POYO+'s final cross-attention layer, it is inserted between that layer and the linear projection head, so that every prediction passes through a sparse code. Two auxiliary instruments carry the interpretation: activation maps, which weight each trial's softmax class probabilities by the absolute latent activation to give each unit a $5\\times8$ receptive field over temporal frequency and orientation, and binary masks that zero out selected latents during inference to test causal necessity. The TopK mechanism is what produces selectivity; the masks are what turn selectivity into causal claims.","core_discovery":"On the paper's own terms, the central discovery is that a TopK sparse autoencoder placed after the transformer's final cross-attention output untangles entangled latent dimensions into units that are biologically readable and causally actionable. Concretely, the authors show three things: the SAE-injected model retains more than 40 percent accuracy on orientation and temporal-frequency decoding versus roughly 45 percent for baseline, with chance at 12.5 percent and 20 percent respectively; running-speed prediction falls from $R^2\\approx0.57$ to $\\approx0.52$; SAE latents occupy a median of 0.4 of the 40 stimulus bins compared to 0.9 for baseline latents, and cluster into sharply tuned receptive fields; and ablating latents associated with the 0-degree orientation selectively reduces 0-degree decoding, while ablating SST-associated latents drops SST Cre-line decoding from 0.73 to 0.56 without affecting VIP decoding. The paper reads the first two findings as evidence that sparsity produces interpretable units rather than reconstruction artifacts, and the ablation results as evidence that the units are causally responsible for the features they seem to encode.","pith_inferences":["Because the receptive fields are built from the model's own softmax labels rather than the true presented stimulus, the reported tuning could partly reflect model confusions; recomputing with ground-truth labels would clarify whether the selectivity is genuine.","The paper's own ablation result leaves SST decoding at 0.56 rather than chance, consistent with its admission of residual entanglement; a stricter test would ablate every unit with any significant SST association and ask whether decoding collapses to chance.","The same sparse-decomposition approach could transfer to other neural decoders and to closed-loop brain-computer interfaces, but that extension is untested and would need to show that ablations remain safe and selective under distribution shift.","The analysis is confined to drifting gratings in primary visual cortex, a deliberately low-dimensional and well-understood stimulus space; generalizing the interpretability claim to natural movies or multi-region recordings is an open empirical question."],"forward_implications":["A brain-decoding transformer can be made interpretable without much accuracy loss: orientation and temporal-frequency decoding drop only from roughly 45 percent to above 40 percent, while running-speed $R^2$ falls from about 0.57 to about 0.52.","SAE latents are markedly more selective than baseline latents, occupying a median of 0.4 stimulus bins versus 0.9, so each unit corresponds to a narrower, more interpretable feature.","Ablating latents tuned to the 0-degree orientation selectively impairs 0-degree decoding while leaving other orientations largely intact, showing that encoding of that stimulus is sparse and causal.","Ablating SST-associated latents drops SST Cre-line decoding from 0.73 to 0.56 without affecting VIP decoding, showing that cell-type information is separable enough to be surgically removed.","Targeting latents associated with emergent features such as genetic background offers a route to remove sensitive or unwanted information from neural decoders without retraining the model."],"supporting_citations":[{"why":"Supplies POYO+, the pretrained transformer whose final cross-attention latents are the object of the interpretability analysis.","marker":"[15]"},{"why":"Supplies the Allen Brain Observatory Visual Coding calcium-imaging dataset and its stimulus structure used throughout.","marker":"[16]"},{"why":"Introduces K-sparse autoencoders, the TopK mechanism that enforces sparse latents.","marker":"[13]"},{"why":"Establishes sparse autoencoders as a method for finding interpretable features in model activations.","marker":"[9]"},{"why":"Demonstrates monosemantic SAE features in language models, motivating the application of SAEs to transformer latents.","marker":"[10]"},{"why":"Provides the classic sparse-coding account of V1 receptive fields, linking sparse representations to visual cortex biology.","marker":"[12]"},{"why":"Defines the Perceiver IO architecture on which POYO+ builds, motivating the analysis of the final cross-attention tokens.","marker":"[20]"},{"why":"Supplies the VIP interneuron directional-selectivity result used to predict and interpret VIP-associated latent coefficients.","marker":"[21]"}],"fun_headline_variants":["Sparse autoencoder turns brain-decoding transformers into open books","Ablating one neuron in brain-decoder transformer removes its chosen feature","Sparse bottleneck gives brain-decoding transformer units with sharp stimulus tuning","Causally probing a brain decoder: remove one unit, lose one visual feature"],"cache_read_input_tokens":14080,"weakest_assumption_plain":"The interpretability conclusion rests on treating the model's own predicted stimulus probabilities as the labels that define each latent's receptive field, even though the model is correct only about 45 percent of the time.","fun_headline_variants_meta":{"raw":{"variants":["Sparse autoencoder turns brain-decoding transformers into open books","Ablating one neuron in brain-decoder transformer removes its chosen feature","Sparse bottleneck gives brain-decoding transformer units with sharp stimulus tuning","Causally probing a brain decoder: remove one unit, lose one visual feature"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001069,"raw_usage":{"total_tokens":4479,"prompt_tokens":944,"completion_tokens":3535,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":3457}},"tokens_in":560,"tokens_out":3535,"duration_ms":23904,"temperature":1.0,"reasoning_tokens":3457,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:24:38.439721+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the activation maps with the true presented orientation and temporal frequency, encoded as one-hot labels, instead of the model's softmax probabilities; if the median selectivity gap (0.4 versus 0.9 occupied bins) and the 0-degree ablation specificity disappear, the claim that SAE latents genuinely encode stimulus features is not supported.","supporting_citations":[{"cited_title":"X., Arora, V ., Knight, I","cited_arxiv_id":null,"evidence_quote":"Supplies POYO+, the pretrained transformer whose final cross-attention latents are the object of the interpretability analysis."},{"cited_title":"E., Lecoq, J","cited_arxiv_id":null,"evidence_quote":"Supplies the Allen Brain Observatory Visual Coding calcium-imaging dataset and its stimulus structure used throughout."},{"cited_title":"L., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y ., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Tamkin, A., Nguyen, K., McLean, B., Burke, J","cited_arxiv_id":null,"evidence_quote":"Demonstrates monosemantic SAE features in language models, motivating the application of SAEs to transformer latents."},{"cited_title":"Olshausen and David J","cited_arxiv_id":null,"evidence_quote":"Provides the classic sparse-coding account of V1 receptive fields, linking sparse representations to visual cortex biology."},{"cited_title":"J., Ocker, G","cited_arxiv_id":null,"evidence_quote":"Supplies the VIP interneuron directional-selectivity result used to predict and interpret VIP-associated latent coefficients."}],"review_version":1}