REVIEW 4 major objections 5 minor 15 references
Concept-SAE: A Controllable and Invertible Concept Interface for Sparse Autoencoders
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper claims that supervising SAE tokens with both existence scores and spatial masks makes model features faithful, disentangled, and causally intervenable, turning sparse autoencoders from passive dictionaries into active probes of…
desk verdict Novel concept/free token architecture worth engaging, but the causal-intervention and vulnerability-localization claims need controls before they can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a tokenizer–aggregator pair. The Concept Tokenizer projects intermediate feature maps into concept-specific embeddings and predicts, for each concept, a binary existence score and a spatial segmentation mask under MSE losses plus an L1 sparsity penalty. The Concept Aggregator, trained after the tokenizer is frozen, multiplies the predicted score and segmentation, fuses them through an MLP, and reconstructs the original feature map; a KL-divergence term aligns the aggregator's per-concept channel distribution with the tokenizer's so that edits stay localized. A parallel Free Tokenizer and Free Aggregator, trained jointly without supervision, reconstruct the residual information. The invertible reconstruction is what turns an edited concept score into a counterfactual feature that can be substituted into the model.
What would settle it
Compute the reconstruction error of the concept aggregator on held-out images and compare downstream accuracy when hidden features are replaced by concept-plus-free reconstructions against a standard SAE reconstruction baseline; if the intervention effect vanishes or predictions change for unrelated concepts, the causal corrections are artifacts of distribution shift rather than evidence of concept control.
Extended reading notes
Core claim
Concept-SAE's central claim is that a hybrid disentanglement strategy—anchoring some SAE tokens to user-defined concepts through dual supervision on concept existence and spatial localization—forges concept representations that are faithful and clean enough to support causal claims. The paper reports that on CelebA and ImageNet, across ResNet-18 and ViT models, concept tokens reconstruct only concept-relevant image regions and achieve higher localization ratios than a concept-embedding baseline. It further reports that concept-score entropy rises on incorrectly predicted and adversarially perturbed inputs, that editing concept scores and feeding the reconstructed feature back into the model changes predictions in the intended direction, and that layers showing large Jensen–Shannon distance between clean and adversarial concept-score distributions are the ones whose fine-tuning yields the largest robustness gains.
Load-bearing premise
The claim leans on the assumption that a feature reconstructed from edited concept scores lies on the target model's feature manifold, so the downstream model responds as it would to an input whose concept actually changed; the paper reports only qualitative reconstruction examples rather than measuring this counterfactual validity directly.
Editorial extensions
If this is right
- Concept scores become a diagnostic signal: if entropy rises systematically on misclassified and adversarial inputs, score distributions alone can flag unreliable or attacked inputs.
- Direct intervention becomes possible: users can change a concept score, reconstruct the feature, and observe a corresponding change in the model's prediction, enabling causal correction of errors.
- Vulnerability localization guides defense: fine-tuning layers with the largest clean-versus-adversarial concept-score shift yields larger robustness gains than fine-tuning other layers.
- Free tokens preserve the exploratory role of standard SAEs, so constraining a subset of tokens to user-defined concepts does not have to sacrifice open-ended feature discovery.
Reading between the lines
- A direct extension the paper leaves implicit: the same tokenizer–aggregator interface could be transferred to language models by replacing spatial masks with token-span masks, turning 'localized concept' supervision into span supervision.
- Concept-score entropy and Jensen–Shannon distance could be reused as a lightweight, training-free out-of-distribution or adversarial detector; the paper reports the signal but does not frame it as a standalone detector.
- The free-token pool is a natural place to grow the concept set: tokens with consistently high activation on coherent image clusters could be promoted into supervised concept tokens, letting the concept vocabulary expand from the model's own structure.
- Because concept fidelity inherits the quality of the external label sources, improvements in foundation-model segmentation and vision-language grounding should directly improve Concept-SAE's faithfulness, even for concepts that are hard to localize today.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Concept-SAE, a method to augment sparse autoencoders with concept tokens that are supervised by VLM-generated existence scores and segmentation masks, while retaining free tokens trained with the usual SAE objective. The authors claim that this dual-supervision and staged training yields concept tokens that are faithful, spatially localized, and disentangled from the residual subspace, enabling causal intervention on predictions and layer-wise localization of adversarial vulnerabilities. Experiments on CelebA and ImageNet compare localization against Concept Embedding Models, report concept-score entropy for correct/incorrect/adversarial examples, and measure adversarial finetuning accuracy after targeting layers selected by JS distance.
Significance. If validated, Concept-SAE would be a valuable contribution to interpretability by providing externally anchored concept tokens that can be manipulated directly. The idea of using both existence labels and segmentation masks as dual supervision is a sensible way to ground token semantics, and the external anchor via GPT-4o and ClipSeg avoids purely self-referential concept labeling. The release of code supports reproducibility. However, the current evidence does not yet substantiate the 'validated fidelity' and 'causal' claims because key controls—reconstruction accuracy, identity-intervention baseline, random-layer baseline, and statistical significance—are missing.
major comments (4)
- [§3.3, Eq. (4); §4.2, Fig. 7] The causal-correction experiment replaces the original activation h with A_concept(T_concept(h)) after editing concept scores, but no quantitative reconstruction fidelity for the aggregator is reported (only qualitative examples in Figs. 4, 5, 12, 13), and there is no control condition in which h is replaced by the unmodified reconstruction. Without such a control, and without a check that edited-score inputs remain on the target model's activation manifold, the prediction flips in Fig. 7 could be artifacts of a lossy or out-of-distribution aggregator rather than evidence about the model's concept usage. This missing control is load-bearing for the abstract's 'validated fidelity' claim and for RQ2's conclusion; §5 acknowledges label noise but does not address this counterfactual-validity gap.
- [Table 3, RQ3] The claim that layers with larger JS distance are more vulnerable is supported only by the correlation between the JS distance column and the after-finetune accuracy column. The experiment lacks a random-layer baseline: any fine-tuning of a single layer might improve adversarial accuracy by itself, and depth or other confounds could explain the pattern. The table also reports no error bars or significance tests, and only FGSM attacks are used, so the 'systematically localizing adversarial vulnerabilities' conclusion is not yet established.
- [Table 2, RQ2] The entropy differences between correct and incorrect predictions are extremely small (e.g., +0.024 to +0.036 for CelebA) and the paper reports no variance, significance tests, or alternative baselines. The conclusion that 'uncertain concept activations are a key factor of model failures' is not supported by these unreplicated point estimates.
- [Table 1, RQ1] The localization comparison against CEM is limited to three shallow layers of ResNet-18 on CelebA, with no error bars and no evaluation on other models or datasets. The claim of consistently superior concept localization is therefore not established beyond this narrow configuration.
minor comments (5)
- [Table 1 caption] The word 'distangled' should be 'disentangled'.
- [Eqs. 3, 4, and 5] The notation λ1, λ2, λ3 is reused with different numerical weightings across the three loss functions; consider using distinct symbols (e.g., α, β, γ) to avoid confusion.
- [§3.2-3.4] The number and dimension of concept tokens are not reported; only the free-token count (36) is given. This information is needed to interpret the capacity of the concept module.
- [§4.2, RQ2] The statement that interventions on deeper ViT layers are more effective is based on Fig. 7, which is a set of qualitative examples; quantitative aggregate statistics would strengthen the claim.
- [Appendix C] The paper provides the full prompt used for ImageNet concept-label generation but not the corresponding CelebA prompt, which makes the external anchor for the key experiments less reproducible.
Circularity Check
Diagnostic evaluations (causal correction and vulnerability localization) are evaluated on the same data used to set their intervention/fine-tuning targets, making the 'predictions' partially in-sample fits.
-
fitted input called prediction
[Section 4.2, RQ2, 'For causal correction' paragraph (around Fig. 7)]
"On CelebA mis-classifications, for example, male images incorrectly predicted as female often exhibit insufficient activations of male-associated concepts such as beard, mustache, and adam’s apple. By increasing the scores of these concepts, the prediction is corrected to male; conversely, reducing them corrects the opposite errors."
The intervention values are chosen after observing the error patterns on the same images. The 'correction' is therefore a post hoc fit: the concept scores are set to values known to be associated with the desired label on those exact inputs, and the resulting prediction flip is presented as evidence of a causal link. No held-out intervention, random-score control, or unmodified-reconstruction control is reported, so the demonstration reduces to showing that manually modifying a feature in a way derived from the target label changes the prediction on that same sample.
-
fitted input called prediction
[Section 4.2, RQ3, 'Vulnerability Localization & Robustness Gains' (Table 3)]
"We compute the Jensen–Shannon (JS) distance (Lin, 2002) between the concept score distributions of clean and adversarial samples. Larger JS distances indicate stronger distributional shifts encountering adversarial samples and thus higher vulnerability. Based on this observation, we hypothesize that layers with higher JS distances are most fragile under attack. We validate this hypothesis through layer-wise finetuning. For each layer, we freeze the rest layers and retrain the chosen layer using mixed clean and adversarial samples for two epochs."
The JS-distance vulnerability metric is computed on the same FGSM adversarial samples used for layer-wise finetuning and for the reported 'Accuracy of the model on adversarial samples after finetuning' (Table 3 caption). The model is retrained on those very adversarial samples, so any accuracy gain on them is partly an in-sample effect. The claimed link between high JS distance and greater robustness improvement is thus evaluated on the data that defined both the metric and the fine-tuning target, rather than on an independent adversarial set; the result is partially forced by construction.
full rationale
The paper's core concept anchoring is grounded externally: concept existence scores come from a VLM (GPT-4o) and spatial masks from ClipSeg, so the tokenizer supervision is not circular. There is no load-bearing self-citation chain or imported uniqueness theorem. The main circularity concerns the two diagnostic validations. In RQ2, the causal-correction demonstration sets concept scores based on the observed error patterns on the same images, making the 'correction' a post hoc fit rather than an independent causal test. In RQ3, the JS-distance vulnerability metric is computed on the same adversarial samples used for fine-tuning and for the reported accuracy, so the robustness improvement is in-sample. These do not invalidate the external anchoring or the comparative localization results, but they mean the paper's headline claims of 'validated fidelity' enabling causal probing are only partially supported by independent evidence. Overall circularity is moderate, not total: score 4.
Assumptions & free parameters
free parameters (4)
- Concept vocabulary =
11 concept groups for CelebA, 14 for ImageNet (LLM-generated)
- Loss weights =
(1,1,0.1); (1,0.01,1); (1,1)
- Free token count =
36
- Intervention thresholds =
concept scores set to 0.0 or 1.0
assumptions (4)
- domain assumption The VLM existence scores and ClipSeg segmentation masks are sufficiently accurate to serve as ground truth for concept supervision.
- domain assumption Sparse autoencoders can linearly decompose model activations into approximately monosemantic features.
- domain assumption The learned concept aggregator reconstructs features accurately enough that modifying concept scores produces in-distribution counterfactual features.
- domain assumption FGSM perturbations are representative of the adversarial vulnerability being localized.
Cite this review
Pith. "Pith review of Concept-SAE: A Controllable and Invertible Concept Interface for Sparse Autoencoders." pith.science (2026). https://pith.science/paper/B637LZBZ
@misc{pith2026250922015,
author = {Pith},
title = {Pith review of: Concept-SAE: A Controllable and Invertible Concept Interface for Sparse Autoencoders},
year = {2026},
howpublished = {\url{https://pith.science/paper/B637LZBZ}},
note = {Machine review of arXiv:2509.22015}
}
read the original abstract
Standard Sparse Autoencoders (SAEs) excel at discovering a dictionary of a model's learned features, providing a powerful lens for passive feature discovery. However, this passive nature makes it difficult to systematically evaluate or analyze concepts that users explicitly care about. We introduce Concept-SAE, a framework that augments SAEs with a structured and controllable interface for probing user-defined concepts. Concept-SAE decomposes an activation subspace into two orthogonal components: Concept Tokens, which are aligned to externally specified semantics through dual supervision on both concept existence and spatial localization, and Free Tokens, which operate like standard SAEs to capture all remaining information. This hybrid disentanglement strategy ensures that Concept Tokens are faithful, spatially grounded, and cleanly separated from the residual subspace while preserving the ability of SAEs for open-ended concept discovery. We conduct extensive experiments demonstrating that Concept-SAE yields high-fidelity, well-localized, and strongly disentangled concept representations, outperforming alternatives in interface quality. Finally, we validate the utility of this conceptual interface through three diagnostic evaluations: a detection test on classifying adversarial image samples, a controllability test focusing on controlled counterfactual editing and a stability test using adversarial perturbations. Together, these results show that Concept-SAE equips SAEs with a reliable mechanism for evaluating, probing, and diagnosing user-defined concepts.
Figures
Figures from the paper (10 more)
Reference graph
Works this paper leans on
-
[1]
Gpt-4 technical report.arXiv preprint arXiv:2303.08774,
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,
-
[6]
Measuring and guiding monosemanticity.arXiv preprint arXiv:2506.19382,
Ruben H ¨arle, Felix Friedrich, Manuel Brack, Stephan W ¨aldchen, Bj ¨orn Deiseroth, Patrick Schramowski, and Kristian Kersting. Measuring and guiding monosemanticity.arXiv preprint arXiv:2506.19382,
-
[9]
Luke Marks, Alasdair Paren, David Krueger, and Fazl Barez. Enhancing neural network inter- pretability with feature-aligned sparse autoencoders.arXiv preprint arXiv:2411.01220,
-
[11]
Charles O’Neill and Thang Bui. Sparse autoencoders enable scalable and reliable circuit identifica- tion in language models.arXiv preprint arXiv:2405.12522,
-
[12]
Gonc ¸alo Paulo and Nora Belrose. Sparse autoencoders trained on the same data learn different features.arXiv preprint arXiv:2501.16615,
-
[15]
Samuel Stevens, Wei-Lun Chao, Tanya Berger-Wolf, and Yu Su. Sparse autoencoders for scientifi- cally rigorous interpretation of vision models.arXiv preprint arXiv:2502.06755,
-
[1997]
Probing the representational power of sparse autoencoders in vision models
Matthew Lyle Olson, Musashi Hinck, Neale Ratzlaff, Changbai Li, Phillip Howard, Vasudev Lal, and Shao-Yen Tseng. Probing the representational power of sparse autoencoders in vision models. arXiv preprint arXiv:2508.11277,
-
[2009]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929,
arXiv 2010
Show all 15 references
-
[2014]
The missing curve detectors of inceptionv1: Applying sparse autoencoders to incep- tionv1 early vision.arXiv preprint arXiv:2406.03662,
Liv Gorton. The missing curve detectors of inceptionv1: Applying sparse autoencoders to incep- tionv1 early vision.arXiv preprint arXiv:2406.03662,
-
[2020]
Hyesu Lim, Jinho Choi, Jaegul Choo, and Steffen Schneider
10 Preprint. Hyesu Lim, Jinho Choi, Jaegul Choo, and Steffen Schneider. Sparse autoencoders reveal selective remapping of visual concepts during adaptation.arXiv preprint arXiv:2412.05276,
-
[2021]
Improving dictionary learning with gated sparse autoen- coders.arXiv preprint arXiv:2404.16014,
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, J ´anos Kram´ar, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoen- coders.arXiv preprint arXiv:2404.16014,
-
[2022]
A survey on sparse autoencoders: Interpreting the internal mechanisms of large language models
Dong Shu, Xuansheng Wu, Haiyan Zhao, Daking Rai, Ziyu Yao, Ninghao Liu, and Mengnan Du. A survey on sparse autoencoders: Interpreting the internal mechanisms of large language models. arXiv preprint arXiv:2503.05613,
-
[2023]
Scaling and evaluating sparse autoencoders.arXiv preprint arXiv:2406.04093,
Leo Gao, Tom Dupr ´e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders.arXiv preprint arXiv:2406.04093,
-
[2024]
Explaining and harnessing adversarial examples.arXiv preprint arXiv:1412.6572,
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples.arXiv preprint arXiv:1412.6572,
-
[2025]
In- terpreting attention layer outputs with sparse autoencoders.arXiv preprint arXiv:2406.17759,
Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, and Neel Nanda. In- terpreting attention layer outputs with sparse autoencoders.arXiv preprint arXiv:2406.17759,
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.