REVIEW 3 major objections 5 minor 22 references
Mammo-SAE: Interpreting Breast Cancer Concept Learning with Sparse Autoencoders
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A sparse autoencoder reveals that just ten latent neurons carry most of the breast-cancer signal in Mammo-CLIP.
desk verdict A useful first transfer of SAEs to mammography with two concrete neuron-level findings, but the headline intervention result needs a random-neuron baseline and a real train/test split before it can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Mammo-SAE: an encoder-decoder sparse autoencoder trained with a reconstruction loss plus an $\ell^1$ sparsity penalty, mapping patch-level CLIP features $x_j \in \mathbb{R}^{2048}$ to a sparse latent $z \in \mathbb{R}^{16384}$ through a ReLU encoder. Class-level relevance scores rank latent neurons by their mean activation over the positive class, and group interventions (top-k activated keeps only the selected neurons; top-k deactivated zeroes them out) turn correlation into a causal test by passing edited latents through the decoder to the classifier. This machinery lets the paper identify, localize, and functionally ablate concept neurons without retraining the vision model.
What would settle it
Train a separate sparse autoencoder on final-layer features from the frozen pretrained Mammo-CLIP model and another on the finetuned model, then repeat the class-separation plots and the top-10 activation and deactivation curves; if the finetuned-versus-pretrained separation and the ten-neuron recovery pattern disappear when each model is measured with its own autoencoder, the conclusions are artifacts of a single SAE specialized to the finetuned feature distribution rather than properties of the models.
Extended reading notes
Core claim
Trained on final-layer patch features of the finetuned Mammo-CLIP suspicious-calcification classifier, a sparse autoencoder with 16,384 latent neurons learns representations whose top class-specific neurons are functionally decisive. Activating only the top-10 neurons for the two classes almost reproduces the model's full AUC-ROC in several settings, and zeroing those neurons out causes a sharp drop, indicating that the model relies on a compact, concept-aligned subset rather than a diffuse feature spread. For suspicious calcification, seven of the top ten latent neurons activate inside the annotated region; for mass, alignment is weaker, consistent with the model's lower AUC. Finetuning the backbone enlarges the separation between class-mean latent activation vectors and yields higher localization mean average precision, which the paper attributes to the model learning more discriminative, spatially meaningful features while still retaining at least one background-sensitive neuron that fires across classes and models.
Load-bearing premise
The conclusions about finetuning and about the mass concept rest on a single sparse autoencoder trained only on final-layer activations of the finetuned suspicious-calcification model being an unbiased measuring instrument for all four model-and-concept settings.
Editorial extensions
If this is right
- Keeping only the top-k class-specific SAE neurons nearly recovers the original AUC-ROC, so a compact set of latent features is sufficient for mass and calcification prediction.
- Zeroing those top-k neurons sharply drops AUC-ROC, showing the model causally depends on those specific neurons.
- In the finetuned model, top-10 latent neurons achieve higher localization mAP than in the pretrained model, indicating that finetuning aligns concept features with ground-truth regions.
- Finetuning makes the class-mean latent vectors more separated, offering an explanation for its AUC gain and improved interpretability.
- A persistent background-sensitive neuron remains active across both classes and both model variants, showing that confounding reliance survives finetuning.
Reading between the lines
- The paper reuses one SAE trained only on the finetuned suspicious-calcification model for all comparisons; I would not trust the finetuned-versus-pretrained separation until it is reproduced with an SAE trained separately on each model.
- If the top-k intervention generalizes, one could build a concept-bottleneck classifier from a few dozen SAE neurons and get nearly the same accuracy, which would make high-stakes screening models auditable by radiologists.
- A natural testable extension is to deactivate the persistent background neuron at inference and measure whether robustness or specificity improves; the paper's data suggest it is spurious but do not run that experiment.
- Applying the same probe to earlier layers and to other concepts such as nipple retraction or skin thickening would test whether monosemantic concept neurons exist throughout the backbone rather than only at the final layer.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Mammo-SAE, a sparse autoencoder trained on patch-level final-layer activations of the Mammo-CLIP breast imaging foundation model, and uses it to identify latent neurons associated with the concepts of mass and suspicious calcification. The authors report that top class-level neurons align with ground-truth regions, that interventions on a small number of top-k neurons can largely recover or destroy classification AUC, and that finetuning produces stronger class separation in the SAE latent space. The main evidence is the intervention analysis in Section 3.1 and Figure 2, the localization mAP in Table 2, and the latent-vector comparisons in Section 3.3.
Significance. If the central intervention claim holds, the paper would be a useful demonstration that sparse autoencoders can localize compact, concept-relevant subspaces in medical vision-language models, with practical implications for auditing confounding behavior and for post-hoc intervention. The reconstruction sanity check in Table 1 is a genuine strength: it shows that inserting the SAE at the final layer costs less than 2% AUC, which supports the faithfulness of the SAE as a probe for the finetuned suspicious-calcification model. The intervention framework itself is appropriate in spirit. However, the claimed quantitative result that 'as few as 10 neurons' capture most of the task-relevant signal is currently undermined by the absence of a clear train/test separation in the neuron-ranking procedure and by the lack of a random-neuron baseline; the cross-model comparison is further weakened by reusing a single SAE trained on one model's features. These issues are load-bearing rather than cosmetic.
major comments (3)
- [Section 3.1 and Eq. (2)] The central intervention result may have selection leakage. The top-k neurons are ranked by class-mean activation computed in Eq. (2) over 'training samples,' but Section 3.1 does not state that the images used to rank neurons are disjoint from the images used to compute the post-intervention AUC in Figure 2. If the ranking and evaluation sets overlap, the top-k activated and deactivated curves become a near-mechanical consequence of selecting the dimensions that best separate the labels on the evaluation set itself, rather than evidence that a compact concept-aligned subspace is causally responsible for the model's predictions. The paper should either explicitly state that a held-out split was used for AUC evaluation, or repeat the analysis with neurons ranked on a training split and evaluated on a test split. In addition, the intervention curves need a random-neuron baseline: without it, the reader cannot tell whether selecting any k high-variance directions would produce a similar AUC recovery, which is essential for interpreting the '10 neurons suffice' claim.
- [Section 3.3 and Section 3, SAE Training] The conclusion that finetuning sharpens class separation in the SAE latent space (Figures 4a, 4b, and Figure 5) is potentially an artifact of the measurement instrument. The paper trains a single SAE on features from the finetuned suspicious-calcification model (d=2048, h=16384) and then reuses that same SAE to analyze the pretrained model and the mass concept. If the SAE has specialized to the finetuned feature distribution, its latent space may exaggerate structure in the finetuned model's activations while misreconstructing the pretrained model's activations. Table 1 shows only that AUC is preserved when the frozen model is run through the SAE, not that the latent geometry is unbiased across both feature distributions. The authors should either train a separate SAE per model and per concept, or provide evidence—for example, matched reconstruction error and per-neuron consistency across models—that the shared SAE is a neutral measurement instrument.
- [Table 2 and Section 3.2] The quantitative localization evidence is weaker than the narrative suggests. Table 2 reports mAP at IoU=0.25 for the top-10 neurons, but the per-neuron mAP values are generally low—for example, finetuned suspicious calcification has values between 0.005 and 0.278, and the mass pretrained row ranges from 0.018 to 0.053. The paper claims that '7 out of the top 10 latent neurons activate strongly within the annotated region' for suspicious calcification, but this claim is not directly backed by the mAP numbers, which are not averaged and are shown for only a small set of representative images. To support the alignment claim, the authors should report the fraction of neurons above a pre-specified mAP threshold over the full evaluation set, and should clarify whether the 7/10 statement refers to qualitative inspection of the figures or to a quantitative criterion.
minor comments (5)
- [Eq. (3)] The notation is inconsistent: the text says T(c)k is a subset of {1,...,k}, but the top-k neurons should be a subset of {1,...,h} of cardinality k. Please correct the set definition.
- [Section 2.1] There is a typo in 'Further implementaton details' and the term 'calcification' has a ligature/formatting artifact in the appendices. Please proofread the final typeset version.
- [Figure 8 caption] The caption for Figure 8 in the appendix reads 'Mass Calcification,' but the concept under study is 'Mass'; suspicious calcification is a separate concept. This appears to be a copy-paste error from Figure 6 and should be corrected.
- [Section 2.2] The notation is mixed: the dataset is described as {x, c} with x an image, but in Eq. (2) xj is a CLIP feature at spatial location j. Please disambiguate image-level and patch-level notation for readability.
- [Appendix, Figure 4 and Figure 5] The y-axes of the mean latent activation plots are unlabeled, and the x-axis is labeled only in units of 'Latent Neuron.' Adding axis labels and a caption describing the normalization would make the 'separation' claim easier to evaluate.
Circularity Check
Section 3.1 intervention claim reduces to in-sample top-k selection as written; no disjoint evaluation set or random-neuron control is reported.
-
fitted input called prediction
[Section 2.2 Eq. (2); Section 2.3 Eqs. (3)-(4); Section 3.1 Fig. 2]
"we compute the class-wise mean latent activation vector ¯z(c) ∈ Rh by averaging over all spatial locations and all training samples in that class ... Dc is the set of training images with class label c ... Latent neurons are ranked in descending order of s(c)t ... we compute the relevance score of each latent neuron with respect to two classes ... identify the top-k neurons per class. We then perform targeted interventions ..."
The top-k set is defined by Eq. (2) as the k latent coordinates with the largest class-mean activation computed on labeled 'training samples' for each class. The Fig. 2 AUC curves are then measured after keeping exactly those coordinates (Eq. (3)) or zeroing them (Eq. (4)). The paper reports no split between the Dc used in Eq. (2) and the images over which post-intervention AUC-ROC is computed. On the literal reading, the 'top-k' coordinates are the k directions that best separate the two classes on the evaluation set itself, so retaining only those directions preserves most of the linear class signal and removing them destroys it.
full rationale
The paper's central quantitative claim is the Section 3.1 intervention experiment. As written, the procedure selects top-k neurons by Eq. (2) on labeled training samples and then evaluates AUC after masking to those same neurons (Eqs. (3)-(4)) without stating that the AUC evaluation uses a disjoint set. Under that reading, the 'small subset of neurons captures the signal' result is an in-sample feature-selection artifact: the chosen coordinates are precisely those that maximize class separation on the evaluation set. This warrants a high circularity score. I did not find other load-bearing circularity: the paper does not rely on author self-citations, uniqueness theorems, or renamed prior results; the shared-SAE design (Section 3) is a measurement-comparability concern rather than a circular reduction, because the SAE is trained with an unsupervised reconstruction loss, not on the labels used for ranking. If a proper train/validation/test split were reported and the AUC curves were computed on a held-out set, the intervention result would be a legitimate supervised feature-selection finding rather than circular; however, the text as it stands does not document such a split.
Assumptions & free parameters
free parameters (5)
- Sparsity penalty lambda =
3e-5
- Latent dimension expansion factor =
8 (h=16384)
- Top-k neuron count for interventions and localization =
10
- Heatmap threshold for bounding boxes =
95th percentile
- SAE training schedule =
lr=3e-4, batch=4096, 200 epochs
assumptions (5)
- domain assumption Mammo-CLIP final-layer patch features are a sufficient substrate for concept-level interpretability.
- domain assumption VinDr-Mammo annotations for mass and suspicious calcification are accurate and define the target concepts.
- ad hoc to paper A single SAE trained on finetuned suspicious-calcification features transfers without bias to pretrained and mass features.
- domain assumption ReLU SAE with L1 sparsity yields monosemantic features in this medical domain.
- domain assumption Class-level neuron ranking and evaluation are performed on disjoint data.
Cite this review
Pith. "Pith review of Mammo-SAE: Interpreting Breast Cancer Concept Learning with Sparse Autoencoders." pith.science (2026). https://pith.science/paper/XRK6OT45
@misc{pith2026250715227,
author = {Pith},
title = {Pith review of: Mammo-SAE: Interpreting Breast Cancer Concept Learning with Sparse Autoencoders},
year = {2026},
howpublished = {\url{https://pith.science/paper/XRK6OT45}},
note = {Machine review of arXiv:2507.15227}
}
read the original abstract
Interpretability is critical in high-stakes domains such as medical imaging, where understanding model decisions is essential for clinical adoption. In this work, we introduce Sparse Autoencoder (SAE)-based interpretability to breast imaging by analyzing {Mammo-CLIP}, a vision--language foundation model pretrained on large-scale mammogram image--report pairs. We train a patch-level \texttt{Mammo-SAE} on Mammo-CLIP to identify and probe latent features associated with clinically relevant breast concepts such as \textit{mass} and \textit{suspicious calcification}. Our findings reveal that top activated class level latent neurons in the SAE latent space often tend to align with ground truth regions, and also uncover several confounding factors influencing the model's decision-making process. Additionally, we analyze which latent neurons the model relies on during downstream finetuning for improving the breast concept prediction. This study highlights the promise of interpretable SAE latent representations in providing deeper insight into the internal workings of foundation models at every layer for breast imaging. The code will be released at https://krishnakanthnakka.github.io/MammoSAE/
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Scien- tific Reports 14(1), 10341 (2024)
Choukali, M.A., Amirani, M.C., Valizadeh, M., Abbasi, A., Komeili, M.: Pseudo- class part prototype networks for interpretable breast cancer classification. Scien- tific Reports 14(1), 10341 (2024)
work page 2024
-
[2]
arXiv preprint arXiv:2309.08600 (2023)
Cunningham, H., Ewart, A., Riggs, L., Huben, R., Sharkey, L.: Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600 (2023)
arXiv 2023
-
[3]
In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion
Dreyer, M., Purelku, E., Vielhaben, J., Samek, W., Lapuschkin, S.: Pure: Turning polysemantic neurons into pure features by identifying relevant circuits. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion. pp. 8212–8217 (2024)
work page 2024
-
[4]
arXiv preprint arXiv:2406.04093 (2024)
Gao, L., la Tour, T.D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., Wu, J.: Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093 (2024)
arXiv 2024
-
[5]
In: International conference on medical image computing and computer- assisted intervention
Ghosh, S., Poynton, C.B., Visweswaran, S., Batmanghelich, K.: Mammo-clip: A vi- sion language foundation model to enhance data efficiency and robustness in mam- mography. In: International conference on medical image computing and computer- assisted intervention. pp. 632–642. Springer (2024)
work page 2024
-
[6]
arXiv preprint arXiv:2410.20526 (2024)
He, Z., Shu, W., Ge, X., Chen, L., Wang, J., Zhou, Y., Liu, F., Guo, Q., Huang, X., Wu, Z., et al.: Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders. arXiv preprint arXiv:2410.20526 (2024)
arXiv 2024
-
[7]
In: Proceedings of the IEEE/CVF international conference on computer vision
Huang, S.C., Shen, L., Lungren, M.P., Yeung, S.: Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 3942–3951 (2021)
2021
-
[8]
Kajala, A., Jaiswal, S., Kumar, R.: ‘breaking the black box: Heatmapdriven trans- parency to breast cancer detection with efficientnet and grad cam. Educ. Admin., Theory Pract 30(5), 4999–5009 (2024)
work page 2024
Show all 22 references
-
[9]
In: Convolutional neural networks with swift for Tensor- flow: image recognition and dataset categorization, pp
Koonce, B.: Efficientnet. In: Convolutional neural networks with swift for Tensor- flow: image recognition and dataset categorization, pp. 109–123. Springer (2021) 1 https://centrescancer.chuv.ch/equipe/rencontrer-lequipe-du-centre-du-sein/ Mammo-SAE: Interpreting Breast Cance...
2021
-
[10]
arXiv preprint arXiv:2408.05147 (2024)
Lieberum, T., Rajamanoharan, S., Conmy, A., Smith, L., Sonnerat, N., Varma, V., Kramár, J., Dragan, A., Shah, R., Nanda, N.: Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05147 (2024)
2024 arXiv
-
[11]
arXiv preprint arXiv:2412.05276 (2024)
Lim, H., Choi, J., Choo, J., Schneider, S.: Sparse autoencoders reveal selective remapping of visual concepts during adaptation. arXiv preprint arXiv:2412.05276 (2024)
2024 arXiv
-
[12]
IEEE Access (2024)
Liu, S., Himel, G.M.S., Wang, J.: Breast cancer classification with enhanced inter- pretability: Dalaresnet50 and dt grad-cam. IEEE Access (2024)
2024
-
[13]
arXiv preprint arXiv:2405.08366 (2024)
Makelov, A., Lange, G., Nanda, N.: Towards principled evaluations of sparse au- toencoders for interpretability and control. arXiv preprint arXiv:2405.08366 (2024)
2024 arXiv
-
[14]
arXiv preprint arXiv:2401.17975 (2024)
Marshall, S.C., Kirchner, J.H.: Understanding polysemanticity in neural networks through coding theory. arXiv preprint arXiv:2401.17975 (2024)
2024 arXiv
-
[15]
Scientific Data10(1), 277 (2023)
Nguyen, H.T., Nguyen, H.Q., Pham, H.H., Lam, K., Le, L.T., Dao, M., Vu, V.: Vindr-mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography. Scientific Data10(1), 277 (2023)
2023
-
[16]
O’Mahony, L., Andrearczyk, V., Müller, H., Graziani, M.: Disentangling neuron representationswithconceptvectors.In:ProceedingsoftheIEEE/CVFConference on Computer Vision and Pattern Recognition. pp. 3770–3775 (2023)
2023
-
[17]
In: World Conference on Explainable Artificial Intelligence
Pathak, S., Schlötterer, J., Veltman, J., Geerdink, J., van Keulen, M., Seifert, C.: Prototype-based interpretable breast cancer prediction models: Analysis and challenges. In: World Conference on Explainable Artificial Intelligence. pp. 21–42. Springer (2024)
2024
-
[18]
Multimedia Tools and Applications 83(19), 57551–57578 (2024)
Raghavan, K., B, S., v, K.: Attention guided grad-cam: an improved explainable artificial intelligence model for infrared breast cancer detection. Multimedia Tools and Applications 83(19), 57551–57578 (2024)
2024
-
[19]
In: European Conference on Computer Vision
Rao, S., Mahajan, S., Böhle, M., Schiele, B.: Discover-then-name: Task-agnostic concept bottlenecks via automated concept discovery. In: European Conference on Computer Vision. pp. 444–461. Springer (2024)
2024
-
[20]
arXiv preprint arXiv:2403.08295 (2024)
Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M.S., Love, J., et al.: Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 (2024)
2024 arXiv
-
[21]
arXiv preprint arXiv:2302.13971 (2023)
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
2023 arXiv
-
[22]
In: Proceedings of the Conference on Empirical Methods in Natural Language Processing
Wang, Z., Wu, Z., Agarwal, D., Sun, J.: Medclip: Contrastive learning from un- paired medical images and text. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing. vol. 2022, p. 3...
2022
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.