REVIEW 3 major objections 6 minor 25 references
Adapting a Segmentation Foundation Model for Medical Image Classification
T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read This paper claims that a frozen SAM encoder improves medical image classification when its multi-depth features are fused through SLCA attention into standard classifiers, especially under scarce labels.
desk verdict Solid incremental paper with a clear method and fair comparisons, but the missing random-encoder control prevents the authors from claiming SAM's features are the cause of the gains. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the frozen SAM image encoder sampled at five depths — patch embedding, Transformer blocks 1, 16, and 32, and the Conv block — together with a fusion pathway. Each extracted feature map $F$ is passed through SLCA, which computes spatially localized channel weights as $\mathrm{SLCA}(F) = \sigma(\mathrm{Conv}_2(\mathrm{ReLU}(\mathrm{Conv}_1(\mathrm{SLAP}(F)))))$, where $\mathrm{SLAP}$ is spatial local average pooling, then applies the weights to the classifier's intermediate features and adds them by residual connection. A lightweight projector head resizes SAM features to the classifier's channel count and resolution before concatenation. The same five extraction points are used for all backbones and datasets.
What would settle it
Run the same training protocol on a held-out medical dataset, comparing the fixed five-block extraction with the best single-block extraction chosen by an ablation on that dataset; if single-block matches or beats the five-block fusion, the paper's core claim that multi-block fusion drives the gain would fail.
Extended reading notes
Core claim
The central claim is that segmentation-oriented features from a frozen SAM image encoder can be injected into existing classifiers at multiple stages without any SAM fine-tuning, and that the way the injection happens matters. Direct additive fusion of SAM features hurts performance, while applying SLCA attention before a residual fusion, plus a small projector head, gives consistent accuracy and AUC gains across three public medical datasets and three classification backbones (ResNet152, SENet154, and Swin Transformer v2). The gains are largest when only 10% of the training labels are used, which the paper reads as evidence of data efficiency. The paper also reports that its fusion approach outperforms the SAMAug-C augmentation baseline on nearly all reported settings.
Load-bearing premise
The fixed set of SAM layers used for feature extraction (patch embedding, blocks 1, 16, 32, and the Conv block) is chosen from an ablation on one dataset and is assumed to transfer to other datasets and backbones without per-dataset revalidation.
Editorial extensions
If this is right
- Using the frozen SAM encoder as a feature source improves classification on RetinaMNIST, BreastMNIST, and ISIC 2017 across ResNet152, SENet154, and Swin Transformer v2.
- The gains grow as training labels shrink; for example, at 10% of RetinaMNIST labels, the method improves Swin Transformer v2 accuracy by 5.75 percentage points, suggesting data efficiency.
- SLCA fusion outperforms both direct additive fusion and a plain sigmoid attention baseline, and the full method beats SAMAug-C on nearly all tested settings.
- Adding the projector head to the SAM feature branch yields a further accuracy gain, indicating that the segmentation encoder benefits from a task-specific readout.
Reading between the lines
- Inference: because the five extraction depths were selected on RetinaMNIST alone, the method would probably gain from per-dataset selection of extraction depths; a search over subsets on BreastMNIST and ISIC 2017 would show whether the fixed choice is the main driver of the reported margins.
- Inference: if SLCA's spatial-localization story is right, the same frozen-encoder fusion should transfer to tasks with small localized findings, such as chest X-ray pathology or histology tiles, and might extend to 3D imaging with a volumetric encoder; neither test appears in the paper.
- Inference: the comparison to SAMAug-C treats SAM as an augmentation source, whereas this method treats it as a feature source; a natural next experiment is to combine both uses and measure the marginal gain.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a framework that adapts the frozen image encoder of the Segment Anything Model (SAM) for medical image classification. Features are extracted from five layers of the SAM ViT backbone (patch embedding, the 1st/16th/32nd Transformer blocks, and the Conv block), passed through a proposed Spatially Localized Channel Attention (SLCA) module, and injected into the stages of a backbone CNN or Transformer via residual connections. A task-specific projector head further concatenates downsampled SAM features with the backbone's late feature map. Experiments on RetinaMNIST, BreastMNIST, and ISIC 2017 show accuracy and AUC improvements over the base backbones (ResNet152, SENet154, Swin Transformer v2) and over the SAMAug-C baseline, with larger gains at 10% training data. Ablations on RetinaMNIST evaluate the SLCA module, the projector head, alternative fusion strategies, and alternative choices of SAM layers.
Significance. If the reported gains are real and attributable to SAM's pretrained segmentation knowledge, then the paper offers a practical recipe for reusing a frozen segmentation foundation model for classification with modest extra parameters. The method is straightforward and the experimental setup covers three public datasets, three backbone families, and several data fractions, which is a reasonable scope for a first demonstration. However, the central mechanistic claim -- that the improvements come from SAM's segmentation-based representations -- is currently not isolated from the effect of simply adding trainable capacity. The paper itself reports that direct addition of SAM features degrades accuracy (Table III), and the SLCA plus projector head account for the entire improvement; without a control using a randomly initialized or non-segmentation pretrained encoder, the attribution is unsupported. The missing standard deviations and significance tests further weaken the quantitative claims, many of which are within 1-3 accuracy points. The paper is clearly written and the empirical exploration is useful, but these gaps are load-bearing for the paper's abstract-level claim.
major comments (3)
- [Section III, Table III and Sections II-A/II-C] The paper attributes the accuracy gains to SAM's pre-trained segmentation-based features, but no control isolates the contribution of SAM's pretrained representation from the additional trainable modules. In Table III, directly adding SAM features to SENet154 drops accuracy from 66.33 to 60.25; replacing SLCA with a plain sigmoid gives 66.75; the full SLCA plus projector head reaches 68.50. This pattern suggests the SLCA/projector machinery, not the segmentation semantics of the frozen encoder, may be responsible for most of the gain. The authors should run the identical pipeline with a randomly initialized ViT-h encoder (and, ideally, with an ImageNet-pretrained ViT that was not trained for segmentation) while keeping all other components fixed. If the gains persist, the claimed role of segmentation-based features is not supported; if they vanish, the claim is validated. This is a necessary experiment before the central conclusion can be accepted.
- [Section III, 'Experimental Results' and Tables I-II] The manuscript states that five runs with different seeds were performed and averages reported, but no standard deviations, confidence intervals, or significance tests are given anywhere. Many of the headline improvements are small (e.g., Table I, Swin Transformer v2 at 100% RetinaMNIST: 68.25 to 69.75; Table II, SENet154 at 10% ISIC 2017: 75.33 to 76.67), and the main data-efficiency claim rests on a 5.75-point gain for a single dataset-backbone combination. Without variance estimates, the reader cannot judge whether these differences are reproducible or within run-to-run noise. The authors should report per-run results or error bars for all entries, and perform a paired significance test (e.g., Wilcoxon signed-rank) for the key comparisons.
- [Section II-C, Table IV] The choice of SAM layers (patch embedding, 1st, 16th, 32nd Transformer blocks, and Conv block) is a fixed design decision that is validated only on RetinaMNIST (Table IV). It is not shown that this selection transfers to BreastMNIST and ISIC 2017, and the text does not clarify whether the selection was made using the test set or a hold-out validation set. If the layer choice was tuned on the same data used for evaluation, the reported numbers are optimistically biased. The authors should either provide the same block-choice ablation on the other two datasets or justify the transferability of the fixed choice, and they should state explicitly how the selection was made.
minor comments (6)
- [Section II-B] The text says 'while our SCLA aims to capture...' but the module is named SLCA; this is a typographical error.
- [Section II-C] The phrase 'we append the SAM image encoder with an additional task-specific projector head for fine-tuning' is ambiguous. Earlier the SAM encoder is described as frozen; clarify that the projector head is trainable while the SAM encoder remains frozen, and use consistent terminology throughout.
- [Section III, Ablation Study] The 'sigmoid' baseline in Table III is not precisely defined. Since SLCA ends with a sigmoid activation, the reader cannot tell what the 'sigmoid' variant actually changes (e.g., removing the spatial local average pooling, or applying sigmoid directly to the feature map). Please specify the exact operation.
- [Section III, Implementation Details] The paper claims SLCA is 'computationally efficient, adding minimal overhead' but reports no parameter counts, FLOPs, or runtime comparisons. A quantitative statement of the added cost would support this claim.
- [Section III, Experimental Results] The sentence 'The less the annotations used, the greater the performance improvements' is informal and imprecise; it should be clarified as 'the accuracy gains over the baseline are larger when fewer labeled training samples are used'.
- [General] No code or model weights are provided. Given that the method relies on a specific set of extraction layers and fusion hyperparameters, releasing the implementation would materially aid reproducibility.
Circularity Check
No significant circularity: the method is an empirical adaptation of a frozen public encoder evaluated on external benchmarks, and no prediction reduces by construction to a fitted input.
full rationale
The paper's central claim is that freezing SAM's public image encoder, extracting features from five fixed blocks, and fusing them via SLCA improves medical image classification accuracy on three external benchmarks. Nothing in the derivation chain equates the prediction to the input by construction. SAM is a publicly released pretrained model with frozen weights, and its features are not fitted to any of the target datasets or to the reported accuracies. SLCA is defined in Equation (1) purely as a function of the input feature map with no label-dependent term, so the attention weights cannot encode the classification outcome. The block choice (patch embedding, 1st, 16th, 32nd Transformer blocks, Conv block) is motivated by an ablation on RetinaMNIST and then transferred to other datasets; this may raise a selection-or-overfitting concern, but it is not circularity because the reported gains are empirical comparisons rather than consequences of how the blocks were defined. The only self-citation is SAMAug-C [13], which is used as a baseline that the proposed method is compared against and outperforms, so the citation is not load-bearing support for the central claim. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in through citation. Consequently, the derivation is self-contained with respect to the paper's inputs, and the appropriate score is 0.
Assumptions & free parameters
free parameters (3)
- SLCA channel reduction ratio r =
not reported
- SAM feature block selection =
patch embedding, 1st, 16th, 32nd Transformer blocks, Conv block
- Projector head output channels C =
C (e.g., 1024 or 2048) depends on DL model
assumptions (3)
- domain assumption Frozen SAM encoder features contain spatially meaningful segmentation information useful for classification
- domain assumption Features from patch embedding, 1st, 16th, 32nd Transformer blocks, and Conv block are complementary
- domain assumption SLCA's spatially localized average pooling followed by channel attention focuses on relevant regions
Cite this review
Pith. "Pith review of Adapting a Segmentation Foundation Model for Medical Image Classification." pith.science (2026). https://pith.science/paper/XEWAKQWP
@misc{pith2026250506217,
author = {Pith},
title = {Pith review of: Adapting a Segmentation Foundation Model for Medical Image Classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/XEWAKQWP}},
note = {Machine review of arXiv:2505.06217}
}
read the original abstract
Recent advancements in foundation models, such as the Segment Anything Model (SAM), have shown strong performance in various vision tasks, particularly image segmentation, due to their impressive zero-shot segmentation capabilities. However, effectively adapting such models for medical image classification is still a less explored topic. In this paper, we introduce a new framework to adapt SAM for medical image classification. First, we utilize the SAM image encoder as a feature extractor to capture segmentation-based features that convey important spatial and contextual details of the image, while freezing its weights to avoid unnecessary overhead during training. Next, we propose a novel Spatially Localized Channel Attention (SLCA) mechanism to compute spatially localized attention weights for the feature maps. The features extracted from SAM's image encoder are processed through SLCA to compute attention weights, which are then integrated into deep learning classification models to enhance their focus on spatially relevant or meaningful regions of the image, thus improving classification performance. Experimental results on three public medical image classification datasets demonstrate the effectiveness and data-efficiency of our approach.
Figures
Reference graph
Works this paper leans on
-
[1]
Deep residual learning for image recognition,
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE / CVF Computer Vision and Pattern Recognition, 2016, pp. 770–778
work page 2016
-
[2]
Squeeze-and-excitation networks,
J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” inPro- ceedings of the IEEE / CVF Computer Vision and Pattern Recognition, 2018, pp. 7132–7141
work page 2018
-
[3]
Swin Transformer v2: Scaling up capacity and resolution,
Z. Liu, H. Hu, Y . Lin, Z. Yao, Z. Xie, Y . Wei, J. Ning, Y . Cao, Z. Zhang, L. Donget al., “Swin Transformer v2: Scaling up capacity and resolution,” inProceedings of the IEEE / CVF Computer Vision and Pattern Recognition, 2022, pp. 12 009–12 019
work page 2022
-
[4]
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, P. Doll ´ar, and R. B. Girshick, “Segment anything,”arXiv preprint arXiv:2304.02643, 2023
arXiv 2023
-
[5]
R. Deng, C. Cui, Q. Liu, T. Yao, L. W. Remedios, S. Bao, B. A. Landman, L. E. Wheless, L. A. Coburn, K. T. Wilson, Y . Wang, S. Zhao, A. B. Fogo, H. Yang, Y . Tang, and Y . Huo, “Segment anything model (SAM) for digital pathology: Assess zero-shot segmentation on whole slide imaging,”arXiv preprint arXiv:2304.04155, 2023
arXiv 2023
-
[6]
C. Mattjie, L. V . de Moura, R. C. Ravazio, L. S. Kupssinsk ¨u, O. Parraga, M. M. Delucis, and R. C. Barros, “Exploring the zero-shot capabil- ities of the segment anything model (SAM) in 2D medical imaging: A comprehensive evaluation and practical guideline,”arXiv preprint arXiv:2305.00109, 2023
work page Pith review arXiv 2023
-
[7]
Segment anything model for medical images?
Y . Huang, X. Yang, L. Liu, H. Zhou, A. Chang, X. Zhou, R. Chen, J. Yu, J. Chen, C. Chen, H. C. Chi, X. Hu, D. Fan, F. Dong, and D. Ni, “Segment anything model for medical images?”arXiv preprint arXiv:2304.14660, 2023
arXiv 2023
-
[8]
Input augmentation with SAM: Boosting medical image segmentation with segmentation foundation model,
Y . Zhang, T. Zhou, P. Liang, and D. Z. Chen, “Input augmentation with SAM: Boosting medical image segmentation with segmentation foundation model,”arXiv preprint arXiv:2304.11332, 2023
arXiv 2023
Show all 25 references
-
[9]
SAMDSK: Combining segment anything model with domain-specific knowledge for semi-supervised learning in medical image segmentation,
Y . Zhang, T. Zhou, S. Wang, Y . Wu, P. Gu, and D. Z. Chen, “SAMDSK: Combining segment anything model with domain-specific knowledge for semi-supervised learning in medical image segmentation,”arXiv preprint arXiv:2308.13759, 2023
2023 arXiv
-
[10]
Segment anything in medical images,
J. Ma and B. Wang, “Segment anything in medical images,”arXiv preprint arXiv:2304.12306, 2023
2023 arXiv
-
[11]
Polyp-SAM: Transfer SAM for polyp segmentation,
Y . Li, M. Hu, and X. Yang, “Polyp-SAM: Transfer SAM for polyp segmentation,”arXiv preprint arXiv:2305.00293, 2023
2023 arXiv
-
[12]
Customized segment anything model for medical image segmentation,
K. Zhang and D. Liu, “Customized segment anything model for medical image segmentation,”arXiv preprint arXiv:2304.13785, 2023
2023 arXiv
-
[13]
Boosting medical image classification with segmentation foundation model,
P. Gu, Z. Zhao, H. Wang, Y . Peng, Y . Zhang, N. Sapkota, C. Wang, and D. Z. Chen, “Boosting medical image classification with segmentation foundation model,” inProceedings of IEEE International Symposium on Biomedical Imaging, 2024
2024
-
[14]
An image is worth 16x16 words: Transformers for image recognition at scale,
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,”arXiv preprint arXiv:2010.11929, 2020
2010 arXiv
-
[15]
Masked autoencoders are scalable vision learners,
K. He, X. Chen, S. Xie, Y . Li, P. Doll ´ar, and R. Girshick, “Masked autoencoders are scalable vision learners,” inProceedings of the IEEE / CVF Computer Vision and Pattern Recognition, 2022, pp. 16 000– 16 009
2022
-
[16]
Data-driven deep supervision for skin lesion classification,
S. Mishra, Y . Zhang, L. Zhang, T. Zhang, X. S. Hu, and D. Z. Chen, “Data-driven deep supervision for skin lesion classification,” in Proceedings of International Conference on Medical Image Computing and Computer Assisted Interventions, 2022, pp. 721–731
2022
-
[17]
IHCSurv: Effective immunohistochemistry priors for cancer survival analysis in gigapixel multi-stain whole slide images,
Y . Zhang, H. Chao, Z. Qiu, W. Liu, Y . Shen, N. Sapkota, P. Gu, D. Z. Chen, L. Lu, K. Yanet al., “IHCSurv: Effective immunohistochemistry priors for cancer survival analysis in gigapixel multi-stain whole slide images,” inProceedings of the International Conference on Medical...
2024
-
[18]
InTracker: An integrated detector-tracker framework for cell detection and tracking,
P. Liang, J. Chen, Y . Zhang, H. Wang, H. Zheng, P. Gu, and D. Chen, “InTracker: An integrated detector-tracker framework for cell detection and tracking,” inProceedings of the 2020 IEEE 33rd International Symposium on Computer-Based Medical Systems, 2020, pp. 332–337
2020
-
[19]
Path- GPTOmic: A balanced multi-modal learning framework for survival outcome prediction,
H. Wang, Y . Yang, Z. Zhao, P. Gu, N. Sapkota, and D. Z. Chen, “Path- GPTOmic: A balanced multi-modal learning framework for survival outcome prediction,” inProceedings of the 2024 IEEE International Symposium on Biomedical Imaging, 2024, pp. 1–5
2024
-
[20]
ECA-Net: Efficient channel attention for deep convolutional neural networks,
Q. Wang, B. Wu, P. Zhu, P. Li, W. Zuo, and Q. Hu, “ECA-Net: Efficient channel attention for deep convolutional neural networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11 534–11 542
2020
-
[21]
MedMNIST classification decathlon: A lightweight autoML benchmark for medical image analysis,
J. Yang, R. Shi, and B. Ni, “MedMNIST classification decathlon: A lightweight autoML benchmark for medical image analysis,” in Proceedings of IEEE International Symposium on Biomedical Imaging, 2021, pp. 191–195
2021
-
[22]
The 2nd diabetic retinopathy – grading and image quality estimation challenge,
D. Dataset, “The 2nd diabetic retinopathy – grading and image quality estimation challenge,” 2020
2020
-
[23]
Dataset of breast ultrasound images,
W. Al-Dhabyani, M. Gomaa, H. Khaled, and A. Fahmy, “Dataset of breast ultrasound images,”Data in Brief, vol. 28, p. 104863, 2020
2020
-
[24]
N. C. Codella, D. Gutman, M. E. Celebi, B. Helba, M. A. Marchetti, S. W. Dusza, A. Kalloo, K. Liopyris, N. Mishra, H. Kittler, and A. Halpern, “Skin lesion analysis toward melanoma detection: A chal- lenge at the 2017 International Symposium on Biomedical Imaging (ISBI), hoste...
2017
-
[25]
Decoupled weight decay regularization,
I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017
2017 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.